We study the dynamics of Staffman-Taylor fingering in terms of topological defects of the flow field. The defects are created and/or annihilated at the interface. The route towards the single-finger steady state is characterized by a detailed mechanism for defect annihilation. For small viscosity contrast this mechanism is impeded, and creation of new defects leads the system away from a single-finger solution. Strong evidence for a drastic reduction of the basin of attraction of the Saffman-Taylor finger is presented.