It's not a big deal at all... that is, if you keep the sights lined-up. For example, if the sights are in line, but you continually drift on and off target, it is really a very minor change in the point of impact (the impact only moves as much as your hands, as the sights are still staying aligned).
However, if the gun pivots (you actually turn the gun, taking the sights out of alignment), it is a much bigger problem. Rather than the sights shifting around the target in unison, your are turning the entire sight axis and you can think of it like standing in the middle of a pic... a little turn at the gun equates to a very wide arch at the point of impact.
So, work on site alignment. You will also (hopefully) learn of a thing called "natural point of aim". While it is very important in prone rifle shooting, it also applies in handgun shooting while standing. The mechanics are the same.
Now, with all that said, at most tactical distances, slight movement is not necessary. Outside of bragging points, there is no benefit to putting the rounds in a hole the size of a silver dollar. All that time you spend on getting a tight group, your threat will be lighting you up!
IMO, a good compromise of speed and accuracy is an 8" circle.
We bring evil things to evil people, kicking in a door near you!
."In theory, there is no difference between theory and practice. But,
in practice, there is."
- Jan L.A. van de Snepscheut
"The difference between 'involvement' and 'commitment' is like
an eggs-and-ham breakfast: the chicken was 'involved' - the pig
was'committed'."
-unknown
Working on a PhD in CQB one doorway at a time.
When the wolf attacks, he will find not all who run with the flock are sheep!