the 180gr. sounds like not big enough feathers if it's doing that
If you have to use more feather to get an arrow to fly straight you are wasting much of the bow's effort. That wagging back and forth as the fletching catches air is called "drag". Hence the bare shaft testing. When a bare shaft flys almost perfectly straight then you can get away with low fletching to impart a degree of helical spin without losing so much speed. Some guys claim that only matters when you are making long shots, but think of it this way: with a badly matched spine your arrow is flopping wildly just off the string as the fletching desperately tries to catch air and stablilize. What happens when the arrow is halfway sideways when it strikes your deer? Reduced penetration, deflection off ribs instead of shearing the rib, drastically increased chance of the arrow shaft smacking into a branch or other obstruction while in flight, and half a dozen other negatives.
Match the arrow to the bow, it's not that hard. Increases accuracy, too...but accurate arrows might take away an excuse for missing! I hate that!
How long are your shafts?
How long is your draw length?
How heavy is the point on the arrow?
And are you left handed or right handed?