Yeah, accuracy definitely improves once you don't have to focus on drawing and holding the bow. A 50 pound bow is much easier to get used to than an 80 lb bow though, and the muscle mass needed to pull a 50 lber will stick around a lot longer when it isn't being used than the additional muscle mass needed to draw the 80. And you're at much less of a risk of injury when drawing a 50 lb bow cold than an 80 lb, and muscles are almost always cold when you are hunting.
I'm not advocating against heavy bows per se, I just hate seeing people use bows they can't shoot accurately. When a bowhunter shows up and can't consistently hit a target, it makes bowhunting look bad. People don't always think to ask about the weight of the bow or how long the hunter's been shooting that weight. All they see is a bowhunter who can't hit the mark.
All the best,
-Eric