Key, sounds like you have a hornets nest. Bees will swarm, and hang on a limb, but then go to a hollow tree, or rock crevice, ect. But you said that you have seen them going in and out of it. Sounds like a hornets nest. Yes definately be careful, they hurt like hell! And indeed if you are alergic, get someone else to get it for you. If you want to try to get it, wait till night, and take a friend and a light, and garbage bag, and a saw, and some twist ties. Shine the light on the nest, and then quickly slip the bag over the nest, and tie the twist tie around the top ( where it connects to the limb ), and then saw off the limb, to length that you want, and then take it home and throw it in the freezer. Next day you an shake out the hornets, and go fishing. It is really a neat nest. It is made in layers, or one on top of the other one, with space in between.
Hope this helps.
Stickbender