Tim, I see smoke as a nature neutral scent and not alarming to animals...but that depends on the source of the smoke. If it is from a controlled fire with natural, local materials like wood and grasses it should not necessarily startle animals but if you use the common campfire smoke you are probably getting beer and soda cans, plastic bottles, plates and cups, uneaten food, etc mixed in and this can be alarming to animals that are being hunted.
That said, I think you hit the nail on the head when you said you hunt into the wind and go into the woods clean. Cover scents make us, the hunter, seem invisible but in fact, no matter what or how much cover scent one uses, if your scent stream hits an animal's nose, the game is over. The same goes with commercial camo. Is it necessary? Is it the pattern you wear or your actions(or inaction's) in the woods that makes you invisible?