Tom,
I might pass a fawn because I didn't think the small amount of meat was worth the work, because I wanted to let a button buck grow up or because I didn't want to use up a tag if they weren't giving out many. There's no ethics issue. In fact, the farmer in Illinois that let me hunt asked me to shoot him a fawn every season. Best eating on the planet. However, I applaud you making a management decision and sticking with it.
George