This is funny
You are worried about the stem
being bad for you but you intend
to smoke tobacco??
Zuma
You cannot compare long term, occasional smoking tobacco with the effect of sudden poisoning caused by hot smoke coming through poisonous wood.
If you for example were to use elderberry wood, which sounds reasonable at first, because it has natural pith in middle, you would get cyanide poisoning.
It seems that bamboo is a good choice for stem. As its not available here as I want (otherwise I would make own arrows), I'll still look around.