After about 15 years of osage dust exposure I became allergic to the stuff, I was already allergic to walnut dust. At first I would get slight hay fever symptoms, this progressed into full blown watery eyes and stopped up nose and sinuses over the years. It finally got so bad I have to wear a cartridge respirator if I work on bows, naturally this apparatus takes a lot of the enjoyment out of wading into an osage stave as it is hot and cumbersome. The paper dust masks just don't work for me, too much gets through them.
I have a complete dust collection system in my shop but it still lets the very fine stuff float around. I sand osage in front of a dust pickup port but still get enough without my mask to make me sick.
Almost everyone I know that is around dust year after year has some problems.
Protect yourself from the get go and you should be OK.
You can see my work station in this picture;