I grabbed some palm sized sandstone rocks on the Lake Superior seashore last year. I tried boring holes so I could sand down shafts. I found that you do have be careful if you bore straight through with a steel bit, sometimes they blow out the material at the back. I bored holes that were 3/8" in size with the idea that I could use it to size the shafts. I found that didn't work so good when one of the shafts got stuck in the hole. The sandstone didn't clear the sawdust fast enough and the hole filled up. Had to break the shaft to get the hole cleared out.
Shallow grooves seem to work best for sanding. I'm thinking that maybe limestone might work better, but I haven't tried that yet.