On my finer rasps, I have been known to rub them with school chalk before using in order to make 'em easier to clean out the clogging. When they load up, hit 'em with a brass brush, and re-apply the chalk.
I also try to avoid laying my rasps on top of each other. Great way to dull them! I keep old computer mouse pads on the work bench for them.