I like to temper when i get it to brace. For me that saves retillering after it's done, and you shouldn't have a lot of material to remove to finish tiller. There's no one right way, think about you're own abilities and experience and try something. One benefit of tempering toward the end as sleek mentions is a greater certainty of making weight, provided you're within 5# or so of desired weight before you temper.
All that said: with elm, best bet is at least do some tempering early, you can temper again at the end if desired. As mentioned, this will make elm less set-prone.