It's quite simple really! I'm surprised nobody has mentioned it.
Yes your tiller was incorrect for the bow design before and you still need to work on a proper elliptical tiller. The reason being that any wood at any given thickness can only bend so far before taking set when released. Make it thinner - it can bend further before taking set. Make it thicker and it can now bend less before taking set. Your bow has a thickness taper - any bow that tapers in thickness along its length MUST bend progressively further as it gets thinner. This concept is absolutely key and especially so when you start using woods that can't shrug off tiller mistakes like osage etc....
The 'problem' with red oak and why it takes set like it does is that it is much stronger in tension than it is in compression. Think of tension and compression properties as two weights on a seesaw - you want the weights equal for both to get a fun ride! Same with a bow - when it is bent the wood gets stretched and it gets compressed - the kicker being that most woods are stronger in tension than compression. So the stronger (actually it isn't 'strength' it is resistance to bending) tension side throws excess stress onto the compression side. As the resistance to compression isn't as great as its resistance to being stretched. Therefore you get set. Set is overstrained bellywood cells that have been crushed past their elastic limit and can therefore no longer spring back.
the way to get around this mismatch of properties is to make the working surfaces proportional to the particular woods set of properties. You can do a simple bend test before starting to work out how your wood behaves and use the information accordingly.
Red oak can make fast, durable, low set bows.
I bought a couple of boards of it years ago and made a whole bunch of pyramid bows and gave them to my friends. They are all still shooting very well. Don't be down on red oak! Just learn how to use its inbuilt properites to your advantage
I made the back 60% of the width of the belly.