Is there a North American hardwood that handles compression well? Something I could have paired up with hickory? I chose hickory backing because I knew the bow was going to have a pretty big bend to it.
That is a good way of thinking, but not only must the backing be able to take the bend, the belly must also be able to withstand the compression. Red oak is pretty poor in compression, so the powerful hickory is likely to overpower the belly with such a sharp bend.
You could have trapped the back, making it about 2/3 the width of the red oak belly. That would have helped a lot. A longer bow length is also a good method, but that is probably not what you wanted.
Hickory on its own would have been great for a design such as this one. It might have gotten some set, (but no chrysals) which could have been avoided by - again - trapping the back.
Elm would have been good as well (no hickory needed on the back). And of course hickory backed osage or maybe even hickory backed hornbeam.