Ty,
I am like Fred, I don't have any experience with tri-lams, but I think your plan is sound. I have only made one bow that was highly reflexed--a sinew backed juniper bow--but I had similar issues bracing it. I think that is pretty common for highly reflexed bows, and it doesn't necessarily mean you are way out of tiller, it means it wasn't ready to brace--I think. Either that, or the glue line breaking exposed/caused a weakness that led to the unbalanced brace. Having a gradual transition at fade--keeping every thing smooth as you work through the tillering, should help, as will having a perfect glue line when you reattach the riser/handle will also be very important.
Good luck, and hang in there. This is a challenging project, but I think you are going to succeed, and you will have a bow to be proud of.