So, Imagine a perfectly round bow, such that the cross section is perfectly circular at all points on the limb. Let's say this bow is also dead straight, and let's say it's made of something other than wood, like a tapered carbon tube. Just for giggles.
This bow can bend in any direction to take the string, right? Back, forth, to each side, basically, any direction you bend it, it goes. It's very hard to torque or twist when unstrung, and seems very stable. Except, once strung, the tips now are out of plane with the handle, and you could push one right and one left fairly easily. Not being co-planar provides them with a certain amount of leverage. Let's say now we want to reflex the bow. Baker says reflexing means that the tip moves more to take the string, aka bends the bow more (straightens from reflex to straight, then bends to the string), and thus provides a tighter string at brace height aka, higher draw weight early in the draw, which leads to higher total stored energy. So, good.
BUT, since the bow is equally stiff in in any direction, you basically COULD NOT POSSIBLY string a perfectly round bow against the direction of it's reflex. IT will simply roll, bend to the side, and take the string in the direction of it's reflex, not against it. Reflexing has again provided the tips with leverge that allows them to undo what you are trying to do. This is the instability that has been mentioned for bows with extreme reflex, and it persists to some degree even if we change the cross section.
OK, so let's take the same type of bow, make it bend to the same curve/tiller, but let's make it WIDER than it is thick, say twice as wide. Now, it can only bend easily back and forth, along its thinnest dimension, and it's much harder to bend it sideways toward it's greater dimension. BUT! This slat-like limb is much easier to twist when unstrung. It bends most easily back and forth, but once bent, pushing the tip from one side to the other is very easy. Also, near the tips, the ratio of thickness to width cannot practically be maintained or the bow will end entirely in the middle. Drat.
So, regardless of, and at the same time depending on, limb design, the more reflex you have, the more of these instability issues you will have. There may be a happy medium, but a limb with more thickness to width will have a rolling problem, and a wider, thinner limb will have a twisting problem. In addition....
Let's say we hit that happy medium, the limb is thinner than wide, but not twisting badly, and very reflexed. Now we have tillering issues. If the reflex comes out strongly from the handle, then we strain the wood mightily just getting back to brace, top and bottom limbs bulging forward then back. If every part of the limb bends the same amount, then later in the draw, the very rounded, bellied curve of the limbs gives very poor leverage from the string, almost like two very short bows attached to a set forward grip. So, the bow stacks. BY the time you tiller it out to 50 lbs what should be good early energy storage has disappeared and the F/D curve sucks. You've tillered what amounts to a good energy storing bow of lower poundage, that then spikes in draw weight at the end.
BUT! WE can avoid this allowing the limbs to bend aggressively near the handle! Right? Because a little bend here translate to a lot of tip movement! Yeah, so the outer limbs can be stiffer, maintaining good leverage out into the draw.......except.......that same principle means that the stiff limbs exert high leverage on those bending inner limbs, so your string tension at brace looks great, but there isn't enough working limb, so you have this long stiff lever, and not enough inner limb bearing the strain. So, you also end up with a LOT of set in the inner limb where it hurts you most. This bow now may look like a straight limb bow with a setback handle, and by the time you get the limbs around, you still have the same issues you did above, though less.
Finally, let's say most of the reflex is toward the tips. Here you may have plenty of working limb, but the abrupt recurves at the tips again provide "handle" that want to torque and invert the limb if allowed to. You rarely see an unstrung bow straight through the handle and straight through most of the limbs with large recurves, because the recurves act like big handles to twist the string off center and generally jack the limbs, AND, since the limbs are getting narrower toward the tips, their ability to bend off to the side is increased. Big recurves or big abrupt reflex requires more mass to keep stable, and that's a problem, too.
If I add big reflex or recurves to anything, it's deflexed first. Properly executed a reflexed stave can be a very fast bow, but thrade-offs add up.