The monkeyball fruit is a dead giveaway, but only the female trees bear fruit, and not every female tree is a heavy producer or makes fruit every year. I know of entire osage rows that never grow the fruit for whatever reasons.
Like Eric said, they're unique enough to be easy to recognize, with a little experience. After a while, ya just kinda know.
From a distance, with all the leaves off, they do give off a greenish yellow cast in the outer branches, relative to other nearby trees.
You could find some likely suspects, take some good pictures of the bark, branches, whole tree, and post them here.