I just upgraded to Apple's 64-bit Java 1.6 on OS X. I can consistently cause a heap overflow on parsing if I do the following:
$FORTRESS_HOME/ProjectFortress/ant cleanCache
$FORTRESS_HOME/bin/fortress [anything here]
The interesting workaround is to ant test after cleaning the cache. Sadly, this is not always an acceptable solution when you're in a compile-edit-run-debug cycle and have a known broken program you're trying to turn around fast.
The fact that we have heap overflows during parsing at all is a big red flag. I'm baffled; if it were a stack overflow I'd guess a right recursion in the grammar where there ought to be a * instead (in general * is preferred to recursion in any PEG). Especially suspicious is the fact that this only occurs if the libraries are unparsed. It looks like we're not traversing the libraries in a sensible order during pre-parse. I'd love it if someone who understood this code would have a look and fix it. I'm guessing Janus is the obvious candidate, but Sukyoung or David may be able to shed light.
Most of our user base is unlikely to run ant test after every cache clean (and they shouldn't have to).