First, for Asia font, the font file is usually bigger than 1M, for example Song Ti of Chinese Simplified is about 10M, and the FOP loads the font file as Array and access the font data directly. As the font data is appended to the PDF in the end. If the document is large and processing the PDF will eat up most of the memory, and there are lots of small memory fragment in the physical memory and it will be inefficient to allocate a large memory block such as 10M block. And OutOfMemoryError will be thrown at this point before the GC can do anything.
See org.apache.fop.fonts.FontFileReader

Second, if the PDF document contains lots of images and the images' dimension is large, allocating memory for the image will throw OutOfMemoryError when there are lots of small memory fragments occupy the physical memory. As one of my project will generate image about 1000x1000, and in the FOP 0.20.5 about 4x1000x1000 + 3x1000x1000, nearly 7M is needed for the image, and there must be 4M big block under fear memory circumstance. Unfortunately, it always throws OutOfMemoryError with VM arguments "-Xms64m -Xmx128m".
See org.apache.fop.image.JimiImage and org.apache.fop.image.FopImageConsumer