Thank you for your post.
Using FileMaker Pro 11.0v4, on a Mac with 3 GB of RAM, running OS X 10.6.8, I am able to print records to PDF and PostScript without memory errors. The file cache is set to 64 MB.
To help identify the source of the error you are getting, I'll need some more details.
Have you been able to test on any other computer? Is it possible to print records without the "placed art" using the File / Print / Print to PDF (PostScript) from this solution? What else can you tell me about the way you are using the images in your file?
Creating a brand new, small, from scratch test file and checking to see if it has the same problem would also be useful. If the test file works fine, then perhaps there is a problem with your file.