There are several ways to display your images in FileMaker. For simple interface elements like buttons I simply copy from photoshop and paste into FileMaker. For more complex images this tends to leave the image artifacts you are referring to. You can also use a container field. In the field set up you can choose to store the actual file, and not a reference to the file. This works good if you want to make your interface elements easily changeable by a end user.
My favorite method is to actually base64 encode your jpgs and send them to a web viewer using a data url. Simply google "filemaker data url" to find more information on this system. The advantage to using this is filemaker uses the underlying OS's web rendering engine to display the image. This means your photos will look exactly like they do in the stock web browser on that operating system.