Apologies for not replying sooner, but a forum glitch was locking out all responses to your question.
I have a database file here on my computer where I can insert files into containers and can then extract the filenames--using FileMaker 10 or 11 so there's no bug here.
The key detail is how was the file originally inserted?
Files can be inserted with Insert Picture, Insert Object (windows only), Insert File or they can be pasted, even dragged and dropped. A physical copy of the file can be stored in the container field or a reference to the file.
The method you describe for extracting the file name works but only if the store by reference option was used and you then have to pay attention to whether insert file or insert picture was used to insert the file. When you select either of these options, there's a check box in the dialog that allows you to choose store by reference when inserting the file. Select this option and you'll now be able to extract the filename, just be careful to select the correct line of text to get that file name--which can differ for inserted files and inserted pictures.
Note: if you have a folder filled with pictures you want to import into your database, you can use Import Records | Folder to import them and then there's an option you can select to import the file name into a separate field at the same time you import the picture.