You can set up an accounting system in FileMaker Pro but it is very complex and unless you are a professional developer (in which case you would not be asking the question), you should not attempt it.
I would suggest that you look at existing FileMaker Pro accounting systems for purchase if you want to go down that path. For example, Absolute Accounting. The expense of that solution gives you an idea of the development hours required.
Depending on your hardware and OSX version, you might investigate buying VMWare, a Windows XP Pro license and Quickbooks Pro for Win.
Filemaker comes with both OSX and Win installers on the CD.
There is a company that sells a Quickbooks to Filemaker plugin.
This is an option that some of my clients use.
I have a small business with very particular requirements and there isn't a software package out there that meets these. To answer your question simply: yes, it's possible to develop complex financial software with FM Pro Advanced. However, a caveat by example: I started developing this solution over two years ago. Although it nearly immediately did basic tasks, I'm STILL working on it! It does what I need now. Whether you're a pro or not, be prepared to accurately identify your needs and make a large time investment.
Had there been an existing alternative I believe I would have used it. Of course then I'd have missed all the fun . . . .