Do you have advanced? Did you try using the Database Design Report? There are also 3rd party apps that do this even better.
Base elements is the 3rd party tool that I use to find unused fields (and layouts and scripts and such). Its very comprehensive. If something is not used or referenced, it will count that and flag it for your viewing.
There is a license to use their system. But it is well worth it. I'm looking at a BE almost every day.
Try that out.
I think you hit the jackpot with me on this issue. I'd never heard of BaseElements. I'm in the process of getting the 30-day trial period and I'll likely find that this product will solve all my issues.