We know nothing about your solution, so we can't really guess why your late fee is not compounded each month.
My guess is that it's calculated instead of transactional. That means that it only ever shows the "current" late fee, rather than saving it along with an invoice each time a new one is created. Invoices should be transactional so that they don't change via calculations once they are generated.
If you need help specifically without sharing your solution file here, you should look for a consultant that can mentor and assist you outside of this public forum.