You've stubbed your two on the main draw back to this method. Portal filters have to evaluate on every related record to control which appear in the portal. If you have a large number of related records (the records that would show if there were no filter), then it will take long time to update the portals with each keystroke.
You could modify the design so that the update doesn't take place keystroke by keystroke by adding a button to update the portal after entering your search text.
You might see if you can use some additional control to limit the records listed to just those that are members of a specific category. This field can be a match field that makes up part of the relationship.