Hello. I need guidance on what's the best way to perform the following:
I have over 240k records with a URL and three categories fields.Many URL's are duplicated but the categories change. Some of those duplicated records have one category, or two, others have three. Some categories are repeated, others aren't. The category fields are cat,cat2 and cat3. I want to gather all categories from the same URL, delete all duplicated categories as well as duplicated URL's. Created a script that will import one category per record along with its URL into an "auxiliary table"(does not take long to do it). So, these 240K initial records could become 600K once they are all imported into the auxiliary table; then, another script sorts the resulting record by URL and category fields, starts analyzing duplicated categories and deleting them all but one, potentially leaving 1 or more records with the same URL and a different category. Last, a portal will show on the first table all related categories (which I could gather them into one field, but that's not a problem).The scripts work.
The problem is that it is taking more than 4 hours to do the second script, and if I were to import more URL's, then it will take more time to script the new URL's. Can anyone guide me on a different procedure to do this, please? Thank you in advance