3 Replies Latest reply on Jul 22, 2014 2:57 PM by philmodjunk

    Preventing Duplicates - CSV Import

    RyanKivi

      Title

      Preventing Duplicates - CSV Import

      Post

           Greetings,

            

           I have two tables setup. one for my customers information and order number, and another for the items in the order.

           I am importing two .csv files and have the customer information setup to match on the order number. 

           I have the order details setup to match on the order number, and the specific line number in the order.

           I am getting duplicates upon import, and I'm very confused why this would be happening.  The CSV file that I am importing from always contains the entire set of orders that have not shipped.  I continue to over write the .csv file to capture any modifications to the order.

            

           Does anyone have any ideas of why I might be receiving duplicates?

        • 1. Re: Preventing Duplicates - CSV Import
          philmodjunk

               Duplicates in which table?

               I would guess that you have this structure in your CSV file from what you report:

               Customer A  Order Number 1    Item 1 data
               Customer A Order Number 1    ITem 2 data
               Customer A Order Number 1    Item 3 data
               Customer B Order Number 2    Item 1 data
               Customer B Order Number 2    Item 2 data
               and so forth....

               If so, when you import into your table of orders from this CSV data, you will get the Order and customer data imported into your table "for my customers information and order number", multiple times, one for each item ordered.

               If that's what you have, you can set up a "unique values, validate always" validation rule on the Order Number field in this table (not the other) and import your records from the CSV. This will then filter out the duplicates during the import.

               On the other hand:

               

                    The CSV file that I am importing from always contains the entire set of orders that have not shipped.  I continue to over write the .csv file to capture any modifications to the order.

               Suggests another issue. If you are importing the same data a second time, because it has not yet shipped, this will generate duplicated data in either table. You may be able to avoid this if you:

               a) show all records in the target table

               b) use the Update matching records in found set option to update existing records with the option to add new records if they don't match selected so that new rows of data produces new records.

               But this assumes that a match field or combination of match fields can be specified to get this update type of import to work...

          • 2. Re: Preventing Duplicates - CSV Import
            RyanKivi

                 I have two individual files exportedCustomers exportedOrders that I am importing from.  I do have match lines setup, and I forgot to mention that I do not ALWAYS get duplicate entries.  Its just occassionally.

                  

                 Your comment about showing all records is intriguing to me.  Do I have to add a step to show all records in my import script?  

            • 3. Re: Preventing Duplicates - CSV Import
              philmodjunk
                   

                        Do I have to add a step to show all records in my import script?

                   yes. Otherwise, there's a chance of getting duplicated data like you describe should the matching record you want to update not happen to be in the current found set of records in the specified target table occurrence.