Forum Discussion

PaulPeterson1's avatar
PaulPeterson1
Qrew Assistant Captain
4 years ago

Remove Duplicate Records

This question is a tribute to the law of unintended consequences.  There is a webhook in table 1 that will create a new record in a table 2 in a different app when a new record is added to the  to table 1 and a checkbox is checked.  That had been working ok for the new records that were initially imported with the checkbox checked.  However, the source data for the import is not always accurate and the checkbox is not checked.   The webhook was modified to fire when a record was added or modified and there a check was added to the checkbox.  That seemed to make sense, but it created duplicate records every day when the data was uploaded (I should mention, table 1 currently has a manual update every morning that includes all previous records plus any new records that were added the previous day).  After two weeks, there are over 125k duplicate records.  Is there way to remove the duplicate records without resorting to brute force manual deletion of the duplicates?

------------------------------
Paul Peterson
------------------------------

4 Replies

  • 125,000 duplicates OMG!

    Can you run a summary report on whatever makes these unique and export to excel.  Delete all the records and then import the summary report?

    ------------------------------
    Mark Shnier (YQC)
    Quick Base Solution Provider
    Your Quick Base Coach
    http://QuickBaseCoach.com
    mark.shnier@gmail.com
    ------------------------------
    • PaulPeterson1's avatar
      PaulPeterson1
      Qrew Assistant Captain
      That brings up the other issue I forgot to include; your suggestion was out first thought.  The problem with that idea is the fact that the table is too large to export.  I'm thinking I might have to break it into smaller chunks.  The data before the problem started can be exported in one file and the duplicates into a couple other files.  Then re-import the files....ugh

      ------------------------------
      Paul Peterson
      ------------------------------
      • MarkShnier__You's avatar
        MarkShnier__You
        Qrew Champion
        yes, you can export in chunks.  Just divide it up by Record ID or Alpha.

        ------------------------------
        Mark Shnier (YQC)
        Quick Base Solution Provider
        Your Quick Base Coach
        http://QuickBaseCoach.com
        mark.shnier@gmail.com
        ------------------------------