Pipelines - Delete unmatched records

Question

I've been using pipelines to replace some synced tables that I need to be updated "live".  Basically two nearly identical tables with the same primary key. 
I've created a pipeline for when a record is created in table A to create a record in table B.
I've created a pipeline for when a record is modified in table A to update the record in table B.
I've created a pipeline for when a record is deleted in table A to delete the record in table B.
Now i'm working on a daily refresh pipeline to ensure the tables are synced in case of errors.  The bulk upsert process is what I have been using for that, and it works great.  My problem is I don't understand how to include the logic to delete records out of table B that no longer exist in table A as part of my daily refresh.  Is there a way to do this within my daily refresh pipeline or another pipeline?  ------------------------------Drew Voge------------------------------

ChayceDuncan · Answer

Do you have some way of binding the two together that you can match between? Such as - when you create a record in Table B from Table A, are you logging the Record ID# in Table B so you can match them back and forth?

If you are - you could run a daily process that takes ALL the records in Table A and imports the current date into table B as a 'Date Last Seen from Table A', and then search for all the records in Table B that don't show as 'Today'. The assumption there is that if you don't update a value in Table B as having been seen today then it must have been deleted from Table A.

------------------------------
Chayce Duncan
------------------------------

Sign up

Login to the community