Notifications
Clear all
Topic starter
Describe the problem/error/question
I am currently scraping TikTok feeds on a daily basis. My goal is to check my Airtable database with the scraped songs and videos. If a scraped UID already exists in my database, I want to update that record. If the UID does not exist, I want to create a new record for both the scraped video and its associated song.
I attempted to use the Airtable filter formula within the list module, but I was unable to access the list of videos that did not meet the filter criteria. This prevents me from creating new entries for videos and songs that are not already in my database.
Additionally, I couldn't get the automation to process all 30-35 videos without using the split in batches module. Is this the correct approach?
I previously had this functionality working in make, but I'm struggling to identify what I'm doing incorrectly here. Any help would be greatly appreciated!
Please share your workflow
Share the output returned by the last node
Information on your n8n setup
- n8n version: Latest cloud
- Database (default: SQLite): Default
- n8n EXECUTIONS_PROCESS setting (default: own, main): Default
- Running n8n via (Docker, npm, n8n cloud, desktop app): n8n cloud
- Operating system: windows 11 pro
Posted : 07/09/2023 4:16 am
Hi @Zac_Magee
For your information, Airtable API makes it possible to “upsert”, that is to update or create if it does not exists
This option is not implemented in n8n Airtable node, so you’ll have to make direct API calls for example:
Maybe this can help
Posted : 07/09/2023 7:11 am
Topic starter
Thank you for your response!
I wanted to add that you don't need to use an API call for the upsert functionality; it's available within the Airtable node's settings.
I've never fully grasped how an upsert works, even in callin.io. Am I correct in assuming it searches for existing records? For instance, if I have 15 videos already in the database and 15 that are not, it will check the parameter I specify. If that ID doesn't exist, it will create a new record in the database.
Is there no way to have both existing and non-existing items pass through the node so I can handle both datasets later in the workflow?
I've updated to use the "Upsert" operation, but it's only creating duplicate records, regardless of the matching field I select.
In callin.io, the workflow functions as illustrated below.
Posted : 07/09/2023 8:10 pm
Topic starter
After numerous attempts, I've successfully implemented this.
Solution:
I utilized the "Upsert" option and employed the UNIX timestamp as the reference point to determine if a record already existed. I'm still exploring methods to use the Airtable ID as a reference, but for the time being, this approach is functional.
Posted : 08/09/2023 2:06 am
Great point!
Why didn't you opt for the Unique ID that you previously attempted to use in your listing?
Also, performing an upsert is significantly more efficient, as it allows you to update or insert 10 records in a single call.
This is particularly valuable in a scenario where Airtable API calls now have a strict limit on the "team plan".
Posted : 08/09/2023 7:24 am
Topic starter
I encountered issues running it smoothly when using the ID. I've since implemented the set node in several places to reduce the data transferred within the workflow, and the Unix timestamp appears to be the most effective method for me to match.
I concur; even on callin.io, that was the reason I transitioned to callin.io. The expenses escalated rapidly due to performing numerous actions that could have been batched together.
Posted : 08/09/2023 10:34 pm
This thread was automatically closed 7 days after the last response. No further replies can be made.
Posted : 15/09/2023 10:35 pm