Notifications
Clear all
Topic starter
Hello Everyone!
I'm looking to send a series of HTTP requests sequentially and save the outcomes to Airtable on a daily basis. The HTTP request I'm making targets a website structured like this: www.testwebsite.com/id=1000. My goal is to send requests for IDs ranging from 1000 to 1050 (as an example), then store the results from these calls. The following day, I plan to execute requests for IDs from 1050 to 1100, and so forth.
I'm having difficulty determining the most effective method to accomplish this.
Any assistance would be greatly appreciated. Thanks in advance.
Posted : 12/01/2022 2:52 pm
Hi Spacewalker, I need a bit more information to assist you, but I believe I can guide you in the right direction.
To clarify your objectives, you are querying a website and need to iterate through IDs, which you are currently doing sequentially. I'm trying to understand if your process involves:
Call 1: www.testwebsite.com/id=1000
Call 2: www.testwebsite.com/id=1001
Call 3: www.testwebsite.com/id=1002
etc.
Call 50: www.testwebsite.com/id=1050
And then the workflow concludes? I'm unsure how you would track progress from day to day, apart from setting a function variable to determine the new starting number. However, your flow should resemble the following, with an initial step to set the function to end at a number other than 51.
Posted : 12/01/2022 5:14 pm
Topic starter
Hi there, thanks a lot for your assistance.
What I'm aiming to do is retrieve data from a website and then store the outcomes in Airtable.
Given that the website is structured somewhat like a database, where each article is housed on its own webpage and the links to these pages are sequential:
www.test.com/id=1200
www.test.com/id=1201
www.test.com/id=1202
www.test.com/id=1204
.
.
.
www.test.com/id=1250
I need to generate 50 URLs at a time automatically and then dispatch those 50 HTTP requests. The requests do not need to be in sequence; in fact, I can pass all 50 URLs together to the HTTP node, send the requests, and save the results.
So, ideally, the workflow would look like this:
- Identify the last ID from the previous iteration (e.g., 1250).
- Generate the next 50 URLs:
www.test.com/id=1251
.
.
www.test.com/id=1300 - Pass these links to the HTTP node.
- Send the requests and capture the responses.
- Store the retrieved data in Airtable.
- Persist the last used ID somewhere so that the next iteration can commence from that point.
I hope this provides a clearer picture, and I greatly appreciate your help.
Posted : 13/01/2022 11:27 am
Yes, that's understandable. Is it feasible to retrieve those IDs from Airtable for individual processing? For instance, if Airtable returns:
www.test.com/id=1200
www.test.com/id=1201
www.test.com/id=1202
www.test.com/id=1204
Then, it would be straightforward to use a split in batches node to break these links into a loop. You could then use the HTTP request node to call on save to Airtable and proceed to the next item.
My main question is whether you have these IDs within Airtable and are retrieving them into your workflow, or if you expect the workflow to generate these IDs. In my experience, it's more efficient if the links are already present in a spreadsheet or another database. This allows you to gather all the IDs and then feed them into your loop to extract data from the page.
Please excuse my unfamiliarity with Airtable; I primarily use Trello, so I'm not entirely sure about retrieving data from Airtable or if it's possible to generate those IDs from it.
Posted : 13/01/2022 4:55 pm