Skip to content
How to split an HTT...
 
Notifications
Clear all

How to split an HTTP GET request into batches based on bundle availability

8 Posts
3 Users
0 Reactions
3 Views
Mars1
(@mars1)
Posts: 4
Active Member
Topic starter
 

The title might be a bit much, apologies.

My scenario is quite straightforward:

HTTP GET (from my database)

Add New Row in Google Sheet for each bundle.

This is currently working fine using the Iterator function.

However, the current challenge is that I need to limit the number of bundles the GET command retrieves. This is because my database contains so many records that callin.io returns an error stating the response exceeds the maximum file size.

I believe I can use a Repeater Function along with some sort of mathematical function or increment to achieve the following:

GET total record number

Divide record number into four segments (or similar)

GET records 1 to [first quarter of total]
→ ADD to Google Sheets

GET records [first quarter +1] to [second quarter of total]
→ add to google sheets

GET records [second quarter +1] to [third quarter of total]

And so forth.

Can anyone assist me with this?

 
Posted : 28/08/2024 4:05 am
LinkYourTech
(@linkyourtech)
Posts: 15
Active Member
 

A Bulk Add Rows module is now available in callin.io for Google Sheets.

 
Posted : 28/08/2024 4:46 am
Mars1
(@mars1)
Posts: 4
Active Member
Topic starter
 

My problem is that the initial HTTP GET request is too large for callin.io to retrieve.

 
Posted : 28/08/2024 5:08 am
samliew
(@samliew)
Posts: 293
Reputable Member
 

Welcome to the callin.io community!

You have asked a duplicate question of

If you need further assistance, please provide the following:

1. Relevant Screenshots

Please share screenshots of your scenario, any error messages, relevant module fields, and filters in question? It would really help other community members to see what you’re looking at.

You can upload images here using the Upload icon in the text editor:

2. Scenario Blueprint

Please export the scenario blueprint file to allow others to view the mapped variables in the module fields. At the bottom of the scenario editor, you can click on the three dots to find the Export Blueprint menu item.

3. Output Bundles of Modules

Please provide the output bundles of the modules by running the scenario (or get from the scenario History tab), then click the white speech bubble on the top-right of each module and select “Download input/output bundles”.

A. Upload as Text File

Save each bundle contents in your text editor as a bundle.txt file, and upload it here into this discussion thread.

B. Insert as Formatted Code Block

If you are unable to upload files on this forum, alternatively you can paste the formatted bundles.
These are the two ways to format text so that it won’t be modified by the forum:

  • Method 1: Type code block manually

    Add three backticks ``` before and after the content/bundle, like this:

    ```
    content goes here
    ```

  • Method 2. Highlight and click the format button in the editor

Providing the input/output bundles will allow others to replicate what is going on in the scenario even if they do not use the external service.

Following these steps will allow others to assist you here. Thanks!

Hope this helps! Let me know if there are any further questions or issues.


P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the callin.io Academy. Investing some effort into it will save you lots of time and frustration using callin.io later!

 
Posted : 28/08/2024 10:42 am
LinkYourTech
(@linkyourtech)
Posts: 15
Active Member
 

There's a very helpful learning module on the callin.io academy regarding pagination. You might want to check that out; it could assist in setting up a solution.

However, are you attempting to input old data into the Google Sheet, or is this new incoming data? If it's new data, consider setting up a webhook or reducing the time intervals between scenario executions.

 
Posted : 28/08/2024 3:06 pm
Mars1
(@mars1)
Posts: 4
Active Member
Topic starter
 

Thanks for those helpful links. However, I believe both of those assume that the API call can be retrieved in its entirety initially, and that subsequent actions fail due to the volume of bundles within that API call.

In my case, the very first action, which is the API call itself, is failing because of the file size. My goal is to retrieve the entire database, perform some filtering on it, and then send the filtered data to Google Sheets as new rows.

My initial approach was to make an API call to retrieve just the "Total Record Count." Following that, I planned to make subsequent API calls, dividing the data into chunks, perhaps by "Total Record Count"/4. I've actually managed to get this working, but it does feel a bit complex.

 
Posted : 28/08/2024 11:38 pm
samliew
(@samliew)
Posts: 293
Reputable Member
 

You will still need a Repeater module.

Screenshot_2024-08-29_080831

Hope this helps! Let me know if there are any further questions or issues.


P.S.: Did you know, the concepts of about 70% of questions asked on this forum are already covered in the callin.io Academy. Investing some effort into it will save you lots of time and frustration using callin.io later!

 
Posted : 29/08/2024 12:00 am
Mars1
(@mars1)
Posts: 4
Active Member
Topic starter
 

Thanks! I will give that a look.

 
Posted : 29/08/2024 3:20 am
Share: