Rate limiting webhooks to execute only once every 5 minutes

Hi Karl,

I'm not certain if a more optimal solution exists, but I'd recommend utilizing a custom variable as a filter (assuming you have the pro plan or a higher tier).

Let's name it $scenarioA_triggered and initialize it to false.

Set up two scenarios. The first should function as the trigger, incorporating a filter and a webhook (or a callin.io module) to initiate the primary scenario, which contains the core logic.

Within your trigger scenario, use $scenarioA_triggered equals false as the filter for sending the webhook to activate the main scenario.

In the main scenario, set $scenarioA_triggered to true during execution and then revert it back to false once completed. You can incorporate delays here as needed to manage the waiting period. You also have the flexibility to add waits before or after your logic execution. The placement is entirely up to your discretion.

An alternative approach would be to create a third scenario solely dedicated to managing the timer for the filter variable. This might be more complex than necessary, but it would offer greater control over the timer, especially if your main scenario has a lengthy or unpredictable execution time.

Hope this proves helpful!

EDIT:

Given your use of Airtable, the "Watch Records" module should inherently address this to some extent. It prevents duplicate records from being returned, so if your record operates on a 5-minute schedule, you should not encounter any issues. Hope this explanation is clear.

Hi

Snippet from the documentation which might answer your question

Source: https://www.callin.io/en/help/tools/webhooks#scheduling-webhooks-processing

Scheduling webhooks processing

By default, when callin.io receives data on a webhook, your scenario executes immediately. If you don’t want to run your scenario immediately after a webhook receives data, you can schedule your scenario to process all webhook requests periodically.

1. Edit the scenario which is triggered by your webhook.
2. Edit the scenario schedule settings. OR Edit the schedule settings of the webhook module.
3. Set up your desired schedule.

When a scheduled webhook receives data, callin.io stores the data in the webhook’s queue. The whole queue is then processed every time your schedule criteria are met.

Glenn - Callinetic

This appears to be a viable solution. I have a question regarding this: is it possible to clear the web queue after each execution concludes? Based on the article, it seems you can configure it to trigger one execution every five minutes. However, if there are 50 calls in the queue, it will continue to trigger them sequentially, meaning it would run continuously for 50 * 5 minutes. In my specific use case, all calls become invalid once processed because the links will have been updated. Therefore, I'd like to clear the queue after each run. Subsequently, if a new, distinct webhook call arrives after the scenario has executed, that new call would then be processed.

Hi Karl,

Could you please briefly describe the scenario?

Do you have X number of Airtable updates, and each update triggers the webhook, which then triggers the scenario? And instead of the scenario running every single time an update is made, you want it to run a maximum of once every 5 minutes?

If that's correct, why wouldn't you simply schedule the scenario to run every five minutes?

Please provide more details.

My use case is as follows: I have a callin.io scenario that refreshes Airtable records/primary key links for reporting purposes. This process needs to be updated only when a relevant event occurs. If no one accesses the system or updates data, there's no need for it to run. Therefore, running it every 5 minutes would be highly inefficient, especially since the system is primarily accessed at the end of the month and is heavily used for only 1-3 days during that period.

I'm looking for a way that when a user starts interacting with the system, Airtable would trigger webhook calls upon updates. Currently, callin.io ends up performing many unnecessary updates because there's no need to do so, for example, once per second. I would prefer the link updates to occur only once every 5 minutes, rather than with every Airtable push. Essentially, the webhook call logic would be: "If the webhook hasn't been called in the past 5 minutes, execute; otherwise, ignore." Naturally, the 5-minute interval is a placeholder and can be adjusted as needed.

With the current scheduling, Airtable might trigger 200 calls to the webhook. If I set it to execute only once every 5 minutes, the queue would take 200 x 5 minutes to complete, even though 199 of those calls are technically redundant.

Thanks for this context, this is very helpful.

Each update that calls the webhook presents an issue because it will consume at least one Operation for callin.io to process the request. The most feasible approach would be to simply disregard the requests until the appropriate time to process them, but this would still utilize at least one Operation per AT change.

Consider utilizing Airtable Automations for this.
Create an additional field, perhaps a simple checkbox.
Configure automations to mark this checkbox whenever a significant change is made to a record that requires the links to be recalculated.
Create a second field, a formula that calculates the number of minutes that have passed since the last update to the record.
Finally, set up another automation that searches for all records where the checkbox is marked and the last update occurred more than 5 minutes ago. If any such records are found, trigger your callin.io scenario.
Your scenario could then be adjusted to search Airtable records based on the same criteria (checkbox marked and last updated more than 5 minutes ago), perform the necessary tasks, update those records, and then clear the checkbox field.

Hope this explanation is clear!

Hi. Thanks a lot for that suggestion using Airtable, it didn’t occur to me to think that way. I’ve now created a simple ‘WebhookScheduler’ table, which tracks the last executed time and last requested time for each webhook I want to schedule/rate limit. Once executed in callin.io, callin.io updates the LastExecutedTime in Airtable. I’ve even added fields to specify the total seconds between executions, allowing me to rate limit/schedule different webhooks differently.

Then, using an Airtable automation, once sufficient time has passed between the last execution and now, if there's a request for a call, the call is made to callin.io. This approach makes the app much more responsive than simply running every 15 or 30 minutes. If anyone needs more details on the actual solution, feel free to ask; I'm happy to share.

A very straightforward solution, thanks for your suggestion!