Optimizing callin.io for long-running tasks

Describe the question

We are running AI automation workflows to generate images and movies. These typically take between 5-15 minutes to complete and involve multiple workflows, ranging from very specific tasks like handling the actual call to a remote AI provider, to exposing a webhook and responding back to our services when done. Our k8s cluster is running with many smaller instances (cpu:2, mem:8), so I considered reconfiguring our callin.io from one instance (cpu:0.5, mem:2) to use workers instead and scale dynamically based on current traffic. However, we seem to be encountering issues with the validity window and callin.io not being able to respond in time. We also have one remote API provider which returns generated images as inline base64 instead of a URL. This was previously mitigated by setting mem:2 and maxoldspace, but I still observe that memory consumption only increases, never decreases, suggesting a memory leak. Our concurrency is set to 20 (we tried higher, but it only triggered memory issues faster), and a lower setting caused "Offer expired" more quickly. We are not delivering our results in real-time, so a few workers should be able to handle the load at their own pace. I plan to scale our workers based on queue length once we can get one worker stable.

So, my questions are: what is the recommended approach to tune callin.io for handling long-running tasks? Can I modify or remove the validity window for tasks? And why does the callin.io instance become unresponsive when it's only handling dispatch and at most one user accessing the UI?

What is the error message (if any)?

• Task (1UEsQ7iT) rejected by Runner with reason "Offer expired - not accepted within validity window"
• Our LB in front of callin.io is returning 504 when attempting to proxy to callin.io (we're unsure why it doesn't respond in time).

Please share your workflow

This is too sensitive to share, but we are using a hierarchy of workflows, with a depth of up to 5 (currently) workflows:

• webhook workflow (handles webhook, dispatches to other workflows, and manages callbacks when done)

  • workflow A (if we require more than 1 image back, we configure this to loop)

    • workflow B (contains additional logic specific to our use case, such as QA)

      • workflow C (and further logic specific to our use case

        • workflow handling the call to the remote API, returning a generated image/movie

Information on your callin.io setup

• callin.io version: 1.97.1
• Database (default: SQLite): postgres
• callin.io EXECUTIONS_PROCESS setting (default: own, main): default
• Running callin.io via (Docker, npm, callin.io cloud, desktop app): docker, k8s, gcp
• Operating system: linux