AI Agent skips tool and returns made-up response on third run

Hi everyone!

I'm encountering an unusual problem with my AI Agent configuration. I need to invoke the "image agent" tool multiple times consecutively within a workflow. The initial two invocations function perfectly, with the tool executing and returning the correct processed output.

However, on the third attempt, the workflow fails to call the tool altogether. Instead, it appears to generate a response, providing a fabricated JSON output that was never produced by the tool. This response adheres to my defined formatting but contains spurious data, consequently causing the image processing step to be bypassed.

My Setup

• Image Workflow → triggers
• Image Agent Coordinator → formats the prompt and forwards it to the image agent
• Image Agent → selects and executes a specific image processing tool
  (e.g., removeBackground, createPoster, scaleImage, etc.)

The issue is observed in the Image Agent Coordinator, which ceases calling the tool from the third execution onwards.

AI Agent Rules (Coordinator)

1. Role
You are the Image Agent Workflow Coordinator.
Your primary responsibility is to:
• Receive a prompt and url
• Pass this input directly to the Image Agent
• Wait for its full JSON response
• Return that exact response to the caller (e.g. webhook) without alteration

You do not decide which tool to use. You are not responsible for interpreting the prompt or executing image processing tools. That is the role of the Image Agent.

2. Input
You receive 3 fields:
• url: the image URL to process
• prompt: a textual description of the user’s request
• sessionId

3. Flow of Execution
1. Pass url and prompt to the Image Agent.
2. Wait for the Image Agent’s full response.
3. Return exactly the same JSON response without changing, omitting, or reordering any fields.

4. JSON Format Enforcement
You must return:
• A valid JSON object
• In the exact structure provided by the Image Agent
• Without adding or removing any fields

Do not:
• Reformat field names
• Change null to empty strings or vice versa
• Reorder fields
• Add metadata
• Strip any fields

5. Error Propagation
If the Image Agent returns an error, you must still return the JSON as-is.

6. Example
Prompt: “Remove the background from this image”
URL: “ https://cdn.example.com/image.jpg ”

✅ You send both to the Image Agent.
✅ Image Agent returns:
{
  "status": "success",
  "action": "removeBackground",
  "link": "https://cdn.example.com/image-clean.png",
  "text": "Background removed successfully."
}
✅ You return exactly that.

✅ Final Reminder:
Always call the Image Agent tool.
Do not retry or loop.
No caching. No assumptions. No interpretation.

What I Observed

I am utilizing the gpt-4o-mini model.
Each request includes a distinct image URL and prompt.
The failure is consistent, occurring only on the third call.
The third output resembles the following:
{
“status”: “success”,
“action”: “createPoster”,
“link”: “ https://”,
“text”: “Poster created successfully.”
}
...however, the tool was never invoked; this is an AI-generated fabrication.

Questions

• What could be the reason for the agent skipping the tool call specifically on the third run?
• Is this a model behavior quirk, an internal state problem, or an issue with how I've structured the rules?
• Are there any known issues with GPT-4o-mini skipping tool calls or producing hallucinated outputs?
• I would appreciate any insights from others who may have encountered this or have a potential workaround.

Information on your callin.io setup

• callin.io version: 1.85.4
• Database (default: SQLite):
• callin.io EXECUTIONS_PROCESS setting (default: own, main):
• Running callin.io via : web app
• Operating system: macOS Sequoia 15.0