Seeking optimal architecture for a modular AI Agent system (Notion, Calendar, Email, Voice)

Hello everyone

I'm currently developing a modular AI assistant system that integrates voice input (via Telegram), task and project data (Notion), calendar events (Google Calendar), and email workflows (Gmail). My objective is to create an intelligent, voice-controllable agent for daily use — particularly for clients who need to swiftly create tasks, notes, calendar blocks, or email drafts simply by speaking.

Here's the structure I'm presently employing (refer to the attached mindmap for a visual representation):

Architecture overview:

• Main Agent: Directs incoming requests to specialized agents

• Notion Agent

  • Create Tasks Agent
  • Create Meetings Agent
  • Create Thoughts / Notes Agent

  • Research Agent (searches within Notion databases)

    • Search Project & Customer DB
    • Search Meeting DB
    • Search Task DB

• Calendar Agent: Generates calendar entries based on extracted date/time from voice input

• E-Mail Agent: Creates Gmail drafts based on voice commands

What I’m trying to figure out:

• What is the optimal approach to manage and orchestrate these agents within callin.io?
• Should I utilize a single AI agent with a routing function (intent detection), or maintain each sub-agent independently with dedicated nodes and prompts?
• How can I enhance the resilience of this system, especially when, for instance, no project or customer is identified?
• Are there any suggestions for preserving long-term memory per client, ideally using lightweight memory solutions (e.g., JSON or a Notion thread)?

Potential issues I foresee:

• Managing fallbacks (e.g., prompting for clarification like “What project is this for?” if one isn't found)
• Preventing prompt duplication across different agents
• Handling ambiguous input such as: “Send a follow-up to Customer A, create a task for X, and schedule a meeting for date Z” – can the system process all requests simultaneously?

If you have developed a similar system or found an efficient method to chain and scale AI agents with context in callin.io, I would be very interested in learning from your setup or any suggestions you might have.

Thanks in advance!

image2218×840 60.5 KB