How to combine multiple markdown files

Hey, thanks for the quick response. Your suggestion regarding 'RAG' models was excellent! I believe this is precisely what I need, as I perform a significant amount of data analysis, including competitor analysis. I'll need to study up and watch more videos/read more about RAG configurations and vector data foundations.

Unfortunately, callin.io is preventing me from sharing the code due to exceeding the character limit (477,555/320,000).

I could add you to the project, but I don't want to take up any more of your time on this, especially if I can rebuild everything with a RAG model (hopefully).

Just to have a small success, the workflow I've built is nearly complete, but I'm encountering one final looping issue. Although my problem is slightly different from before, given your apparent understanding of how callin.io flows handle arrays (looping) versus strings (not looping), let me run this by you one last time in this thread – I promise. (This is part of the previous flow, so I can't share it because it's too large).

The gist of it is:

1. I've finished my data enrichment process after scraping company websites; the data is in a Google Sheet.
   • Each row represents a company (e.g., concert golf, acushnet, topgolf).
   • Each column is a characteristic of that company (e.g., business description, industry focus, etc.).
2. I'm providing this Google Sheet to my AI agent as a tool and instructing the agent to perform a competitor analysis. (the screenshot)
   • The agent should begin by comparing company 1 to company 2 (concert golf vs. acushnet).
   • Then, compare company 1 to company 3 (concert golf vs. topgolf).
   • Then, compare company 1 to company 4, and so on – essentially, loop the task.

The agent executes the task correctly on the first iteration, comparing company 1 to company 2 (concert golf to acushnet). However, for subsequent rows, it duplicates the result from the first row (see screenshot). This indicates the loop isn't functioning correctly. I was wondering if this might be related to me not providing the AI agent with memory? I'm having trouble connecting memory to a previous node, as the AI agent doesn't use a previous node in its task execution, only the tool/Google Sheet. Alternatively, could the problem lie in how I've written my system and user prompts for the AI agent?

loop2039×1002 302 KB

Wouter - thanks again for your time and invaluable input!

To overcome the size limitation, you can capture a screenshot for context and then select only the nodes pertinent to the issue by highlighting them. We don't always require the entire workflow to diagnose a specific problem.

Here are a few suggestions: I recommend avoiding the creation of excessively large workflows. It's better to divide them into smaller, functional modules. For instance, one workflow for data enrichment, another for processing, and so on. If you need to transfer data between workflows, utilize the "Execute A Sub Workflow" node.

You can view RAG as a knowledge base for an LLM, providing it with specific knowledge from data you own. It might not address your use case here.

To attempt an answer to your challenge: incorporating memory into an agent is only beneficial if you're adding it to a chatbot and require it to retain context, such as in a sequence like Q1: What's the weather like at X? Q2: And at Y? This enables the agent to "remember" that the context pertains to weather inquiries. I would much prefer to equip it with the Think tool to transform the model into a thinking agent.

However, I would design the aforementioned process quite differently, making the data input more explicit. This would involve calling the Google Sheet before the AI agent and then inputting that data, rather than relying on the agent to deduce that it needs to read all rows (unless your prompt was highly detailed). I also believe Claude performs better when comparing data.

I will construct a workflow to test my theory before sharing it here.

1. I will certainly consider your advice about breaking down the process into sub-workflows for future projects. This was my initial attempt at an analysis, and even with a limited number of companies and attributes, the agent encountered difficulties.
2. Regarding rag, after my research, I concur. For my current needs, Excel might suffice, but I should explore solutions like Supabase to accommodate larger datasets in the future.
3. Concerning my loop issue, please don't invest any more time on it; you've already provided ample assistance! I plan to construct a smaller workflow that utilizes the output from the data-enrichment process from the worksheet and maps the parameters to AI. I'll also try using Claude. Regardless of whether my workflow is successful, I will share my findings here.

Thank you once again for your patience and help!!!

Here's what I've put together. I need to fine-tune the prompt and data input, as it's currently only performing one comparison. I have a meeting to attend now, but I'll address this this afternoon after my meeting and update this post. However, this should give you a general idea. The input formatting needs improvement; it was quite rough and hastily done.

image1636×1844 243 KB

image1920×1019 140 KB