Trouble with AI Agents

An update on this; reviewing the agent logs, I'm seeing a potential issue. I'm repeatedly encountering these prompts (which were hardcoded and not input by me) that inform the AI that its response was either incorrect or that an error occurred.

Okay, so what is the response to my last comment? If you use information obtained from the tools, you must mention it explicitly without naming the tools – I have forgotten all TOOL RESPONSES! Remember to reply with a markdown code snippet of a JSON blob containing a single action, and nothing else.
AI:
json
{
“action”: “respond”,
“action_input”: “I apologize for the earlier confusion. Based on the research I’ve gathered, trust is a critical factor in various relationship types. Numerous studies highlight both the cognitive and affective aspects of trust in relationship building. Neuroscience research indicates that cultivating a culture of trust can substantially boost productivity, collaboration, and employee loyalty. Furthermore, research underscores the importance of trustworthiness in qualitative research and emphasizes the significance of trust between research participants and researchers. Would you like me to elaborate on any specific aspect of trust in relationships?”
}


Human: TOOL RESPONSE:
---------------------
respond is not a valid tool, try another one.

Human: TOOL RESPONSE:
---------------------
There was an error: "There was an error: "The workflow did not return an item with the property 'commit'""

USER'S INPUT
--------------------

Okay, so what is the response to my last comment? If using information obtained from the tools you must mention it explicitly without mentioning the tool names - I have forgotten all TOOL RESPONSES! Remember to respond with a markdown code snippet of a json blob with a single action, and NOTHING else.
AI: ```json
{
"action": "commit_data",
"action_input": {
"type": "research",
"commit": "Trust plays a crucial role in various types of relationships, according to research. Key findings include:n1. Trust impacts attachment styles, relationship patterns, and the likelihood of relational success.n2. Establishing trust takes time and effort in most relationships.n3. Trust is naturally established between parents and children, spouses, partnerships, and friendships.n4. Trust in relationships with parents may influence future health and relational developments.n5. Lack of trust and adverse childhood experiences can contribute to unhealthy outcomes in the future.n6. Children who lack trust with parents may be less likely to engage in relational developments with other children.n7. Trust is built by recognizing and responding to one's negative affect, which fosters attunement and comfort in discussing conflict and stress.n8. The Adverse Childhood Experiences Study (ACES) showed that children who experienced conflicts such as abuse, neglect, or familial conflict were more likely to develop depressive symptoms, substance abuse issues, and increased antisocial behavior.n9. A Harvard study found that adults who described their parental relationships as lacking warmth and closeness were 45% more likely to have developed serious medical illnesses in mid-life.nThese findings highlight the significant impact of trust on relationship dynamics, mental health, and long-term well-being."
}
}

Human: TOOL RESPONSE:

There was an error: "There was an error: “The workflow did not return an item with the property ‘commit’”

Hey,

Selecting the right tool can sometimes be challenging. The model you're employing needs to be intelligent enough to recognize when to utilize a specific tool. I would have expected Claude to be capable of selecting the tool, but I'm uncertain about the data flow between different tools, which might be your objective.

Do you have any insights on this, Oleg?

Unfortunately, even with two tools, I'm unable to get this working. Claude continues to invent tools not specified in the prompt, leading to the error "this is not a valid tool." Meanwhile, GPT-4o is frequently rate-limited because its output is flagged as not being in the correct format. I'll need to proceed without using the agent, making API calls to retrieve context. My current challenge is figuring out how to implement a loop to maintain a back-and-forth chat experience.

This is proving a bit challenging as I'm developing a solution for a client. My goal is a chatbot capable of making a couple of API calls: one to a vector database (specifically qdrant with several collections) and another to Perplexity. However, despite clearly defining the available tools and their input/output structures, the chatbot continues to make errors.

Interesting, what happens if you use a ‘Tools Agent’ instead of the conversational one? Also, as Jon mentioned, can you try with a different language model? I am using callin.io's gpt-4o-mini and it’s pretty good for what I need. I use about 12 tools right now.

GPT-4o appears to grasp tool usage more effectively, but it consistently returns an “improper output.” This consumes additional tokens and leads to frequent rate limiting.
I’ve encountered a similar problem with the tools agent. It seems related to the internal prompting that instructs the LLM to produce a specific output format. Whenever the model attempts to generate markdown or a similar structure, the internal prompt flags the output as not being valid JSON, necessitating several back-and-forth attempts. This process contributes to rate limiting, especially given that this is a relatively new account with OpenAI.

Hello!

Some observations on your tool descriptions:

Example 1:

Call this tool to get context from a vector database that will assist in writing the book The Evolved Man.

For me, this is a classic case of focusing too much on the “how” (see my previous response to this). What response should the agent expect from this tool? How can it determine that the response was a success? If it can’t, it’ll just keep retrying/asking and you’ll get stuck.
In this scenario, I try to put myself in the virtual shoes of the agent:

• Why should I use this tool? it helps the “user” write their book.
• When should I use this tool? when user is mentioning/asking about something that was discussed previously/in the past.

Try this or some variation of:

Call this tool to search for past context that was saved whilst helping the user write their book.

Example 2:

Call this tool to make an API call to Perplexity AI that will do a real-time web lookup for research papers on a topic.

Be careful of keywords in your tool description as these also act as trigger words for tool use:

• “Perplexity AI” - did the user specifically request to use Perplexity?
• “real-time web lookup” - did the user specifically request for real-time or does it matter if it’s slightly delayed?
• “research papers” - did the user specifically ask for research papers or just wants the summary of the research?

Sometimes being overly specific or service-dependent means the tool won’t be used or used incorrectly.

Try this instead:

Call this tool to research a topic suggested by the user.

• “research a topic” - doesn’t care about what service is being used, focuses on purpose; just send the topic to be researched.
• “suggested by the user” - only trigger this when the user mentions or requests for it or when it is a good time to do so. Good for limiting ways extraneous calls.

Overall I think you have quite an ambitious project on your hands but don’t give up, I’ll be cheering you on!

I cooked up this example workflow to hopefully help you debug your tool problem.

Screenshot 2024-08-21 at 13.32.571920×1081 124 KB

Screenshot 2024-08-21 at 13.34.213084×1736 508 KB

Screenshot 2024-08-21 at 13.35.561920×1081 111 KB

I really appreciate your work on this. While I couldn't utilize your in-memory vector (we have a crucial first draft in our current vector db), it seems your simplified prompts were effective!

Thank you.

This discussion was automatically closed 7 days following the last response. New replies are no longer permitted.