Review: Vector Store Tool (+ Langchain Code Alternative!)

Jim_Le · 2024-07-15T11:52:02Z

Frame 6800×444 76.6 KB In version 1.49, callin.io finally included the Vector Store Tool as part of the officially supported tools for AI Agent nodes. Hurrah! I spent a few hours familiarising myself with the tool and wanted to share some thoughts on how I’d use it going forward. Note:These views are my totally my own. I enjoy talking about callin.io and AI and if you do too, check out my other posts in the forum! What is the Vector Store Tool? As with other callin.io’s AI agent tools, the Vector Store Tool extends the knowledge fetching capabilities of your agents and offers a new standard for connecting to your vector store of choice without the additional complexity of introducing sub-workflows. To start using this tool, Find it in the “tools” options for your AI agent. Connect your vector store and to that, connect your embedding model. Connect an additional language model (LLM) to the vector store. image1018×717 84.3 KB image1588×777 79.3 KB Screenshot 2024-07-15 1207191588×779 60.6 KB How it works The Vector Store Tool performs very much like the Vector Store Retriever but a key difference is that rather than returning raw results, the tool uses an additional LLM to return an “answer” to the agent’s query. Pros For most knowledge retrieval use-cases, this works great and introduces what is effectively a multi-agent workflow by having two LLMs talk to each other. Very easy to understand and put together - as mentioned previously, no need for a sub-workflow tool to use the vector store nodes. Great choice by the callin.io team with UI choices to keep these tools similar to their original counterparts. Cons In my particular use-case - which I acknowledge may not be how others typically use vector stores - I noticed that prompt instructions from the root agent may not be fully transferred to the tool’s LLM prompt, which may miss some key details you wanted to pick out from the retrieved information. Additionally, if you use the metadata as part of the response and not only for filtering, then you may lose this information in the tool’s LLM response as well. It’s not possible to modify the generic prompt instructions of the tool’s LLM. This means a lack of post-retrieval processing before sending the tool’s response and possibly a fix for the above. Verdict: Nice! A great addition to the AI agent’s arsenal and looking forward to more! Should now be the standard way AI Agents (with tools) get access to vector store resources. Suitable for a majority of vector store retrieval use-cases without additional modification. It may be difficult to get exact wording/extracts from retrieved documents as the result is “summarised” twice (tool’s LLM > agent’s LLM). If you use vector store metadata for more than just filtering, you may have trouble getting this data into the response. Ok, Let’s Go Custom Langchain Code! So, obviously with hitting the above limitations, I had to investigate how to go about overwriting the tool’s LLM prompt. The scenario is for my recipe library in my Qdrant Vectorstore where I store the recipe ID, categories and URL in the metadata and would like to pull these back out in the tool’s response. Here’s what I came up with… 1. Build your own Vector Store Tool to Override Tool’s LLM Prompt After trying a few things, the only way I found to do this was to use a custom Langchain Code Node implementation: A “supply data” sub-node using Langchain Code node which replicates the vector tool node shape, basically one-to-one. Inside, it uses a createRetrievalChain which connects to the vector store and LLM. Since we’re building the entire chain from scratch, we can now override the system prompt with SystemMessagePromptTemplate. Finally, returning our chain’s “invoke” wrapped in a DynamicTool allows this to be used by our Agent. Don’t forget to set the name and description! So we kinda just rebuilt the Vector Stool Tool but this worked pretty well in my opinion. By customising the tool’s prompt, I could get the recipe attributes I needed. More control over tool response to surface information that is needed. Agent and tool prompts need to be aligned. More risk of breaking due to upgrades. Screenshot 2024-07-15 1213091594×793 62.6 KB ### 2. Build your own Vector Store Tool Without LLM After implementing the above, it got me thinking if the tool really needed the LLM in my scenario. So I tried an alternative implementation without the LLM and instead with the tool returns the vector store documents directly. * Similar to the above implementation but does not use the __createRetrievalChain__ and so no need for the LLM. * The tool simply performs a `` similaritySearch() `` and returns them as the tool’s response. * Wrapped in __DynamicTool__ as before. After testing, the response were much more detailed for my particular use-case. Additionally, I also found possibility for post-retrieval processing which may help if you’re worried about token count. Keep all prompts in one place for easier maintenance...

Page 2 / 2 Prev

Features

Last Post by frknbr 7 months ago

16 Posts

6 Users

0 Reactions

7 Views

RSS

frknbr

(@frknbr)

Posts: 1

New Member

Hello,

Thanks, that's interesting. I'd like to try your custom approach.

However, I'm unable to locate the module you're using for your Vector Store with Chain. It appears there isn't one available.

Cheers,

Posted : 31/12/2024 4:20 pm

Page 2 / 2 Prev

8 Forums
1,441 Topics
8,083 Posts
0 Online
2,387 Members

Forum Icons: Forum contains no unread posts Forum contains unread posts

Topic Icons: Not Replied Replied Active Hot Sticky Unapproved Solved Private Closed