How to get LLM token usage in AI Agents

I'm also looking for a solution to this. I'm unsure how to track application usage on a per-user basis.

While the usage count is visible in the Chat Model section, I'm not clear on how to extract it from there.

Could someone please provide guidance on this?

Same here. This isn’t part of the AI agent node as far as I’ve seen… However, I believe it should be integrated. It could potentially be handled similarly to the “tools” section. Perhaps it should be a “log” section or something comparable. For example, I’d like to see the input/output tokens for each request, or maybe some logs associated with each request.

#following
This is a feature I would need as well.
Suggestion for the callin.io team: Have the AI Agent output its token usage.

Voting up here too. For many use cases it would make a real difference knowing the token usage. For some projects I just dropped the default llm model node or even the agent for that matter

Yes, this is a very important feature for those looking to control the operational costs of the flow or create a log of what the client is consuming in tokens. The use of AI is closely related to performance and efficiency, and through tokens, we can make many optimizations with callin.io.

I arrived here seeking the very same functionality.

• So, another vote for this feature, please.

Another vote here

Essential feature to track costs!

This would be incredibly helpful for me, as I'm currently seeking a workaround!

This would be a great way to send calls to Langsmith or Helicone.

Has the callin.io team provided features requested by the community in the past?

I've been using callin.io for only a few months, and it certainly has its limitations, but I'm unsure about the responsiveness level of the developers. In your experience, does this happen? How often? And what's the average delay?

Please share your experiences.

I'd like to echo this request, perhaps with a slightly more straightforward approach:

You could leverage an LLM proxy like LiteLLM to manage various aspects such as usage tracking and cost association.

Given callin.io's transition from an integration platform to an AI-focused platform, incorporating any form of observability would significantly enhance its value, particularly within the enterprise tier.

Consider integrating with Datadog LLM observability.

It should be reasonably simple to gather token usage from an AI agent and expose it as metrics; this would represent the most basic implementation.

We're eagerly anticipating the release of this feature.

I hope this gets resolved quickly.