I’ve been using gptel in Emacs for increasingly complicated workflows: conversations that involve tools, multiple model requests, and tasks delegated to other agents. One of the first things I did was add a simple tool-call logging system. With this, I was able to introspect my chats and improve the top-level and subagent system prompts, as well as several tool definitions. This was useful, but I was running into limitations and wanted to step up to something that captured whole traces with all the metadata I’d need.
At work we’ve looked into a lot of different solutions along these lines. There are several options that support OpenTelemetry tracing, including Arize Phoenix, LangSmith, and Braintrust. We’ve had great success with Langfuse, which also has the benefit of being self-hostable.
I couldn’t find an existing integration for gptel, so I built gptel-otel, an OpenTelemetry tracing package. It records model generations and tool executions, including timing, model information, and token usage when available. If you’re also using gptel-agent, it connects delegated agents to the tool calls that launched them, so you can follow a task through its child requests rather than seeing a collection of unrelated model calls.
Let me know if you find it useful!