Semantic Caching in Workflow Graphs: How ChorusGraph Reduces AI Agent Costs by 76% in 2026

MarGib October 16, 2026

🌐 🇵🇱 Polski · 🇬🇧 EN

In 2026, the costs of LLM calls are becoming a barrier to scaling AI agents. chorusgraph proposes a solution: semantic caching in workflow graphs, which reduces the number of queries by up to 76%. How does it work and is it worth implementing?

AI agent workflow graph with highlighted semantically cached nodes — Illustration: AI agent workflow graph with semantic caching in ChorusGraph (2026)

AI agents are becoming increasingly complex, but their practical application is still hindered by the high costs and latency associated with large language model (LLM) calls. In response to these challenges, new optimization techniques are emerging, and one of the most promising is semantic caching. In this article, we will examine how the chorusgraph framework integrates this technology with workflow graphs, offering a reduction in LLM calls of up to 76%.

What is semantic caching?

Traditional caching involves storing query results and reusing them when the exact same query appears again. However, this approach has limitations – in real-world scenarios, users rarely ask identical questions. Instead, they phrase them differently, but with the same meaning. This is where semantic caching comes in.

This mechanism is based on the concept of embeddings – vector representations of text that capture its meaning. When a user asks a question, the system converts it into an embedding and compares it with previously stored vectors. If the similarity (measured, for example, using cosine similarity) exceeds a set threshold, the system returns the stored result instead of calling the LLM. This allows for savings even when queries are not identical but are semantically similar.

Example: The query "How to reset my password?" may be considered similar to "How to change my password?" if their embeddings are close enough. As a result, the system avoids an unnecessary model call, which translates into lower costs and faster response times.

Technical foundations of semantic caching

The key elements of this mechanism are:

Embeddings: Text is converted into a numerical vector using models such as text-embedding-ada-002 from openai or sentence-transformers/all-MiniLM-L6-v2. These vectors have hundreds or thousands of dimensions and capture the semantics of the text.
Semantic similarity: Vectors are compared using metrics such as cosine similarity, which measures the angle between vectors (a value from 0 to 1, where 1 means identical meaning).
Acceptance threshold: Set by the developer (e.g., 0.92). If the similarity exceeds this threshold, the system considers the queries similar enough to return the cached result.
Search algorithms: To quickly find similar embeddings in large datasets, structures such as Locality-Sensitive Hashing (LSH) or Hierarchical Navigable Small World (HNSW) are used.

Unlike traditional caching, which requires an exact text match, semantic caching is more flexible and handles the natural diversity of language better.

chorusgraph: architecture and semantic caching integration

chorusgraph is an open-source framework for building AI agents based on workflow graphs. Its unique feature is the integration of semantic caching directly into the graph structure, which allows for optimization at the level of individual nodes.

Key components of chorusgraph

The framework's architecture consists of several layers:

Knowledge Graph: Represents the structure of tasks and the dependencies between them. Each node is a task (e.g., query analysis, response generation), and edges define the order of execution.
Semantic Cache: Caches LLM call results at the graph node level. It uses embeddings to compare queries and decide whether to return a stored result.
LLM Gateway: An abstraction layer enabling integration with various models (openai, Mistral, local models).
Monitoring and fallback: Tracks cache hits/misses and calls the LLM when necessary, ensuring reliability.

Use cases

chorusgraph excels in scenarios where AI agents perform repetitive tasks. Examples from the documentation include:

Customer support: Caching answers to frequently asked questions (FAQ), e.g., "How to return a product?" or "How to track an order?".
Document analysis: Repetitive queries to RAG (Retrieval-Augmented Generation) systems, where semantic caching reduces the number of LLM calls.
Business process automation: Agents for processing invoices, orders, or complaints, where many queries have a similar nature.

Thanks to the integration with workflow graphs, chorusgraph allows for precise definition of which nodes should use caching and which require direct LLM calls.

Where did the 76% fewer LLM calls figure come from?

This number comes from internal tests by the creators of chorusgraph, described in the benchmarks/README.md file in the repository. The test was conducted on a set of 10,000 queries to an agent handling an online store's FAQ (topics: returns, payments, delivery). The experiment conditions were as follows:

LLM Model: gpt-3.5-turbo (openai API).
Semantic similarity threshold: 0.92 (cosine similarity).
Cache size: 5,000 unique queries.
The query set contained 30% repetitions (semantically similar questions).

The results were impressive:

Without caching: 10,000 LLM calls.
With semantic caching: 2,400 LLM calls (a 76% reduction).
Average response time dropped from 1.2s to 0.3s.

However, it is worth emphasizing that this result depends on query repeatability. In scenarios where queries are unique (e.g., analysis of custom documents), the reduction will be smaller. The creators of chorusgraph do not provide details regarding the distribution of query similarity in the test set, which makes it difficult to assess how representative these results are.

Comparison with other solutions

Similar caching mechanisms are offered by other frameworks, but with varying results:

langchain: The SemanticCache module allows for a 40–60% reduction in calls in FAQ scenarios (source: langchain documentation). However, it is not integrated with a graphical workflow representation.
llamaindex: Offers caching at the RAG query level but does not provide specific cost reduction benchmarks.

chorusgraph stands out because caching works at the graph node level, which provides greater flexibility in optimizing complex agents.

Limitations and challenges of semantic caching

Despite promising results, semantic caching is not without its challenges. Here are the most important ones:

1. False Positives

The biggest risk is a situation where the system considers two semantically different queries as similar and returns an incorrect result. Example:

Query: "How to delete my account?"
Cached query: "How to change my password?"

Although both concern account management, the answers will be completely different. To minimize this risk, you should:

Fine-tune the similarity threshold (e.g., 0.95 for critical queries).
Test the system on real data and monitor cache hits.

2. Stale data

If cached results are based on outdated information (e.g., a change in company policy), the agent will return incorrect answers. Solutions:

Time-to-Live (TTL): Setting an expiration time for cache entries (e.g., 24h).
Manual cache clearing: In case of critical updates.
Dynamic data: Disabling caching for nodes that use frequently changing information.

3. Computational cost

Generating embeddings and comparing vectors requires computational power, especially for large caches. Ways to optimize:

Using optimized libraries such as FAISS from Meta.
Storing embeddings in vector databases (Pinecone, Weaviate).
Using local embedding models (e.g., sentence-transformers/all-MiniLM-L6-v2) instead of paid APIs.

4. Configuration

Semantic caching requires tuning several parameters:

Similarity threshold: Too low → false positives, too high → low call reduction.
Cache size: Too small → low efficiency, too large → high memory usage.
Embedding model: Choosing between quality (e.g., text-embedding-ada-002) and cost (local models).

There are no universal settings – each use case requires an individual approach.

Alternatives to chorusgraph

chorusgraph is not the only solution for optimizing AI agents. Here is a comparison with alternatives:

Solution	Type	Semantic Caching	Graph Integration	Pros	Cons
chorusgraph	Framework	✅ Yes	✅ Yes	Optimization for graphical workflows, flexibility	Young project, smaller community
langchain	Library	✅ Yes (semanticcache)	❌ No	Large community, many integrations	No native graph support
llamaindex	Library	✅ Yes (Query Cache)	❌ No	Strong RAG support	Less flexible than chorusgraph
Haystack	Framework	❌ No	❌ No	Good for document search	No semantic caching
Custom RAG	Self-built	✅ Possible	✅ Possible	Full control	Requires significant effort

langchain vs. chorusgraph

langchain offers SemanticCache as part of the data_connection module, but it is not integrated with a graphical workflow representation. This makes chorusgraph a better choice for projects where agents have a complex structure of dependencies between tasks. On the other hand, langchain has a larger community and more ready-made integrations, which may be decisive for smaller projects.

Costs of implementing semantic caching

Semantic caching can bring savings, but it also involves certain costs. Here is what you need to consider:

Costs of chorusgraph

License: MIT (free, open-source).
Hardware requirements:
- RAM: Depends on cache size. For example, 10,000 embeddings of 1536 dimensions (model text-embedding-ada-002) take up about 120 MB.
- Processor: Generating embeddings can strain the CPU/GPU, especially with large datasets. Using libraries like FAISS is recommended.
External costs:
- Embedding models: openai API (text-embedding-ada-002) costs $0.0001 per 1K tokens (as of 2026).
- Vector databases: Services like Pinecone cost from $70/month for small projects.

Example cost calculation

Assume an agent handling 100,000 queries per month, with a 50% cache hit rate:

Without caching: 100,000 LLM calls × $0.002 (gpt-3.5-turbo) = $200.
With caching: 50,000 LLM calls + 100,000 embeddings × $0.0001 = $105 (47.5% savings).

It is worth noting that savings grow with query repeatability. In scenarios with low repeatability (e.g., 10%), cost reduction may be minimal.

Development prospects in 2026

Semantic caching is gaining popularity, and chorusgraph is one of the projects that may accelerate its adoption. Here is what we can expect in the coming months:

Market trends

Cost optimization: As companies scale their LLM-based solutions, cost optimization is becoming a priority. Semantic caching is one of the most effective tools in this regard.
Graph integration: More and more projects are exploring the combination of AI agents with workflow graphs, which opens up new possibilities for chorusgraph.
Local embedding models: To reduce dependence on paid APIs, developers are increasingly turning to local models, such as sentence-transformers/all-MiniLM-L6-v2.

ChorusGraph Roadmap

Based on the ROADMAP.md file in the repository, the creators plan:

Q3 2026: Integration with langgraph – a new project from the creators of langchain for building graph-based agents.
Q4 2026: Support for local embedding models, which will allow for further cost reduction.
2027: Automatic similarity threshold tuning using machine learning, which will simplify system configuration.

New features in competition

LangChain: Plans to add hierarchical caching, which will allow for caching results of sub-tasks in chains.
LlamaIndex: Is testing adaptive caching, where the similarity threshold is dynamically adjusted based on context.

Summary: Is it worth implementing semantic caching?

Semantic caching is a powerful tool for optimizing AI agents, especially in scenarios with repetitive queries. ChorusGraph stands out by integrating this mechanism with workflow graphs, which provides greater flexibility and control over optimization. However, implementing this technology requires careful configuration and monitoring to avoid errors related to false positives or stale data.

If your projects rely on AI agents performing repetitive tasks (e.g., customer support, document analysis), semantic caching can bring significant savings. However, it is worth remembering that this is not a universal solution – in scenarios with unique queries, its effectiveness will be limited.

For developers looking for a ready-made solution, ChorusGraph is an interesting option, especially if they plan to integrate with workflow graphs. Alternatives such as LangChain or LlamaIndex may be a better choice for smaller projects or those that do not require an advanced graphical structure.

As this technology evolves, we can expect better tools for automatic parameter tuning and integration with local models, which will further lower the barrier to entry.

The author is not affiliated with the creators of ChorusGraph or any of the mentioned projects. The analysis is based on publicly available sources and tests conducted up to July 13, 2026.