Reducing Latency and Cost with Semantic Caching on Amazon Bedrock
Reducing Latency and Cost with Semantic Caching on Amazon Bedrock
“There are only two hard things in Computer Science: cache invalidation and naming things.” The line is attributed to Phil Karlton, and the first of the two becomes especially interesting when software stops communicating exclusively through deterministic parameters and starts communicating through natural language.
In a traditional application, caching is relatively deterministic. A request arrives with a known set of attributes, such as a user ID, product ID, API path, or query parameters. If another request arrives with the same key, the application can look up the previously computed result and return it without repeating the work. This can reduce calls to backend systems, improve response times, and lower operating costs when the cache is designed correctly.
Generative AI applications make that problem more difficult. A large portion of the request is often expressed in natural language rather than in a fixed collection of key-value pairs. Two users can ask the same question using completely different words. Two requests can look nearly identical but have completely different intent. Conversely, two requests can be phrased differently while requiring the same operation and ultimately producing an equivalent answer.
This creates a problem for traditional exact-match caching. A cache that uses the raw request string as its key may consider the following requests different:
- “Where can I find the instructions for resetting my password?”
- “How do I reset my password?”
- “What are the steps to change my account password?”
From a string-matching perspective, these are three very different keys. From a semantic perspective, they may represent the same user intent.
This is where semantic caching can help. Rather than asking whether two requests are exactly the same, a semantic cache asks whether they are similar enough in meaning that the result of one request can safely be reused for the other. The key idea is to represent natural-language requests as embeddings, compare those vectors in a vector store, and reuse a previously generated response when the similarity exceeds a configured threshold. AWS has published multiple reference architectures for this pattern, including implementations using Amazon Bedrock with Amazon OpenSearch Serverless and Amazon ElastiCache for Valkey.
In this post, we will look at what semantic caching is, how it fits into an Amazon Bedrock architecture, how to implement a read-through semantic cache using Amazon ElastiCache for Valkey and Amazon Titan Text Embeddings V2, and the trade-offs that should be considered before introducing semantic caching into a production generative AI application.
Why Traditional Caching Breaks Down for Generative AI
Traditional caching works well when the application can deterministically identify whether two requests are equivalent. A database query, REST endpoint, or computation can often be represented by a stable cache key. For example, an application might cache the result of a request based on a customer ID and a product ID. When those values are the same, the cache can safely retrieve the previous result.
Generative AI applications are different because the request frequently contains natural-language instructions, questions, or context. The exact characters in the request are often less important than the intent represented by those characters.
Consider an enterprise support chatbot. One employee asks, “How do I connect to the corporate VPN from my laptop?” Another asks, “What’s the process for setting up VPN access?” A third asks, “My company has a VPN. I need instructions to connect my computer to it.” These prompts are not exact matches, but they can correspond to the same underlying task.
Embedding models provide a way to represent this semantic meaning numerically. Amazon Bedrock provides Amazon Titan Text Embeddings V2, which converts text into a vector representation and can be used for retrieval and semantic similarity workloads. Titan Text Embeddings V2 supports configurable output dimensions of 256, 512, or 1,024 dimensions, with 1,024 dimensions being the default [1]. AWS documentation also describes embeddings as numerical representations of the semantic relationships in text, while the actual similarity search is performed by a vector database or vector search engine.
Once a request has been transformed into a vector, the application can compare that vector with vectors generated from previous requests. Vector search systems can use similarity or distance metrics such as cosine distance, Euclidean distance, or inner product to determine which stored vectors are closest to the new request [2]. Amazon ElastiCache for Valkey supports vector search using HNSW and FLAT algorithms and supports cosine, Euclidean, and inner product distance metrics [3]. If you are weighing where those vectors should live and at what cost, our post on tiered vector search with Amazon OpenSearch Service and S3 Vectors covers the storage side of that decision.
This changes the basic caching question from “Have I seen this exact request before?” to “Have I seen a sufficiently similar request before?”
That distinction is the foundation of semantic caching.
What Is Semantic Caching?
As mentioned above, a semantic cache is a request-response cache that uses semantic similarity rather than exact string equality to identify reusable results.
At a high level, the application stores two things together: the embedding of a request and the response generated for that request. When a new request arrives, the application generates an embedding for the new request in real time and performs a vector similarity search against the cached embeddings. When an existing request is sufficiently similar, the cached response can be returned directly. When no sufficiently similar result exists, the application proceeds to the normal generative AI path, invokes the model or agent, and stores the newly generated response for future reuse. This is the read-through semantic caching pattern described in AWS guidance for Amazon Bedrock workloads.
This architecture introduces an additional step before model inference. The application first calls the embedding model, searches the semantic cache, and only invokes the larger foundation model when the request is a cache miss. That means a semantic cache is not the same thing as prompt caching in Amazon Bedrock. Amazon Bedrock prompt caching is designed to reuse eligible prompt content, particularly repeated prompt prefixes, so the model does not have to repeatedly process the same context. Bedrock supports implicit and explicit prompt caching for supported models and APIs [4]. Semantic caching operates at a different layer: it stores request-response pairs and determines whether a new request is similar enough to a previous request to reuse the previous response. The two mechanisms can therefore complement each other. Prompt caching can reduce the cost and latency associated with repeatedly processing shared prompt context, while a semantic cache can potentially avoid an entire model invocation when a sufficiently similar request has already been answered.
A Semantic Cache Architecture on Amazon Bedrock
A practical architecture can place the semantic cache directly in front of an Amazon Bedrock model or agent. The application receives a user request, generates an embedding for the request using Amazon Bedrock, searches a vector-enabled cache, and then chooses between a cache-hit path and a cache-miss path.
For this example, we use Amazon ElastiCache for Valkey as the semantic cache store and Amazon Titan Text Embeddings V2 as the embedding model. AWS provides a documented implementation of this pattern using ElastiCache for Valkey, including vector search, similarity thresholds, TTL-based expiration, and a read-through cache flow [5].
The architecture can be represented conceptually as follows: a user request first reaches the application layer. The application invokes Amazon Titan Text Embeddings V2 to convert the request into an embedding vector. That vector is then sent to Amazon ElastiCache for Valkey, which performs a vector search against previously cached requests. If the nearest cached request satisfies the configured similarity threshold, the application returns the stored response. The application does not need to invoke the downstream foundation model or agent for that request. If no cached request meets the threshold, the application invokes the selected Amazon Bedrock foundation model or agent. The generated response is returned to the user and the request embedding, response, and associated metadata are written into the cache.
The important architectural property is that the cache sits on the request path before expensive inference. A cache hit therefore changes the work performed by the application, rather than simply accelerating access to the same backend call.
How the Request Flow Works
The first step is embedding generation. Amazon Titan Text Embeddings V2 accepts text and returns a vector representation. In the AWS documentation example, the model is invoked with “amazon.titan-embed-text-v2:0”, a configurable dimension size, and an optional normalization setting.
A simplified request might look like this:
import json
import boto3
bedrock_runtime = boto3.client(
"bedrock-runtime",
region_name="us-east-1"
)
response = bedrock_runtime.invoke_model(
modelId="amazon.titan-embed-text-v2:0",
body=json.dumps({
"inputText": user_message,
"dimensions": 1024,
"normalize": True
})
)
embedding = json.loads(response["body"].read())["embedding"]
The application then uses the resulting vector to search the cache. A cache entry can contain the original request, its embedding, the generated response, and metadata describing the context in which that response was created. For example, the application could associate the entry with a model identifier, tenant identifier, application version, knowledge base version, or expiration time.
The metadata is important because semantic similarity alone does not necessarily mean two responses are interchangeable. For example, suppose an enterprise assistant answers a question using a particular version of an internal policy. Six months later, another employee asks a semantically similar question after the policy has changed. The requests may be close in embedding space, but returning the historical answer may be incorrect. The same issue can occur with personalized applications. Two requests may have identical semantic intent but require different answers because they belong to different users, accounts, regions, authorization domains, or product configurations. For this reason, semantic matching should generally be combined with deterministic constraints. The vector search answers the question, “Which previous request is most similar?” Additional metadata and application-level checks should answer the question, “Is this response actually eligible to be reused here?” Amazon Bedrock supports metadata filtering in Knowledge Bases, providing an example of the broader architectural pattern of combining semantic retrieval with deterministic metadata constraints. Retrieval filters can restrict results based on metadata attributes before results are returned [6]. Ultimately, this can lead to faster, cheaper and more accurate retrieval if applied correctly.
Choosing a Similarity Threshold
This is where semantic caching can become a bit of a gray area. A cache hit is no longer an absolute fact. With exact-match caching, two requests either have the same cache key or they do not. With semantic caching, the application has to decide how similar is similar enough.
A threshold that is too high makes the cache conservative. The application might only reuse responses for nearly identical requests, which protects answer quality but results in fewer cache hits. A threshold that is too low makes the cache aggressive. The hit rate may increase substantially, but the system also becomes more likely to reuse a response for a request that only appears similar but actually has a different intent. AWS has demonstrated this trade-off in an evaluation of semantic caching with Amazon ElastiCache for Valkey and Amazon Bedrock. In that specific experiment, a similarity threshold of 0.9 produced a 74.5% cache hit ratio, a reported 72.5% reduction in total daily cost, and 92.3% accuracy for cached responses. Lowering the threshold to 0.75 increased the hit ratio to 90.3% and cost savings to 86.3%, while reported cached-response accuracy fell to 91.2% [7].
The important lesson is not that 0.75 or 0.9 is the correct threshold for every application. It is that threshold selection is an application-specific quality and cost decision. A customer-support chatbot answering stable questions about account setup may tolerate a lower threshold than a system generating legal, financial, medical, or security-sensitive recommendations. Even within the same application, different classes of requests may require different policies.
A useful production approach is to treat the similarity threshold as a configurable product and engineering parameter rather than a constant buried in application code. Your organization can evaluate cache hit rate, answer quality, latency, and model cost against a representative validation set and then choose the operating point that makes sense for the concrete workload.
The AWS verified semantic cache pattern demonstrates a similar concept. In that architecture, highly similar questions could return curated answers directly, while partially similar questions could be used as examples to help guide the model rather than bypassing inference altogether. Lower-similarity requests fell back to normal LLM processing [11]. This suggests that semantic caching could be more sophisticated than a simple binary decision.
Preventing Stale or Incorrect Cached Responses
The largest risk in semantic caching is not that the cache fails to find a match. It is that the cache finds a match that looks correct but should not have been reused.
Generative AI responses are often dependent on context. That context can include the model being used, the system instructions, retrieved documents, tools, user permissions, business rules, and external data. A cache entry should therefore be considered valid only within the context for which it was produced. One practical solution is to make cache eligibility part of the keying and metadata strategy. A cache entry can carry information such as the model ID, application version, tenant or authorization scope, prompt or system-instruction version, knowledge-base version, and creation time. The application can then enforce deterministic checks before accepting a semantic match.
Expiration is another important mechanism. Semantic caches generally contain a hot set of frequently accessed requests rather than a permanent system of record. Amazon ElastiCache for Valkey provides standard cache capabilities such as time-to-live and eviction policies that can be used to control the lifetime of semantic cache entries. The AWS semantic caching implementation explicitly uses TTLs for cache records. The appropriate TTL depends on the data being served. A cache for static product documentation may tolerate a relatively long expiration period. A cache for rapidly changing operational information may need a short expiration period or explicit invalidation when the source data changes.
Sensitive data requires additional care. A semantic cache can contain both user requests and model-generated answers, which means a poorly scoped cache can accidentally expose one user's sensitive data or personally identifiable information (PII) to another user. Tenant isolation, authorization-aware metadata, and conservative cache eligibility rules should therefore be part of the design rather than an afterthought. The same principle applies to safety controls. When a cache hit bypasses model inference, the user input does not pass through the same inference path as a cache miss. Amazon Bedrock provides the “ApplyGuardrail” API for applying a guardrail independently, which can be used where an application requires additional validation of input or response before returning it [8].
Monitoring the Cache in Production
A semantic cache changes the behavior of the application, so observability becomes especially important. A traditional cache can typically be measured with familiar metrics such as hit rate, miss rate, entry count, and latency. A semantic cache needs those metrics as well as quality-oriented measures. For example, the application should distinguish requests served from the exact cache, requests served from the semantic cache, and requests that reached the foundation model. It should measure the embedding-generation latency separately from vector-search latency and model-inference latency. It should also monitor the similarity scores associated with accepted cache hits. This provides visibility into whether the cache is behaving as intended.
Amazon Bedrock publishes runtime metrics to Amazon CloudWatch, including invocation volume, latency, token consumption, and errors [9]. Bedrock also supports model invocation logging to CloudWatch Logs and Amazon S3 when enabled [10]. For the semantic cache itself, the application can publish additional metrics such as cache hit ratio, semantic cache hit ratio, average accepted similarity score, cache-hit response latency, cache-miss response latency, stale-response rate, and invalidated-entry count. One metric is particularly important: cache-hit quality. A semantic cache can report an excellent hit rate while making the application worse if the responses it returns are not appropriate for the incoming requests. That means organizations should evaluate cache quality using a representative test set or downstream business metrics rather than optimizing the hit rate in isolation.
In other words, the objective is not “maximize cache hits”. The objective is “maximize safe reuse”.
What This Means for You
Semantic caching introduces a different way of thinking about performance in generative AI applications. For product teams, it can provide faster responses for common questions and repeated workflows without requiring users to phrase their requests exactly the same way. For platform teams, it creates an explicit control point for reducing downstream model traffic. For engineering teams, it provides another architectural layer where response latency and inference costs can be optimized. For data and AI teams, the most important benefit may be the ability to reason about reuse in terms of intent rather than strings. The system does not have to know that two requests are identical. It only needs a reliable mechanism for determining that they are similar enough for the application.
Amazon Bedrock provides the model layer needed to generate those semantic representations through services such as Amazon Titan Text Embeddings V2. Vector-capable services such as Amazon ElastiCache for Valkey and Amazon OpenSearch Serverless provide the search layer needed to compare those representations efficiently. These capabilities make it possible to introduce semantic caching without changing the fundamental model-serving workflow of the application. The trade-off is that semantic caching adds another decision to the system: whether a previous answer is sufficiently close to the current request to be reused. That decision should be conservative, observable, and driven by application requirements.
What’s Next
Caching has always been about avoiding work that has already been done. Generative AI does not change that goal. It changes the definition of what it means for work to be reusable.
In traditional systems, the answer might be determined by an exact key. In generative AI applications, the answer can depend on semantic intent. Embeddings provide a way to represent that intent as vectors, and vector search provides a mechanism for finding previously answered requests that are close in meaning. Amazon Bedrock provides the foundation-model and embedding capabilities, while services such as Amazon ElastiCache for Valkey can provide the low-latency vector search and cache-management layer. The result is an architecture that can reduce unnecessary model invocations, lower latency, and improve the responsiveness of generative AI applications when users repeatedly ask for similar information.
The key is to treat semantic caching as an engineering control rather than a magic optimization. Similarity thresholds, expiration, authorization scope, model and prompt versions, source-data freshness, and response quality all matter. A semantic cache that is designed carefully can become an effective layer between users and expensive downstream inference. A semantic cache that is designed carelessly can return an answer that is fast, cheap, and wrong.
For teams building generative AI applications on AWS, semantic caching is therefore worth considering as part of the broader performance architecture. The right approach, however, will depend on the application, the data, the users, and the business requirements. There is no single semantic caching architecture that fits every workload.
If you are exploring how semantic caching could work in your own environment, we would be happy to help you assess the opportunities and trade-offs. Together, we can look at your existing generative AI architecture, identify where semantic caching could provide the most value, and design an approach around your specific requirements for performance, cost, security, and response quality. Reach out to us to explore how we can adapt this architecture to your business and build a solution that fits your environment.
References
[1] “Amazon Titan Text Embeddings models”, AWS Docs,
https://docs.aws.amazon.com/bedrock/latest/userguide/titan-embedding-models.html
[2] Yujian Tang, “Similarity Metrics for Vector Search”, Zilliz, 11 Mar 2025,
https://zilliz.com/blog/similarity-metrics-for-vector-search
[3] Chaitanya Nuthalapati, Chenhe Liang, Christoph Koerner, and Ohad Shacham, “Announcing vector search for Amazon ElastiCache”, AWS Blogs, 13 Oct 2025,
https://aws.amazon.com/blogs/database/announcing-vector-search-for-amazon-elasticache/
[4] “Prompt caching for faster model inference”, AWS Docs,
https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html
[5] “Implementing a semantic cache with ElastiCache for Valkey”, AWS Docs,
https://docs.aws.amazon.com/AmazonElastiCache/latest/dg/semantic-caching-implementation.html
[6] Corvus Lee, Ahmed Ewis, and Chris Pecora, “Amazon Bedrock Knowledge Bases now supports metadata filtering to improve retrieval accuracy”, AWS Blogs, 08 Apr 2024,
[7] Meet Bhagdev, Chaitanya Nuthalapati, Jungwoo Song, and Utkarsh Shah, “Lower cost and latency for AI using Amazon ElastiCache as a semantic cache with Amazon Bedrock”, AWS Blogs,
[8] “Use the ApplyGuardrail API in your application”, AWS Docs,
https://docs.aws.amazon.com/bedrock/latest/userguide/guardrails-use-independent-api.html
[9] “Model Invocations”, AWS Docs,
https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/model-invocations.html
[10] “Monitor model invocation using CloudWatch Logs and Amazon S3”, AWS Docs,
https://docs.aws.amazon.com/bedrock/latest/userguide/model-invocation-logging.html
[11] “Reducing hallucinations in LLM agents with a verified semantic cache using Amazon Bedrock Knowledge Bases”, AWS Blogs,
Relevant Success Stories
.png)
.png)
Book a meeting
Ready to unlock more value from your cloud? Whether you're exploring a migration, optimizing costs, or building with AI—we're here to help. Book a free consultation with our team and let's find the right solution for your goals.