Reduce Gemini Costs and Latency with Vertex AI Context Caching

As developers build increasingly complex AI applications, they often face the challenge of repeatedly sending large amounts of contextual information to their models. This can include lengthy documents, detailed instructions, or extensive codebases. While this context is crucial for accurate responses, it can significantly increase both costs and latency. To address this, Google Cloud introduced Vertex AI context caching in 2024, a feature designed to optimize Gemini model performance.

What is Vertex AI Context Caching?

Vertex AI context caching allows developers to save and reuse precomputed input tokens, reducing the need for redundant processing. This results in both cost savings and improved latency. The system offers two primary types of caching: implicit and explicit.

Implicit Caching

Implicit caching is enabled by default for all Google Cloud projects. It automatically caches tokens when repeated content is detected. The system then reuses these cached tokens in subsequent requests. This process happens seamlessly, without requiring any modifications to your API calls. Cost savings are automatically passed on when cache hits occur. Caches are typically deleted within 24 hours, based on overall load and reuse frequency.

Explicit Caching

Explicit caching provides users with greater control. You explicitly declare the content to be cached, allowing you to manage which information is stored and reused. This method guarantees predictable cost savings. Furthermore, explicit caches can be encrypted using Customer Managed Encryption Keys (CMEKs) to enhance security and compliance.

Vertex AI context caching supports a wide range of use cases and prompt sizes. Caching is enabled from a minimum of 2,048 tokens up to the model’s context window size – over 1 million tokens for Gemini 2.5 Pro. Cached content can include text, PDFs, images, audio, and video, making it versatile for various applications. Both implicit and explicit caching work across global and regional endpoints. Implicit caching is integrated with Provisioned Throughput to ensure production-grade traffic benefits from caching.

Ideal Use Cases for Context Caching

Context caching is beneficial across many applications. Here are a few examples:

• Large-Scale Document Processing: Cache extensive documents like contracts, case files, or research papers. This allows for efficient querying of specific clauses or information without repeatedly processing the entire document. For instance, a financial analyst could upload and cache numerous annual reports to facilitate repeated analysis and summarization requests.
• Customer Support Chatbots/Conversational Agents: Cache detailed instructions and persona definitions for chatbots. This ensures consistent responses and allows chatbots to quickly access relevant information, leading to faster response times and reduced costs.
• Coding: Improve codebase Q&A, autocomplete, bug fixing, and feature development by caching your codebase.
• Enterprise Knowledge Bases (Q&A): Cache complex technical documentation or internal wikis to provide employees with quick answers to questions about internal processes or technical specifications.

Cost Implications: Implicit vs. Explicit Caching

Understanding the cost implications of each caching method is crucial for optimization.

• Implicit Caching: Enabled by default, you are charged standard input token costs for writing to the cache, but you automatically receive a discount when cache hits occur.
• Explicit Caching: When creating a CachedContent object, you pay a one-time fee for the initial caching of tokens (standard input token cost). Subsequent usage of cached content in a generate_content request is billed at a 90% discount compared to regular input tokens. You are also charged for the storage duration (TTL – Time-To-Live), based on an hourly rate per million tokens, prorated to the minute.

Best Practices and Optimization

To maximize the benefits of context caching, consider the following best practices:

• Check Limitations: Ensure you are within the caching limitations, such as the minimum cache size and supported models.
• Granularity: Place the cached/repeated portion of your context at the beginning of your prompt. Avoid caching small, frequently changing pieces.
• Monitor Usage and Costs: Regularly review your Google Cloud billing reports to understand the impact of caching on your expenses. The cachedContentTokenCount in the UsageMetadata provides insights into the number of tokens cached.
• TTL Management (Explicit Caching): Carefully set the TTL. A longer TTL reduces recreation overhead but incurs more storage costs. Balance this based on the relevance and access frequency of your context.

Context caching is a powerful tool for optimizing AI application performance and cost-efficiency. By intelligently leveraging this feature, you can significantly reduce redundant token processing, achieve faster response times, and build more scalable and cost-effective generative AI solutions. Implicit caching is enabled by default for all GCP projects, so you can get started today.

For explicit caching, consult the official documentation and explore the provided Colab notebook for examples and code snippets.

By using Vertex AI context caching, Google Cloud users can significantly reduce costs and latency when working with Gemini models. This technology, available since 2024, offers both implicit and explicit caching options, each with unique advantages. The financial analyst, the customer support chatbot, and the coder can improve their workflow by using context caching. By following best practices and understanding the cost implications, developers can build more efficient and scalable AI applications. Explicit Caching allows for more control over the data that is cached.

To get started with explicit caching check out our documentation and a Colab notebook with common examples and code.

Source: Google Cloud Blog