IntermediatePython

LMCacheOpen-Source KV Cache Layer for Lower First-Token Latency

LMCache is an open-source KV cache layer that reuses attention state across LLM requests and engines, cutting first-token latency for long-context and RAG workloads. It is primarily written in Python and licensed under Apache-2.0. As of collection time, it has 10,183 GitHub stars.

10.2K Stars

1.5K Forks

392 Issues

110 Views

Python

Apache-2.0

IndexedJuly 9, 2026

GitHub repository Online Demo

Project Overview

LMCache is an open-source KV cache layer designed to improve the performance of LLM services by reusing attention states. It enables KV cache reuse across requests and engines, reducing redundant computation and lowering first-token latency, especially for long-context and RAG workloads.

Core Features

Reuse KV cache across requests: Share attention states between requests to avoid repeated computation.
Compatibility across engines: Supports multiple inference engines, facilishing integration into existing systems.
Reduced first-token latency: Speeds up response time via cache reuse.

Tech Stack and License

The project is primarily written in Python and follows the Apache-2.0 license. As of collection time, it has 10,183 stars on GitHub.

Getting Started

According to the README, users can start by cloning the repository and installing dependencies. For detailed installation and configuration steps, please refer to the project documentation.

LLM inference accelerationKV cacheLLM optimizationopen-source toolslatency reductionlarge model deploymentPython utilityLMCacheefficiency toolcaching layer

Sources & Verification

This entry is compiled and reviewed against the project’s primary sources:

Last verified:: July 9, 2026
Reviewed by:: SoFarBot Editorial Team

We confirm the project resolves to its real repository, classify it, and label its open-source license and language. Metrics such as stars and last commit reflect the upstream repository and are refreshed as it changes.

Project Rating

0.0 (0 Reviews)

Frequently Asked Questions

What is LMCache: Open-Source KV Cache Layer for Lower First-Token Latency?

What language is LMCache: Open-Source KV Cache Layer for Lower First-Token Latency written in?

LMCache: Open-Source KV Cache Layer for Lower First-Token Latency is primarily written in Python.

What license is LMCache: Open-Source KV Cache Layer for Lower First-Token Latency under?

LMCache: Open-Source KV Cache Layer for Lower First-Token Latency is released under the Apache-2.0 license.

Related Projects

No results yet

Explore More

Similar Tools

Nika

Nika is a suite of AI companion products, including the Nika AI girlfriend, the Sebastian AI boyfriend and the Aurora City dating simulator. It offers text, voice and multimedia roleplay through Telegram and a web app, with a level-based bond system.

Filently

Filently is a Swiss-built AI tool that automatically renames and files Google Drive documents, learning your existing folders and reading dates, vendors and amounts by OCR.

PakBot

PakBot is billed as the first AI assistant made for Pakistan, with free chat in Urdu, English, Punjabi, Sindhi and Pashto, plus image generation, voice chat and web search.

WeiClaw

WeiClaw is a hardware device that pairs with Agent-running computers, watching Agent status and toggling the host between sleep and active on demand.

Ruler Online Free

Ruler Online Free is an online ruler tool that displays a real-size ruler on your screen, calibrated to your exact display for accurate measurements. It supports cm, mm, and inches, and can print to PDF (12-inch, 8-inch, protractor). No signup, no ads, no install. Works on any device with a modern browser.