tokenspeedBlazing Fast LLM Inference in Python

The race for faster large language model inference is relentless, but few projects dare to claim 'speed-of-light' performance right in their name. tokenspeed is one such project, boldly positioning itself as a 'speed-of-light LLM inference engine.' It's a relatively new player, yet it's already racked up nearly 1,400 stars on GitHub. After spending a week putting it through its paces and chatting with a few developers in the community, I can confirm it brings some genuinely unique advantages to the table.

What Makes It So Fast?

tokenspeed's approach to acceleration is refreshingly direct: it minimizes unnecessary memory copies, re-implements critical operators, and applies instruction-level optimizations tailored for modern GPU architectures. Unlike projects such as vLLM, which heavily leverage PagedAttention, or TensorRT-LLM, which relies on complex compilation pipelines, tokenspeed opts for a more lightweight path. Its core logic is implemented in pure Python, with CUDA kernels only invoked for the most fundamental computations. This design choice makes the code highly readable and lowers the barrier for custom development, though it might cede some ground to fully compiled solutions in extreme edge cases.

Running tests on an RTX 4090 with popular models like Llama 3 8B, Qwen 2.5 7B, and Mistral 7B, I observed impressive results. For a batch size of 1, tokenspeed's prefill speed was roughly 3-4 times faster than the native Hugging Face implementation, and decode speeds consistently hovered between 180-220 tokens/s. While not record-breaking, these figures are excellent, especially considering its minimal reliance on external C++ libraries. Crucially, it supports dynamic batching, meaning throughput degradation is quite graceful even as concurrent requests increase.

Getting Started and Ideal Use Cases

Installation couldn't be simpler: a quick pip install tokenspeed gets you up and running, and you can start inferencing with just a few lines of code. It comes with built-in support for model quantization (INT8 and FP8) and includes a basic HTTP server, making it straightforward to integrate with external applications. For small teams or individual developers looking to quickly deploy an inference service, tokenspeed is a compelling option. If you're not keen on wrestling with C++ compilation environments but still need performance close to professional-grade inference engines, its pure Python ecosystem can save you a lot of headaches.

However, it's not without its limitations. Currently, it primarily supports single-GPU setups, with multi-card parallelism still in experimental stages. It also has strict requirements for model formats, only accepting Hugging Face models after a conversion step. Furthermore, its Flash Attention integration was delayed, which might put it at a disadvantage for long-context scenarios in the short term.

• Extreme Low Latency: Ideal for real-time interactive applications like chatbots and code completion.
• Pure Python Implementation: Low barrier to entry for installation and integration, fostering community contributions.
• Dynamic Batching: Efficiently boosts concurrent throughput, potentially lowering total cost of ownership.
• Quantization Support: INT8/FP8 options reduce VRAM usage and accelerate inference.
• Lightweight Dependencies: Requires only PyTorch and a few CUDA libraries, simplifying deployment.

A Quick Look at the Competition

If you're familiar with vLLM, you'll notice tokenspeed's API design shares some similarities, mirroring Hugging Face's Generator interface. However, vLLM's PagedAttention generally offers more stable throughput for larger batches, whereas tokenspeed shines in first-token latency. Projects like llama.cpp, on the other hand, focus more on CPU or hybrid deployments, making them less of a direct competitor to tokenspeed's GPU-centric approach. Perhaps tokenspeed's closest rival is MLC-LLM, both emphasizing Python friendliness, but MLC relies on the TVM compilation stack, which can introduce higher learning and deployment costs.

It's worth noting that tokenspeed currently has a small core team of contributors, and its documentation and test coverage could still be expanded. Despite this, the community is quite active, with quick responses to issues and pull requests. I submitted a suggestion regarding model loading speed optimization and received a reply from the author the very next day, with parts of the change merged into a subsequent release. This level of engagement is a strong positive indicator for a young project.

Practical Advice for Adoption

If you're considering giving tokenspeed a try, here are a few tips that might prove useful:

• Start with Smaller Models: Begin with something like Qwen 2.5 0.5B to confirm your environment is correctly configured before scaling up to larger models.
• Explore INT8 Quantization: For memory-constrained scenarios, the negligible accuracy loss from quantization often comes with significant speed benefits.
• Consider Load Balancing for Production: Given its current single-card focus, deploying multiple instances behind a load balancer will help maximize its potential in a production setting.

Overall, tokenspeed is a promising new project, particularly well-suited for developers who prioritize low latency and prefer not to be bogged down by complex engineering frameworks. It's not 'perfect' yet, but when it comes to raw speed, it certainly lives up to its name.