Project Overview
FlashInfer is an open-source CUDA library designed to accelerate large language model (LLM) inference on NVIDIA GPUs. It includes various core kernels such as attention, GEMM, and MoE, aiming to improve inference performance.
Core Features
- Optimized attention kernels for LLM inference.
- Efficient GEMM (General Matrix Multiplication) kernels.
- Kernel implementations for Mixture of Experts (MoE) models.
Technology Stack
The project is primarily written in Python, built on CUDA technology, and supports NVIDIA GPU architectures from Turing through Blackwell.
License
FlashInfer is licensed under Apache-2.0, allowing users freedom to use, modify, and distribute.
Open Source Status
As an open-source project, FlashInfer is publicly available on GitHub and currently has 5876 stars, reflecting an active community.










Comments
No comments yet
Be the first to comment