KVCache.AI is a joint research project between MADSys and top industry collaborators, focusing on efficient LLM serving.
15
Public repositories
22,965
Total stars
1,146
Followers
kvcache-ai has a significant presence on GitHub, showcasing several widely used projects focused on efficient LLM serving. The organization primarily uses languages such as Python, Cuda, C++, Go, and JavaScript. Notable repositories include ktransformers, Mooncake, and vllm, reflecting its commitment to advancing large language model technologies.
Top languages
Public repositories
ktransformers
A Flexible Framework for Experiencing Heterogeneous LLM Inference/Fine-tune Optimizations
Mooncake
Mooncake is the serving platform for Kimi, a leading LLM service provided by Moonshot AI.
TrEnv-X
No description provided for this repository.
vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
kvcache-blog
No description provided for this repository.
sglang
SGLang is a fast serving framework for large language models and vision language models.
custom_flashinfer
FlashInfer: Kernel Library for LLM Serving
DeepEP_fault_tolerance
DeepEP: an efficient expert-parallel communication library that supports fault tolerance
sglang_awq
SGLang is a fast serving framework for large language models and vision language models.
accelerate
🚀 A simple way to launch, train, and use PyTorch models on almost any device and distributed configuration, automatic mixed precision (including fp8), and easy-to-configure FSDP and DeepSpeed support
Model-Optimizer
A unified library of SOTA model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.
evalscope
A streamlined and customizable framework for efficient large model (LLM, VLM, AIGC) evaluation and performance benchmarking.
transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
gpustack
GPU cluster manager for optimized AI model deployment
sglang-npu
SGLang is a fast serving framework for large language models and vision language models.
Frequently asked questions
What does kvcache-ai build on GitHub?
kvcache-ai builds a variety of projects on GitHub, primarily focused on efficient serving of large language models. Key repositories include ktransformers and Mooncake, which contribute to advancements in LLM inference and optimization.
Which programming languages does kvcache-ai use?
kvcache-ai utilizes several programming languages for its projects, including Python, Cuda, C++, Go, and JavaScript. This diverse language use supports their focus on efficient LLM serving and enhances project flexibility.
Are kvcache-ai's repositories public?
Yes, kvcache-ai's repositories are public on GitHub. This transparency allows the community to access and contribute to their projects, fostering collaboration and innovation in the field of large language models.
Is this exposure intended?
Monitor kvcache.ai with RepoGuard and get alerted the moment a new public repository appears.
Monitor this account