Inference optimization AI Tool Directory

BitNet

Official inference framework for 1-bit LLMs

Open Source Framework Inference optimization

llama.cpp

LLM inference in C/C++

Open Source Inference optimization

llama2.c

Inference Llama 2 in one file of pure C

Open Source Inference optimization

cutlass

CUDA Templates for Linear Algebra Subroutines

Open Source Inference optimization

vector-quantize-pytorch

Vector (and Scalar) Quantization, in Pytorch

Open Source Inference optimization

llamafile

Distribute and run LLMs with a single file.

Open Source Inference optimization

ao

PyTorch native quantization and sparsity for training and inference

Open Source Inference optimization

whisper.cpp

Port of OpenAI's Whisper model in C/C++

Open Source Artificial Intelligence Inference optimization

flash-attention

Fast and memory-efficient exact attention

Open Source Inference optimization

ggml

Tensor library for machine learning

Open Source Inference optimization

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Open Source Inference optimization

mlc-llm

Universal LLM Deployment Engine with ML Compilation

Open Source Inference optimization

Filter

All Inference optimization Listings

Loading

Loading

Loading

Loading

Loading

Loading

Loading

Loading

Loading

Loading

Loading

Loading

Filter

BitNet

llama.cpp

llama2.c

cutlass

vector-quantize-pytorch

llamafile

ao

whisper.cpp

flash-attention

ggml

smoothquant

mlc-llm

Loading

Loading

Loading

Loading

Loading

Loading

BitNet

llama.cpp

llama2.c

cutlass

vector-quantize-pytorch

llamafile

ao

whisper.cpp

flash-attention

ggml

smoothquant

mlc-llm