Inference Optimization

Filter

Category

Search with AI

All Inference optimization Listings

Cover Image

Model Development 📊🧠

llama.cpp

LLM inference in C/C++

Open Source Inference optimization

Cover Image

Model Development 📊🧠

TensorRT-LLM

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

Open Source Inference optimization

Cover Image

Model Development 📊🧠

flash-attention

Fast and memory-efficient exact attention

Open Source Inference optimization

Cover Image

Model Development 📊🧠

ggml

Tensor library for machine learning

Open Source Inference optimization

Cover Image

Model Development 📊🧠

unsloth

Finetune Llama 3.3, Mistral, Phi, Qwen 2.5 & Gemma LLMs 2-5x faster with 80% less memory

Open Source Inference optimization

Cover Image

Model Development 📊🧠

whisper.cpp

Port of OpenAI's Whisper model in C/C++

Open Source Artificial Intelligence Inference optimization

Cover Image

Model Development 📊🧠

vector-quantize-pytorch

Vector (and Scalar) Quantization, in Pytorch

Open Source Inference optimization

Cover Image

Model Development 📊🧠

mlc-llm

Universal LLM Deployment Engine with ML Compilation

Open Source Inference optimization

Cover Image

Model Development 📊🧠

Torch-Pruning

[CVPR 2023] DepGraph: Towards Any Structural Pruning

Open Source Inference optimization

Cover Image

Model Development 📊🧠

BitNet

Official inference framework for 1-bit LLMs

Open Source Framework Inference optimization

Cover Image

Model Development 📊🧠

llama2.c

Inference Llama 2 in one file of pure C

Open Source Inference optimization

Cover Image

Model Development 📊🧠

llm.c

LLM training in simple, raw C/CUDA

Open Source Inference optimization

Listings per page

Showing1 - 12of16listings

Page 1 of 2