Deployment center

Choose a runtime from product constraints

Every path records fit, verified versions, startup commands, production responsibilities, and known limitations.

Paths
10
Languages
2
Verified
2026-08-13
Community verified 2026-08-13

Native FunASR on vLLM

Use vLLM's built-in FunASR architecture for multilingual and hotword-aware transcription through an OpenAI-compatible API on NVIDIA GPUs.

Hardware
nvidia-gpu / kubernetes
Models
Fun-ASR-Nano-2512 (community vLLM conversion)
Limitation

The native vLLM path currently depends on a community-converted checkpoint; use the official FunASR split-engine when an official-weight chain is required. Evidence covers one H100 only, so retest the target GPU, audio, languages, hotwords, and traffic.

View deployment contract
Production verified 2026-08-04

SenseVoice TensorRT / Triton

Build SenseVoiceSmall as a native FP16 TensorRT engine and serve batched GPU inference through Triton.

Hardware
nvidia-gpu / kubernetes
Models
SenseVoiceSmall
Limitation

A TensorRT plan is not portable across arbitrary GPU architectures or TensorRT versions; rebuild and revalidate accuracy, memory, and capacity on the target stack.

View deployment contract
Production verified 2026-08-11

llama.cpp / GGUF standalone

Use the nine v0.2.0 release packages or source builds to run FunASR GGUF models on CPU, Vulkan, CUDA, and edge devices.

Hardware
cpu / desktop-edge-gpu
Models
SenseVoiceSmall-GGUF / Paraformer-GGUF
Limitation

Prebuilt GPU packages cover only the documented backend and architecture; the Windows AMD Vulkan crash fix still awaits reporter hardware confirmation, and every other target also requires hardware validation.

View deployment contract
Production verified 2026-08-13

SenseVoice native OpenAI realtime server

Use one C++ binary with no Python runtime to serve SenseVoice over REST, SSE, SRT/VTT, and OpenAI realtime WebSocket transcription.

Hardware
cpu / nvidia-gpu
Models
SenseVoiceSmall-GGUF / FSMN-VAD-GGUF
Limitation

Official verification currently covers Linux source builds; the server does not include production authentication, TLS, tenant quotas, or durable queues, so provide them at the gateway and infrastructure layers.

View deployment contract
Community verified 2026-08-13

audio.cpp native Fun-ASR-Nano and SenseVoice

Run Fun-ASR-Nano and SenseVoice Q8 with native C++ and GGML on CPU or GPU, using offline CLI, a local OpenAI-compatible API, and buffered SenseVoice streaming results.

Hardware
cpu / nvidia-gpu / desktop-edge-gpu
Models
Fun-ASR-Nano-2512 / SenseVoice-Small
Limitation

SenseVoice is merged into audio.cpp main@979e070f but is not yet in a tagged release. Pin that main commit until a release is published. Neither path currently provides word-level timestamps.

View deployment contract
Production verified 2026-07-26

OpenAI-compatible private API

Serve FunASR models through `/v1/audio/transcriptions` for existing SDKs, agents, and workflows.

Hardware
cpu / nvidia-gpu / kubernetes
Models
sensevoice / paraformer
Limitation

The example server does not include production authentication, tenant quotas, or full observability; provide them at the gateway and infrastructure layers.

View deployment contract
Community verified 2026-08-17

Realtime streaming and captions

Use online or two-pass WebSocket protocols for microphones, meeting captions, and continuous speech streams.

Hardware
cpu / nvidia-gpu / kubernetes
Models
Paraformer-online / FSMN-VAD
Limitation

Capacity depends on realtime ratio, chunks, VAD, connection count, and client backpressure; file RTFx does not replace concurrency tests.

View deployment contract
Community verified 2026-07-26

Containers and Kubernetes

Validate with Docker Compose, then deploy with the Kubernetes template, persistent model cache, probes, and private ClusterIP.

Hardware
kubernetes / cpu / nvidia-gpu
Models
sensevoice / paraformer
Limitation

The CPU example is not a CUDA production image; GPU drivers, image, scheduling, and capacity still require platform validation.

View deployment contract
Community verified 2026-07-26

ONNX / C++ CPU runtime

Deploy offline, online, and two-pass ASR on general CPUs with ONNX Runtime and C++ paths.

Hardware
cpu
Models
Paraformer / Paraformer-online
Limitation

Model conversion, operator support, and thread settings vary by platform; measure capacity on the target CPU with real audio.

View deployment contract
Production verified 2026-07-26

Production rollout checklist

Move from a runnable model to an observable, scalable, reversible production service with explicit boundaries.

Hardware
cpu / nvidia-gpu / desktop-edge-gpu / kubernetes
Models
All supported deployment models
Limitation

A checklist does not replace workload, accuracy, failure, and security exercises for the target product.

View deployment contract