Deployment contract
SenseVoice native OpenAI realtime server
Use one C++ binary with no Python runtime to serve SenseVoice over REST, SSE, SRT/VTT, and OpenAI realtime WebSocket transcription.
Workload boundary
Decide whether it fits your production constraints
Good fit
- A local or private OpenAI-compatible speech endpoint
- CPU-first deployment with optional CUDA and few runtime dependencies
- File transcription, streaming partial/final results, and server-side VAD
Not a fit
- Prebuilt cross-platform release packages are required
- Built-in multi-tenant authentication, billing, or durable job queues are required
- Public exposure before capacity testing on the target hardware
| Models | SenseVoiceSmall-GGUF / FSMN-VAD-GGUF |
|---|---|
| Hardware | cpu / nvidia-gpu |
| Operating systems | Linux |
| Interfaces | OpenAI-compatible HTTP / OpenAI realtime WebSocket / WS /v1/realtime?intent=transcription / SSE / SRT/VTT |
Run path
From installation to known-audio verification
These commands come from the current verification registry. Pin dependencies, model, and hardware before rollout.
Install
git clone https://github.com/QwenAudio/SenseVoice.git && cd SenseVoicegit checkout b054623cca8f015b73ec471dce4f473ac47413dapython3 -m pip install -U huggingface_hubbash runtime/llama.cpp/download-funasr-model.sh sensevoice modelcmake -S runtime/llama.cpp -B runtime/llama.cpp/build -DCMAKE_BUILD_TYPE=Releasecmake --build runtime/llama.cpp/build --target sensevoice-server -jLaunch
runtime/llama.cpp/build/bin/sensevoice-server -m model/sensevoice-small-q8.gguf -vad model/fsmn-vad.gguf --threads 8 --max-connections 4 --max-audio-seconds 300 127.0.0.1 8040Health check
curl -fsS http://127.0.0.1:8040/healthcurl -fsS http://127.0.0.1:8040/v1/modelsSmoke test
Minimum verification before promotion
curl -fsS http://127.0.0.1:8040/v1/audio/transcriptions -F file=@speech.wavcurl -fsS http://127.0.0.1:8040/v1/audio/transcriptions -F file=@speech.wav -F response_format=vtt | tee transcript.vtt && grep -q WEBVTT transcript.vttcurl -fsSN http://127.0.0.1:8040/v1/audio/transcriptions -F file=@speech.wav -F stream=true | grep -q transcript.text.donepython3 runtime/llama.cpp/tests/stream_client.py 127.0.0.1 8040 speech_16k.wav 200Operations and capacity
Move from runnable to operable
Operational checks
- Pin the merge commit plus the SenseVoice and FSMN-VAD GGUF files
- Set the connection limit, per-request audio limit, and read timeout explicitly
- Load-test REST, SSE, and WebSocket separately before rollout, and retain the previous binary for rollback
Capacity variables
- Audio duration, language, channels, and VAD segment distribution
- Concurrency, queue time, warmup, and model-cache state
- Exact hardware, driver, runtime, and thread configuration
Troubleshooting
- Validate CPU, a known 16 kHz WAV, and the REST JSON path first
- Confirm the model files are named sensevoice-small-q8.gguf and fsmn-vad.gguf
- Send base64 chunks of 16 kHz mono PCM16 from WebSocket clients
Security boundary
Treat production ingress as untrusted
- Bind to 127.0.0.1 or a trusted private network by default
- Enforce TLS, authentication, MIME, request-body size, rate, and concurrency at the reverse proxy
- Load only GGUF files with verified provenance and digests
Known limitation
Official verification currently covers Linux source builds; the server does not include production authentication, TLS, tenant quotas, or durable queues, so provide them at the gateway and infrastructure layers.
Public benchmarks are reproduction starting points, not substitutes for target-workload testing.
Evidence and feedback