Deployment contract
Native FunASR on vLLM
Use vLLM's built-in FunASR architecture for multilingual and hotword-aware transcription through an OpenAI-compatible API on NVIDIA GPUs.
Workload boundary
Decide whether it fits your production constraints
Good fit
- A standard /v1/audio/transcriptions endpoint
- Concurrent or batch Fun-ASR-Nano transcription
- An NVIDIA GPU Linux serving environment
Not a fit
- Deployments that accept only officially published weights and cannot use a community conversion
- CPU-only or edge devices
- Internet-facing service before memory, concurrency, and workload tests
| Models | Fun-ASR-Nano-2512 (community vLLM conversion) |
|---|---|
| Hardware | nvidia-gpu / kubernetes |
| Operating systems | Linux |
| Interfaces | OpenAI-compatible HTTP |
Run path
From installation to known-audio verification
These commands come from the current verification registry. Pin dependencies, model, and hardware before rollout.
Install
curl -fL https://github.com/vllm-project/vllm/releases/download/v0.27.1/vllm-0.27.1%2Bcu129-cp38-abi3-manylinux_2_28_x86_64.whl -o vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whlecho "bf0d52faa2a51e7a01c6856a7a8a2d1307fd0ff711415d34168a67ffac0fa47b vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl" | sha256sum -c -uv venv --python 3.12 .venv && uv pip install --python .venv/bin/python --torch-backend=auto "vllm[audio] @ file://$PWD/vllm-0.27.1+cu129-cp38-abi3-manylinux_2_28_x86_64.whl"Launch
CUDA_VISIBLE_DEVICES=0 .venv/bin/vllm serve allendou/Fun-ASR-Nano-2512-vllm --revision e718b36e2578203ec893e9b488239225f8d668e2 --served-model-name fun-asr-nano --host 127.0.0.1 --port 8899 --dtype float32 --gpu-memory-utilization 0.40 --enforce-eagerHealth check
curl -fsS http://127.0.0.1:8899/healthcurl -fsS http://127.0.0.1:8899/v1/modelsSmoke test
Minimum verification before promotion
curl -fL https://huggingface.co/allendou/Fun-ASR-Nano-2512-vllm/resolve/e718b36e2578203ec893e9b488239225f8d668e2/example/zh.mp3 -o zh.mp3 && echo "0e64de19e4ff9a02e682955c9112f32d2317cfdbb5bc2f3504664044c993f195 zh.mp3" | sha256sum -c -curl -fsS http://127.0.0.1:8899/v1/audio/transcriptions -F file=@zh.mp3 -F model=fun-asr-nano -F language=zh -F response_format=jsoncurl -fsS http://127.0.0.1:8899/v1/audio/transcriptions -F file=@zh.mp3 -F model=fun-asr-nano -F language=zh -F 'hotwords=开放时间,开放时间,开放时间' -F response_format=jsonOperations and capacity
Move from runnable to operable
Operational checks
- Pin the vLLM wheel, Torch, CUDA, model revision, and audio dependencies
- Wait for /health and warm the model before measuring capacity
- Measure queue time, final-result latency, memory, and hotword hit rate
Capacity variables
- Audio duration, language, channels, and VAD segment distribution
- Concurrency, queue time, warmup, and model-cache state
- Exact hardware, driver, runtime, and thread configuration
Troubleshooting
- For audio HTTP 400 responses, confirm vllm[audio] is installed
- If startup reports insufficient KV cache, raise GPU utilization or lower max-model-len only when the workload permits
- Pass language explicitly for non-English audio; comma-separate hotwords and validate bias strength on production samples
Security boundary
Treat production ingress as untrusted
- Put authentication, TLS, rate limits, and audio size/duration limits at the gateway
- Bind workers to a private address instead of exposing them directly
- Verify wheel, model revision, and sample digests; isolate model cache and temporary uploads
Known limitation
The native vLLM path currently depends on a community-converted checkpoint; use the official FunASR split-engine when an official-weight chain is required. Evidence covers one H100 only, so retest the target GPU, audio, languages, hotwords, and traffic.
Public benchmarks are reproduction starting points, not substitutes for target-workload testing.
Evidence and feedback
