plugnmeet-local-insights

A self-hosted, CPU-friendly transcription and translation service for plugNmeet. Speaks the minimal WebSocket protocol expected by the local provider in plugNmeet.

• Transcription: faster-whisper with int8 quantization and streaming via whisper_streaming.
• Translation (optional): NLLB-200-distilled-600M via CTranslate2. See the license notice below.

The reference container runs end-to-end on a modest CPU (tested on ARM64 Neoverse-N1, 10 cores, ~2 GHz).

Quick start

docker build -t plugnmeet-local-insights .
docker run -d --name whisper-local \
  --network plugnmeet_net \
  -p 8002:8002 \
  plugnmeet-local-insights

Then point plugNmeet's config.yaml at it:

insights:
  enabled: true
  providers:
    local:
      - id: "local-01"
        credentials: { api_key: "", region: "" }
        options:
          whisper_url: "ws://whisper-local:8002/ws/transcribe"
          translate_url: "http://whisper-local:8002/translate"
  services:
    transcription: { provider: "local", id: "local-01" }
    translation:   { provider: "local", id: "local-01" }

Protocol

WebSocket /ws/transcribe

Real-time transcription. First message is a JSON handshake, followed by binary PCM16 audio frames at 16000 Hz mono.

Client → Server  {"type":"start","lang":"de","transLangs":["en"],"userName":"Alice"}
Client → Server  <binary PCM16 frames, any size>
Server → Client  {"type":"partial","text":"hallo","lang":"de"}
Server → Client  {"type":"final","text":"hallo welt","lang":"de"}
Server → Client  {"type":"final","text":"hello world","lang":"en"}
Server → Client  {"type":"error","error":"..."}
Client → Server  {"type":"end"}

Behaviour:

• partial messages are speculative (may be revised).
• final messages are confirmed, stable text.
• Per-target-language translations are emitted only for final results (translating unstable partials wastes cycles).
• The server drops the connection on malformed input; reconnect is the client's responsibility.

HTTP POST /translate

Azure Translation API–compatible for one-shot batches:

curl -X POST "http://whisper-local:8002/translate?from=de&to=en&to=fr" \
  -H "Content-Type: application/json" \
  -d '[{"Text":"Guten Morgen"}]'

[{"translations":[
  {"text":"Good morning","to":"en"},
  {"text":"Bonjour","to":"fr"}
]}]

HTTP GET /health

curl http://whisper-local:8002/health
{"status":"ok"}

Environment variables

Variable	Default	Purpose
WHISPER_MODEL	small	faster-whisper model size
WHISPER_DEVICE	cpu	cpu or cuda
WHISPER_COMPUTE_TYPE	int8	quantization
TRANSLATE_DEVICE	cpu	NLLB device
NLLB_CT2_DIR	/app/models/nllb-ct2	pre-converted CT2 model path
PORT	8002	HTTP port

Latency & resource notes

Defaults are tuned for CPU real-time:

• 500 ms audio chunks (trades precision for responsiveness).
• beam_size=1 for Whisper decoding.
• VAD filter enabled — silence is skipped instead of hallucinated.
• buffer_trimming=("segment", 8) — caps re-transcription window to 8 s.
• NLLB: beam_size=2, all target languages batched in a single CTranslate2 call (multi-language translation takes roughly the same time as one language).
• Partial results are emitted from the uncommitted prefix of the transcription buffer for lower perceived latency; translations run only on confirmed finals.

Raise WHISPER_MODEL to medium or large-v3 on GPU for higher quality.

Licenses

The code in this repository is MIT. Model licenses differ:

• faster-whisper / Whisper — MIT.
• NLLB-200 (translation) — CC-BY-NC 4.0, non-commercial. If you need translation for commercial use, swap in a permissive model (e.g. opus-mt-*, MIT/CC-BY) or a commercial API. Transcription (Whisper) is unaffected.

Building this image downloads the NLLB-200 checkpoint from Hugging Face at build time. If you only need transcription, point plugNmeet's config.yaml at the service without setting translate_url — the translation model will never load.