deepstream-sop · v1.0.0 · 2026-07-01 · sha256 9030b35061302ba3
deepstream-sop v1.0.0A
Immutable. This exact content is served forever at /api/v1/blob/9030b35061302ba3.
---
name: "deepstream-sop"
description: >
Use this skill when building, deploying, evaluating, debugging, or measuring
latency for the DeepStream SOP Inference Microservice — a GPU-accelerated
FastAPI service that detects whether operators perform assembly-line steps in
order via event boundary detection (GEBD) plus VLM classification. Trigger
even if the user does not name it: verify operator step sequence, detect
missing or out-of-order SOP steps, score factory/work-cell video for
procedure compliance, run VLM-based SOP checking on industrial cameras, or
call /v1/chat/completions with a file, RTSP, or Basler camera. Also trigger
for its internals: SOPVideoProcessor, DeepStream GEBD model (e.g. DDM) via
Triton CAPI, nvds_custom_postprocess, Cosmos Reason 1/2 vLLM, SSE streaming,
Kafka NvProto/JSON output, Basler/Pylon camera + emulation, Docker compose,
chunk-level latency. Do NOT trigger for generic DeepStream pipelines, object
detection/tracking, NIM imports, or video summarization.
owner: "windy@nvidia.com"
service: "deepstream-sop"
version: "1.0.0"
license: "CC-BY-4.0 AND Apache-2.0"
reviewed: "2026-04-08"
metadata:
author: "Wind Yuan <windy@nvidia.com>"
tags:
- deepstream
- sop
- vlm
- triton
- gpu
languages:
- python
frameworks:
- deepstream
- triton
- fastapi
domain: video-analytics
---
# DeepStream SOP Inference Microservice Skill
This skill guides AI coding assistants in building, extending, and debugging the
**NVIDIA DeepStream SOP (Standard Operating Procedure) Inference Microservice** —
a GPU-accelerated pipeline for temporal action detection and VLM-based SOP compliance
monitoring on industrial video feeds.
**Reference repository**: https://github.com/NVIDIA/sop-monitoring-blueprints/tree/main/microservices/sop-inference-bp
**Local reference code**: `sop-inference-bp/` directory (from a local clone of the repository)
---
## Models
Model-agnostic at both inference stages — swap via env var (and Triton dir for GEBD).
| Stage | Role | Model class | Default | Swap via |
|------|------|-------------|---------|----------|
| Stage 1 (CV) | Per-frame boundary scoring → chunk segmentation | **Generic Event Boundary Detection (GEBD)** | **DDM** ([MCG-NJU/DDM](https://github.com/MCG-NJU/DDM)) via Triton Python backend | Replace `triton_model_repo/<model>/` + `DDM_MODEL_PATH` (§ 5) |
| Stage 3 (VLM) | Per-chunk action classification | Vision-language model via vLLM | **Cosmos Reason 1 7B** (Reason 2 also supported) | Set `VLLM_MODEL_PATH` to a different HF ID or local path |
"GEBD" = swappable Stage-1 slot; "DDM" = the default architecture (terms used interchangeably).
**Chunking is selectable per request** (§ 2): default `ddm-net` uses GEBD; `uniform` produces fixed-length chunks and **bypasses Stage-1 GEBD** (§ 3, § 6). DDM temporal window is configurable via `FRAMES_PER_SIDE` / `SEQUENCE_BATCH` (§ 4, § 5), with optional **TensorRT** (§ 5).
---
## Architecture Overview
Runs in a Docker container (`nvds-action-sop`) alongside a Kafka container. Full diagram: [`references/sop_architecture.svg`](references/sop_architecture.svg).
**Data flow through the 4-stage `SOPVideoProcessor` pipeline (per-request):**
```
Input Sources Docker Container: nvds-action-sop
───────────── ──────────────────────────────────────────────────
Video Files ──┐ FastAPI Server (port 8300)
RTSP Streams ─┤── base64/ ├─ /v1/chat/completions → SOPProcessManager
Basler Camera ┘ file/rtsp/ │
camera │ ModelInitializer: VLM first, then DDM dummy pipeline
│ 4 Thread Pools: cv(32), clip(32), vlm(64), vlm_req(64)
│
▼ SOPVideoProcessor (per-request)
┌────────────────────────────────────────────────┐
│ Stage 1: DeepStream Pipeline (GPU) │
│ Source → nvstreammux → tee1 │
│ ├─[inference] queue1 → nvdspreprocess │
│ │ → nvinferserver (Triton CAPI + DDM) │
│ │ → InferOutputTensorParser → score_queue │
│ ├─[frames] queue3 → nvvideoconvert │
│ │ → capsfilter → appsink │
│ │ → DecodedFrameRetriever → frame_queue │
│ └─[RTSP out] queue → convert → H.264 enc │ (optional, § 18)
│ → rtppay → udpsink → RTSPServer (§ 18) │ opt-in only
│ │ boundary scores │
│ ▼ │
│ Stage 2: Clip Post-Process │
│ Boundary detection → chunk segmentation │
│ │ video frames + timestamps │
│ ▼ │
│ Stage 3: VLM Inference │
│ Embedded vLLM (Cosmos Reason 1/2) │
│ Frame sampling at VLM_FPS → classification │
│ │ action labels │
│ ▼ │
│ Stage 4: SOP Checker │
│ Sequence validation → missing/misordered │
│ │ chunk results │
│ ▼ │
│ final_queue │
└────────────────────────────────────────────────┘
│
Output ▼
────── ┌─────────────────┐
SSE Stream (chat.completion.chunk) │ Kafka Messages │
Non-streaming (chat.completion) │ (JSON/Protobuf) │
Prometheus metrics (/v1/metrics) └────────┬────────┘
▼
Docker Container: kafka
(apache/kafka:3.7.0)
```
---
## Section Index
Each section is a standalone file in `references/` — load only what your task needs.
| § | File | Responsibility |
|---|------|---------------|
| 1 | [`skill_01_fastapi_endpoints.md`](references/skill_01_fastapi_endpoints.md) | FastAPI endpoints, server init, Prometheus metrics |
| 2 | [`skill_02_pydantic_schemas.md`](references/skill_02_pydantic_schemas.md) | Request/response Pydantic models (`api_types.py`) |
| 3 | [`skill_03_deepstream_pipeline.md`](references/skill_03_deepstream_pipeline.md) | DeepStream pyservicemaker pipeline, tensor parser, dummy pipeline |
| 4 | [`skill_04_config_templates.md`](references/skill_04_config_templates.md) | nvdspreprocess / nvinferserver config templates + rendering |
| 5 | [`skill_05_triton_ddm_model.md`](references/skill_05_triton_ddm_model.md) | Triton model repo, config.pbtxt, model.py, ddm_net.py |
| 5b | [`skill_05b_custom_postprocess.md`](references/skill_05b_custom_postprocess.md) | C++ postprocess plugin, Makefile, IOptions API |
| 6 | [`skill_06_sop_process_manager.md`](references/skill_06_sop_process_manager.md) | SOPProcessManager, SOPVideoProcessor, VLLMInference, Kafka |
| 6b | [`skill_06b_sop_checker.md`](references/skill_06b_sop_checker.md) | SOP sequence and checker compliance: MissingNumberDetector, SopCheckerCache, SopCheckerRequest/Response |
| 7 | [`skill_07_sse_streaming.md`](references/skill_07_sse_streaming.md) | SSE generator, stream response formatting, dummy test mode |
| 8 | [`skill_08_basler_camera.md`](references/skill_08_basler_camera.md) | Basler camera support, Pylon SDK, emulation, formats |
| 9 | [`skill_09_docker_build_deploy.md`](references/skill_09_docker_build_deploy.md) | Docker build, deploy, .env configuration |
| 10 | [`skill_10_test_suite.md`](references/skill_10_test_suite.md) | Test suite coverage, assertions, running tests |
| 11 | [`skill_11_env_variables.md`](references/skill_11_env_variables.md) | All environment variables reference |
| 12 | [`skill_12_evaluation_workflow.md`](references/skill_12_evaluation_workflow.md) | End-to-end eval workflow: static checks, build, launch, tests, API/camera/Kafka checks, report |
| 13 | [`skill_13_verification_curl.md`](references/skill_13_verification_curl.md) | Verification steps and curl examples |
| 14 | [`skill_14_implementation_checklist.md`](references/skill_14_implementation_checklist.md) | Implementation checklist: file copy list, generated files, Docker prereqs, verification |
| 15 | [`skill_15_latency_measurement.md`](references/skill_15_latency_measurement.md) | TTFC and C2C latency measurement for file input via SSE streaming |
| 16 | [`skill_16_message_schema.md`](references/skill_16_message_schema.md) | Kafka message schema selection (`JSON` default vs `NvProtoSchema`) and extending messages with custom data |
| 17 | [`skill_17_camera_latency_measurement.md`](references/skill_17_camera_latency_measurement.md) | Camera / live-stream chunk_e2e latency measurement using internal pipeline timestamps |
| 18 | [`skill_18_rtsp_streaming_output.md`](references/skill_18_rtsp_streaming_output.md) | **OPT-IN** RTSP streaming output: `tee1`-tap re-stream, `RTSPStreamingServer`, `SW_ENCODER` toggle. Generate **only** when user explicitly requests RTSP |
For end-to-end evaluation, read § 12 first; load build/test/curl/latency/camera/Kafka as needed.
**§ 18 is opt-in** — generate only when the user explicitly requests RTSP output; otherwise skip § 18 and the `RTSP_*` rules below.
---
## Key Files Map
The full source-to-target file mapping lives in
[`skill_14_implementation_checklist.md`](references/skill_14_implementation_checklist.md):
- **Files copied verbatim** from `references/` (non-trivial algorithms — cycle
detection, qwen_vl_utils preprocessing, DeepStream `IOptions` API, protobuf
sources) with the rationale per file.
- **Files copied as adaptable templates** (Dockerfile, compose.yaml, Triton
config and `model.py`, `ddm_net.py`, Pylon emulation config, etc.).
- **Files generated from skill sections** — each annotated with the Critical
Rules below that the generation must follow exactly.
- Docker build prerequisites and post-build verification checklist.
Config files (`nvds_preprocess_template.txt`, `nvds_inference_template.txt`,
`vlm_prompts.txt`) are used as-is from `configs/`.
When `skill_06b` is loaded, read `configs/actions.json` from the project root and run the
§ 6b-G generation workflow to produce `nvds_action_detector/missing_number_detector.py`.
If `configs/actions.json` is absent or invalid, fall back to copying the reference file.
---
## Critical Rules
> Each rule's full detail lives in the linked `skill_NN_*.md` reference file.
| Tag | Rule summary | Details in |
|-----|---|---|
| `MANAGER_INIT_IN_MAIN` | `SOPProcessManager` init in `main()` before `uvicorn.run()` — not inside `lifespan()` | `skill_01_fastapi_endpoints.md` |
| `NAMED_KWARGS` | `create_video_processor()` uses named kwargs; camera args as separate kwargs | `skill_06_sop_process_manager.md` |
| `LIVE_REQUIRES_STREAM_TRUE` | `stream: true` required for live inputs (RTSP / camera) | `skill_08_basler_camera.md` |
| `VLM_DISABLED_DISABLES_SOP_CHECKER` | `DISABLE_VLM_INFERENCE=true` auto-disables SOP checker at import | `skill_06_sop_process_manager.md` |
| `CHUNK_PARAMS_MAX_LENGTH` | `ChunkParams.max_length_sec` = 10s internal; 60s API default | `skill_06_sop_process_manager.md` |
| `VLM_WARMUP_BEFORE_DDM` | `ModelInitializer`: VLM warmup FIRST, then CV dummy pipeline | `skill_06_sop_process_manager.md` |
| `VLM_WARMUP_3_FRAMES` | VLM warmup needs 3 frames (`torch.zeros`) — Qwen3VL hangs on < 3 | `skill_06_sop_process_manager.md` |
| `THREAD_POOL_SIZES` | 4 thread pools: `cv`(32), `clip`(32), `vlm_inference`(64), `vlm_request`(64) | `skill_06_sop_process_manager.md` |
| `MEDIA_INFO_PYMEDIAINFO` | Media info via `pymediainfo`; live sources set fps=30/duration=inf directly | `skill_06_sop_process_manager.md` |
| `CAMERA_EMULATION_PYLON_CAMEMU` | `PYLON_CAMEMU=1` for camera emulation (serial `0815-0000`) | `skill_08_basler_camera.md` |
| `DEEPSTREAM_LIB_HIDE` | DeepStream lib hide trick: rename lib → lib.tmp during gst-plugin-pylon build | `skill_08_basler_camera.md` |
| `VLM_REAL_GPU_FRAMES` | VLM uses real GPU frames via `DecodedFrameRetriever`; never `torch.zeros` for inference | `skill_06_sop_process_manager.md` |
| `BUFFER_RETRIEVER_STATIC_BASE` | `DecodedFrameRetriever` MUST inherit `BufferRetriever` statically via `super().__init__()`; runtime `__class__.__bases__` mutation hangs `pipeline.attach()` | `skill_06_sop_process_manager.md` |
| `FRAME_RETRIEVER_PRIORITY` | `create_inference_pipeline`: `frame_retriever=` kwarg takes priority over `frame_queue` | `skill_03_deepstream_pipeline.md` |
| `MUX_ORIGINAL_RESOLUTION` | nvstreammux uses original resolution (not 224); pass `mux_width/mux_height` from `get_media_info()` (probe live RTSP for non-camera inputs; camera path unaffected) | `skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md` |
| `FILE_URI_NO_DOUBLE_PREFIX` | `create_inference_pipeline` file source: check `file_path.startswith("file://")` before prepending — API passes `file://` URLs directly | `skill_03_deepstream_pipeline.md` |
| `CLEANUP_ON_DISCONNECT` | Pipeline cleanup on client disconnect via `trigger_stop_processors` in `try/finally` | `skill_07_sse_streaming.md` |
| `UNIFIED_CLIP_POST_PROCESS` | Unified `clip_post_process()` for file + live; `stop()` puts `None` in `_score_queue` | `skill_06_sop_process_manager.md` |
| `ABORT_INFLIGHT_VLM` | Abort in-flight VLM requests on `stop()` via `llm.abort(req_id)` | `skill_06_sop_process_manager.md` |
| `LOGGER_EXPORT_GET_LOGGER` | `ds_logger.py` must export `get_logger` | `skill_06_sop_process_manager.md` |
| `KAFKA_USE_CREATE_PRODUCER` | Kafka: use `create_producer()` from `messager.py`; no `Messager` class | `skill_06_sop_process_manager.md` |
| `USER_PROMPT_PRIORITY` | User request text takes priority over `VLM_PROMPT_PATH` file; `{"type":"text"}` in the request overrides the config-file prompt | `skill_06_sop_process_manager.md` |
| `EVAL_USE_CONFIG_PROMPT` | Eval/latency requests omit request text by default so the VLM uses `VLM_PROMPT_PATH` | `skill_12_evaluation_workflow.md`, `skill_13_verification_curl.md`, `skill_15_latency_measurement.md`, `skill_17_camera_latency_measurement.md` |
| `CHUNK_SCHEMA_FIELD_NAMES` | Chunk schema: `chunk_idx`, `cv_boundary_score`, `checker_result`; summary `chunk_idx=-1` | `skill_06_sop_process_manager.md` |
| `SEQUENTIAL_FRAME_DRAIN` | Drain `decoded_frame_queue` (FIFO, shared across chunks) in a SINGLE thread and submit VLM per chunk incrementally; parallel drain steals frames → 0-frame chunks / wrong VLM input | `skill_06_sop_process_manager.md` |
| `WALL_CLOCK_BEFORE_GPU` | `DecodedFrameRetriever.consume()`: capture `wall_clock_entry = time.time()` BEFORE GPU dlpack; queue 3-tuple `(timestamp, wall_clock_entry, tensor)` | `skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md` |
| `CHUNK_E2E_PIPELINE_TIMESTAMPS` | Write `pipeline_chunk_end_timestamp` (last frame wall_clock) and `pipeline_vlm_ready_timestamp` (`tm_e2e.now()`) into `chunk_info` for camera latency (§ 17) | `skill_06_sop_process_manager.md`, `skill_17_camera_latency_measurement.md` |
| `VLM_INFERENCE_REQUIRED_KWARGS` | Every `VLLMInference.inference()` call must pass `video_fps`, `system_prompt`, `max_completion_tokens` | `skill_06_sop_process_manager.md` |
| `UNIFORM_CHUNKING_BYPASSES_DDM` | `chunking_options.algorithm="uniform"` → fixed-length chunks; `create_inference_pipeline(uniform_chunk=True)` skips DDM but keeps `tee1` fanout; Stage 2 uses `uniform_clip_post_process` | `skill_02_pydantic_schemas.md`, `skill_03_deepstream_pipeline.md`, `skill_06_sop_process_manager.md` |
| `DDM_TEMPORAL_CONFIGURABLE` | `SLIDING_WINDOWS_SIZE = 2*FRAMES_PER_SIDE + SEQUENCE_BATCH` rendered into preprocess/nvinferserver (no hard-coded 18); Triton `config.pbtxt` sequence dim `-1` | `skill_04_config_templates.md`, `skill_05_triton_ddm_model.md` |
| `DDM_TRT_OPTIONAL_PATH` | `DDM_TRT_OPTIMIZATION=true` runs DDM via TensorRT (per-thread contexts, fixed batch = SEQUENCE_BATCH); PyTorch fallback; never both. PyTorch is default | `skill_05_triton_ddm_model.md` |
| `DDM_TRT_STREAM_ORDERING` | `DDMTensorRTEngine.infer()`: `wait_stream(current)` → `execute_async_v3` → **`torch.cuda.synchronize(device)`** (NOT per-stream). Per-stream sync leaves TRT aux-stream work in flight → gst-CV SIGSEGV (NVBug 6289256) | `skill_05_triton_ddm_model.md` |
| `METADATA_LICENSE_FROM_FILE` | `/v1/metadata` reads `licenseInfo` from `DS_SOP_LICENSE_PATH` (default `/opt/nvidia/nvds_sop/license.txt`); never hard-code license text | `skill_01_fastapi_endpoints.md` |
| `CAMERA_EMULATION_FRAMES_RGB` | Pylon emulation PNGs must be explicit 3-channel RGB (matches `Emulation_0815-0000.pfs PixelFormat=RGB8Packed`); generate via `nvvideoconvert ! videoconvert ! "video/x-raw,format=RGB" ! pngenc` | `skill_08_basler_camera.md` |
| `COMPOSE_ENV_PASSTHROUGH` | `docker compose` only substitutes `${VAR}` references; every runtime env var must be explicitly listed under `environment:` to reach the container. | `skill_09_docker_build_deploy.md` |
> The four `RTSP_*` rules below apply **only when the optional RTSP streaming-output feature
> (§ 18) is requested**. They do not apply to the default build — skip them if the user did
> not ask for RTSP output.
| `RTSP_OUTPUT_TAPS_TEE1` | RTSP output branch links from the existing `tee1` (added after the main inference link) only when `rtsp_port` is present. | `skill_18_rtsp_streaming_output.md` |
| `RTSP_LEAKY_QUEUE_TINY` | RTSP branch queue must be `leaky=2` + tiny cap (`max-size-buffers=2`) to prevent backpressure and NVMM pool exhaustion. | `skill_18_rtsp_streaming_output.md` |
| `RTSP_KEYINT_MAX_30` | RTSP H.264 encoder must set `key-int-max=30` (and B-frames disabled) to allow downstream seeking. | `skill_18_rtsp_streaming_output.md` |
| `RTSP_ENCODER_FALLBACK` | Select software/hardware H.264 encoder based on `SW_ENCODER` with MJPEG fallback. | `skill_18_rtsp_streaming_output.md` |
---