StreetPrompt

Mehmet Kerem Turkcan
Columbia University, Center for Smart Streetscapes (CS3)

StreetPrompt classifies street camera video into classes that you write as text. Type the situations you care about, such as "the street is flooded" and "a dry street", and the model scores each clip against them. No labelled data, no retraining and no video generation are needed to add a class.

It was trained without class labels, on the synthetic clips of Synthetic Street Scenes and on unlabelled real traffic footage captioned by a vision language model. Three variants target three classes of hardware. The Nano variant has 4.35 million parameters, needs 6.0 GMACs for a 16 frame clip and fits a Jetson Orin Nano.

The figure shows held-out test clips with prompts written for it. With those prompts, the Nano variant ranks the correct prompt first for 14 of 14 severe flood clips, 6 of 6 fog clips, 8 of 10 tampering clips and 7 of 10 collision clips in the test sets.

Variants

Variant	Video encoder	Input	Parameters	GMACs per clip	ONNX size	A100, TensorRT FP16	Target hardware
nano	MobileNetV4 Conv Small	16 frames, 320 px	4.35 M	6.0	17 MB	1.2 ms	Jetson Orin Nano
agx	MobileNetV4 Conv Medium	16 frames, 352 px	10.3 M	32.6	41 MB	3.4 ms	Jetson AGX Orin
rtx4090	ConvNeXt Small	12 frames, 320 px	50.8 M	212.6	199 MB	6.9 ms	RTX 4090 class GPU

Every variant shares one text encoder, the frozen CLIP ViT-B/32 text tower (text_encoder.onnx, 243 MB). It runs once per prompt list, not per clip, and can stay off the device entirely: encode the prompts on any computer and ship the 512 dimensional class embeddings. Latency is for the video encoder at batch 1, measured with TensorRT 11.3 on one A100; it has not yet been measured on Jetson hardware.

Results

Zero-shot balanced accuracy (mean per class recall). No benchmark label was used in training, and the BDD100K images, the never seen scene types and the egocentric prompts were never used in any form.

Benchmark	Classes	Chance	Nano	AGX	RTX 4090	CLIP ViT-B/32 image tower
Synthetic test sets, class prompts, 7 tasks	3 to 4	0.29	0.535	0.563	0.565	0.365
Synthetic test sets, yes or no prompts, 7 tasks	2	0.50	0.756	0.746	0.731	0.605
BDD100K weather, real dashcam	5	0.20	0.624	0.646	0.599	0.417
BDD100K scene, real dashcam	3	0.33	0.727	0.721	0.741	0.644
BDD100K time of day, real dashcam	3	0.33	0.808	0.813	0.811	0.641
Scene types never seen in training	7	0.14	0.595	0.423	0.560	0.768
Egocentric categories, 3 tasks	3 to 4	0.31	0.388	0.358	0.393	0.327

• Synthetic test sets are the test splits of SynFLOOD, SynSNOW, SynJAM, SynCRASH, SynAFTERMATH, SynBIKELANE and SynTIC (266 clips). Class prompts name every class of a task, for example the four flood depth grades; yes or no prompts ask one binary question per task, for example "is the road flooded". For the Nano variant, the yes or no scores are 0.72 for flooding, 0.92 for snow, 0.71 for congestion, 0.70 for collisions and near misses, 0.61 for road obstruction, 0.75 for blocked bike lanes and 0.90 for tampering.
• BDD100K is 2,000 real dashcam images from the 10,000 image BDD100K subset, held out from training, scored with the weather, scene and time of day attributes shipped with the dataset. Other images of the same subset were used as unlabelled captioned footage.
• Scene types never seen in training are the 36 UnseenClasses clips of the dataset (fog, wildfire smoke, a construction zone, a sinkhole, a crowd, emergency vehicles) plus 8 free flowing traffic clips as the normal class. With 44 clips, differences below about 0.1 are not reliable. All three variants score low on sinkholes and emergency vehicles: the two larger variants assign most sinkhole and normal clips to the construction prompt, whose cones and barriers appear in many street scenes, and the Nano variant tends to call emergency scenes normal traffic.
• Egocentric categories ask abstract questions, such as whether a situation is a problem the city should fix. Scores near chance show that such categories do not transfer through text alone.
• The last column scores the CLIP ViT-B/32 image tower on the same clips, averaging 12 frames: the model whose text tower StreetPrompt shares, and one of its training teachers.

Run streetprompt benchmark from the accompanying code to reproduce every number in this table.

Quick start

pip install onnxruntime-gpu av tokenizers huggingface_hub opencv-python-headless numpy
huggingface-cli download mehmetkeremturkcan/StreetPrompt --local-dir StreetPrompt
python StreetPrompt/inference.py --variant nano --video clip.mp4 \
    --prompts "the street is flooded with water" "a dry street, no flooding"

In Python, with the streetprompt package from the accompanying code:

from streetprompt import StreetPrompt

model = StreetPrompt("nano")                       # downloads nano/ and the text encoder
classes = {
    "flooded": ["the street is flooded with water", "water covering the road"],
    "dry": ["a dry street, no flooding"],
}
for window in model.classify("camera.mp4", classes, window=15):
    print(window.start, window.label, window.probs)

A class may be a single prompt or a list of prompts that describe the same thing; the class embedding is their normalised mean. Long videos are split into windows; 15 seconds matches the training clips.

Without the text encoder on the device. Encode the prompts once, copy classes.npz to the device, and run only the video encoder:

streetprompt encode-prompts --prompts "a traffic jam|traffic is congested and stopped" "traffic is flowing" --out classes.npz
streetprompt stream rtsp://camera/stream --class-embeddings classes.npz --window 15 --every 1

TensorRT on Jetson. Build an engine on the device itself; engines do not transfer between GPUs or TensorRT versions.

/usr/src/tensorrt/bin/trtexec --onnx=nano/video_encoder.onnx --fp16 --saveEngine=nano_fp16.engine

Writing prompts

• Describe the visible state of the scene in plain words: "snow covering the road", "a car accident or a near collision", "someone covering the roadside camera".
• Always include the normal case as its own class, for example "normal traffic, nothing unusual". Scores are relative to the prompts you give.
• Two or three phrasings per class make scores steadier than one.
• Avoid prompts that name objects most scenes contain. A "construction zone with cones and barriers" prompt also attracts sinkholes, emergency scenes and ordinary streets with barriers.
• Abstract judgements, such as "a problem the city should fix", do not work; name the visible situation instead.

Inputs

Each clip is reduced to 16 frames taken at the centres of 16 equal segments of the clip or window, each resized to 672 x 378 with bicubic filtering and centre cropped to the variant's input size (320 or 352 px). The 12 frame variant uses 12 of those 16 frames. Pixels are scaled to 0 to 1 and normalised with the ImageNet mean and standard deviation. meta.json in each variant folder records these settings, and the accompanying code reproduces the training preprocessing. Still images are repeated to 16 frames.

Files

text_encoder.onnx            CLIP ViT-B/32 text tower, input_ids and attention_mask (N x 77) -> text_embedding (N x 512)
tokenizer.json               CLIP BPE tokenizer
<variant>/video_encoder.onnx clip (1 x T x 3 x R x R) -> video_embedding (1 x 512), unit length
<variant>/model.safetensors  PyTorch weights of the video encoder, for fine tuning
<variant>/meta.json          input format and training run
inference.py                 dependency light single file inference
examples/prompts_streetscape.json   a starting set of prompts for street cameras

How it was trained

Architecture. A timm image backbone runs on every frame with a temporal shift module at the input of each residual block, frame features are averaged, and a two layer MLP projects them to the 512 dimensional embedding space of the frozen CLIP ViT-B/32 text tower.

Training signals. None of them uses a class label.

1. Text of the synthetic clips. Each training clip of Synthetic Street Scenes is paired with its generation prompt, the situation clause of that prompt and eight shorter paraphrases written by a language model, and the video and text embeddings are aligned with a symmetric contrastive loss.
2. Captions of unlabelled footage. 9,638 clips and stills without labels were captioned by Qwen3-VL-32B and added as pairs: 1,780 dashcam videos from the Nexar collision prediction dataset, 1,560 traffic camera stills from the Montreal snow covered roads dataset, 5,152 BDD100K dashcam images, and 1,146 synthetic clips that never enter any evaluation.
3. Grounding of short class prompts. For the seven synthetic tasks, the model matches the zero-shot answer distribution that Qwen3-VL-32B gives over the class prompts on training clips (weight 0.3).
4. Distillation of the CLIP image tower. The video embedding is pulled toward the frozen CLIP ViT-B/32 image embedding of the same clip, which carries open vocabulary knowledge that the captions lack (weight 0.3 for nano, 1.0 for rtx4090, not used for agx).

Recipe. AdamW with a one cycle schedule, weight decay 0.05 and contrastive temperature 0.05; the backbone learns at 0.3 times the rate of the projection. nano: learning rate 0.001, batch 24, 24 epochs. agx: 0.001, batch 16, 30 epochs. rtx4090: 0.0005, batch 12, 12 epochs. Each variant trained on one A100.

Evaluation hygiene. Evaluation uses the exact test clips listed in the accompanying code. Train, validation and test splits follow the dataset's splits. The 1,146 synthetic clips used as unlabelled captioned data sit outside every evaluation list, including the ones that belong to test splits. The 2,000 BDD100K evaluation images were excluded from training.

Intended use and limitations

StreetPrompt is intended for research and for prototyping monitoring tools for streets, roads and public spaces: flagging flooding, snow, congestion, collisions, obstructions and tampering for a person to review.

• It was trained mostly on synthetic video. Scores on real footage (the BDD100K rows) are lower than a model trained on labelled real data would reach; validate on footage from your own cameras before relying on it.
• Confidence depends on the prompt list. A probability of 0.9 means the clip matches that prompt better than the others given, not that the event is certain.
• Brief events, small objects and abstract categories are the weakest cases.
• Do not use it for automated high risk decisions, for identifying people, or for surveillance of individuals.

License and attribution

The weights are released under CC BY 4.0; inference.py is released under the MIT license. Wherever the model or work derived from it is used or shared, display this reference where readers of that work can clearly see it:

StreetPrompt, Mehmet Kerem Turkcan, Columbia University, Center for Smart Streetscapes (CS3). https://huggingface.co/mehmetkeremturkcan/StreetPrompt. Licensed under CC BY 4.0.

Components and training data keep their own terms: the CLIP ViT-B/32 text tower (MIT license), the timm backbone initialisations, the MiniMax H3 Community License Agreement for the synthetic training clips, and the licenses of the Nexar collision prediction dataset, the Montreal snow covered roads dataset and BDD100K. The Nexar license asks every work that uses its data to cite it; the citation is below.

Citation

@misc{turkcan2026streetprompt,
  title        = {StreetPrompt: Prompt Driven Video Classification for Street Cameras on Edge Hardware},
  author       = {Turkcan, Mehmet Kerem},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/mehmetkeremturkcan/StreetPrompt}},
  note         = {Columbia University, Center for Smart Streetscapes (CS3)}
}

@misc{turkcan2026syntheticstreetscenes,
  title        = {Synthetic Street Scenes: Video Datasets of Rare Street Events for Training and Evaluating Video Models},
  author       = {Turkcan, Mehmet Kerem},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/datasets/mehmetkeremturkcan/SyntheticStreetScenes}},
  note         = {Columbia University, Center for Smart Streetscapes (CS3)}
}

Training data:

• Moura, Daniel C., and Zvitia, Orly. "Nexar Collison Dataset." Hugging Face, 2025. https://huggingface.co/datasets/nexar-ai/nexar_collision_prediction
• Yu, Fisher, et al. "BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learning." CVPR 2020.
• Snow Covered Roads Dataset, https://github.com/mohamedkaraa/Snow-Covered-Roads-Dataset

Acknowledgements

This work was supported by the NSF Engineering Research Center for Smart Streetscapes under Award EEC-2133516.