Hugging Face's logo

qualcomm
/
Video-MAE

Video Classification
PyTorch
backbone
android
Model card Files
xet
 Community
Video-MAE
File size: 10,890 Bytes
101f643
 
d88fe06
101f643
 
 
 
 
 
 
 
 
cbcfae9
101f643
 
 
cbcfae9
1cdf057
cbcfae9
 
 
 
 
 
 
 
 
 
 
 
1cdf057
 
 
 
 
cbcfae9
 
 
 
 
 
1cdf057
cbcfae9
 
 
 
 
 
1cdf057
cbcfae9
 
 
 
 
 
 
f240fef
cbcfae9
f240fef
cbcfae9
 
 
 
1b4484b
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
101f643
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
---
library_name: pytorch
license: cc-by-4.0
tags:
- backbone
- android
pipeline_tag: video-classification

---

![](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/video_mae/web-assets/model_demo.png)

# Video-MAE: Optimized for Qualcomm Devices

Video MAE (Masked Auto Encoder) is a network for doing video classification that uses the ViT (Vision Transformer) backbone.

This is based on the implementation of Video-MAE found [here](https://github.com/MCG-NJU/VideoMAE).
This repository contains pre-exported model files optimized for Qualcomm® devices. You can use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.62.2/src/qai_hub_models/models/video_mae) library to export with custom configurations. More details on model performance across various devices, can be found [here](#performance-summary).

Qualcomm AI Hub Models uses [Qualcomm AI Hub Workbench](https://workbench.aihub.qualcomm.com) to compile, profile, and evaluate this model. [Sign up](https://myaccount.qualcomm.com/signup) to run these models on a hosted Qualcomm® device.

## Getting Started
There are two ways to deploy this model on your device:

### Option 1: Download Pre-Exported Models

Below are pre-exported model assets ready for deployment.

| Runtime | Precision | Chipset | SDK Versions | Download |
|---|---|---|---|---|
| ONNX | float | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/video_mae/releases/v0.62.2/video_mae-onnx-float.zip)
| ONNX | w8a16 | Universal | QAIRT 2.45, ONNX Runtime 1.27.1 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/video_mae/releases/v0.62.2/video_mae-onnx-w8a16.zip)
| QNN_DLC | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/video_mae/releases/v0.62.2/video_mae-qnn_dlc-float.zip)
| QNN_DLC | w8a16 | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/video_mae/releases/v0.62.2/video_mae-qnn_dlc-w8a16.zip)
| TFLITE | float | Universal | QAIRT 2.45 | [Download](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/models/video_mae/releases/v0.62.2/video_mae-tflite-float.zip)

For more device-specific assets and performance metrics, visit **[Video-MAE on Qualcomm® AI Hub](https://aihub.qualcomm.com/models/video_mae)**.


### Option 2: Export with Custom Configurations

Use the [Qualcomm® AI Hub Models](https://github.com/qualcomm/ai-hub-models/blob/v0.62.2/src/qai_hub_models/models/video_mae) Python library to compile and export the model with your own:
- Custom weights (e.g., fine-tuned checkpoints)
- Custom input shapes
- Target device and runtime configurations

This option is ideal if you need to customize the model beyond the default configuration provided here.

See our repository for [Video-MAE on GitHub](https://github.com/qualcomm/ai-hub-models/blob/v0.62.2/src/qai_hub_models/models/video_mae) for usage instructions.

## Model Details

**Model Type:** Model_use_case.video_classification

**Model Stats:**
- Input resolution: 224x224
- Model checkpoint: Kinectics-400
- Model size (float): 335 MB
- Number of parameters: 87.7M

## Performance Summary
| Model | Runtime | Precision | Chipset | Inference Time (ms) | Peak Memory Range (MB) | Primary Compute Unit
|---|---|---|---|---|---|---
| Video-MAE | ONNX | float | Snapdragon® X2 Elite | 465.657 ms | 9 - 9 MB | NPU
| Video-MAE | ONNX | float | Snapdragon® X Elite | 615.161 ms | 188 - 188 MB | NPU
| Video-MAE | ONNX | float | Snapdragon® 8 Gen 3 Mobile | 399.949 ms | 1 - 1085 MB | NPU
| Video-MAE | ONNX | float | Snapdragon® 8 Gen 1 Mobile | 1390.512 ms | 11 - 955 MB | NPU
| Video-MAE | ONNX | float | Qualcomm® Dragonwing™ IQ-8275 | 558.465 ms | 9 - 22 MB | NPU
| Video-MAE | ONNX | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 594.644 ms | 0 - 217 MB | NPU
| Video-MAE | ONNX | float | Qualcomm® QCS8450 | 1390.512 ms | 11 - 955 MB | NPU
| Video-MAE | ONNX | float | Qualcomm® Dragonwing™ IQ-9075 | 560.516 ms | 9 - 21 MB | NPU
| Video-MAE | ONNX | float | Qualcomm® Dragonwing™ IQ-X7181 | 615.161 ms | 188 - 188 MB | NPU
| Video-MAE | ONNX | float | Qualcomm® Dragonwing™ Q-8750 | 379.001 ms | 1 - 787 MB | NPU
| Video-MAE | ONNX | float | Snapdragon® 8 Elite Mobile | 379.001 ms | 1 - 787 MB | NPU
| Video-MAE | ONNX | float | Snapdragon® 8 Elite Gen 5 Mobile | 469.425 ms | 4 - 795 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® X2 Elite | 328.077 ms | 5 - 5 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® X Elite | 494.636 ms | 100 - 100 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® 8 Gen 3 Mobile | 371.206 ms | 0 - 1091 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® 8 Gen 1 Mobile | 1448.884 ms | 5 - 1156 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS6490 | 2319.296 ms | 0 - 7 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 544.73 ms | 1 - 9 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 487.645 ms | 0 - 111 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® QCS8450 | 1448.884 ms | 5 - 1156 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 487.856 ms | 0 - 7 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 494.636 ms | 100 - 100 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 1079.643 ms | 5 - 1356 MB | NPU
| Video-MAE | ONNX | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 290.863 ms | 2 - 875 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® 8 Elite Mobile | 290.863 ms | 2 - 875 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 362.553 ms | 1 - 897 MB | NPU
| Video-MAE | ONNX | w8a16 | Snapdragon® 7 Gen 4 Mobile | 1079.643 ms | 5 - 1356 MB | NPU
| Video-MAE | QNN_DLC | float | Snapdragon® X2 Elite | 255.624 ms | 9 - 9 MB | NPU
| Video-MAE | QNN_DLC | float | Snapdragon® X Elite | 470.017 ms | 9 - 9 MB | NPU
| Video-MAE | QNN_DLC | float | Snapdragon® 8 Gen 3 Mobile | 383.664 ms | 10 - 986 MB | NPU
| Video-MAE | QNN_DLC | float | Snapdragon® 8 Gen 1 Mobile | 562.401 ms | 9 - 898 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-8275 | 519.983 ms | 9 - 21 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 456.357 ms | 10 - 12 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® SA8775P | 480.128 ms | 0 - 679 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® SA8650P | 480.128 ms | 0 - 679 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® SA8255P | 480.128 ms | 0 - 679 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® QCS8450 | 562.401 ms | 9 - 898 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-9075 | 476.202 ms | 9 - 20 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® Dragonwing™ IQ-X7181 | 470.017 ms | 9 - 9 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® Dragonwing™ Q-8750 | 266.012 ms | 7 - 684 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® SA7255P | 1125.532 ms | 3 - 680 MB | NPU
| Video-MAE | QNN_DLC | float | Qualcomm® SA8295P | 616.266 ms | 1 - 651 MB | NPU
| Video-MAE | QNN_DLC | float | Snapdragon® 8 Elite Mobile | 266.012 ms | 7 - 684 MB | NPU
| Video-MAE | QNN_DLC | float | Snapdragon® 8 Elite Gen 5 Mobile | 291.16 ms | 9 - 702 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Snapdragon® X2 Elite | 336.034 ms | 5 - 5 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Snapdragon® X Elite | 518.054 ms | 5 - 5 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Snapdragon® 8 Gen 3 Mobile | 378.224 ms | 5 - 1247 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-8275 | 519.134 ms | 5 - 12 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 511.946 ms | 5 - 7 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® SA8775P | 514.977 ms | 2 - 798 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® SA8650P | 514.977 ms | 2 - 798 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® SA8255P | 514.977 ms | 2 - 798 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-9075 | 508.939 ms | 3 - 9 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ IQ-X7181 | 518.054 ms | 5 - 5 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-6690 | 3549.028 ms | 5 - 1342 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-7790 | 713.062 ms | 5 - 1154 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® Dragonwing™ Q-8750 | 292.97 ms | 5 - 782 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Qualcomm® SA7255P | 980.449 ms | 1 - 832 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Snapdragon® 8 Elite Mobile | 292.97 ms | 5 - 782 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Snapdragon® 8 Elite Gen 5 Mobile | 334.391 ms | 5 - 809 MB | NPU
| Video-MAE | QNN_DLC | w8a16 | Snapdragon® 7 Gen 4 Mobile | 713.062 ms | 5 - 1154 MB | NPU
| Video-MAE | TFLITE | float | Snapdragon® 8 Gen 3 Mobile | 124.051 ms | 0 - 977 MB | NPU
| Video-MAE | TFLITE | float | Snapdragon® 8 Gen 1 Mobile | 260.283 ms | 1 - 935 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® Dragonwing™ IQ-8275 | 184.639 ms | 0 - 208 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® Dragonwing™ QCS8550 (Proxy) | 158.207 ms | 0 - 4 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® SA8775P | 181.366 ms | 0 - 709 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® SA8650P | 181.366 ms | 0 - 709 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® SA8255P | 181.366 ms | 0 - 709 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® QCS8450 | 260.283 ms | 1 - 935 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® Dragonwing™ IQ-9075 | 178.838 ms | 0 - 207 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® Dragonwing™ Q-8750 | 91.622 ms | 0 - 688 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® SA7255P | 469.667 ms | 1 - 681 MB | NPU
| Video-MAE | TFLITE | float | Qualcomm® SA8295P | 239.007 ms | 0 - 649 MB | NPU
| Video-MAE | TFLITE | float | Snapdragon® 8 Elite Mobile | 91.622 ms | 0 - 688 MB | NPU
| Video-MAE | TFLITE | float | Snapdragon® 8 Elite Gen 5 Mobile | 63.749 ms | 0 - 729 MB | NPU

## License
* The license for the original implementation of Video-MAE can be found
  [here](https://github.com/MCG-NJU/VideoMAE/blob/main/LICENSE).

## References
* [Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://arxiv.org/abs/2203.12602)
* [Source Model Implementation](https://github.com/MCG-NJU/VideoMAE)

## Community
* Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
* For questions or feedback please [reach out to us](mailto:ai-hub-support@qti.qualcomm.com).