Zhan-Lun commited on
Commit
8e92bfb
·
verified ·
1 Parent(s): 5b4004b

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +19 -17
README.md CHANGED
@@ -78,10 +78,12 @@ as synthetically generated data.
78
 
79
  ## 📊 Evaluation results
80
 
81
- The table reports NDCG@10 scores on the eight **public** ViDoRe V3 tasks as
82
- percentages rather than values between 0 and 1 (for example, 0.80 is shown as
83
- 80.00). Each task value is the mean of its six language subsets; the public
84
- average is the unweighted mean of the eight public task values.
 
 
85
 
86
  The evaluation software versions and setup are documented under
87
  [Reproducing the evaluation environment](#reproducing-the-evaluation-environment).
@@ -100,18 +102,18 @@ All table values are shown to two decimal places. ColVec1.1 values are rounded
100
  from the submitted artifacts using round-half-up. Comparator values were read
101
  from the live
102
  [ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)
103
- on July 22, 2026.
104
-
105
- | Model | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Pharmaceuticals | Physics | **Avg. public** |
106
- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
107
- | **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 67.88 | 51.50 | **65.32** |
108
- | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | **68.19** | **51.73** | 64.72 |
109
- | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 67.32 | 48.38 | 64.45 |
110
- | **webAI-ColVec1.1-4b (this model)** | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 67.25 | 51.24 | 64.24 |
111
- | [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | 67.45 | 50.18 | 63.72 |
112
- | [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 67.19 | 50.84 | 63.54 |
113
- | [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 65.68 | 50.15 | 63.39 |
114
- | [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 66.36 | 50.13 | 61.60 |
115
 
116
  The current MTEB leaderboard entries named `webAI-ColVec1-4b` and
117
  `webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1
@@ -304,7 +306,7 @@ The requirements include FlashAttention 2 as an optional supported backend;
304
  its presence does not change the SDPA configuration used for the reported
305
  scores.
306
 
307
- The public ViDoRe V3 scores used the following core software versions:
308
 
309
  ```text
310
  Python 3.12
 
78
 
79
  ## 📊 Evaluation results
80
 
81
+ The table reports NDCG@10 scores on the ViDoRe V3 tasks as percentages rather
82
+ than values between 0 and 1 (for example, 0.80 is shown as 80.00). Each task
83
+ value is the mean of its six language subsets. Mean (Public) is the unweighted
84
+ mean of the eight public task values, Mean (Private) covers the Nuclear and
85
+ Telecom tasks, and the final mean is the unweighted mean across all ten tasks.
86
+ Models are sorted by the final mean.
87
 
88
  The evaluation software versions and setup are documented under
89
  [Reproducing the evaluation environment](#reproducing-the-evaluation-environment).
 
102
  from the submitted artifacts using round-half-up. Comparator values were read
103
  from the live
104
  [ViDoRe V3 MTEB leaderboard](https://mteb-leaderboard.hf.space/benchmark/ViDoRe%28v3%29)
105
+ on July 31, 2026.
106
+
107
+ | Model | **Final mean** | **Mean (Public)** | **Mean (Private)** | Computer Science | Energy | FinanceEn | FinanceFr | HR | Industrial | Nuclear | Pharmaceuticals | Physics | Telecom |
108
+ | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
109
+ | **[webAI-ColVec1.1-8b](https://huggingface.co/webAI-Official/webAI-ColVec1.1-8b)** | **64.95** | **65.32** | **63.47** | 80.08 | 70.12 | **71.90** | **54.87** | 68.55 | **57.65** | 53.66 | 67.88 | 51.50 | **73.29** |
110
+ | [VultronRetriever Prime](https://huggingface.co/vultr/VultronRetrieverPrime-Qwen3.5-8B) | 64.26 | 64.72 | 62.43 | 79.81 | **70.26** | 69.01 | 54.51 | 66.82 | 57.41 | 53.59 | **68.19** | **51.73** | 71.27 |
111
+ | **webAI-ColVec1.1-4b (this model)** | 63.90 | 64.24 | 62.53 | 80.34 | 69.50 | 69.18 | 53.13 | 66.90 | 56.36 | 53.30 | 67.25 | 51.24 | 71.76 |
112
+ | [VultronRetriever Core](https://huggingface.co/vultr/VultronRetrieverCore-Qwen3.5-4.5B) | 63.57 | 63.72 | 63.00 | 79.77 | 69.19 | 68.93 | 52.02 | 66.10 | 56.11 | **54.90** | 67.45 | 50.18 | 71.10 |
113
+ | [Nemotron ColEmbed VL 8B V2](https://huggingface.co/nvidia/nemotron-colembed-vl-8b-v2) | 63.42 | 63.54 | 62.92 | 79.29 | 69.82 | 67.29 | 51.54 | 66.32 | 56.03 | 53.84 | 67.19 | 50.84 | 72.00 |
114
+ | [webAI-ColVec1-9b](https://huggingface.co/webAI-Official/webAI-ColVec1-9b) | 63.00 | 64.45 | 57.20 | **80.92** | 69.77 | 68.28 | 53.72 | **70.04** | 57.18 | 47.66 | 67.32 | 48.38 | 66.74 |
115
+ | [webAI-ColVec1-4b](https://huggingface.co/webAI-Official/webAI-ColVec1-4b) | 62.22 | 63.39 | 57.55 | 79.84 | 68.70 | 68.49 | 51.11 | 67.40 | 55.73 | 48.22 | 65.68 | 50.15 | 66.88 |
116
+ | [Tomoro ColQwen3 Embed 8B](https://huggingface.co/TomoroAI/tomoro-colqwen3-embed-8b) | 61.59 | 61.60 | 61.56 | 75.35 | 68.41 | 65.08 | 49.10 | 63.98 | 54.41 | 52.65 | 66.36 | 50.13 | 70.46 |
117
 
118
  The current MTEB leaderboard entries named `webAI-ColVec1-4b` and
119
  `webAI-ColVec1-9b` refer to the previous ColVec1 release, not these ColVec1.1
 
306
  its presence does not change the SDPA configuration used for the reported
307
  scores.
308
 
309
+ The ViDoRe V3 scores used the following core software versions:
310
 
311
  ```text
312
  Python 3.12