Hugging Face's logo

TakoData
/
chart-reranker

Text Ranking
sentence-transformers
Safetensors
new
cross-encoder
reranker
Generated from Trainer
dataset_size:24588
loss:BinaryCrossEntropyLoss
custom_code
Eval Results (legacy)
text-embeddings-inference
Model card Files
xet
 Community
chart-reranker / README.md
noahjax's picture
noahjax
Update README.md
0f7b530 verified
preview code
|
Raw
History Blame Contribute Delete
17.5 kB
metadata 
tags:
  - sentence-transformers
  - cross-encoder
  - reranker
  - generated_from_trainer
  - dataset_size:24588
  - loss:BinaryCrossEntropyLoss
base_model: Alibaba-NLP/gte-multilingual-reranker-base
pipeline_tag: text-ranking
library_name: sentence-transformers
metrics:
  - pearson
  - spearman
model-index:
  - name: CrossEncoder based on Alibaba-NLP/gte-multilingual-reranker-base
    results:
      - task:
          type: cross-encoder-correlation
          name: Cross Encoder Correlation
        dataset:
          name: validation
          type: validation
        metrics:
          - type: pearson
            value: 0.875500492479389
            name: Pearson
          - type: spearman
            value: 0.8709281334702662
            name: Spearman

CrossEncoder based on Alibaba-NLP/gte-multilingual-reranker-base

DEPRECATED

This is a Cross Encoder model finetuned from Alibaba-NLP/gte-multilingual-reranker-base using the sentence-transformers library. It computes scores for pairs of texts, which can be used for text reranking and semantic search.

Model Details

Model Description

Model Sources

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import CrossEncoder

# Download from the 🤗 Hub
model = CrossEncoder("cross_encoder_model_id")
# Get scores for pairs of texts
pairs = [
    ['What is the average rent price in Canada?', 'Title: "How many hours do Americans sleep at night (United States)"\nCollections: YouGov Trackers\nDatasets: YouGovTrackerValueV2\nChart Type: survey:timeseries\nSources: YouGov'],
    ['for the topic digital foortprint and identity use "\t " to give a description on if there was an provided teaching materials for this activity.', 'Title: "Different ways Americans define gender for someone who says they are transgender (United States)"\nCollections: YouGov Trackers\nDatasets: YouGovTrackerValueV2\nChart Type: survey:timeseries\nSources: YouGov'],
    ['Which U.S. cities or counties have the highest rates of aggravated assault involving a deadly weapon per 100,000 residents?', 'Title: "U.S. Bank Overview, CITY Overview"\nCollections: Companies\nDatasets: InstrumentClosePrice1Day\nChart Type: timeseries:eav_v3\nCanonical forms: "U.S. Bancorp"="closing_price", "Club De Futbol Intercity Sad"="closing_price"'],
    ['Black identity topics', 'Title: "Different ways Americans define gender for someone who says they are transgender (United States)"\nCollections: YouGov Trackers\nDatasets: YouGovTrackerValueV2\nChart Type: survey:timeseries\nSources: YouGov'],
    ['Which company in the Interactive Media and Services category has the highest market capitalization?', 'Title: "DigiPlus Interactive. Capital Expenditure (Quarterly)"\nCollections: Companies\nDatasets: StandardIncomeStatement\nChart Type: timeseries:eav_v3\nCanonical forms: "Capital Expenditure"="capital_expenditure"\nSources: S&P Global'],
]
scores = model.predict(pairs)
print(scores.shape)
# (5,)

# Or rank different texts based on similarity to a single text
ranks = model.rank(
    'What is the average rent price in Canada?',
    [
        'Title: "How many hours do Americans sleep at night (United States)"\nCollections: YouGov Trackers\nDatasets: YouGovTrackerValueV2\nChart Type: survey:timeseries\nSources: YouGov',
        'Title: "Different ways Americans define gender for someone who says they are transgender (United States)"\nCollections: YouGov Trackers\nDatasets: YouGovTrackerValueV2\nChart Type: survey:timeseries\nSources: YouGov',
        'Title: "U.S. Bank Overview, CITY Overview"\nCollections: Companies\nDatasets: InstrumentClosePrice1Day\nChart Type: timeseries:eav_v3\nCanonical forms: "U.S. Bancorp"="closing_price", "Club De Futbol Intercity Sad"="closing_price"',
        'Title: "Different ways Americans define gender for someone who says they are transgender (United States)"\nCollections: YouGov Trackers\nDatasets: YouGovTrackerValueV2\nChart Type: survey:timeseries\nSources: YouGov',
        'Title: "DigiPlus Interactive. Capital Expenditure (Quarterly)"\nCollections: Companies\nDatasets: StandardIncomeStatement\nChart Type: timeseries:eav_v3\nCanonical forms: "Capital Expenditure"="capital_expenditure"\nSources: S&P Global',
    ]
)
# [{'corpus_id': ..., 'score': ...}, {'corpus_id': ..., 'score': ...}, ...]

Evaluation

Metrics

Cross Encoder Correlation

Metric Value
pearson 0.8755
spearman 0.8709

Training Details

Training Dataset

Unnamed Dataset

  • Size: 24,588 training samples
  • Columns: sentence_0, sentence_1, and label
  • Approximate statistics based on the first 1000 samples:
    sentence_0 sentence_1 label
    type string string float
    details
    • min: 3 characters
    • mean: 88.65 characters
    • max: 998 characters
    • min: 73 characters
    • mean: 169.97 characters
    • max: 352 characters
    • min: 0.0
    • mean: 0.41
    • max: 1.0
  • Samples:
    sentence_0 sentence_1 label
    What is the average rent price in Canada? Title: "How many hours do Americans sleep at night (United States)"
    Collections: YouGov Trackers
    Datasets: YouGovTrackerValueV2
    Chart Type: survey:timeseries
    Sources: YouGov    		 0.0
    for the topic digital foortprint and identity use " " to give a description on if there was an provided teaching materials for this activity. Title: "Different ways Americans define gender for someone who says they are transgender (United States)"
    Collections: YouGov Trackers
    Datasets: YouGovTrackerValueV2
    Chart Type: survey:timeseries
    Sources: YouGov    		 0.25
    Which U.S. cities or counties have the highest rates of aggravated assault involving a deadly weapon per 100,000 residents? Title: "U.S. Bank Overview, CITY Overview"
    Collections: Companies
    Datasets: InstrumentClosePrice1Day
    Chart Type: timeseries:eav_v3
    Canonical forms: "U.S. Bancorp"="closing_price", "Club De Futbol Intercity Sad"="closing_price"    		 0.0
  • Loss: BinaryCrossEntropyLoss with these parameters:
    {
    "activation_fn": "torch.nn.modules.linear.Identity",
    "pos_weight": null
}

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: steps
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • num_train_epochs: 5
  • fp16: True

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: steps
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 1
  • eval_accumulation_steps: None
  • torch_empty_cache_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1
  • num_train_epochs: 5
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.0
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • bf16: False
  • fp16: True
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: False
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • parallelism_config: None
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • project: huggingface
  • trackio_space_id: trackio
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: None
  • hub_always_push: False
  • hub_revision: None
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • include_for_metrics: []
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: no
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • eval_on_start: False
  • use_liger_kernel: False
  • liger_kernel_config: None
  • eval_use_gather_object: False
  • average_tokens_across_devices: True
  • prompts: None
  • batch_sampler: batch_sampler
  • multi_dataset_batch_sampler: proportional
  • router_mapping: {}
  • learning_rate_mapping: {}

Training Logs

Epoch Step Training Loss validation_spearman
0.1300 100 - 0.7581
0.2601 200 - 0.7928
0.3901 300 - 0.8105
0.5202 400 - 0.8252
0.6502 500 0.4726 0.8306
0.7802 600 - 0.8338
0.9103 700 - 0.8398
1.0 769 - 0.8406
1.0403 800 - 0.8412
1.1704 900 - 0.8479
1.3004 1000 0.4027 0.8525
1.4304 1100 - 0.8521
1.5605 1200 - 0.8549
1.6905 1300 - 0.8591
1.8205 1400 - 0.8619
1.9506 1500 0.3793 0.8614
2.0 1538 - 0.8627
2.0806 1600 - 0.8623
2.2107 1700 - 0.8641
2.3407 1800 - 0.8598
2.4707 1900 - 0.8655
2.6008 2000 0.3534 0.8641
2.7308 2100 - 0.8651
2.8609 2200 - 0.8656
2.9909 2300 - 0.8668
3.0 2307 - 0.8660
3.1209 2400 - 0.8678
3.2510 2500 0.3387 0.8654
3.3810 2600 - 0.8654
3.5111 2700 - 0.8667
3.6411 2800 - 0.8676
3.7711 2900 - 0.8674
3.9012 3000 0.3335 0.8704
4.0 3076 - 0.8703
4.0312 3100 - 0.8698
4.1612 3200 - 0.8709

Framework Versions

  • Python: 3.12.11
  • Sentence Transformers: 5.1.2
  • Transformers: 4.57.1
  • PyTorch: 2.8.0+cu128
  • Accelerate: 1.11.0
  • Datasets: 4.2.0
  • Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}