Hugging Face's logo

srikarvar
/
fine_tuned_model_9

Sentence Similarity
sentence-transformers
Safetensors
bert
feature-extraction
Generated from Trainer
dataset_size:2476
loss:OnlineContrastiveLoss
Eval Results
text-embeddings-inference
Inference Endpoints
Model card Files Community
fine_tuned_model_9 / README.md
srikarvar's picture
srikarvar
Add new SentenceTransformer model.
844fc0d verified
preview code
|
raw
history blame contribute delete
28.1 kB
metadata
base_model: intfloat/multilingual-e5-small
library_name: sentence-transformers
metrics:
  - cosine_accuracy
  - cosine_accuracy_threshold
  - cosine_f1
  - cosine_f1_threshold
  - cosine_precision
  - cosine_recall
  - cosine_ap
  - dot_accuracy
  - dot_accuracy_threshold
  - dot_f1
  - dot_f1_threshold
  - dot_precision
  - dot_recall
  - dot_ap
  - manhattan_accuracy
  - manhattan_accuracy_threshold
  - manhattan_f1
  - manhattan_f1_threshold
  - manhattan_precision
  - manhattan_recall
  - manhattan_ap
  - euclidean_accuracy
  - euclidean_accuracy_threshold
  - euclidean_f1
  - euclidean_f1_threshold
  - euclidean_precision
  - euclidean_recall
  - euclidean_ap
  - max_accuracy
  - max_accuracy_threshold
  - max_f1
  - max_f1_threshold
  - max_precision
  - max_recall
  - max_ap
pipeline_tag: sentence-similarity
tags:
  - sentence-transformers
  - sentence-similarity
  - feature-extraction
  - generated_from_trainer
  - dataset_size:2476
  - loss:OnlineContrastiveLoss
widget:
  - source_sentence: Would you want to be President?
    sentences:
      - Can you help me with my homework?
      - How to bake cookies?
      - Why do you want to be to president?
  - source_sentence: Velocity of sound waves in the atmosphere
    sentences:
      - What is the speed of sound in air?
      - What is the best/most memorable thing you've ever eaten and why?
      - >-
        The `safe` option in the `to_spreadsheet` method controls whether a safe
        conversion or not is needed for certain plant attributes to store the
        data in a SpreadsheetTable or Row.
  - source_sentence: Number of countries in the European Union
    sentences:
      - How many countries are in the European Union?
      - Who painted the Sistine Chapel ceiling?
      - >-
        The RecipeManager class is used to manage the downloading and extraction
        of recipes.
  - source_sentence: Official currency of the USA
    sentences:
      - What is purpose of life?
      - >-
        Files inside ZIP archives are accessed and yielded sequentially using
        iter_zip().
      - What is the currency of the United States?
  - source_sentence: Who wrote the book "1984"?
    sentences:
      - What is the speed of light?
      - How to set up a home gym?
      - Who wrote the book "To Kill a Mockingbird"?
model-index:
  - name: SentenceTransformer based on intfloat/multilingual-e5-small
    results:
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: pair class dev
          type: pair-class-dev
        metrics:
          - type: cosine_accuracy
            value: 0.8623188405797102
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.8491722345352173
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.8856304985337243
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.8245140314102173
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.8435754189944135
            name: Cosine Precision
          - type: cosine_recall
            value: 0.9320987654320988
            name: Cosine Recall
          - type: cosine_ap
            value: 0.9266759550807792
            name: Cosine Ap
          - type: dot_accuracy
            value: 0.8623188405797102
            name: Dot Accuracy
          - type: dot_accuracy_threshold
            value: 0.8491722941398621
            name: Dot Accuracy Threshold
          - type: dot_f1
            value: 0.8856304985337243
            name: Dot F1
          - type: dot_f1_threshold
            value: 0.8245140910148621
            name: Dot F1 Threshold
          - type: dot_precision
            value: 0.8435754189944135
            name: Dot Precision
          - type: dot_recall
            value: 0.9320987654320988
            name: Dot Recall
          - type: dot_ap
            value: 0.9266759550807792
            name: Dot Ap
          - type: manhattan_accuracy
            value: 0.8623188405797102
            name: Manhattan Accuracy
          - type: manhattan_accuracy_threshold
            value: 8.599637031555176
            name: Manhattan Accuracy Threshold
          - type: manhattan_f1
            value: 0.8856304985337243
            name: Manhattan F1
          - type: manhattan_f1_threshold
            value: 9.221129417419434
            name: Manhattan F1 Threshold
          - type: manhattan_precision
            value: 0.8435754189944135
            name: Manhattan Precision
          - type: manhattan_recall
            value: 0.9320987654320988
            name: Manhattan Recall
          - type: manhattan_ap
            value: 0.9260061788962293
            name: Manhattan Ap
          - type: euclidean_accuracy
            value: 0.8623188405797102
            name: Euclidean Accuracy
          - type: euclidean_accuracy_threshold
            value: 0.5491920709609985
            name: Euclidean Accuracy Threshold
          - type: euclidean_f1
            value: 0.8856304985337243
            name: Euclidean F1
          - type: euclidean_f1_threshold
            value: 0.5924187898635864
            name: Euclidean F1 Threshold
          - type: euclidean_precision
            value: 0.8435754189944135
            name: Euclidean Precision
          - type: euclidean_recall
            value: 0.9320987654320988
            name: Euclidean Recall
          - type: euclidean_ap
            value: 0.9266759550807792
            name: Euclidean Ap
          - type: max_accuracy
            value: 0.8623188405797102
            name: Max Accuracy
          - type: max_accuracy_threshold
            value: 8.599637031555176
            name: Max Accuracy Threshold
          - type: max_f1
            value: 0.8856304985337243
            name: Max F1
          - type: max_f1_threshold
            value: 9.221129417419434
            name: Max F1 Threshold
          - type: max_precision
            value: 0.8435754189944135
            name: Max Precision
          - type: max_recall
            value: 0.9320987654320988
            name: Max Recall
          - type: max_ap
            value: 0.9266759550807792
            name: Max Ap
      - task:
          type: binary-classification
          name: Binary Classification
        dataset:
          name: pair class test
          type: pair-class-test
        metrics:
          - type: cosine_accuracy
            value: 0.8659420289855072
            name: Cosine Accuracy
          - type: cosine_accuracy_threshold
            value: 0.8320531249046326
            name: Cosine Accuracy Threshold
          - type: cosine_f1
            value: 0.8875379939209727
            name: Cosine F1
          - type: cosine_f1_threshold
            value: 0.8320531249046326
            name: Cosine F1 Threshold
          - type: cosine_precision
            value: 0.874251497005988
            name: Cosine Precision
          - type: cosine_recall
            value: 0.9012345679012346
            name: Cosine Recall
          - type: cosine_ap
            value: 0.9257692996006023
            name: Cosine Ap
          - type: dot_accuracy
            value: 0.8659420289855072
            name: Dot Accuracy
          - type: dot_accuracy_threshold
            value: 0.8320530652999878
            name: Dot Accuracy Threshold
          - type: dot_f1
            value: 0.8875379939209727
            name: Dot F1
          - type: dot_f1_threshold
            value: 0.8320530652999878
            name: Dot F1 Threshold
          - type: dot_precision
            value: 0.874251497005988
            name: Dot Precision
          - type: dot_recall
            value: 0.9012345679012346
            name: Dot Recall
          - type: dot_ap
            value: 0.9257692996006023
            name: Dot Ap
          - type: manhattan_accuracy
            value: 0.8623188405797102
            name: Manhattan Accuracy
          - type: manhattan_accuracy_threshold
            value: 8.854782104492188
            name: Manhattan Accuracy Threshold
          - type: manhattan_f1
            value: 0.8875739644970415
            name: Manhattan F1
          - type: manhattan_f1_threshold
            value: 9.349273681640625
            name: Manhattan F1 Threshold
          - type: manhattan_precision
            value: 0.8522727272727273
            name: Manhattan Precision
          - type: manhattan_recall
            value: 0.9259259259259259
            name: Manhattan Recall
          - type: manhattan_ap
            value: 0.9255387736459155
            name: Manhattan Ap
          - type: euclidean_accuracy
            value: 0.8659420289855072
            name: Euclidean Accuracy
          - type: euclidean_accuracy_threshold
            value: 0.5795620679855347
            name: Euclidean Accuracy Threshold
          - type: euclidean_f1
            value: 0.8875379939209727
            name: Euclidean F1
          - type: euclidean_f1_threshold
            value: 0.5795620679855347
            name: Euclidean F1 Threshold
          - type: euclidean_precision
            value: 0.874251497005988
            name: Euclidean Precision
          - type: euclidean_recall
            value: 0.9012345679012346
            name: Euclidean Recall
          - type: euclidean_ap
            value: 0.9257692996006023
            name: Euclidean Ap
          - type: max_accuracy
            value: 0.8659420289855072
            name: Max Accuracy
          - type: max_accuracy_threshold
            value: 8.854782104492188
            name: Max Accuracy Threshold
          - type: max_f1
            value: 0.8875739644970415
            name: Max F1
          - type: max_f1_threshold
            value: 9.349273681640625
            name: Max F1 Threshold
          - type: max_precision
            value: 0.874251497005988
            name: Max Precision
          - type: max_recall
            value: 0.9259259259259259
            name: Max Recall
          - type: max_ap
            value: 0.9257692996006023
            name: Max Ap

SentenceTransformer based on intfloat/multilingual-e5-small

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-small. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

  • Model Type: Sentence Transformer
  • Base model: intfloat/multilingual-e5-small
  • Maximum Sequence Length: 512 tokens
  • Output Dimensionality: 384 tokens
  • Similarity Function: Cosine Similarity

Model Sources

Full Model Architecture 

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("srikarvar/fine_tuned_model_9")
# Run inference
sentences = [
    'Who wrote the book "1984"?',
    'Who wrote the book "To Kill a Mockingbird"?',
    'What is the speed of light?',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Binary Classification

Metric Value
cosine_accuracy 0.8623
cosine_accuracy_threshold 0.8492
cosine_f1 0.8856
cosine_f1_threshold 0.8245
cosine_precision 0.8436
cosine_recall 0.9321
cosine_ap 0.9267
dot_accuracy 0.8623
dot_accuracy_threshold 0.8492
dot_f1 0.8856
dot_f1_threshold 0.8245
dot_precision 0.8436
dot_recall 0.9321
dot_ap 0.9267
manhattan_accuracy 0.8623
manhattan_accuracy_threshold 8.5996
manhattan_f1 0.8856
manhattan_f1_threshold 9.2211
manhattan_precision 0.8436
manhattan_recall 0.9321
manhattan_ap 0.926
euclidean_accuracy 0.8623
euclidean_accuracy_threshold 0.5492
euclidean_f1 0.8856
euclidean_f1_threshold 0.5924
euclidean_precision 0.8436
euclidean_recall 0.9321
euclidean_ap 0.9267
max_accuracy 0.8623
max_accuracy_threshold 8.5996
max_f1 0.8856
max_f1_threshold 9.2211
max_precision 0.8436
max_recall 0.9321
max_ap 0.9267

Binary Classification

Metric Value
cosine_accuracy 0.8659
cosine_accuracy_threshold 0.8321
cosine_f1 0.8875
cosine_f1_threshold 0.8321
cosine_precision 0.8743
cosine_recall 0.9012
cosine_ap 0.9258
dot_accuracy 0.8659
dot_accuracy_threshold 0.8321
dot_f1 0.8875
dot_f1_threshold 0.8321
dot_precision 0.8743
dot_recall 0.9012
dot_ap 0.9258
manhattan_accuracy 0.8623
manhattan_accuracy_threshold 8.8548
manhattan_f1 0.8876
manhattan_f1_threshold 9.3493
manhattan_precision 0.8523
manhattan_recall 0.9259
manhattan_ap 0.9255
euclidean_accuracy 0.8659
euclidean_accuracy_threshold 0.5796
euclidean_f1 0.8875
euclidean_f1_threshold 0.5796
euclidean_precision 0.8743
euclidean_recall 0.9012
euclidean_ap 0.9258
max_accuracy 0.8659
max_accuracy_threshold 8.8548
max_f1 0.8876
max_f1_threshold 9.3493
max_precision 0.8743
max_recall 0.9259
max_ap 0.9258

Training Details

Training Dataset

Unnamed Dataset

  • Size: 2,476 training samples
  • Columns: sentence2, label, and sentence1
  • Approximate statistics based on the first 1000 samples:
    sentence2 label sentence1
    type string int string
    details
    • min: 4 tokens
    • mean: 16.06 tokens
    • max: 98 tokens
    • 0: ~40.20%
    • 1: ~59.80%
    • min: 6 tokens
    • mean: 16.35 tokens
    • max: 98 tokens
  • Samples:
    sentence2 label sentence1
    A model is trained using the ImageNet dataset to classify images into distinct categories. 1 The ImageNet dataset is used for training models to classify images into various categories.
    Version 5.3.1 does not contain it. 1 No, it doesn't exist in version 5.3.1.
    Can you do my homework for me? 0 Can you help me with my homework?
  • Loss: OnlineContrastiveLoss

Evaluation Dataset

Unnamed Dataset

  • Size: 276 evaluation samples
  • Columns: sentence2, label, and sentence1
  • Approximate statistics based on the first 276 samples:
    sentence2 label sentence1
    type string int string
    details
    • min: 5 tokens
    • mean: 15.34 tokens
    • max: 86 tokens
    • 0: ~41.30%
    • 1: ~58.70%
    • min: 6 tokens
    • mean: 15.56 tokens
    • max: 87 tokens
  • Samples:
    sentence2 label sentence1
    How is AI used to enhance cybersecurity? 0 What are the challenges of AI in cybersecurity?
    The SYSTEM log documentation can be accessed by clicking on the link which will take you to the main version. 1 You can find the SYSTEM log documentation on the main version. Click on the provided link to redirect to the main version of the documentation.
    Name the capital city of Italy 1 What is the capital of Italy?
  • Loss: OnlineContrastiveLoss

Training Hyperparameters

Non-Default Hyperparameters

  • eval_strategy: epoch
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • gradient_accumulation_steps: 2
  • num_train_epochs: 4
  • warmup_ratio: 0.1
  • load_best_model_at_end: True
  • optim: adamw_torch_fused
  • batch_sampler: no_duplicates

All Hyperparameters

Click to expand
  • overwrite_output_dir: False
  • do_predict: False
  • eval_strategy: epoch
  • prediction_loss_only: True
  • per_device_train_batch_size: 32
  • per_device_eval_batch_size: 32
  • per_gpu_train_batch_size: None
  • per_gpu_eval_batch_size: None
  • gradient_accumulation_steps: 2
  • eval_accumulation_steps: None
  • learning_rate: 5e-05
  • weight_decay: 0.0
  • adam_beta1: 0.9
  • adam_beta2: 0.999
  • adam_epsilon: 1e-08
  • max_grad_norm: 1.0
  • num_train_epochs: 4
  • max_steps: -1
  • lr_scheduler_type: linear
  • lr_scheduler_kwargs: {}
  • warmup_ratio: 0.1
  • warmup_steps: 0
  • log_level: passive
  • log_level_replica: warning
  • log_on_each_node: True
  • logging_nan_inf_filter: True
  • save_safetensors: True
  • save_on_each_node: False
  • save_only_model: False
  • restore_callback_states_from_checkpoint: False
  • no_cuda: False
  • use_cpu: False
  • use_mps_device: False
  • seed: 42
  • data_seed: None
  • jit_mode_eval: False
  • use_ipex: False
  • bf16: False
  • fp16: False
  • fp16_opt_level: O1
  • half_precision_backend: auto
  • bf16_full_eval: False
  • fp16_full_eval: False
  • tf32: None
  • local_rank: 0
  • ddp_backend: None
  • tpu_num_cores: None
  • tpu_metrics_debug: False
  • debug: []
  • dataloader_drop_last: False
  • dataloader_num_workers: 0
  • dataloader_prefetch_factor: None
  • past_index: -1
  • disable_tqdm: False
  • remove_unused_columns: True
  • label_names: None
  • load_best_model_at_end: True
  • ignore_data_skip: False
  • fsdp: []
  • fsdp_min_num_params: 0
  • fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
  • fsdp_transformer_layer_cls_to_wrap: None
  • accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
  • deepspeed: None
  • label_smoothing_factor: 0.0
  • optim: adamw_torch_fused
  • optim_args: None
  • adafactor: False
  • group_by_length: False
  • length_column_name: length
  • ddp_find_unused_parameters: None
  • ddp_bucket_cap_mb: None
  • ddp_broadcast_buffers: False
  • dataloader_pin_memory: True
  • dataloader_persistent_workers: False
  • skip_memory_metrics: True
  • use_legacy_prediction_loop: False
  • push_to_hub: False
  • resume_from_checkpoint: None
  • hub_model_id: None
  • hub_strategy: every_save
  • hub_private_repo: False
  • hub_always_push: False
  • gradient_checkpointing: False
  • gradient_checkpointing_kwargs: None
  • include_inputs_for_metrics: False
  • eval_do_concat_batches: True
  • fp16_backend: auto
  • push_to_hub_model_id: None
  • push_to_hub_organization: None
  • mp_parameters:
  • auto_find_batch_size: False
  • full_determinism: False
  • torchdynamo: None
  • ray_scope: last
  • ddp_timeout: 1800
  • torch_compile: False
  • torch_compile_backend: None
  • torch_compile_mode: None
  • dispatch_batches: None
  • split_batches: None
  • include_tokens_per_second: False
  • include_num_input_tokens_seen: False
  • neftune_noise_alpha: None
  • optim_target_modules: None
  • batch_eval_metrics: False
  • batch_sampler: no_duplicates
  • multi_dataset_batch_sampler: proportional

Training Logs

Epoch Step Training Loss loss pair-class-dev_max_ap pair-class-test_max_ap
0 0 - - 0.7876 -
0.2564 10 1.6257 - - -
0.5128 20 0.8138 - - -
0.7692 30 0.7276 - - -
1.0 39 - 0.8190 0.9089 -
1.0256 40 0.6423 - - -
1.2821 50 0.5168 - - -
1.5385 60 0.3583 - - -
1.7949 70 0.3182 - - -
2.0 78 - 0.7351 0.9215 -
2.0513 80 0.3521 - - -
2.3077 90 0.2037 - - -
2.5641 100 0.1293 - - -
2.8205 110 0.1374 - - -
3.0 117 - 0.7223 0.9258 -
3.0769 120 0.198 - - -
3.3333 130 0.0667 - - -
3.5897 140 0.0526 - - -
3.8462 150 0.0652 - - -
4.0 156 - 0.7327 0.9267 0.9258
  • The bold row denotes the saved checkpoint.

Framework Versions

  • Python: 3.10.12
  • Sentence Transformers: 3.1.0
  • Transformers: 4.41.2
  • PyTorch: 2.1.2+cu121
  • Accelerate: 0.34.2
  • Datasets: 2.19.1
  • Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers 

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}