distilroberta-base-finetuned-amazon-products

This model is a fine-tuned version of distilroberta-base on the on 1 million sentences sampled randomly from the pool of approximately 20 million product titles, product descriptions and product bullet points extracted out of Amazon India Scrapped Dataset in 2023. The model is evaluated on 0.1 million sentences which were held out. It achieves the following results on the evaluation set:

• Loss: 2.1609
• Perplexity: 8.68

Usage

from transformers import AutoTokenizer, AutoModelForMaskedLM, pipeline

model_checkpoint = "RishiDarkDevil/distilroberta-base-finetuned-amazon-products"

tokenizer = AutoTokenizer.from_pretrained("distilroberta-base", use_fast=True)
model = AutoModelForMaskedLM.from_pretrained(model_checkpoint, output_hidden_states = True)

fill_mask = pipeline("fill-mask", model=model, tokenizer=tokenizer)

fill_mask('Made with Super Hard Coating and Shock <mask> technology and it protects the display') 
# actual word: Absorption, predicted: 'proof' (0.36), 'absorption' (0.23)

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 8
• eval_batch_size: 8
• seed: 42
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 3.0

Training results

Training Loss	Epoch	Step	Validation Loss
2.5129	1.0	125000	2.3504
2.3418	2.0	250000	2.2269
2.2978	3.0	375000	2.1161

Framework versions

• Transformers 4.28.0
• Pytorch 2.0.0
• Datasets 2.1.0
• Tokenizers 0.13.3