|
--- |
|
license: apache-2.0 |
|
language: |
|
- ru |
|
metrics: |
|
- accuracy |
|
base_model: |
|
- ai-forever/ruRoberta-large |
|
pipeline_tag: text-classification |
|
tags: |
|
- reviews |
|
- e-commercy |
|
- foodtech |
|
--- |
|
# Food Delivery Feedback Multi-Label Classification Model |
|
|
|
This model was developed for multi-label classification of customer feedback in the food delivery domain. It can identify up to 50 different aspects/issues from user reviews and feedback messages. |
|
|
|
## Model Description |
|
|
|
The model is designed to analyze customer feedback and automatically categorize it into multiple relevant labels, helping businesses quickly identify and address various aspects of their food delivery service. |
|
|
|
### Key Features: |
|
- Multi-label classification across 50 distinct categories |
|
- Specialized for food delivery domain feedback |
|
- Handles various aspects including delivery speed, food quality, packaging, customer service, etc. |
|
- Supports Russian language feedback processing |
|
|
|
### Use Cases: |
|
- Automated feedback categorization |
|
- Customer satisfaction analysis |
|
- Service quality monitoring |
|
- Issue identification and prioritization |
|
- Operational improvement insights |
|
|
|
### Performance: |
|
- Trained on about 3000 customer feedback messages |
|
- Achieves **Full Match Accuracy** on test set |
|
- Optimized for real-world food delivery feedback scenarios |
|
|
|
## Evaluation Metric: Full Match Accuracy |
|
|
|
The model is evaluated using a strict full match accuracy metric that requires exact matching between predicted and target labels. |
|
|
|
### Metric Description: |
|
- Type: Exact Match / Perfect Match Accuracy |
|
- Range: 0 to 1 (or 0% to 100%) |
|
- Format: Labels are represented as space-separated indices (e.g., "0 1 10") |
|
|
|
### Example: |
|
```python |
|
# Correct prediction (100% accuracy): |
|
Target: "0 1 10" |
|
Prediction: "0 1 10" |
|
|
|
# Incorrect predictions (0% accuracy): |
|
Target: "0 1 10" |
|
Prediction: "0 1" # Partial match is considered incorrect |
|
Target: "0 1 10" |
|
Prediction: "0 1 10 2" # Extra label is considered incorrect |
|
``` |
|
|
|
### Labels Include: |
|
- Delivery time issues |
|
- Food quality concerns |
|
- Order accuracy |
|
- Packaging condition |
|
- Courier behavior |
|
- App/website functionality |
|
- Payment issues |
|
|
|
## Usage |
|
|
|
The model can be used for: |
|
- Real-time feedback classification |
|
- Batch processing of historical feedback |
|
- Customer support automation |
|
- Service quality analytics |
|
|
|
## Training |
|
|
|
The model was trained as part of an NLP competition focused on improving customer feedback analysis in the food delivery industry. It uses state-of-the-art transformer architecture optimized for multi-label classification tasks. |
|
|
|
## Limitations |
|
|
|
- Primarily optimized for Russian language feedback |
|
- May require fine-tuning for specific regional contexts |
|
- Best suited for food delivery domain specifically |
|
|
|
## How to use with PyTorch and Transfomers |
|
|
|
```python |
|
import torch |
|
from transformers import RobertaTokenizer, RobertaForSequenceClassification |
|
|
|
|
|
model_name = 'ai-forever/ruRoberta-large' |
|
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') |
|
|
|
tokenizer = RobertaTokenizer.from_pretrained(model_name) |
|
class_descriptions = pd.read_csv('files/trends_description.csv') |
|
|
|
class BERTClass(torch.nn.Module): |
|
def __init__(self): |
|
super(BERTClass, self).__init__() |
|
self.bert_model = RobertaForSequenceClassification.from_pretrained( |
|
'metanovus/ruroberta-ecom-tech-best', |
|
return_dict=True, |
|
problem_type='multi_label_classification', |
|
num_labels=50 |
|
) |
|
|
|
def forward(self, input_ids, attn_mask, token_type_ids): |
|
output = self.bert_model( |
|
input_ids, |
|
attention_mask=attn_mask, |
|
token_type_ids=token_type_ids |
|
) |
|
return output.logits |
|
|
|
model = BERTClass().to(device) |
|
``` |
