Commit
•
c2d27e4
1
Parent(s):
da6a129
Update README.md
Browse files
README.md
CHANGED
|
@@ -40,9 +40,19 @@ The **isy-thl/multilingual-e5-base-course-skill-tuned** is a finetuned version o
|
|
| 40 |
|
| 41 |
## Performance
|
| 42 |
|
| 43 |
-
|
| 44 |
|
| 45 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 46 |
|
| 47 |
## Usage
|
| 48 |
|
|
|
|
| 40 |
|
| 41 |
## Performance
|
| 42 |
|
| 43 |
+
To evaluate the model, all ESCO (x=13895) and GRETA (x=23) skills were embedded using the model under assessment and stored in a vector database. For each query in the evaluation dataset, the top 30 most relevant candidates were retrieved based on cosine similarity. Metrics such as accuracy, precision, recall, NDCG, MRR, and MAP were then calculated. For reranker evaluation, the reranker was used to re-rank the top 30 candidates chosen by the fine-tuned bi-encoder model. The evaluation results were split for the ESCO and GRETA use cases:
|
| 44 |
|
| 45 |
+
**ESCO Use Case**
|
| 46 |
+
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
**GRETA Use Case**
|
| 50 |
+
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
The results demonstrate that fine-tuning significantly enhanced the performance of the model, often more than doubling the performance of the non-fine-tuned base model. Notably, fine-tuning with training data from both use cases outperformed fine-tuning with training data from only the target skill taxonomy. This suggests that the models learn more than just specific skills from the training data and are capable of generalizing. Further research could evaluate the model's performance on an unknown skill taxonomy, where we expect it to perform better as well.
|
| 54 |
+
|
| 55 |
+
The fine-tuned BI-Encoder model (isy-thl/multilingual-e5-base-course-skill-tuned) shows exceptional performance for the target task, providing significant improvements over the base model. To maximize retrieval success, it is recommended to complement the BI-Encoder model with the reranker (isy-thl/bge-reranker-base-course-skill-tuned), especially in scenarios where the computational cost is justified by the need for higher accuracy and precision.
|
| 56 |
|
| 57 |
## Usage
|
| 58 |
|