README.md

metadata

license: apache-2.0
base_model: IlyaGusev/saiga_mistral_7b_lora
language:
  - ru
library_name: peft
pipeline_tag: text-generation
inference: false
tags:
  - mistral
  - saiga
  - search-queries
  - instruct-fine-tuned
  - search-queries-parser
  - zero-shot
  - llm
  - instuct
  - zero-shot
  - query parsing
  - synthetic
  - search-queries
metrics:
  - accuracy
  - precision
  - recall
  - f1
datasets:
  - EmbeddingStudio/query-parsing-instructions-saiga

Model Card for the Query Parser LLM using Falcon-7B-Instruct

EmbeddingStudio is the open-source framework, that allows you transform a joint "Embedding Model + Vector DB" into a full-cycle search engine: collect clickstream -> improve search experience-> adapt embedding model and repeat out of the box.

It's a highly rare case when a company will use unstructured search as is. And by searching brick red houses san francisco area for april user definitely wants to find some houses in San Francisco for a month-long rent in April, and then maybe brick-red houses. Unfortunately, for the 15th January 2024 there is no such accurate embedding model. So, companies need to mix structured and unstructured search.

The very first step of mixing it - to parse a search query. Usual approaches are:

• Implement a bunch of rules, regexps, or grammar parsers (like NLTK grammar parser).
• Collect search queries and to annotate some dataset for NER task.

It takes some time to do, but at the end you can get controllable and very accurate query parser. EmbeddingStudio team decided to dive into LLM instruct fine-tuning for Zero-Shot query parsing task to close the first gap while a company doesn't have any rules and data being collected, or even eliminate exhausted rules implementation, but in the future.

The main idea is to align an LLM to being to parse short search queries knowing just a company market and a schema of search filters. Moreover, being oriented on applied NLP, we are trying to serve only light-weight LLMs a.k.a not heavier than 7B parameters.

Model Details

Model Description

This is only IlyaGusev/saiga_mistral_7b_lora aligned to follow instructions like:

<s>system: Эксперт по разбору поисковых запросов</s>
<s>user: Преобразование запросов в JSON, соответствие схеме, обеспечение правильного написания.
Категория: Mobile App Development
Схема: ```[{"Name": "Project-Budget", "Representations": [{"Name": "Budget-in-USD", "Type": "float", "Examples": [1000.0, 5000.0, 10000.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}, {"Name": "Budget-in-EUR", "Type": "float", "Examples": [850.0, 4250.0, 8500.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}, {"Name": "Budget-in-JPY", "Type": "float", "Examples": [110000.0, 550000.0, 1100000.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}, {"Name": "Budget-in-AUD", "Type": "float", "Examples": [1300.0, 6500.0, 13000.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}]}, {"Name": "Project-Duration", "Representations": [{"Name": "Duration-in-Minutes", "Type": "int", "Examples": [43200, 259200, 525600, 5, 5, 5]}]}, {"Name": "Project-End-Date", "Representations": [{"Name": "Day-Month-Year", "Type": "str", "Examples": ["01 January 2022", "15 February 2023", "31 December 2024", "01 января 2022 года", "15 февраля 2023 года", "31 декабря 2024"], "Pattern": ["dd Month YYYY", "дд Месяц ГГГГ"]}]}, {"Name": "Project-Start-Date", "Representations": [{"Name": "Day-Month-Year", "Type": "str", "Examples": ["01 January 2022", "15 February 2023", "31 December 2024", "01 января 2022 года", "15 февраля 2023 года", "31 декабря 2024"], "Pattern": ["dd Month YYYY", "дд Месяц ГГГГ"]}, {"Name": "Month-Day-Year", "Type": "str", "Examples": ["January 01 2022", "February 15 2023", "December 31 2024", "01 января 2022 года", "15 февраля 2023", "31 декабря 2024"], "Pattern": ["Month dd YYYY", "Месяц dd ГГГГ"]}, {"Name": "Month-Day-Year", "Type": "str", "Examples": ["01-01-2022", "02-15-2023", "12-31-2024", "01.01.2022", "15-02-2023", "31-12-2024"], "Pattern": ["mm-dd-YYYY", "mm-dd-YYYY"]}]}]```
Запрос: приложение для новогодней акции, дедлайн 31 декабря</s>
<s>bot:
[{"Value": "приложение для новогодней акции, дедлайн 31 декабря", "Name": "Correct"}, {"Name": "Project-End-Date.Day-Month-Year", "Value": "31 декабря текущего года"}]</s>

Important: Additionally, we are trying to fine-tune the Large Language Model (LLM) to not only parse unstructured search queries but also to correct spelling.

• Developed by EmbeddingStudio team:
  • Aleksandr Iudaev [LinkedIn] [Email]
  • Andrei Kostin [LinkedIn] [Email]
  • ML Doom [AI Assistant]
• Funded by EmbeddingStudio team
• Model type: Instruct Fine-Tuned Large Language Model
• Model task: Zero-shot search query parsing
• Language(s) (NLP): English
• License: apache-2.0
• Finetuned from model: IlyaGusev/saiga_mistral_7b_lora
• !Maximal Length Size: we used 1024 for fine-tuning, this is highly different from the original model max_seq_length = 2048
• Tuning Epochs: 3 for now, but will be more later.

Disclaimer: As a small startup, this direction forms a part of our Minimum Viable Product (MVP). It's more of an attempt to test the 'product-market fit' rather than a well-structured scientific endeavor. Once we check it and go with a round, we definitely will:

• Curating a specific dataset for more precise analysis.
• Exploring various approaches and Large Language Models (LLMs) to identify the most effective solution.
• Publishing a detailed paper to ensure our findings and methodologies can be thoroughly reviewed and verified.

We acknowledge the complexity involved in utilizing Large Language Models, particularly in the context of Zero-Shot search query parsing and AI Alignment. Given the intricate nature of this technology, we emphasize the importance of rigorous verification. Until our work is thoroughly reviewed, we recommend being cautious and critical of the results.

Model Sources

• Repository: code of inference the model will be here
• Paper: Work In Progress
• Demo: Work In Progress

Uses

We strongly recommend only the direct usage of this fine-tuned version of IlyaGusev/saiga_mistral_7b_lora:

• Zero-shot Search Query Parsing with porived company market name and filters schema
• Search Query Spell Correction

For any other needs the behaviour of the model in unpredictable, please utilize the IlyaGusev/saiga_mistral_7b_lora or fine-tune your own.

Instruction format

<s>system: Эксперт по разбору поисковых запросов</s>
<s>user: Преобразование запросов в JSON, соответствие схеме, обеспечение правильного написания.
Категория: {your_company_category}
Схема: ```{filters_schema}```
Запрос: {query}
<s>bot:

Filters schema is JSON-readable line in the format (we highly recommend you to use it): List of filters (dict):

• Name - name of filter (better to be meaningful).
• Representations - list of possible filter formats (dict):
  • Name - name of representation (better to be meaningful).
  • Type - python base type (int, float, str, bool).
  • Examples - list of examples.
  • Enum - if a representation is enumeration, provide a list of possible values, LLM should map parsed value into this list.
  • Pattern - if a representation is pattern-like (datetime, regexp, etc.) provide a pattern text in any format.

Example:

[{"Name": "Customer_Ratings", "Representations": [{"Name": "Exact_Rating", "Type": "float", "Examples": [4.5, 3.2, 5.0, "4.5", "Unstructured"]}, {"Name": "Minimum_Rating", "Type": "float", "Examples": [4.0, 3.0, 5.0, "4.5"]}, {"Name": "Star_Rating", "Type": "int", "Examples": [4, 3, 5], "Enum": [1, 2, 3, 4, 5]}]}, {"Name": "Date", "Representations": [{"Name": "Day_Month_Year", "Type": "str", "Examples": ["01.01.2024", "15.06.2023", "31.12.2022", "25.12.2021", "20.07.2024", "15.06.2023"], "Pattern": "dd.mm.YYYY"}, {"Name": "Day_Name", "Type": "str", "Examples": ["Понедельник", "Вторник", "пн", "вт", "Среда", "Четверг"], "Enum": ["Понедельник", "Вторник", "Среда", "Четверг", "Пятница", "Суббота", "Воскресенье"]}]}, {"Name": "Date_Period", "Representations": [{"Name": "Specific_Period", "Type": "str", "Examples": ["01.01.2024 - 31.01.2024", "01.06.2023 - 30.06.2023", "01.12.2022 - 31.12.2022"], "Pattern": "dd.mm.YYYY - dd.mm.YYYY"}, {"Name": "Month", "Type": "str", "Examples": ["Январь", "Янв", "Декабрь"], "Enum": ["Январь", "Февраль", "Март", "Апрель", "Май", "Июнь", "Июль", "Август", "Сентябрь", "Октябрь", "Ноябрь", "Декабрь"]}, {"Name": "Quarter", "Type": "str", "Examples": ["Q1", "Q2", "Q3"], "Enum": ["Q1", "Q2", "Q3", "Q4"]}, {"Name": "Season", "Type": "str", "Examples": ["Winter", "Summer", "Autumn"], "Enum": ["Winter", "Spring", "Summer", "Autumn"]}]}, {"Name": "Destination_Country", "Representations": [{"Name": "Country_Name", "Type": "str", "Examples": ["United States", "Germany", "China"]}, {"Name": "Country_Code", "Type": "str", "Examples": ["US", "DE", "CN"]}, {"Name": "Country_Abbreviation", "Type": "str", "Examples": ["USA", "GER", "CHN"]}]}]

As the result, response will be JSON-readable line in the format:

[{"Value": "Corrected search phrase", "Name": "Correct"}, {"Name": "filter-name.representation", "Value": "some-value"}]

Field and representation names will be aligned with the provided schema. Example:

[{"Value": "приложение для новогодней акции, дедлайн 31 декабря", "Name": "Correct"}, {"Name": "Project-End-Date.Day-Month-Year", "Value": "31 декабря текущего года"}]

Used for fine-tuning system phrases:

[
    "Эксперт по разбору поисковых запросов",
    "Мастер анализа поисковых запросов",
    "Первоклассный интерпретатор поисковых запросов",
    "Продвинутый декодер поисковых запросов",
    "Гений разбора поисковых запросов",
    "Волшебник разбора поисковых запросов",
    "Непревзойденный механизм разбора запросов",
    "Виртуоз разбора поисковых запросов",
    "Маэстро разбора запросов",
]

Used for fine-tuning instruction phrases:

[
    "Преобразование запросов в JSON, соответствие схеме, обеспечение правильного написания.",
    "Анализ и структурирование запросов в JSON, поддержание схемы, проверка орфографии.",
    "Организация запросов в JSON, соблюдение схемы, верификация орфографии.",
    "Декодирование запросов в JSON, следование схеме, исправление орфографии.",
    "Разбор запросов в JSON, соответствие схеме, правильное написание.",
    "Преобразование запросов в структурированный JSON, соответствие схеме и орфографии.",
    "Реструктуризация запросов в JSON, соответствие схеме, точное написание.",
    "Перестановка запросов в JSON, строгое соблюдение схемы, поддержание орфографии.",
    "Гармонизация запросов с JSON схемой, обеспечение точности написания.",
    "Эффективное преобразование запросов в JSON, соответствие схеме, правильная орфография."
]

Direct Use

import json

from json import JSONDecodeError

from transformers import AutoTokenizer, AutoModelForCausalLM

from transformers import StoppingCriteria

class EosListStoppingCriteria(StoppingCriteria):
    def __init__(self, eos_sequence = [2]):
        self.eos_sequence = eos_sequence

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        last_ids = input_ids[:,-len(self.eos_sequence):].tolist()
        return self.eos_sequence in last_ids

INSTRUCTION_TEMPLATE = """<s>system: Эксперт по разбору поисковых запросов</s>
<s>user: Преобразование запросов в JSON, соответствие схеме, обеспечение правильного написания.
Категория: {0}
Схема: ```{1}```
Запрос: {2}
<s>bot:
"""


def parse(
        query: str, 
        company_category: str, 
        filter_schema: dict,
        model: AutoModelForCausalLM,
        tokenizer: AutoTokenizer
):
    input_text = INSTRUCTION_TEMPLATE.format(
      company_category,
      json.dumps(filter_schema),
      query
    )
    input_ids = tokenizer.encode(input_text, return_tensors='pt')

    # Generating text
    output = model.generate(input_ids.to('cuda'), 
                            max_new_tokens=1024, 
                            do_sample=True, 
                            temperature=0.05
    )
    try:
        generated = tokenizer.decode(output[0]).replace('<s> ', '<s>').split('<s>bot:\n')[-1].replace('</s>', '').strip()
        parsed = json.loads(generated)
    except JSONDecodeError as e:
        parsed = dict()
        
    return parsed

Bias, Risks, and Limitations

Bias

Again, this model was fine-tuned for following the zero-shot query parsing instructions. So, all ethical biases are inherited by the original model.

Model was fine-tuned to be able to work with the unknown company domain and filters schema. But, can be better with the training company categories:

Artificial Intelligence and Machine Learning, Automotive, Automotive Dealerships, Banking Services, Books and Media, Cloud Computing Services, Cloud-based Development Environments, Collaborative Development Environments, Commercial Real Estate, Continuous Integration/Continuous Deployment, Credit Services, Customer Support Services, Customer Support and Feedback, Cybersecurity Software, Data Analytics and Business Intelligence, Dating Apps, Digital and Mobile Banking, Documentation and Knowledge Sharing, E-commerce Platforms, Eco-Friendly and Sustainable Properties, Educational Institutions, Electronics, Enterprise Software Development, Entertainment and Media Platforms, Event Planning Services, Fashion and Apparel, Financial Planning and Advisory, Food and Grocery, Game Development, Government Services, Health and Beauty, Healthcare Providers, Home and Garden, Image Stock Platforms, Insurance Services, International Real Estate, Internet of Things (IoT) Development, Investment Services, Issue Tracking and Bug Reporting, Job Recruitment Agencies, Land Sales and Acquisitions, Legal Services, Logistics and Supply Chain Management, Luxury and High-End Properties, Market Research Firms, Mobile App Development, Mortgage and Real Estate Services, Payment Processing, Pet Supplies, Professional Social Networks, Project Management Tools, Property Management, Real Estate Consulting, Real Estate Development, Real Estate Investment, Residential Real Estate, Restaurants and Food Delivery Services, Retail Stores (Online and Offline), Risk Management and Compliance, Social Networks, Sports and Outdoors, Task and Time Management, Taxation Services, Team Communication and Chat Tools, Telecommunication Companies, Toys and Games, Travel and Booking Agencies, Travelers and Consumers, User Interface/User Experience Design, Version Control Systems, Video Hosting and Portals, Web Development```

Risks and Limitations

Known limitations:

1. Can add extra spaces or remove spaces: 1-2 -> 1 - 2.
2. Can add extra words: 5 -> 5 years.
3. Can not differentiate between <>= and theirs HTML versions <, >, &eq;.
4. Bad with abbreviations.
5. Can add extra .0 for floats and integers.
6. Can add extra 0 or remove 0 for integers with a char postfix: 10M -> 1m.
7. Can hallucinate with integers. For the case like list of positions exactly 7 openings available result can be {'Name': 'Job_Type.Exact_Match', 'Value': 'Full Time'}.
8. We fine-tuned this model with max sequence length = 1024, so it may happen that response will not be JSON-readable.

The list will be extended in the future.

Recommendations

1. We used synthetic data for the first version of this model. So, we suggest you to precisely test this model on your company's domain, even it's in the list.
2. Use meaningful names for filters and theirs representations.
3. Provide examples for each representation.
4. Try to be compact, model was fine-tuned with max sequence length equal 1024.
5. During the generation use greedy strategy with tempertature 0.05.
6. The result will be better if you align a filters schema with a schema type of the training data.

How to Get Started with the Model

Use the code below to get started with the model.

MODEL_ID = 'EmbeddingStudio/query-parser-saiga-mistral-7b-lora'

Initialize tokenizer:

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID, 
    trust_remote_code=True,
    add_prefix_space=True,
    use_fast=False,
)

Initialize model:

import torch

from peft import LoraConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig

peft_config = LoraConfig(
      lora_alpha=16,
      lora_dropout=0.1,
      r=64,
      bias="none",
      task_type="CAUSAL_LM",
)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    load_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)

device_map = {"": 0}

model = AutoModelForCausalLM.from_pretrained(
  MODEL_ID, 
  quantization_config=bnb_config, 
  device_map=device_map, 
  torch_dtype=torch.float16
)

Use for parsing:

import json

from json import JSONDecodeError

from transformers import StoppingCriteria

class EosListStoppingCriteria(StoppingCriteria):
    def __init__(self, eos_sequence = [2]):
        self.eos_sequence = eos_sequence

    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        last_ids = input_ids[:,-len(self.eos_sequence):].tolist()
        return self.eos_sequence in last_ids

INSTRUCTION_TEMPLATE = """<s>system: Эксперт по разбору поисковых запросов</s>
<s>user: Преобразование запросов в JSON, соответствие схеме, обеспечение правильного написания.
Категория: {0}
Схема: ```{1}```
Запрос: {2}
<s>bot:
"""


def parse(
        query: str, 
        company_category: str, 
        filter_schema: dict,
        model: AutoModelForCausalLM,
        tokenizer: AutoTokenizer
):
    input_text = INSTRUCTION_TEMPLATE.format(
      company_category,
      json.dumps(filter_schema),
      query
    )
    input_ids = tokenizer.encode(input_text, return_tensors='pt')

    # Generating text
    output = model.generate(input_ids.to('cuda'), 
                            max_new_tokens=1024, 
                            do_sample=True, 
                            temperature=0.05
    )
    try:
        generated = tokenizer.decode(output[0]).replace('<s> ', '<s>').split('<s>bot:\n')[-1].replace('</s>', '').strip()
        parsed = json.loads(generated)
    except JSONDecodeError as e:
        parsed = dict()
        
    return parsed

category = 'Mobile App Development'
query = 'приложение для новогодней акции, дедлайн 31 декабря'
schema = [{"Name": "Project-Budget", "Representations": [{"Name": "Budget-in-USD", "Type": "float", "Examples": [1000.0, 5000.0, 10000.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}, {"Name": "Budget-in-EUR", "Type": "float", "Examples": [850.0, 4250.0, 8500.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}, {"Name": "Budget-in-JPY", "Type": "float", "Examples": [110000.0, 550000.0, 1100000.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}, {"Name": "Budget-in-AUD", "Type": "float", "Examples": [1300.0, 6500.0, 13000.0, "The project is currently on hold", "The project is currently on hold", "The project is currently on hold"]}]}, {"Name": "Project-Duration", "Representations": [{"Name": "Duration-in-Minutes", "Type": "int", "Examples": [43200, 259200, 525600, 5, 5, 5]}]}, {"Name": "Project-End-Date", "Representations": [{"Name": "Day-Month-Year", "Type": "str", "Examples": ["01 January 2022", "15 February 2023", "31 December 2024", "01 января 2022 года", "15 февраля 2023 года", "31 декабря 2024"], "Pattern": ["dd Month YYYY", "дд Месяц ГГГГ"]}]}, {"Name": "Project-Start-Date", "Representations": [{"Name": "Day-Month-Year", "Type": "str", "Examples": ["01 January 2022", "15 February 2023", "31 December 2024", "01 января 2022 года", "15 февраля 2023 года", "31 декабря 2024"], "Pattern": ["dd Month YYYY", "дд Месяц ГГГГ"]}, {"Name": "Month-Day-Year", "Type": "str", "Examples": ["January 01 2022", "February 15 2023", "December 31 2024", "01 января 2022 года", "15 февраля 2023", "31 декабря 2024"], "Pattern": ["Month dd YYYY", "Месяц dd ГГГГ"]}, {"Name": "Month-Day-Year", "Type": "str", "Examples": ["01-01-2022", "02-15-2023", "12-31-2024", "01.01.2022", "15-02-2023", "31-12-2024"], "Pattern": ["mm-dd-YYYY", "mm-dd-YYYY"]}]}]

output = parse(query, category, schema)
print(output)

# [out]: [{"Value": "приложение для новогодней акции, дедлайн 31 декабря", "Name": "Correct"}, {"Name": "Project-End-Date.Day-Month-Year", "Value": "31 декабря текущего года"}]

Training Details

Training Data

We used synthetically generated query parsing instructions:

• We generated lists of possible filters for 72 company categories:
  • Raw version of filters dataset
  • Split by representations
• Select randomly up-to 150 possible combinations (1-3 filters in each combination) of filters, the way each filter's representation appears maximum twice.
• For a given category and combination we generated with GPT-4 Turbo:
  • 2 search queries and theirs parsed version with unstructured parts.
  • 2 search queries and theirs parsed version without unstructured part.
• Using filters, queries and parsed version we prepared 27.42k saiga format instruction

Warning: EmbeddingStudio team aware you that generated queries weren't enough curated, and will be curated later once we finish our product market fit stage

Principles of train / test splitting

As we are trying to fine-tune LLM to follow zero-shot query parsing instructions, so we want to test:

• Ability to work well with unseen domain
• Ability to work well with unseen filters
• Ability to work well with unseen queries

For these purposes we:

1. We put into test split 5 categories, completely separared from train: Automotive, Educational Institutions, Enterprise Software Development, Payment Processing, Professional Social Networks.
2. Also out of each appearing in train company categories, we put aside / removed one filter and queries related to it.

Filters generation details

We used GPT-4 Turbo to generate several possible filters for 63 company categroies. For each filter we also generated some possible representations. For examples filter Date can be represented as dd/mm/YYYY, YYYY-mm-dd, as words 2024 Янв 17, etc.

Queries generation details

We also used GPT-4 Turbo for generation of search queries and theirs parsed version. Main principles were:

• If passed schema doesn't contain possible filter, do not generate query itself or a possible filter
• If a selected representations combination contains enumeration, so we ask to map values in a search query and a parsed version.
• If a selected representations combination contains pattern, so we ask GPT-4 Turbo to be aligned with a pattern

Instructions generation details

For the generation instructions we used following ideas:

1. Zero-Shot query parser should be schema agnostic. Cases like snake_case, CamelCase, http-headers-like should not ruin generation process.

2. Zero-Shot query parser should be spelling errors insensitive.

3. Training instructions should be in the following order:

   • Category
   • Schema
   • Query

   So LLM can be used in the following way: just generate embedding of category -> schema part, so inference will be faster.

We assume, that schema agnostic termin means something wider, like to be able to work not only with JSONs, but also with HTML, Markdown, YAML, etc. We are working on it.

So, what was our approach as an attempt to achieve these abilities:

1. For each query we generated a version with a mistake
2. Passed to each parsed version an additional field Correct, which contains a corrected version of a search query.
3. For each query we randomly selected and used a case for schema fields and a case for filter and representation names.
4. For each query we additionally generated two instuctions:

• Where did we remove from a provided schema and parsed version one filter
• Where did we remove from a provided schema and parsed version all related filters

Warning: EmbeddingStudio team ask you to curate datasets on your own precisely.

Training Procedure

1. Mixed Precision Regime
2. Supervised Fine-Tuning
3. Three epochs with cosine scheduler

All details in Training Hyperparameters

Preprocessing [optional]

The preprocessing steps are not detailed in the provided code. Typically, preprocessing involves tokenization, normalization, data augmentation, and handling of special tokens. In this training setup, the tokenizer was configured with add_prefix_space=True and use_fast=False, which might indicate special considerations for tokenizing certain languages or text formats.

Training Hyperparameters

Hyperparameter	Value	Description
Training Regime	Mixed Precision (bfloat16)	Utilizes bfloat16 for efficient memory usage and training speed.
Model Configuration	Causal Language Model	Incorporates LoRA (Low-Rank Adaptation) for training efficiency.
Quantization Configuration	Bits and Bytes (BnB)	Uses settings like load_in_4bit and bnb_4bit_quant_type for model quantization.
Training Environment	CUDA-enabled Device	Indicates GPU acceleration for training.
Learning Rate	0.003	Determines the step size at each iteration while moving toward a minimum of a loss function.
Warmup Ratio	0.03	Fraction of total training steps used for the learning rate warmup.
Optimizer	Paged AdamW (32-bit)	Optimizes the training process with efficient memory usage.
Gradient Accumulation Steps	2	Reduces memory consumption and allows for larger effective batch sizes.
Max Grad Norm	0.3	Maximum norm for the gradients.
LR Scheduler Type	Cosine	Specifies the learning rate schedule.
PEFT Configurations	LoraConfig	Details like lora_alpha, lora_dropout, and r for LoRA adaptations.
Training Dataset Segmentation	Train and Test Sets	Segmentation of the dataset for training and evaluation.
Max Sequence Length	1024	Maximum length of the input sequences.

Testing Data, Factors & Metrics

Testing Data

All information is provided in Training Data section.

Factors Influencing Saiga-Mistral-7B Model Performance

1. Company Category and Domain Knowledge

• Performance may vary based on the specific company category or domain.
• Enhanced performance in domains specifically trained on, such as Educational Institutions, Banking Services, Logistics, etc.

2. Filter Schema Adaptability

• Ability to adapt to various filter schemas.
• Performance in parsing and organizing queries according to different schemas.

3. Handling of Spelling and Syntax Errors

• Robustness in handling spelling errors and syntax variations in queries.

4. Representation and Type Handling

• Capability to handle different data representations (e.g., date formats, enumerations, patterns).
• Accurate processing of various base types (int, float, str, bool).

5. Length and Complexity of Queries

• Impact of the length and complexity of queries on performance.
• Maximum sequence length of 1024 could pose limitations for longer or complex queries.

6. Bias and Ethical Considerations

• Inherited ethical biases from the original model.
• Importance of understanding these biases in different contexts.

7. Limitations in Fine-Tuning and Data Curation

• Limitations such as extra spaces, handling of abbreviations, etc.
• Influence of the extent of training data curation on model accuracy.

8. Specific Use Cases

• Recommended primarily for zero-shot search query parsing and search query spell correction.
• Performance in other use cases might be unpredictable.

9. Training Data Quality and Diversity

• Quality and diversity of synthetic training data.
• Influence on the model's effectiveness across different scenarios.

Testing Procedure

Metrics

Metric Overview

Our zero-shot search query parsing model is designed to extract structured information from unstructured search queries with high precision. The primary metric for evaluating our model's performance is the True Positive (TP) rate, which is assessed using a specialized token-wise Levenshtein distance. This approach is aligned with our goal to achieve semantic accuracy in parsing user queries.

True Positives (TP)

• Definition: A True Positive in our model is counted when the model correctly identifies both the 'Name' and 'Value' in a query, matching the expected results.
• Measurement Method: The TP rate is quantified using the levenshtein_tokenwise function, which calculates the distance between predicted and actual key-value pairs at a token level. We consider a Levenshtein distance of 0.25 or less as acceptable for matching.
• Importance:
  • Token-Level Accuracy: We use token-wise accuracy over traditional character-level Levenshtein distance, which can be overly strict, especially for minor spelling variations. Our token-wise approach prioritizes semantic accuracy.
  • Relevance to Search Queries: Accuracy at the token level is more indicative of the model's ability to understand and parse user intent in search queries.

Generation Strategy

• Approach: The model generates responses based on input queries with a maximum token length set to 1000, employing a sampling strategy (do_sample=True), and a low temperature setting of 0.05. This controlled randomness in generation ensures a variety of accurate and relevant responses.
• Impact on TP:
  • The low temperature setting directly influences the TP rate by reducing the randomness in the model's predictions. With a lower temperature, the model is more likely to choose the most probable word in a given context, leading to more accurate and consistent outputs. This is particularly crucial in search query parsing, where understanding and interpreting user input with high precision is vital.

Additional Metrics

• False Positives (FP) and False Negatives (FN): These metrics are monitored to provide a comprehensive view of the model's predictive capabilities.
• Precision, Recall, F1 Score, Accuracy: These standard metrics complement our TP-focused assessment, providing a rounded picture of the model's performance in various aspects.

Motivation for Metric Choice

• Alignment with User Intent: Focusing on token-wise accuracy ensures the model's performance closely mirrors the structure and intent typical in search queries.
• Robustness Against Query Variations: This metric approach makes the model adaptable to the varied formulations of real-world search queries.
• Balancing Precision and Recall: Our method aims to balance the model's ability not to miss relevant key-value pairs (high recall) while not over-identifying irrelevant ones (high precision).

Total metrics

Category	Recall	Precision	F1	Accuracy
Educational Institutions [+]	0.74	0.71	0.73	0.57
Enterprise Software Development [+]	0.80	0.73	0.76	0.62
Professional Social Networks [+]	0.82	0.72	0.76	0.62
Automotive [+]	0.77	0.64	0.70	0.54
Payment Processing [+]	0.80	0.73	0.76	0.62
Continuous Integration/Continuous Deployment	0.85	0.83	0.84	0.72
Digital and Mobile Banking	0.79	0.83	0.81	0.68
Web Development	0.93	0.73	0.82	0.69
Banking Services	0.74	0.79	0.76	0.62
Customer Support and Feedback	0.88	0.93	0.90	0.83
Video Hosting and Portals	0.86	0.88	0.87	0.77
Cloud Computing Services	0.72	0.62	0.67	0.50
Health and Beauty	0.78	0.81	0.79	0.66
Game Development	0.65	0.62	0.63	0.46
Artificial Intelligence and Machine Learning	0.80	0.83	0.82	0.69
Social Networks	0.92	0.82	0.87	0.76
Mobile App Development	0.89	0.88	0.88	0.79
Customer Support Services	0.81	0.80	0.81	0.68
Commercial Real Estate	0.91	0.80	0.85	0.74
Cloud-based Development Environments	0.90	0.82	0.86	0.75
Event Planning Services	0.92	0.69	0.79	0.65
Project Management Tools	0.88	0.83	0.86	0.75
Version Control Systems	0.70	0.67	0.69	0.52
Automotive Dealerships	0.60	0.67	0.63	0.46
Insurance Services	0.81	0.60	0.69	0.53
Telecommunication Companies	0.68	0.73	0.71	0.55
Image Stock Platforms	0.95	0.91	0.93	0.87
Toys and Games	0.79	0.79	0.79	0.65
Books and Media	0.74	0.74	0.74	0.58
Residential Real Estate	0.78	0.63	0.69	0.53
Legal Services	0.91	0.83	0.87	0.76
Job Recruitment Agencies	0.84	0.73	0.78	0.64
International Real Estate	0.97	0.84	0.90	0.81
Dating Apps	0.92	0.84	0.88	0.79
Home and Garden	0.70	0.59	0.64	0.47
User Interface/User Experience Design	0.84	0.73	0.78	0.64
Logistics and Supply Chain Management	0.78	0.69	0.73	0.57
Sports and Outdoors	0.80	0.72	0.76	0.61
Team Communication and Chat Tools	0.71	0.61	0.66	0.49
Mortgage and Real Estate Services	0.77	0.67	0.71	0.55
Taxation Services	0.67	0.64	0.66	0.49
Electronics	0.79	0.53	0.63	0.47
Travelers and Consumers	0.91	0.86	0.89	0.80
Financial Planning and Advisory	0.90	0.85	0.88	0.78
Real Estate Consulting	0.77	0.69	0.73	0.57
Property Management	0.86	0.72	0.79	0.65
Government Services	0.96	0.93	0.94	0.89
E-commerce Platforms	0.93	0.89	0.91	0.83
Data Analytics and Business Intelligence	0.96	0.88	0.92	0.85
Documentation and Knowledge Sharing	0.84	0.77	0.80	0.67
Real Estate Investment	0.79	0.73	0.76	0.62
Eco-Friendly and Sustainable Properties	0.85	0.72	0.78	0.64
Task and Time Management	0.91	0.87	0.89	0.80
Issue Tracking and Bug Reporting	0.70	0.57	0.63	0.46
Restaurants and Food Delivery Services	0.80	0.72	0.76	0.61
Luxury and High-End Properties	0.93	0.77	0.84	0.73
Food and Grocery	0.85	0.60	0.70	0.54
Entertainment and Media Platforms	0.80	0.85	0.83	0.70
Real Estate Development	0.86	0.74	0.79	0.66
Market Research Firms	0.90	0.82	0.86	0.75
Investment Services	0.76	0.86	0.81	0.67
Collaborative Development Environments	0.72	0.62	0.66	0.50
Retail Stores (Online and Offline)	0.78	0.71	0.75	0.60
Fashion and Apparel	0.68	0.72	0.70	0.54
Healthcare Providers	0.68	0.61	0.64	0.48
Travel and Booking Agencies	0.80	0.72	0.76	0.61
Credit Services	0.94	0.94	0.94	0.88
Land Sales and Acquisitions	0.90	0.76	0.83	0.70
Internet of Things (IoT) Development	0.90	0.92	0.91	0.83
Risk Management and Compliance	0.74	0.77	0.75	0.60
Pet Supplies	0.85	0.70	0.77	0.62
Aggregate	0.82	0.75	0.78	0.65

Unseen domains metrics

Category	Recall	Precision	F1	Accuracy
Educational Institutions [+]	0.74	0.71	0.73	0.57
Enterprise Software Development [+]	0.80	0.73	0.76	0.62
Professional Social Networks [+]	0.82	0.72	0.76	0.62
Automotive [+]	0.77	0.64	0.70	0.54
Payment Processing [+]	0.80	0.73	0.76	0.62
Aggregate	0.78	0.71	0.74	0.59

Environmental Impact

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: NVIDIA Tesla V100
• Hours used: 72
• Cloud Provider: Google Cloud
• Compute Region: us-west-1
• Carbon Emitted: 6.48

Technical Specifications

Model Architecture and Objective

• Base model: IlyaGusev/saiga_mistral_7b_lora.
• Quantization Configuration: Uses settings like load_in_4bit and bnb_4bit_quant_type for model quantization.

Compute Infrastructure

[To be added]

Hardware

[To be added]

Software

• Python 3.9+
• CUDA 11.7.1
• NVIDIA Compatible Drivers
• Torch 2.0.0

More Information / About us

EmbeddingStudio is an innovative open-source framework designed to seamlessly convert a combined "Embedding Model + Vector DB" into a comprehensive search engine. With built-in functionalities for clickstream collection, continuous improvement of search experiences, and automatic adaptation of the embedding model, it offers an out-of-the-box solution for a full-cycle search engine.

Features

1. 🔄 Turn your vector database into a full-cycle search engine
2. 🖱️ Collect users feedback like clickstream
3. 🚀 (*) Improve search experience on-the-fly without frustrating wait times
4. 📊 (*) Monitor your search quality
5. 🎯 Improve your embedding model through an iterative metric fine-tuning procedure
6. 🆕 (*) Use the new version of the embedding model for inference

(*) - features in development

EmbeddingStudio is highly customizable, so you can bring your own:

1. Data source
2. Vector database
3. Clickstream database
4. Embedding model

For more details visit GitHub Repo.

Model Card Authors and Contact

• Aleksandr Iudaev [LinkedIn] [Email]
• Andrei Kostin [LinkedIn] [Email]
• ML Doom [AI Assistant]

Framework versions

• PEFT 0.5.0
• Datasets 2.16.1
• BitsAndBytes 0.41.0
• PyTorch 2.0.0
• Transformers 4.36.2
• TRL 0.7.7