davidberenstein1957
HF staff
docs: Update local deployment info and getting the right HF token
8bedfed
| import gradio as gr | |
| with gr.Blocks() as app: | |
| with gr.Row(): | |
| with gr.Column(scale=1): | |
| pass | |
| with gr.Column(scale=3): | |
| gr.HTML( | |
| """ | |
| <div style="text-align: justify;"> | |
| <h4 style="text-align: center;">What is Distilabel Synthetic Data Generator?</h4> | |
| <p>Distilabel Synthetic Data Generator is an experimental tool that allows you to easily create high-quality datasets for training and fine-tuning language models. It leverages the power of distilabel and advanced language models to generate synthetic data tailored to your specific needs.</p> | |
| <p>This tool simplifies the process of creating custom datasets, enabling you to:</p> | |
| <ul> | |
| <li>Define the characteristics of your desired application</li> | |
| <li>Generate system prompts automatically</li> | |
| <li>Create sample datasets for quick iteration</li> | |
| <li>Produce full-scale datasets with customizable parameters</li> | |
| <li>Push your generated datasets directly to the Hugging Face Hub</li> | |
| </ul> | |
| <p>By using Distilabel Synthetic Data Generator, you can rapidly prototype and create datasets for, accelerating your AI development process.</p> | |
| <h4 style="text-align: center;">How is this free?</h4> | |
| <p>The current implementation is based on <a href="https://huggingface.co/docs/api-inference/index" target="_blank">Free Serverless Hugging Face Inference Endpoints</a>. They are rate limited but free to use for anyone on the Hugging Face Hub. You can re-use the underlying pipeline to generate data with other <a href="https://distilabel.argilla.io/dev/components-gallery/llms/" target="_blank">distilabel LLM integrations</a>.</p> | |
| <h4 style="text-align: center;">Can I run this locally?</h4> | |
| <p>Yes, you can run this locally by <a href="https://huggingface.co/spaces/argilla/synthetic-data-generator?clone=true" target="_blank">cloning the Space</a> and installing the requirements with `pip install -r requirements.txt` and running `python app.py`. Alternatively, you can install the <a href="https://github.com/argilla-io/distilabel" target="_blank">distilabel library</a> with `pip install distilabel[hf-inference-endpoints]` and use the pipeline code at the bottom of each application tab. Distilabel also supports running the pipeline with <a href="https://distilabel.argilla.io/latest/components-gallery/llms/" target="_blank">other LLMs</a>. Do make sure to get a valid <a href="https://huggingface.co/settings/tokens/new?ownUserPermissions=repo.content.read&ownUserPermissions=repo.write&globalPermissions=inference.serverless.write&canReadGatedRepos=true&tokenType=fineGrained" target="_blank">Hugging Face Token</a> that allows for calling serverless inference endpoints and create datasets on the Hugging Face Hub.</p> | |
| <h4 style="text-align: center;">What is distilabel?</h4> | |
| <p>Distilabel is the framework for synthetic data and AI feedback for engineers who need fast, reliable and scalable pipelines based on verified research papers.</p> | |
| <h4 style="text-align: center;">What is synthetic data?</h4> | |
| <p>Synthetic data is data generated by an AI model, instead of being collected from the real world.</p> | |
| <h4 style="text-align: center;">What is AI feedback?</h4> | |
| <p>AI feedback is feedback provided by an AI model, instead of being provided by a human.</p> | |
| <h4 style="text-align: center;">How is distilabel different from other frameworks?</h4> | |
| <p>Distilabel can be used for generating synthetic data and AI feedback for a wide variety of projects including traditional predictive NLP (classification, extraction, etc.), or generative and large language model scenarios (instruction following, dialogue generation, judging etc.). Distilabel's programmatic approach allows you to build scalable pipelines for data generation and AI feedback. The goal of distilabel is to accelerate your AI development by quickly generating high-quality, diverse datasets based on verified research methodologies for generating and judging with AI feedback. So, Distilabel is focused and specifically designed to be a tool that for scalable and reliable synthetic data generation.</p> | |
| <h4 style="text-align: center;">What do people use distilabel for?</h4> | |
| <p>The Argilla community uses distilabel to create amazing <a href="https://huggingface.co/datasets?other=distilabel" target="_blank">datasets</a> and <a href="https://huggingface.co/models?other=distilabel" target="_blank">models</a>.</p> | |
| <ul> | |
| <li>The <a href="https://huggingface.co/datasets/argilla/OpenHermesPreferences" target="_blank">1M OpenHermesPreference</a> is a dataset of ~1 million AI preferences derived from teknium/OpenHermes-2.5. It shows how we can use distilabel to <strong>synthesize data on an immense scale</strong>.</li> | |
| <li>Our <a href="https://huggingface.co/datasets/argilla/distilabel-intel-orca-dpo-pairs" target="_blank">distilabeled Intel Orca DPO dataset</a> and the <a href="https://huggingface.co/argilla/distilabeled-OpenHermes-2.5-Mistral-7B" target="_blank">improved OpenHermes model</a>, show how we <strong>improve model performance by filtering out 50%</strong> of the original dataset through <strong>AI feedback</strong>.</li> | |
| <li>The <a href="https://github.com/davanstrien/haiku-dpo" target="_blank">haiku DPO data</a> outlines how anyone can create a <strong>dataset for a specific task</strong> and <strong>the latest research papers</strong> to improve the quality of the dataset.</li> | |
| </ul> | |
| </div> | |
| """ | |
| ) | |
| with gr.Column(scale=1): | |
| pass | |