|
--- |
|
base_model: sentence-transformers/all-MiniLM-L6-v2 |
|
datasets: [] |
|
language: |
|
- en |
|
library_name: sentence-transformers |
|
license: apache-2.0 |
|
metrics: |
|
- cosine_accuracy@1 |
|
- cosine_accuracy@3 |
|
- cosine_accuracy@5 |
|
- cosine_accuracy@10 |
|
- cosine_precision@1 |
|
- cosine_precision@3 |
|
- cosine_precision@5 |
|
- cosine_precision@10 |
|
- cosine_recall@1 |
|
- cosine_recall@3 |
|
- cosine_recall@5 |
|
- cosine_recall@10 |
|
- cosine_ndcg@10 |
|
- cosine_mrr@10 |
|
- cosine_map@100 |
|
pipeline_tag: sentence-similarity |
|
tags: |
|
- sentence-transformers |
|
- sentence-similarity |
|
- feature-extraction |
|
- generated_from_trainer |
|
- dataset_size:1490 |
|
- loss:MatryoshkaLoss |
|
- loss:MultipleNegativesRankingLoss |
|
widget: |
|
- source_sentence: What are some examples of long-lived credentials used in ZenML |
|
for authentication? |
|
sentences: |
|
- ' gs://zenml-core_cloudbuild ┃┃ │ gs://zenml-datasets ┃ |
|
|
|
|
|
┃ │ gs://zenml-internal-artifact-store ┃ |
|
|
|
|
|
┃ │ gs://zenml-kubeflow-artifact-store ┃ |
|
|
|
|
|
┃ │ gs://zenml-project-time-series-bucket ┃ |
|
|
|
|
|
┠───────────────────────┼─────────────────────────────────────────────────┨ |
|
|
|
|
|
┃ 🌀 kubernetes-cluster │ zenml-test-cluster ┃ |
|
|
|
|
|
┠───────────────────────┼─────────────────────────────────────────────────┨ |
|
|
|
|
|
┃ 🐳 docker-registry │ gcr.io/zenml-core ┃ |
|
|
|
|
|
┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ |
|
|
|
|
|
Long-lived credentials (API keys, account keys) |
|
|
|
|
|
This is the magic formula of authentication methods. When paired with another |
|
ability, such as automatically generating short-lived API tokens, or impersonating |
|
accounts or assuming roles, this is the ideal authentication mechanism to use, |
|
particularly when using ZenML in production and when sharing results with other |
|
members of your ZenML team. |
|
|
|
|
|
As a general best practice, but implemented particularly well for cloud platforms, |
|
account passwords are never directly used as a credential when authenticating |
|
to the cloud platform APIs. There is always a process in place that exchanges |
|
the account/password credential for another type of long-lived credential: |
|
|
|
|
|
AWS uses the aws configure CLI command |
|
|
|
|
|
GCP offers the gcloud auth application-default login CLI commands |
|
|
|
|
|
Azure provides the az login CLI command |
|
|
|
|
|
None of your original login information is stored on your local machine or used |
|
to access workloads. Instead, an API key, account key or some other form of intermediate |
|
credential is generated and stored on the local host and used to authenticate |
|
to remote cloud service APIs.' |
|
- '─────────────────────────────────────────────────┨┃ VERSION │ |
|
1 ┃ |
|
|
|
|
|
┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┨ |
|
|
|
|
|
┃ VERSION_DESCRIPTION │ Run #1 of the mlflow_training_pipeline. ┃ |
|
|
|
|
|
┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┨ |
|
|
|
|
|
┃ CREATED_AT │ 2023-03-01 09:09:06.899000 ┃ |
|
|
|
|
|
┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┨ |
|
|
|
|
|
┃ UPDATED_AT │ 2023-03-01 09:09:06.899000 ┃' |
|
- 'un.py |
|
|
|
|
|
Read more in the production guide. |
|
|
|
|
|
CleanupMake sure you no longer need the resources before deleting them. The instructions |
|
and commands that follow are DESTRUCTIVE. |
|
|
|
|
|
Delete any AWS resources you no longer use to avoid additional charges. You''ll |
|
want to do the following: |
|
|
|
|
|
# delete the S3 bucket |
|
|
|
|
|
aws s3 rm s3://your-bucket-name --recursive |
|
|
|
|
|
aws s3api delete-bucket --bucket your-bucket-name |
|
|
|
|
|
# delete the SageMaker domain |
|
|
|
|
|
aws sagemaker delete-domain --domain-id <DOMAIN_ID> |
|
|
|
|
|
# delete the ECR repository |
|
|
|
|
|
aws ecr delete-repository --repository-name zenml-repository --force |
|
|
|
|
|
# detach policies from the IAM role |
|
|
|
|
|
aws iam detach-role-policy --role-name zenml-role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess |
|
|
|
|
|
aws iam detach-role-policy --role-name zenml-role --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess |
|
|
|
|
|
aws iam detach-role-policy --role-name zenml-role --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess |
|
|
|
|
|
# delete the IAM role |
|
|
|
|
|
aws iam delete-role --role-name zenml-role |
|
|
|
|
|
Make sure to run these commands in the same AWS region where you created the resources. |
|
|
|
|
|
By running these cleanup commands, you will delete the S3 bucket, SageMaker domain, |
|
ECR repository, and IAM role, along with their associated policies. This will |
|
help you avoid any unnecessary charges for resources you no longer need. |
|
|
|
|
|
Remember to be cautious when deleting resources and ensure that you no longer |
|
require them before running the deletion commands. |
|
|
|
|
|
Conclusion |
|
|
|
|
|
In this guide, we walked through the process of setting up an AWS stack with ZenML |
|
to run your machine learning pipelines in a scalable and production-ready environment. |
|
The key steps included: |
|
|
|
|
|
Setting up credentials and the local environment by creating an IAM role with |
|
the necessary permissions. |
|
|
|
|
|
Creating a ZenML service connector to authenticate with AWS services using the |
|
IAM role. |
|
|
|
|
|
Configuring stack components, including an S3 artifact store, a SageMaker Pipelines |
|
orchestrator, and an ECR container registry.' |
|
- source_sentence: Can you provide more information on how to fetch the last successful |
|
run of a pipeline in ZenML? |
|
sentences: |
|
- 'Connecting remote storage |
|
|
|
|
|
Transitioning to remote artifact storage. |
|
|
|
|
|
In the previous chapters, we''ve been working with artifacts stored locally on |
|
our machines. This setup is fine for individual experiments, but as we move towards |
|
a collaborative and production-ready environment, we need a solution that is more |
|
robust, shareable, and scalable. Enter remote storage! |
|
|
|
|
|
Remote storage allows us to store our artifacts in the cloud, which means they''re |
|
accessible from anywhere and by anyone with the right permissions. This is essential |
|
for team collaboration and for managing the larger datasets and models that come |
|
with production workloads. |
|
|
|
|
|
When using a stack with remote storage, nothing changes except the fact that the |
|
artifacts get materialized in a central and remote storage location. This diagram |
|
explains the flow: |
|
|
|
|
|
Provisioning and registering a remote artifact store |
|
|
|
|
|
Out of the box, ZenML ships with many different supported artifact store flavors. |
|
For convenience, here are some brief instructions on how to quickly get up and |
|
running on the major cloud providers: |
|
|
|
|
|
You will need to install and set up the AWS CLI on your machine as a prerequisite, |
|
as covered in the AWS CLI documentation, before you register the S3 Artifact Store. |
|
|
|
|
|
The Amazon Web Services S3 Artifact Store flavor is provided by the S3 ZenML integration, |
|
you need to install it on your local machine to be able to register an S3 Artifact |
|
Store and add it to your stack: |
|
|
|
|
|
zenml integration install s3 -y |
|
|
|
|
|
Having trouble with this command? You can use poetry or pip to install the requirements |
|
of any ZenML integration directly. In order to obtain the exact requirements of |
|
the AWS S3 integration you can use zenml integration requirements s3. |
|
|
|
|
|
The only configuration parameter mandatory for registering an S3 Artifact Store |
|
is the root path URI, which needs to point to an S3 bucket and take the form s3://bucket-name. |
|
In order to create a S3 bucket, refer to the AWS documentation.' |
|
- 'Load a Model in code |
|
|
|
|
|
There are a few different ways to load a ZenML Model in code: |
|
|
|
|
|
Load the active model in a pipeline |
|
|
|
|
|
You can also use the active model to get the model metadata, or the associated |
|
artifacts directly as described in the starter guide: |
|
|
|
|
|
from zenml import step, pipeline, get_step_context, pipeline, Model |
|
|
|
|
|
@pipeline(model=Model(name="my_model")) |
|
|
|
|
|
def my_pipeline(): |
|
|
|
|
|
... |
|
|
|
|
|
@step |
|
|
|
|
|
def my_step(): |
|
|
|
|
|
# Get model from active step context |
|
|
|
|
|
mv = get_step_context().model |
|
|
|
|
|
# Get metadata |
|
|
|
|
|
print(mv.run_metadata["metadata_key"].value) |
|
|
|
|
|
# Directly fetch an artifact that is attached to the model |
|
|
|
|
|
output = mv.get_artifact("my_dataset", "my_version") |
|
|
|
|
|
output.run_metadata["accuracy"].value |
|
|
|
|
|
Load any model via the Client |
|
|
|
|
|
Alternatively, you can use the Client: |
|
|
|
|
|
from zenml import step |
|
|
|
|
|
from zenml.client import Client |
|
|
|
|
|
from zenml.enums import ModelStages |
|
|
|
|
|
@step |
|
|
|
|
|
def model_evaluator_step() |
|
|
|
|
|
... |
|
|
|
|
|
# Get staging model version |
|
|
|
|
|
try: |
|
|
|
|
|
staging_zenml_model = Client().get_model_version( |
|
|
|
|
|
model_name_or_id="<INSERT_MODEL_NAME>", |
|
|
|
|
|
model_version_name_or_number_or_id=ModelStages.STAGING, |
|
|
|
|
|
except KeyError: |
|
|
|
|
|
staging_zenml_model = None |
|
|
|
|
|
... |
|
|
|
|
|
PreviousControlling Model versions |
|
|
|
|
|
NextPromote a Model |
|
|
|
|
|
Last updated 19 days ago' |
|
- ' more information. |
|
|
|
|
|
Get the last run of a pipelineTo access the most recent run of a pipeline, you |
|
can either use the last_run property or access it through the runs list: |
|
|
|
|
|
last_run = pipeline_model.last_run # OR: pipeline_model.runs[0] |
|
|
|
|
|
If your most recent runs have failed, and you want to find the last run that has |
|
succeeded, you can use the last_successful_run property instead. |
|
|
|
|
|
Get the latest run from a pipeline |
|
|
|
|
|
Calling a pipeline executes it and then returns the response of the freshly executed |
|
run. |
|
|
|
|
|
run = training_pipeline() |
|
|
|
|
|
The run that you get back is the model stored in the ZenML database at the point |
|
of the method call. This means the pipeline run is still initializing and no steps |
|
have been run. To get the latest state can get a refreshed version from the client: |
|
|
|
|
|
from zenml.client import Client |
|
|
|
|
|
Client().get_pipeline_run(run.id) to get a refreshed version |
|
|
|
|
|
Get a run via the client |
|
|
|
|
|
If you already know the exact run that you want to fetch (e.g., from looking at |
|
the dashboard), you can use the Client.get_pipeline_run() method to fetch the |
|
run directly without having to query the pipeline first: |
|
|
|
|
|
from zenml.client import Client |
|
|
|
|
|
pipeline_run = Client().get_pipeline_run("first_pipeline-2023_06_20-16_20_13_274466") |
|
|
|
|
|
Similar to pipelines, you can query runs by either ID, name, or name prefix, and |
|
you can also discover runs through the Client or CLI via the Client.list_pipeline_runs() |
|
or zenml pipeline runs list commands. |
|
|
|
|
|
Run information |
|
|
|
|
|
Each run has a collection of useful information which can help you reproduce your |
|
runs. In the following, you can find a list of some of the most useful pipeline |
|
run information, but there is much more available. See the PipelineRunResponse |
|
definition for a comprehensive list. |
|
|
|
|
|
Status |
|
|
|
|
|
The status of a pipeline run. There are five possible states: initialized, failed, |
|
completed, running, and cached. |
|
|
|
|
|
status = run.status |
|
|
|
|
|
Configuration' |
|
- source_sentence: How does the code in the given documentation visualize the embeddings |
|
using t-SNE and UMAP? |
|
sentences: |
|
- ' |
|
|
|
|
|
embeddings_2d[mask, 0], |
|
|
|
|
|
embeddings_2d[mask, 1],c=[section_color_dict[section]], |
|
|
|
|
|
label=section, |
|
|
|
|
|
plt.title("t-SNE Visualization") |
|
|
|
|
|
plt.legend() |
|
|
|
|
|
plt.show() |
|
|
|
|
|
# Dimensionality reduction using UMAP |
|
|
|
|
|
def umap_visualization(embeddings, parent_sections): |
|
|
|
|
|
umap_2d = umap.UMAP(n_components=2, random_state=42) |
|
|
|
|
|
embeddings_2d = umap_2d.fit_transform(embeddings) |
|
|
|
|
|
plt.figure(figsize=(8, 8)) |
|
|
|
|
|
for section in unique_parent_sections: |
|
|
|
|
|
if section in section_color_dict: |
|
|
|
|
|
mask = [section == ps for ps in parent_sections] |
|
|
|
|
|
plt.scatter( |
|
|
|
|
|
embeddings_2d[mask, 0], |
|
|
|
|
|
embeddings_2d[mask, 1], |
|
|
|
|
|
c=[section_color_dict[section]], |
|
|
|
|
|
label=section, |
|
|
|
|
|
plt.title("UMAP Visualization") |
|
|
|
|
|
plt.legend() |
|
|
|
|
|
plt.show() |
|
|
|
|
|
In this stage, we have utilized the ''parent directory'', which we had previously |
|
stored in the vector store as an additional attribute, as a means to color the |
|
values. This approach allows us to gain some insight into the semantic space inherent |
|
in our data. It demonstrates that you can visualize the embeddings and observe |
|
how similar chunks are grouped together based on their semantic meaning and context. |
|
|
|
|
|
So this step iterates through all the chunks and generates embeddings representing |
|
each piece of text. These embeddings are then stored as an artifact in the ZenML |
|
artifact store as a NumPy array. We separate this generation from the point where |
|
we upload those embeddings to the vector database to keep the pipeline modular |
|
and flexible; in the future we might want to use a different vector database so |
|
we can just swap out the upload step without having to re-generate the embeddings. |
|
|
|
|
|
In the next section, we''ll explore how to store these embeddings in a vector |
|
database to enable fast and efficient retrieval of relevant chunks at inference |
|
time. |
|
|
|
|
|
Code Example |
|
|
|
|
|
To explore the full code, visit the Complete Guide repository. The embeddings |
|
generation step can be found here. |
|
|
|
|
|
PreviousData ingestion and preprocessing |
|
|
|
|
|
NextStoring embeddings in a vector database |
|
|
|
|
|
Last updated 2 months ago' |
|
- 'Amazon SageMaker |
|
|
|
|
|
Executing individual steps in SageMaker. |
|
|
|
|
|
SageMaker offers specialized compute instances to run your training jobs and has |
|
a comprehensive UI to track and manage your models and logs. ZenML''s SageMaker |
|
step operator allows you to submit individual steps to be run on Sagemaker compute |
|
instances. |
|
|
|
|
|
When to use it |
|
|
|
|
|
You should use the SageMaker step operator if: |
|
|
|
|
|
one or more steps of your pipeline require computing resources (CPU, GPU, memory) |
|
that are not provided by your orchestrator. |
|
|
|
|
|
you have access to SageMaker. If you''re using a different cloud provider, take |
|
a look at the Vertex or AzureML step operators. |
|
|
|
|
|
How to deploy it |
|
|
|
|
|
Create a role in the IAM console that you want the jobs running in SageMaker to |
|
assume. This role should at least have the AmazonS3FullAccess and AmazonSageMakerFullAccess |
|
policies applied. Check here for a guide on how to set up this role. |
|
|
|
|
|
Infrastructure Deployment |
|
|
|
|
|
A Sagemaker step operator can be deployed directly from the ZenML CLI: |
|
|
|
|
|
zenml orchestrator deploy sagemaker_step_operator --flavor=sagemaker --provider=aws |
|
... |
|
|
|
|
|
You can pass other configurations specific to the stack components as key-value |
|
arguments. If you don''t provide a name, a random one is generated for you. For |
|
more information about how to work use the CLI for this, please refer to the dedicated |
|
documentation section. |
|
|
|
|
|
How to use it |
|
|
|
|
|
To use the SageMaker step operator, we need: |
|
|
|
|
|
The ZenML aws integration installed. If you haven''t done so, runCopyzenml integration |
|
install aws |
|
|
|
|
|
Docker installed and running. |
|
|
|
|
|
An IAM role with the correct permissions. See the deployment section for detailed |
|
instructions. |
|
|
|
|
|
An AWS container registry as part of our stack. Take a look here for a guide on |
|
how to set that up.' |
|
- 'Deploy with Docker |
|
|
|
|
|
Deploying ZenML in a Docker container. |
|
|
|
|
|
The ZenML server container image is available at zenmldocker/zenml-server and |
|
can be used to deploy ZenML with a container management or orchestration tool |
|
like Docker and docker-compose, or a serverless platform like Cloud Run, Container |
|
Apps, and more! This guide walks you through the various configuration options |
|
that the ZenML server container expects as well as a few deployment use cases. |
|
|
|
|
|
Try it out locally first |
|
|
|
|
|
If you''re just looking for a quick way to deploy the ZenML server using a container, |
|
without going through the hassle of interacting with a container management tool |
|
like Docker and manually configuring your container, you can use the ZenML CLI |
|
to do so. You only need to have Docker installed and running on your machine: |
|
|
|
|
|
zenml up --docker |
|
|
|
|
|
This command deploys a ZenML server locally in a Docker container, then connects |
|
your client to it. Similar to running plain zenml up, the server and the local |
|
ZenML client share the same SQLite database. |
|
|
|
|
|
The rest of this guide is addressed to advanced users who are looking to manually |
|
deploy and manage a containerized ZenML server. |
|
|
|
|
|
ZenML server configuration options |
|
|
|
|
|
If you''re planning on deploying a custom containerized ZenML server yourself, |
|
you probably need to configure some settings for it like the database it should |
|
use, the default user details, and more. The ZenML server container image uses |
|
sensible defaults, so you can simply start a container without worrying too much |
|
about the configuration. However, if you''re looking to connect the ZenML server |
|
to an external MySQL database or secrets management service, to persist the internal |
|
SQLite database, or simply want to control other settings like the default account, |
|
you can do so by customizing the container''s environment variables. |
|
|
|
|
|
The following environment variables can be passed to the container:' |
|
- source_sentence: What is the purpose of the `enable_step_logs` parameter in ZenML? |
|
sentences: |
|
- 'Collect information from your SQL database serviceUsing an external MySQL-compatible |
|
database service is optional, but is recommended for production deployments. If |
|
omitted, ZenML will default to using an embedded SQLite database, which has the |
|
following limitations: |
|
|
|
|
|
the SQLite database is not persisted, meaning that it will be lost if the ZenML |
|
server pod is restarted or deleted |
|
|
|
|
|
the SQLite database does not scale horizontally, meaning that you will not be |
|
able to use more than one replica at a time for the ZenML server pod |
|
|
|
|
|
If you decide to use an external MySQL-compatible database service, you will need |
|
to collect and prepare the following information for the Helm chart configuration: |
|
|
|
|
|
the hostname and port where the SQL database is reachable from the Kubernetes |
|
cluster |
|
|
|
|
|
the username and password that will be used to connect to the database. It is |
|
recommended that you create a dedicated database user for the ZenML server and |
|
that you restrict its privileges to only access the database that will be used |
|
by ZenML. Enforcing secure SSL connections for the user/database is also recommended. |
|
See the MySQL documentation for more information on how to set up users and privileges. |
|
|
|
|
|
the name of the database that will be used by ZenML. The database does not have |
|
to exist prior to the deployment ( ZenML will create it on the first start). However, |
|
you need to create the database if you follow the best practice of restricting |
|
database user privileges to only access it. |
|
|
|
|
|
if you plan on using SSL to secure the client database connection, you may also |
|
need to prepare additional SSL certificates and keys:the TLS CA certificate that |
|
was used to sign the server TLS certificate, if you''re using a self-signed certificate |
|
or signed by a custom certificate authority that is not already trusted by default |
|
by most operating systems.the TLS client certificate and key. This is only needed |
|
if you decide to use client certificates for your DB connection (some managed |
|
DB services support this, CloudSQL is an example).' |
|
- ' │ SHARED │ OWNER │ EXPIRES IN │ LABELS ┃┠────────┼────────────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼─────────────────────────┼────────┼─────────┼────────────┼────────┨ |
|
|
|
|
|
┃ │ gcp-multi │ 9d953320-3560-4a78-817c-926a3898064d │ 🔵 gcp |
|
│ 🔵 gcp-generic │ <multiple> │ ➖ │ default │ │ ┃ |
|
|
|
|
|
┃ │ │ │ │ |
|
📦 gcs-bucket │ │ │ │ │ ┃ |
|
|
|
|
|
┃ │ │ │ │ |
|
🌀 kubernetes-cluster │ │ │ │ │ ┃ |
|
|
|
|
|
┃ │ │ │ │ |
|
🐳 docker-registry │ │ │ │ │ ┃ |
|
|
|
|
|
┠────────┼────────────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼─────────────────────────┼────────┼─────────┼────────────┼────────┨ |
|
|
|
|
|
┃ │ gcs-multi │ ff9c0723-7451-46b7-93ef-fcf3efde30fa │ 🔵 gcp |
|
│ 📦 gcs-bucket │ <multiple> │ ➖ │ default │ │ ┃ |
|
|
|
|
|
┠────────┼────────────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼─────────────────────────┼────────┼─────────┼────────────┼────────┨ |
|
|
|
|
|
┃ │ gcs-langchain-slackbot │ cf3953e9-414c-4875-ba00-24c62a0dc0c5 │ 🔵 gcp |
|
│ 📦 gcs-bucket │ gs://langchain-slackbot │ ➖ │ default │ │ ┃ |
|
|
|
|
|
┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┛ |
|
|
|
|
|
Local and remote availability' |
|
- 'onfiguration, if specified overrides for this stepenable_artifact_metadata: True |
|
|
|
|
|
enable_artifact_visualization: True |
|
|
|
|
|
enable_cache: False |
|
|
|
|
|
enable_step_logs: True |
|
|
|
|
|
# Same as pipeline level configuration, if specified overrides for this step |
|
|
|
|
|
extra: {} |
|
|
|
|
|
# Same as pipeline level configuration, if specified overrides for this step |
|
|
|
|
|
model: {} |
|
|
|
|
|
# Same as pipeline level configuration, if specified overrides for this step |
|
|
|
|
|
settings: |
|
|
|
|
|
docker: {} |
|
|
|
|
|
resources: {} |
|
|
|
|
|
# Stack component specific settings |
|
|
|
|
|
step_operator.sagemaker: |
|
|
|
|
|
estimator_args: |
|
|
|
|
|
instance_type: m7g.medium |
|
|
|
|
|
Deep-dive |
|
|
|
|
|
enable_XXX parameters |
|
|
|
|
|
These are boolean flags for various configurations: |
|
|
|
|
|
enable_artifact_metadata: Whether to associate metadata with artifacts or not. |
|
|
|
|
|
enable_artifact_visualization: Whether to attach visualizations of artifacts. |
|
|
|
|
|
enable_cache: Utilize caching or not. |
|
|
|
|
|
enable_step_logs: Enable tracking step logs. |
|
|
|
|
|
enable_artifact_metadata: True |
|
|
|
|
|
enable_artifact_visualization: True |
|
|
|
|
|
enable_cache: True |
|
|
|
|
|
enable_step_logs: True |
|
|
|
|
|
build ID |
|
|
|
|
|
The UUID of the build to use for this pipeline. If specified, Docker image building |
|
is skipped for remote orchestrators, and the Docker image specified in this build |
|
is used. |
|
|
|
|
|
build: <INSERT-BUILD-ID-HERE> |
|
|
|
|
|
Configuring the model |
|
|
|
|
|
Specifies the ZenML Model to use for this pipeline. |
|
|
|
|
|
model: |
|
|
|
|
|
name: "ModelName" |
|
|
|
|
|
version: "production" |
|
|
|
|
|
description: An example model |
|
|
|
|
|
tags: ["classifier"] |
|
|
|
|
|
Pipeline and step parameters |
|
|
|
|
|
A dictionary of JSON-serializable parameters specified at the pipeline or step |
|
level. For example: |
|
|
|
|
|
parameters: |
|
|
|
|
|
gamma: 0.01 |
|
|
|
|
|
steps: |
|
|
|
|
|
trainer: |
|
|
|
|
|
parameters: |
|
|
|
|
|
gamma: 0.001 |
|
|
|
|
|
Corresponds to: |
|
|
|
|
|
from zenml import step, pipeline |
|
|
|
|
|
@step |
|
|
|
|
|
def trainer(gamma: float): |
|
|
|
|
|
# Use gamma as normal |
|
|
|
|
|
print(gamma) |
|
|
|
|
|
@pipeline |
|
|
|
|
|
def my_pipeline(gamma: float): |
|
|
|
|
|
# use gamma or pass it into the step |
|
|
|
|
|
print(0.01) |
|
|
|
|
|
trainer(gamma=gamma)' |
|
- source_sentence: Can I use ZenML to register an S3 Artifact Store after setting |
|
up the IAM user and generating the access key? |
|
sentences: |
|
- ' to install and configure the AWS CLI on your hostyou don''t need to care about |
|
enabling your other stack components (orchestrators, step operators, and model |
|
deployers) to have access to the artifact store through IAM roles and policies |
|
|
|
|
|
you can combine the S3 artifact store with other stack components that are not |
|
running in AWS |
|
|
|
|
|
Note: When you create the IAM user for your AWS access key, please remember to |
|
grant the created IAM user permissions to read and write to your S3 bucket (i.e. |
|
at a minimum: s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject) |
|
|
|
|
|
After having set up the IAM user and generated the access key, as described in |
|
the AWS documentation, you can register the S3 Artifact Store as follows: |
|
|
|
|
|
# Store the AWS access key in a ZenML secret |
|
|
|
|
|
zenml secret create s3_secret \ |
|
|
|
|
|
--aws_access_key_id=''<YOUR_S3_ACCESS_KEY_ID>'' \ |
|
|
|
|
|
--aws_secret_access_key=''<YOUR_S3_SECRET_KEY>'' |
|
|
|
|
|
# Register the S3 artifact-store and reference the ZenML secret |
|
|
|
|
|
zenml artifact-store register s3_store -f s3 \ |
|
|
|
|
|
--path=''s3://your-bucket'' \ |
|
|
|
|
|
--authentication_secret=s3_secret |
|
|
|
|
|
# Register and set a stack with the new artifact store |
|
|
|
|
|
zenml stack register custom_stack -a s3_store ... --set |
|
|
|
|
|
Advanced Configuration |
|
|
|
|
|
The S3 Artifact Store accepts a range of advanced configuration options that can |
|
be used to further customize how ZenML connects to the S3 storage service that |
|
you are using. These are accessible via the client_kwargs, config_kwargs and s3_additional_kwargs |
|
configuration attributes and are passed transparently to the underlying S3Fs library: |
|
|
|
|
|
client_kwargs: arguments that will be transparently passed to the botocore client |
|
. You can use it to configure parameters like endpoint_url and region_name when |
|
connecting to an S3-compatible endpoint (e.g. Minio). |
|
|
|
|
|
config_kwargs: advanced parameters passed to botocore.client.Config. |
|
|
|
|
|
s3_additional_kwargs: advanced parameters that are used when calling S3 API, typically |
|
used for things like ServerSideEncryption and ACL.' |
|
- 'Evidently |
|
|
|
|
|
How to keep your data quality in check and guard against data and model drift |
|
with Evidently profiling |
|
|
|
|
|
The Evidently Data Validator flavor provided with the ZenML integration uses Evidently |
|
to perform data quality, data drift, model drift and model performance analyses, |
|
to generate reports and run checks. The reports and check results can be used |
|
to implement automated corrective actions in your pipelines or to render interactive |
|
representations for further visual interpretation, evaluation and documentation. |
|
|
|
|
|
When would you want to use it? |
|
|
|
|
|
Evidently is an open-source library that you can use to monitor and debug machine |
|
learning models by analyzing the data that they use through a powerful set of |
|
data profiling and visualization features, or to run a variety of data and model |
|
validation reports and tests, from data integrity tests that work with a single |
|
dataset to model evaluation tests to data drift analysis and model performance |
|
comparison tests. All this can be done with minimal configuration input from the |
|
user, or customized with specialized conditions that the validation tests should |
|
perform. |
|
|
|
|
|
Evidently currently works with tabular data in pandas.DataFrame or CSV file formats |
|
and can handle both regression and classification tasks. |
|
|
|
|
|
You should use the Evidently Data Validator when you need the following data and/or |
|
model validation features that are possible with Evidently: |
|
|
|
|
|
Data Quality reports and tests: provides detailed feature statistics and a feature |
|
behavior overview for a single dataset. It can also compare any two datasets. |
|
E.g. you can use it to compare train and test data, reference and current data, |
|
or two subgroups of one dataset. |
|
|
|
|
|
Data Drift reports and tests: helps detects and explore feature distribution changes |
|
in the input data by comparing two datasets with identical schema.' |
|
- ' us know! |
|
|
|
|
|
Configuration at pipeline or step levelWhen running your ZenML pipeline with the |
|
Sagemaker orchestrator, the configuration set when configuring the orchestrator |
|
as a ZenML component will be used by default. However, it is possible to provide |
|
additional configuration at the pipeline or step level. This allows you to run |
|
whole pipelines or individual steps with alternative configurations. For example, |
|
this allows you to run the training process with a heavier, GPU-enabled instance |
|
type, while running other steps with lighter instances. |
|
|
|
|
|
Additional configuration for the Sagemaker orchestrator can be passed via SagemakerOrchestratorSettings. |
|
Here, it is possible to configure processor_args, which is a dictionary of arguments |
|
for the Processor. For available arguments, see the Sagemaker documentation . |
|
Currently, it is not possible to provide custom configuration for the following |
|
attributes: |
|
|
|
|
|
image_uri |
|
|
|
|
|
instance_count |
|
|
|
|
|
sagemaker_session |
|
|
|
|
|
entrypoint |
|
|
|
|
|
base_job_name |
|
|
|
|
|
env |
|
|
|
|
|
For example, settings can be provided in the following way: |
|
|
|
|
|
sagemaker_orchestrator_settings = SagemakerOrchestratorSettings( |
|
|
|
|
|
processor_args={ |
|
|
|
|
|
"instance_type": "ml.t3.medium", |
|
|
|
|
|
"volume_size_in_gb": 30 |
|
|
|
|
|
They can then be applied to a step as follows: |
|
|
|
|
|
@step(settings={"orchestrator.sagemaker": sagemaker_orchestrator_settings}) |
|
|
|
|
|
For example, if your ZenML component is configured to use ml.c5.xlarge with 400GB |
|
additional storage by default, all steps will use it except for the step above, |
|
which will use ml.t3.medium with 30GB additional storage. |
|
|
|
|
|
Check out this docs page for more information on how to specify settings in general. |
|
|
|
|
|
For more information and a full list of configurable attributes of the Sagemaker |
|
orchestrator, check out the SDK Docs . |
|
|
|
|
|
S3 data access in ZenML steps' |
|
model-index: |
|
- name: zenml/finetuned-all-MiniLM-L6-v2 |
|
results: |
|
- task: |
|
type: information-retrieval |
|
name: Information Retrieval |
|
dataset: |
|
name: dim 384 |
|
type: dim_384 |
|
metrics: |
|
- type: cosine_accuracy@1 |
|
value: 0.3313253012048193 |
|
name: Cosine Accuracy@1 |
|
- type: cosine_accuracy@3 |
|
value: 0.6024096385542169 |
|
name: Cosine Accuracy@3 |
|
- type: cosine_accuracy@5 |
|
value: 0.6927710843373494 |
|
name: Cosine Accuracy@5 |
|
- type: cosine_accuracy@10 |
|
value: 0.7710843373493976 |
|
name: Cosine Accuracy@10 |
|
- type: cosine_precision@1 |
|
value: 0.3313253012048193 |
|
name: Cosine Precision@1 |
|
- type: cosine_precision@3 |
|
value: 0.2008032128514056 |
|
name: Cosine Precision@3 |
|
- type: cosine_precision@5 |
|
value: 0.13855421686746985 |
|
name: Cosine Precision@5 |
|
- type: cosine_precision@10 |
|
value: 0.07710843373493974 |
|
name: Cosine Precision@10 |
|
- type: cosine_recall@1 |
|
value: 0.3313253012048193 |
|
name: Cosine Recall@1 |
|
- type: cosine_recall@3 |
|
value: 0.6024096385542169 |
|
name: Cosine Recall@3 |
|
- type: cosine_recall@5 |
|
value: 0.6927710843373494 |
|
name: Cosine Recall@5 |
|
- type: cosine_recall@10 |
|
value: 0.7710843373493976 |
|
name: Cosine Recall@10 |
|
- type: cosine_ndcg@10 |
|
value: 0.551850042031417 |
|
name: Cosine Ndcg@10 |
|
- type: cosine_mrr@10 |
|
value: 0.4812392426850258 |
|
name: Cosine Mrr@10 |
|
- type: cosine_map@100 |
|
value: 0.48905601510986996 |
|
name: Cosine Map@100 |
|
- task: |
|
type: information-retrieval |
|
name: Information Retrieval |
|
dataset: |
|
name: dim 256 |
|
type: dim_256 |
|
metrics: |
|
- type: cosine_accuracy@1 |
|
value: 0.3253012048192771 |
|
name: Cosine Accuracy@1 |
|
- type: cosine_accuracy@3 |
|
value: 0.5903614457831325 |
|
name: Cosine Accuracy@3 |
|
- type: cosine_accuracy@5 |
|
value: 0.6867469879518072 |
|
name: Cosine Accuracy@5 |
|
- type: cosine_accuracy@10 |
|
value: 0.7710843373493976 |
|
name: Cosine Accuracy@10 |
|
- type: cosine_precision@1 |
|
value: 0.3253012048192771 |
|
name: Cosine Precision@1 |
|
- type: cosine_precision@3 |
|
value: 0.19678714859437746 |
|
name: Cosine Precision@3 |
|
- type: cosine_precision@5 |
|
value: 0.13734939759036144 |
|
name: Cosine Precision@5 |
|
- type: cosine_precision@10 |
|
value: 0.07710843373493974 |
|
name: Cosine Precision@10 |
|
- type: cosine_recall@1 |
|
value: 0.3253012048192771 |
|
name: Cosine Recall@1 |
|
- type: cosine_recall@3 |
|
value: 0.5903614457831325 |
|
name: Cosine Recall@3 |
|
- type: cosine_recall@5 |
|
value: 0.6867469879518072 |
|
name: Cosine Recall@5 |
|
- type: cosine_recall@10 |
|
value: 0.7710843373493976 |
|
name: Cosine Recall@10 |
|
- type: cosine_ndcg@10 |
|
value: 0.5441628856415894 |
|
name: Cosine Ndcg@10 |
|
- type: cosine_mrr@10 |
|
value: 0.4714883342895392 |
|
name: Cosine Mrr@10 |
|
- type: cosine_map@100 |
|
value: 0.4790082728748276 |
|
name: Cosine Map@100 |
|
- task: |
|
type: information-retrieval |
|
name: Information Retrieval |
|
dataset: |
|
name: dim 128 |
|
type: dim_128 |
|
metrics: |
|
- type: cosine_accuracy@1 |
|
value: 0.3313253012048193 |
|
name: Cosine Accuracy@1 |
|
- type: cosine_accuracy@3 |
|
value: 0.5120481927710844 |
|
name: Cosine Accuracy@3 |
|
- type: cosine_accuracy@5 |
|
value: 0.6144578313253012 |
|
name: Cosine Accuracy@5 |
|
- type: cosine_accuracy@10 |
|
value: 0.6987951807228916 |
|
name: Cosine Accuracy@10 |
|
- type: cosine_precision@1 |
|
value: 0.3313253012048193 |
|
name: Cosine Precision@1 |
|
- type: cosine_precision@3 |
|
value: 0.1706827309236948 |
|
name: Cosine Precision@3 |
|
- type: cosine_precision@5 |
|
value: 0.12289156626506023 |
|
name: Cosine Precision@5 |
|
- type: cosine_precision@10 |
|
value: 0.06987951807228915 |
|
name: Cosine Precision@10 |
|
- type: cosine_recall@1 |
|
value: 0.3313253012048193 |
|
name: Cosine Recall@1 |
|
- type: cosine_recall@3 |
|
value: 0.5120481927710844 |
|
name: Cosine Recall@3 |
|
- type: cosine_recall@5 |
|
value: 0.6144578313253012 |
|
name: Cosine Recall@5 |
|
- type: cosine_recall@10 |
|
value: 0.6987951807228916 |
|
name: Cosine Recall@10 |
|
- type: cosine_ndcg@10 |
|
value: 0.5108893388836802 |
|
name: Cosine Ndcg@10 |
|
- type: cosine_mrr@10 |
|
value: 0.45126936316695354 |
|
name: Cosine Mrr@10 |
|
- type: cosine_map@100 |
|
value: 0.4605530012141939 |
|
name: Cosine Map@100 |
|
- task: |
|
type: information-retrieval |
|
name: Information Retrieval |
|
dataset: |
|
name: dim 64 |
|
type: dim_64 |
|
metrics: |
|
- type: cosine_accuracy@1 |
|
value: 0.2891566265060241 |
|
name: Cosine Accuracy@1 |
|
- type: cosine_accuracy@3 |
|
value: 0.4759036144578313 |
|
name: Cosine Accuracy@3 |
|
- type: cosine_accuracy@5 |
|
value: 0.5542168674698795 |
|
name: Cosine Accuracy@5 |
|
- type: cosine_accuracy@10 |
|
value: 0.6265060240963856 |
|
name: Cosine Accuracy@10 |
|
- type: cosine_precision@1 |
|
value: 0.2891566265060241 |
|
name: Cosine Precision@1 |
|
- type: cosine_precision@3 |
|
value: 0.15863453815261044 |
|
name: Cosine Precision@3 |
|
- type: cosine_precision@5 |
|
value: 0.1108433734939759 |
|
name: Cosine Precision@5 |
|
- type: cosine_precision@10 |
|
value: 0.06265060240963854 |
|
name: Cosine Precision@10 |
|
- type: cosine_recall@1 |
|
value: 0.2891566265060241 |
|
name: Cosine Recall@1 |
|
- type: cosine_recall@3 |
|
value: 0.4759036144578313 |
|
name: Cosine Recall@3 |
|
- type: cosine_recall@5 |
|
value: 0.5542168674698795 |
|
name: Cosine Recall@5 |
|
- type: cosine_recall@10 |
|
value: 0.6265060240963856 |
|
name: Cosine Recall@10 |
|
- type: cosine_ndcg@10 |
|
value: 0.45650145038804574 |
|
name: Cosine Ndcg@10 |
|
- type: cosine_mrr@10 |
|
value: 0.40227337923121054 |
|
name: Cosine Mrr@10 |
|
- type: cosine_map@100 |
|
value: 0.4137670603435629 |
|
name: Cosine Map@100 |
|
--- |
|
|
|
# zenml/finetuned-all-MiniLM-L6-v2 |
|
|
|
This is a [sentence-transformers](https://www.SBERT.net) model finetuned from [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2). It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
- **Model Type:** Sentence Transformer |
|
- **Base model:** [sentence-transformers/all-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) <!-- at revision 8b3219a92973c328a8e22fadcfa821b5dc75636a --> |
|
- **Maximum Sequence Length:** 256 tokens |
|
- **Output Dimensionality:** 384 tokens |
|
- **Similarity Function:** Cosine Similarity |
|
<!-- - **Training Dataset:** Unknown --> |
|
- **Language:** en |
|
- **License:** apache-2.0 |
|
|
|
### Model Sources |
|
|
|
- **Documentation:** [Sentence Transformers Documentation](https://sbert.net) |
|
- **Repository:** [Sentence Transformers on GitHub](https://github.com/UKPLab/sentence-transformers) |
|
- **Hugging Face:** [Sentence Transformers on Hugging Face](https://huggingface.co/models?library=sentence-transformers) |
|
|
|
### Full Model Architecture |
|
|
|
``` |
|
SentenceTransformer( |
|
(0): Transformer({'max_seq_length': 256, 'do_lower_case': False}) with Transformer model: BertModel |
|
(1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True}) |
|
(2): Normalize() |
|
) |
|
``` |
|
|
|
## Usage |
|
|
|
### Direct Usage (Sentence Transformers) |
|
|
|
First install the Sentence Transformers library: |
|
|
|
```bash |
|
pip install -U sentence-transformers |
|
``` |
|
|
|
Then you can load this model and run inference. |
|
```python |
|
from sentence_transformers import SentenceTransformer |
|
|
|
# Download from the 🤗 Hub |
|
model = SentenceTransformer("zenml/finetuned-all-MiniLM-L6-v2") |
|
# Run inference |
|
sentences = [ |
|
'Can I use ZenML to register an S3 Artifact Store after setting up the IAM user and generating the access key?', |
|
" to install and configure the AWS CLI on your hostyou don't need to care about enabling your other stack components (orchestrators, step operators, and model deployers) to have access to the artifact store through IAM roles and policies\n\nyou can combine the S3 artifact store with other stack components that are not running in AWS\n\nNote: When you create the IAM user for your AWS access key, please remember to grant the created IAM user permissions to read and write to your S3 bucket (i.e. at a minimum: s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject)\n\nAfter having set up the IAM user and generated the access key, as described in the AWS documentation, you can register the S3 Artifact Store as follows:\n\n# Store the AWS access key in a ZenML secret\n\nzenml secret create s3_secret \\\n\n--aws_access_key_id='<YOUR_S3_ACCESS_KEY_ID>' \\\n\n--aws_secret_access_key='<YOUR_S3_SECRET_KEY>'\n\n# Register the S3 artifact-store and reference the ZenML secret\n\nzenml artifact-store register s3_store -f s3 \\\n\n--path='s3://your-bucket' \\\n\n--authentication_secret=s3_secret\n\n# Register and set a stack with the new artifact store\n\nzenml stack register custom_stack -a s3_store ... --set\n\nAdvanced Configuration\n\nThe S3 Artifact Store accepts a range of advanced configuration options that can be used to further customize how ZenML connects to the S3 storage service that you are using. These are accessible via the client_kwargs, config_kwargs and s3_additional_kwargs configuration attributes and are passed transparently to the underlying S3Fs library:\n\nclient_kwargs: arguments that will be transparently passed to the botocore client . You can use it to configure parameters like endpoint_url and region_name when connecting to an S3-compatible endpoint (e.g. Minio).\n\nconfig_kwargs: advanced parameters passed to botocore.client.Config.\n\ns3_additional_kwargs: advanced parameters that are used when calling S3 API, typically used for things like ServerSideEncryption and ACL.", |
|
' us know!\n\nConfiguration at pipeline or step levelWhen running your ZenML pipeline with the Sagemaker orchestrator, the configuration set when configuring the orchestrator as a ZenML component will be used by default. However, it is possible to provide additional configuration at the pipeline or step level. This allows you to run whole pipelines or individual steps with alternative configurations. For example, this allows you to run the training process with a heavier, GPU-enabled instance type, while running other steps with lighter instances.\n\nAdditional configuration for the Sagemaker orchestrator can be passed via SagemakerOrchestratorSettings. Here, it is possible to configure processor_args, which is a dictionary of arguments for the Processor. For available arguments, see the Sagemaker documentation . Currently, it is not possible to provide custom configuration for the following attributes:\n\nimage_uri\n\ninstance_count\n\nsagemaker_session\n\nentrypoint\n\nbase_job_name\n\nenv\n\nFor example, settings can be provided in the following way:\n\nsagemaker_orchestrator_settings = SagemakerOrchestratorSettings(\n\nprocessor_args={\n\n"instance_type": "ml.t3.medium",\n\n"volume_size_in_gb": 30\n\nThey can then be applied to a step as follows:\n\n@step(settings={"orchestrator.sagemaker": sagemaker_orchestrator_settings})\n\nFor example, if your ZenML component is configured to use ml.c5.xlarge with 400GB additional storage by default, all steps will use it except for the step above, which will use ml.t3.medium with 30GB additional storage.\n\nCheck out this docs page for more information on how to specify settings in general.\n\nFor more information and a full list of configurable attributes of the Sagemaker orchestrator, check out the SDK Docs .\n\nS3 data access in ZenML steps', |
|
] |
|
embeddings = model.encode(sentences) |
|
print(embeddings.shape) |
|
# [3, 384] |
|
|
|
# Get the similarity scores for the embeddings |
|
similarities = model.similarity(embeddings, embeddings) |
|
print(similarities.shape) |
|
# [3, 3] |
|
``` |
|
|
|
<!-- |
|
### Direct Usage (Transformers) |
|
|
|
<details><summary>Click to see the direct usage in Transformers</summary> |
|
|
|
</details> |
|
--> |
|
|
|
<!-- |
|
### Downstream Usage (Sentence Transformers) |
|
|
|
You can finetune this model on your own dataset. |
|
|
|
<details><summary>Click to expand</summary> |
|
|
|
</details> |
|
--> |
|
|
|
<!-- |
|
### Out-of-Scope Use |
|
|
|
*List how the model may foreseeably be misused and address what users ought not to do with the model.* |
|
--> |
|
|
|
## Evaluation |
|
|
|
### Metrics |
|
|
|
#### Information Retrieval |
|
* Dataset: `dim_384` |
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
| Metric | Value | |
|
|:--------------------|:-----------| |
|
| cosine_accuracy@1 | 0.3313 | |
|
| cosine_accuracy@3 | 0.6024 | |
|
| cosine_accuracy@5 | 0.6928 | |
|
| cosine_accuracy@10 | 0.7711 | |
|
| cosine_precision@1 | 0.3313 | |
|
| cosine_precision@3 | 0.2008 | |
|
| cosine_precision@5 | 0.1386 | |
|
| cosine_precision@10 | 0.0771 | |
|
| cosine_recall@1 | 0.3313 | |
|
| cosine_recall@3 | 0.6024 | |
|
| cosine_recall@5 | 0.6928 | |
|
| cosine_recall@10 | 0.7711 | |
|
| cosine_ndcg@10 | 0.5519 | |
|
| cosine_mrr@10 | 0.4812 | |
|
| **cosine_map@100** | **0.4891** | |
|
|
|
#### Information Retrieval |
|
* Dataset: `dim_256` |
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
| Metric | Value | |
|
|:--------------------|:----------| |
|
| cosine_accuracy@1 | 0.3253 | |
|
| cosine_accuracy@3 | 0.5904 | |
|
| cosine_accuracy@5 | 0.6867 | |
|
| cosine_accuracy@10 | 0.7711 | |
|
| cosine_precision@1 | 0.3253 | |
|
| cosine_precision@3 | 0.1968 | |
|
| cosine_precision@5 | 0.1373 | |
|
| cosine_precision@10 | 0.0771 | |
|
| cosine_recall@1 | 0.3253 | |
|
| cosine_recall@3 | 0.5904 | |
|
| cosine_recall@5 | 0.6867 | |
|
| cosine_recall@10 | 0.7711 | |
|
| cosine_ndcg@10 | 0.5442 | |
|
| cosine_mrr@10 | 0.4715 | |
|
| **cosine_map@100** | **0.479** | |
|
|
|
#### Information Retrieval |
|
* Dataset: `dim_128` |
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
| Metric | Value | |
|
|:--------------------|:-----------| |
|
| cosine_accuracy@1 | 0.3313 | |
|
| cosine_accuracy@3 | 0.512 | |
|
| cosine_accuracy@5 | 0.6145 | |
|
| cosine_accuracy@10 | 0.6988 | |
|
| cosine_precision@1 | 0.3313 | |
|
| cosine_precision@3 | 0.1707 | |
|
| cosine_precision@5 | 0.1229 | |
|
| cosine_precision@10 | 0.0699 | |
|
| cosine_recall@1 | 0.3313 | |
|
| cosine_recall@3 | 0.512 | |
|
| cosine_recall@5 | 0.6145 | |
|
| cosine_recall@10 | 0.6988 | |
|
| cosine_ndcg@10 | 0.5109 | |
|
| cosine_mrr@10 | 0.4513 | |
|
| **cosine_map@100** | **0.4606** | |
|
|
|
#### Information Retrieval |
|
* Dataset: `dim_64` |
|
* Evaluated with [<code>InformationRetrievalEvaluator</code>](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) |
|
|
|
| Metric | Value | |
|
|:--------------------|:-----------| |
|
| cosine_accuracy@1 | 0.2892 | |
|
| cosine_accuracy@3 | 0.4759 | |
|
| cosine_accuracy@5 | 0.5542 | |
|
| cosine_accuracy@10 | 0.6265 | |
|
| cosine_precision@1 | 0.2892 | |
|
| cosine_precision@3 | 0.1586 | |
|
| cosine_precision@5 | 0.1108 | |
|
| cosine_precision@10 | 0.0627 | |
|
| cosine_recall@1 | 0.2892 | |
|
| cosine_recall@3 | 0.4759 | |
|
| cosine_recall@5 | 0.5542 | |
|
| cosine_recall@10 | 0.6265 | |
|
| cosine_ndcg@10 | 0.4565 | |
|
| cosine_mrr@10 | 0.4023 | |
|
| **cosine_map@100** | **0.4138** | |
|
|
|
<!-- |
|
## Bias, Risks and Limitations |
|
|
|
*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* |
|
--> |
|
|
|
<!-- |
|
### Recommendations |
|
|
|
*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* |
|
--> |
|
|
|
## Training Details |
|
|
|
### Training Dataset |
|
|
|
#### Unnamed Dataset |
|
|
|
|
|
* Size: 1,490 training samples |
|
* Columns: <code>positive</code> and <code>anchor</code> |
|
* Approximate statistics based on the first 1000 samples: |
|
| | positive | anchor | |
|
|:--------|:----------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------| |
|
| type | string | string | |
|
| details | <ul><li>min: 9 tokens</li><li>mean: 21.16 tokens</li><li>max: 49 tokens</li></ul> | <ul><li>min: 21 tokens</li><li>mean: 239.82 tokens</li><li>max: 256 tokens</li></ul> | |
|
* Samples: |
|
| positive | anchor | |
|
|:------------------------------------------------------------------------------------------------------------------------------------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
|
| <code>Could you explain how to configure step-specific settings like the experiment_tracker and step_operator in ZenML docs?</code> | <code>docs.<br><br>failure_hook_source and success_hook_sourceThe source of the failure and success hooks can be specified.<br><br>Step-specific configuration<br><br>A lot of pipeline-level configuration can also be applied at a step level (as we have already seen with the enable_cache flag). However, there is some configuration that is step-specific, meaning it cannot be applied at a pipeline level, but only at a step level.<br><br>experiment_tracker: Name of the experiment_tracker to enable for this step. This experiment_tracker should be defined in the active stack with the same name.<br><br>step_operator: Name of the step_operator to enable for this step. This step_operator should be defined in the active stack with the same name.<br><br>outputs: This is configuration of the output artifacts of this step. This is further keyed by output name (by default, step outputs are named output). The most interesting configuration here is the materializer_source, which is the UDF path of the materializer in code to use for this output (e.g. materializers.some_data.materializer.materializer_class). Read more about this source path here.<br><br>PreviousHow to configure a pipeline with a YAML<br><br>NextRuntime settings for Docker, resources, and stack components<br><br>Last updated 19 days ago</code> | |
|
| <code>How do I configure ZenML to use the Azure Key Vault as a secrets store backend?</code> | <code>COUNT_NAME>@<PROJECT_NAME>.iam.gserviceaccount.comUsing the Azure Key Vault as a secrets store backend<br><br>The Azure Secrets Store uses the ZenML Azure Service Connector under the hood to authenticate with the Azure Key Vault API. This means that you can use any of the authentication methods supported by the Azure Service Connector to authenticate with the Azure Key Vault API.<br><br>Example configuration for the Azure Key Vault Secrets Store:<br><br>zenml:<br><br># ...<br><br># Secrets store settings. This is used to store centralized secrets.<br><br>secretsStore:<br><br># Set to false to disable the secrets store.<br><br>enabled: true<br><br># The type of the secrets store<br><br>type: azure<br><br># Configuration for the Azure Key Vault secrets store<br><br>azure:<br><br># The name of the Azure Key Vault. This must be set to point to the Azure<br><br># Key Vault instance that you want to use.<br><br>key_vault_name:<br><br># The Azure Service Connector authentication method to use.<br><br>authMethod: service-principal<br><br># The Azure Service Connector configuration.<br><br>authConfig:<br><br># The Azure application service principal credentials to use to<br><br># authenticate with the Azure Key Vault API.<br><br>client_id: <your Azure client ID><br><br>client_secret: <your Azure client secret><br><br>tenant_id: <your Azure tenant ID><br><br>Using the HashiCorp Vault as a secrets store backend<br><br>To use the HashiCorp Vault service as a Secrets Store back-end, it must be configured in the Helm values:<br><br>zenml:<br><br># ...<br><br># Secrets store settings. This is used to store centralized secrets.<br><br>secretsStore:<br><br># Set to false to disable the secrets store.<br><br>enabled: true<br><br># The type of the secrets store<br><br>type: hashicorp<br><br># Configuration for the HashiCorp Vault secrets store<br><br>hashicorp:<br><br># The url of the HashiCorp Vault server to use<br><br>vault_addr: https://vault.example.com<br><br># The token used to authenticate with the Vault server<br><br>vault_token: <your Vault token><br><br># The Vault Enterprise namespace. Not required for Vault OSS.<br><br>vault_namespace: <your Vault namespace><br><br>Using a custom secrets store backend implementation</code> | |
|
| <code>How do I register a service connector with AWS using ZenML?</code> | <code>r the local cloud provider CLI (AWS in this case):AWS_PROFILE=connectors zenml service-connector register aws-sts-token --type aws --auto-configure --auth-method sts-token<br><br>Example Command Output<br><br>⠸ Registering service connector 'aws-sts-token'...<br><br>Successfully registered service connector `aws-sts-token` with access to the following resources:<br><br>┏━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓<br><br>┃ RESOURCE TYPE │ RESOURCE NAMES ┃<br><br>┠───────────────────────┼──────────────────────────────────────────────┨<br><br>┃ 🔶 aws-generic │ us-east-1 ┃<br><br>┠───────────────────────┼──────────────────────────────────────────────┨<br><br>┃ 📦 s3-bucket │ s3://zenfiles ┃<br><br>┃ │ s3://zenml-demos ┃<br><br>┃ │ s3://zenml-generative-chat ┃<br><br>┃ │ s3://zenml-public-datasets ┃<br><br>┠───────────────────────┼──────────────────────────────────────────────┨<br><br>┃ 🌀 kubernetes-cluster │ zenhacks-cluster ┃<br><br>┠───────────────────────┼──────────────────────────────────────────────┨<br><br>┃ 🐳 docker-registry │ 715803424590.dkr.ecr.us-east-1.amazonaws.com ┃<br><br>┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛<br><br>The Service Connector is now configured with a short-lived token that will expire after some time. You can verify this by inspecting the Service Connector:<br><br>zenml service-connector describe aws-sts-token<br><br>Example Command Output<br><br>Service connector 'aws-sts-token' of type 'aws' with id '63e14350-6719-4255-b3f5-0539c8f7c303' is owned by user 'default' and is 'private'.<br><br>'aws-sts-token' aws Service Connector Details<br><br>┏━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓<br><br>┃ PROPERTY │ VALUE ┃</code> | |
|
* Loss: [<code>MatryoshkaLoss</code>](https://sbert.net/docs/package_reference/sentence_transformer/losses.html#matryoshkaloss) with these parameters: |
|
```json |
|
{ |
|
"loss": "MultipleNegativesRankingLoss", |
|
"matryoshka_dims": [ |
|
384, |
|
256, |
|
128, |
|
64 |
|
], |
|
"matryoshka_weights": [ |
|
1, |
|
1, |
|
1, |
|
1 |
|
], |
|
"n_dims_per_step": -1 |
|
} |
|
``` |
|
|
|
### Training Hyperparameters |
|
#### Non-Default Hyperparameters |
|
|
|
- `eval_strategy`: epoch |
|
- `per_device_train_batch_size`: 32 |
|
- `per_device_eval_batch_size`: 16 |
|
- `gradient_accumulation_steps`: 16 |
|
- `learning_rate`: 2e-05 |
|
- `num_train_epochs`: 4 |
|
- `lr_scheduler_type`: cosine |
|
- `warmup_ratio`: 0.1 |
|
- `bf16`: True |
|
- `tf32`: True |
|
- `load_best_model_at_end`: True |
|
- `optim`: adamw_torch_fused |
|
- `batch_sampler`: no_duplicates |
|
|
|
#### All Hyperparameters |
|
<details><summary>Click to expand</summary> |
|
|
|
- `overwrite_output_dir`: False |
|
- `do_predict`: False |
|
- `eval_strategy`: epoch |
|
- `prediction_loss_only`: True |
|
- `per_device_train_batch_size`: 32 |
|
- `per_device_eval_batch_size`: 16 |
|
- `per_gpu_train_batch_size`: None |
|
- `per_gpu_eval_batch_size`: None |
|
- `gradient_accumulation_steps`: 16 |
|
- `eval_accumulation_steps`: None |
|
- `learning_rate`: 2e-05 |
|
- `weight_decay`: 0.0 |
|
- `adam_beta1`: 0.9 |
|
- `adam_beta2`: 0.999 |
|
- `adam_epsilon`: 1e-08 |
|
- `max_grad_norm`: 1.0 |
|
- `num_train_epochs`: 4 |
|
- `max_steps`: -1 |
|
- `lr_scheduler_type`: cosine |
|
- `lr_scheduler_kwargs`: {} |
|
- `warmup_ratio`: 0.1 |
|
- `warmup_steps`: 0 |
|
- `log_level`: passive |
|
- `log_level_replica`: warning |
|
- `log_on_each_node`: True |
|
- `logging_nan_inf_filter`: True |
|
- `save_safetensors`: True |
|
- `save_on_each_node`: False |
|
- `save_only_model`: False |
|
- `restore_callback_states_from_checkpoint`: False |
|
- `no_cuda`: False |
|
- `use_cpu`: False |
|
- `use_mps_device`: False |
|
- `seed`: 42 |
|
- `data_seed`: None |
|
- `jit_mode_eval`: False |
|
- `use_ipex`: False |
|
- `bf16`: True |
|
- `fp16`: False |
|
- `fp16_opt_level`: O1 |
|
- `half_precision_backend`: auto |
|
- `bf16_full_eval`: False |
|
- `fp16_full_eval`: False |
|
- `tf32`: True |
|
- `local_rank`: 0 |
|
- `ddp_backend`: None |
|
- `tpu_num_cores`: None |
|
- `tpu_metrics_debug`: False |
|
- `debug`: [] |
|
- `dataloader_drop_last`: False |
|
- `dataloader_num_workers`: 0 |
|
- `dataloader_prefetch_factor`: None |
|
- `past_index`: -1 |
|
- `disable_tqdm`: True |
|
- `remove_unused_columns`: True |
|
- `label_names`: None |
|
- `load_best_model_at_end`: True |
|
- `ignore_data_skip`: False |
|
- `fsdp`: [] |
|
- `fsdp_min_num_params`: 0 |
|
- `fsdp_config`: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False} |
|
- `fsdp_transformer_layer_cls_to_wrap`: None |
|
- `accelerator_config`: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None} |
|
- `deepspeed`: None |
|
- `label_smoothing_factor`: 0.0 |
|
- `optim`: adamw_torch_fused |
|
- `optim_args`: None |
|
- `adafactor`: False |
|
- `group_by_length`: False |
|
- `length_column_name`: length |
|
- `ddp_find_unused_parameters`: None |
|
- `ddp_bucket_cap_mb`: None |
|
- `ddp_broadcast_buffers`: False |
|
- `dataloader_pin_memory`: True |
|
- `dataloader_persistent_workers`: False |
|
- `skip_memory_metrics`: True |
|
- `use_legacy_prediction_loop`: False |
|
- `push_to_hub`: False |
|
- `resume_from_checkpoint`: None |
|
- `hub_model_id`: None |
|
- `hub_strategy`: every_save |
|
- `hub_private_repo`: False |
|
- `hub_always_push`: False |
|
- `gradient_checkpointing`: False |
|
- `gradient_checkpointing_kwargs`: None |
|
- `include_inputs_for_metrics`: False |
|
- `eval_do_concat_batches`: True |
|
- `fp16_backend`: auto |
|
- `push_to_hub_model_id`: None |
|
- `push_to_hub_organization`: None |
|
- `mp_parameters`: |
|
- `auto_find_batch_size`: False |
|
- `full_determinism`: False |
|
- `torchdynamo`: None |
|
- `ray_scope`: last |
|
- `ddp_timeout`: 1800 |
|
- `torch_compile`: False |
|
- `torch_compile_backend`: None |
|
- `torch_compile_mode`: None |
|
- `dispatch_batches`: None |
|
- `split_batches`: None |
|
- `include_tokens_per_second`: False |
|
- `include_num_input_tokens_seen`: False |
|
- `neftune_noise_alpha`: None |
|
- `optim_target_modules`: None |
|
- `batch_eval_metrics`: False |
|
- `batch_sampler`: no_duplicates |
|
- `multi_dataset_batch_sampler`: proportional |
|
|
|
</details> |
|
|
|
### Training Logs |
|
| Epoch | Step | dim_128_cosine_map@100 | dim_256_cosine_map@100 | dim_384_cosine_map@100 | dim_64_cosine_map@100 | |
|
|:----------:|:-----:|:----------------------:|:----------------------:|:----------------------:|:---------------------:| |
|
| 0.6667 | 1 | 0.4231 | 0.4458 | 0.4525 | 0.3907 | |
|
| 2.0 | 3 | 0.4526 | 0.4745 | 0.4875 | 0.4114 | |
|
| **2.6667** | **4** | **0.4606** | **0.479** | **0.4891** | **0.4138** | |
|
|
|
* The bold row denotes the saved checkpoint. |
|
|
|
### Framework Versions |
|
- Python: 3.10.14 |
|
- Sentence Transformers: 3.0.1 |
|
- Transformers: 4.41.2 |
|
- PyTorch: 2.3.1+cu121 |
|
- Accelerate: 0.31.0 |
|
- Datasets: 2.19.1 |
|
- Tokenizers: 0.19.1 |
|
|
|
## Citation |
|
|
|
### BibTeX |
|
|
|
#### Sentence Transformers |
|
```bibtex |
|
@inproceedings{reimers-2019-sentence-bert, |
|
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks", |
|
author = "Reimers, Nils and Gurevych, Iryna", |
|
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing", |
|
month = "11", |
|
year = "2019", |
|
publisher = "Association for Computational Linguistics", |
|
url = "https://arxiv.org/abs/1908.10084", |
|
} |
|
``` |
|
|
|
#### MatryoshkaLoss |
|
```bibtex |
|
@misc{kusupati2024matryoshka, |
|
title={Matryoshka Representation Learning}, |
|
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi}, |
|
year={2024}, |
|
eprint={2205.13147}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.LG} |
|
} |
|
``` |
|
|
|
#### MultipleNegativesRankingLoss |
|
```bibtex |
|
@misc{henderson2017efficient, |
|
title={Efficient Natural Language Response Suggestion for Smart Reply}, |
|
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil}, |
|
year={2017}, |
|
eprint={1705.00652}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL} |
|
} |
|
``` |
|
|
|
<!-- |
|
## Glossary |
|
|
|
*Clearly define terms in order to be accessible across audiences.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Authors |
|
|
|
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* |
|
--> |
|
|
|
<!-- |
|
## Model Card Contact |
|
|
|
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* |
|
--> |
