diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -31,1008 +31,1060 @@ tags: - loss:MatryoshkaLoss - loss:MultipleNegativesRankingLoss widget: -- source_sentence: Can you explain how to override the extract_metadata() method to - track custom metadata for artifacts in ZenML? +- source_sentence: What are some examples of long-lived credentials used in ZenML + for authentication? sentences: - - 'tional) Which Metadata to Extract for the ArtifactOptionally, you can override - the extract_metadata() method to track custom metadata for all artifacts saved - by your materializer. Anything you extract here will be displayed in the dashboard - next to your artifacts. + - ' gs://zenml-core_cloudbuild ┃┃ │ gs://zenml-datasets ┃ - src.zenml.metadata.metadata_types that are displayed in a dedicated way in the - dashboard. See + ┃ │ gs://zenml-internal-artifact-store ┃ - src.zenml.metadata.metadata_types.MetadataType for more details. + ┃ │ gs://zenml-kubeflow-artifact-store ┃ - By default, this method will only extract the storage size of an artifact, but - you can overwrite it to track anything you wish. E.g., the zenml.materializers.NumpyMaterializer - overwrites this method to track the shape, dtype, and some statistical properties - of each np.ndarray that it saves. + ┃ │ gs://zenml-project-time-series-bucket ┃ - If you would like to disable artifact metadata extraction altogether, you can - set enable_artifact_metadata at either pipeline or step level via @pipeline(enable_artifact_metadata=False) - or @step(enable_artifact_metadata=False). + ┠───────────────────────┼─────────────────────────────────────────────────┨ - Skipping materialization + ┃ 🌀 kubernetes-cluster │ zenml-test-cluster ┃ - Skipping materialization might have unintended consequences for downstream tasks - that rely on materialized artifacts. Only skip materialization if there is no - other way to do what you want to do. + ┠───────────────────────┼─────────────────────────────────────────────────┨ - While materializers should in most cases be used to control how artifacts are - returned and consumed from pipeline steps, you might sometimes need to have a - completely unmaterialized artifact in a step, e.g., if you need to know the exact - path to where your artifact is stored. + ┃ 🐳 docker-registry │ gcr.io/zenml-core ┃ - An unmaterialized artifact is a zenml.materializers.UnmaterializedArtifact. Among - others, it has a property uri that points to the unique path in the artifact store - where the artifact is persisted. One can use an unmaterialized artifact by specifying - UnmaterializedArtifact as the type in the step: + ┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ - from zenml.artifacts.unmaterialized_artifact import UnmaterializedArtifact + Long-lived credentials (API keys, account keys) - from zenml import step + This is the magic formula of authentication methods. When paired with another + ability, such as automatically generating short-lived API tokens, or impersonating + accounts or assuming roles, this is the ideal authentication mechanism to use, + particularly when using ZenML in production and when sharing results with other + members of your ZenML team. - @step + As a general best practice, but implemented particularly well for cloud platforms, + account passwords are never directly used as a credential when authenticating + to the cloud platform APIs. There is always a process in place that exchanges + the account/password credential for another type of long-lived credential: + + + AWS uses the aws configure CLI command + + + GCP offers the gcloud auth application-default login CLI commands + + + Azure provides the az login CLI command + + + None of your original login information is stored on your local machine or used + to access workloads. Instead, an API key, account key or some other form of intermediate + credential is generated and stored on the local host and used to authenticate + to remote cloud service APIs.' + - '─────────────────────────────────────────────────┨┃ VERSION │ + 1 ┃ + + ┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┨ - def my_step(my_artifact: UnmaterializedArtifact): # rather than pd.DataFrame + ┃ VERSION_DESCRIPTION │ Run #1 of the mlflow_training_pipeline. ┃ - pass + ┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┨ - Example' - - ' ┃┃ │ │ │ │ - HTTP response headers: HTTPHeaderDict({''Audit-Id'': ''72558f83-e050-4fe3-93e5-9f7e66988a4c'', - ''Cache-Control'': ┃ + ┃ CREATED_AT │ 2023-03-01 09:09:06.899000 ┃ - ┃ │ │ │ │ - ''no-cache, private'', ''Content-Type'': ''application/json'', ''Date'': ''Fri, - 09 Jun 2023 18:59:02 GMT'', ┃ + ┠────────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┨ - ┃ │ │ │ │ - ''Content-Length'': ''129''}) ┃ + ┃ UPDATED_AT │ 2023-03-01 09:09:06.899000 ┃' + - 'un.py - ┃ │ │ │ │ - HTTP response body: ┃ + Read more in the production guide. - ┃ │ │ │ │ - {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauth - ┃ + CleanupMake sure you no longer need the resources before deleting them. The instructions + and commands that follow are DESTRUCTIVE. - ┃ │ │ │ │ - orized","code":401} ┃ + Delete any AWS resources you no longer use to avoid additional charges. You''ll + want to do the following: - ┃ │ │ │ │ ┃ + # delete the S3 bucket - ┃ │ │ │ │ ┃ + aws s3 rm s3://your-bucket-name --recursive - ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛' - - 's/30267569827/locations/global/workloadIdentityP ┃┃ │ ools/mypool/providers/myprovider", ┃ + aws s3api delete-bucket --bucket your-bucket-name - ┃ │ "subject_token_type": "urn:ietf:params:aws:token-type:aws4_request", ┃ + # delete the SageMaker domain - ┃ │ "service_account_impersonation_url": ┃ + aws sagemaker delete-domain --domain-id - ┃ │ "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/myrole@ ┃ + # delete the ECR repository - ┃ │ zenml-core.iam.gserviceaccount.com:generateAccessToken", ┃ + aws ecr delete-repository --repository-name zenml-repository --force - ┃ │ "token_url": "https://sts.googleapis.com/v1/token", ┃ + # detach policies from the IAM role - ┃ │ "credential_source": { ┃ + aws iam detach-role-policy --role-name zenml-role --policy-arn arn:aws:iam::aws:policy/AmazonS3FullAccess - ┃ │ "environment_id": "aws1", ┃ + aws iam detach-role-policy --role-name zenml-role --policy-arn arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryFullAccess - ┃ │ "region_url": ┃ + aws iam detach-role-policy --role-name zenml-role --policy-arn arn:aws:iam::aws:policy/AmazonSageMakerFullAccess - ┃ │ "http://169.254.169.254/latest/meta-data/placement/availability-zone", ┃ + # delete the IAM role - ┃ │ "url": ┃ + aws iam delete-role --role-name zenml-role - ┃ │ "http://169.254.169.254/latest/meta-data/iam/security-credentials", ┃ + Make sure to run these commands in the same AWS region where you created the resources. - ┃ │ "regional_cred_verification_url": ┃ + By running these cleanup commands, you will delete the S3 bucket, SageMaker domain, + ECR repository, and IAM role, along with their associated policies. This will + help you avoid any unnecessary charges for resources you no longer need. - ┃ │ "https://sts.{region}.amazonaws.com?Action=GetCallerIdentity&Version=2011-06- - ┃ + Remember to be cautious when deleting resources and ensure that you no longer + require them before running the deletion commands. - ┃ │ 15" ┃ + Conclusion - ┃ │ } ┃ + In this guide, we walked through the process of setting up an AWS stack with ZenML + to run your machine learning pipelines in a scalable and production-ready environment. + The key steps included: - ┃ │ } ┃ + Setting up credentials and the local environment by creating an IAM role with + the necessary permissions. - ┗━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛ + Creating a ZenML service connector to authenticate with AWS services using the + IAM role. - GCP OAuth 2.0 token' -- source_sentence: How does the `sync_new_data_to_label_studio` step ensure that new - annotations remain in sync with the ZenML artifact store? + + Configuring stack components, including an S3 artifact store, a SageMaker Pipelines + orchestrator, and an ECR container registry.' +- source_sentence: Can you provide more information on how to fetch the last successful + run of a pipeline in ZenML? sentences: - - 'nf, + - 'Connecting remote storage - step_config: "StepConfiguration", + Transitioning to remote artifact storage. - ) -> None:"""Configures Spark to handle backends like YARN, Mesos or Kubernetes.""" + In the previous chapters, we''ve been working with artifacts stored locally on + our machines. This setup is fine for individual experiments, but as we move towards + a collaborative and production-ready environment, we need a solution that is more + robust, shareable, and scalable. Enter remote storage! - def _io_configuration( + Remote storage allows us to store our artifacts in the cloud, which means they''re + accessible from anywhere and by anyone with the right permissions. This is essential + for team collaboration and for managing the larger datasets and models that come + with production workloads. - self, + When using a stack with remote storage, nothing changes except the fact that the + artifacts get materialized in a central and remote storage location. This diagram + explains the flow: - spark_config: SparkConf + Provisioning and registering a remote artifact store - ) -> None: + Out of the box, ZenML ships with many different supported artifact store flavors. + For convenience, here are some brief instructions on how to quickly get up and + running on the major cloud providers: - """Configures Spark to handle different input/output sources.""" + You will need to install and set up the AWS CLI on your machine as a prerequisite, + as covered in the AWS CLI documentation, before you register the S3 Artifact Store. - def _additional_configuration( + The Amazon Web Services S3 Artifact Store flavor is provided by the S3 ZenML integration, + you need to install it on your local machine to be able to register an S3 Artifact + Store and add it to your stack: - self, + zenml integration install s3 -y - spark_config: SparkConf + Having trouble with this command? You can use poetry or pip to install the requirements + of any ZenML integration directly. In order to obtain the exact requirements of + the AWS S3 integration you can use zenml integration requirements s3. - ) -> None: + The only configuration parameter mandatory for registering an S3 Artifact Store + is the root path URI, which needs to point to an S3 bucket and take the form s3://bucket-name. + In order to create a S3 bucket, refer to the AWS documentation.' + - 'Load a Model in code - """Appends the user-defined configuration parameters.""" + There are a few different ways to load a ZenML Model in code: - def _launch_spark_job( + Load the active model in a pipeline - self, + You can also use the active model to get the model metadata, or the associated + artifacts directly as described in the starter guide: - spark_config: SparkConf, + from zenml import step, pipeline, get_step_context, pipeline, Model - entrypoint_command: List[str] + @pipeline(model=Model(name="my_model")) - ) -> None: + def my_pipeline(): - """Generates and executes a spark-submit command.""" + ... - def launch( + @step - self, + def my_step(): - info: "StepRunInfo", + # Get model from active step context - entrypoint_command: List[str], + mv = get_step_context().model - ) -> None: + # Get metadata - """Launches the step on Spark.""" + print(mv.run_metadata["metadata_key"].value) - Under the base configuration, you will see the main configuration parameters: + # Directly fetch an artifact that is attached to the model - master is the master URL for the cluster where Spark will run. You might see different - schemes for this URL with varying cluster managers such as Mesos, YARN, or Kubernetes. + output = mv.get_artifact("my_dataset", "my_version") - deploy_mode can either be ''cluster'' (default) or ''client'' and it decides where - the driver node of the application will run. + output.run_metadata["accuracy"].value - submit_args is the JSON string of a dictionary, which will be used to define additional - parameters if required ( Spark has a wide variety of parameters, thus including - them all in a single class was deemed unnecessary.). + Load any model via the Client - In addition to this configuration, the launch method of the step operator gets - additional configuration parameters from the DockerSettings and ResourceSettings. - As a result, the overall configuration happens in 4 base methods: + Alternatively, you can use the Client: - _resource_configuration translates the ZenML ResourceSettings object to Spark''s - own resource configuration. + from zenml import step - _backend_configuration is responsible for cluster-manager-specific configuration. + from zenml.client import Client - _io_configuration is a critical method. Even though we have materializers, Spark - might require additional packages and configuration to work with a specific filesystem. - This method is used as an interface to provide this configuration. + from zenml.enums import ModelStages - _additional_configuration takes the submit_args, converts, and appends them to - the overall configuration.' - - 'Azure Blob Storage + @step - Storing artifacts using Azure Blob Storage + def model_evaluator_step() - The Azure Artifact Store is an Artifact Store flavor provided with the Azure ZenML - integration that uses the Azure Blob Storage managed object storage service to - store ZenML artifacts in an Azure Blob Storage container. + ... - When would you want to use it? + # Get staging model version - Running ZenML pipelines with the local Artifact Store is usually sufficient if - you just want to evaluate ZenML or get started quickly without incurring the trouble - and the cost of employing cloud storage services in your stack. However, the local - Artifact Store becomes insufficient or unsuitable if you have more elaborate needs - for your project: + try: - if you want to share your pipeline run results with other team members or stakeholders - inside or outside your organization + staging_zenml_model = Client().get_model_version( - if you have other components in your stack that are running remotely (e.g. a Kubeflow - or Kubernetes Orchestrator running in a public cloud). + model_name_or_id="", - if you outgrow what your local machine can offer in terms of storage space and - need to use some form of private or public storage service that is shared with - others + model_version_name_or_number_or_id=ModelStages.STAGING, - if you are running pipelines at scale and need an Artifact Store that can handle - the demands of production-grade MLOps + except KeyError: - In all these cases, you need an Artifact Store that is backed by a form of public - cloud or self-hosted shared object storage service. + staging_zenml_model = None - You should use the Azure Artifact Store when you decide to keep your ZenML artifacts - in a shared object storage and if you have access to the Azure Blob Storage managed - service. You should consider one of the other Artifact Store flavors if you don''t - have access to the Azure Blob Storage service. + ... - How do you deploy it? + PreviousControlling Model versions - The Azure Artifact Store flavor is provided by the Azure ZenML integration, you - need to install it on your local machine to be able to register an Azure Artifact - Store and add it to your stack: + NextPromote a Model + + + Last updated 19 days ago' + - ' more information. + + + Get the last run of a pipelineTo access the most recent run of a pipeline, you + can either use the last_run property or access it through the runs list: + + + last_run = pipeline_model.last_run # OR: pipeline_model.runs[0] + + + If your most recent runs have failed, and you want to find the last run that has + succeeded, you can use the last_successful_run property instead. + + + Get the latest run from a pipeline + + + Calling a pipeline executes it and then returns the response of the freshly executed + run. + + + run = training_pipeline() + + + The run that you get back is the model stored in the ZenML database at the point + of the method call. This means the pipeline run is still initializing and no steps + have been run. To get the latest state can get a refreshed version from the client: + + + from zenml.client import Client + + + Client().get_pipeline_run(run.id) to get a refreshed version + + + Get a run via the client + + + If you already know the exact run that you want to fetch (e.g., from looking at + the dashboard), you can use the Client.get_pipeline_run() method to fetch the + run directly without having to query the pipeline first: + + + from zenml.client import Client + + pipeline_run = Client().get_pipeline_run("first_pipeline-2023_06_20-16_20_13_274466") - zenml integration install azure -y' - - 'th the appropriate label config with Label Studio.get_labeled_data step - This - step will get all labeled data available for a particular dataset. Note that these - are output in a Label Studio annotation format, which will subsequently be converted - into a format appropriate for your specific use case. + Similar to pipelines, you can query runs by either ID, name, or name prefix, and + you can also discover runs through the Client or CLI via the Client.list_pipeline_runs() + or zenml pipeline runs list commands. - sync_new_data_to_label_studio step - This step is for ensuring that ZenML is handling - the annotations and that the files being used are stored and synced with the ZenML - artifact store. This is an important step as part of a continuous annotation workflow - since you want all the subsequent steps of your workflow to remain in sync with - whatever new annotations are being made or have been created. + Run information - Helper Functions + Each run has a collection of useful information which can help you reproduce your + runs. In the following, you can find a list of some of the most useful pipeline + run information, but there is much more available. See the PipelineRunResponse + definition for a comprehensive list. - Label Studio requires the use of what it calls ''label config'' when you are creating/registering - your dataset. These are strings containing HTML-like syntax that allow you to - define a custom interface for your annotation. ZenML provides three helper functions - that will construct these label config strings in the case of object detection, - image classification, and OCR. See the integrations.label_studio.label_config_generators - module for those three functions. + Status - PreviousArgilla + The status of a pipeline run. There are five possible states: initialized, failed, + completed, running, and cached. - NextPigeon + status = run.status - Last updated 15 days ago' -- source_sentence: How can I register an orchestrator in our active stack using a - Service Connector? + + Configuration' +- source_sentence: How does the code in the given documentation visualize the embeddings + using t-SNE and UMAP? sentences: - - 'in our active stack. This can be done in two ways:If you have a Service Connector - configured to access the remote Kubernetes cluster, you no longer need to set - the kubernetes_context attribute to a local kubectl context. In fact, you don''t - need the local Kubernetes CLI at all. You can connect the stack component to the - Service Connector instead:Copy$ zenml orchestrator register - --flavor tekton + - ' - Running with active workspace: ''default'' (repository) - Running with active stack: ''default'' (repository) + embeddings_2d[mask, 0], - Successfully registered orchestrator ``. + embeddings_2d[mask, 1],c=[section_color_dict[section]], - $ zenml service-connector list-resources --resource-type kubernetes-cluster -e - The following ''kubernetes-cluster'' resources can be accessed by service connectors - configured in your workspace: + label=section, - ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━━━┯━━━━━━━━━━━━━━━━━━━━━┓ - ┃ CONNECTOR ID │ CONNECTOR NAME │ CONNECTOR TYPE - │ RESOURCE TYPE │ RESOURCE NAMES ┃ + plt.title("t-SNE Visualization") - ┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨ - ┃ e33c9fac-5daa-48b2-87bb-0187d3782cde │ aws-iam-multi-eu │ 🔶 aws │ - 🌀 kubernetes-cluster │ kubeflowmultitenant ┃ + plt.legend() - ┃ │ │ │ │ - zenbox ┃ - ┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨ + plt.show() - ┃ ed528d5a-d6cb-4fc4-bc52-c3d2d01643e5 │ aws-iam-multi-us │ 🔶 aws │ - 🌀 kubernetes-cluster │ zenhacks-cluster ┃ - ┠──────────────────────────────────────┼───────────────────────┼────────────────┼───────────────────────┼─────────────────────┨ + # Dimensionality reduction using UMAP - ┃ 1c54b32a-4889-4417-abbd-42d3ace3d03a │ gcp-sa-multi │ 🔵 gcp │ - 🌀 kubernetes-cluster │ zenml-test-cluster ┃ - ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━┛' - - 'e by passing the page argument to the list method.You can further restrict your - search by passing additional arguments that will be used to filter the results. - E.g., most resources have a user_id associated with them that can be set to only - list resources created by that specific user. The available filter argument options - are different for each list method; check out the method declaration in the Client - SDK documentation to find out which exact arguments are supported or have a look - at the fields of the corresponding filter model class. + def umap_visualization(embeddings, parent_sections): - Except for pipeline runs, all other resources will by default be ordered by creation - time ascending. E.g., client.list_artifacts() would return the first 50 artifacts - ever created. You can change the ordering by specifying the sort_by argument when - calling list methods. + umap_2d = umap.UMAP(n_components=2, random_state=42) - Get Methods + embeddings_2d = umap_2d.fit_transform(embeddings) - Fetch a specific instance of a resource by either resource ID, name, or name prefix, - e.g.: + plt.figure(figsize=(8, 8)) - client.get_pipeline_run("413cfb42-a52c-4bf1-a2fd-78af2f7f0101") # ID + for section in unique_parent_sections: - client.get_pipeline_run("first_pipeline-2023_06_20-16_20_13_274466") # Name + if section in section_color_dict: - client.get_pipeline_run("first_pipeline-2023_06_20-16") # Name prefix + mask = [section == ps for ps in parent_sections] - Create, Update, and Delete Methods + plt.scatter( - Methods for creating / updating / deleting resources are only available for some - of the resources and the required arguments are different for each resource. Checkout - the Client SDK Documentation to find out whether a specific resource supports - write operations through the Client and which arguments are required. + embeddings_2d[mask, 0], - Active User and Active Stack + embeddings_2d[mask, 1], - For some use cases you might need to know information about the user that you - are authenticated as or the stack that you have currently set as active. You can - fetch this information via the client.active_user and client.active_stack_model - properties respectively, e.g.: + c=[section_color_dict[section]], - my_runs_on_current_stack = client.list_pipeline_runs( + label=section, - stack_id=client.active_stack_model.id, # on current stack + plt.title("UMAP Visualization") - user_id=client.active_user.id, # ran by you + plt.legend() - Resource Models' - - ' build to finish. More information: Build Timeout.We can register the image builder - and use it in our active stack: + plt.show() - zenml image-builder register \ + In this stage, we have utilized the ''parent directory'', which we had previously + stored in the vector store as an additional attribute, as a means to color the + values. This approach allows us to gain some insight into the semantic space inherent + in our data. It demonstrates that you can visualize the embeddings and observe + how similar chunks are grouped together based on their semantic meaning and context. - --flavor=gcp \ + So this step iterates through all the chunks and generates embeddings representing + each piece of text. These embeddings are then stored as an artifact in the ZenML + artifact store as a NumPy array. We separate this generation from the point where + we upload those embeddings to the vector database to keep the pipeline modular + and flexible; in the future we might want to use a different vector database so + we can just swap out the upload step without having to re-generate the embeddings. - --cloud_builder_image= \ + In the next section, we''ll explore how to store these embeddings in a vector + database to enable fast and efficient retrieval of relevant chunks at inference + time. - --network= \ + Code Example - --build_timeout= + To explore the full code, visit the Complete Guide repository. The embeddings + generation step can be found here. - # Register and activate a stack with the new image builder + PreviousData ingestion and preprocessing - zenml stack register -i ... --set + NextStoring embeddings in a vector database - You also need to set up authentication required to access the Cloud Build GCP - services. + Last updated 2 months ago' + - 'Amazon SageMaker - Authentication Methods + Executing individual steps in SageMaker. - Integrating and using a GCP Image Builder in your pipelines is not possible without - employing some form of authentication. If you''re looking for a quick way to get - started locally, you can use the Local Authentication method. However, the recommended - way to authenticate to the GCP cloud platform is through a GCP Service Connector. - This is particularly useful if you are configuring ZenML stacks that combine the - GCP Image Builder with other remote stack components also running in GCP. + SageMaker offers specialized compute instances to run your training jobs and has + a comprehensive UI to track and manage your models and logs. ZenML''s SageMaker + step operator allows you to submit individual steps to be run on Sagemaker compute + instances. - This method uses the implicit GCP authentication available in the environment - where the ZenML code is running. On your local machine, this is the quickest way - to configure a GCP Image Builder. You don''t need to supply credentials explicitly - when you register the GCP Image Builder, as it leverages the local credentials - and configuration that the Google Cloud CLI stores on your local machine. However, - you will need to install and set up the Google Cloud CLI on your machine as a - prerequisite, as covered in the Google Cloud documentation , before you register - the GCP Image Builder. + When to use it - Stacks using the GCP Image Builder set up with local authentication are not portable - across environments. To make ZenML pipelines fully portable, it is recommended - to use a GCP Service Connector to authenticate your GCP Image Builder to the GCP - cloud platform.' -- source_sentence: How can ZenML load and display visualizations from the artifact - store? - sentences: - - 'Connect with a Service Account + You should use the SageMaker step operator if: - Sometimes you may need to authenticate to a ZenML server from a non-interactive - environment where the web login is not possible, like a CI/CD workload or a serverless - function. In these cases, you can configure a service account and an API key and - use the API key to authenticate to the ZenML server: + one or more steps of your pipeline require computing resources (CPU, GPU, memory) + that are not provided by your orchestrator. - zenml service-account create + you have access to SageMaker. If you''re using a different cloud provider, take + a look at the Vertex or AzureML step operators. - This command creates a service account and an API key for it. The API key is displayed - as part of the command output and cannot be retrieved later. You can then use - the issued API key to connect your ZenML client to the server through one of the - following methods: + How to deploy it - using the CLI: + Create a role in the IAM console that you want the jobs running in SageMaker to + assume. This role should at least have the AmazonS3FullAccess and AmazonSageMakerFullAccess + policies applied. Check here for a guide on how to set up this role. - zenml connect --url https://... --api-key + Infrastructure Deployment - setting the ZENML_STORE_URL and ZENML_STORE_API_KEY environment variables when - you set up your ZenML client for the first time. This method is particularly useful - when you are using the ZenML client in an automated CI/CD workload environment - like GitHub Actions or GitLab CI or in a containerized environment like Docker - or Kubernetes: + A Sagemaker step operator can be deployed directly from the ZenML CLI: - export ZENML_STORE_URL=https://... + zenml orchestrator deploy sagemaker_step_operator --flavor=sagemaker --provider=aws + ... - export ZENML_STORE_API_KEY= + You can pass other configurations specific to the stack components as key-value + arguments. If you don''t provide a name, a random one is generated for you. For + more information about how to work use the CLI for this, please refer to the dedicated + documentation section. - To see all the service accounts you''ve created and their API keys, use the following - commands: + How to use it - zenml service-account list + To use the SageMaker step operator, we need: - zenml service-account api-key list + The ZenML aws integration installed. If you haven''t done so, runCopyzenml integration + install aws - Additionally, the following command allows you to more precisely inspect one of - these service accounts and an API key: + Docker installed and running. - zenml service-account describe + An IAM role with the correct permissions. See the deployment section for detailed + instructions. - zenml service-account api-key describe + An AWS container registry as part of our stack. Take a look here for a guide on + how to set that up.' + - 'Deploy with Docker - API keys don''t have an expiration date. For increased security, we recommend - that you regularly rotate the API keys to prevent unauthorized access to your - ZenML server. You can do this with the ZenML CLI: + Deploying ZenML in a Docker container. - zenml service-account api-key rotate ' - - 'd_data_loader_step() + The ZenML server container image is available at zenmldocker/zenml-server and + can be used to deploy ZenML with a container management or orchestration tool + like Docker and docker-compose, or a serverless platform like Cloud Run, Container + Apps, and more! This guide walks you through the various configuration options + that the ZenML server container expects as well as a few deployment use cases. - # Load the model to finetune# If no version is specified, the latest version of - "my_model" is used + Try it out locally first - model = client.get_artifact_version( + If you''re just looking for a quick way to deploy the ZenML server using a container, + without going through the hassle of interacting with a container management tool + like Docker and manually configuring your container, you can use the ZenML CLI + to do so. You only need to have Docker installed and running on your machine: - name_id_or_prefix="my_model", version=model_version + zenml up --docker - # Finetune the model + This command deploys a ZenML server locally in a Docker container, then connects + your client to it. Similar to running plain zenml up, the server and the local + ZenML client share the same SQLite database. - # This automatically creates a new version of "my_model" + The rest of this guide is addressed to advanced users who are looking to manually + deploy and manage a containerized ZenML server. - model_finetuner_step(model=model, dataset=dataset) + ZenML server configuration options - def main(): + If you''re planning on deploying a custom containerized ZenML server yourself, + you probably need to configure some settings for it like the database it should + use, the default user details, and more. The ZenML server container image uses + sensible defaults, so you can simply start a container without worrying too much + about the configuration. However, if you''re looking to connect the ZenML server + to an external MySQL database or secrets management service, to persist the internal + SQLite database, or simply want to control other settings like the default account, + you can do so by customizing the container''s environment variables. - # Save an untrained model as first version of "my_model" + The following environment variables can be passed to the container:' +- source_sentence: What is the purpose of the `enable_step_logs` parameter in ZenML? + sentences: + - 'Collect information from your SQL database serviceUsing an external MySQL-compatible + database service is optional, but is recommended for production deployments. If + omitted, ZenML will default to using an embedded SQLite database, which has the + following limitations: - untrained_model = SVC(gamma=0.001) + the SQLite database is not persisted, meaning that it will be lost if the ZenML + server pod is restarted or deleted - save_artifact( + the SQLite database does not scale horizontally, meaning that you will not be + able to use more than one replica at a time for the ZenML server pod - untrained_model, name="my_model", version="1", tags=["SVC", "untrained"] + If you decide to use an external MySQL-compatible database service, you will need + to collect and prepare the following information for the Helm chart configuration: - # Create a first version of "my_dataset" and train the model on it + the hostname and port where the SQL database is reachable from the Kubernetes + cluster - model_finetuning_pipeline() + the username and password that will be used to connect to the database. It is + recommended that you create a dedicated database user for the ZenML server and + that you restrict its privileges to only access the database that will be used + by ZenML. Enforcing secure SSL connections for the user/database is also recommended. + See the MySQL documentation for more information on how to set up users and privileges. - # Finetune the latest model on an older version of the dataset + the name of the database that will be used by ZenML. The database does not have + to exist prior to the deployment ( ZenML will create it on the first start). However, + you need to create the database if you follow the best practice of restricting + database user privileges to only access it. - model_finetuning_pipeline(dataset_version="1") + if you plan on using SSL to secure the client database connection, you may also + need to prepare additional SSL certificates and keys:the TLS CA certificate that + was used to sign the server TLS certificate, if you''re using a self-signed certificate + or signed by a custom certificate authority that is not already trusted by default + by most operating systems.the TLS client certificate and key. This is only needed + if you decide to use client certificates for your DB connection (some managed + DB services support this, CloudSQL is an example).' + - ' │ SHARED │ OWNER │ EXPIRES IN │ LABELS ┃┠────────┼────────────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼─────────────────────────┼────────┼─────────┼────────────┼────────┨ - # Run inference with the latest model on an older version of the dataset + ┃ │ gcp-multi │ 9d953320-3560-4a78-817c-926a3898064d │ 🔵 gcp + │ 🔵 gcp-generic │ │ ➖ │ default │ │ ┃ - latest_trained_model = load_artifact("my_model") + ┃ │ │ │ │ + 📦 gcs-bucket │ │ │ │ │ ┃ - old_dataset = load_artifact("my_dataset", version="1") + ┃ │ │ │ │ + 🌀 kubernetes-cluster │ │ │ │ │ ┃ - latest_trained_model.predict(old_dataset[0]) + ┃ │ │ │ │ + 🐳 docker-registry │ │ │ │ │ ┃ - if __name__ == "__main__": + ┠────────┼────────────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼─────────────────────────┼────────┼─────────┼────────────┼────────┨ - main() + ┃ │ gcs-multi │ ff9c0723-7451-46b7-93ef-fcf3efde30fa │ 🔵 gcp + │ 📦 gcs-bucket │ │ ➖ │ default │ │ ┃ - This would create the following pipeline run DAGs: + ┠────────┼────────────────────────┼──────────────────────────────────────┼────────┼───────────────────────┼─────────────────────────┼────────┼─────────┼────────────┼────────┨ - Run 1: + ┃ │ gcs-langchain-slackbot │ cf3953e9-414c-4875-ba00-24c62a0dc0c5 │ 🔵 gcp + │ 📦 gcs-bucket │ gs://langchain-slackbot │ ➖ │ default │ │ ┃ - Run 2: + ┗━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━━━━━━━━━━━━━━━━━━┷━━━━━━━━┷━━━━━━━━━┷━━━━━━━━━━━━┷━━━━━━━━┛ - PreviousCache previous executions + Local and remote availability' + - 'onfiguration, if specified overrides for this stepenable_artifact_metadata: True - NextTrack ML models + enable_artifact_visualization: True - Last updated 19 days ago' - - 'into play when the component is ultimately in use.The design behind this interaction - lets us separate the configuration of the flavor from its implementation. This - way we can register flavors and components even when the major dependencies behind - their implementation are not installed in our local setting (assuming the CustomArtifactStoreFlavor - and the CustomArtifactStoreConfig are implemented in a different module/path than - the actual CustomArtifactStore). + enable_cache: False - Enabling Artifact Visualizations with Custom Artifact Stores + enable_step_logs: True - ZenML automatically saves visualizations for many common data types and allows - you to view these visualizations in the ZenML dashboard. Under the hood, this - works by saving the visualizations together with the artifacts in the artifact - store. + # Same as pipeline level configuration, if specified overrides for this step - In order to load and display these visualizations, ZenML needs to be able to load - and access the corresponding artifact store. This means that your custom artifact - store needs to be configured in a way that allows authenticating to the back-end - without relying on the local environment, e.g., by embedding the authentication - credentials in the stack component configuration or by referencing a secret. + extra: {} - Furthermore, for deployed ZenML instances, you need to install the package dependencies - of your artifact store implementation in the environment where you have deployed - ZenML. See the Documentation on deploying ZenML with custom Docker images for - more information on how to do that. + # Same as pipeline level configuration, if specified overrides for this step - PreviousAzure Blob Storage + model: {} - NextContainer Registries + # Same as pipeline level configuration, if specified overrides for this step - Last updated 19 days ago' -- source_sentence: How do you configure ED25519 key based authentication for the HyperAI - orchestrator in ZenML? - sentences: - - 'authentication. + settings: - ED25519 key based authentication.SSH private keys configured in the connector - will be distributed to all clients that use them to run pipelines with the HyperAI - orchestrator. SSH keys are long-lived credentials that give unrestricted access - to HyperAI instances. + docker: {} - When configuring the Service Connector, it is required to provide at least one - hostname via hostnames and the username with which to login. Optionally, it is - possible to provide an ssh_passphrase if applicable. This way, it is possible - to use the HyperAI service connector in multiple ways: + resources: {} - Create one service connector per HyperAI instance with different SSH keys. + # Stack component specific settings - Configure a reused SSH key just once for multiple HyperAI instances, then select - the individual instance when creating the HyperAI orchestrator component. + step_operator.sagemaker: - Auto-configuration + estimator_args: - This Service Connector does not support auto-discovery and extraction of authentication - credentials from HyperAI instances. If this feature is useful to you or your organization, - please let us know by messaging us in Slack or creating an issue on GitHub. + instance_type: m7g.medium - Stack Components use + Deep-dive - The HyperAI Service Connector can be used by the HyperAI Orchestrator to deploy - pipeline runs to HyperAI instances. + enable_XXX parameters - PreviousAzure Service Connector + These are boolean flags for various configurations: - NextManage stacks + enable_artifact_metadata: Whether to associate metadata with artifacts or not. - Last updated 15 days ago' - - 'nswer is satisfactory, or ''NO'' if it is not.", + enable_artifact_visualization: Whether to attach visualizations of artifacts. - },"role": "user", + enable_cache: Utilize caching or not. - "content": f"Question: {question}\nExpected Answer: {expected_answer}\nGenerated - Answer: {generated_answer}\nIs the generated answer relevant and accurate?", + enable_step_logs: Enable tracking step logs. - }, + enable_artifact_metadata: True - ], + enable_artifact_visualization: True - model="gpt-3.5-turbo", + enable_cache: True - judgment = chat_completion.choices[0].message.content.strip().lower() + enable_step_logs: True - return judgment == "yes" + build ID - retrieval_scores = [] + The UUID of the build to use for this pipeline. If specified, Docker image building + is skipped for remote orchestrators, and the Docker image specified in this build + is used. - generation_scores = [] + build: - for item in eval_data: + Configuring the model - retrieval_score = evaluate_retrieval( + Specifies the ZenML Model to use for this pipeline. - item["question"], item["expected_answer"], corpus + model: - retrieval_scores.append(retrieval_score) + name: "ModelName" - generated_answer = answer_question(item["question"], corpus) + version: "production" - generation_score = evaluate_generation( + description: An example model - item["question"], item["expected_answer"], generated_answer + tags: ["classifier"] - generation_scores.append(generation_score) + Pipeline and step parameters - retrieval_accuracy = sum(retrieval_scores) / len(retrieval_scores) + A dictionary of JSON-serializable parameters specified at the pipeline or step + level. For example: - generation_accuracy = sum(generation_scores) / len(generation_scores) + parameters: - print(f"Retrieval Accuracy: {retrieval_accuracy:.2f}") + gamma: 0.01 - print(f"Generation Accuracy: {generation_accuracy:.2f}") + steps: - As you can see, we''ve added two evaluation functions: evaluate_retrieval and - evaluate_generation. The evaluate_retrieval function checks if the retrieved chunks - contain any words from the expected answer. The evaluate_generation function uses - OpenAI''s chat completion LLM to evaluate the quality of the generated answer. + trainer: - We then loop through the evaluation data, which contains questions and expected - answers, and evaluate the retrieval and generation components of our RAG pipeline. - Finally, we calculate the accuracy of both components and print the results: + parameters: - As you can see, we get 100% accuracy for both retrieval and generation in this - example. Not bad! The sections that follow will provide a more detailed and sophisticated - implementation of RAG evaluation, but this example shows how you can think about - it at a high level! + gamma: 0.001 - PreviousEvaluation and metrics + Corresponds to: - NextRetrieval evaluation + from zenml import step, pipeline - Last updated 19 days ago' - - ' have a look at the SDK docs . + @step - How do you use it?To log information from a ZenML pipeline step using the Neptune - Experiment Tracker component in the active stack, you need to enable an experiment - tracker using the @step decorator. Then fetch the Neptune run object and use logging - capabilities as you would normally do. For example: + def trainer(gamma: float): - import numpy as np + # Use gamma as normal - import tensorflow as tf + print(gamma) - from neptune_tensorflow_keras import NeptuneCallback + @pipeline - from zenml.integrations.neptune.experiment_trackers.run_state import ( + def my_pipeline(gamma: float): - get_neptune_run, + # use gamma or pass it into the step - from zenml import step + print(0.01) - @step(experiment_tracker="") + trainer(gamma=gamma)' +- source_sentence: Can I use ZenML to register an S3 Artifact Store after setting + up the IAM user and generating the access key? + sentences: + - ' to install and configure the AWS CLI on your hostyou don''t need to care about + enabling your other stack components (orchestrators, step operators, and model + deployers) to have access to the artifact store through IAM roles and policies - def tf_trainer( + you can combine the S3 artifact store with other stack components that are not + running in AWS - x_train: np.ndarray, + Note: When you create the IAM user for your AWS access key, please remember to + grant the created IAM user permissions to read and write to your S3 bucket (i.e. + at a minimum: s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject) - y_train: np.ndarray, + After having set up the IAM user and generated the access key, as described in + the AWS documentation, you can register the S3 Artifact Store as follows: - x_val: np.ndarray, + # Store the AWS access key in a ZenML secret - y_val: np.ndarray, + zenml secret create s3_secret \ - epochs: int = 5, + --aws_access_key_id='''' \ - lr: float = 0.001 + --aws_secret_access_key='''' - ) -> tf.keras.Model: + # Register the S3 artifact-store and reference the ZenML secret - ... + zenml artifact-store register s3_store -f s3 \ - neptune_run = get_neptune_run() + --path=''s3://your-bucket'' \ - model.fit( + --authentication_secret=s3_secret - x_train, + # Register and set a stack with the new artifact store - y_train, + zenml stack register custom_stack -a s3_store ... --set - epochs=epochs, + Advanced Configuration - validation_data=(x_val, y_val), + The S3 Artifact Store accepts a range of advanced configuration options that can + be used to further customize how ZenML connects to the S3 storage service that + you are using. These are accessible via the client_kwargs, config_kwargs and s3_additional_kwargs + configuration attributes and are passed transparently to the underlying S3Fs library: - callbacks=[ + client_kwargs: arguments that will be transparently passed to the botocore client + . You can use it to configure parameters like endpoint_url and region_name when + connecting to an S3-compatible endpoint (e.g. Minio). - NeptuneCallback(run=neptune_run), + config_kwargs: advanced parameters passed to botocore.client.Config. - ], + s3_additional_kwargs: advanced parameters that are used when calling S3 API, typically + used for things like ServerSideEncryption and ACL.' + - 'Evidently - metric = ... + How to keep your data quality in check and guard against data and model drift + with Evidently profiling - neptune_run[""] = metric + The Evidently Data Validator flavor provided with the ZenML integration uses Evidently + to perform data quality, data drift, model drift and model performance analyses, + to generate reports and run checks. The reports and check results can be used + to implement automated corrective actions in your pipelines or to render interactive + representations for further visual interpretation, evaluation and documentation. - Instead of hardcoding an experiment tracker name, you can also use the Client - to dynamically use the experiment tracker of your active stack: + When would you want to use it? - from zenml.client import Client + Evidently is an open-source library that you can use to monitor and debug machine + learning models by analyzing the data that they use through a powerful set of + data profiling and visualization features, or to run a variety of data and model + validation reports and tests, from data integrity tests that work with a single + dataset to model evaluation tests to data drift analysis and model performance + comparison tests. All this can be done with minimal configuration input from the + user, or customized with specialized conditions that the validation tests should + perform. - experiment_tracker = Client().active_stack.experiment_tracker + Evidently currently works with tabular data in pandas.DataFrame or CSV file formats + and can handle both regression and classification tasks. - @step(experiment_tracker=experiment_tracker.name) + You should use the Evidently Data Validator when you need the following data and/or + model validation features that are possible with Evidently: - def tf_trainer(...): + Data Quality reports and tests: provides detailed feature statistics and a feature + behavior overview for a single dataset. It can also compare any two datasets. + E.g. you can use it to compare train and test data, reference and current data, + or two subgroups of one dataset. - ... + Data Drift reports and tests: helps detects and explore feature distribution changes + in the input data by comparing two datasets with identical schema.' + - ' us know! - Additional configuration + Configuration at pipeline or step levelWhen running your ZenML pipeline with the + Sagemaker orchestrator, the configuration set when configuring the orchestrator + as a ZenML component will be used by default. However, it is possible to provide + additional configuration at the pipeline or step level. This allows you to run + whole pipelines or individual steps with alternative configurations. For example, + this allows you to run the training process with a heavier, GPU-enabled instance + type, while running other steps with lighter instances. - You can pass a set of tags to the Neptune run by using the NeptuneExperimentTrackerSettings - class, like in the example below: + Additional configuration for the Sagemaker orchestrator can be passed via SagemakerOrchestratorSettings. + Here, it is possible to configure processor_args, which is a dictionary of arguments + for the Processor. For available arguments, see the Sagemaker documentation . + Currently, it is not possible to provide custom configuration for the following + attributes: - import numpy as np + image_uri - import tensorflow as tf + instance_count - from zenml import step + sagemaker_session - from zenml.integrations.neptune.experiment_trackers.run_state import ( + entrypoint - get_neptune_run, + base_job_name - from zenml.integrations.neptune.flavors import NeptuneExperimentTrackerSettings + env - neptune_settings = NeptuneExperimentTrackerSettings(tags={"keras", "mnist"}) + For example, settings can be provided in the following way: - @step( + sagemaker_orchestrator_settings = SagemakerOrchestratorSettings( - experiment_tracker="", + processor_args={ - settings={ + "instance_type": "ml.t3.medium", - "experiment_tracker.neptune": neptune_settings + "volume_size_in_gb": 30 - def my_step( + They can then be applied to a step as follows: - x_test: np.ndarray, + @step(settings={"orchestrator.sagemaker": sagemaker_orchestrator_settings}) - y_test: np.ndarray, + For example, if your ZenML component is configured to use ml.c5.xlarge with 400GB + additional storage by default, all steps will use it except for the step above, + which will use ml.t3.medium with 30GB additional storage. - model: tf.keras.Model, + Check out this docs page for more information on how to specify settings in general. - ) -> float: + For more information and a full list of configurable attributes of the Sagemaker + orchestrator, check out the SDK Docs . - """Log metadata to Neptune run"""' + S3 data access in ZenML steps' model-index: - name: zenml/finetuned-all-MiniLM-L6-v2 results: @@ -1047,46 +1099,46 @@ model-index: value: 0.3313253012048193 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.5963855421686747 + value: 0.6024096385542169 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.6867469879518072 + value: 0.6927710843373494 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.8072289156626506 + value: 0.7710843373493976 name: Cosine Accuracy@10 - type: cosine_precision@1 value: 0.3313253012048193 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.1987951807228915 + value: 0.2008032128514056 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.13734939759036144 + value: 0.13855421686746985 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.08072289156626504 + value: 0.07710843373493974 name: Cosine Precision@10 - type: cosine_recall@1 value: 0.3313253012048193 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.5963855421686747 + value: 0.6024096385542169 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.6867469879518072 + value: 0.6927710843373494 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.8072289156626506 + value: 0.7710843373493976 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.56403366334234 + value: 0.551850042031417 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.48697408682348453 + value: 0.4812392426850258 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.4930812678607045 + value: 0.48905601510986996 name: Cosine Map@100 - task: type: information-retrieval @@ -1096,49 +1148,49 @@ model-index: type: dim_256 metrics: - type: cosine_accuracy@1 - value: 0.3373493975903614 + value: 0.3253012048192771 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.5963855421686747 + value: 0.5903614457831325 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.6746987951807228 + value: 0.6867469879518072 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.7771084337349398 + value: 0.7710843373493976 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.3373493975903614 + value: 0.3253012048192771 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.1987951807228915 + value: 0.19678714859437746 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.13493975903614455 + value: 0.13734939759036144 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.07771084337349397 + value: 0.07710843373493974 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.3373493975903614 + value: 0.3253012048192771 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.5963855421686747 + value: 0.5903614457831325 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.6746987951807228 + value: 0.6867469879518072 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.7771084337349398 + value: 0.7710843373493976 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.5551681743429259 + value: 0.5441628856415894 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.4842656339644293 + value: 0.4714883342895392 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.4928559286742408 + value: 0.4790082728748276 name: Cosine Map@100 - task: type: information-retrieval @@ -1148,49 +1200,49 @@ model-index: type: dim_128 metrics: - type: cosine_accuracy@1 - value: 0.3132530120481928 + value: 0.3313253012048193 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.5060240963855421 + value: 0.5120481927710844 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.5843373493975904 + value: 0.6144578313253012 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.7289156626506024 + value: 0.6987951807228916 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.3132530120481928 + value: 0.3313253012048193 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.1686746987951807 + value: 0.1706827309236948 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.11686746987951806 + value: 0.12289156626506023 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.07289156626506023 + value: 0.06987951807228915 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.3132530120481928 + value: 0.3313253012048193 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.5060240963855421 + value: 0.5120481927710844 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.5843373493975904 + value: 0.6144578313253012 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.7289156626506024 + value: 0.6987951807228916 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.5072091899807667 + value: 0.5108893388836802 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.43864744693057955 + value: 0.45126936316695354 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.44825117444575807 + value: 0.4605530012141939 name: Cosine Map@100 - task: type: information-retrieval @@ -1200,49 +1252,49 @@ model-index: type: dim_64 metrics: - type: cosine_accuracy@1 - value: 0.2469879518072289 + value: 0.2891566265060241 name: Cosine Accuracy@1 - type: cosine_accuracy@3 - value: 0.4819277108433735 + value: 0.4759036144578313 name: Cosine Accuracy@3 - type: cosine_accuracy@5 - value: 0.536144578313253 + value: 0.5542168674698795 name: Cosine Accuracy@5 - type: cosine_accuracy@10 - value: 0.6506024096385542 + value: 0.6265060240963856 name: Cosine Accuracy@10 - type: cosine_precision@1 - value: 0.2469879518072289 + value: 0.2891566265060241 name: Cosine Precision@1 - type: cosine_precision@3 - value: 0.1606425702811245 + value: 0.15863453815261044 name: Cosine Precision@3 - type: cosine_precision@5 - value: 0.10722891566265058 + value: 0.1108433734939759 name: Cosine Precision@5 - type: cosine_precision@10 - value: 0.0650602409638554 + value: 0.06265060240963854 name: Cosine Precision@10 - type: cosine_recall@1 - value: 0.2469879518072289 + value: 0.2891566265060241 name: Cosine Recall@1 - type: cosine_recall@3 - value: 0.4819277108433735 + value: 0.4759036144578313 name: Cosine Recall@3 - type: cosine_recall@5 - value: 0.536144578313253 + value: 0.5542168674698795 name: Cosine Recall@5 - type: cosine_recall@10 - value: 0.6506024096385542 + value: 0.6265060240963856 name: Cosine Recall@10 - type: cosine_ndcg@10 - value: 0.4454935545722587 + value: 0.45650145038804574 name: Cosine Ndcg@10 - type: cosine_mrr@10 - value: 0.38048384012239445 + value: 0.40227337923121054 name: Cosine Mrr@10 - type: cosine_map@100 - value: 0.39358543662682755 + value: 0.4137670603435629 name: Cosine Map@100 --- @@ -1296,9 +1348,9 @@ from sentence_transformers import SentenceTransformer model = SentenceTransformer("zenml/finetuned-all-MiniLM-L6-v2") # Run inference sentences = [ - 'How do you configure ED25519 key based authentication for the HyperAI orchestrator in ZenML?', - 'authentication.\n\nED25519 key based authentication.SSH private keys configured in the connector will be distributed to all clients that use them to run pipelines with the HyperAI orchestrator. SSH keys are long-lived credentials that give unrestricted access to HyperAI instances.\n\nWhen configuring the Service Connector, it is required to provide at least one hostname via hostnames and the username with which to login. Optionally, it is possible to provide an ssh_passphrase if applicable. This way, it is possible to use the HyperAI service connector in multiple ways:\n\nCreate one service connector per HyperAI instance with different SSH keys.\n\nConfigure a reused SSH key just once for multiple HyperAI instances, then select the individual instance when creating the HyperAI orchestrator component.\n\nAuto-configuration\n\nThis Service Connector does not support auto-discovery and extraction of authentication credentials from HyperAI instances. If this feature is useful to you or your organization, please let us know by messaging us in Slack or creating an issue on GitHub.\n\nStack Components use\n\nThe HyperAI Service Connector can be used by the HyperAI Orchestrator to deploy pipeline runs to HyperAI instances.\n\nPreviousAzure Service Connector\n\nNextManage stacks\n\nLast updated 15 days ago', - ' have a look at the SDK docs .\n\nHow do you use it?To log information from a ZenML pipeline step using the Neptune Experiment Tracker component in the active stack, you need to enable an experiment tracker using the @step decorator. Then fetch the Neptune run object and use logging capabilities as you would normally do. For example:\n\nimport numpy as np\n\nimport tensorflow as tf\n\nfrom neptune_tensorflow_keras import NeptuneCallback\n\nfrom zenml.integrations.neptune.experiment_trackers.run_state import (\n\nget_neptune_run,\n\nfrom zenml import step\n\n@step(experiment_tracker="")\n\ndef tf_trainer(\n\nx_train: np.ndarray,\n\ny_train: np.ndarray,\n\nx_val: np.ndarray,\n\ny_val: np.ndarray,\n\nepochs: int = 5,\n\nlr: float = 0.001\n\n) -> tf.keras.Model:\n\n...\n\nneptune_run = get_neptune_run()\n\nmodel.fit(\n\nx_train,\n\ny_train,\n\nepochs=epochs,\n\nvalidation_data=(x_val, y_val),\n\ncallbacks=[\n\nNeptuneCallback(run=neptune_run),\n\n],\n\nmetric = ...\n\nneptune_run[""] = metric\n\nInstead of hardcoding an experiment tracker name, you can also use the Client to dynamically use the experiment tracker of your active stack:\n\nfrom zenml.client import Client\n\nexperiment_tracker = Client().active_stack.experiment_tracker\n\n@step(experiment_tracker=experiment_tracker.name)\n\ndef tf_trainer(...):\n\n...\n\nAdditional configuration\n\nYou can pass a set of tags to the Neptune run by using the NeptuneExperimentTrackerSettings class, like in the example below:\n\nimport numpy as np\n\nimport tensorflow as tf\n\nfrom zenml import step\n\nfrom zenml.integrations.neptune.experiment_trackers.run_state import (\n\nget_neptune_run,\n\nfrom zenml.integrations.neptune.flavors import NeptuneExperimentTrackerSettings\n\nneptune_settings = NeptuneExperimentTrackerSettings(tags={"keras", "mnist"})\n\n@step(\n\nexperiment_tracker="",\n\nsettings={\n\n"experiment_tracker.neptune": neptune_settings\n\ndef my_step(\n\nx_test: np.ndarray,\n\ny_test: np.ndarray,\n\nmodel: tf.keras.Model,\n\n) -> float:\n\n"""Log metadata to Neptune run"""', + 'Can I use ZenML to register an S3 Artifact Store after setting up the IAM user and generating the access key?', + " to install and configure the AWS CLI on your hostyou don't need to care about enabling your other stack components (orchestrators, step operators, and model deployers) to have access to the artifact store through IAM roles and policies\n\nyou can combine the S3 artifact store with other stack components that are not running in AWS\n\nNote: When you create the IAM user for your AWS access key, please remember to grant the created IAM user permissions to read and write to your S3 bucket (i.e. at a minimum: s3:PutObject, s3:GetObject, s3:ListBucket, s3:DeleteObject)\n\nAfter having set up the IAM user and generated the access key, as described in the AWS documentation, you can register the S3 Artifact Store as follows:\n\n# Store the AWS access key in a ZenML secret\n\nzenml secret create s3_secret \\\n\n--aws_access_key_id='' \\\n\n--aws_secret_access_key=''\n\n# Register the S3 artifact-store and reference the ZenML secret\n\nzenml artifact-store register s3_store -f s3 \\\n\n--path='s3://your-bucket' \\\n\n--authentication_secret=s3_secret\n\n# Register and set a stack with the new artifact store\n\nzenml stack register custom_stack -a s3_store ... --set\n\nAdvanced Configuration\n\nThe S3 Artifact Store accepts a range of advanced configuration options that can be used to further customize how ZenML connects to the S3 storage service that you are using. These are accessible via the client_kwargs, config_kwargs and s3_additional_kwargs configuration attributes and are passed transparently to the underlying S3Fs library:\n\nclient_kwargs: arguments that will be transparently passed to the botocore client . You can use it to configure parameters like endpoint_url and region_name when connecting to an S3-compatible endpoint (e.g. Minio).\n\nconfig_kwargs: advanced parameters passed to botocore.client.Config.\n\ns3_additional_kwargs: advanced parameters that are used when calling S3 API, typically used for things like ServerSideEncryption and ACL.", + ' us know!\n\nConfiguration at pipeline or step levelWhen running your ZenML pipeline with the Sagemaker orchestrator, the configuration set when configuring the orchestrator as a ZenML component will be used by default. However, it is possible to provide additional configuration at the pipeline or step level. This allows you to run whole pipelines or individual steps with alternative configurations. For example, this allows you to run the training process with a heavier, GPU-enabled instance type, while running other steps with lighter instances.\n\nAdditional configuration for the Sagemaker orchestrator can be passed via SagemakerOrchestratorSettings. Here, it is possible to configure processor_args, which is a dictionary of arguments for the Processor. For available arguments, see the Sagemaker documentation . Currently, it is not possible to provide custom configuration for the following attributes:\n\nimage_uri\n\ninstance_count\n\nsagemaker_session\n\nentrypoint\n\nbase_job_name\n\nenv\n\nFor example, settings can be provided in the following way:\n\nsagemaker_orchestrator_settings = SagemakerOrchestratorSettings(\n\nprocessor_args={\n\n"instance_type": "ml.t3.medium",\n\n"volume_size_in_gb": 30\n\nThey can then be applied to a step as follows:\n\n@step(settings={"orchestrator.sagemaker": sagemaker_orchestrator_settings})\n\nFor example, if your ZenML component is configured to use ml.c5.xlarge with 400GB additional storage by default, all steps will use it except for the step above, which will use ml.t3.medium with 30GB additional storage.\n\nCheck out this docs page for more information on how to specify settings in general.\n\nFor more information and a full list of configurable attributes of the Sagemaker orchestrator, check out the SDK Docs .\n\nS3 data access in ZenML steps', ] embeddings = model.encode(sentences) print(embeddings.shape) @@ -1345,42 +1397,42 @@ You can finetune this model on your own dataset. | Metric | Value | |:--------------------|:-----------| | cosine_accuracy@1 | 0.3313 | -| cosine_accuracy@3 | 0.5964 | -| cosine_accuracy@5 | 0.6867 | -| cosine_accuracy@10 | 0.8072 | +| cosine_accuracy@3 | 0.6024 | +| cosine_accuracy@5 | 0.6928 | +| cosine_accuracy@10 | 0.7711 | | cosine_precision@1 | 0.3313 | -| cosine_precision@3 | 0.1988 | -| cosine_precision@5 | 0.1373 | -| cosine_precision@10 | 0.0807 | +| cosine_precision@3 | 0.2008 | +| cosine_precision@5 | 0.1386 | +| cosine_precision@10 | 0.0771 | | cosine_recall@1 | 0.3313 | -| cosine_recall@3 | 0.5964 | -| cosine_recall@5 | 0.6867 | -| cosine_recall@10 | 0.8072 | -| cosine_ndcg@10 | 0.564 | -| cosine_mrr@10 | 0.487 | -| **cosine_map@100** | **0.4931** | +| cosine_recall@3 | 0.6024 | +| cosine_recall@5 | 0.6928 | +| cosine_recall@10 | 0.7711 | +| cosine_ndcg@10 | 0.5519 | +| cosine_mrr@10 | 0.4812 | +| **cosine_map@100** | **0.4891** | #### Information Retrieval * Dataset: `dim_256` * Evaluated with [InformationRetrievalEvaluator](https://sbert.net/docs/package_reference/sentence_transformer/evaluation.html#sentence_transformers.evaluation.InformationRetrievalEvaluator) -| Metric | Value | -|:--------------------|:-----------| -| cosine_accuracy@1 | 0.3373 | -| cosine_accuracy@3 | 0.5964 | -| cosine_accuracy@5 | 0.6747 | -| cosine_accuracy@10 | 0.7771 | -| cosine_precision@1 | 0.3373 | -| cosine_precision@3 | 0.1988 | -| cosine_precision@5 | 0.1349 | -| cosine_precision@10 | 0.0777 | -| cosine_recall@1 | 0.3373 | -| cosine_recall@3 | 0.5964 | -| cosine_recall@5 | 0.6747 | -| cosine_recall@10 | 0.7771 | -| cosine_ndcg@10 | 0.5552 | -| cosine_mrr@10 | 0.4843 | -| **cosine_map@100** | **0.4929** | +| Metric | Value | +|:--------------------|:----------| +| cosine_accuracy@1 | 0.3253 | +| cosine_accuracy@3 | 0.5904 | +| cosine_accuracy@5 | 0.6867 | +| cosine_accuracy@10 | 0.7711 | +| cosine_precision@1 | 0.3253 | +| cosine_precision@3 | 0.1968 | +| cosine_precision@5 | 0.1373 | +| cosine_precision@10 | 0.0771 | +| cosine_recall@1 | 0.3253 | +| cosine_recall@3 | 0.5904 | +| cosine_recall@5 | 0.6867 | +| cosine_recall@10 | 0.7711 | +| cosine_ndcg@10 | 0.5442 | +| cosine_mrr@10 | 0.4715 | +| **cosine_map@100** | **0.479** | #### Information Retrieval * Dataset: `dim_128` @@ -1388,21 +1440,21 @@ You can finetune this model on your own dataset. | Metric | Value | |:--------------------|:-----------| -| cosine_accuracy@1 | 0.3133 | -| cosine_accuracy@3 | 0.506 | -| cosine_accuracy@5 | 0.5843 | -| cosine_accuracy@10 | 0.7289 | -| cosine_precision@1 | 0.3133 | -| cosine_precision@3 | 0.1687 | -| cosine_precision@5 | 0.1169 | -| cosine_precision@10 | 0.0729 | -| cosine_recall@1 | 0.3133 | -| cosine_recall@3 | 0.506 | -| cosine_recall@5 | 0.5843 | -| cosine_recall@10 | 0.7289 | -| cosine_ndcg@10 | 0.5072 | -| cosine_mrr@10 | 0.4386 | -| **cosine_map@100** | **0.4483** | +| cosine_accuracy@1 | 0.3313 | +| cosine_accuracy@3 | 0.512 | +| cosine_accuracy@5 | 0.6145 | +| cosine_accuracy@10 | 0.6988 | +| cosine_precision@1 | 0.3313 | +| cosine_precision@3 | 0.1707 | +| cosine_precision@5 | 0.1229 | +| cosine_precision@10 | 0.0699 | +| cosine_recall@1 | 0.3313 | +| cosine_recall@3 | 0.512 | +| cosine_recall@5 | 0.6145 | +| cosine_recall@10 | 0.6988 | +| cosine_ndcg@10 | 0.5109 | +| cosine_mrr@10 | 0.4513 | +| **cosine_map@100** | **0.4606** | #### Information Retrieval * Dataset: `dim_64` @@ -1410,21 +1462,21 @@ You can finetune this model on your own dataset. | Metric | Value | |:--------------------|:-----------| -| cosine_accuracy@1 | 0.247 | -| cosine_accuracy@3 | 0.4819 | -| cosine_accuracy@5 | 0.5361 | -| cosine_accuracy@10 | 0.6506 | -| cosine_precision@1 | 0.247 | -| cosine_precision@3 | 0.1606 | -| cosine_precision@5 | 0.1072 | -| cosine_precision@10 | 0.0651 | -| cosine_recall@1 | 0.247 | -| cosine_recall@3 | 0.4819 | -| cosine_recall@5 | 0.5361 | -| cosine_recall@10 | 0.6506 | -| cosine_ndcg@10 | 0.4455 | -| cosine_mrr@10 | 0.3805 | -| **cosine_map@100** | **0.3936** | +| cosine_accuracy@1 | 0.2892 | +| cosine_accuracy@3 | 0.4759 | +| cosine_accuracy@5 | 0.5542 | +| cosine_accuracy@10 | 0.6265 | +| cosine_precision@1 | 0.2892 | +| cosine_precision@3 | 0.1586 | +| cosine_precision@5 | 0.1108 | +| cosine_precision@10 | 0.0627 | +| cosine_recall@1 | 0.2892 | +| cosine_recall@3 | 0.4759 | +| cosine_recall@5 | 0.5542 | +| cosine_recall@10 | 0.6265 | +| cosine_ndcg@10 | 0.4565 | +| cosine_mrr@10 | 0.4023 | +| **cosine_map@100** | **0.4138** |