{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "pdcMxVGEA9Cd" }, "source": [ "# **Fine-tuning for Image Classification with 🤗 Optimum Graphcore**\n", "\n", "This notebook shows how to fine-tune any pretrained Vision model for Image Classification on an IPU. The idea is to add a randomly initialized classification head on top of a pre-trained encoder, and fine-tune the model altogether on a labeled dataset.\n", "\n", "## ImageFolder\n", "\n", "This notebook leverages the [ImageFolder](https://huggingface.co/docs/datasets/v2.0.0/en/image_process#imagefolder) feature to easily run the notebook on a custom dataset (namely, [EuroSAT](https://github.com/phelber/EuroSAT) in this tutorial). You can either load a `Dataset` from local folders or from local/remote files, like zip or tar.\n", "\n", "## Any model\n", "\n", "This notebook is built to run on any image classification dataset with any vision model checkpoint from the [Model Hub](https://huggingface.co/) as long as that model has a version with a Image Classification head and is supported by [🤗 Optimum Graphcore](https://github.com/huggingface/optimum-graphcore). The IPU config files of the supported models are available in Graphcore's [Hugging Face account](https://huggingface.co/Graphcore). You can also create your own IPU config file locally. \n", "Currently supported models:\n", "* [ViT](https://huggingface.co/docs/transformers/model_doc/vit#transformers.ViTForImageClassification)\n", "* [ConvNeXT](https://huggingface.co/docs/transformers/master/en/model_doc/convnext#transformers.ConvNextForImageClassification)\n", "\n", "---\n", "\n", "In this notebook, we are using both data parallelism and pipeline parallelism (see this [tutorial](https://github.com/graphcore/examples/tree/master/tutorials/tutorials/pytorch/efficient_data_loading) for more). Therefore the global batch size, which is the actual number of samples used for the weight update, is determined with three factors:\n", "- global batch size = micro batch size * gradient accumulation steps * replication factor\n", "\n", "and replication factor is determined by `pod_type`, which will be used as a key to select the replication factor from a dictionary defined in the IPU config file. For example, the dictionary in the IPU config file [Graphcore/roberta-base-ipu](https://huggingface.co/Graphcore/roberta-base-ipu/blob/main/ipu_config.json) looks like this:\n", "- \"replication_factor\": {\"pod4\": 1, \"pod8\": 2, \"pod16\": 4, \"pod32\": 8, \"pod64\": 16, \"default\": 1}\n", "\n", "Depending on you model and the pod machine you are using, you might need to adjust these three batch-size-related arguments.\n", "\n", "In this notebook, we'll fine-tune from the [google/vit-base-patch16-224-in21k](https://huggingface.co/google/vit-base-patch16-224-in21k) checkpoint, but note that there are many more available on the [hub](https://huggingface.co/models?other=vision).\n" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "execution": { "iopub.execute_input": "2023-06-21T21:42:56.324166Z", "iopub.status.busy": "2023-06-21T21:42:56.323923Z", "iopub.status.idle": "2023-06-21T21:42:56.327242Z", "shell.execute_reply": "2023-06-21T21:42:56.326624Z", "shell.execute_reply.started": "2023-06-21T21:42:56.324147Z" }, "id": "5WMEawzyCEyG" }, "outputs": [], "source": [ "model_checkpoint = \"facebook/dino-vitb16\" # pre-trained model from which to fine-tune\n", "\n", "ipu_config_name = \"Graphcore/vit-base-ipu\" # config specific to the IPU\n", "micro_batch_size = 1 # micro batch size for training and evaluation\n", "gradient_accumulation_steps = 32" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Values for machine size and cache directories can be configured through environment variables or directly in the notebook:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2023-06-21T21:43:04.407323Z", "iopub.status.busy": "2023-06-21T21:43:04.407084Z", "iopub.status.idle": "2023-06-21T21:43:04.411027Z", "shell.execute_reply": "2023-06-21T21:43:04.410205Z", "shell.execute_reply.started": "2023-06-21T21:43:04.407303Z" } }, "outputs": [], "source": [ "import os\n", "\n", "pod_type = os.getenv(\"GRAPHCORE_POD_TYPE\", \"pod4\")\n", "executable_cache_dir = os.getenv(\"POPLAR_EXECUTABLE_CACHE_DIR\", \"/tmp/exe_cache/\") + \"/image_classification\"\n", "dataset_dir = os.getenv(\"DATASETS_DIR\", \"./\")" ] }, { "cell_type": "markdown", "metadata": { "id": "NlArTG8KChJf" }, "source": [ "First of all, make sure your environment has installed the latest version of [🤗 Optimum Graphcore](https://github.com/huggingface/optimum-graphcore) and any packages we will need." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "In order to improve usability and support for future users, Graphcore would like to collect information about the\n", "applications and code being run in this notebook. The following information will be anonymised before being sent to Graphcore:\n", "\n", "- User progression through the notebook\n", "- Notebook details: number of cells, code being run and the output of the cells\n", "- Environment details\n", "\n", "You can disable logging at any time by running `%unload_ext gc_logger` from any cell.\n" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "execution": { "iopub.execute_input": "2023-06-21T21:43:08.202242Z", "iopub.status.busy": "2023-06-21T21:43:08.202000Z", "iopub.status.idle": "2023-06-21T21:44:37.490120Z", "shell.execute_reply": "2023-06-21T21:44:37.489336Z", "shell.execute_reply.started": "2023-06-21T21:43:08.202223Z" }, "id": "L1532RVbJgQV", "outputId": "1d92a15b-0efd-4b09-b006-56384c64943b" }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n", "Collecting examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable\n", " Cloning https://github.com/graphcore/examples-utils (to revision latest_stable) to /tmp/pip-install-a926yjkj/examples-utils_a82a79d52fff4e08a47c0e0e1ece1ac6\n", " Running command git clone --filter=blob:none --quiet https://github.com/graphcore/examples-utils /tmp/pip-install-a926yjkj/examples-utils_a82a79d52fff4e08a47c0e0e1ece1ac6\n", " Running command git checkout -q 40c62e6646db8f9d60d1707a61204c95a15c7ccb\n", " Resolved https://github.com/graphcore/examples-utils to commit 40c62e6646db8f9d60d1707a61204c95a15c7ccb\n", " Preparing metadata (setup.py) ... \u001b[?25ldone\n", "\u001b[?25hRequirement already satisfied: awscli>=1.24.10 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.27.143)\n", "Collecting boto3>=1.26 (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading boto3-1.26.158-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.9/135.9 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting cppimport>=22.07.17 (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading cppimport-22.8.2.tar.gz (26 kB)\n", " Installing build dependencies ... \u001b[?25ldone\n", "\u001b[?25h Getting requirements to build wheel ... \u001b[?25ldone\n", "\u001b[?25h Preparing metadata (pyproject.toml) ... \u001b[?25ldone\n", "\u001b[?25hRequirement already satisfied: filelock>=3.9.0 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.12.2)\n", "Collecting gitpython>=3.1 (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading GitPython-3.1.31-py3-none-any.whl (184 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m184.3/184.3 kB\u001b[0m \u001b[31m24.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting ipynbname>=2021.3.2 (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading ipynbname-2023.1.0.0-py3-none-any.whl (4.3 kB)\n", "Requirement already satisfied: nbformat>=5.7.3 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.9.0)\n", "Requirement already satisfied: psutil>=5.7.0 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.9.5)\n", "Requirement already satisfied: pyyaml>=5.4.1 in /usr/local/lib/python3.8/dist-packages (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.4.1)\n", "Collecting simple-parsing==0.0.19.post1 (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading simple_parsing-0.0.19.post1-py3-none-any.whl (79 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m79.9/79.9 kB\u001b[0m \u001b[31m25.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting typing-inspect (from simple-parsing==0.0.19.post1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n", "Requirement already satisfied: botocore==1.29.143 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.29.143)\n", "Requirement already satisfied: docutils<0.17,>=0.10 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.16)\n", "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.6.1)\n", "Requirement already satisfied: colorama<0.4.5,>=0.2.5 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.4.4)\n", "Requirement already satisfied: rsa<4.8,>=3.1.2 in /usr/local/lib/python3.8/dist-packages (from awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.7.2)\n", "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/local/lib/python3.8/dist-packages (from botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.0.1)\n", "Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/local/lib/python3.8/dist-packages (from botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.8.2)\n", "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3/dist-packages (from botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.25.8)\n", "INFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while.\n", "Collecting boto3>=1.26 (from examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading boto3-1.26.157-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.9/135.9 kB\u001b[0m \u001b[31m33.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.156-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m33.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.155-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.154-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m34.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.153-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m33.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.152-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.151-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m32.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hINFO: pip is looking at multiple versions of boto3 to determine which version is compatible with other requirements. This could take a while.\n", " Downloading boto3-1.26.150-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m35.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.149-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m34.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.148-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m32.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.147-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m33.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.146-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m35.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hINFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.\n", " Downloading boto3-1.26.145-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m36.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.144-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m35.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25h Downloading boto3-1.26.143-py3-none-any.whl (135 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m135.6/135.6 kB\u001b[0m \u001b[31m31.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting mako (from cppimport>=22.07.17->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading Mako-1.2.4-py3-none-any.whl (78 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m78.7/78.7 kB\u001b[0m \u001b[31m23.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting pybind11 (from cppimport>=22.07.17->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading pybind11-2.10.4-py3-none-any.whl (222 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m222.3/222.3 kB\u001b[0m \u001b[31m53.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hCollecting gitdb<5,>=4.0.1 (from gitpython>=3.1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading gitdb-4.0.10-py3-none-any.whl (62 kB)\n", "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m62.7/62.7 kB\u001b[0m \u001b[31m11.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", "\u001b[?25hRequirement already satisfied: ipykernel in /usr/local/lib/python3.8/dist-packages (from ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.5.6)\n", "Requirement already satisfied: fastjsonschema in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.17.1)\n", "Requirement already satisfied: jsonschema>=2.6 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.17.3)\n", "Requirement already satisfied: jupyter-core in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.3.0)\n", "Requirement already satisfied: traitlets>=5.1 in /usr/local/lib/python3.8/dist-packages (from nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.9.0)\n", "Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->gitpython>=3.1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading smmap-5.0.0-py3-none-any.whl (24 kB)\n", "Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (23.1.0)\n", "Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.12.0)\n", "Requirement already satisfied: pkgutil-resolve-name>=1.3.10 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.3.10)\n", "Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.8/dist-packages (from jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.19.3)\n", "Requirement already satisfied: pyasn1>=0.1.3 in /usr/local/lib/python3.8/dist-packages (from rsa<4.8,>=3.1.2->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.5.0)\n", "Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.2.0)\n", "Requirement already satisfied: ipython>=5.0.0 in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (7.16.3)\n", "Requirement already satisfied: jupyter-client in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (8.2.0)\n", "Requirement already satisfied: tornado>=4.2 in /usr/local/lib/python3.8/dist-packages (from ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (6.3.2)\n", "Requirement already satisfied: platformdirs>=2.5 in /usr/local/lib/python3.8/dist-packages (from jupyter-core->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.5.1)\n", "Requirement already satisfied: MarkupSafe>=0.9.2 in /usr/local/lib/python3.8/dist-packages (from mako->cppimport>=22.07.17->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.1.2)\n", "Collecting mypy-extensions>=0.3.0 (from typing-inspect->simple-parsing==0.0.19.post1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable)\n", " Downloading mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n", "Requirement already satisfied: typing-extensions>=3.7.4 in /usr/local/lib/python3.8/dist-packages (from typing-inspect->simple-parsing==0.0.19.post1->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.6.2)\n", "Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.8/dist-packages (from importlib-resources>=1.4.0->jsonschema>=2.6->nbformat>=5.7.3->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.15.0)\n", "Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (67.8.0)\n", "Requirement already satisfied: jedi<=0.17.2,>=0.10 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.17.2)\n", "Requirement already satisfied: decorator in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (5.1.1)\n", "Requirement already satisfied: pickleshare in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.7.5)\n", "Requirement already satisfied: prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0 in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (3.0.38)\n", "Requirement already satisfied: pygments in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (2.15.1)\n", "Requirement already satisfied: backcall in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.2.0)\n", "Requirement already satisfied: pexpect in /usr/local/lib/python3.8/dist-packages (from ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (4.8.0)\n", "Requirement already satisfied: six>=1.5 in /usr/lib/python3/dist-packages (from python-dateutil<3.0.0,>=2.1->botocore==1.29.143->awscli>=1.24.10->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (1.14.0)\n", "Requirement already satisfied: importlib-metadata>=4.8.3 in /usr/local/lib/python3.8/dist-packages (from jupyter-client->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (6.6.0)\n", "Requirement already satisfied: pyzmq>=23.0 in /usr/local/lib/python3.8/dist-packages (from jupyter-client->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (25.1.0)\n", "Requirement already satisfied: parso<0.8.0,>=0.7.0 in /usr/local/lib/python3.8/dist-packages (from jedi<=0.17.2,>=0.10->ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.7.1)\n", "Requirement already satisfied: wcwidth in /usr/local/lib/python3.8/dist-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.2.6)\n", "Requirement already satisfied: ptyprocess>=0.5 in /usr/local/lib/python3.8/dist-packages (from pexpect->ipython>=5.0.0->ipykernel->ipynbname>=2021.3.2->examples-utils[common]@ git+https://github.com/graphcore/examples-utils@latest_stable) (0.7.0)\n", "Building wheels for collected packages: cppimport, examples-utils\n", " Building wheel for cppimport (pyproject.toml) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for cppimport: filename=cppimport-22.8.2-py3-none-any.whl size=17499 sha256=9a686bdcf202f0cd615f5344f1ec24a1bbd07c7bc90fe92ef042eaeca162492f\n", " Stored in directory: /root/.cache/pip/wheels/6e/d2/91/94b56e3d502b51c6fb096384d6257632cde02f9a9df23f20c1\n", " Building wheel for examples-utils (setup.py) ... \u001b[?25ldone\n", "\u001b[?25h Created wheel for examples-utils: filename=examples_utils-0.1.0-py3-none-any.whl size=75817 sha256=c87a47bb56f4cf9cb7c33ae4f520ebde76868b9a5165776a27a6b60b52c27959\n", " Stored in directory: /tmp/pip-ephem-wheel-cache-e77qgaub/wheels/7b/ef/40/036b06152d165a6d3c2ade41a7f53c22e31a76a515dba9b89b\n", "Successfully built cppimport examples-utils\n", "Installing collected packages: smmap, pybind11, mypy-extensions, mako, typing-inspect, gitdb, cppimport, simple-parsing, gitpython, ipynbname, boto3, examples-utils\n", "Successfully installed boto3-1.26.143 cppimport-22.8.2 examples-utils-0.1.0 gitdb-4.0.10 gitpython-3.1.31 ipynbname-2023.1.0.0 mako-1.2.4 mypy-extensions-1.0.0 pybind11-2.10.4 simple-parsing-0.0.19.post1 smmap-5.0.0 typing-inspect-0.9.0\n", "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n", "\u001b[0mNote: you may need to restart the kernel to use updated packages.\n", "Loading extensions from ~/.ipython/extensions is deprecated. We recommend managing extensions like any other Python packages, in site-packages.\n" ] } ], "source": [ "%pip install -q \"optimum-graphcore==0.6.1\" scikit-learn torchvision==0.14.1+cpu -f https://download.pytorch.org/whl/torch_stable.html\n", "%pip install examples-utils[common]@git+https://github.com/graphcore/examples-utils@latest_stable\n", "from examples_utils import notebook_logging\n", "%load_ext gc_logger" ] }, { "cell_type": "markdown", "metadata": { "id": "snZ1tmaOC412" }, "source": [ "If you're opening this notebook locally, make sure your environment has an install from the last version of those libraries.\n", "\n", "To be able to share your model with the community and generate results like the one shown in the picture below via the inference API, there are a few more steps to follow.\n", "\n", "First you have to store your authentication token from the Hugging Face website (sign up [here](https://huggingface.co/join) if you haven't already!) then execute the following cell and input your token:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 386, "referenced_widgets": [ "f1760b8ccf9b4c32977a1e83f3a3af3d", "e276e653c187474dba7e1c4fede10b79", "a6d024d44a6c49eebef7966ef5a836f1", "344c041a34b5458eb1bbb3ea8fa5b315", "18f7dc9f5e6241138534b7cf9aa30adb", "d5e3a1de2a4645639c029a404d04dc1c", "e7e938eb6baf486e829ea5d4734087cf", "730378e114f944908aa06f42bb2faa3d", "f73b5464140d4723b1b3f46796d9b1ca", "16ffe85c44764fa9ad8a31fb21e6432f", "40db3808e98d424cacc5d0fed54b9eaa", "e9bad1a707f0442da6f117fdd2804f72", "a0bac01e342b4793b66d7d4f5bfac2e2", "b447ca05136342d0a167bfb133a353bd", "4dcbcf9b086f452d9d1bc07b4a4cc1d3", "724b578bac3f4b7cb6806ee3c45aff01", "6ca86c47e547426e8a0d4487a786c46c" ] }, "execution": { "iopub.execute_input": "2023-06-21T21:44:37.492165Z", "iopub.status.busy": "2023-06-21T21:44:37.491744Z", "iopub.status.idle": "2023-06-21T21:44:37.717449Z", "shell.execute_reply": "2023-06-21T21:44:37.716929Z", "shell.execute_reply.started": "2023-06-21T21:44:37.492146Z" }, "id": "Bkpk_JPlCww8", "outputId": "d80cb8c7-5382-427b-e90b-bfec7afdc052" }, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "aa278fd06d734ba184bb23865fe22631", "version_major": 2, "version_minor": 0 }, "text/plain": [ "VBox(children=(HTML(value='