{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to be able to access the data on Hugging Face Hub, we must import the\n",
    "necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_dataset           # Loading datasets from Hugging Face Hub\n",
    "from pprint import pprint                   # Pretty print"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After importing the modules, we set a few variables that will be used throughout\n",
    "this demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to the dataset repository on the Hugging Face Hub\n",
    "path = \"molssiai-hub/pubchemqc-b3lyp\"\n",
    "\n",
    "# set the dataset configuration/subset name\n",
    "name = \"b3lyp_pm6\"\n",
    "\n",
    "# set the dataset split\n",
    "split = \"train\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we are going to work with the\n",
    "[PubChemQC-B3LYP/6-31G*//PM6\n",
    "Dataset](https://huggingface.co/datasets/molssiai-hub/pubchemqc-b3lyp)\n",
    "(PubChemQC-B3LYP for short) from the [PubChemQC dataset\n",
    "collection](https://huggingface.co/collections/molssiai-hub/pubchemqc-datasets-669e5482260861ba7cce3d1c).\n",
    "Let us load the dataset as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset\n",
    "hub_dataset = load_dataset(path=path,\n",
    "                           name=name,\n",
    "                           split=split,\n",
    "                           streaming=True,\n",
    "                           trust_remote_code=True)\n",
    "\n",
    "hub_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PubChemQC datasets are very large and downloading them on your local machine can\n",
    "be a heavy lift for your internet network and disk storage. Therefore, we set\n",
    "the `streaming` parameter to `True` to avoid downloading the dataset on disk and\n",
    "ensure streaming the data from the hub. In this mode, the `load_dataset`\n",
    "function returns an `IterableDataset` object that can be iterated over to access\n",
    "the data. The `trust_remote_code` argument is also set to `True` to allow the\n",
    "usage of a custom [load\n",
    "script](https://huggingface.co/datasets/molssiai-hub/pubchemqc-b3lyp/blob/main/pubchemqc-b3lyp.py)\n",
    "for the data.\n",
    "\n",
    "The PubChemQC-B3LYP dataset is made of several files called `shards` that enable\n",
    "multiprocessing and parallelization of the data loading process. Multiprocessing\n",
    "can speed up the loading process significantly if you are absolutely sure that\n",
    "you have enough CPU cores to handle the load and enough memory and storage space\n",
    "to download and store the data.\n",
    "\n",
    "You can choose the number of processes to use for loading the data by setting\n",
    "the `num_proc` parameter in the `load_dataset` function.\n",
    "\n",
    "```python\n",
    "\n",
    "    >>> dataset = load_dataset(path=path,\n",
    "                               split=split,\n",
    "                               streaming=True,\n",
    "                               trust_remote_code=True,\n",
    "                               num_proc=4)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we create our `Dataset` or `IterableDataset` instance, we can access its\n",
    "features or column names using the `features` or `column_names` attributes of\n",
    "the dataset object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the column names\n",
    "hub_dataset.column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `take(n)` to fetch the first $n$ examples from the dataset.\n",
    "For demonstration purposes, we set $n$ to 2 which yields a list of\n",
    "two dictionaries, each containing the features of the corresponding\n",
    "data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(hub_dataset.take(2))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hugface",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}