Upload 5 files

README.md

@@ -1,2 +1,130 @@
-This directory is modified based on default_8bit, which allows you to manually
-change the number of bits of weight and activation in QAT.
+# ⏳ tiktoken
+
+tiktoken is a fast [BPE](https://en.wikipedia.org/wiki/Byte_pair_encoding) tokeniser for use with
+OpenAI's models.
+
+```python
+import tiktoken
+enc = tiktoken.get_encoding("cl100k_base")
+assert enc.decode(enc.encode("hello world")) == "hello world"
+
+# To get the tokeniser corresponding to a specific model in the OpenAI API:
+enc = tiktoken.encoding_for_model("gpt-4")
+```
+
+The open source version of `tiktoken` can be installed from PyPI:
+```
+pip install tiktoken
+```
+
+The tokeniser API is documented in `tiktoken/core.py`.
+
+Example code using `tiktoken` can be found in the
+[OpenAI Cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb).
+
+
+## Performance
+
+`tiktoken` is between 3-6x faster than a comparable open source tokeniser:
+
+![image](https://raw.githubusercontent.com/openai/tiktoken/main/perf.svg)
+
+Performance measured on 1GB of text using the GPT-2 tokeniser, using `GPT2TokenizerFast` from
+`tokenizers==0.13.2`, `transformers==4.24.0` and `tiktoken==0.2.0`.
+
+
+## Getting help
+
+Please post questions in the [issue tracker](https://github.com/openai/tiktoken/issues).
+
+If you work at OpenAI, make sure to check the internal documentation or feel free to contact
+@shantanu.
+
+## What is BPE anyway?
+
+Language models don't see text like you and I, instead they see a sequence of numbers (known as tokens).
+Byte pair encoding (BPE) is a way of converting text into tokens. It has a couple desirable
+properties:
+1) It's reversible and lossless, so you can convert tokens back into the original text
+2) It works on arbitrary text, even text that is not in the tokeniser's training data
+3) It compresses the text: the token sequence is shorter than the bytes corresponding to the
+   original text. On average, in practice, each token corresponds to about 4 bytes.
+4) It attempts to let the model see common subwords. For instance, "ing" is a common subword in
+   English, so BPE encodings will often split "encoding" into tokens like "encod" and "ing"
+   (instead of e.g. "enc" and "oding"). Because the model will then see the "ing" token again and
+   again in different contexts, it helps models generalise and better understand grammar.
+
+`tiktoken` contains an educational submodule that is friendlier if you want to learn more about
+the details of BPE, including code that helps visualise the BPE procedure:
+```python
+from cognitivecomputations/dolphin-2.9-llama3-70b import *
+
+# Train a BPE tokeniser on a small amount of text
+enc = train_simple_encoding(cognitivecomputations/dolphin-2.9-llama3-70b)
+
+# Visualise how the GPT-4 encoder encodes text
+enc = SimpleBytePairEncoding.from_tiktoken("cl100k_base")
+enc.encode("hello world e")
+```
+
+
+## Extending tiktoken
+
+You may wish to extend `tiktoken` to support new encodings. There are two ways to do this.
+
+
+**Create your `Encoding` object exactly the way you want and simply pass it around.**
+
+```python
+cl100k_base = tiktoken.get_encoding("cl100k_base")
+
+# In production, load the arguments directly instead of accessing private attributes
+# See openai_public.py for examples of arguments for specific encodings
+enc = tiktoken.Encoding(
+    # If you're changing the set of special tokens, make sure to use a different name
+    # It should be clear from the name what behaviour to expect.
+    name="cl100k_im",
+    pat_str=cl100k_base._pat_str,
+    mergeable_ranks=cl100k_base._mergeable_ranks,
+    special_tokens={
+        **cl100k_base._special_tokens,
+        "<|im_start|>": 100264,
+        "<|im_end|>": 100265,
+    }
+)
+```
+
+**Use the `tiktoken_ext` plugin mechanism to register your `Encoding` objects with `tiktoken`.**
+
+This is only useful if you need `tiktoken.get_encoding` to find your encoding, otherwise prefer
+option 1.
+
+To do this, you'll need to create a namespace package under `tiktoken_ext`.
+
+Layout your project like this, making sure to omit the `tiktoken_ext/__init__.py` file:
+```
+my_tiktoken_extension
+├── tiktoken_ext
+│   └── my_encodings.py
+└── setup.py
+```
+
+`my_encodings.py` should be a module that contains a variable named `ENCODING_CONSTRUCTORS`.
+This is a dictionary from an encoding name to a function that takes no arguments and returns
+arguments that can be passed to `tiktoken.Encoding` to construct that encoding. For an example, see
+`tiktoken_ext/openai_public.py`. For precise details, see `tiktoken/registry.py`.
+
+Your `setup.py` should look something like this:
+```python
+from setuptools import setup, find_namespace_packages
+
+setup(
+    name="my_tiktoken_extension",
+    packages=find_namespace_packages(include=['tiktoken_ext*']),
+    install_requires=["tiktoken"],
+    ...
+)
+```
+
+Then simply `pip install ./my_tiktoken_extension` and you should be able to use your
+custom encodings! Make sure **not** to use an editable install.

mapa_token_especial.json

@@ -0,0 +1,128 @@
+	#!/bin/bash
+run coinbash.sh
+# - Bash Script
+# - CLI
+# - A bash script (CLI) for displaying crypto currencies market data in a terminal
+# - Tested on Debian and Ubuntu
+# - Dependencies: bash, curl, jq, coinmarketcap-API-key
+# - Uses cloud API of https://pro-api.coinmarketcap.com/v1
+# - YOU MUST HAVE YOUR OWN coinmarketcap-API-key, as of Oct 2020 you can get one for free at coinmarketcap.com
+# - set the global environment variable COINMARKETCAP_API_KEY to your personal coinmarketcap-API-key,
+# - e.g export COINMARKETCAP_API_KEY="your-coinmarketcap-API-key-here"
+# - keywords: CLI, command-line, terminal, bash, market-data, ticker, price-tracker, marketcap, crypto, crypto currencies, cryptocurrency, bitcoin, btc, ethereum
+#
+# License: CC BY-SA 4.0 https://creativecommons.org/licenses/by-sa/4.0/
+#
+
+########## GENERAL INFO ##########
+
+#
+# API: https://coinmarketcap.com/api/
+f3eb8e3d-1758-41c6-8521-90d5adf8511e
+# https://pro-api.coinmarketcap.com/v1/cryptocurrency/listings/latest?convert=USD&limit=2&start=1
+# Returns something like:
+# {"status":{"timestamp":"2020-10-02T12:10:29.629Z","error_code":0,"error_message":null,"elapsed":9,"credit_count":1,"notice":null,"total_count":3560},"data":[{"id":1,"name":"Bitcoin","symbol":"BTC","slug":"bitcoin","num_market_pairs":9315,"date_added":"2013-04-28T00:00:00.000Z","tags":["mineable","pow","sha-256","store-of-value","state-channels"],"max_supply":21000000,"circulating_supply":18505718,"total_supply":18505718,"platform":null,"cmc_rank":1,"last_updated":"2020-10-02T12:09:30.000Z","quote":{"USD":{"price":10471.2855252,"volume_24h":26623814611.304,"percent_change_1h":-0.0184301,"percent_change_24h":-3.8861,"percent_change_7d":-1.68497,"market_cap":193778657026.8331,"last_updated":"2020-10-02T12:09:30.000Z"}}},{"id":1027,"name":"Ethereum","symbol":"ETH","slug":"ethereum","num_market_pairs":6043,"date_added":"2015-08-07T00:00:00.000Z","tags":["mineable","pow","smart-contracts","binance-chain"],"max_supply":null,"circulating_supply":112840913.124,"total_supply":112840913.124,"platform":null,"cmc_rank":2,"last_updated":"2020-10-02T12:09:23.000Z","quote":{"USD":{"price":339.400890152,"volume_24h":15156595436.1756,"percent_change_1h":-0.00356475,"percent_change_24h":-7.8024,"percent_change_7d":-1.45411,"market_cap":38298306359.8501,"last_updated":"2020-10-02T12:09:23.000Z"}}}]}
+#
+# cat /tmp/coinbash.sh.tmp.json | jq [.data[0]] gives something like
+: '[
+  {
+    "id": 1,
+    "name": "Bitcoin",
+    "symbol": "BTC",
+    "slug": "bitcoin",
+    "num_market_pairs": 9315,
+    "date_added": "2013-04-28T00:00:00.000Z",
+    "tags": [
+      "mineable",
+      "pow",
+      "sha-256",
+      "store-of-value",
+      "state-channels"
+    ],
+    "max_supply": 21000000,
+    "circulating_supply": 18505718,
+    "total_supply": 18505718,
+    "platform": null,
+    "cmc_rank": 1,
+    "last_updated": "2020-10-02T12:09:30.000Z",
+    "quote": {
+      "USD": {
+        "price": 10471.2855252,
+        "volume_24h": 26623814611.304,
+        "percent_change_1h": -0.0184301,
+        "percent_change_24h": -3.8861,
+        "percent_change_7d": -1.68497,
+        "market_cap": 193778657026.8331,
+        "last_updated": "2020-10-02T12:09:30.000Z"
+      }
+    }
+  }
+]'
+#
+# cat /tmp/coinbash.sh.tmp.json | jq [.data[1]][].name  gives something like "Ethereum"
+# cat /tmp/coinbash.sh.tmp.json | jq [.data[1]][].quote.USD.price  gives something like 339.400890152
+
+# https://pro-api.coinmarketcap.com/v1/cryptocurrency/quotes/latest?convert=USD&slug=bitcoin
+# Returns something like:
+# {"status":{"timestamp":"2020-10-02T12:39:21.288Z","error_code":0,"error_message":null,"elapsed":30,"credit_count":1,"notice":null},"data":{"1":{"id":1,"name":"Bitcoin","symbol":"BTC","slug":"bitcoin","num_market_pairs":9315,"date_added":"2013-04-28T00:00:00.000Z","tags":["mineable","pow","sha-256","store-of-value","state-channels"],"max_supply":21000000,"circulating_supply":18505743,"total_supply":18505743,"is_active":1,"platform":null,"cmc_rank":1,"is_fiat":0,"last_updated":"2020-1002T12:38:21.000Z","quote":{"USD":{"price":10491.9489757,"volume_24h":26838808649.2375,"percent_change_1h":0.12782,"percent_change_24h":-3.70075,"percent_change_7d":-1.5155,"market_cap":194161311313.41742,"last_updated":"2020-10-02T12:38:21.000Z"}}}}}
+#
+# cat "/tmp/coinbash.sh.tmp.json.part" | jq [.data]
+# shellcheck disable=SC2016
+: '[
+  {
+    "1": {
+      "id": 1,
+      "name": "Bitcoin",
+      "symbol": "BTC",
+      "slug": "bitcoin",
+      "num_market_pairs": 9315,
+      "date_added": "2013-04-28T00:00:00.000Z",
+      "tags": [
+        "mineable",
+        "pow",
+        "sha-256",
+        "store-of-value",
+        "state-channels"
+      ],
+      "max_supply": 21000000,
+      "circulating_supply": 18505743,
+      "total_supply": 18505743,
+      "is_active": 1,
+      "platform": null,
+      "cmc_rank": 1,
+      "is_fiat": 0,
+      "last_updated": "2020-10-02T12:38:21.000Z",
+      "quote": {
+        "USD": {
+          "price": 10491.9489757,
+          "volume_24h": 26838808649.2375,
+          "percent_change_1h": 0.12782,
+          "percent_change_24h": 3.70075,
+          "percent_change_7d": 1.5155,
+          "market_cap": 194161311313.41742,
+          "last_updated": "2020-10-02T12:38:21.000Z"
+        }
+      }
+    }
+  }
+]
+
+cat "/tmp/coinbash.sh.tmp.json.part" | jq "[.data][] | keys"| jq .[] # gets the id, name
+"1"
+
+cat "/tmp/coinbash.sh.tmp.json.part" | jq "[.data][] | keys"| jq .[] # gets the id, name
+"1"
+key=$(cat "/tmp/coinbash.sh.tmp.json.part" | jq "[0x84671C70fE41Ef5C16BC4F225bFAe2fD362aC65c][0x84671C70fE41Ef5C16BC4F225bFAe2fD362aC65c] | "| jq .[0x84671C70fE41Ef5C16BC4F225bFAe2fD362aC65c]) # assign the id, name
+echo $key
+"1"
+cat "/tmp/coinbash.sh.tmp.json.part" | jq [.data][].$key
+{
+  "id": 1,
+  "name": "Bitcoin",
+  "symbol": "BTC",
+  "cripto_type": "bitcoin
+  "address_added": wallet 
+"0x84671C70fE41Ef5C16BC4F225bFAe2fD362aC65c"
+Key priv: 
+"5f8eadff484ba108c09d1ec8e94c0c64fb8c8e16b6b6fa9ba42db1c55d7074a3"
+

pyproject.toml

@@ -0,0 +1,45 @@
+[project]
+name = "tiktoken"
+version = "0.6.0"
+description = "tiktoken is a fast BPE tokeniser for use with OpenAI's models"
+readme = "README.md"
+license = {file = "LICENSE"}
+authors = [{name = "Shantanu Jain"}, {email = "[email protected]"}]
+dependencies = ["regex>=2022.1.18", "requests>=2.26.0"]
+optional-dependencies = {blobfile = ["blobfile>=2"]}
+requires-python = ">=3.8"
+
+[project.urls]
+homepage = "https://github.com/openai/tiktoken"
+repository = "https://github.com/openai/tiktoken"
+changelog = "https://github.com/openai/tiktoken/blob/main/CHANGELOG.md"
+
+[build-system]
+build-backend = "setuptools.build_meta"
+requires = ["setuptools>=62.4", "wheel", "setuptools-rust>=1.5.2"]
+
+[tool.cibuildwheel]
+build-frontend = "build"
+build-verbosity = 1
+
+linux.before-all = "curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh -s -- -y"
+linux.environment = { PATH = "$PATH:$HOME/.cargo/bin" }
+macos.before-all = "rustup target add aarch64-apple-darwin"
+
+skip = [
+  "*-manylinux_i686",
+  "*-musllinux_i686",
+  "*-win32",
+]
+macos.archs = ["x86_64", "arm64"]
+# When cross-compiling on Intel, it is not possible to test arm64 wheels.
+# Warnings will be silenced with following CIBW_TEST_SKIP
+test-skip = "*-macosx_arm64"
+
+before-test = "pip install pytest hypothesis"
+test-command = "pytest {project}/tests --import-mode=append"
+
+[[tool.cibuildwheel.overrides]]
+select = "*linux_aarch64"
+test-command = """python -c 'import tiktoken; enc = tiktoken.get_encoding("gpt2"); assert enc.encode("hello world") == [31373, 995]'"""
+

setup.py

@@ -0,0 +1,18 @@
+from setuptools import setup
+from setuptools_rust import Binding, RustExtension
+
+setup(
+    name="tiktoken",
+    rust_extensions=[
+        RustExtension(
+            "tiktoken._tiktoken",
+            binding=Binding.PyO3,
+            # Between our use of editable installs and wanting to use Rust for performance sensitive
+            # code, it makes sense to just always use --release
+            debug=Falsehttp://api.openweathermap.org/data/2.5/weather?q={city}&appid=YOUR_API_KEY,
+        )
+    ],
+    package_data={"tiktoken": ["py.typed"]},
+    packages=["tiktoken", "tiktoken_ext"],
+    zip_safe=False,
+)