Update README.md

README.md

@@ -64,70 +64,48 @@ _Kotoba-Whisper_ is a collection of distilled [Whisper](https://arxiv.org/abs/22
 we employ OpenAI's [Whisper large-v3](https://huggingface.co/openai/whisper-large-v3) as the teacher model, and the student model that consists the full encoder of the 
 teacher whisper model, and a decoder with two layers initialized from the first and last layer of the whisper model. 
 As the initial version, we release ***kotoba-whisper-v1.0*** trained on the `large` subset of [ReazonSpeech](https://huggingface.co/datasets/reazon-research/reazonspeech), 
-which amounts 1,253 hours of audio with 16,861,235 characters of transcriptions (5 sec audio with 18 text tokens in average).
-Kotoba-whisper-v1.0 is competitive or even outpeform the largest whisper model in Japanese ASR benchmarks, while being 6.3 times faster than the whisper model.
+which amounts 1,253 hours of audio with 16,861,235 characters of transcriptions (5 sec audio with 18 text tokens in average) after 
+those transcriptions more than 10 WER are removed (see [WER Filter](https://huggingface.co/distil-whisper/distil-large-v3#wer-filter)).
+The model was trained for 8 epochs with batch size 256 with sampling rate of 16kHz, and the raining and evaluation code to reproduce kotoba-whisper is available at [https://github.com/kotoba-tech/kotoba-whisper](https://github.com/kotoba-tech/kotoba-whisper).
 
 
-
-## Table of Contents
-Since the sequential algorithm is the "de-facto" transcription algorithm across the most popular Whisper libraries 
-(Whisper cpp, Faster-Whisper, OpenAI Whisper), this distilled model is designed to be compatible with these libraries. 
-You can expect significant performance gains by switching from previous Distil-Whisper checkpoints to distil-large-v3 
-when using these libraries. For convenience, the weights for the most popular libraries are already converted, 
-with instructions for getting started below.
-
-1. [Evaluation Results](#evaluation-results)
-2. [Transformers Usage](#transformers-usage)
-   * [Short-Form Transcription](#short-form-transcription)
-   * [Sequential Long-Form](#sequential-long-form)
-   * [Chunked Long-Form](#chunked-long-form)
-   * [Speculative Decoding](#speculative-decoding)
-   * [Additional Speed and Memory Improvements](#additional-speed--memory-improvements)
-2. [Library Integrations](#library-integrations)
-   * [Whisper cpp](#whispercpp)
-   * [Faster Whisper](#faster-whisper)
-3. [Model Details](#model-details)
-
-
-## Evaluation Results
-***kotoba-whisper-v1.0*** achieves better CER and WER than the [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) in the in-domain held-out test set from ReazonSpeech, and
+Kotoba-whisper-v1.0 achieves better CER and WER than the [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) in the in-domain held-out test set from ReazonSpeech, and
 achieves competitive CER and WER on the out-of-domain test set including [JSUT basic 5000](https://sites.google.com/site/shinnosuketakamichi/publication/jsut) and
 the Japanese subset from [CommonVoice 8.0](https://huggingface.co/datasets/common_voice).
 
-### CER
+
+- ***CER***
 
 | Model                                                                                           | CommonVoice 8.0 (Japanese) | JSUT Basic 5000 | ReazonSpeech Test |
 |:------------------------------------------------------------------------------------------------|---------------------------:|----------------:|------------------:|
-| [***kotoba-tech/kotoba-whisper-v1.0***](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0) |                       9.44 |            8.48 |             12.60  |
+| [**kotoba-tech/kotoba-whisper-v1.0**](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)                       **9.44** |        **8.48** |         **12.60** |
 | [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                       |                       8.52 |            7.18 |             15.18 |
 | [openai/whisper-medium](https://huggingface.co/openai/whisper-medium)                           |                      11.34 |            9.87 |             29.56 |
 | [openai/whisper-small](https://huggingface.co/openai/whisper-small)                             |                      15.26 |           14.22 |             34.29 |
 | [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny)                               |                      46.86 |           35.69 |             96.69 |
 
-### WER
+- ***WER***
 
 | Model                                                                                           | CommonVoice 8.0 (Japanese) | JSUT Basic 5000 | ReazonSpeech Test |
 |:------------------------------------------------------------------------------------------------|---------------------------:|----------------:|------------------:|
-| [***kotoba-tech/kotoba-whisper-v1.0***](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0) |                      59.27 |           64.36 |             56.62 |
+| [**kotoba-tech/kotoba-whisper-v1.0**](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)   |                  **59.27** |       **64.36** |         **56.62** |
 | [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                       |                      55.41 |           59.34 |             60.23 |
 | [openai/whisper-medium](https://huggingface.co/openai/whisper-medium)                           |                      63.64 |           69.52 |             76.04 |
 | [openai/whisper-small](https://huggingface.co/openai/whisper-small)                             |                      74.21 |           82.02 |             82.99 |
 | [openai/whisper-tiny](https://huggingface.co/openai/whisper-tiny)                               |                      93.78 |           97.72 |             94.85 |
 
-###  Latency
-As kotoba-whisper uses the same architecture as [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3),
+- ***Latency***: As kotoba-whisper uses the same architecture as [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3),
 it inherits the benefit of the improved latency compared to [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3) 
 (**6.3x faster than large-v3**, see the table below taken from [distil-whisper/distil-large-v3](https://huggingface.co/distil-whisper/distil-large-v3)).
 
-| Model                                                                        | Params / M | Rel. Latency |
-|------------------------------------------------------------------------------|------------|--------------|
+| Model                                                                                        | Params / M | Rel. Latency |
+|----------------------------------------------------------------------------------------------|------------|--------------|
 | **[kotoba-tech/kotoba-whisper-v1.0](https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0)**| **756**    | **6.3**      |
-| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                   | 1550       | 1.0          |
+| [openai/whisper-large-v3](https://huggingface.co/openai/whisper-large-v3)                    | 1550       | 1.0          |
 
 
 ## Transformers Usage
-
-distil-large-v3 is supported in the Hugging Face 🤗 Transformers library from version 4.39 onwards. To run the model, first 
+Kotoba-Whisper is supported in the Hugging Face 🤗 Transformers library from version 4.39 onwards. To run the model, first 
 install the latest version of Transformers. For this example, we'll also install 🤗 Datasets to load a toy audio dataset 
 from the Hugging Face Hub:
 
@@ -137,7 +115,6 @@ pip install --upgrade transformers accelerate datasets[audio]
 ```
 
 ### Short-Form Transcription
-
 The model can be used with the [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline)
 class to transcribe short-form audio files (< 30-seconds) as follows:
 
@@ -146,19 +123,15 @@ import torch
 from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
 from datasets import load_dataset
 
-
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
-
+# config
 model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
 
-model = AutoModelForSpeechSeq2Seq.from_pretrained(
-    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
-)
+# load model
+model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
 model.to(device)
-
 processor = AutoProcessor.from_pretrained(model_id)
-
 pipe = pipeline(
     "automatic-speech-recognition",
     model=model,
@@ -169,110 +142,55 @@ pipe = pipeline(
     device=device,
 )
 
-dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
+# load sample audio
+dataset = load_dataset("japanese-asr/ja_asr.common_voice_8_0", split="test")
 sample = dataset[0]["audio"]
 
+# run inference
 result = pipe(sample)
 print(result["text"])
 ```
 
-To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:
+- To transcribe a local audio file, simply pass the path to your audio file when you call the pipeline:
 ```diff
 - result = pipe(sample)
 + result = pipe("audio.mp3")
 ```
 
-For segment-level timestamps, pass the argument `return_timestamps=True` and return the `"chunks"` output:
+- For segment-level timestamps, pass the argument `return_timestamps=True` and return the `"chunks"` output:
 ```python
 result = pipe(sample, return_timestamps=True)
 print(result["chunks"])
 ```
 
-<details>
-
-<summary> For more control over the generation parameters, use the model + processor API directly: </summary>
-
-Ad-hoc generation arguments can be passed to `model.generate`, including `num_beams` for beam-search, `return_timestamps` 
-for segment-level timestamps, and `prompt_ids` for prompting. See the [docstrings](https://huggingface.co/docs/transformers/en/model_doc/whisper#transformers.WhisperForConditionalGeneration.generate)
-for more details.
-
-```python
-import torch
-from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
-from datasets import Audio, load_dataset
-
-
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
-
-model_id = "kotoba-tech/kotoba-whisper-v1.0"
-
-model = AutoModelForSpeechSeq2Seq.from_pretrained(
-    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
-)
-model.to(device)
-
-processor = AutoProcessor.from_pretrained(model_id)
-
-dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
-dataset = dataset.cast_column("audio", Audio(processor.feature_extractor.sampling_rate))
-sample = dataset[0]["audio"]
-
-input_features = processor(
-  sample["array"], sampling_rate=sample["sampling_rate"], return_tensors="pt"
-).input_features
-
-input_features = input_features.to(device, dtype=torch_dtype)
-
-gen_kwargs = {
-  "max_new_tokens": 128,
-  "num_beams": 1,
-  "return_timestamps": False,
-}
-
-pred_ids = model.generate(input_features, **gen_kwargs)
-pred_text = processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=gen_kwargs["return_timestamps"])
-
-print(pred_text)
-```
-
-</details>
-
 ### Sequential Long-Form
-
-Unlike previous Distil-Whisper releases, distil-large-v3 is specifically designed to be compatible with OpenAI's sequential
-long-form transcription algorithm. This algorithm uses a sliding window for buffered inference of long audio files (> 30-seconds),
-and returns more accurate transcriptions compared to the [chunked long-form algorithm](#chunked-long-form).
-
+Kotoba-whisper is designed to be compatible with OpenAI's sequential long-form transcription algorithm. This algorithm uses a sliding window for buffered 
+inference of long audio files (> 30-seconds), and returns more accurate transcriptions compared to the [chunked long-form algorithm](#chunked-long-form).
 The sequential long-form algorithm should be used in either of the following scenarios:
+
 1. Transcription accuracy is the most important factor, and latency is less of a consideration
 2. You are transcribing **batches** of long audio files, in which case the latency of sequential is comparable to chunked, while being up to 0.5% WER more accurate
 
 If you are transcribing single long audio files and latency is the most important factor, you should use the chunked algorithm
 described [below](#chunked-long-form). For a detailed explanation of the different algorithms, refer to Sections 5 of 
-the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf).
-
-The [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) 
+the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf). The [`pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines#transformers.AutomaticSpeechRecognitionPipeline) 
 class can be used to transcribe long audio files with the sequential algorithm as follows: 
 
 ```python
 import torch
+import numpy as np
 from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
 from datasets import load_dataset
 
-
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
-
+# config
 model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
 
-model = AutoModelForSpeechSeq2Seq.from_pretrained(
-    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
-)
+# load model
+model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
 model.to(device)
-
 processor = AutoProcessor.from_pretrained(model_id)
-
 pipe = pipeline(
     "automatic-speech-recognition",
     model=model,
@@ -283,75 +201,19 @@ pipe = pipeline(
     device=device,
 )
 
-dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
-sample = dataset[0]["audio"]
+# load sample audio (concatenate instances to creaete a long audio)
+dataset = load_dataset("japanese-asr/ja_asr.common_voice_8_0", split="test")
+sample = {"array": np.concatenate([i["array"] for i in dataset[:20]["audio"]]), "sampling_rate": dataset[0]['audio']['sampling_rate'], "path": "tmp"}
 
+# run inference
 result = pipe(sample)
 print(result["text"])
 ```
 
-<details>
-
-<summary> For more control over the generation parameters, use the model + processor API directly: </summary>
-
-```python
-import torch
-from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
-from datasets import Audio, load_dataset
-
-
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
-
-model_id = "kotoba-tech/kotoba-whisper-v1.0"
-
-model = AutoModelForSpeechSeq2Seq.from_pretrained(
-    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
-)
-model.to(device)
-
-processor = AutoProcessor.from_pretrained(model_id)
-
-dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
-dataset = dataset.cast_column("audio", Audio(processor.feature_extractor.sampling_rate))
-sample = dataset[0]["audio"]
-
-inputs = processor(
-    sample["array"],
-    sampling_rate=sample["sampling_rate"],
-    return_tensors="pt",
-    truncation=False,
-    padding="longest",
-    return_attention_mask=True,
-)
-inputs = inputs.to(device, dtype=torch_dtype)
-
-gen_kwargs = {
-    "max_new_tokens": 448,
-    "num_beams": 1,
-    "condition_on_prev_tokens": False,
-    "compression_ratio_threshold": 1.35,  # zlib compression ratio threshold (in token space)
-    "temperature": (0.0, 0.2, 0.4, 0.6, 0.8, 1.0),
-    "logprob_threshold": -1.0,
-    "no_speech_threshold": 0.6,
-    "return_timestamps": True,
-}
-
-pred_ids = model.generate(**i   nputs, **gen_kwargs)
-pred_text = processor.batch_decode(pred_ids, skip_special_tokens=True, decode_with_timestamps=False)
-
-print(pred_text)
-```
-
-</details>
 
 ### Chunked Long-Form
-
-distil-large-v3 remains compatible with the Transformers chunked long-form algorithm. This algorithm should be used when 
-a single large audio file is being transcribed and the fastest possible inference is required. In such circumstances, 
-the chunked algorithm is up to 9x faster than OpenAI's sequential long-form implementation (see Table 7 of the 
-[Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf)).
-
+This algorithm should be used when a single large audio file is being transcribed and the fastest possible inference is required. In such circumstances, 
+the chunked algorithm is up to 9x faster than OpenAI's sequential long-form implementation (see Table 7 of the [Distil-Whisper paper](https://arxiv.org/pdf/2311.00430.pdf)).
 To enable chunking, pass the `chunk_length_s` parameter to the `pipeline`. For distil-large-v3, a chunk length of 25-seconds
 is optimal. To activate batching over long audio files, pass the argument `batch_size`:
 
@@ -360,19 +222,15 @@ import torch
 from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
 from datasets import load_dataset
 
-
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
-
+# config
 model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
 
-model = AutoModelForSpeechSeq2Seq.from_pretrained(
-    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
-)
+# load model
+model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
 model.to(device)
-
 processor = AutoProcessor.from_pretrained(model_id)
-
 pipe = pipeline(
     "automatic-speech-recognition",
     model=model,
@@ -385,17 +243,17 @@ pipe = pipeline(
     device=device,
 )
 
-dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
-sample = dataset[0]["audio"]
+# load sample audio (concatenate instances to creaete a long audio)
+dataset = load_dataset("japanese-asr/ja_asr.common_voice_8_0", split="test")
+sample = {"array": np.concatenate([i["array"] for i in dataset[:20]["audio"]]), "sampling_rate": dataset[0]['audio']['sampling_rate'], "path": "tmp"}
 
+# run inference
 result = pipe(sample)
 print(result["text"])
 ```
 
-
 ### Additional Speed & Memory Improvements
-
-You can apply additional speed and memory improvements to Distil-Whisper to further reduce the inference speed and VRAM 
+You can apply additional speed and memory improvements to further reduce the inference speed and VRAM 
 requirements. These optimisations primarily target the attention kernel, swapping it from an eager implementation to a 
 more efficient flash attention version.
 
@@ -438,106 +296,10 @@ Once a valid PyTorch version is installed, SDPA is activated by default. It can
 + model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True, attn_implementation="sdpa")
 ```
 
-## Library Integrations
-
-### Whisper.cpp
-
-Distil-Whisper can be run with the [Whisper.cpp](https://github.com/ggerganov/whisper.cpp) package with the original 
-sequential long-form transcription algorithm. In a provisional benchmark on Mac M1, distil-large-v3 is over 5x faster 
-than Whisper large-v3, while performing to within 0.8% WER over long-form audio.
-
-Steps for getting started:
-
-1. Clone the Whisper.cpp repository:
-```
-git clone https://github.com/ggerganov/whisper.cpp.git
-cd whisper.cpp
-```
-2. Install the Hugging Face Hub Python package:
-```bash
-pip install --upgrade huggingface_hub
-```
-And download the GGML weights for distil-large-v3 using the following Python snippet:
-
-```python
-from huggingface_hub import hf_hub_download
-
-hf_hub_download(repo_id='kotoba-tech/kotoba-whisper-v1.0-ggml', filename='ggml-distil-large-v3.bin', local_dir='./models')
-```
-
-Note that if you do not have a Python environment set-up, you can also download the weights directly with `wget`:
-
-```bash
-wget https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0-ggml/resolve/main/ggml-distil-large-v3.bin -P ./models
-```
-
-3. Run inference using the provided sample audio:
-
-```bash
-make -j && ./main -m models/ggml-distil-large-v3.bin -f samples/jfk.wav
-```
-
-### Faster-Whisper
-
-Faster-Whisper is a reimplementation of Whisper using [CTranslate2](https://github.com/OpenNMT/CTranslate2/), a fast 
-inference engine for Transformer models.
-
-First, install the Faster-Whisper package according to the [official instructions](https://github.com/SYSTRAN/faster-whisper#installation).
-For this example, we'll also install 🤗 Datasets to load a toy audio dataset from the Hugging Face Hub:
-
-```bash
-pip install --upgrade pip
-pip install --upgrade git+https://github.com/SYSTRAN/faster-whisper datasets[audio]
-```
-
-The following code snippet loads the distil-large-v3 model and runs inference on an example file from the LibriSpeech ASR
-dataset:
-
-```python
-import torch
-from faster_whisper import WhisperModel
-from datasets import load_dataset
-
-# define our torch configuration
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-compute_type = "float16" if torch.cuda.is_available() else "float32"
-
-# load model on GPU if available, else cpu
-model = WhisperModel("distil-large-v3", device=device, compute_type=compute_type)
-
-# load toy dataset for example
-dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
-sample = dataset[1]["audio"]["path"]
-
-segments, info = model.transcribe(sample, beam_size=1)
-
-for segment in segments:
-    print("[%.2fs -> %.2fs] %s" % (segment.start, segment.end, segment.text))
-```
-
-To transcribe a local audio file, simply pass the path to the audio file as the `audio` argument to transcribe:
-
-```python
-segments, info = model.transcribe("audio.mp3", beam_size=1)
-```
-
 
 ## Model Details
+See [https://huggingface.co/distil-whisper/distil-large-v3#model-details](https://huggingface.co/distil-whisper/distil-large-v3#model-details).
 
-Distil-Whisper inherits the encoder-decoder architecture from Whisper. The encoder maps a sequence of speech vector 
-inputs to a sequence of hidden-state vectors. The decoder auto-regressively predicts text tokens, conditional on all 
-previous tokens and the encoder hidden-states. Consequently, the encoder is only run forward once, whereas the decoder 
-is run as many times as the number of tokens generated. In practice, this means the decoder accounts for over 90% of 
-total inference time. Thus, to optimise for latency, the focus is on minimising the inference time of the decoder.
-
-To distill the Whisper model, we reduce the number of decoder layers while keeping the encoder fixed. 
-The encoder (shown in green) is entirely copied from the teacher to the student and frozen during training. 
-The student's decoder consists of a subset of the teacher decoder layers, which are intialised from maximally spaced layers.
-The model is then trained on a weighted sum of the KL divergence and pseudo-label loss terms.
-
-<p align="center">
-  <img src="https://huggingface.co/datasets/distil-whisper/figures/resolve/main/architecture.png?raw=true" width="600"/>
-</p>
 
 ## Evaluation
 
@@ -557,123 +319,62 @@ Evaluation can then be run end-to-end with the following example:
 
 ```python
 from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor
-from datasets import load_dataset
+from datasets import load_dataset, features
 from evaluate import load
 import torch
 from tqdm import tqdm
 
-# define our torch configuration
-device = "cuda:0" if torch.cuda.is_available() else "cpu"
-torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
-
+# config
 model_id = "kotoba-tech/kotoba-whisper-v1.0"
+torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
+device = "cuda:0" if torch.cuda.is_available() else "cpu"
+audio_column = 'audio'
+text_column = 'transcription'
+batch_size = 16
 
-# load the model + processor
-model =  AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, use_safetensors=True, low_cpu_mem_usage=True)
-model = model.to(device)
+# load model
+model = AutoModelForSpeechSeq2Seq.from_pretrained(model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True)
+model.to(device)
 processor = AutoProcessor.from_pretrained(model_id)
 
-# load the dataset with streaming mode
-dataset = load_dataset("librispeech_asr", "clean", split="validation", streaming=True)
+# load the dataset and sample the audio with 16kHz
+dataset = load_dataset("japanese-asr/ja_asr.common_voice_8_0", split="test")
+dataset = dataset.cast_column(audio_column, features.Audio(sampling_rate=processor.feature_extractor.sampling_rate))
+dataset = dataset.select([0, 1, 2, 3, 4, 5, 6])
 
-# define the evaluation metric
-wer_metric = load("wer")
+# preprocess and batch the dataset
 
 def inference(batch):
     # 1. Pre-process the audio data to log-mel spectrogram inputs
     audio = [sample["array"] for sample in batch["audio"]]
     input_features = processor(audio, sampling_rate=batch["audio"][0]["sampling_rate"], return_tensors="pt").input_features
     input_features = input_features.to(device, dtype=torch_dtype)
-    
     # 2. Auto-regressively generate the predicted token ids
     pred_ids = model.generate(input_features, max_new_tokens=128)
-    
     # 3. Decode the token ids to the final transcription
     batch["transcription"] = processor.batch_decode(pred_ids, skip_special_tokens=True)
-    batch["reference"] = batch["text"]
+    batch["reference"] = batch[text_column]
     return batch
 
-# batch size 16 inference
-dataset = dataset.map(function=inference, batched=True, batch_size=16)
+dataset = dataset.map(function=inference, batched=True, batch_size=batch_size)
 
+# iterate over the dataset and run inference
 all_transcriptions = []
 all_references = []
-
-# iterate over the dataset and run inference
 for result in tqdm(dataset, desc="Evaluating..."):
     all_transcriptions.append(result["transcription"])
     all_references.append(result["reference"])
 
 # normalize predictions and references
-all_transcriptions = [processor.normalize(transcription) for transcription in all_transcriptions]
-all_references = [processor.normalize(reference) for reference in all_references]
-
-# compute the WER metric
-wer = 100 * wer_metric.compute(predictions=all_transcriptions, references=all_references)
-print(wer)
+all_transcriptions = [transcription.replace(" ", "") for transcription in all_transcriptions]
+all_references = [reference.replace(" ", "") for reference in all_references]
 
+# compute the CER metric
+cer_metric = load("cer")
+cer = 100 * cer_metric.compute(predictions=all_transcriptions, references=all_references)
+print(cer)
 ```
-**Print Output:**
-```
-2.428920763531516
-```
-
-
-## Data
-
-Distil-Whisper is trained on 22,000 hours of audio data from nine open-source, permissively licensed speech datasets on the 
-Hugging Face Hub:
-
-| Dataset                                                                                 | Size / h | Speakers | Domain                      | Licence         |
-|-----------------------------------------------------------------------------------------|----------|----------|-----------------------------|-----------------|
-| [People's Speech](https://huggingface.co/datasets/MLCommons/peoples_speech)             | 12,000   | unknown  | Internet Archive            | CC-BY-SA-4.0    |
-| [Common Voice 13](https://huggingface.co/datasets/mozilla-foundation/common_voice_13_0) | 3,000    | unknown  | Narrated Wikipedia          | CC0-1.0         |
-| [GigaSpeech](https://huggingface.co/datasets/speechcolab/gigaspeech)                    | 2,500    | unknown  | Audiobook, podcast, YouTube | apache-2.0      |
-| Fisher                                                                                  | 1,960    | 11,900   | Telephone conversations     | LDC             |
-| [LibriSpeech](https://huggingface.co/datasets/librispeech_asr)                          | 960      | 2,480    | Audiobooks                  | CC-BY-4.0       |
-| [VoxPopuli](https://huggingface.co/datasets/facebook/voxpopuli)                         | 540      | 1,310    | European Parliament         | CC0             |
-| [TED-LIUM](https://huggingface.co/datasets/LIUM/tedlium)                                | 450      | 2,030    | TED talks                   | CC-BY-NC-ND 3.0 |
-| SwitchBoard                                                                             | 260      | 540      | Telephone conversations     | LDC             |
-| [AMI](https://huggingface.co/datasets/edinburghcstr/ami)                                | 100      | unknown  | Meetings                    | CC-BY-4.0       |
-||||||
-| **Total**                                                                               | 21,770   | 18,260+  |                             |                 |
-
-The combined dataset spans 10 distinct domains and over 50k speakers. The diversity of this dataset is crucial to ensuring 
-the distilled model is robust to audio distributions and noise. 
-
-The audio data is then pseudo-labelled using the Whisper large-v3 model: we use Whisper to generate predictions for all 
-the audio in our training set and use these as the target labels during training. Using pseudo-labels ensures that the 
-transcriptions are consistently formatted across datasets and provides sequence-level distillation signal during training.
-
-## WER Filter
-
-The Whisper pseudo-label predictions are subject to mis-transcriptions and hallucinations. To ensure we only train on 
-accurate pseudo-labels, we employ a simple WER heuristic during training. First, we normalise the Whisper pseudo-labels
-and the ground truth labels provided by each dataset. We then compute the WER between these labels. If the WER exceeds
-a specified threshold, we discard the training example. Otherwise, we keep it for training.
-
-Section 9.2 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430) demonstrates the effectiveness of this filter 
-for improving downstream performance of the distilled model. We also partially attribute Distil-Whisper's robustness to 
-hallucinations to this filter.
-
-## Training
-
-The model was trained for 80,000 optimisation steps (or 11 epochs) with batch size 256. The Tensorboard training logs can 
-be found under: https://huggingface.co/kotoba-tech/kotoba-whisper-v1.0/tensorboard?params=scalars#frame
-
-## Results
-
-The distilled model performs to within 1.5% WER of Whisper large-v3 on out-of-distribution (OOD) short-form audio, within
-1% WER on sequential long-form decoding, and outperforms large-v3 by 0.1% on chunked long-form. This performance gain is 
-attributed to lower hallucinations.
-
-For a detailed per-dataset breakdown of the evaluation results, refer to Tables 16 and 17 of the [Distil-Whisper paper](https://arxiv.org/abs/2311.00430)
-
-Distil-Whisper is also evaluated on the [ESB benchmark](https://arxiv.org/abs/2210.13352) datasets as part of the [OpenASR leaderboard](https://huggingface.co/spaces/hf-audio/open_asr_leaderboard),
-where it performs to within 0.2% WER of Whisper.
 
-## Reproducing Kotoba-Whisper
-Training and evaluation code to reproduce Kotoba-Whisper is available at the repository: [TBA](TBA).
 
 ## Acknowledgements
 * OpenAI for the Whisper [model](https://huggingface.co/openai/whisper-large-v3).