metadata

license: mit
pipeline_tag: text-generation
tags:
  - ONNX
  - ONNXRuntime
  - ONNXRuntimeWeb
  - phi3
  - transformers.js
  - transformers
  - nlp
  - conversational
  - custom_code
inference: false

Phi-3 Mini-4K-Instruct ONNX model for in-browser inference

Running Phi3-mini-4K entirely in the browser! Check out this demo.

This repository hosts the optimized Web version of ONNX Phi-3-mini-4k-instruct model to accelerate inference in the browser with ONNX Runtime Web.

The Phi-3-Mini-4K-Instruct is a 3.8B parameters, lightweight, state-of-the-art open model trained with the Phi-3 datasets that includes both synthetic data and the filtered publicly available websites data with a focus on high-quality and reasoning dense properties. When assessed against benchmarks testing common sense, language understanding, math, code, long context and logical reasoning, Phi-3 Mini-4K-Instruct showcased a robust and state-of-the-art performance among models with less than 13 billion parameters.

How to run

ONNX Runtime Web is a JavaScript library to enable web developers to deploy machine learning models directly in web browsers, offering multiple backends leveraging hardware acceleration. WebGPU backend is recommended to run Phi-3-mini efficiently.

Here is an E2E example for running this optimized Phi3-mini-4K for the web, with ONNX Runtime harnessing WebGPU.

Supported devices and browser with WebGPU: Chrome 113+ and Edge 113+ for Mac, Windows, ChromeOS, and Chrome 121+ for Android. Pls visit here for tracking WebGPU support in browsers

Performance Metrics

Performance vary between GPUs. The more powerful the GPU, the faster the speed. On a NVIDIA GeForce RTX 4090: ~42 tokens/second

Additional Details

To obtain other optimized Phi3-mini-4k ONNX models for server platforms, Windows, Linux, Mac desktops, and mobile, please visit Phi-3-mini-4k-instruct onnx model. The model differences in the web version compared to other versions:

the model is fp16 with int4 block quantization for weights
the 'logits' output is fp32
the model uses MHA instead of GQA
onnx and external data file need to stay below 2GB to be cacheable in chromium

To optimize a fine-tuned Phi3-mini-4k model to run with ONNX Runtime Web, please follow this Olive example. Olive is an easy-to-use model optimization tool for generating an optimized ONNX model to efficiently run with ONNX Runtime across platforms.

Model Description

Developed by: Microsoft
Model type: ONNX
Inference Language(s) (NLP): JavaScript
License: MIT
Model Description: This is the web version of the Phi-3 Mini-4K-Instruct model for ONNX Runtime inference.

Model Card Contact

guschmue, qining