README.md

metadata

license: other
model-index:
  - name: Wizard-Vicuna-13B-Uncensored-SpQR
    results:
      - task:
          type: text-generation-inference
          name: Text Generation
        dataset:
          type: c4
          name: C4
        metrics:
          - type: perplexity
            value: 7.354
      - task:
          type: text-generation-inference
          name: Text Generation
        dataset:
          type: wikitext2
          name: WikiText-2
        metrics:
          - type: perplexity
            value: 5.685
      - task:
          type: text-generation-inference
          name: Text Generation
        dataset:
          type: ptb
          name: PTB
        metrics:
          - type: perplexity
            value: 20.822
pipeline_tag: text-generation

Wizard-Vicuna-13B-Uncensored-SpQR

Overview

This model is an SpQR 4-bit quantization of the original Wizard-Vicuna-13B-Uncensored-HF

Quantization Specifications

• Quantization: 4-bit, group size of 16, per-channel with scale and zero-point of 3 bits.
• Outliers: Threshold set at 0.2.
• Permutation Order: act_order.
• Dampening: Set at 1e0.
• Sampling: 128 samples.
• Logging: Via Weights & Biases.

Evaluation Metrics

The following perplexity scores were obtained on various datasets:

Dataset	Perplexity
c4	7.354
wikitext2	5.685
ptb	20.822