cygu
/

llama-2-7b-logit-watermark-distill-aar-k3

Text Generation

Generated from Trainer

text-generation-inference

Inference Endpoints

Model card Files Files and versions Community

Edit model card

Model description

Logits-based watermark distilled Llama 2 7B using the Aar $k = 3$ watermarking strategy in the paper On the Learnability of Watermarks for Language Models.

Training hyperparameters

The following hyperparameters were used during training:

learning_rate: 1e-05
train_batch_size: 16
eval_batch_size: 8
seed: 42
distributed_type: multi-GPU
num_devices: 4
total_train_batch_size: 64
total_eval_batch_size: 32
optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
lr_scheduler_type: cosine
lr_scheduler_warmup_steps: 500
training_steps: 5000

Framework versions

Transformers 4.29.2
Pytorch 2.0.1+cu117
Datasets 2.13.1
Tokenizers 0.13.3

Downloads last month: 14

Inference Examples

Text Generation

This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train cygu/llama-2-7b-logit-watermark-distill-aar-k3

Collection including cygu/llama-2-7b-logit-watermark-distill-aar-k3

On the Learnability of Watermarks for Language Models

Paper: https://arxiv.org/abs/2312.04469 (ICLR 2024), GitHub: https://github.com/chenchenygu/watermark-learnability • 50 items • Updated May 23