CLIP Sparse Autoencoder Checkpoint
This model is a sparse autoencoder trained on CLIP's internal representations.
Model Details
Architecture
- Layer: 4
- Layer Type: hook_resid_post
- Model: open-clip:laion/CLIP-ViT-B-32-DataComp.XL-s13B-b90K
- Dictionary Size: 49152.0
- Input Dimension: 768.0
- Expansion Factor: 64.0
- CLS Token Only: False
Training
- Training Images: 1299936.0000
- Learning Rate: 0.0002
- L1 Coefficient: 0.0000
- Batch Size: 4096.0
- Context Size: 49.0
Performance Metrics
Sparsity
L0 (Active Features): 965.4101
Dead Features: 0.0000
Mean Passes Since Fired: 2.1601
Reconstruction
- Explained Variance: 0.9987
- Explained Variance Std: 0.0032
- MSE Loss: 0.0000
- L1 Loss: 1456.6568
- Overall Loss: 0.0001
Training Details
- Training Duration: 3927 seconds
- Final Learning Rate: 0.0000
- Warm Up Steps: 200.0
- Gradient Clipping: 1.0
Additional Information
- Original Checkpoint Path: /network/scratch/p/praneet.suresh/imgnet_checkpoints/c8de8491-tinyclip_sae_16_hyperparam_sweep_lr/n_images_1300020.pt
- Wandb Run: https://wandb.ai/perceptual-alignment/vanilla-imagenet-spatial_only-sweep/runs/7dfdq3cz
- Random Seed: 42.0