Initial commit.

PLDRv51G-106M-2-model-checkpoint.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0280da71ea5da0f9b1745084a66f3c772b4c47fa1f38702c56d001ec3e9409ae
+size 422476082

PLDRv51G_106M_2_hyperparameters.py

@@ -0,0 +1,24 @@
+import torch.nn.functional as F
+
+hpdict={'num_layers': 5,
+        'd_model': 896,
+        'num_heads': 14,
+        'dff': 2389,
+        'Gcachelst': './predefined_G_LM_cache_list_IDENTITY_5layer_14head_64x64_paper.pkl',
+        'input_vocab_size': 32000,
+        'max_seq_len': 1024,
+        'epochs': 1,
+        'save_model_path': './PLDRv51G-106M-2-checkpoint',
+        'warmup_steps': 2000,
+        'lr_total_steps': 250000,
+        'learning_rate': 0.0003,
+        'lr_alpha': 0.1,
+        'adamw_decay': 0.1,
+        'activation': F.silu,
+        'disable_amp': False,
+        'auto_size_minimum': None,
+        'disable_fsdp_mixed_precision': False,
+        'fsdp_cpu_offload': False,
+        'fsdp_sharding_strategy': 'HYBRID_SHARD',
+        'backward_prefetch': 'PRE',
+        'save_type': 'torch'}

README.md

@@ -1,3 +1,67 @@
----
-license: apache-2.0
----
+---
+language:
+- en
+tags:
+- text-generation
+- large-language-model
+- power-law-decoder-representations
+- power-law-graph-attention
+- pldr-llm
+- kv-cache
+- g-cache
+- kvg-cache
+- pytorch
+license: apache-2.0
+datasets:
+- tiiuae/falcon-refinedweb
+---
+
+# PLDR-LLM-v51G-106M-2
+
+## Model Description
+
+PLDR-LLM-v51G-106M-2 is a large language model from power law decoder representations with KV-cache and G-cache support, which is a new foundational language model architecture that utilizes power law graph attention to generate deductive and inductive outputs. This model has a parameter size of 106M. It refers to PLDRv51G-106M-2 whose architecture and training details are provided in Table 1 of the research paper titled [PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inference](https://arxiv.org/abs/2502.13502).
+
+## Training data
+
+PLDR-LLM-v51G-106M-2 was pretrained on the [RefinedWeb](https://huggingface.co/datasets/tiiuae/falcon-refinedweb), a publicly available English web dataset with extensive filtering and deduplication.
+
+## Training procedure
+
+This model was trained for ~8B tokens on RefinedWeb over 250k steps per rank. It was trained  autoregressively with cross-entropy loss.
+
+## Intended Use and Limitations
+
+This model is intended to be used for research purposes. Given text as input prompt, it carries out next token prediction to generate continuation text. The context length for this model is 1024 tokens. 
+
+### How to use
+
+- The model checkpoint and tokenizer can be loaded into the PLDR-LLM framework to generate text as described in the code repository for training this model: [PLDR-LLM-with-KVG-cache](https://github.com/burcgokden/PLDR-LLM-with-KVG-cache).
+
+### LM Evaluation Harness Support
+
+- The model can be used with a fork of LM-Evaluation-Harness Suite with PLDR-LLM with KV-cache and G-cache support: [lm-evaluation-harness-with-PLDR-LLM-kvg-cache](https://github.com/burcgokden/lm-evaluation-harness-with-PLDR-LLM-kvg-cache).
+
+### Limitations and Biases
+
+Large Language Models may generate text that is profane, lewd, socially unacceptable or offensive based on the contents of the dataset it was pretrained. RefinedWeb is a dataset that is as toxic and biased as the Pile. Please see the papers for [RefinedWeb](https://arxiv.org/abs/2306.01116) and [the Pile](https://arxiv.org/pdf/2101.00027) for more information. Moreover, large language models are also susceptible to hallucinations and may generate text that contains incorrect, irrelevant or misleading information. Since it is very hard to expect the contents of generated text ahead of time, the output of the large language models need to be heavily moderated and curated to avoid undesired content to appear without warning.
+
+## Eval results
+
+The evaluation results on benchmarks with zero-shot setting and their comparison to LLM models of similar size reported in the literature can be found in Tables 3-5 and 7 of the [research paper](https://arxiv.org/abs/2502.13502).
+
+### BibTeX entry and citation info
+
+Please cite this model as:
+
+```bibtex
+@misc{gokden2025pldrllmkvgcache,
+      title={PLDR-LLMs Learn A Generalizable Tensor Operator That Can Replace Its Own Deep Neural Net At Inference}, 
+      author={Burc Gokden},
+      year={2025},
+      eprint={2502.13502},
+      archivePrefix={arXiv},
+      primaryClass={cs.CL},
+      url={https://arxiv.org/abs/2502.13502}, 
+}
+```

predefined_G_LM_cache_list_IDENTITY_5layer_14head_64x64_paper.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4e8eb98538109cb263369958fa73fcaa30773435fc0a25d4f13d957942f772a8
+size 1148520

refinedweb-tokenizer-pldrllm-kvg-cache-paper.tar.gz

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4485f0cb53da6c9a25d99153b70f52c2e643789b54786bed4168f11e83091818
+size 616297