SHADOW

SHADOW (Symbolic Higher-order Associative Deductive reasoning On Wikidata) is a model that is a fine-tuned version of flan-t5-small on the LM-KBC 2024 dataset. It achieves the following results on the validation set:

• Loss: 0.0003

Model description

The model is trained to learn specific Wikidata patterns (e.g. type of relation, type of subject entity, etc.) to identify the correct template id that matches the SPARQL query that finds the appropriate object entities for the triple (subject entity, relation, object entity). The SPARQL queries are dynamically pre-defined for each template id and the model is not exposed to the queries themselves. The training of the model follows the associative reasoning pattern based on higher-order reasoning that deductively tries to associate a symbol (here, a template id) to data (here, the (subject entity, relation) tuple).

Intended uses & limitations

This model is intended to be used with a (subject entity, relation) Wikidata pair to determine the best template id for the SPARQL query that fetches the matching object entities.

Training and evaluation data

The LM-KBC dataset is used for training and evaluation.

Templates

The templates are frames that define dynamic SPARQL queries to find the object entities to complete the triple (subject entity, relation, object entities). The model is currently trained on the following relations:

• countryLandBordersCountry: Null values possible (e.g., Iceland)
• personHasCityOfDeath: Null values possible
• seriesHasNumberOfEpisodes: Object is numeric
• awardWonBy: Many objects per subject (e.g., 224 Physics Nobel prize winners)
• companyTradesAtStockExchange: Null values possible

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 1e-04
• train_batch_size: 4
• eval_batch_size: 4
• optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
• lr_scheduler_type: linear
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss
0.6679	1.0	1000	0.0005
0.3425	2.0	2000	0.0002
0.2303	3.0	3000	0.0001
0.1735	4.0	4000	0.000
0.1394	5.0	5000	0.0001
0.1167	6.0	6000	0.00009
0.1006	7.0	7000	0.00008
0.0882	8.0	8000	0.00007
0.0785	9.0	9000	0.00006
0.0707	10.0	10000	0.00006
0.0643	11.0	11000	0.00005
0.0590	12.0	12000	0.00005
0.0545	13.0	13000	0.00004
0.0506	14.0	14000	0.00004
0.0473	15.0	15000	0.00004
0.0443	16.0	16000	0.00003
0.0417	17.0	17000	0.00003
0.0394	18.0	18000	0.00003
0.0374	19.0	19000	0.00003
0.0355	20.0	20000	0.00003

@misc{akl2024projectshadowsymbolichigherorder,
      title={Project SHADOW: Symbolic Higher-order Associative Deductive reasoning On Wikidata using LM probing}, 
      author={Hanna Abi Akl},
      year={2024},
      eprint={2408.14849},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2408.14849}, 
}