Exploration of the Impact from Early Injection of Symbolic Knowledge into a Language Model

Large Language Models (LLMs) have demonstrated remarkable capabilities but continue to exhibit fundamental limitations in areas humans find intuitive, such as spatial reasoning. This paper investigates whether early injection of symbolic knowledge during training could address these limitations. Unlike post-training alignment techniques, our approach aims to influence how concepts are initially represented within model parameters. Through controlled experiments with a very simple model, word2vec models, we demonstrate that early injection of spatial symbolic knowledge produces qualitatively different representations of spatial concepts, particularly regarding dynamic relationships and potential movements. Models trained with this approach show enhanced understanding of spatial dynamics, capturing not just static positions but causal consequences. While quantitative differences were modest in our small-scale experiment, the qualitative improvements in semantic retrieval suggest promising directions for integrating symbolic knowledge in more complex language models. This work contributes a novel perspective on the timing of symbolic knowledge integration, challenging the prevailing paradigm of large-scale pretraining followed by alignment.