Uncovering a curious case of distribution shift

Date: 2026-07-18

The source code for the experiment mentioned in this article is available on GitHub.

I was studying chapter 9 of D2L recently on natural language processing (NLP) using recurrent neural networks (RNN). To validate and further reinforce my understanding of the concepts presented in the chapter, I decided to train a word-level language model from scratch based on the King James Version (KJV) of the Christian Bible.

Various factors informed my decision to use the Bible as the data source for my more realistic (compared to the toy example in D2L) NLP deep learning experiment.

1. The Bible is among one of the English books with the most words, surpassed only by a few select novels. This makes it a prime candidate for performing word-level analysis and deep learning as opposed to shorter texts such as The Time Machine by H. G. Wells
2. Growing up in a community with moderate Christian influence, the Bible was familiar to me compared to novels that I may not have read or even heard of
3. The raw Bible text taken straight from Open Bible was already sufficiently structured and well formatted, perfect for practicing how to create and implement simple data pre-processing pipelines without ingesting structured CSV data directly

With the problem statement defined, I proceeded to develop a Python script to train my Elman RNN-based language model on the entire corpus of the KJV Bible. With the help of DeepSeek, I was able to quickly clarify the key concepts required to implement the model training pipeline with Huawei’s Ascend-optimized deep learning framework known as MindSpore and quickly triage and debug issues which otherwise would have wasted me precious hours or days chasing them.

As an aside, I started migrating from Jupyter notebooks to plain Python scripts on my Orange Pi AIpro (20T) recently. While Jupyter notebooks made it easy to visualize the results from my deep learning experiments and explain my work to myself and others via inline text, using them effectively meant I had to keep my browser open for the entire duration of the experiment and wait for the entire training pipeline to finish. This was fine for short experiments lasting up to an hour but quickly proved unwieldy with more complex neural networks such as ResNet and DenseNet which easily took hours or even days to complete. Migrating to a vanilla Python approach enabled me to run my deep learning pipelines in the background with convenient command-line tools such as tmux, checking on the progress occasionally with MLflow without keeping the monitor attached.

With my NLP pipeline humming along, I was set to celebrate the success of training my first word-level language model from scratch using a dataset I selected and prepared. Little did I know that this was just the start of my debugging journey …

A classic textbook example of overfitting, or is it?

I divided the full text of the KJV Bible directly with an 80-20 ratio with 80% of the text for my training set and the remaining 20% for my validation set. What caveats can you think of that could render this approach flawed or suboptimal? ;-)

Within the first few hours of training my language model, the training loss and perplexity was steadily decreasing over time. However, the validation loss (perplexity) quickly reached its optimum at the 2nd epoch, with the loss steadily increasing thereafter.

Since I used a 2-layer Elman RNN without regularization techniques such as weight decay, momentum and dropout, I initially assumed it to be a classic textbook example of overfitting. After all, my network was deep in both the conventional and time-oriented sense - it had 2 stacked recurrent layers and the model was trained on sequences of 32 consecutive words. While the Bible contains many words for a mere human, such a dataset is considered tiny by today’s standards of training LLMs such as ChatGPT and DeepSeek.

With these metrics and my initial hypothesis in mind, I asked DeepSeek for tips and tricks to reduce overfitting. Not surprisingly, it gave the following general recommendations typical for most deep learning tasks.

1. Reduce the learning rate
2. Add weight decay and momentum
3. Apply dropout within the stacked RNN layers
4. Reduce the number of units of hidden state within the stacked RNN - I used 256, DeepSeek recommended 128 or fewer

I decided to adopt (1) and (2) as a starting point to introduce regularization and mitigate overfitting.

Mitigating overfitting with abysmal results

With the 2nd run of my NLP pipeline using weight decay and momentum as my primary technique to combat overfitting, the training and validation losses actually surged initially before following a steady decline after 5 or so epochs. Both the training and validation losses followed this same general pattern which suggested that the regularization techniques were kicking in. Unfortunately, the overall performance of my model was abysmal compared to my initial run.

1. During my initial run, the optimal results were obtained at the end of the 2nd epoch with the training perplexity reduced to around 50 and the validation perplexity remaining around 200
2. By the end of the 2nd run, my training perplexity stabilized at around 90 while my validation perplexity remained at around 1500

Considering the vocabulary of the KJV Bible consisted of around 5600 unique words after removing infrequent words appearing less than 5 times throughout the entire text, my updated model was only performing marginally better than random guessing on the validation set! What was going on? Surely the adoption of regularization techniques such as weight decay and momentum, even if ineffective, should not exacerbate overfitting?

The “Aha!” moment

On a Friday evening, a crucial detail on how the Bible is structured surfaced in my mind. Unlike typical English novels which follow a consistent vocabulary and writing style throughout the entire book, the Bible is divided into 2 major sections, each with its own distinct vocabulary, style and phrasing.

1. The Old Testament focuses primarily on the Ten Commandments and the chosen people, with adherence to the Law and discipline as its recurring theme
2. The New Testament on the other hand focuses on belief and salvation, with love, forgiveness and understanding as its recurring theme

With this key observation in mind, I asked DeepSeek what percentage of words across the entire KJV Bible belong to the Old vs. New Testaments. Unsurprisingly, the numbers coincided heavily with my naive 80-20 train/test split.

1. The Old Testament accounts for approximately 77% of total words within the KJV Bible
2. In contrast, the New Testament accounts for just under 23% of total words within the KJV Bible

Simply put, my model was effectively trained to recognize phrases from the Old Testament but asked to predict phrases from the New Testament, both of which are characterized by distinct vocabulary and writing styles, a classic example of distribution shift. The regularization techniques only served to widen the gap since the sliver of New Testament phrases which made it to the training set were forgotten after a dozen or so epochs through regularization.

The solution? Shuffle the phrases across the entire Bible before applying the 80-20 train/test split. With this simple optimization, the training and validation perplexities both stabilize near the 20-30 range over 20+ epochs, without compromising on the learning rate or applying regularization techniques such as dropout.

Lesson learned - spend time understanding the data you are dealing with before rushing to train your next model!

Concluding remarks and going further

This was an interesting albeit unexpected lesson in data preprocessing and handling with designing and implementing deep learning experiments.

I hope you enjoyed reading this article as much as I did authoring it and stay tuned for updates ;-)

Subscribe:

Return to homepage