{"id":843,"date":"2025-04-27T20:09:08","date_gmt":"2025-04-28T00:09:08","guid":{"rendered":"https:\/\/brian.digitalmaddox.com\/blog\/?p=843"},"modified":"2025-04-27T20:09:08","modified_gmt":"2025-04-28T00:09:08","slug":"a-short-guide-to-nlp-part-3-how-we-do-nlp-today","status":"publish","type":"post","link":"https:\/\/brian.digitalmaddox.com\/blog\/?p=843","title":{"rendered":"A Short Guide to NLP \u2013 Part 3: How We Do NLP Today"},"content":{"rendered":"

OK I will admit it took me longer than I had planned to finish this up. \u00a0Life got in the way. \u00a0But now I think is a good time to finish up the series and move on to another.<\/p>\n

In the first parts of this series, we looked at why English is such a challenging language for natural language processing (NLP), and how early methods like rules-based systems and Bag of Words models approached the problem.\u00a0 <\/span>But like everything else, access to power GPU\u2019s and AI has had a big impact on modern NLP techniques.<\/p>\n

Today, NLP has evolved dramatically, powered by new methods that are far more powerful \u2014 and much better at handling the complexity and ambiguity of human language.<\/p>\n

Let us look at some of the modern techniques that have reshaped NLP in recent years.<\/p>\n

Word Embeddings: Giving Words Meaning<\/h1>\n

One major leap forward came with word embeddings<\/a> \u2014 ways of representing words as vectors in a multi-dimensional space, where words with similar meanings are close together.<\/p>\n

Unlike older methods that treated words as isolated tokens, embeddings like Word2Vec<\/a>, GloVe<\/a>, and FastText<\/a> actually learned that words like \u201cking\u201d and \u201cqueen\u201d are related, or that \u201cParis\u201d and \u201cFrance\u201d have a strong connection.<\/p>\n

These embeddings helped NLP systems recognize relationships and similarities between words \u2014 even when they were not identical \u2014 which made downstream tasks like translation, search, and classification much more accurate.<\/p>\n

Let us dig a little deeper into how these things work by taking a look at Word2Vec.\u00a0 <\/span>Word2Vec generates a vector representation of words in a multi-dimensional space.\u00a0 <\/span>In this space, similar words end up being \u201ccloser\u201d to one another.\u00a0 <\/span>The idea behind it is that words that appear in similar contexts tend to have similar meanings.<\/p>\n

Word2Vec actually generates a shallow neural network to learn the relationships between words.\u00a0 <\/span>It typically has a dense hidden layer that takes generated embeddings from the inputs to predict the outputs.\u00a0 <\/span>It does this in two different ways.<\/p>\n

The first way is the Skip-Gram Model<\/strong><\/a> and is the most common method of generating the network. \u00a0We select a word in the middle of the sentence and train the model to predict the nearby words within a certain window size. \u00a0These word pairs (such as cat and furry) are fed into the network for training.\u00a0 The model learns to predict context words given an input word. \u00a0<\/span>A very simplified example of the network is given below.<\/p>\n

Input Layer (one-hot encoding) : (cat, furry)\r\n          |\r\nHidden Layer (dense layer)\r\n          |\r\nOutput Layer (Softmax layer that outputs probabilities for words being a nearby word)<\/pre>\n
The second method is the Continuous Bag of Words<\/strong><\/a> model.\u00a0 <\/span>Here the model is trained by giving it context words and it then tries to predict the target word.\u00a0 <\/span>It can be thought of as a reverse of the previous technique. It uses a fixed-sized window of context words around the target word.\u00a0 <\/span>So if our target word in a sentence is cat, we could have context words of furry and purrs.\u00a0 <\/span>Another of my masterful graphic arts shows how this works below.<\/p>\n
Input Layer (words are one-hot encoded, averaged into an input vector, and the vectors summed depending on the implementation)\r\n          |\r\nHidden Layer (dense layer)\r\n          |\u00a0<\/span>\r\n\r\nOutput Layer (Softmax layer that outputs probabilities of a single word being the target word)<\/pre>\n
But traditional embeddings had one big weakness: each word only had one vector, no matter how it was used. \u201cBank\u201d always meant the same thing, whether you were talking about rivers or money.<\/p>\n
That led to the next big innovation: contextual embeddings.<\/p>\n
Contextual Embeddings: Understanding Words in Context<\/h1>\n
With models like ELMo<\/strong><\/a> and BERT<\/strong><\/a>, NLP systems moved beyond static word meanings.\u00a0 <\/span>Now, the context of a word \u2014 the words around it \u2014 could change its meaning. If you wrote \u201che deposited cash at the bank\u201d versus \u201cthey sat by the river bank,\u201d modern models could understand that bank means two very different things.<\/p>\n
This made a huge difference in tasks like question answering, translation, and search, where understanding the nuance of a sentence is critical.<\/p>\n
Transformers: The Engine Behind Modern NLP<\/h2>\n
All of this was made possible by a groundbreaking architecture introduced in 2017: the Transformer.<\/p>\n
Transformers, introduced in the paper Attention is All You Need<\/strong><\/a>, replaced older models like RNNs<\/strong><\/a> and LSTMs<\/strong><\/a> by doing something surprisingly simple: instead of reading words one by one, they looked at the entire sentence (or even paragraph) all at once.<\/p>\n
At the heart of transformers is the attention mechanism<\/strong><\/a>, which lets the model figure out which words in a sentence are most important when trying to understand a given word.<\/p>\n
This means a model can understand relationships across an entire sentence \u2014 or even multiple sentences \u2014 no matter how far apart the words are.<\/p>\n
For an example, we will look at how BERT (Bi-directional Encoder Representations from Transformers) works.\u00a0 <\/span>BERT is an encoder only transformer architecture that consists of four main modules:<\/p>\n
    \n
  1. The Tokenizer Module<\/strong> converts the words in a sentence into a series of tokens.<\/li>\n
  2. The Embeddings Module<\/strong> converts the tokens into embeddings.<\/li>\n
  3. The Encoder Module<\/strong> is a stack of Transformer blocks that use self-attention without causal masking.\u00a0 <\/span>Self-attention here basically means the blocks determine the relative importance of component in a sequence relative to the other components of the sentence.\u00a0 <\/span>This lets the model learn the relationships between words no matter where they appear in a sentence.<\/li>\n
  4. The Task-Head Module<\/strong> uses the final embeddings to predict outputs, such as masked words or next sentence predictions, typically through a classification layer.<\/li>\n<\/ol>\n
    BERT processes the entire sentence at once, verses other methods that look at text sequentially.\u00a0 <\/span>As a whole model, BERT is trained and fine-tuned using two unsupervised-learning tasks. \u00a0<\/span><\/p>\n
      \n
    1. Masked Language Modeling<\/strong><\/a> (MLM) masks random words in a sentence and the model learns how to predict the masked words based on the context provided by the other words in the sentence.<\/li>\n
    2. Next Sentence Prediction<\/strong><\/a> (NSP) works by giving the model pairs of sentences and it learns to predict whether or not the second sentence logically follows the first. \u00a0<\/span><\/li>\n<\/ol>\n
      Pre-trained Language Models: A Giant Leap<\/h1>\n
      Once transformers became popular, researchers realized they could pre-train huge language models on massive amounts of text \u2014 and then fine-tune them for specific tasks.<\/p>\n
      Instead of training a model from scratch every time you wanted to do translation, or summarization, or classification, you could just start with a giant model that already knew a lot about language, and tweak it slightly.<\/p>\n
      Some of the most important pre-trained models today include:<\/p>\n