Paid episode

The full episode is only available to paid subscribers of The AiEdge Newsletter

How Do We Create Tokens From Words in LLMs

Introduction to LLMs
Damien Benveniste
Nov 30, 2023
∙ Paid
Share

This is the 5th video in the Introduction to LLMs series. Check out the Table of Contents for more information.

  • Word-level vs Character-level vs Subword level embeddings

  • The Byte Pair Encoding Strategy

  • Special Tokens

  • The Hugging Face Tokenizer

  • Visualizing the Attentions with the Padding Tokken

Here are the latest articles you may have missed:

Listen to this episode with a 7-day free trial

Subscribe to The AiEdge Newsletter to listen to this post and get 7 days of free access to the full post archives.

The AiEdge Newsletter
The AiEdge Newsletter
Authors
Damien Benveniste
Recent Posts
The Transformer Architecture (V2)
 • Damien Benveniste
How To Get a Job in Machine Learning!
 • Damien Benveniste
Training a Deep Q-Network to Play Video Games
 • Damien Benveniste
Advanced Deep Learning Techniques
 • Damien Benveniste
Introduction to Deep Learning
 • Damien Benveniste
Optimizing Models with AutoML
 • Damien Benveniste
GBM vs XGBoost vs LightGBM vs CatBoost
 • Damien Benveniste
Introduction to Machine Learning System Design
 • Damien Benveniste