<aside> ❔
What is Fine-Tuning?
Fine-tuning means taking a pre-trained language model—something that already learned English from millions of examples—and training it further to match a particular writing style you choose.
You can do this with just about 100 examples. If you haven’t, see How to Prepare your Dataset
</aside>
The first step to fine-tune a language model… is finding a language model!
We'll use DistilGPT2, a simplified and smaller version of GPT-2: the popular language model developed by OpenAI in 2019.
DistilGPT2 is a Text Generation Model, which you can find on 🤗 HuggingFace: an online platform hosting models and datasets.
You can tell it's small by looking at the size of its .safetensors file [see image].
To fine-tune DistilGPT2, you'll use Python (a programming language) and a few specialized tools available through Google Colab, Google's online coding platform.
If you don't have an account yet, create one on Google Colab.
Make sure your dataset (text you wrote or collected) is ready. If you haven't prepared it yet, check out How to Prepare your Dataset
Here’s the full notebook code you’ll use:
If you know Google Colab already, run the code directly. Otherwise, follow the steps below carefully.
After opening our code notebook on Google Colab, you’ll see "cells" of text and code.
You can't edit this notebook directly because it's shared from my account. First, click File → Save a copy in Drive. This opens your own editable copy.
Run each code-cell by clicking the PLAY button.
First, install Python packages (tools):
!pip install -U transformers[torch] tf-keras datasets
This installs: