GPT-2 BPE tokenization

load the vocabulary and merge ranks      # learned during training
split the input text into words         # "my name" → ["my", " name"]
for each piece, from left to right:     # outer loop
    start with its byte tokens          # preserve special tokens
    while an adjacent pair has a merge rank:
        take the pair with the lowest rank
        merge its non-overlapping occurrences in this piece
    append the remaining token IDs to the output
return the token IDs