T5 Unigram tokenization

load the vocabulary and log scores     # add scores along a path
normalize the text and mark word starts # "my name" → ["␣my", "␣name"]
for each text piece:
    if it is a special token, append its ID and continue
    best[0] = 0; all other scores = −∞
    for each position, find all tokens that start here:
        if no single-character token exists, add an unknown
        score each match = best[position] + token_score
        if score > best[end], save score and this token
    follow saved tokens backward from the end
    combine adjacent unknowns; append IDs in reading order
return the token IDs