A Byte-Pair-encoded translation of the Universal declaration of human rights.
The “tokenization” is a crucial step that happens before any interaction with a language or text based machine learning model. A text of any language is translated into an array of numbers, which represent common repetitions of characters. Most commonly this is achieved using the Byte-Pair-encoding algorithm (BPE), which iteratively abstracts the most common sequence of byte-pairs from a certain dataset into a dictionary of id’s – this repeats until every unique character and repeated series of characters has an id. Leaving no repetition and only numbers for the models to digest.
Tokenization is itself as well based on large quantities of text-analysis, which–as any training data–already imposes great biases and reproduces inequalities. Large Language Models that proclaim “world knowledge” and pursue an idea of Universality, are therefore already inscribing a non-neutral encoding before even running the actual model itself.
