Large language models like ChatGPT process text by converting words into numerical tokens rather than understanding the actual meaning of the words.

LLM breakdown 1/6: Tokenization (words to integers) hypothes.is/a/giWe8jyIEfGSTEcmPXX9XQ Here’s the thing: ChatGPT doesn’t know how many “r”s are in strawberry, because what we see as the word “ strawberry” GPT sees as token 41236. The letter “r” is token 81.thats interesting I thought Chat was actually reading and processing the words we type

LLM breakdown 1/6: Tokenization (words to integers) hypothes.is/a/giWe8jyIEfGSTEcmPXX9XQ Here’s the thing: ChatGPT doesn’t know how many “r”s are in strawberry, because what we see as the word “ strawberry” GPT sees as token 41236. The letter “r” is token 81.thats interesting I thought Chat was actually reading and processing the words we type