The problem
A language described for two thousand years, and a machine that cannot read the description
Tamil grammar has been written down, argued over and taught continuously since தொல்காப்பியம். The rules are not missing. What is missing is any of it in a form a machine can use.
What "low-resource" actually claims
It is a statement about data, not about literature. A language is low-resource when there is little annotated, machine-readable material to train and test on: tagged corpora, treebanks, lexicons with structure, benchmarks.
Tamil is rich in இலக்கியம் and இலக்கணம் and poor in all of that. So a model learns Tamil the way a tourist does, from exposure, and it never gets taught the grammar that Tamil scholars have spent two millennia making explicit.
Worth saying to anyone who finds the phrase insulting: it describes a pipeline failure, not the language. And it is fixable by building the data, which is what this project does as a by-product of being used.
What happens to a Tamil word inside a model
Before a model reasons about text it splits it into tokens, using a vocabulary learned mostly from English. Tamil carries its grammar inside the word, so those vocabularies have no entry for what a Tamil word is made of, and the word shatters into fragments that mean nothing.
We measured it rather than quoting it. Six Tamil words, two OpenAI encodings, the count of tokens each word becomes, next to the number of உறுப்பு the word actually has.
| Word | English | cl100k | o200k | morphemes | tokens under cl100k |
|---|---|---|---|---|---|
| மரம் | tree | 7 | 2 | 1 | |
| மரத்தில் | in the tree | 11 | 3 | 3 | |
| வந்தான் | he came | 11 | 3 | 3 | |
| வருகிறான் | he is coming | 14 | 3 | 3 | |
| கொடுத்தான் | he gave | 16 | 4 | 3 | |
| படித்துக்கொண்டிருந்தார்கள் | they were reading | 35 | 7 | — |
Measured 2026-08-18 with tiktoken 0.14.0. Reproduce it:
scripts/measure-tokens.py in this repo.
The last row is the one to sit with. படித்துக்கொண்டிருந்தார்கள் is one Tamil word carrying root, aspect, tense, person, number and honorific. Under the older encoding it becomes 35 tokens. Its English translation, "they were reading", becomes 3.
The newer encoding is far better, and that is worth saying plainly rather than picking the number that suits us: the same word drops from 35 tokens to 7. The problem is shrinking, and it has not gone away. Seven fragments still carry no morphology, so nothing downstream can tell that the word contains a tense.
What we do not claim: we have not verified which shipping products use which encoding, so this page names encodings and not models. If you know the mapping for a model you care about, tell us and we will cite you.
Somebody has already worked on the fix
The fragmentation is not a law of nature. It follows from choices made when a tokenizer is built, and the choice that matters most turns out to be the one made before the algorithm even runs.
Egalitarian Language Representation in Language Models: It All Begins with Tokenizers
Menan Velayuthan, Kengatharaiyer Sarveswaran
Proceedings of the 31st International Conference on Computational Linguistics (COLING 2025), pp. 5987–5996, Abu Dhabi, UAE, 2025. Code: vmenan/tokenizers-coling2025.
Their finding is that pre-tokenization, the step that decides what counts as a character before any merging happens, matters more for fair representation than the tokenization algorithm itself. Their fix builds the Tamil writing system into that step: Grapheme Pair Encoding, which merges graphemes rather than bytes, and which beats byte-level tokenizers on Tamil, Sinhala and Hindi.
We cite this for two reasons. It is the scholarly version of the table above, done properly by people who then did something about it. And the second author is Sarveswaran, whose morphological analyser this whole project is built on, which says something about how small the group of people working on computable Tamil actually is.
A grapheme is not a morpheme, and the distinction matters for what comes later. Their work respects the writing system: a Tamil character stays whole instead of being split into bytes. Our long-term stage asks the same question one level up, about units of meaning rather than units of writing. Both are needed and neither replaces the other.
Why the fragments cost more than context
A long Tamil prompt fills the window faster than the same content in English, which is the cost everybody notices. Two quieter costs matter more. Reasoning degrades across long runs of meaningless fragments. And in retrieval, an embedding built from fragments does not sit near the meaning of the word, so the search fails before the model ever sees the text.
The measurement somebody else already made
ILAKKANAM is the first Tamil linguistics benchmark: 820 school examination questions across Grades 1 to 13, covering phonetics, phonology, morphology, syntax and semantics.
From Phonemes to Meaning: Evaluating Large Language Models on Tamil
J. Varsha, Menan Velayuthan, S. Karunakaran, R. Nivethiga, Kengatharaiyer Sarveswaran
arXiv:2511.12387, 2025.
Same senior author as the tokenizer paper above, and as the analyser this project runs on. That is not a coincidence we can take credit for.
- The best frontier model scored 79.6%. Claude Sonnet 4.5 scored 71.1%. Open-source models scored between 37.9% and 60.7%.
- Accuracy falls as the grade rises, which is the opposite of what understanding would predict.
- A model's ability to name a question's linguistic category does not predict getting the question right.
The authors' conclusion is the sentence this whole project turns on: the performance reflects exposure, not understanding.
What we are not doing
Retraining a base model on more Tamil. It costs money a nonprofit does not have, it needs the very data that does not exist yet, and it fixes nothing about citation. A larger model that is confidently wrong about வேற்றுமை is still wrong, and a teacher still cannot check it.
What we do instead
Put the grammar where a model can look it up, and make every answer name its source. The model keeps doing what it is good at, which is meaning. The parts that are rule-governed get handled by rules that cannot hallucinate, and the answer arrives with the நூற்பா attached so a scholar can overrule it.