ILAKKANAM
The first benchmark that tests a model on Tamil linguistics rather than on translation. 820 school examination questions, Grades 1 to 13, from the University of Jaffna. It is how we would find out whether any of this actually helps.
Paper
From Phonemes to Meaning (arXiv:2511.12387)
Senior author
Kengatharaiyer Sarveswaran, University of Jaffna
Size
820 questions, Grades 1–13, six categories
Status
The dataset was not public when we last checked
From Phonemes to Meaning: Evaluating Large Language Models on Tamil
J. Varsha, Menan Velayuthan, S. Karunakaran, R. Nivethiga, Kengatharaiyer Sarveswaran
arXiv:2511.12387, 2025.
The senior author is the same person behind ThamizhiMorph, the analyser this project runs on, and behind the tokenizer work cited on our why page. Four of the things this project stands on trace back to one research group.
What it found
- The best frontier model scored 79.6%. Claude Sonnet 4.5 scored 71.1%. Open-source models scored between 37.9% and 60.7%.
- Accuracy falls as the school grade rises, which is the opposite of what understanding would predict.
- Whether a model can correctly name the linguistic category of a question does not predict whether it answers correctly.
The authors' conclusion is the sentence this project turns on: the performance reflects exposure, not understanding. A model has read a great deal of Tamil. It has not been taught any இலக்கணம்.
Their morphology category is where our tools do their work, and their finding that accuracy collapses on harder items is the shape of problem a lookup layer can fix and a larger model cannot.
How we would use it
The measurement that matters for this project is not our own test suite passing. It is whether a model answers Tamil linguistics questions better with these tools attached than without them. Same model, same questions, tools on and tools off, scored per category and grade band.
That number is the honest test of whether the project deserves anyone's attention, and we do not have it yet. We built the harness, ran a smoke test, and stopped.
The smoke test found something more useful than a score. Under a neutral prompt, a model with the tools attached called them 0% of the time. The server was returning correct answers. Nothing was telling the model to ask. Tool descriptions turned out to be a product surface rather than documentation, and "did the model call the tool at all" is now a permanent evaluation metric rather than a bug we fixed once.
What we are asking
Access to the dataset, or word when it publishes. Until then we test against fixtures we wrote ourselves, which are useful for catching regressions and are not an independent measurement, because we wrote both the questions and the thing being tested.
If it publishes, it becomes our held-out test set and our own fixtures get demoted to development data. We would also want to check the reverse direction: our fixture words are flagged in the store so that they can never leak into any dataset we publish, and a benchmark is only worth something while it stays uncontaminated. thamizh@ief-global.org