இயல்பு தமிழ் படிமம் a Tamil-native model

Why data, and not a bigger model

Tamil is short of machine-readable linguistic data rather than of literature. So the useful move is to make the missing data, and the way to make it without funding a labelling project is to produce it as a by-product of something people already want.

Every analysis the server resolves is stored with its sources and the date. Verified, provenance-tagged, and growing whenever anyone uses the tool.

1. Somebody asks about a Tamil word.
2. The engine answers from cited sources, or returns an honest gap.
3. The answer is stored with every claim's provenance attached.
4. Curation separates verified from generated from disputed, and drops anything a licence will not let us publish.
5. Published datasets of exactly the kinds that do not currently exist for Tamil.
6. Better Tamil tooling, ours and other people's, which brings more use, which returns to step one.

The eval fixtures are flagged in the store so they can never leak into a published dataset. A benchmark is worth nothing once the thing being measured has read it.

Predicting the next Tamil word is three problems

Only one of them actually needs a trained model, which is the observation this whole section rests on:

So the target is not a Tamil model that does everything. It is a pairing: a large model proposes a word and its grammatical features, and this engine turns that into correct Tamil. The hard, error-prone morphology gets handled by rules that cannot hallucinate.

The stages

Each one is independently useful, and each is a prerequisite for the next. If the last stage never happens, everything before it still stands on its own.

1. A

   Word in context

   Take a sentence, split it into words, and choose the right analysis for each one in context rather than returning every possibility.

   Tamil surface forms are massively ambiguous in isolation and context resolves most of it. Nothing further is possible without this.

2. B

   Structure

   Parse the sentence: which word governs which.

   It lets us ask what grammatical slot comes next, rather than what token comes next.

3. C

   The realizer

   Give it a lemma and a bundle of grammatical features, get back the correct surface form. Join words with the right புணர்ச்சி.

   This needs no training at all. The transducer already runs in this direction and we simply do not use it. Highest value for the least work in this whole list.

4. D

   The validator

   Check Tamil that something else produced: is every form parseable, does agreement hold, is case government right? Return corrections with the rule that was broken.

   This is a Tamil grammar checker, and it is plausibly wanted by more people than the analyser is.

5. E

   The first predictor

   De-agglutinate a corpus into sequences of meaningful units, then train a modest model over those units to predict the next one.

   Laptop scale, no specialised hardware, weeks rather than months. It is a Tamil next-word predictor that reasons over morphemes rather than fragments.

6. F

   Tokenizer, then a model

   A tokenizer whose units are morphemes, and then a small Tamil model trained with it.

   The segmentation data that trains the tokenizer is what the server has been accumulating since the beginning.

Stage C before stage E, always. A predictor without a realizer just produces more fragmented output, which is the problem we started with.

Stage F is the one place where somebody has already published a result we would build on rather than repeat:

The cheap experiment that would settle it

Before committing to any of the expensive stages, there is a few days of work that would tell us whether the whole thesis holds: take Tamil Wikipedia, split it into morphemes with the analyser we already have, then train two small models, one over ordinary tokens and one over morphemes, and compare how well each predicts the next unit.

If morphemes win clearly, the long-term argument is proved and it is a publishable result nobody has produced. If they do not, we have saved a year. It is the highest information per hour available to this project, and it needs no funding.

Two questions we have to answer before any of this

Honest gaps do not survive generation

"No attested analysis" is a fine answer for an analyser. In the middle of a sentence it is not, because something has to be produced. Refuse the whole sentence? Emit with a confidence flag? Fall back and mark it ungrounded? Whichever we choose, choosing it late means the honesty rule erodes quietly rather than being changed on purpose.

Rule-attested is not source-attested

A realizer necessarily produces correct Tamil forms that no dictionary has ever recorded. That is fine, and it is a different kind of evidence from a word somebody attested. The rule against inventing words has to be scoped narrowly to coining a new word, or it will accidentally forbid inflecting an existing one.

When this becomes real

There is a written trigger, so the decision is not made on enthusiasm: roughly a hundred thousand verified records in the corpus, and evaluation that runs as routine rather than as an event. Until both hold, this stays where it is.

Everything on the current status page is what produces those two conditions. That is the actual relationship between the near work and the far work, and it is why the far work is not being solutioned now.

What we still need to get there →