'Cheap Chinese Models' Aren't Cheap

Motivation

I have a small side project in the works: Kanjiland. It’s a reading comprehension engine for Japanese language learners. It’s a model trained to help identify the structure of Japanese text. Raw Japanese goes in, structured grammar and annotated furigana comes out. If you’re an intermediate or advanced Japanese language learner, this model is for you™.

To train this model, I need more data than just English ⇔ Japanese translations. What I really need is the annotated grammar and furigana (i.e. the readings of the individual characters). That kind of data doesn’t really exist on the open web. So I decided to distill the training data from the big labs. In other words, I’m going to pay the big labs some amount of money and have their models produce a large dataset that I can use to train my own model.

It’s likely that I’ll need to build a training set of around 300k samples. That’s a lot cost-wise, especially for a side project that won’t make any money. So consider me a prime target for the “Cheap Chinese Models”. Thought I, “I hear the Chinese models are a fraction of OpenAI/Anthropic! I’ll be so smart to use them for this project.”

In an uncanny moment of clarity, I decided to be a little more empirical and get a data-driven sense of the cost / quality curve. So I decided to stage a “bake off” between OpenAI, Anthropic, and the Chinese Models.

The method

For this experiment, I used 150 sentences from the KFTT corpus. The idea is to give these models a prompt like so:

SYSTEM:
You annotate Japanese sentences for a reading-comprehension dataset. You are
given a sentence already segmented into morpheme TOKENS (with ids, surface,
POS, dictionary form). Do NOT re-segment. Produce ONLY a JSON object:

{
  "tokens":    [{"id": <int>, "gloss": "<contextual English meaning HERE>"}, ...],
  "words":     [{"span": [start,end], "dict": "<dictionary form>", "gloss": "<meaning>"}],
  "sentences": [{"span": [start,end], "translation": "<natural English>"}],
  "grammar":   [{"rule": "<RULE_ID>", "roles": {"<role>": <target>, ...}}]
}

Rules: gloss EVERY token exactly once (punctuation = ""); sentence spans must
tile the whole paragraph; grammar rules come from a closed 120-rule inventory
(TOPIC_WA, GENITIVE_NO, PASSIVE, COND_BA, ...) with required roles per rule.

USER:
Sentence: 正室は松平正和の娘。
Tokens:
  0: 正室  [NOUN]   dict=正室
  1: は    [PART]   dict=は
  2: 松平  [NOUN]   dict=松平
  3: 正和  [NOUN]   dict=正和
  4: の    [PART]   dict=の
  5: 娘    [NOUN]   dict=娘
  6: 。    [PUNCT]  dict=。

And then the expected output would look something like this:

{
  "tokens": [
    {"id": 0, "gloss": "legal wife / official consort"},
    {"id": 1, "gloss": "(topic marker)"},
    {"id": 2, "gloss": "Matsudaira (surname)"},
    {"id": 3, "gloss": "Masakazu (given name)"},
    {"id": 4, "gloss": "of"},
    {"id": 5, "gloss": "daughter"},
    {"id": 6, "gloss": ""}
  ],
  "words": [
    {"span": [2, 4], "dict": "松平正和", "gloss": "Matsudaira Masakazu (person)"}
  ],
  "sentences": [
    {"span": [0, 7], "translation": "His legal wife was a daughter of Matsudaira Masakazu."}
  ],
  "grammar": [
    {"rule": "TOPIC_WA",    "roles": {"topic": [0, 1], "marker": 1}},
    {"rule": "GENITIVE_NO", "roles": {"modifier": [2, 4], "marker": 4, "head": [5, 6]}}
  ]
}

And then evaluate that output for correctness. The model that has the most amount of correct outputs wins the bake off.

The results

I entered the following models into this bake off: gpt-5.6-luna, deepseek-v4-flash, qwen3.6-plus, kimi-k2.6, and Claude Opus 4.8[1]

This Experiment:

The yield and gloss-rate columns are important. They’re basically telling us: “how many extra queries do you need to send the model to reach your (300k) target”. That’s the key piece when working with cheap models – you need to hit them multiple times to get the desired results.

Total Cost (i.e the estimated total cost of generating 300k samples based on the above data)

Surprise #1: the cheap models billed like expensive ones

So were the “cheap models” cheap? No. The premise died on contact with the invoice. qwen3.6-plus and deepseek-v4-flash billed 8–13× the completion tokens luna did for the same task (qwen ~8k billed tokens per call vs luna’s ~640). Result: qwen cost 27× more per valid item than luna and ran 48× slower. My $2.50 cost projection came in at $5.68 — the overage entirely reasoning tokens. Meanwhile luna costs $1 per thousand sentences. At my scale, the entire data expansion costs ~$360 with the luna.

Surprise #2: the cheapest option was the worst teacher

DeepSeek-V4-Flash turned out to actually be cheaper (especially with its $0.03/M cached-input pricing). But it produced the dirtiest data by a wide margin: 42% yield, 36% of tokens left unglossed, and 39/150 responses truncated (even at max_tokens 8000) because it couldn’t stop deliberating.

Surprise #3: Kimi never got to compete

kimi-k2.6 hit three account-level walls: it rejects any temperature≠1, my account has a 3-concurrent / 20-RPM cap, and a 1.5M token/day ceiling — which the first (truncation-plagued) run exhausted, burning $1.12 for zero clean samples. Whatever the model’s quality, those limits disqualify it as a 300k-sentence batch teacher. A more patient person would have persevered, but I had to cut my losses here.

Surprise #4: Anthropic is really, really good

Claude Opus 4.8 (run as 10 parallel subagents in Claude Code, 15 sentences each) posted the best funnel: 92% yield, zero empty glosses, zero JSON failures, and the strongest grammar agreement with every peer. It also handled the proper-noun minefield well — 島津貞久 read correctly as “Sadahisa” where DeepSeek invented “Sadaisa.”

Takeaways

OpenAI’s luna was the clear winner in terms of performance per dollar. It was the best value by 27×, fastest, and its one systematic defect (lazy particle glosses) is fixable with one prompt line, per the Anthropic result.

Fine print

This post should be considered amateur and nobody should be making budgetary decisions based off it. I’m literally running these experiments in my garage. Second, the eval domain (KFTT/Wikipedia-history) is pretty brutal for machine translation. That’s both a blessing and a curse for my use case, but it’s pretty orthogonal to the primary use-case these models are intended to tackle.

Prices are July-2026 list; the luna cost is measured actuals, the Opus cost is a stated-assumptions estimate. Total spend for everything in this post: $5.68 of API money plus ~1.06M Claude-plan tokens.

[1] Opus was tested in-harness (via Claude Code) via my subscription account. So cost is just an estimate.