Compute & Cost Transparency

We run an open-weight LLM benchmark for structured clinical-data extraction from de-identified physician notes — an academic study with a manuscript in preparation. This page is a transparent account of the compute the study consumes, the cost drivers, and the controls we run it under. It is written for compute partners and research sponsors.

No patient-identifying information appears anywhere in this benchmark or on this page; the corpus is de-identified, and institutional details are omitted by design.

The benchmark, in one paragraph

A factorial benchmark measures how completely and accurately different models extract a fixed set of structured clinical fields from a corpus of 110 de-identified notes, under three extraction methods (“harnesses”). It runs entirely on open-weight models served by Nebius Token Factory.

Harness	Description	Model calls per note
plain	A single structured-extraction call.	1
agent	A multi-step clinical agent (planning → retrieval tools → reply) whose answer is then structured.	several
grader	A specialized extraction/completeness agent.	several

Full design: 4 models × 3 harnesses × 110 notes = 1,320 extraction cells. This is standard model-evaluation research — no production traffic, no bulk generation for redistribution. The output is a comparative analysis of open models for a healthcare NLP task, which the resulting publication will cite.

Models under test

Four models from the Nebius Token Factory catalog:

Model	Role
openai/gpt-oss-120b	Baseline
moonshotai/Kimi-K3	Reasoning model under test
zai-org/GLM-5.2	Reasoning model under test
MiniMaxAI/MiniMax-M3	Efficient model under test

Compute footprint to date

Measured spend on the completed and partial passes (authoritative figures from the provider’s usage dashboard, 09 Jul – 08 Aug 2026):

Model	Input (1M tokens)	Output (1M tokens)	Cost
gpt-oss-120b	5.06	2.49	$2.26
Kimi-K3	1.63	1.34	$25.06
GLM-5.2	12.54	4.93	$39.24
MiniMax-M3	0.93	0.31	$0.65
Total			$67.21

What drives the cost

1. Reasoning-heavy models dominate. GLM-5.2 ($39.24) and Kimi-K3 ($25.06) together are ~95% of the spend, almost entirely in output tokens (large chain-of-thought / reasoning streams by design). The cheaper baseline and the efficient model were negligible by comparison. This is itself a useful benchmark finding: reasoning-model output volume, not input size, is the real cost of an evaluation like this.
2. Multi-step harnesses make several calls per note. The agent and grader methods run a planning → tools → reply → structuring chain, so a single “cell” is multiple billable calls.
3. Infrastructure bring-up. Early concurrency tuning against rate limits produced retried attempts before we established the safe ceiling — now fixed (see controls below).

To complete the benchmark

Remaining: a clean pass of the three non-baseline models across all three harnesses (~990 cells). Because the reasoning models’ measured per-cell cost is high, completing this is estimated at ~$150–250, dominated by GLM-5.2 and Kimi-K3. We meter the true per-cell cost on a small batch first and run the remainder under strict per-wave budgets.

How we control spend

• Per-wave human checkpoints — the benchmark runs one harness at a time, and results are reviewed before spending on the next wave. Credit is never spent without a result to show for it.
• Metered pre-flight — a small batch establishes the true per-cell cost before any full run.
• Concurrency tuned to rate limits — the safe concurrency ceiling is measured, so the API is never over-driven into retries.

Team

• Dr. Erick Zamora Tehozol — Rheumatology & clinical validation
• Ing. Ángel Meléndez Córdoba — Engineering

For sponsors and compute partners

This benchmark is citable academic use of an open-model catalog for a healthcare-NLP publication. If you host or fund open-weight inference and want your models represented in the comparison — or want to sponsor the compute that completes it — the study is a transparent, well-instrumented, and publication-bound use of that compute. Reach out through the contacts on poktacare.com.