Local inference lab · LM Studio

Benchmark the models running on your machine.

A live harness for LM Studio’s local server. Load a few models, pin a set of workloads, and get measured throughput, latency, and quality — ranked on the hardware in front of you. Nothing is uploaded, nothing leaves localhost.

no cloud  ·  no uploads  ·  localhost:1234

benchmark archive — on this machine
benchmark archive · 0 models

No archived runs yet — run a benchmark and the models will be kept here.

3 workload classes code · reasoning · prose
4 metrics per run tokens · time · tok/s · quality
0 bytes uploaded localhost only
1 machine involved the one in front of you

Inside the harness

Built to measure what your hardware actually delivers.

Six design choices keep the numbers honest. No synthetic scoring, no remote backend, no calibration to look good.

Direct connection

The page talks to the LM Studio HTTP server on this machine. Loaded models are discovered through the standard OpenAI-compatible endpoint. There is no relay, no tunnel, and no account.

GET /v1/models

Deliberate defaults

Sampling is pinned so models are compared under the same conditions: temperature 0.2, top-p 0.9, and no system prompt unless the workload requires one. Differences come from the model, not the settings.

temp 0.2 · top_p 0.9

Token accounting

Throughput uses the exact completion-token count divided by decode time only — prompt processing (TTFT) is measured separately.

decode tok/s · TTFT split out

Three workload classes

Code generation, constrained reasoning, and long-form writing. Each stresses a different bottleneck: decode speed, planning depth, and context discipline.

code · reason · prose

Ranks, not ratings

Models are ordered by measured tokens per second. Quality is reported as a badge alongside, so a fast model that misses the task is easy to spot.

sort: mean tok/s

History stays on this machine

Completed runs are kept in the browser’s local storage. Reload the page, swap a quant, load a new model, and compare against the last run without a spreadsheet.

localStorage · archived on finish

The live run

Choose models, choose workloads, measure what’s loaded.

Requires the LM Studio local server to be running (Settings → Server → Start Server). Models already loaded in LM Studio appear below.

Checking server…

live throughput idle — waiting for a run
0.0 tok/s
instant rolling 2s tokens estimated during the stream, exact at completion
run log idle

Methodology

The short version of how the numbers are made.

Five questions, answered plainly. If a number looks wrong, start here before blaming the model.

How is throughput measured?

Tokens per second is the completion-token count divided by wall-clock time, timed from the request leaving this page to the full response arriving. When a workload runs more than one repeat, the cells are averaged before ranking.

Why do the numbers move between runs?

Local inference is noisy. Thermal throttling, memory-bandwidth contention, and background processes all shift decode speed. Run several repeats and compare averages, not single spikes.

How is quality scored?

Each workload ships with a lightweight rubric checked against the response text: code fences present, the numeric answer extracted, a word-count band, or valid JSON. It is a coarse gate, not a judge.

What about context length?

Context is reported next to each model when the server exposes it. The harness never pads the prompt; the longest workload stays well under 2k tokens so decode speed remains the story.

Why are sampling defaults pinned?

Temperature and top-p are locked per workload so differences in output reflect the model. Change them if you like — the run header records exactly what was used.

Archive

Previous runs, kept on this machine.

Every finished run is written to local storage. Compare a new quant against last week without copying anything out.