Fast is a feeling. Measure the parts behind it.
A reproducible evaluation of dictation latency, transcription quality, recovery, memory, and battery use without invented benchmark results.
Separate response time from transcription time
A recording control should respond immediately, even when a model needs time to load. Measure shortcut-to-indicator latency separately from release-to-text latency. Otherwise a fast animation can conceal a slow transcription pipeline.
Record the device, operating system, model revision, language, clip length, and power mode. Separate cold starts from an already-loaded model. Report a distribution, including slower requests, rather than publishing the fastest attempt. This article describes the evaluation method; it does not report comparative performance results.
Use material that resembles real work
Include quiet and noisy speech, proper names, numbers, corrections, and the languages your team actually uses. Use consented or synthetic samples with a reference transcript. Count meaning-changing errors separately from harmless punctuation differences.
For meetings, test a long session as well as a short clip. Watch queue growth and memory over time. Pause, resume, disconnect, and interrupt the process. Verify the saved record after restarting; a successful stop animation alone does not establish that the recording is durable.
Measure the quiet moments too
Profile idle, recording-only, active transcription, and background states separately. A graphics scene or polling loop that stays busy after the user stops working can matter as much as the model itself.
Compare repeated runs under the same conditions and retain the raw measurements. Investigate energy regressions alongside quality and latency. A smaller model that consumes less power may also miss critical words; choosing a default means evaluating those tradeoffs together, on declared supported hardware.