A fast number means nothing
without a fair measurement.
Our proposed benchmark protocol makes each deployment comparable. The current catalog contains sample data; automated measurements and provider verification are in development.
Measure the request
Decode speed per request, end-to-end latency, and instrumented prefill are separate metrics. Aggregate server throughput is labeled separately.
Identify the deployment
A model name is not enough. Pin the weight revision, tokenizer, quantization, adapters, serving engine, and runtime configuration.
Show the evidence
Publish sample count, test conditions, observation window, and verification level. A new provider starts with insufficient data, not an invented score.
One protocol. Reproducible conditions.
Planned default text profile. Long-context, concurrency, and warm-cache tests will have their own profiles.
| Metric | Meaning | What we will report |
|---|---|---|
| Decode speed | Output tokens generated after the first token, divided by generation time. | Per-request tok/s; p50 and p95 latency; exact output-token counting. |
| Prefill speed | Input tokens processed during the engine’s measured prefill phase. | Instrumented tok/s. Not input tokens divided by TTFT. |
| Time to first token | Client-observed time from request start to first output token. | p50 and p95 in seconds, including network and queueing; test region disclosed. |
| Context window | Total supported input and output tokens for this deployment. | Boundary tests, maximum output length, and runtime limits. |
| Availability | Successful scheduled health checks in the observation window. | Uptime, error rate, observation period and sample count. |
| Cost | Price for the measured task under the exact test settings. | Separate input/output prices, cache fees where applicable, and modality-specific units. |
Every modality has its own units.
Images
Seconds per image at p50/p95, resolution, steps, sampler, batch size, precision and adapters. Price is tied to these settings.
Audio
Transcription and speech synthesis measured separately. Realtime factor, first-chunk latency, language, audio duration, sample rate and cost per audio minute.
Video & embeddings
Video: seconds per clip with duration, resolution and fps. Embeddings: latency, input throughput, dimensions, batch size and cost per input token.
Trust is evidence, not a badge.
| Level | Evidence | Limit |
|---|---|---|
| Declared | Provider-supplied model and runtime information. | Not independently verified. |
| Manifest checked | Observed artifact hashes match a known weight manifest. | Does not prove which model served a request. |
| Benchmarked | Deployment passed the applicable performance and behavior checks. | Behavioral testing is evidence, not proof of exact weights. |
| Runtime attested · planned | Trusted measurements bind the runtime, model artifacts, and execution environment. | Depends on supported hardware and the full attestation chain. |
Technical references: NVIDIA inference metrics and confidential inference architecture.