★★★★★ 4.5 CE

Groq Performance: Benchmarks, Latency & Limits 2026

Groq is an inference host built for speed. Independent benchmarks rank it first or near-first on output speed for open models, from ~300 to over 900 tokens a second.

Large Language ModelsVerified today

Full review Pricing Official site ↗

Groq Performance verdict

Verified today·4 sources checked

Groq is an inference host whose whole proposition is speed, and Artificial Analysis's independent live benchmarks back it.

It ranks first or near-first among providers on output speed for the open models it serves, from 297 tokens a second on Llama 3.3 70B to over 900 on small gpt-oss models. Time to first token is around 0.73 to 1.04 seconds.

How to size it

Choose Groq when latency and throughput on open models matter more than picking a proprietary frontier model. It is independently the fastest or near-fastest provider for Llama and gpt-oss class models, at 721 tokens a second on Llama 3.1 8B, more than double the next host. Time to first token runs sub-second to about 1 second, with blended prices from $0.05 per 1M. It is ideal for interactive chat, high-throughput pipelines and agentic loops where speed compounds, and it drops in behind the OpenAI SDK. Match the model to the job, small gpt-oss for raw speed or 70B Llama for quality, and use Flex or Batch tiers to cut cost on deferrable work. Look elsewhere if you need a proprietary frontier model, very long context beyond 131k, or guaranteed fixed throughput rather than live-measured speed.

Honest limits

Groq is a host, not a model. Its intelligence is whatever open model you pick, so judge quality by the model and speed and price by Groq.
Output speed varies up to 97% across Groq's own models, so the headline 900-plus tokens a second is the small gpt-oss models, not the 70B-class ones at around 300.
Figures are live 72-hour first-party API measurements and can shift with Groq's infrastructure and load, so check the over-time charts, not one snapshot.
The catalog is open-weight models only, 11 tracked, at a uniform 131k context. It is not a home for proprietary frontier models.

Top output speed: 938 t/s (gpt-oss-20B)
Lowest TTFT: 0.73 s
vs other hosts: #1 on Llama 3.1 8B (721 t/s)
Models: 11 open-weight, 131k
Blended price: $0.05-$0.84 / 1M

View sources

Time to first token by model

Model on Groq	TTFT	Notes
gpt-oss-120b (low)	0.73 s	Lowest time to first token on Groq
gpt-oss-120b (high)	0.74 s	Reasoning effort high
gpt-oss-20B (high)	0.78 s	Smallest, fast start
Qwen3 32B	0.86 s	Reasoning model
Llama 3.3 70B	1.02 s	70B-class, ~1s start
Llama 3.1 8B	1.04 s	Small, broadly served

Output speed by model

Model on Groq	Output speed	Notes
gpt-oss-20B (low)	938 t/s	Fastest model on Groq
gpt-oss-20B (high)	905 t/s	Reasoning effort high
Llama 3.1 8B	652 t/s	Small dense model
gpt-oss-120b (high)	477 t/s	120B-class at high effort
Llama 4 Scout	450 t/s	Multimodal MoE
Llama 3.3 70B	297 t/s	70B dense, still ~300 t/s

What Groq serves

Dimension	Value	Notes
Models offered	11 (all open-weight)	Llama, gpt-oss, Qwen families
Context window	131k tokens (all models)	Uniform across the catalog
Function calling	All 11 models	Tool use supported catalog-wide
JSON mode	All 11 models	Structured output catalog-wide
Reasoning models	6 of 11	Extended thinking variants
Processing tiers	On-demand, Flex, Batch	Trade latency for cost or scale

Groq reliability and architecture

Groq's value is hardware-accelerated inference: it delivers the field's top output speeds, over 900 tokens/sec on small models, well above the cross-provider median, with speed varying up to 97% across its own model sizes
Performance is independently and continuously measured, not self-reported: Artificial Analysis benchmarks are live over a 72-hour window, taken eight times a day for single requests and twice a day for parallel requests
The API is OpenAI-compatible, so Groq can be dropped in behind existing OpenAI SDK integrations as a faster, cheaper backend with minimal code change
Groq exposes processing tiers (on-demand, Flex Processing for discounted deferred work, and Batch Processing) plus a Performance Tier, so reliability and cost can be tuned per workload
It ships production-readiness tooling: a Production Checklist, latency-optimization guidance, Prometheus metrics and spend limits in the GroqCloud console
Provider performance can vary over time with infrastructure changes and load, which Artificial Analysis tracks in historical 'over time' charts rather than a single snapshot

Benchmarked against other hosts

For Llama 3.1 8B, Groq is the fastest of all tracked providers at 721.3 tokens/sec, more than double FriendliAI (325.3) and over three times Azure (211.7)
For Llama 3.3 70B, Groq leads the providers at 316.1 tokens/sec, ahead of SambaNova (301.6) and Makora FP8 (292.7)
Its fastest model overall is gpt-oss-20B at 938 tokens/sec, and end-to-end a full 500-token Llama 3.1 8B response completes in about 1.81 seconds
Speed does not cost a premium: blended prices run from $0.05 per 1M tokens (Llama 3.1 8B) to $0.84 (Qwen3.6 27B), a 16x spread, with the fastest small models among the cheapest
The benchmarked figures are first-party Groq API measurements, so they reflect what an application actually gets from GroqCloud rather than a lab number
Methodology is independent and ongoing: Artificial Analysis measures live over the past 72 hours, eight times a day for single requests and twice a day for parallel requests

Groq Performance FAQ

How fast is Groq compared to other providers?

On Artificial Analysis's independent live benchmarks, Groq is the fastest or near-fastest host for the open models it serves. For Llama 3.1 8B it runs at 721.3 tokens a second, more than double the next provider, FriendliAI at 325, and over three times Azure at 211. For Llama 3.3 70B it leads at 316.1 tokens a second, ahead of SambaNova and Makora. Its fastest model overall is gpt-oss-20B at 938 tokens a second.

What is Groq's latency, or time to first token?

Low. The gpt-oss-120b models answer in about 0.73 to 0.74 seconds to first token, and gpt-oss-20B in 0.78. Even 70B-class models like Llama 3.3 70B start in about 1.02, with Llama 3.1 8B at 1.04. Combined with high output speed, a full 500-token Llama 3.1 8B response completes end-to-end in roughly 1.81 seconds.

Which models does Groq run, and how fast are the big ones?

Groq tracks 11 open-weight models across the Llama, gpt-oss and Qwen families, all at a 131k context with function calling and JSON mode. Speed varies up to 97% by size. Small gpt-oss models exceed 900 tokens a second, while a 70B model like Llama 3.3 70B runs around 297, still fast for its class. Six of the models are reasoning models.

Is Groq reliable, and how is its speed measured?

Its performance is measured independently and continuously, not self-reported. Artificial Analysis benchmarks live over the past 72 hours, eight times a day for single requests and twice for parallel requests, and tracks change over time. Groq also exposes processing tiers of on-demand, Flex and Batch, a production checklist, latency-optimization guidance and Prometheus metrics for production reliability.

Does Groq's speed cost more?

No. Blended prices run from $0.05 per 1M tokens for Llama 3.1 8B up to $0.84 for Qwen3.6 27B, a 16x spread, and the fastest small models are among the cheapest. Groq pairs top-of-field speed with low, open-model pricing, and offers Flex and Batch tiers to discount deferrable or bulk workloads further.

Sources & verification

Verified by ComparEdgeMethod: Vendor docs, official pages, and selected independent sources

Source	What was checked	Last checked
Groq Official	Official product page	July 10, 2026
Artificial Analysis	Independent reference	July 10, 2026
Groq Openai	Openai	July 10, 2026
Groq Quickstart	Quickstart	July 10, 2026

Every fact on this Groq page is tied to a named source and a verification date. Freshness-sensitive figures trace to the sources above; verify against the vendor before relying on them.