★★★★☆ 4.4 CE

Replicate Performance: Benchmarks, Latency & Limits 2026

Replicate runs any containerized model on demand, so latency is dominated by cold boots. Warm speed is competitive; cold, time to first token is 10-27 seconds.

Large Language ModelsVerified today

Full review Pricing Official site ↗

Replicate Performance verdict

Verified today·4 sources checked

Replicate's performance profile is the opposite of a dedicated speed host.

It runs any containerized model on demand, so latency is dominated by cold boots. On the two LLMs Artificial Analysis tracks, output speed once warm is competitive, at 437 tokens a second on Granite 4.0 H Small and 389 on Granite 3.3 8B.

How to size it

Choose Replicate when breadth and ease beat guaranteed latency. It runs any public model, from LLMs to SDXL and Flux images, video and audio, or your own custom model with one API call and no infrastructure, billed per run-second. It shines for batch and background generation, and for prototyping across many models. It also fits image, video and audio work where a 10 to 30 second cold start is acceptable and you only pay for run time. For interactive, latency-sensitive LLM serving it is the wrong tool out of the box. Time to first token runs 10 to 27 seconds cold. So either keep a deployment warm, with minimum instances of 1 or more, paying for idle capacity, or use a dedicated low-latency host like Groq. Use versions for reproducibility and webhooks for long jobs.

Honest limits

Latency is dominated by cold boots. A cold model loads several GB of weights and can take seconds to minutes to start, so first-call time to first token is high, 10 to 27 seconds on tracked LLMs.
Replicate is a general model host, not an LLM speed provider. Artificial Analysis tracks only two LLMs, and Replicate's value is running any image, video, audio or custom model.
For production latency, you must keep models warm with a deployment, at minimum instances of 1 or more, which means paying for idle warm capacity.
Execution is asynchronous. Long jobs run as predictions, up to 30 minutes, that you poll or receive via webhook, not a synchronous low-latency call.

Cold-boot TTFT: 10-27 s (tracked LLMs)
Warm output speed: 389-437 t/s
Keep warm: Deployment min instances >= 1
Execution: Async predictions, 30-min cap
Runs: Any model (LLM/image/video/custom)

View sources

Latency: cold versus warm

Scenario	Time to first token / response	Notes
Granite 4.0 H Small (measured)	10.40 s TTFT	Fastest-starting LLM AA tracks on Replicate
Granite 3.3 8B (measured)	27.29 s TTFT	Higher start latency on a smaller model
Cold boot ('starting')	Seconds to several minutes	Fetches and loads several GB of weights
Warm model	Fast	Already running from recent activity
Deployment (min instances >= 1)	No cold boot	Keep a model always-on to remove cold starts
Granite 4.0 H Small end-to-end	11.54 s (500 tok)	Start latency dominates the full response

Throughput and the prediction model

Aspect	Value	Notes
Granite 4.0 H Small speed	437 t/s	Output speed once warm
Granite 3.3 8B speed	389 t/s	Smaller model, slightly slower
Execution model	Asynchronous predictions	Poll status for long-running jobs
Autoscaling	Multiple copies across machines	Scales on demand; new copies take time to ready
Prediction timeout	30 minutes	Contact Replicate for longer
Billing basis	Per second of run time	Cold-boot wait not billed to your prediction

What Replicate can run

Dimension	Value	Notes
Catalogue	Any public model	LLMs, SDXL/Flux image, video, audio, custom
Custom models	Push your own	Public or private; fine-tune or build from scratch
Versions	Pinned, reproducible	A model behaves consistently regardless of when run
Deployments	Custom hardware + scaling	Set min/max instances; keep warm
Hardware	Selectable model hardware (GPU tiers)	Pick GPU per model/deployment
Organizations	Shared models, tokens, billing, private models	Team collaboration
API rate limits	Per-account request limits	See HTTP API reference

Replicate reliability and architecture

Reliability hinges on warmth: a warm model responds fast, a cold one is slow, popular public models stay warm from activity, and less-used or custom models cold-boot more often
Cold boots are removable for production: a deployment with minimum instances set to one or more keeps a model always running and ready to respond
Every run is an explicit prediction with a lifecycle (starting, processing, succeeded, failed, canceled), so applications poll or use webhooks rather than assume a synchronous response
Outputs are ephemeral: input and output files for API predictions are automatically deleted after an hour, so you must save copies of anything you need to keep
Cost is decoupled from cold-start: Replicate only charges for the time a prediction is actually running, so a cold boot adds latency but not directly to that prediction's bill
Long jobs are first-class: predictions can run up to 30 minutes, and webhooks deliver results so you do not have to hold a connection open

Benchmarked LLM performance

Artificial Analysis tracks only two LLMs on Replicate (Granite 4.0 H Small and Granite 3.3 8B), reflecting that Replicate is a general model-hosting platform rather than a dedicated LLM speed provider
Output speed once warm is competitive: Granite 4.0 H Small reaches 437 tokens/sec and Granite 3.3 8B 389 tokens/sec, both at a 128k context
Time to first token is high (10.40s for Granite 4.0 H Small, 27.29s for Granite 3.3 8B), the visible cost of cold starts on a general-purpose host versus a dedicated provider's sub-second start
Blended prices are low for the tracked LLMs: $0.05 per 1M tokens for Granite 3.3 8B and $0.08 for Granite 4.0 H Small, a 2x spread
End-to-end a 500-token response lands in about 11.5s for Granite 4.0 H Small and 28.6s for Granite 3.3 8B, dominated by the start latency rather than generation
Figures are first-party Replicate API measurements taken live over a 72-hour window, eight times a day for single requests and twice a day for parallel requests

Replicate Performance FAQ

How fast is Replicate?

It depends on whether the model is warm or cold. Once warm, the LLMs Artificial Analysis tracks run at competitive speed, at 437 tokens a second on Granite 4.0 H Small and 389 on Granite 3.3 8B. But time to first token is high, 10.40 and 27.29 seconds respectively, because a cold model has to load several gigabytes of weights, which can take seconds to minutes. Popular models stay warm, and less-used ones cold-boot more often.

What are cold boots on Replicate and how do I avoid them?

Replicate only runs models that are being used, so an idle model is turned off and must cold-boot, reloading its weights, on the next request, which adds latency. To avoid it in production, create a deployment with minimum instances set to 1 or more, to keep the model always running and ready. You are only charged for the time a prediction is actually running, so cold-boot wait does not add to that prediction's bill.

Is Replicate fast enough for interactive LLM apps?

Out of the box, no. Cold-start time to first token runs 10 to 27 seconds on the tracked LLMs, far above a dedicated host. For interactive use you must keep a deployment warm, at minimum instances of 1 or more, paying for idle capacity, or use a dedicated low-latency provider. Replicate is better suited to batch, background and image, video and audio generation where a cold start is acceptable.

What can Replicate run, and how does it scale?

Replicate runs any public model, from LLMs to image models like SDXL and Flux, video and audio, plus models you push yourself, public or private, with pinned versions for reproducibility. It autoscales by running multiple copies across machines, and deployments let you customize hardware, across GPU tiers, and min and max instances. Execution is asynchronous predictions, up to 30 minutes, that you poll or receive via webhooks.

How is Replicate's performance measured?

The LLM figures come from Artificial Analysis's independent provider benchmarks, measured live over the past 72 hours, eight times a day for single requests and twice for parallel, on Replicate's first-party API. The cold-boot, prediction-lifecycle and deployment behavior is from Replicate's own documentation. Provider performance varies over time with infrastructure and load.

Sources & verification

Verified by ComparEdgeMethod: Vendor docs, official pages, and selected independent sources

Source	What was checked	Last checked
Replicate Official	Official product page	July 10, 2026
Artificial Analysis	Independent reference	July 10, 2026
Replicate Reference How Does Replicate Work	How Does Replicate Work	July 10, 2026
Replicate Reference Http	Http	July 10, 2026

Every fact on this Replicate page is tied to a named source and a verification date. Freshness-sensitive figures trace to the sources above; verify against the vendor before relying on them.