Naturally, a fast LLM can still be expensive to run. A cheap deployment can often feel painfully slow to its users. That is why LLM serving cannot be judged solely by GPU specifications.

Teams need to understand how quickly the system generates tokens, how many users it can serve at once, and how long each request waits. Moreover, they need to understand how much every generated token costs. 

Two metrics define this decision: tokens per second and cost per token. 

Tokens per second vs. cost per token

Metric	What it tells you	Why it matters
Tokens per second	How quickly the system generates output	Affects response speed and throughput
Cost per token	How much compute spending produces each token	Helps measure inference efficiency
Time to first token	How long users wait before output begins	Affects perceived responsiveness
Concurrent requests	How many requests can run together	Important for production scale
GPU utilization	How much available GPU capacity is being used	Shows infrastructure efficiency

If you look at only one of these numbers, it may create the wrong impression or expectation for you. 

A GPU may generate tokens very quickly for a single request but become inefficient at scale. Another configuration may produce slightly fewer tokens per second but handle many more users at a lower overall cost.

What does tokens per second mean in LLM serving? 

Objectively, tokens per second measures how swiftly a model generates output. LLMs work with tokens rather than complete words. A token may represent a word (or just a part of it), punctuation, or another piece of text entirely.

If a model generates 100 tokens per second, it can produce 100 output tokens during one second under that test configuration. Higher throughput can make applications feel faster.

This is crucial for things like:

• AI chatbots
• Coding assistants
• AI agents
• Search assistants
• Document generation
• Customer support systems

The number also needs some context. You see, model size, batch size, prompt length, output length, precision, and GPU configuration all affect the result.

Time to first token matters just as much

Users experience “waiting” rather than benchmark numbers. Time to first token measures the delay between submitting a prompt and receiving the first token of the response. Now, consider two systems:

System A: starts responding almost immediately, then generates at a moderate rate.

System B: waits several seconds before generating output at a higher token rate.

System B may win in a raw tokens-per-second sense while System A feels much faster to the user. 

Interactive applications need to track both time to first token and output tokens per second. This separates initial response delay from generation speed.

Prefill and decode create different workloads for the GPU

LLM inference has two distinct stages: 

• Prefill processes the input prompt. A long document or large conversation history creates more work during this stage. And 
• decode generates the output one token at a time. 

These stages place different demands on your GPU. A document analysis tool with quite a long prompt may spend significant compute on prefill. A chatbot generating long responses may place more emphasis on decode performance. 

This is why a single tokens-per-second number cannot truly capture every LLM workload. Thus, teams should benchmark realistic prompt and response lengths. 

Why does batch size change tokens per second

Serving one request at a time can leave GPU capacity unused. To address this problem, batching allows the system to process multiple requests simultaneously. This can improve overall throughput because more of the GPU performs useful work at once.

Yet, there is a trade-off you must keep in mind.

Larger batches can increase the total tokens processed per second while also causing individual requests to wait longer. Teams then need to decide whether the application values these things:

• Low latency: Often useful for interactive chat, coding tools, and live assistants.
• High throughput: Useful for batch document processing, offline generation, and large request queues. 

Production serving often needs a balance between the two. 

Cost per token shows infrastructure efficiency. 

Tokens per second tells teams how quickly the system works. Cost per token shows you how efficiently money is being converted into model output. A simple way to think about this is to consider the cost per token as the infrastructure cost during the serving period plus the total tokens generated. 

Your team can also calculate cost per million tokens to make the number easier to compare across deployments. 

This allows two GPU configurations to be evaluated on the actual output. This lets two GPU configurations be evaluated on actual output. For instance, a more expensive GPU may still deliver a lower cost per token if it processes substantially more requests during the same period. 

The hourly GPU price alone therefore gives an incomplete picture. 

GPU utilization can change the economics a little

Often, an expensive GPU sitting mostly idle can produce a poor cost-per-token result. Something we should avoid! Utilization measures how much of the available GPU processing capacity the workload truly uses.

Low utilization frequently happens when: 

• Request traffic is lower than expected
• Batch sizes are very small
• CPU processing delays the GPU
• Data movement becomes a bottleneck
• Models are poorly configured
• Too much GPU capacity is provisioned

Higher utilization enables more useful work to be done during each GPU hour you pay for. Thus, infrastructure sizing should be based on actual traffic.

Where the H200 fits into LLM serving

The NVIDIA H200 is designed for demanding AI workloads and provides a massive HBM3e memory pool with a high memory bandwidth. That’s something valuable for LLMs since inference regularly moves model weights and cached data through GPU memory. 

Teams evaluating the H200 GPU price should thus compare the hourly infrastructure cost with the throughput the GPU produces for their specific model. Ask yourself, “how many useful tokens can this GPU produce for the money I spent?” 

This creates a much more practical comparison than purchase price or hourly rates. 

If a team wants to test this class of hardware through cloud infrastructure, they can also evaluate an H200 GPU cloud before planning larger production capacity. 

Model size can change the serving equation.

In perspective, a 7-billion-parameter model and a much larger language model create very different infrastructure requirements. Larger models generally require more GPU memory and may require several GPUs when the full model no longer fits on a single device. 

This may affect both metrics: tokens per second may fall as more computation is required per generated token. Cost per token may rise as more GPU capacity becomes necessary.

It’s ideal for teams to compare hardware using the exact model planned for production. A GPU benchmark using another model may provide limited guidance for the actual deployment process.

Picking the right metric for the workload 

Different AI products should prioritize different measurements. Take a look at which fits your context:

• Interactive chatbot: Focus on time to first token and generation speed.
• Coding assistant: Prioritize low latency and consistent token generation.
• Batch document processing: Focus on total throughput and cost per token.
• High-volume API: Track concurrency, throughput, latency, and cost together.
• Internal AI tool: Look at utilization so oversized infrastructure does not sit idle.

This decision should follow the user experience the application needs to deliver. 

Conclusion 

In a nutshell, tokens per second tells teams how fast an LLM serving system can generate output. Cost per token indicates how efficiently that output is produced. So, neither metric is enough on its own. 

The best GPU is therefore not always the one with the lowest hourly price or the highest benchmark result. Context matters! It is the configuration that delivers the required user experience at a sustainable cost per token. 

Frequently asked questions

What does tokens per second entail for an LLM?

Tokens per second measures how quickly an LLM generates output tokens during inference.

What is cost per token?

Cost per token measures the amount of infrastructure spending required to generate each token. Teams often convert this into cost per million tokens for easier comparison. 

Is higher tokens per second always better?

Higher generation speed is useful, but production systems also need acceptable latency, concurrency, utilization, and cost. 

Which metric matters most for LLM serving?

This depends on the application. Interactive tools usually prioritize latency, while large-batch workloads may place greater importance on throughput and cost per token.