Whenever you optimize the inference efficiency of an LLM, that you must know find out how to measure it. With out measurement, it’s simple to make a mannequin extra difficult with out making it quicker, or to enhance throughput whereas making user-visible latency worse.
An LLM service has a number of sorts of efficiency. A person cares about how lengthy it takes to see the primary token and the way shortly the remainder of the reply streams. An operator cares about what number of requests the {hardware} can serve, how a lot reminiscence is used, and the way a lot every generated token prices. A researcher could care about whether or not an optimization modifications the mannequin’s output high quality.
On this chapter, you’ll study:
- Latency and throughput metrics
- Time to first token and time per output token
- Measuring CPU and GPU inference
- Utilizing CUDA occasions
- Benchmarking a number of requests
- Serious about a number of GPUs and a number of machines
Let’s get began.
Â
Measuring Efficiency of Transformer Inference
Picture by Tomas Anton Escobar. Some rights reserved.
Overview
This chapter is split into eight components; they’re:
- Metrics for LLM Inference
- Measuring a Single Request
- Warmup and Synchronization
- Measuring GPU Work with CUDA Occasions
- Measuring Reminiscence Utilization
- Measuring Concurrent Requests
- A number of GPUs and A number of Machines
- Price per Token
Metrics for LLM Inference
The most typical inference metrics are:
- Latency:Â How lengthy a request takes from begin to end.
- Time to first token (TTFT): How lengthy the person waits earlier than the primary output token seems.
- Time per output token (TPOT):Â The common time between generated tokens after the primary token.
- Throughput: What number of tokens or requests are processed per second.
- Reminiscence utilization:Â How a lot CPU reminiscence or GPU reminiscence is used.
- Utilization: How busy the accelerator is throughout the benchmark.
- Price per token:Â The {hardware} or service value divided by the variety of tokens processed.
For LLMs, a single latency quantity is often not sufficient. Contemplate two requests:
- Request A: 2,000 immediate tokens and 20 output tokens
- Request B: 20 immediate tokens and a pair of,000 output tokens
Request A stresses prefill. Request B stresses decode. They might have the identical complete variety of tokens, however they’ve totally different efficiency profiles. Because of this it is best to file immediate tokens and output tokens individually.
Tail latency additionally issues. If most requests full in a single second however a number of take ten seconds, customers will discover. Report high-percentile latencies resembling p90, p95, and p99 along with the imply or median. The excessive percentiles describe the worst circumstances higher. You may simply discover these percentiles from an inventory of values utilizing NumPy:
|
import numpy as np  def summarize(values):     values = np.asarray(values, dtype=np.float64)     return {         “imply”: values.imply(),         “median”: np.percentile(values, 50),         “p90”: np.percentile(values, 90),         “p95”: np.percentile(values, 95),         “p99”: np.percentile(values, 99),     } |
These numbers are easy, however they forestall a typical mistake: optimizing the typical whereas making the worst circumstances slower.
Measuring a Single Request
The best measurement makes use of time.perf_counter(). It’s a built-in high-resolution wall-clock timer appropriate for measuring elapsed time in Python. It’s extra correct than time.time().
The next instance measures prefill and decode individually for a Hugging Face causal language mannequin:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
import time import torch from transformers import AutoModelForCausalLM, AutoTokenizer   def load_model(model_name=“sshleifer/tiny-gpt2”, gadget=“cpu”):     tokenizer = AutoTokenizer.from_pretrained(model_name)     mannequin = AutoModelForCausalLM.from_pretrained(model_name).to(gadget)     mannequin.eval()     return tokenizer, mannequin   @torch.no_grad() def measure_one_request(mannequin, tokenizer, immediate, max_new_tokens=50, gadget=“cpu”):     input_ids = tokenizer(immediate, return_tensors=“pt”).input_ids.to(gadget)      begin = time.perf_counter()     outputs = mannequin(input_ids, use_cache=True)     prefill_end = time.perf_counter()      past_key_values = outputs.past_key_values     next_token = outputs.logits[:, –1, :].argmax(dim=–1, keepdim=True)     generated = [next_token]      decode_times = []      for _ in vary(max_new_tokens – 1):         step_start = time.perf_counter()         outputs = mannequin(             next_token,             past_key_values=past_key_values,             use_cache=True,         )         # Be aware: Chances are you’ll want torch.cuda.synchronize() right here         step_end = time.perf_counter()          decode_times.append(step_end – step_start)         past_key_values = outputs.past_key_values         next_token = outputs.logits[:, –1, :].argmax(dim=–1, keepdim=True)         generated.append(next_token)          if tokenizer.eos_token_id is not None:             if next_token.merchandise() == tokenizer.eos_token_id:                 break      finish = time.perf_counter()     output_ids = torch.cat([input_ids] + generated, dim=1)      return {         “textual content”: tokenizer.decode(output_ids[0], skip_special_tokens=True),         “prompt_tokens”: input_ids.measurement(1),         “output_tokens”: len(generated),         “prefill_seconds”: prefill_end – begin,         “decode_seconds”: sum(decode_times),         “total_seconds”: finish – begin,         “ttft_seconds”: prefill_end – begin,         “seconds_per_output_token”: (             sum(decode_times) / max(1, len(decode_times))         ),     } |
This operate doesn’t use the mannequin’s generate() technique. That’s intentional. The objective is to show prefill and decode to allow them to be measured individually. The variety of output_tokens contains the tokens generated by each prefill and decode. The seconds_per_output_token is the typical time per output token within the decode section.
There are two particulars to note:
use_cache=Trueasks the mannequin to return the KV cache.- Throughout decode, the mannequin receives solely
next_token, not the entire sequence.
This is identical thought as Chapter 1, however utilizing a library mannequin.
Warmup and Synchronization
Whenever you measure efficiency, notice that some one-time prices mustn’t dominate the end result. In Python, import of a module could be sluggish however subsequent import of the identical module is immediate. Equally, the primary execution of some code could also be slower than subsequent executions because of initialization of knowledge buildings or warmup of caches. You need to measure steady-state work, not that setup overhead.
Due to this fact, benchmarks ought to embody warmup. The primary few iterations could also be slower for varied causes. As an alternative of measuring the whole time and dividing by the variety of iterations, it is best to measure the time for every iteration and analyze the steady-state ones. For instance, should you use the mannequin to generate a number of tokens, you’ll probably put the era in a loop. Measure every iteration as follows, then ignore the primary few outcomes:
|
def iterations(mannequin, tokenizer, immediate, gadget, steps=100, warmup=10): Â Â Â Â outcomes = [] Â Â Â Â for _ in vary(steps): Â Â Â Â Â Â Â Â end result = measure_one_request( Â Â Â Â Â Â Â Â Â Â Â Â mannequin, Â Â Â Â Â Â Â Â Â Â Â Â tokenizer, Â Â Â Â Â Â Â Â Â Â Â Â immediate, Â Â Â Â Â Â Â Â Â Â Â Â max_new_tokens=8, Â Â Â Â Â Â Â Â Â Â Â Â gadget=gadget, Â Â Â Â Â Â Â Â ) Â Â Â Â Â Â Â Â outcomes.append(end result) Â Â Â Â regular = outcomes[warmup:] Â Â Â Â return summarize([item[“total_seconds”] for merchandise in regular]) |
If you happen to use GPU to run your LLM inference, you additionally must initialize the kernels if you first run them. Sadly, many GPU operations are asynchronous. That’s, when you launched an operation on GPU, Python could proceed along with your code instantly whereas the GPU continues to be working. Due to this fact, a naive strategy to measure the time could be incorrect. As an alternative, it is best to use torch.cuda.synchronize() to attend for the GPU to complete the operation earlier than you cease the timer:
|
def sync_if_needed(gadget): Â Â Â Â if gadget.startswith(“cuda”): Â Â Â Â Â Â Â Â torch.cuda.synchronize() Â begin = time.perf_counter() outputs = mannequin(input_ids, use_cache=True) sync_if_needed(gadget) elapsed = time.perf_counter() – begin |
This provides a wall-clock measurement that features the precise GPU work. For correct prefill and per-token decode timings on GPU, name sync_if_needed(gadget) after every timed mannequin(...) name in measure_one_request(), not solely as soon as on the finish of the request.
Measuring GPU Work with CUDA Occasions
CUDA occasions measure elapsed time the GPU spent executing kernels, not the end-to-end person latency. This time doesn’t embody any Python overhead. Beneath is an instance of find out how to use CUDA occasions to measure the time:
|
def cuda_event_time(fn): Â Â Â Â begin = torch.cuda.Occasion(enable_timing=True) Â Â Â Â finish = torch.cuda.Occasion(enable_timing=True) Â Â Â Â Â begin.file() Â Â Â Â end result = fn() Â Â Â Â finish.file() Â Â Â Â Â torch.cuda.synchronize() Â Â Â Â milliseconds = begin.elapsed_time(finish) Â Â Â Â return end result, milliseconds / 1000.0 |
You should use it to measure one ahead go:
|
with torch.no_grad(): Â Â Â Â outputs, seconds = cuda_event_time( Â Â Â Â Â Â Â Â lambda: mannequin(input_ids, use_cache=True) Â Â Â Â ) Â print(f“GPU ahead time: {seconds:.6f} seconds”) |
CUDA occasion timing and wall-clock timing reply totally different questions:
- Wall-clock timing measures what the applying experiences.
- CUDA occasion timing measures how lengthy the GPU work took.
For an inference service, wall-clock timing is often the first metric as a result of customers expertise queues, tokenization, scheduling, community overhead, and streaming. CUDA occasions are helpful if you end up optimizing kernels or evaluating mannequin execution paths.
For deeper GPU profiling, use instruments resembling PyTorch Profiler, Nsight Techniques, Nsight Compute, or CUPTI-based monitoring. These instruments can report kernel timelines, reminiscence copies, GPU utilization, and operator-level breakdowns. They’re extra advanced than a timer, however they’re vital when a easy benchmark says the mannequin is sluggish and that you must know why.
Measuring Reminiscence Utilization
Reminiscence is a unique dimension to measure as a result of it limits not velocity for one person a lot as what number of customers your system can serve. Normally the GPU reminiscence is the bottleneck. In PyTorch, you may report allotted and reserved reminiscence like the next:
|
def gpu_memory_summary(gadget=“cuda”): Â Â Â Â torch.cuda.synchronize() Â Â Â Â return { Â Â Â Â Â Â Â Â “allocated_gb”: torch.cuda.memory_allocated(gadget) / 1e9, Â Â Â Â Â Â Â Â “reserved_gb”: torch.cuda.memory_reserved(gadget) / 1e9, Â Â Â Â Â Â Â Â “max_allocated_gb”: torch.cuda.max_memory_allocated(gadget) / 1e9, Â Â Â Â } |
The allotted worth is reminiscence utilized by tensors. The reserved worth is reminiscence held by PyTorch’s caching allocator. The utmost allotted worth is commonly probably the most helpful quantity for capability planning.
The allotted and reserved reminiscence are real-time snapshots however the max allotted worth is a peak over time. For correct measurement, it is best to reset the height statistic earlier than a benchmark:
|
torch.cuda.reset_peak_memory_stats() end result = measure_one_request(mannequin, tokenizer, immediate, gadget=“cuda”) reminiscence = gpu_memory_summary(“cuda”) print(reminiscence) |
Reminiscence ought to be measured along with tokens. A run with an extended immediate or extra generated tokens will naturally use extra KV cache reminiscence.
Measuring Concurrent Requests
To create a server that runs a language mannequin, consider the system by what number of requests you may serve per second. Throughput is dependent upon each how briskly you fulfill one request and what number of requests you may run concurrently, although concurrency doesn’t scale linearly underneath rivalry.
Manufacturing methods ought to deal with a number of customers, and the scheduler could batch their work collectively. The next easy benchmark runs a number of requests concurrently utilizing Python threads. This doesn’t implement steady batching. It solely measures how a mannequin wrapper behaves when a number of callers use it on the identical time.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
from concurrent.futures import ThreadPoolExecutor, as_completed  def run_prompt(mannequin, tokenizer, immediate, gadget):     begin = time.perf_counter()     end result = measure_one_request(         mannequin,         tokenizer,         immediate,         max_new_tokens=32,         gadget=gadget,     )     finish = time.perf_counter()     end result[“wall_seconds”] = finish – begin     return end result   def benchmark_concurrent(mannequin, tokenizer, prompts, gadget=“cpu”, employees=4):     outcomes = []     begin = time.perf_counter()      with ThreadPoolExecutor(max_workers=employees) as pool:         futures = [             pool.submit(run_prompt, model, tokenizer, prompt, device)             for prompt in prompts         ]         for future in as_completed(futures):             outcomes.append(future.end result())      finish = time.perf_counter()     total_output_tokens = sum(merchandise[“output_tokens”] for merchandise in outcomes)      return {         “requests”: len(outcomes),         “total_seconds”: finish – begin,         “output_tokens”: total_output_tokens,         “output_tokens_per_second”: total_output_tokens / (finish – begin),         “latency_summary”: summarize([item[“wall_seconds”] for merchandise in outcomes]),     } |
This benchmark is for illustration solely. It’s not a alternative for an actual serving benchmark. It doesn’t mannequin HTTP overhead, streaming, request queues, batching, cancellation, or cache eviction. However it’s a helpful subsequent step after a single-request benchmark the place you may run the mannequin in parallel and observe the per-request latency. The Python GIL (World Interpreter Lock) is often not the primary concern right here as a result of heavyweight mannequin execution is commonly offloaded to compiled code. Concurrent use of the identical mannequin or tensors from a number of threads is unsafe with out synchronization, so share one mannequin on CUDA solely with a lock or a single employee thread.
When benchmarking an actual server, file no less than:
- Variety of concurrent customers
- Immediate token distribution
- Output token distribution
- Request fee
- TTFT (Time to first token) percentiles
- Inter-token latency percentiles
- Complete tokens per second
- Error fee and timeout fee
The distributions matter. A benchmark with all prompts at precisely 128 tokens and all outputs at precisely 128 tokens is straightforward to check, however it might not signify your software.
A number of GPUs and A number of Machines
A number of GPUs can be utilized for inference in a number of alternative ways. Completely different approaches can drastically change the efficiency of your system.
The best strategy is replication. You load one copy of the mannequin on every GPU and route totally different requests to totally different replicas. This will increase throughput and is straightforward to motive about, however every GPU should have sufficient reminiscence for the total mannequin and its KV cache.
One other strategy is to separate one mannequin throughout a number of GPUs. Tensor parallelism splits weight matrices throughout units. Pipeline parallelism locations totally different layers on totally different units. Context parallelism partitions sequence work. Knowledgeable parallelism is used for mixture-of-experts fashions. These methods permit bigger fashions to run, however they introduce communication overhead and might improve latency.
A number of machines add one other layer. A system could use many replicas throughout machines for prime request quantity. It might additionally cut up a single massive mannequin throughout machines, however that is tougher as a result of community communication is slower than communication inside one machine. For low-latency serving, crossing machine boundaries inside one ahead go ought to be handled as costly.
Measuring efficiency of a system with a number of GPUs or a number of machines provides a brand new dimension of communication and synchronization overhead. Earlier than selecting a multi-GPU or multi-machine design, reply these questions:
- Are you serving one massive mannequin or many smaller fashions?
- Are you restricted by mannequin weight reminiscence or KV cache reminiscence?
- Do you want decrease latency, increased throughput, or each?
- Are requests impartial, or do they share lengthy immediate prefixes?
- Can one GPU maintain the mannequin, or should the mannequin be partitioned?
These questions matter as a result of the very best design is dependent upon the bottleneck. Including GPUs doesn’t robotically make a single request quicker. It might assist throughput by means of replication, or it might make a bigger mannequin doable by means of partitioning. The benchmark ought to present which impact you’re getting.
Price per Token
Price is a efficiency metric. A quicker system that makes use of way more costly {hardware} might not be higher for an software.
A easy value estimate is:
|
cost_per_output_token = hardware_cost_per_second / output_tokens_per_second |
If a GPU occasion prices 3 {dollars} per hour and the service generates 1,000 output tokens per second:
|
hardware_cost_per_second = 3.00 / 3600 = 0.000833 cost_per_output_token = 0.000833 / 1000 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â = 0.000000833 |
That is lower than one millionth of a greenback per output token for {hardware} alone. An actual calculation may additionally embody idle capability, storage, networking, engineering time, orchestration overhead, and failed requests.
Price ought to be in contrast with high quality. Quantization, smaller fashions, and routing can scale back value, however they might change mannequin conduct. An environment friendly inference system just isn’t merely the quickest one. It’s the one which meets high quality and reliability necessities on the lowest sensible value.
Additional Studying
Beneath are some sources chances are you’ll discover helpful:
- Little’s legislation, on Wikipedia.
It is a helpful queueing-theory end result for relating common concurrency, arrival fee, and response time. It’s a useful psychological mannequin when reasoning about request fee, latency, and the variety of in-flight inference requests. - Metrics, in NVIDIA NIM LLMs Benchmarking.
This web page defines frequent LLM inference metrics resembling time to first token, end-to-end latency, inter-token latency, tokens per second, and requests per second. - MLPerf Inference, by MLCommons.
It is a extensively used benchmark suite for measuring inference efficiency throughout deployment eventualities. It’s not restricted to LLMs, however it offers helpful self-discipline round repeatable benchmarking and reporting. - torch.profiler, within the PyTorch documentation.
That is the primary PyTorch profiling interface for gathering CPU and accelerator exercise, operator timings, reminiscence data, tensor shapes, and traces that may be inspected later. - NVIDIA Nsight Techniques Person Information, by NVIDIA.
Nsight Techniques is helpful when wall-clock timers usually are not sufficient and also you want a timeline of CUDA API calls, GPU kernels, reminiscence copies, CPU work, and synchronization. - Taming the Titans: A Survey of Environment friendly LLM Inference Serving, by Zhen et al.
This survey provides a broader view of LLM inference serving, together with request scheduling, mannequin placement, storage administration, disaggregation, load balancing, and cluster-level serving points.
Abstract
On this chapter, you discovered find out how to measure LLM inference efficiency. You noticed why prefill and decode ought to be measured individually, find out how to use wall-clock timers and CUDA occasions, find out how to file reminiscence utilization, and find out how to report latency percentiles. You additionally discovered that a number of GPUs can imply replication for extra throughput or partitioning for bigger fashions, and that the benchmark ought to make this distinction clear.
Within the subsequent a part of the ebook, you’ll start finding out methods for making one mannequin quicker, beginning with floating-point precision.
Â
Whenever you optimize the inference efficiency of an LLM, that you must know find out how to measure it. With out measurement, it’s simple to make a mannequin extra difficult with out making it quicker, or to enhance throughput whereas making user-visible latency worse.
An LLM service has a number of sorts of efficiency. A person cares about how lengthy it takes to see the primary token and the way shortly the remainder of the reply streams. An operator cares about what number of requests the {hardware} can serve, how a lot reminiscence is used, and the way a lot every generated token prices. A researcher could care about whether or not an optimization modifications the mannequin’s output high quality.
On this chapter, you’ll study:
- Latency and throughput metrics
- Time to first token and time per output token
- Measuring CPU and GPU inference
- Utilizing CUDA occasions
- Benchmarking a number of requests
- Serious about a number of GPUs and a number of machines
Let’s get began.
Â
Measuring Efficiency of Transformer Inference
Picture by Tomas Anton Escobar. Some rights reserved.
Overview
This chapter is split into eight components; they’re:
- Metrics for LLM Inference
- Measuring a Single Request
- Warmup and Synchronization
- Measuring GPU Work with CUDA Occasions
- Measuring Reminiscence Utilization
- Measuring Concurrent Requests
- A number of GPUs and A number of Machines
- Price per Token
Metrics for LLM Inference
The most typical inference metrics are:
- Latency:Â How lengthy a request takes from begin to end.
- Time to first token (TTFT): How lengthy the person waits earlier than the primary output token seems.
- Time per output token (TPOT):Â The common time between generated tokens after the primary token.
- Throughput: What number of tokens or requests are processed per second.
- Reminiscence utilization:Â How a lot CPU reminiscence or GPU reminiscence is used.
- Utilization: How busy the accelerator is throughout the benchmark.
- Price per token:Â The {hardware} or service value divided by the variety of tokens processed.
For LLMs, a single latency quantity is often not sufficient. Contemplate two requests:
- Request A: 2,000 immediate tokens and 20 output tokens
- Request B: 20 immediate tokens and a pair of,000 output tokens
Request A stresses prefill. Request B stresses decode. They might have the identical complete variety of tokens, however they’ve totally different efficiency profiles. Because of this it is best to file immediate tokens and output tokens individually.
Tail latency additionally issues. If most requests full in a single second however a number of take ten seconds, customers will discover. Report high-percentile latencies resembling p90, p95, and p99 along with the imply or median. The excessive percentiles describe the worst circumstances higher. You may simply discover these percentiles from an inventory of values utilizing NumPy:
|
import numpy as np  def summarize(values):     values = np.asarray(values, dtype=np.float64)     return {         “imply”: values.imply(),         “median”: np.percentile(values, 50),         “p90”: np.percentile(values, 90),         “p95”: np.percentile(values, 95),         “p99”: np.percentile(values, 99),     } |
These numbers are easy, however they forestall a typical mistake: optimizing the typical whereas making the worst circumstances slower.
Measuring a Single Request
The best measurement makes use of time.perf_counter(). It’s a built-in high-resolution wall-clock timer appropriate for measuring elapsed time in Python. It’s extra correct than time.time().
The next instance measures prefill and decode individually for a Hugging Face causal language mannequin:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
import time import torch from transformers import AutoModelForCausalLM, AutoTokenizer   def load_model(model_name=“sshleifer/tiny-gpt2”, gadget=“cpu”):     tokenizer = AutoTokenizer.from_pretrained(model_name)     mannequin = AutoModelForCausalLM.from_pretrained(model_name).to(gadget)     mannequin.eval()     return tokenizer, mannequin   @torch.no_grad() def measure_one_request(mannequin, tokenizer, immediate, max_new_tokens=50, gadget=“cpu”):     input_ids = tokenizer(immediate, return_tensors=“pt”).input_ids.to(gadget)      begin = time.perf_counter()     outputs = mannequin(input_ids, use_cache=True)     prefill_end = time.perf_counter()      past_key_values = outputs.past_key_values     next_token = outputs.logits[:, –1, :].argmax(dim=–1, keepdim=True)     generated = [next_token]      decode_times = []      for _ in vary(max_new_tokens – 1):         step_start = time.perf_counter()         outputs = mannequin(             next_token,             past_key_values=past_key_values,             use_cache=True,         )         # Be aware: Chances are you’ll want torch.cuda.synchronize() right here         step_end = time.perf_counter()          decode_times.append(step_end – step_start)         past_key_values = outputs.past_key_values         next_token = outputs.logits[:, –1, :].argmax(dim=–1, keepdim=True)         generated.append(next_token)          if tokenizer.eos_token_id is not None:             if next_token.merchandise() == tokenizer.eos_token_id:                 break      finish = time.perf_counter()     output_ids = torch.cat([input_ids] + generated, dim=1)      return {         “textual content”: tokenizer.decode(output_ids[0], skip_special_tokens=True),         “prompt_tokens”: input_ids.measurement(1),         “output_tokens”: len(generated),         “prefill_seconds”: prefill_end – begin,         “decode_seconds”: sum(decode_times),         “total_seconds”: finish – begin,         “ttft_seconds”: prefill_end – begin,         “seconds_per_output_token”: (             sum(decode_times) / max(1, len(decode_times))         ),     } |
This operate doesn’t use the mannequin’s generate() technique. That’s intentional. The objective is to show prefill and decode to allow them to be measured individually. The variety of output_tokens contains the tokens generated by each prefill and decode. The seconds_per_output_token is the typical time per output token within the decode section.
There are two particulars to note:
use_cache=Trueasks the mannequin to return the KV cache.- Throughout decode, the mannequin receives solely
next_token, not the entire sequence.
This is identical thought as Chapter 1, however utilizing a library mannequin.
Warmup and Synchronization
Whenever you measure efficiency, notice that some one-time prices mustn’t dominate the end result. In Python, import of a module could be sluggish however subsequent import of the identical module is immediate. Equally, the primary execution of some code could also be slower than subsequent executions because of initialization of knowledge buildings or warmup of caches. You need to measure steady-state work, not that setup overhead.
Due to this fact, benchmarks ought to embody warmup. The primary few iterations could also be slower for varied causes. As an alternative of measuring the whole time and dividing by the variety of iterations, it is best to measure the time for every iteration and analyze the steady-state ones. For instance, should you use the mannequin to generate a number of tokens, you’ll probably put the era in a loop. Measure every iteration as follows, then ignore the primary few outcomes:
|
def iterations(mannequin, tokenizer, immediate, gadget, steps=100, warmup=10): Â Â Â Â outcomes = [] Â Â Â Â for _ in vary(steps): Â Â Â Â Â Â Â Â end result = measure_one_request( Â Â Â Â Â Â Â Â Â Â Â Â mannequin, Â Â Â Â Â Â Â Â Â Â Â Â tokenizer, Â Â Â Â Â Â Â Â Â Â Â Â immediate, Â Â Â Â Â Â Â Â Â Â Â Â max_new_tokens=8, Â Â Â Â Â Â Â Â Â Â Â Â gadget=gadget, Â Â Â Â Â Â Â Â ) Â Â Â Â Â Â Â Â outcomes.append(end result) Â Â Â Â regular = outcomes[warmup:] Â Â Â Â return summarize([item[“total_seconds”] for merchandise in regular]) |
If you happen to use GPU to run your LLM inference, you additionally must initialize the kernels if you first run them. Sadly, many GPU operations are asynchronous. That’s, when you launched an operation on GPU, Python could proceed along with your code instantly whereas the GPU continues to be working. Due to this fact, a naive strategy to measure the time could be incorrect. As an alternative, it is best to use torch.cuda.synchronize() to attend for the GPU to complete the operation earlier than you cease the timer:
|
def sync_if_needed(gadget): Â Â Â Â if gadget.startswith(“cuda”): Â Â Â Â Â Â Â Â torch.cuda.synchronize() Â begin = time.perf_counter() outputs = mannequin(input_ids, use_cache=True) sync_if_needed(gadget) elapsed = time.perf_counter() – begin |
This provides a wall-clock measurement that features the precise GPU work. For correct prefill and per-token decode timings on GPU, name sync_if_needed(gadget) after every timed mannequin(...) name in measure_one_request(), not solely as soon as on the finish of the request.
Measuring GPU Work with CUDA Occasions
CUDA occasions measure elapsed time the GPU spent executing kernels, not the end-to-end person latency. This time doesn’t embody any Python overhead. Beneath is an instance of find out how to use CUDA occasions to measure the time:
|
def cuda_event_time(fn): Â Â Â Â begin = torch.cuda.Occasion(enable_timing=True) Â Â Â Â finish = torch.cuda.Occasion(enable_timing=True) Â Â Â Â Â begin.file() Â Â Â Â end result = fn() Â Â Â Â finish.file() Â Â Â Â Â torch.cuda.synchronize() Â Â Â Â milliseconds = begin.elapsed_time(finish) Â Â Â Â return end result, milliseconds / 1000.0 |
You should use it to measure one ahead go:
|
with torch.no_grad(): Â Â Â Â outputs, seconds = cuda_event_time( Â Â Â Â Â Â Â Â lambda: mannequin(input_ids, use_cache=True) Â Â Â Â ) Â print(f“GPU ahead time: {seconds:.6f} seconds”) |
CUDA occasion timing and wall-clock timing reply totally different questions:
- Wall-clock timing measures what the applying experiences.
- CUDA occasion timing measures how lengthy the GPU work took.
For an inference service, wall-clock timing is often the first metric as a result of customers expertise queues, tokenization, scheduling, community overhead, and streaming. CUDA occasions are helpful if you end up optimizing kernels or evaluating mannequin execution paths.
For deeper GPU profiling, use instruments resembling PyTorch Profiler, Nsight Techniques, Nsight Compute, or CUPTI-based monitoring. These instruments can report kernel timelines, reminiscence copies, GPU utilization, and operator-level breakdowns. They’re extra advanced than a timer, however they’re vital when a easy benchmark says the mannequin is sluggish and that you must know why.
Measuring Reminiscence Utilization
Reminiscence is a unique dimension to measure as a result of it limits not velocity for one person a lot as what number of customers your system can serve. Normally the GPU reminiscence is the bottleneck. In PyTorch, you may report allotted and reserved reminiscence like the next:
|
def gpu_memory_summary(gadget=“cuda”): Â Â Â Â torch.cuda.synchronize() Â Â Â Â return { Â Â Â Â Â Â Â Â “allocated_gb”: torch.cuda.memory_allocated(gadget) / 1e9, Â Â Â Â Â Â Â Â “reserved_gb”: torch.cuda.memory_reserved(gadget) / 1e9, Â Â Â Â Â Â Â Â “max_allocated_gb”: torch.cuda.max_memory_allocated(gadget) / 1e9, Â Â Â Â } |
The allotted worth is reminiscence utilized by tensors. The reserved worth is reminiscence held by PyTorch’s caching allocator. The utmost allotted worth is commonly probably the most helpful quantity for capability planning.
The allotted and reserved reminiscence are real-time snapshots however the max allotted worth is a peak over time. For correct measurement, it is best to reset the height statistic earlier than a benchmark:
|
torch.cuda.reset_peak_memory_stats() end result = measure_one_request(mannequin, tokenizer, immediate, gadget=“cuda”) reminiscence = gpu_memory_summary(“cuda”) print(reminiscence) |
Reminiscence ought to be measured along with tokens. A run with an extended immediate or extra generated tokens will naturally use extra KV cache reminiscence.
Measuring Concurrent Requests
To create a server that runs a language mannequin, consider the system by what number of requests you may serve per second. Throughput is dependent upon each how briskly you fulfill one request and what number of requests you may run concurrently, although concurrency doesn’t scale linearly underneath rivalry.
Manufacturing methods ought to deal with a number of customers, and the scheduler could batch their work collectively. The next easy benchmark runs a number of requests concurrently utilizing Python threads. This doesn’t implement steady batching. It solely measures how a mannequin wrapper behaves when a number of callers use it on the identical time.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 |
from concurrent.futures import ThreadPoolExecutor, as_completed  def run_prompt(mannequin, tokenizer, immediate, gadget):     begin = time.perf_counter()     end result = measure_one_request(         mannequin,         tokenizer,         immediate,         max_new_tokens=32,         gadget=gadget,     )     finish = time.perf_counter()     end result[“wall_seconds”] = finish – begin     return end result   def benchmark_concurrent(mannequin, tokenizer, prompts, gadget=“cpu”, employees=4):     outcomes = []     begin = time.perf_counter()      with ThreadPoolExecutor(max_workers=employees) as pool:         futures = [             pool.submit(run_prompt, model, tokenizer, prompt, device)             for prompt in prompts         ]         for future in as_completed(futures):             outcomes.append(future.end result())      finish = time.perf_counter()     total_output_tokens = sum(merchandise[“output_tokens”] for merchandise in outcomes)      return {         “requests”: len(outcomes),         “total_seconds”: finish – begin,         “output_tokens”: total_output_tokens,         “output_tokens_per_second”: total_output_tokens / (finish – begin),         “latency_summary”: summarize([item[“wall_seconds”] for merchandise in outcomes]),     } |
This benchmark is for illustration solely. It’s not a alternative for an actual serving benchmark. It doesn’t mannequin HTTP overhead, streaming, request queues, batching, cancellation, or cache eviction. However it’s a helpful subsequent step after a single-request benchmark the place you may run the mannequin in parallel and observe the per-request latency. The Python GIL (World Interpreter Lock) is often not the primary concern right here as a result of heavyweight mannequin execution is commonly offloaded to compiled code. Concurrent use of the identical mannequin or tensors from a number of threads is unsafe with out synchronization, so share one mannequin on CUDA solely with a lock or a single employee thread.
When benchmarking an actual server, file no less than:
- Variety of concurrent customers
- Immediate token distribution
- Output token distribution
- Request fee
- TTFT (Time to first token) percentiles
- Inter-token latency percentiles
- Complete tokens per second
- Error fee and timeout fee
The distributions matter. A benchmark with all prompts at precisely 128 tokens and all outputs at precisely 128 tokens is straightforward to check, however it might not signify your software.
A number of GPUs and A number of Machines
A number of GPUs can be utilized for inference in a number of alternative ways. Completely different approaches can drastically change the efficiency of your system.
The best strategy is replication. You load one copy of the mannequin on every GPU and route totally different requests to totally different replicas. This will increase throughput and is straightforward to motive about, however every GPU should have sufficient reminiscence for the total mannequin and its KV cache.
One other strategy is to separate one mannequin throughout a number of GPUs. Tensor parallelism splits weight matrices throughout units. Pipeline parallelism locations totally different layers on totally different units. Context parallelism partitions sequence work. Knowledgeable parallelism is used for mixture-of-experts fashions. These methods permit bigger fashions to run, however they introduce communication overhead and might improve latency.
A number of machines add one other layer. A system could use many replicas throughout machines for prime request quantity. It might additionally cut up a single massive mannequin throughout machines, however that is tougher as a result of community communication is slower than communication inside one machine. For low-latency serving, crossing machine boundaries inside one ahead go ought to be handled as costly.
Measuring efficiency of a system with a number of GPUs or a number of machines provides a brand new dimension of communication and synchronization overhead. Earlier than selecting a multi-GPU or multi-machine design, reply these questions:
- Are you serving one massive mannequin or many smaller fashions?
- Are you restricted by mannequin weight reminiscence or KV cache reminiscence?
- Do you want decrease latency, increased throughput, or each?
- Are requests impartial, or do they share lengthy immediate prefixes?
- Can one GPU maintain the mannequin, or should the mannequin be partitioned?
These questions matter as a result of the very best design is dependent upon the bottleneck. Including GPUs doesn’t robotically make a single request quicker. It might assist throughput by means of replication, or it might make a bigger mannequin doable by means of partitioning. The benchmark ought to present which impact you’re getting.
Price per Token
Price is a efficiency metric. A quicker system that makes use of way more costly {hardware} might not be higher for an software.
A easy value estimate is:
|
cost_per_output_token = hardware_cost_per_second / output_tokens_per_second |
If a GPU occasion prices 3 {dollars} per hour and the service generates 1,000 output tokens per second:
|
hardware_cost_per_second = 3.00 / 3600 = 0.000833 cost_per_output_token = 0.000833 / 1000 Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â = 0.000000833 |
That is lower than one millionth of a greenback per output token for {hardware} alone. An actual calculation may additionally embody idle capability, storage, networking, engineering time, orchestration overhead, and failed requests.
Price ought to be in contrast with high quality. Quantization, smaller fashions, and routing can scale back value, however they might change mannequin conduct. An environment friendly inference system just isn’t merely the quickest one. It’s the one which meets high quality and reliability necessities on the lowest sensible value.
Additional Studying
Beneath are some sources chances are you’ll discover helpful:
- Little’s legislation, on Wikipedia.
It is a helpful queueing-theory end result for relating common concurrency, arrival fee, and response time. It’s a useful psychological mannequin when reasoning about request fee, latency, and the variety of in-flight inference requests. - Metrics, in NVIDIA NIM LLMs Benchmarking.
This web page defines frequent LLM inference metrics resembling time to first token, end-to-end latency, inter-token latency, tokens per second, and requests per second. - MLPerf Inference, by MLCommons.
It is a extensively used benchmark suite for measuring inference efficiency throughout deployment eventualities. It’s not restricted to LLMs, however it offers helpful self-discipline round repeatable benchmarking and reporting. - torch.profiler, within the PyTorch documentation.
That is the primary PyTorch profiling interface for gathering CPU and accelerator exercise, operator timings, reminiscence data, tensor shapes, and traces that may be inspected later. - NVIDIA Nsight Techniques Person Information, by NVIDIA.
Nsight Techniques is helpful when wall-clock timers usually are not sufficient and also you want a timeline of CUDA API calls, GPU kernels, reminiscence copies, CPU work, and synchronization. - Taming the Titans: A Survey of Environment friendly LLM Inference Serving, by Zhen et al.
This survey provides a broader view of LLM inference serving, together with request scheduling, mannequin placement, storage administration, disaggregation, load balancing, and cluster-level serving points.
Abstract
On this chapter, you discovered find out how to measure LLM inference efficiency. You noticed why prefill and decode ought to be measured individually, find out how to use wall-clock timers and CUDA occasions, find out how to file reminiscence utilization, and find out how to report latency percentiles. You additionally discovered that a number of GPUs can imply replication for extra throughput or partitioning for bigger fashions, and that the benchmark ought to make this distinction clear.
Within the subsequent a part of the ebook, you’ll start finding out methods for making one mannequin quicker, beginning with floating-point precision.
Â
