Picture by Editor | ChatGPT# Introduction
Hugging Face has develop into the usual for a lot of AI builders and knowledge scientists as a result of it drastically lowers the barrier to working with superior AI. Fairly than working with AI fashions from scratch, builders can entry a variety of pretrained fashions with out trouble. Customers may also adapt these fashions with customized datasets and deploy them shortly.
One of many Hugging Face framework API wrappers is the Transformers Pipelines, a collection of packages that consists of the pretrained mannequin, its tokenizer, pre- and post-processing, and associated parts to make an AI use case work. These pipelines summary advanced code and supply a easy, seamless API.
Nevertheless, working with Transformers Pipelines can get messy and should not yield an optimum pipeline. That’s the reason we are going to discover 5 other ways you’ll be able to optimize your Transformers Pipelines.
Let’s get into it.
# 1. Batch Inference Requests
Typically, when utilizing Transformers Pipelines, we don’t totally make the most of the graphics processing unit (GPU). Batch processing of a number of inputs can considerably enhance GPU utilization and improve inference effectivity.
As a substitute of processing one pattern at a time, you should utilize the pipeline’s batch_size parameter or move a listing of inputs so the mannequin processes a number of inputs in a single ahead move. Here’s a code instance:
from transformers import pipeline
pipe = pipeline(
activity="text-classification",
mannequin="distilbert-base-uncased-finetuned-sst-2-english",
device_map="auto"
)
texts = [
"Great product and fast delivery!",
"The UI is confusing and slow.",
"Support resolved my issue quickly.",
"Not worth the price."
]
outcomes = pipe(texts, batch_size=16, truncation=True, padding=True)
for r in outcomes:
print(r)
By batching requests, you’ll be able to obtain increased throughput with solely a minimal influence on latency.
# 2. Use Decrease Precision And Quantization
Many pretrained fashions fail at inference as a result of improvement and manufacturing environments would not have sufficient reminiscence. Decrease numerical precision helps cut back reminiscence utilization and hurries up inference with out sacrificing a lot accuracy.
For instance, right here is find out how to use half precision on the GPU in a Transformers Pipeline:
import torch
from transformers import AutoModelForSequenceClassification
mannequin = AutoModelForSequenceClassification.from_pretrained(
model_id,
torch_dtype=torch.float16
)
Equally, quantization methods can compress mannequin weights with out noticeably degrading efficiency:
# Requires bitsandbytes for 8-bit quantization
from transformers import AutoModelForCausalLM
mannequin = AutoModelForCausalLM.from_pretrained(
model_id,
load_in_8bit=True,
device_map="auto"
)
Utilizing decrease precision and quantization in manufacturing normally hurries up pipelines and reduces reminiscence use with out considerably impacting mannequin accuracy.
# 3. Choose Environment friendly Mannequin Architectures
In lots of purposes, you do not want the biggest mannequin to unravel the duty. Choosing a lighter transformer structure, comparable to a distilled mannequin, typically yields higher latency and throughput with a suitable accuracy trade-off.
Compact fashions or distilled variations, comparable to DistilBERT, retain a lot of the authentic mannequin’s accuracy however with far fewer parameters, leading to quicker inference.
Select a mannequin whose structure is optimized for inference and fits your activity’s accuracy necessities.
# 4. Leverage Caching
Many techniques waste compute by repeating costly work. Caching can considerably improve efficiency by reusing the outcomes of expensive computations.
with torch.inference_mode():
output_ids = mannequin.generate(
**inputs,
max_new_tokens=120,
do_sample=False,
use_cache=True
)
Environment friendly caching reduces computation time and improves response occasions, reducing latency in manufacturing techniques.
# 5. Use An Accelerated Runtime Through Optimum (ONNX Runtime)
Many pipelines run in a PyTorch not-so-optimal mode, which provides Python overhead and further reminiscence copies. Utilizing Optimum with Open Neural Community Alternate (ONNX) Runtime — through ONNX Runtime — converts the mannequin to a static graph and fuses operations, so the runtime can use quicker kernels on a central processing unit (CPU) or GPU with much less overhead. The result’s normally quicker inference, particularly on CPU or blended {hardware}, with out altering the way you name the pipeline.
Set up the required packages with:
pip set up -U transformers optimum[onnxruntime] onnxruntime
Then, convert the mannequin with code like this:
from optimum.onnxruntime import ORTModelForSequenceClassification
ort_model = ORTModelForSequenceClassification.from_pretrained(
model_id,
from_transformers=True
)
By changing the pipeline to ONNX Runtime by means of Optimum, you’ll be able to preserve your present pipeline code whereas getting decrease latency and extra environment friendly inference.
# Wrapping Up
Transformers Pipelines is an API wrapper within the Hugging Face framework that facilitates AI utility improvement by condensing advanced code into less complicated interfaces. On this article, we explored 5 tricks to optimize Hugging Face Transformers Pipelines, from batch inference requests, to deciding on environment friendly mannequin architectures, to leveraging caching and past.
I hope this has helped!
Cornellius Yudha Wijaya is an information science assistant supervisor and knowledge author. Whereas working full-time at Allianz Indonesia, he likes to share Python and knowledge suggestions through social media and writing media. Cornellius writes on quite a lot of AI and machine studying matters.
Picture by Editor | ChatGPT# Introduction
Hugging Face has develop into the usual for a lot of AI builders and knowledge scientists as a result of it drastically lowers the barrier to working with superior AI. Fairly than working with AI fashions from scratch, builders can entry a variety of pretrained fashions with out trouble. Customers may also adapt these fashions with customized datasets and deploy them shortly.
One of many Hugging Face framework API wrappers is the Transformers Pipelines, a collection of packages that consists of the pretrained mannequin, its tokenizer, pre- and post-processing, and associated parts to make an AI use case work. These pipelines summary advanced code and supply a easy, seamless API.
Nevertheless, working with Transformers Pipelines can get messy and should not yield an optimum pipeline. That’s the reason we are going to discover 5 other ways you’ll be able to optimize your Transformers Pipelines.
Let’s get into it.
# 1. Batch Inference Requests
Typically, when utilizing Transformers Pipelines, we don’t totally make the most of the graphics processing unit (GPU). Batch processing of a number of inputs can considerably enhance GPU utilization and improve inference effectivity.
As a substitute of processing one pattern at a time, you should utilize the pipeline’s batch_size parameter or move a listing of inputs so the mannequin processes a number of inputs in a single ahead move. Here’s a code instance:
from transformers import pipeline
pipe = pipeline(
activity="text-classification",
mannequin="distilbert-base-uncased-finetuned-sst-2-english",
device_map="auto"
)
texts = [
"Great product and fast delivery!",
"The UI is confusing and slow.",
"Support resolved my issue quickly.",
"Not worth the price."
]
outcomes = pipe(texts, batch_size=16, truncation=True, padding=True)
for r in outcomes:
print(r)
By batching requests, you’ll be able to obtain increased throughput with solely a minimal influence on latency.
# 2. Use Decrease Precision And Quantization
Many pretrained fashions fail at inference as a result of improvement and manufacturing environments would not have sufficient reminiscence. Decrease numerical precision helps cut back reminiscence utilization and hurries up inference with out sacrificing a lot accuracy.
For instance, right here is find out how to use half precision on the GPU in a Transformers Pipeline:
import torch
from transformers import AutoModelForSequenceClassification
mannequin = AutoModelForSequenceClassification.from_pretrained(
model_id,
torch_dtype=torch.float16
)
Equally, quantization methods can compress mannequin weights with out noticeably degrading efficiency:
# Requires bitsandbytes for 8-bit quantization
from transformers import AutoModelForCausalLM
mannequin = AutoModelForCausalLM.from_pretrained(
model_id,
load_in_8bit=True,
device_map="auto"
)
Utilizing decrease precision and quantization in manufacturing normally hurries up pipelines and reduces reminiscence use with out considerably impacting mannequin accuracy.
# 3. Choose Environment friendly Mannequin Architectures
In lots of purposes, you do not want the biggest mannequin to unravel the duty. Choosing a lighter transformer structure, comparable to a distilled mannequin, typically yields higher latency and throughput with a suitable accuracy trade-off.
Compact fashions or distilled variations, comparable to DistilBERT, retain a lot of the authentic mannequin’s accuracy however with far fewer parameters, leading to quicker inference.
Select a mannequin whose structure is optimized for inference and fits your activity’s accuracy necessities.
# 4. Leverage Caching
Many techniques waste compute by repeating costly work. Caching can considerably improve efficiency by reusing the outcomes of expensive computations.
with torch.inference_mode():
output_ids = mannequin.generate(
**inputs,
max_new_tokens=120,
do_sample=False,
use_cache=True
)
Environment friendly caching reduces computation time and improves response occasions, reducing latency in manufacturing techniques.
# 5. Use An Accelerated Runtime Through Optimum (ONNX Runtime)
Many pipelines run in a PyTorch not-so-optimal mode, which provides Python overhead and further reminiscence copies. Utilizing Optimum with Open Neural Community Alternate (ONNX) Runtime — through ONNX Runtime — converts the mannequin to a static graph and fuses operations, so the runtime can use quicker kernels on a central processing unit (CPU) or GPU with much less overhead. The result’s normally quicker inference, particularly on CPU or blended {hardware}, with out altering the way you name the pipeline.
Set up the required packages with:
pip set up -U transformers optimum[onnxruntime] onnxruntime
Then, convert the mannequin with code like this:
from optimum.onnxruntime import ORTModelForSequenceClassification
ort_model = ORTModelForSequenceClassification.from_pretrained(
model_id,
from_transformers=True
)
By changing the pipeline to ONNX Runtime by means of Optimum, you’ll be able to preserve your present pipeline code whereas getting decrease latency and extra environment friendly inference.
# Wrapping Up
Transformers Pipelines is an API wrapper within the Hugging Face framework that facilitates AI utility improvement by condensing advanced code into less complicated interfaces. On this article, we explored 5 tricks to optimize Hugging Face Transformers Pipelines, from batch inference requests, to deciding on environment friendly mannequin architectures, to leveraging caching and past.
I hope this has helped!
Cornellius Yudha Wijaya is an information science assistant supervisor and knowledge author. Whereas working full-time at Allianz Indonesia, he likes to share Python and knowledge suggestions through social media and writing media. Cornellius writes on quite a lot of AI and machine studying matters.
