feeling when all the things appears to be working simply high quality, till you look underneath the hood and notice your system is burning 10× extra gasoline than it must?
We had a consumer script firing off requests to validate our prompts, constructed with async Python code and working easily in a Jupyter pocket book. Clear, easy, and quick. We ran it usually to check our fashions and acquire analysis information. No crimson flags. No warnings.
However beneath that polished floor, one thing was quietly going incorrect.
We weren’t seeing failures. We weren’t getting exceptions. We weren’t even noticing slowness. However our system was doing much more work than it wanted to, and we didn’t notice it.
On this submit, we’ll stroll by way of how we found the difficulty, what prompted it, and the way a easy structural change in our async code diminished LLM site visitors and value by 90%, with nearly no loss in pace or performance.
Now, truthful warning, studying this submit received’t magically slash your LLM prices by 90%. However the takeaway right here is broader: small, missed design choices, generally only a few traces of code, can result in huge inefficiencies. And being intentional about how your code runs can prevent time, cash, and frustration in the long term.
The repair itself may really feel area of interest at first. It entails the subtleties of Python’s asynchronous habits, how duties are scheduled and dispatched. When you’re acquainted with Python and async/await, you’ll get extra out of the code examples, however even when you’re not, there’s nonetheless loads to remove. As a result of the actual story right here isn’t nearly LLMs or Python, it’s about accountable, environment friendly engineering.
Let’s dig in.
The Setup
To automate validation, we use a predefined dataset and set off our system by way of a consumer script. The validation focuses on a small subset of the dataset, so the consumer code solely stops after receiving a sure variety of responses.
Right here’s a simplified model of our consumer in Python:
import asyncio
from aiohttp import ClientSession
from tqdm.asyncio import tqdm_asyncio
URL = "http://localhost:8000/instance"
NUMBER_OF_REQUESTS = 100
STOP_AFTER = 10
async def fetch(session: ClientSession, url: str) -> bool:
async with session.get(url) as response:
physique = await response.json()
return physique["value"]
async def principal():
outcomes = []
async with ClientSession() as session:
duties = [fetch(session, URL) for _ in range(NUMBER_OF_REQUESTS)]
for future in tqdm_asyncio.as_completed(duties, whole=NUMBER_OF_REQUESTS, desc="Fetching"):
response = await future
if response is True:
outcomes.append(response)
if len(outcomes) >= STOP_AFTER:
print(f"n✅ Stopped after receiving {STOP_AFTER} true responses.")
break
asyncio.run(principal())This script reads requests from a dataset, fires them concurrently, and stops as soon as we acquire sufficient true responses for our analysis. In manufacturing, the logic is extra advanced and primarily based on the variety of responses we want. However the construction is identical.
Let’s use a dummy FastAPI server to simulate actual habits:
import asyncio
import fastapi
import uvicorn
import random
app = fastapi.FastAPI()
@app.get("/instance")
async def instance():
sleeping_time = random.uniform(1, 2)
await asyncio.sleep(sleeping_time)
return {"worth": random.alternative([True, False])}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=8000)Now let’s hearth up that dummy server and run the consumer. You’ll see one thing like this from the consumer terminal:
Can You Spot the Drawback?
Good! Quick, clear, and… wait is all the things working as anticipated?
On the floor, it appears just like the consumer is doing the precise factor: sending requests, getting 10 true responses, then stopping.
However is it?
Let’s add a number of print statements to our server to see what it’s truly doing underneath the hood:
import asyncio
import fastapi
import uvicorn
import random
app = fastapi.FastAPI()
@app.get("/instance")
async def instance():
print("Acquired a request")
sleeping_time = random.uniform(1, 2)
print(f"Sleeping for {sleeping_time:.2f} seconds")
await asyncio.sleep(sleeping_time)
worth = random.alternative([True, False])
print(f"Returning worth: {worth}")
return {"worth": worth}
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0", port=8000)Now re-run all the things.
You’ll begin seeing logs like this:
Acquired a request
Sleeping for 1.11 seconds
Acquired a request
Sleeping for 1.29 seconds
Acquired a request
Sleeping for 1.98 seconds
...
Returning worth: True
Returning worth: False
Returning worth: False
...Take a better have a look at the server logs. You’ll discover one thing surprising: as an alternative of processing simply 14 requests like we see within the progress bar, the server handles all 100. Although the consumer stops after receiving 10 true responses, it nonetheless sends each request up entrance. In consequence, the server should course of all of them.
It’s a straightforward mistake to overlook, particularly as a result of all the things seems to be working appropriately from the consumer’s perspective: responses are available shortly, the progress bar advances, and the script exits early. However behind the scenes, all 100 requests are despatched instantly, no matter after we resolve to cease listening. This ends in 10× extra site visitors than wanted, driving up prices, rising load, and risking fee limits.
So the important thing query turns into: why is that this taking place, and the way can we be sure we solely ship the requests we really want? The reply turned out to be a small however highly effective change.
The basis of the difficulty lies in how the duties are scheduled. In our unique code, we create a listing of 100 duties all of sudden:
duties = [fetch(session, URL) for _ in range(NUMBER_OF_REQUESTS)]
for future in tqdm_asyncio.as_completed(duties, whole=NUMBER_OF_REQUESTS, desc="Fetching"):
response = await futureWhile you move a listing of coroutines to as_completed, Python instantly wraps every coroutine in a Activity and schedules it on the occasion loop. This occurs earlier than you begin iterating over the loop physique. As soon as a coroutine turns into a Activity, the occasion loop begins working it within the background straight away.
as_completed itself doesn’t management concurrency, it merely waits for duties to complete and yields them one after the other within the order they full. Consider it as an iterator over accomplished futures, not a site visitors controller. Which means by the point you begin looping, all 100 requests are already in progress. Breaking out after 10 true outcomes stops you from processing the remaining, however it doesn’t cease them from being despatched.
To repair this, we launched a semaphore to restrict concurrency. The semaphore provides a light-weight lock inside fetch in order that solely a set variety of requests can begin on the identical time. The remaining stay paused, ready for a slot. As soon as we hit our stopping situation, the paused duties by no means purchase the lock, in order that they by no means ship their requests.
Right here’s the adjusted model:
import asyncio
from aiohttp import ClientSession
from tqdm.asyncio import tqdm_asyncio
URL = "http://localhost:8000/instance"
NUMBER_OF_REQUESTS = 100
STOP_AFTER = 10
async def fetch(session: ClientSession, url: str, semaphore: asyncio.Semaphore) -> str:
async with semaphore:
async with session.get(url) as response:
physique = await response.json()
return physique["value"]
async def principal():
outcomes = []
semaphore = asyncio.Semaphore(int(STOP_AFTER * 1.5))
async with ClientSession() as session:
duties = [fetch(session, URL, semaphore) for _ in range(NUMBER_OF_REQUESTS)]
for future in tqdm_asyncio.as_completed(duties, whole=NUMBER_OF_REQUESTS, desc="Fetching"):
response = await future
if response:
outcomes.append(response)
if len(outcomes) >= STOP_AFTER:
print(f"n✅ Stopped after receiving {STOP_AFTER} true responses.")
break
asyncio.run(principal())With this transformation, we nonetheless outline 100 requests upfront, however solely a small group is allowed to run on the identical time, 15 in that instance. If we attain our stopping situation early, the occasion loop stops earlier than launching extra requests. This retains the habits responsive whereas lowering pointless calls.
Now, the server logs will show solely round 20 "Acquired a request/Returning response" entries. On the consumer aspect, the progress bar will seem equivalent to the unique.
With this transformation in place, we noticed quick influence: 90% discount in request quantity and LLM price, with no noticeable degradation in consumer expertise. It additionally improved throughput throughout the group, diminished queuing, and eradicated rate-limit points from our LLM suppliers.
This small structural adjustment made our validation pipeline dramatically extra environment friendly, with out including a lot complexity to the code. It’s a great reminder that in async programs, management circulate doesn’t all the time behave the best way you assume until you’re express about how duties are scheduled and when they need to run.
Bonus Perception: Closing the Occasion Loop
If we had run the unique consumer code with out asyncio.run, we’d have seen the issue earlier.
For instance, if we had used handbook occasion loop administration like this:
loop = asyncio.get_event_loop()
loop.run_until_complete(principal())
loop.shut()Python would have printed warnings equivalent to:
These warnings seem when this system exits whereas there are nonetheless unfinished async duties scheduled within the loop. If we had seen a display stuffed with these warnings, it possible would’ve triggered a crimson flag a lot sooner.
So why didn’t we see that warning when utilizing asyncio.run()?
As a result of asyncio.run() takes care of cleanup behind the scenes. It doesn’t simply run your coroutine and exit, it additionally cancels any remaining duties, waits for them to complete, and solely then shuts down the occasion loop. This built-in security internet prevents these “pending job” warnings from displaying up, even when your code quietly launched extra duties than it wanted to.
In consequence, it suppresses these “pending job” warnings if you manually shut the loop with loop.shut() after run_until_complete(), any leftover duties that haven’t been awaited will nonetheless be hanging round. Python detects that you simply’re forcefully shutting down the loop whereas work remains to be scheduled, and warns you about it.
This isn’t to say that each async Python program ought to keep away from asyncio.run() or all the time use loop.run_until_complete() with a handbook loop.shut(). However it does spotlight one thing vital: you have to be conscious of what duties are nonetheless working when your program exits. On the very least, it’s a good suggestion to watch or log any pending duties earlier than shutdown.
Remaining Ideas
By stepping again and rethinking the management circulate, we have been in a position to make our validation course of dramatically extra environment friendly — not by including extra infrastructure, however through the use of what we already had extra fastidiously. A couple of traces of code change led to a 90% price discount with nearly no added complexity. It resolved rate-limit errors, diminished system load, and allowed the group to run evaluations extra continuously with out inflicting bottlenecks.
It is a vital reminder that “clear” async code doesn’t all the time imply environment friendly code, being intentional about how we use system assets is essential. Accountable, environment friendly engineering is about extra than simply writing code that works. It’s about designing programs that respect time, cash, and shared assets, particularly in collaborative environments. While you deal with compute as a shared asset as an alternative of an infinite pool, everybody advantages: programs scale higher, groups transfer quicker, and prices keep predictable.
So, whether or not you’re making LLM calls, launching Kubernetes jobs, or processing information in batches, pause and ask your self: am I solely utilizing what I actually need?
Usually, the reply and the development are only one line of code away.
