Do you want extra GPUs or a contemporary GPU? How do you make infrastructure choices?
How does it really feel when a gaggle of customers out of the blue begin utilizing an app that solely you and your dev workforce have used earlier than?
That’s the million-dollar query of transferring from prototype to manufacturing.
So far as LLMs are involved, you are able to do a number of dozen tweaks to run your app inside the price range and acceptable qualities. For example, you possibly can select a quantized mannequin for decrease reminiscence utilization. Or you possibly can fine-tune a tiny mannequin and beat the efficiency of large LLMs.
You’ll be able to even tweak your infrastructure to realize higher outcomes. For instance, it’s possible you’ll need to double the variety of GPUs you employ or select the latest-generation GPU.
However how may you say Possibility A performs higher than Possibility B and C?
This is a crucial query to ask ourselves on the earliest levels of going into manufacturing. All these choices have their prices…
