Learn to use Google’s Germini-1.5-pro-latest mannequin to develop a generative AI app for calorie counting
Have you ever ever questioned the quantity of energy you devour while you eat your dinner, for instance? I do this on a regular basis. Wouldn’t or not it’s fantastic when you may merely move an image of your plate by means of an app and get an estimate of the whole variety of energy earlier than you determine how far in you need to dip?
This calorie counter app that I created might help you obtain this. It’s a Python software that makes use of Google’s Gemini-1.5-Professional-Newest mannequin to estimate the variety of energy in meals objects.
The app takes two inputs: a query in regards to the meals and a picture of the meals or meals objects, or just, a plate of meals. It outputs a solution to the query, the whole variety of energy within the picture and a breakdown of energy by every meals merchandise within the picture.
On this article, I’ll clarify the complete end-to-end strategy of constructing the app from scratch, utilizing Google’s Gemini-1.5-pro-latest (a Massive Language generative AI mannequin launched by Google), and the way I developed the front-end of the appliance utilizing Streamlit.
It’s value noting right here that with developments on the earth of AI, it’s incumbent on knowledge scientists to regularly shift from conventional deep studying to generative AI strategies so as to revolutionize their position. That is my principal function of training on this topic.
Let me begin by briefly explaining Gemini-1.5-pro-latest and the streamlit framework, as they’re the most important parts within the infrastructure of this calorie counter app.
Gemini-1.5-pro-latest is a complicated AI language mannequin developed by Google. Since it’s the newest model, it has enhanced capabilities over earlier variations within the mild of quicker response instances and improved accuracy when utilized in pure language processing and constructing functions.
This can be a multi-modal mannequin that works with each texts and pictures — an development from Google Gemini-pro mannequin which solely works with textual content prompts.
The mannequin works by understanding and producing textual content, like people, based mostly on prompts given to it. On this article, this mannequin shall be used to to generate textual content for our energy counter app.
Gemini-1.5-pro-latest might be built-in into different functions to bolster their AI capabilities. On this present software, the mannequin makes use of generative AI strategies to interrupt the uploaded picture into particular person meals objects . Primarily based on its contextual understanding of the meals objects from its dietary database, it makes use of picture recognition and object detection to estimate the variety of energy, after which totals up the energy for all objects within the picture.
Streamlit is an open-source Python framework that may handle the consumer interface. This framework simplifies internet improvement in order that all through the venture, you don’t want to put in writing any HTML and CSS codes for the entrance finish.
Allow us to dive into constructing the app.
I’ll present you tips on how to construct the app in 5 clear steps.
1. Arrange your Folder construction
For a begin, go into your favourite code editor (mine is VS Code) and begin a venture file. Name it Energy-Counter, for instance. That is the present working listing. Create a digital surroundings (venv), activate it in your terminal, after which create the next recordsdata: .env, energy.py, necessities.txt.
Right here’s a suggestion for the look of your folder construction:
Energy-Counter/
├── venv/
│ ├── xxx
│ ├── xxx
├── .env
├── energy.py
└── necessities.txt
Please be aware that Gemini-1.5-Professional works greatest with Python variations 3.9 and larger.
2. Get the Google API key
Like different Gemini fashions, Gemini-1.5-pro-latest is at present free for public use. Accessing it requires that you simply acquire an API key, which you will get from Google AI Studio by going to “Get API key” on this hyperlink. As soon as the hot button is generated, copy it for subsequent use in your code. Save this key as an surroundings variable within the .env file as follows.
GOOGLE_API_KEY="paste the generated key right here"
3. Set up dependencies
Sort the next libraries into your necessities.txt file.
- streamlit
- google-generativeai
- python-dotenv
Within the terminal, set up the libraries in necessities.txt with:
python -m pip set up -r necessities.txt
4. Write the Python script
Now, let’s begin writing the Python script in energy.py. With the next code, import all required libraries:
# import the libraries
from dotenv import load_dotenv
import streamlit as st
import os
import google.generativeai as genai
from PIL import Picture
Right here’s how the varied modules imported shall be used:
- dotenv — Since this software shall be configured from a Google API key surroundings variable, dotenv is used to load configuration from the .env file.
- Streamlit — to create an interactive consumer interface for front-end
- os module is used to deal with the present working listing whereas performing file operations like getting the API key from the .env file
- google.generativeai module, in fact, offers us entry to the Gemini mannequin we’re about to make use of.
- PIL is a Python imaging library used for managing picture file codecs.
The next traces will configure the API keys and cargo them from the surroundings variables retailer.
genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))load_dotenv()
Outline a operate that, when known as, will load the Gemini-1.5-pro-latest and get the response, as follows:
def get_gemini_reponse(input_prompt,picture,user_prompt):
mannequin=genai.GenerativeModel('gemini-1.5-pro-latest')
response=mannequin.generate_content([input_prompt,image[0],user_prompt])
return response.textual content
Within the above operate, you see that it takes as enter, the enter immediate that shall be specified additional down within the script, a picture that shall be equipped by the consumer, and a consumer immediate/query that shall be equipped by the consumer. All that goes into the gemini mannequin to return the response textual content.
Since Gemini-1.5-pro expects enter pictures within the type of byte arrays, the following factor to do is write a operate that processes the uploaded picture, changing it to bytes.
def input_image_setup(uploaded_file):
# Test if a file has been uploaded
if uploaded_file shouldn't be None:
# Learn the file into bytes
bytes_data = uploaded_file.getvalue()image_parts = [
{
"mime_type": uploaded_file.type, # Get the mime type of the uploaded file
"data": bytes_data
}
]
return image_parts
else:
increase FileNotFoundError("No file uploaded")
Subsequent, specify the enter immediate that may decide the behaviour of your app. Right here, we’re merely telling Gemini what to do with the textual content and picture that the app shall be fed with by the consumer.
input_prompt="""
You're an professional nutritionist.
You need to reply the query entered by the consumer within the enter based mostly on the uploaded picture you see.
You must also have a look at the meals objects discovered within the uploaded picture and calculate the whole energy.
Additionally, present the main points of each meals merchandise with energy consumption within the format beneath:1. Merchandise 1 - no of energy
2. Merchandise 2 - no of energy
----
----
"""
The subsequent step is to initialize streamlit and create a easy consumer interface to your calorie counter app.
st.set_page_config(page_title="Gemini Calorie Counter App")
st.header("Calorie Counter App")
enter=st.text_input("Ask any query associated to your meals: ",key="enter")
uploaded_file = st.file_uploader("Add a picture of your meals", kind=["jpg", "jpeg", "png"])
picture=""
if uploaded_file shouldn't be None:
picture = Picture.open(uploaded_file)
st.picture(picture, caption="Uploaded Picture.", use_column_width=True) #present the picturesubmit=st.button("Submit & Course of") #creates a "Submit & Course of" button
The above steps have all of the items of the app. At this level, the consumer is ready to open the app, enter a query and add a picture.
Lastly, let’s put all of the items collectively such that when the “Submit & Course of” button is clicked, the consumer will get the required response textual content.
# As soon as submit&Course of button is clicked
if submit:
image_data=input_image_setup(uploaded_file)
response=get_gemini_reponse(input_prompt,image_data,enter)
st.subheader("The Response is")
st.write(response)
5. Run the script and work together along with your app
Now that the app improvement is full, you may execute it within the terminal utilizing the command:
streamlit run energy.py
To work together along with your app and see the way it performs, view your Streamlit app in your browser utilizing the native url or community URL generated.
This how your Streamlit app appears like when it’s first opened on the browser.
As soon as the consumer asks a query and uploads a picture, right here is the show:
As soon as the consumer pushes the “Submit & Course of” button, the response within the picture beneath is generated on the backside of the display screen.
For exterior entry, take into account deploying your app utilizing cloud providers like AWS, Heroku, Streamlit Neighborhood Cloud. On this case, let’s use Streamlit Neighborhood Cloud to deploy the app at no cost.
On the highest proper of the app display screen, click on ‘Deploy’ and comply with the prompts to finish the deployment.
After deployment, you may share the generated app URL to different customers.
Identical to different AI functions, the outcomes outputed are the very best estimates of the mannequin, so, earlier than fully counting on the app, please be aware the next as a number of the potential dangers:
- The calorie counter app could misclassify sure meals objects and thus, give the mistaken variety of energy.
- The app doesn’t have a reference level to estimate the dimensions of the meals — portion — based mostly on the uploaded picture. This may result in errors.
- Over-reliance on the app can result in stress and psychological well being points as one could grow to be obsessive about counting energy and worrying about outcomes that will not be too correct.
To assist scale back the dangers that include utilizing the calorie counter, listed below are doable enhancements that may very well be built-in into its improvement:
- Including contextual evaluation of the picture, which can assist to gauge the dimensions of the meals portion being analysed. For example, the app may very well be constructed such that a typical object like a spoon, included within the meals picture, may very well be used as a reference level for measuring the sizes of the meals objects. It will scale back errors in ensuing whole energy.
- Google may enhance the range of the meals objects of their coaching set to scale back misclassification errors. They may broaden it to incorporate meals from extra cultures in order that even uncommon African meals objects shall be recognized.
