The right way to Construct a Easy AI Net Scraper with Python

Net scraping is the method of gathering info from web sites routinely. A standard scraper normally extracts uncooked textual content, HTML parts, or the total web page content material. However when you’re constructing AI brokers or giant language mannequin (LLM) functions, sending your entire webpage to the mannequin is just not all the time the very best method.

A greater manner is to first clear the web page, convert it into Markdown, after which use an LLM to grasp the content material and return solely the reply the consumer wants. This makes the output cleaner, simpler to learn, and simpler to make use of in one other workflow.

It additionally helps cut back token utilization. As an alternative of passing a messy webpage stuffed with navigation hyperlinks, buttons, scripts, footers, and repeated content material, we solely ship the helpful web page content material to the mannequin. The LLM then returns a targeted reply in Markdown as an alternative of dumping the entire web page again to the consumer.

On this information, we’ll construct a easy AI internet scraper in Python utilizing Jupyter Pocket book. It’ll fetch a webpage, clear the HTML, convert it into Markdown, settle for a consumer question, and return a transparent Markdown reply primarily based on the web page content material.

# Setting Up

We are going to use Jupyter Pocket book for this undertaking. It makes it simpler to check every step first earlier than turning the scraper into a correct utility programming interface (API) or utility.

Begin by putting in the required Python packages:

!pip set up requests beautifulsoup4 markdownify openai ftfy python-dotenv

We are going to use:

Within the subsequent cell, import the required libraries:

import os
import re
import requests

from bs4 import BeautifulSoup, Remark
from ftfy import fix_text
from markdownify import markdownify as markdownify_html
from openai import OpenAI
from dotenv import load_dotenv
from IPython.show import Markdown, show

Subsequent, be certain your OpenAI API secret’s out there as an setting variable. The safer manner is to create a .env file in the identical folder as your pocket book and add your key there:

OPENAI_API_KEY=your_api_key_here

Then load it contained in the pocket book:

load_dotenv()

consumer = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

You too can examine that the important thing was loaded appropriately:

if not os.getenv("OPENAI_API_KEY"):
    increase ValueError("OPENAI_API_KEY is lacking. Add it to your .env file first.")

Additionally be certain your OpenAI platform account has billing arrange. For brand spanking new API accounts, you might want so as to add pay as you go credit earlier than you may run API calls. If a mannequin is just not out there in your account, use one other mannequin out of your OpenAI dashboard.

Now outline the mannequin title:

MODEL_NAME = "gpt-5.4-nano"

We’re utilizing a smaller mannequin right here as a result of this job doesn’t want a big reasoning mannequin. The aim is straightforward: learn the cleaned webpage content material, perceive the consumer question, and return a targeted Markdown reply.

# Fetching the Webpage

Now we’ll create the primary operate. This operate will fetch the webpage utilizing the requests package deal and return the uncooked HTML.

def fetch_page(url: str) -> str:
    """
    Obtain the HTML content material from a webpage.
    """
    headers = {
        "Person-Agent": "SimpleAIScraper/1.0"
    }

    response = requests.get(url, headers=headers, timeout=15)
    response.raise_for_status()

    return response.textual content

The Person-Agent header tells the web site that the request is coming from our scraper. Some web sites block requests that don’t embrace a consumer agent, so including one makes the request a bit extra dependable.

We additionally use timeout to keep away from ready indefinitely if the web site doesn’t reply. The raise_for_status() name will cease the code if the request fails — for instance, if the web page returns a 404 or 500 error.

Now let’s check the operate with an actual web site:

uncooked = fetch_page("https://www.olostep.com/")
print(uncooked[:500])

This may obtain the uncooked HTML from the webpage and print the primary 500 characters.

Raw HTML output from the fetch_page function

Uncooked HTML output | Picture by Writer

At this stage, the output will nonetheless look messy as a result of it incorporates the total web page HTML, together with tags, scripts, structure parts, and different content material we don’t want.

# Cleansing the HTML

The uncooked HTML from a webpage normally incorporates plenty of content material we don’t want. It could possibly embrace scripts, styling, navigation menus, buttons, varieties, headers, footers, popups, and different structure parts.

Earlier than sending the web page content material to the LLM, we have to clear the HTML. This helps cut back noise and makes the ultimate Markdown a lot simpler for the mannequin to grasp.

We are going to use BeautifulSoup to parse the HTML and take away pointless parts.

def clean_html(html):
    html = fix_text(html)

    soup = BeautifulSoup(html, "html.parser")

    # Take away apparent noisy tags
    for tag in soup([
        "script", "style", "noscript", "svg", "img", "iframe",
        "nav", "header", "footer", "aside", "form", "button"
    ]):
        tag.decompose()

    noise_words = [
        "cursor",
        "modal",
        "popup",
        "floating",
        "signup",
        "login",
        "cookie",
        "banner",
        "navbar",
        "menu",
        "footer",
        "header",
        "subscribe",
        "newsletter",
        "loading",
        "wait",
        "success",
        "auth",
        "w-nav",
        "w-form"
    ]

    # First acquire noisy tags
    tags_to_remove = []

    for tag in soup.find_all(True):
        if tag.attrs is None:
            proceed

        class_value = tag.get("class", [])
        id_value = tag.get("id", "")

        if isinstance(class_value, checklist):
            class_text = " ".be part of(class_value).decrease()
        else:
            class_text = str(class_value).decrease()

        id_text = str(id_value).decrease()

        if any(phrase in class_text or phrase in id_text for phrase in noise_words):
            tags_to_remove.append(tag)

    # Then take away them safely
    for tag in tags_to_remove:
        tag.decompose()

    physique = soup.physique if soup.physique else soup

    return str(physique)

First, we use fix_text() to wash any damaged or unusual textual content encoding points. Then BeautifulSoup parses the HTML so we will take away the components we don’t want.

We take away apparent noisy tags like script, type, nav, header, footer, kind, and button. These sections normally don’t assist reply the consumer question and may waste tokens.

After that, we search for noisy class names and IDs. Many web sites use phrases like popup, cookie, navbar, e-newsletter, or modal inside their HTML. If a tag incorporates these phrases, we acquire it and take away it safely.

Now let’s run the operate on the uncooked HTML:

clear = clean_html(uncooked)
print(clear[:500])

As you may see, the webpage is now a lot cleaner. It nonetheless incorporates helpful HTML tags and textual content, however a lot of the noisy structure, scripts, navigation, and popups have been eliminated.

Cleaned HTML output after removing noisy elements

Cleaned HTML output | Picture by Writer

# Changing HTML to Markdown

Now we’ll convert the cleaned HTML into Markdown. Markdown is less complicated to learn, simpler to avoid wasting, and simpler for the LLM to grasp in comparison with uncooked HTML.

This step additionally helps cut back enter tokens as a result of we take away pointless formatting, pictures, clean traces, and repeated textual content. For the conversion, we’ll use markdownify.

def html_to_markdown(html):
    markdown_text = markdownify_html(
        html,
        heading_style="ATX",
        bullets="-"
    )

    markdown_text = fix_text(markdown_text)

    # Take away picture markdown
    markdown_text = re.sub(r"![.*?](.*?)", "", markdown_text)

    # Take away further areas and clean traces
    markdown_text = re.sub(r"[ t]+", " ", markdown_text)
    markdown_text = re.sub(r"n{3,}", "nn", markdown_text)

    traces = []

    skip_lines = [
        "click to try",
        "wait...",
        "you've successfully reserved your spot.",
        "thank you! your submission has been received!",
        "oops! something went wrong while submitting the form.",
        "product",
        "resources",
        "company"
    ]

    for line in markdown_text.splitlines():
        line = line.strip()

        if not line:
            proceed

        if line.decrease() in skip_lines:
            proceed

        traces.append(line)

    return "n".be part of(traces)

First, we use markdownify to transform the cleaned HTML into Markdown. We set the heading type to ATX, which suggests headings will use customary Markdown syntax with #, ##, and ###.

Then we run fix_text() once more to wash any remaining encoding points. After that, we take away picture Markdown as a result of picture hyperlinks are normally not helpful for answering text-based questions.

We additionally take away further areas and clean traces so the ultimate content material is compact. This makes the web page simpler to examine and helps cut back the variety of tokens despatched to the mannequin.

The skip_lines checklist removes repeated web site textual content resembling kind messages, navigation labels, and small call-to-action textual content. You’ll be able to replace this checklist primarily based on the web site you’re scraping.

Now let’s run the operate:

md = html_to_markdown(clear)
print(md[:500])

As you may see, the textual content is now a lot cleaner and nearer to the format we wish. As an alternative of uncooked HTML, we now have readable Markdown with helpful headings, paragraphs, and bullet factors.

Markdown output after converting cleaned HTML

Markdown output | Picture by Writer

# Asking a Person Question In opposition to the Web page

Now we’ll create the operate that sends the cleaned Markdown content material to the LLM. This operate takes two inputs: the webpage content material in Markdown and the consumer question.

As an alternative of asking the mannequin to summarize the entire web page, we ask it to reply a selected query utilizing solely the web page content material. This makes the response extra targeted and helpful.

def answer_query_from_page(markdown_text, user_query):
    immediate = f"""
You might be an AI internet scraping assistant.

You'll obtain Markdown extracted from a webpage.

Your job is to reply the consumer's question utilizing solely the helpful web page content material.

Person question:
{user_query}

Webpage Markdown:
{markdown_text}

Directions:
- Return solely clear Markdown.
- Use solely info from the webpage Markdown.
- Don't invent lacking particulars.
- Ignore navigation hyperlinks, buttons, CTAs, popups, ornamental labels, picture captions, and repeated advertising fragments.
- Ignore traces like "Begin totally free", "Contact Gross sales", "Your AI Agent", and ornamental workflow examples except they immediately reply the question.
- Deal with headings, paragraphs, product descriptions, function sections, pricing particulars, documentation textual content, and factual claims.
- If the web page doesn't comprise the reply, say: "The web page doesn't comprise this info."
- Preserve the reply brief, clear, and targeted.
"""

    response = consumer.responses.create(
        mannequin=MODEL_NAME,
        enter=immediate
    )

    return response.output_text

The immediate is crucial a part of this step. It tells the mannequin what position it ought to play, what content material it may use, and how much reply it ought to return.

We additionally inform the mannequin to make use of solely the supplied Markdown. That is vital as a result of we are not looking for the mannequin to guess or add info that isn’t current on the webpage.

The instruction to return solely clear Markdown makes the output simpler to show in a pocket book, save to a file, or go into one other AI workflow.

This operate is the place the AI internet scraper turns into genuinely helpful. We’re now not simply extracting web page textual content — we’re asking the LLM to grasp the cleaned web page and return the precise reply the consumer is in search of.

# Creating the Full AI Net Scraper

Now we’ll create the ultimate operate that connects every thing collectively.

This operate will take the URL and the consumer question as inputs. It’ll then fetch the webpage, clear the HTML, convert the content material into Markdown, and return the reply utilizing the gpt-5.4-nano mannequin.

def ai_web_scraper(url, user_query):
    raw_html = fetch_page(url)
    cleaned_html = clean_html(raw_html)
    markdown_text = html_to_markdown(cleaned_html)
    reply = answer_query_from_page(markdown_text, user_query)

    return reply

That is our full AI internet scraper pipeline. As an alternative of manually operating every step one after the other, we will now name a single operate and get a clear Markdown reply from any webpage.

The move is straightforward:

Fetch the webpage.
Clear the HTML.
Convert it into Markdown.
Ask the LLM a query.
Return the ultimate reply.

This retains the code easy and simple to reuse later in an API, chatbot, or agent workflow.

# Testing the AI Net Scraper

Now let’s check our AI internet scraper. We are going to present it with a web site URL and ask what the corporate does.

url = "https://www.olostep.com/"
user_query = "What does this firm do?"
outcome = ai_web_scraper(url, user_query)

show(Markdown(outcome))

In return, we get a correct Markdown response in regards to the firm and its product. That is a lot better than returning the total webpage content material as a result of the reply is targeted, readable, and immediately associated to the consumer question.

AI web scraper output answering what the company does

Scraper output for an organization overview question | Picture by Writer

Now let’s attempt a distinct web page and ask about pricing.

url = "https://www.olostep.com/pricing"
user_query = "Assist me perceive the pricing"
outcome = ai_web_scraper(url, user_query)

show(Markdown(outcome))

In a couple of seconds, we get a clear response that’s straightforward to grasp. As an alternative of manually visiting the pricing web page and looking for the related info, the scraper extracts the web page, cleans it, and asks the LLM to clarify solely what issues.

AI web scraper output summarizing pricing information

Scraper output for a pricing question | Picture by Writer

We will additionally save the ultimate response as a Markdown file.

with open("ai_scraper_result.md", "w", encoding="utf-8") as file:
    file.write(outcome)

print("Markdown saved to ai_scraper_result.md")

Output:

Markdown saved to ai_scraper_result.md

Now the result’s saved as a Markdown file, which you’ll be able to open, edit, share, or use in one other workflow.

# Closing Ideas

Constructing your personal AI instruments is far simpler now. With a couple of traces of Python and an LLM, we turned a traditional webpage right into a easy question-answering engine that may learn the web page, perceive the consumer question, and return a clear Markdown reply.

That is highly effective as a result of you don’t all the time want a fancy system to resolve a selected drawback. Typically, a small specialised resolution is sufficient.

However it is usually vital to keep in mind that every thing has a price. Working the app on a server prices cash. Calling an LLM prices cash. Sustaining the scraper, fixing damaged pages, dealing with errors, and enhancing the system over time additionally prices money and time.

So earlier than constructing your personal customized resolution, it’s price taking a look at present instruments like Olostep, Firecrawl, or Exa. In some instances, paying for a ready-made scraping or internet intelligence API could make extra sense. In different instances — particularly if the duty is small, native, or very particular — constructing your personal light-weight resolution could be the higher choice.

Abid Ali Awan (@1abidaliawan) is a licensed information scientist skilled who loves constructing machine studying fashions. At the moment, he’s specializing in content material creation and writing technical blogs on machine studying and information science applied sciences. Abid holds a Grasp’s diploma in know-how administration and a bachelor’s diploma in telecommunication engineering. His imaginative and prescient is to construct an AI product utilizing a graph neural community for college students battling psychological sickness.

Official Zlibrary Area: Search Engine Indexing Latency

Why Your Salon Enterprise Wants a Skilled App Growth Firm |