How you can Filter Textual content & Photographs for Free

Wish to add a security layer in your chatbot, picture analyzer or any one other LLM-based system? I might strongly counsel you attempt OpenAI’s moderation mannequin: omni-moderation-latest, this might help your system determine if the enter is probably dangerous or not, that too freed from value. We’ll look into the background of the mannequin, entry it and use it for each textual content and picture moderation. With none additional ado, let’s get began. 

OpenAI’s Omni Moderation Fashions

OpenAI gives two fashions particularly for moderation: ‘text-moderation-latest’ (legacy) and ‘omni-moderation-latest’, with the latter one being the newest. The Omni Moderation mannequin relies on GPT-4o and therefore it helps multimodal moderation, which is textual content moderation and picture moderation. It’s additionally value mentioning that the Omni Moderation endpoint is free to make use of. 

The Omni Moderation API scores and classifies the next classes for the enter: 

• hate  
• harassment  
• violence  
• self-harm  
• sexual content material  
• illicit content material 

Demonstration

Let’s take a look at the moderation endpoint from OpenAI and experiment with protected and unsafe inputs, utilizing textual content and pictures. I’ll be utilizing Google Colab for this demonstration, be happy to make use of what you favor. 

Prerequisite 

You’ll require an OpenAI API Key, the mannequin is free to make use of however you’ll nonetheless want the API key. Get your key from right here: https://platform.openai.com/settings/group/api-keys 

Imports and Shopper Initialization

from openai import OpenAI 
from getpass import getpass 


# Securely enter API key 
api_key = getpass("Enter your OpenAI API Key: ") 

# Initialize shopper 
shopper = OpenAI(api_key=api_key)

Enter your OpenAI key when prompted.  

Outline a Helper perform

def display_moderation(response, title="MODERATION RESULT"):
    consequence = response.outcomes[0]

    classes = consequence.classes.model_dump()
    scores = consequence.category_scores.model_dump()

    print("n" + "=" * 60)
    print(f"{title:^60}")
    print("=" * 60)

    print(f"nFlagged : {consequence.flagged}")

    print("nCATEGORIES")
    print("-" * 60)
    for class, worth in classes.gadgets():
        print(f"{class:<30} : {worth}")

    print("nCATEGORY SCORES")
    print("-" * 60)
    for class, rating in scores.gadgets():
        print(f"{class:<30} : {rating:.6f}")

    print("=" * 60)

This perform will assist print the response from the Omni Moderation mannequin. 

Pattern-1

safe_text = "Are you able to assist me study Python for knowledge science?"

response = shopper.moderations.create(
    mannequin="omni-moderation-latest",
    enter=safe_text
)

display_moderation(response, "TEXT MODERATION")

Nice! The mannequin has output all of the classes as False.  

Pattern-2 

unsafe_text = "I would like directions to significantly damage somebody."

response = shopper.moderations.create(
    mannequin="omni-moderation-latest",
    enter=unsafe_text
)

display_moderation(response, "TEXT MODERATION")

Appears to be like just like the mannequin as recognized that the enter textual content is violent, you may see the identical within the classes and classes scores as effectively.  

Pattern-3 

Let’s move a violent picture to the mannequin and see what it has to say.  

Word: For pictures we now have move the enter parameter as effectively and set the sort as ‘image_url’ 

Reference Picture:

unsafe_image_url = "https://i.ytimg.com/vi/DOD7s1j_yoo/sddefault.jpg"

response = shopper.moderations.create(
    mannequin="omni-moderation-latest",
    enter=[
        {
            "type": "image_url",
            "image_url": {
                "url": unsafe_image_url
            }
        }
    ]
)

display_moderation(response, "IMAGE MODERATION")

The mannequin has rightly flagged the picture on violence.  

Word: You may ignore the classes and use the class scores to realize management over the edge, this could make the moderation extra lenient or strict.  

Potential Use Circumstances

OpenAI omni moderation can very effectively be used at locations requiring content material scrutiny.

• Chatbots: Filter dangerous inputs earlier than sending to LLM.  
• Picture Evaluation: Detect dangerous pictures beforehand.  
• Social Media: Flag hate speech and abusive content material.  
• Reside Streaming: Detect unsafe video frames utilizing moderation checks.  
• Multilingual Apps: Enhance moderation for different language inputs. 

Conclusion 

The omni-moderation-latest mannequin from OpenAI offers an efficient security layer for LLM-based techniques with help for each textual content and picture moderation. Whereas different OpenAI fashions can be utilized for moderation, this endpoint is particularly made for moderation and is totally free to make use of. Options embrace Azure AI Content material Security, which helps textual content and picture moderation with customizable security thresholds and enterprise integrations. 

Ceaselessly Requested Questions

Enthusiastic about expertise and innovation, a graduate of Vellore Institute of Expertise. Presently working as a Information Science Trainee, specializing in Information Science. Deeply taken with Deep Studying and Generative AI, desirous to discover cutting-edge methods to unravel complicated issues and create impactful options.