The important thing steps within the workflow lie in structuring the transcript in paragraphs (step 2) earlier than grouping the paragraphs into chapters from which a desk of contents is derived (step 4). Notice that these two steps might depend on completely different LLMs: A quick and low-cost LLM akin to LLama 3 8B for the straightforward job of textual content enhancing and paragraph identification, and a extra subtle LLM akin to GPT-4o-mini for the technology of the desk of contents. In between, TF-IDF is used so as to add again timestamp data to the structured paragraphs.
The remainder of the submit describes every step in additional element.
Take a look at the accompanying Github repository and Colab pocket book to discover by yourself!
Allow us to use for example the primary lecture of the course ‘MIT 6.S191: Introduction to Deep Studying’ (IntroToDeepLearning.com) by Alexander Amini and Ava Amini (licensed below the MIT License).
Notice that chapters are already supplied within the video description.
This supplies us with a baseline to qualitatively examine our chaptering later on this submit.
YouTube transcript API
For YouTube movies, an routinely generated transcript is normally made obtainable by YouTube. A handy approach to retrieve that transcript is by calling the get_transcript technique of the Python youtube_transcript_api library. The tactic takes the YouTube video_id library as argument:
# https://www.youtube.com/watch?v=ErnWZxJovaM
video_id = "ErnWZxJovaM" # MIT Introduction to Deep Studying - 2024# Retrieve transcript with the youtube_transcript_api library
from youtube_transcript_api import YouTubeTranscriptApi
transcript = YouTubeTranscriptApi.get_transcript(video_id, languages=["en"])
This returns the transcript as an inventory of textual content and timestamp key-value pairs:
[{'text': '[Music]', 'begin': 1.17},
{'textual content': 'good afternoon everybody and welcome to', 'begin': 10.28},
{'textual content': 'MIT sus1 191 my title is Alexander amini', 'begin': 12.88},
{'textual content': "and I will be one in every of your instructors for", 'begin': 16.84},
...]
The transcript is nonetheless poorly formatted: it lacks punctuation and incorporates typos (‘MIT sus1 191’ as a substitute of ‘MIT 6.S191′, or ‘amini’ as a substitute of ‘Amini’).
Speech-to-text with Whisper
Alternatively, a speech-to-text library can be utilized to deduce the transcript from a video or audio file. We suggest utilizing faster-whisper, which is a quick implementation of the state-of-the-art open-source whisper mannequin.
The fashions come in numerous measurement. Essentially the most correct is the ‘large-v3’, which is ready to transcribe about quarter-hour of audio per minute on a T4 GPU (obtainable free of charge on Google Colab).
from faster_whisper import WhisperModel# Load Whisper mannequin
whisper_model = WhisperModel("large-v3",
gadget="cuda" if torch.cuda.is_available() else "cpu",
compute_type="float16",
)
# Name the Whisper transcribe perform on the audio file
initial_prompt = "Use punctuation, like this."
segments, transcript_info = whisper_model.transcribe(audio_file, initial_prompt=initial_prompt, language="en")
The results of the transcription is supplied as segments which will be simply transformed in an inventory of textual content and timestamps as with the youtube_transcript_api library.
Tip: Whisper might generally not embrace the punctuation. The initial_prompt argument can be utilized to nudge the mannequin so as to add punctuation by offering a small sentence containing punctuation.
Under is an excerpt of the transcription of the our video instance with whisper large-v3:
[{'start': 0.0, 'text': ' Good afternoon, everyone, and welcome to MIT Success 191.'},
{'start': 15.28, 'text': " My name is Alexander Amini, and I'll be one of your instructors for the course this year"},
{'start': 19.32, 'duration': 2.08, 'text': ' along with Ava.'}
...]
Notice that in comparison with the YouTube transcription, the punctuation is added. Some transcription errors nonetheless nonetheless stay (‘MIT Success 191’ as a substitute of ‘MIT 6.S191′).
As soon as a transcript is accessible, the second stage consists in enhancing and structuring the transcript in paragraphs.
Transcript enhancing refers to adjustments made to enhance readability. This entails, for instance, including punctuation whether it is lacking, correcting grammatical errors, eradicating verbal tics, and so on.
The structuring in paragraphs additionally improves readability, and additionnally serves as a preprocessing step for figuring out chapters in stage 4, since chapters shall be fashioned by grouping paragraphs collectively.
Paragraph enhancing and structuring will be carried out in a single operation, utilizing an LLM. We illustrated under the anticipated results of this stage:
This job doesn’t require a really subtle LLM because it largely consists in reformulating content material. On the time of writing this text, first rate outcomes might be obtained with for instance GPT-4o-mini or Llama 3 8B, and the next system immediate:
You’re a useful assistant.
Your job is to enhance the person enter’s readability: add punctuation if wanted and take away verbal tics, and construction the textual content in paragraphs separated with ‘nn’.
Maintain the wording as devoted as attainable to the unique textual content.
Put your reply inside
tags.
We depend on OpenAI appropriate chat completion API for LLM calling, with messages having the roles of both ‘system’, ‘person’ or ‘assistant’. The code under illustrates the instantiation of an LLM consumer with Groq, utilizing LLama 3 8B:
# Connect with Groq with a Groq API key
llm_client = Groq(api_key=api_key)
mannequin = "llama-8b-8192"# Extract textual content from transcript
transcript_text = ' '.be a part of([s['text'] for s in transcript])
# Name LLM
response = consumer.chat.completions.create(
messages=[
{
"role": "system",
"content": system_prompt
},
{
"role": "user",
"content": transcript_text
}
],
mannequin=mannequin,
temperature=0,
seed=42
)
Given a bit of uncooked ‘transcript_text’ as enter, this returns an edited piece of textual content inside
response_content=response.selections[0].message.content materialprint(response_content)
"""
Good afternoon, everybody, and welcome to MIT 6.S191. My title is Alexander Amini, and I will be one in every of your instructors for the course this 12 months, together with Ava. We're actually excited to welcome you to this unimaginable course.
It is a fast-paced and intense one-week course that we're about to undergo collectively. We'll be overlaying the foundations of a quickly altering discipline, and a discipline that has been revolutionizing many areas of science, arithmetic, physics, and extra.
Over the previous decade, AI and deep studying have been quickly advancing and fixing issues that we did not assume had been solvable in our lifetimes. As we speak, AI is fixing issues past human efficiency, and annually, this lecture is getting tougher and tougher to show as a result of it is imagined to cowl the foundations of the sphere.
"""
Allow us to then extract the edited textual content from the
import re
sample = re.compile(r'(.*?) ', re.DOTALL)
response_content_edited = sample.findall(response_content)
paragraphs = response_content_edited.strip().cut up('nn')
paragraphs_dict = [{'paragraph_number': i, 'paragraph_text': paragraph} for i, paragraph in enumerate(paragraphs)print(paragraph_dict)
[{'paragraph_number': 0,
'paragraph_text': "Good afternoon, everyone, and welcome to MIT 6.S191. My name is Alexander Amini, and I'll be one of your instructors for the course this year, along with Ava. We're really excited to welcome you to this incredible course."},
{'paragraph_number': 1,
'paragraph_text': "This is a fast-paced and intense one-week course that we're about to go through together. We'll be covering the foundations of a rapidly changing field, and a field that has been revolutionizing many areas of science, mathematics, physics, and more."},
{'paragraph_number': 2,
'paragraph_text': "Over the past decade, AI and deep learning have been rapidly advancing and solving problems that we didn't think were solvable in our lifetimes. Today, AI is solving problems beyond human performance, and each year, this lecture is getting harder and harder to teach because it's supposed to cover the foundations of the field."}]
Notice that the enter shouldn’t be too lengthy because the LLM will in any other case ‘neglect’ a part of the textual content. For lengthy inputs, the transcript have to be cut up in chunks to enhance reliability. We observed that GPT-4o-mini handles nicely as much as 5000 characters, whereas Llama 3 8B can solely deal with as much as 1500 characters. The pocket book supplies the perform transcript_to_paragraphs which takes care of splitting the transcript in chunks.
The transcript is now structured as an inventory of edited paragraphs, however the timestamps have been misplaced within the course of.
The third stage consists in including again timestamps, by inferring which section within the uncooked transcript is the closest to every paragraph.
We rely for this job on the TF-IDF metric. TF-IDF stands for time period frequency–inverse doc frequency and is a similarity measure for evaluating two items of textual content. The measure works by computing the variety of related phrases, giving extra weight to phrases which seem much less incessantly.
As a preprocessing step, we modify the transcript segments and paragraph beginnings in order that they comprise the identical variety of phrases. The textual content items ought to be lengthy sufficient in order that paragraph beginnings will be efficiently matched to a novel transcript section. We discover that utilizing 50 phrases works nicely in follow.
num_words = 50transcript_num_words = transform_text_segments(transcript, num_words=num_words)
paragraphs_start_text = [{"start": p['paragraph_number'], "textual content": p['paragraph_text']} for p in paragraphs]
paragraphs_num_words = transform_text_segments(paragraphs_start_text, num_words=num_words)
We then depend on the sklearn library and its TfidfVectorizer and cosine_similarity perform to run TF-IDF and compute similarities between every paragraph starting and transcript section. under is an instance of code for locating the most effective match index within the transcript segments for the primary paragraph.
from sklearn.feature_extraction.textual content import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity# Paragraph for which to search out the timestamp
paragraph_i = 0
# Create a TF-IDF vectorizer
vectorizer = TfidfVectorizer().fit_transform(transcript_num_words + paragraphs_num_words)
# Get the TF-IDF vectors for the transcript and the excerpt
vectors = vectorizer.toarray()
# Extract the TF-IDF vector for the paragraph
paragraph_vector = vectors[len(transcript_num_words) + paragraph_i]
# Calculate the cosine similarity between the paragraph vector and every transcript chunk
similarities = cosine_similarity(vectors[:len(transcript_num_words)], paragraph_vector.reshape(1, -1))
# Discover the index of essentially the most related chunk
best_match_index = int(np.argmax(similarities))
We wrapped the method in a add_timestamps_to_paragraphs perform, which provides timestamps to paragraphs, along with the matched section index and textual content:
paragraphs = add_timestamps_to_paragraphs(transcript, paragraphs, num_words=50)#Instance of output for the primary paragraph:
print(paragraphs[0])
{'paragraph_number': 0,
'paragraph_text': "Good afternoon, everybody, and welcome to MIT 6.S191. My title is Alexander Amini, and I will be one in every of your instructors for the course this 12 months, together with Ava. We're actually excited to welcome you to this unimaginable course.",
'matched_index': 1,
'matched_text': 'good afternoon everybody and welcome to',
'start_time': 10}
Within the instance above, the primary paragraph (numbered 0) is discovered to match the transcript section number one that begins at time 10 (in seconds).
The desk of content material is then discovered by grouping consecutive paragraphs into chapters and figuring out significant chapter titles. The duty is usually carried out by an LLM, which is instructed to rework an enter consisting in an inventory of JSON paragraphs into an output consisting in an inventory of JSON chapter titles with the beginning paragraph numbers:
system_prompt_paragraphs_to_toc = """You're a useful assistant.
You're given a transcript of a course in JSON format as an inventory of paragraphs, every containing 'paragraph_number' and 'paragraph_text' keys.
Your job is to group consecutive paragraphs in chapters for the course and establish significant chapter titles.
Listed here are the steps to comply with:
1. Learn the transcript fastidiously to grasp its common construction and the principle subjects coated.
2. Search for clues {that a} new chapter is about to begin. This might be a change of matter, a change of time or setting, the introduction of latest themes or subjects, or the speaker's express point out of a brand new half.
3. For every chapter, preserve observe of the paragraph quantity that begins the chapter and establish a significant chapter title.
4. Chapters ought to ideally be equally spaced all through the transcript, and focus on a particular matter.
Format your end in JSON, with an inventory dictionaries for chapters, with 'start_paragraph_number':integer and 'title':string as key:worth.
Instance:
{"chapters":
[{"start_paragraph_number": 0, "title": "Introduction"},
{"start_paragraph_number": 10, "title": "Chapter 1"}
]
}
"""
An necessary ingredient is to particularly ask for a JSON output, which will increase the probabilities to get a accurately formatted JSON output that may later be loaded again in Python.
GPT-4o-mini is used for this job, as it’s cheaper than OpenAI’s GPT-4o and usually supplies good outcomes. The directions are supplied by means of the ‘system’ position, and paragraphs are supplied in JSON format by means of the ‘person’ position.
# Connect with OpenAI with an OpenAI API key
llm_client_get_toc = OpenAI(api_key=api_key)
model_get_toc = "gpt-4o-mini-2024-07-18"# Dump JSON paragraphs as textual content
paragraphs_number_text = [{'paragraph_number': p['paragraph_number'], 'paragraph_text': p['paragraph_text']} for p in paragraphs]
paragraphs_json_dump = json.dumps(paragraphs_number_text)
# Name LLM
response = client_get_toc.chat.completions.create(
messages=[
{
"role": "system",
"content": system_prompt_paragraphs_to_toc
},
{
"role": "user",
"content": paragraphs_json_dump
}
],
mannequin=model_get_toc,
temperature=0,
seed=42
)
Et voilà! The decision returns the checklist of chapter titles along with the beginning paragraph quantity in JSON format:
print(response){
"chapters": [
{
"start_paragraph_number": 0,
"title": "Introduction to the Course"
},
{
"start_paragraph_number": 17,
"title": "Foundations of Intelligence and Deep Learning"
},
{
"start_paragraph_number": 24,
"title": "Course Structure and Expectations"
}
....
]
}
As in step 2, the LLM might battle with lengthy inputs and dismiss a part of the enter. The answer consists once more in splitting the enter into chunks, which is applied within the pocket book with the paragraphs_to_toc perform and the chunk_size parameter.
This final stage combines the paragraphs and the desk of content material to create a structured JSON file with chapters, an instance of which is supplied within the accompanying Github repository.
We illustrate under the ensuing chaptering (proper), in comparison with the baseline chaptering that was obtainable from the YouTube description (left):
