Exploring Music Transcription with Multi-Modal Language Fashions | by Jon Flynn | Nov, 2024

Utilizing Qwen2-Audio to transcribe music into sheet music

Automated music transcription is the method of changing audio information like MP3 and WAV into sheet music, guitar tablature, and any format a musician might need to be taught a tune on their instrument.

We’ll go over one of the best present instruments for doing this, which occur to be deep learning-based, and a novel strategy for it.

The present state-of-the-art for this activity comes from Magenta, an open-source analysis mission developed by the now defunct (as of April 2023) Google Mind Staff.

They launched a paper Sequence-to-Sequence Piano Transcription with Transformers in 2021 which used a T5-inspired transformer mannequin (much like “t5-small”) with 54 million parameters and the Maestro dataset, reaching nice outcomes. The issue is approached as a sequence-to-sequence activity utilizing an encoder-decoder Transformer structure. The encoder processes mel spectrogram frames as enter and produces embeddings, whereas the decoder makes use of these embeddings by way of cross-attention to autoregressively generate a sequence of MIDI-like tokens. Their vocabulary consisted of 4 varieties of tokens:

• Observe tokens (128 values for MIDI pitches)
• Velocity tokens (128 values together with zero for note-off)
• Time tokens (6,000 values in 10ms bins for absolute timing)
• EOS token (to mark sequence finish)

See the picture beneath for a visualisation of the structure and an instance sequence of their customized MIDI tokens:

Our mannequin is a generic encoder-decoder Transformer structure the place every enter place comprises a single spectrogram body and every output place comprises an occasion from our MIDI-like vocabulary. Outputs tokens are autoregressively sampled from the decoder, at every step taking the token with most chance.

In 2022, they launched a paper, MT3: Multi-Job Multitrack Music Transcription. This experiment used the identical strategy because the final one however added further instrument tokens to characterize the totally different devices. Once more, they used the same T5 mannequin and achieved nice efficiency towards most of the datasets educated on, notably Slakh, Maestro and MusicNet.

MR-MT3 was launched the next yr as a slight enchancment to MT3.

Compute/GPU sources

Enormous sources had been wanted to coach this from scratch, regardless of being a lot smaller in dimension in comparison with even the smallest language fashions. The 2021 paper famous:

“We educated all fashions on 32 TPUv3 cores, leading to a per-core batch dimension of 8. Primarily based on validation set outcomes, overfitting didn’t appear to be an issue, so we allowed coaching to progress for 400K steps, which took about 2.5 days for our baseline fashions.”

The MT3 paper doesn’t present as particular particulars on coaching, stating they prepare for 1 million steps.

Different limitations

These fashions have some inherent limitations of their output flexibility. Whereas language fashions usually have giant vocabularies (usually 30,000+ tokens) which can be extensively pre-trained on numerous pure language information, MT3 and related music transcription fashions use a a lot smaller, specialised token vocabulary (only some thousand tokens) targeted solely on musical occasions. This specialisation signifies that including new tokens, resembling for brand new devices or enjoying strategies like palm muting on guitars or pizzicato on violins, is probably going not simple — it requires important retraining to combine these new tokens successfully with the present vocabulary, and infrequently requires substantial coaching information demonstrating these strategies. This differs from giant language fashions which might usually describe such musical nuances in pure language with out modification, as they’ve encountered these ideas throughout their broad pre-training.

Switch studying and zero-shot

We are able to leverage switch studying from giant open-source pre-trained audio and language fashions. Examples of music era fashions embrace OpenAI’s Jukebox and Meta’s MusicGen.

GPT-4o is designed to deal with textual content, audio and pictures “natively”. Though OpenAI has not launched the technical particulars on this, it’s assumed that some weights within the community will course of all modalities. It’s potential that the mannequin makes use of a decoder-only structure like language solely GPT fashions with out the necessity for encoder parts to transform totally different modalities to a dense illustration first. This design permits the mannequin to seamlessly course of and interpret inputs like textual content and pictures collectively, probably providing efficiency advantages each computationally and by way of mannequin understanding.

Many multi-modal fashions take a less complicated strategy harking back to the encoder-decoder structure: they mix two pre-trained fashions — an encoder for the particular enter modality (like ViT for imaginative and prescient or an audio encoder for sound) and a Giant Language Mannequin (resembling LLaMA, Gemma, or Qwen). These fashions are related via projection layers that align their representations in a shared latent house, usually utilizing only a single linear layer. These projection layers be taught to transform the encoder’s output right into a format that matches the LLM’s anticipated enter dimensions and traits. The projection creates new embeddings/tokens from the enter modality that may then be injected into the LLM’s enter sequence. LLaVA is a major instance of this structure for vision-language duties, whereas Spotify’s Llark and Qwen-Audio apply the identical precept utilizing audio encoders as a substitute of imaginative and prescient encoders.

Right here’s some pseudocode on how the fashions are stitched collectively:

# Extract options from closing layer of audio encoder
# Form: [batch_size, audio_seq_len, encoder_dim=1024]
audio_features = audio_model(audio_input)
# Venture audio options to match LLM's embedding dimension
# Form: [batch_size, audio_seq_len, llm_embed_dim=4096]
audio_embeddings = projection_layer(audio_features)
# Get textual content embeddings from LLM's embedding layer
# Form: [batch_size, text_seq_len, llm_embed_dim=4096]
text_embeddings = llm.embed_text(text_input)
# Concatenate alongside sequence size dimension
# Form: [batch_size, audio_seq_len + text_seq_len, llm_embed_dim=4096]
combined_input = concatenate([audio_embeddings, text_embeddings], dim=1)
# Feed them into the LLM as regular for era
output = llm(combined_input)

Overview of structure

Llark makes use of OpenAI’s Jukebox and Qwen2-Audio makes use of OpenAI’s Whisper for the audio towers. Jukebox is a music era mannequin however it will possibly additionally soak up audio clips as enter and outputs a continuation of the audio clip. Whisper is used for transcribing voice to textual content.

Given their function, the selection of audio module is obvious: Llark specialises in music evaluation, whereas Qwen2Audio primarily focuses on responding to voice directions with some primary audio and music evaluation capabilities.

Figuring out the optimum supply for extracting embeddings from giant pre-trained fashions entails analysis and experimentation. Moreover, deciding whether or not to fine-tune your entire module or freeze components of it’s a essential design alternative. For example, LlaVa’s coaching technique entails freezing the imaginative and prescient tower and specializing in fine-tuning the projection layer and language mannequin. We’ll go over this facet of every mannequin beneath.

Llark: why Jukebox? Are these embeddings one of the best as of September 2024?

Figuring out the optimum location to extract embeddings from giant fashions usually requires intensive probing. This entails testing varied activations or extracted layers of the mannequin on totally different classification duties via a strategy of trial and error. For music era fashions, this might embrace duties like style recognition, instrument detection, emotion detection, in addition to evaluation of harmonic constructions and temporal patterns. Many business embedding fashions (like OpenAI’s embedding fashions) are educated particularly for embedding era with specialised architectures and coaching aims, slightly than being fine-tuned variations of current language fashions.

The 2 largest publicly accessible music era and music continuation (i.e.: ready to soak up audio as enter) fashions are Jukebox and MusicGen. MusicGen is newer and sooner, and due to this fact appeared like it could be the plain option to me. Nevertheless, in line with this paper on probing MusicGen, embeddings extracted from Jukebox seem to outperform MusicGen on common in classification duties. The findings from this paper led to the authors of Llark utilizing the next strategy for extracting embeddings:

1. Embeddings are derived from the output of the thirty sixth layer of the Jukebox encoder following the strategy described in Castellon et al. (2021)
2. Authentic Jukebox encoding:
   * 4800-dimensional vectors at 345Hz
   * For a 25s clip: over 4.14 * 10⁷ floating-point values
3. The authors use a downsampling strategy: Imply-pooling inside 100ms frames, leading to:
   * Downsampled frequency: 10Hz
   * Embedding dimension: 1.2 × 10⁶ for a 25s audio clip. Meaning a 2D array with form [240, 4800].
   * Retains temporal data (in contrast to Castellon et al. who common over the time dimension)

(The downsampled embedding dimension is roughly 6x bigger than CLIP ViT-L14 fashions utilized in many multimodal imaginative and prescient fashions)

Qwen2Audio: Whisper

The embedding extraction for Qwen2Audio isn’t talked about intimately within the paper. Whisper is an encoder-decoder structure the place the encoder generates deeply discovered representations of the audio and the decoder decodes the representations to textual content (the transcription). In Qwen2Audio, it seems they extract embeddings from the ultimate layer of Whisper’s encoder, though they don’t point out whether or not they freeze it throughout coaching.

Pre-trained weights, coaching information and datasets

Sadly Spotify has not supplied any datasets or their educated mannequin weights to the general public, noting:

“With respect to inputs: the inputs to our mannequin are public, open-source, Artistic Commons-licensed audio and related annotations. Nevertheless, every particular person audio file can have its personal, probably extra restrictive license. Most of the audio information embrace “no derivatives” licenses. We encourage customers of the datasets to familiarize themselves with the restrictions of those licenses; with a purpose to honor such licenses, we don’t launch any derivatives from the coaching information on this paper (together with query- response pairs or educated mannequin weights).”

They used the next datasets:

• MusicCaps (Agostinelli et al., 2023)
• YouTube8M-MusicTextClips (McKee et al., 2023)
• MusicNet (Thickstun et al., 2017)
• FMA (Defferrard et al., 2017)
• MTG-Jamendo (Bogdanov et al., 2019)
• MagnaTagATune (Regulation et al., 2009)

Llark particulars it’s coaching information era course of within the following extract:

“We use variants of ChatGPT to extract the instruction- tuning information for all experiments. Nevertheless, the precise language mannequin used varies by dataset. We choose the OpenAI mannequin as follows: We use GPT-4 for all reasoning duties. We discovered that GPT-4 was far more adept at following the complicated directions within the Reasoning activity household. For datasets with greater than 25k samples, we restrict Reasoning information to a random subsample of 25k tracks.”

This leads to Q&An information like this:

The datasets used for coaching Qwen2Audio should not shared both, however the educated mannequin is broadly accessible and in addition is applied within the transformers library:

For this mission, fine-tuning off a pre-trained Llark mannequin would have been optimum, given it’s reportedly good efficiency towards the analysis benchmarks Spotify acknowledged within the paper.

Nevertheless, given they didn’t launch the weights for it, it’s unfeasible to begin coaching a mannequin like this from scratch with no good bit of experience and cash. Spotify educated it on:

Our mannequin is educated on 4 80GB NVIDIA A100 GPUs. Coaching takes roughly 54 hours.

This is able to price round $700 utilizing a supplier like LambdaLabs.

Due to the above, I went with Qwen. Nevertheless, Qwen2-Audio doesn’t carry out that properly throughout primary music duties like tempo and instrument detection. I element this beneath within the analysis part. Which means the mannequin might be not giant sufficient or pre-trained sufficient to realize this activity, however my hope is I may a minimum of set a place to begin and framework for fine-tuning on this activity sooner or later. As Alibaba state of their Qwen2-Audio weblog submit:

We additionally plan to construct bigger Qwen2-Audio fashions to discover the scaling legal guidelines of audio language fashions.

For my very own studying although, I did have a go at re-creating the mannequin utilizing torch and pre-trained fashions with the transformers library.

I additionally created datasets for Q&An information and embeddings. I generated brief kind Q&An information for the URMP dataset, e.g.: “What’s the tempo of this observe”, “What devices are enjoying on this audio”.

Right here’s a pocket book for operating Jukebox in a Colab surroundings to make the most of a budget T4 GPU’s. I uploaded each Q&A and embeddings datasets to HuggingFace right here.

Right here’s a pocket book with Llark replicated.

Transcription format

I selected ABC music notation because the output format that the language mannequin is anticipated to transcribe the music in. Right here’s an instance of it:

X:1
M:4/4
L:1/16
Okay:none
Q:67
V:1 title="Electrical Bass (finger)"
%%octave-default C4
GAA^2E3A2A^4 A^2E2 A2A^4A^2 E2 | A2A^4 |
V:2 title="Shiny Acoustic Piano"
%%octave-default C5
[E3C3][E3C3][E3C3] [E3C3][A^,2E2A^2] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3][E3A^3][E3A^3][E3A^3][E3A^3] |
[E3A^3][E3A^3][E3A^3] [E3A^3][E3A^3] | [E3A^3] |
V:3 title="Electrical Guitar (jazz)"
%%octave-default C5
E'3C'3A^4E'3C'3 | A^4E'3 C'3A^4E'3C'3 | A^4 E'3C'3A^4 E'3C'3 | A^4E'3C'3A^4E'3C'3 |
A^4E'3C'3 A^4E'3C'3 | A^4 |

I transformed the MIDI information supplied by the datasets to ABC notation utilizing this library. A pocket book for creating the datasets is right here.

To guage each the unique mannequin and every stage of fine-tuning I carried out thereafter, I randomly chosen 30 samples of various complexity from the URMP dataset and ran the mannequin 3 times on every pattern, manually analyzing all responses.

By handbook testing, I discovered the optimum decoding parameters to be a temperature of 0.7 and a top_p of 1.2. The utmost variety of tokens to return was capped at 2048. Adjusting the max appeared to have little distinction on efficiency.

The unique mannequin carried out poorly on this analysis set. Whereas it sometimes predicted the tempo and devices appropriately, it largely failed to take action. A textual content file with the analysis outcomes is on the market right here.

Given this place to begin, it’s unlikely that we’ll see robust outcomes from this experiment with no sturdy pre-trained mannequin. Nevertheless, the objective is to develop methods that may be utilized sooner or later as extra superior pre-trained fashions turn out to be accessible.

I first tried fine-tuning with primary cross-entropy loss. Supervised fine-tuning with cross-entropy loss is a fast approach to begin educating the mannequin however a primary loss operate like this has limitations as we’ll see beneath. The instinct behind this stage of coaching is that it could nudge the mannequin in the suitable path and it could choose up any patterns or any customised ABC notation the dataset might have which the mannequin might not have seen earlier than.

Cross-entropy loss with trainer forcing

First, we educated it in a typical supervised fine-tuning method for language fashions. I used the SFTtrainer from the trl library for this, which makes use of cross-entropy loss with trainer forcing outlined step-by-step beneath:

1. The mannequin predicts the subsequent token within the sequence.
2. The loss is calculated based mostly on the distinction between the expected possibilities (logits) and the precise subsequent token.
3. For the subsequent prediction, the mannequin is given the precise right token (floor fact), slightly than its personal prediction. This is named trainer forcing, it helps stabilise coaching and considerably pace it up, particularly within the early levels.

The outcomes from this coaching part had been poor. It degraded the efficiency of the unique mannequin. The mannequin, which beforehand dealt with tempo and instrument recognition properly, now largely bought these fallacious. It additionally started producing garbled textual content output with infinite repetition. This occurred even when setting a low studying price, making use of gradient clipping, and utilizing low LoRA ranks to mitigate giant adjustments to the mannequin. Total, it appeared the mannequin was very delicate to the coaching utilized.

Nevertheless, whereas this coaching part might provide some enhancements, it received’t result in optimum efficiency as a result of limitations of our primary loss operate. This operate struggles to totally seize the mannequin’s efficiency nuances. For instance, when utilizing trainer forcing, instrument predictions can yield deceptively low loss throughout sure token sections. If an instrument title begins with “V”, the mannequin would possibly confidently predict “Violin” or “Viola” based mostly on our dataset, no matter accuracy. Moreover, the loss operate might not precisely replicate near-misses, resembling predicting a tempo of 195 as a substitute of 200 — a small distinction that’s moderately correct however probably penalised closely depending on the distribution of possibilities amongst logits. It’s potential that neighbouring numbers even have excessive possibilities.

RLHF with PPO

Due to these limitations, we are able to create our personal customized loss operate that may extra precisely rating the response from the mannequin. That’s, given a predicted sequence from the mannequin, the loss operate may give it a rating between 0 and 1 on how good it’s.

Nevertheless, integrating this tradition loss operate into supervised fine-tuning presents a major problem. The problem stems from the non-linearity launched by the customized loss operate, which prevents the direct calculation of gradients. Let’s break this down:

In conventional SFT with cross-entropy loss:

• The mannequin outputs logits (uncooked scores) for every token in its vocabulary
• These logits instantly characterize the mannequin’s prediction possibilities
• The loss operate compares these possibilities to the bottom fact
• Gradients may be computed instantly via this comparability
• The chain rule of calculus permits us to propagate these gradients again via the mannequin

With our customized loss operate:

• The mannequin should first generate full textual content output
• This era course of entails sampling from chance distributions
• Our loss operate then analyses this textual content output (checking tempo, notes, and so forth.)
• This creates a non-differentiable step between the mannequin’s logits and our loss calculation
• The sampling and textual content evaluation steps break the gradient chain wanted for backpropagation

To beat this, reinforcement studying strategies like Proximal Coverage Optimisation (PPO) may be employed. PPO is particularly designed to deal with non-differentiable loss features and might optimise the mannequin by contemplating your entire coverage (the mannequin’s output distribution), slightly than counting on gradient data from logits.

Observe, there’s a lot of nice articles on right here explaining PPO!

The important thing perception of PPO is that as a substitute of attempting to instantly backpropagate via the non-differentiable steps, it:

1. Treats the mannequin’s outputs as actions in a reinforcement studying framework
2. Makes use of the customized loss operate as a reward sign
3. Updates the mannequin’s coverage (its chance distributions over tokens) to maximise anticipated reward
4. Does this whereas guaranteeing the up to date coverage doesn’t deviate too removed from the present one

This strategy permits us to successfully prepare the mannequin with the customized loss operate, guaranteeing efficiency enhancements with out disrupting the core coaching dynamics. The PPO algorithm’s conservative replace technique helps keep stability throughout coaching, which is especially vital when working with giant language fashions.

Often, this scoring operate could be applied as a separate LLM within the type of a “reward mannequin” generally used when fine-tuning fashions by way of RLHF, which was a breakthrough first launched when ChatGPT got here out. Because of the nature of this activity, we are able to manually write code to attain the responses, which makes use of fewer sources and is faster.

For time signature and tempo recognition that is simple to calculate. We extract all predicted objects with regex, for instance extracting the metre:

def extract_metre(self, abc_string):
return re.search(r'M:(S+)', abc_string).group(1)

The mannequin ought to be taught the syntax and construction we would like it to output within the SFT stage. If it outputs one thing that may trigger our regex to not discover something or error, we are able to simply skip that pattern, assuming it’s a small minority of the dataset.

We extract the expected tempo and write a operate that’s extra forgiving for small errors however penalises bigger errors extra closely:

• For small variations (≤10 BPM), it makes use of linear scaling.
• For bigger variations, it switches to exponential scaling.
• The ultimate loss is capped between 0 and 1.

Let’s break down the important thing parts of this tradition loss:

Code for the customized loss is right here

1. Metre Loss

The metre loss focuses on the time signature of the piece. It compares the expected metre with the bottom fact, contemplating each the numerator and denominator individually, in addition to their ratio. This strategy permits for a nuanced analysis that may deal with varied time signatures precisely.

The metre loss makes use of a mixture of linear and exponential scaling to penalise variations. Small discrepancies end in a linear improve in loss, whereas bigger variations result in an exponential improve, capped at a most worth of 1.

2. Tempo Loss

Tempo loss evaluates the accuracy of the expected beats per minute (BPM). Just like the metre loss, it makes use of a mixture of linear and exponential scaling.

For small tempo variations (≤10 BPM), the operate applies linear scaling. Bigger variations set off exponential scaling, guaranteeing that important tempo mismatches are penalised extra closely.

3. Pitch Loss

The pitch loss is probably probably the most essential element, because it assesses the accuracy of the transcribed notes. This operate makes use of the Levenshtein distance to match the sequence of notes in every voice.

The pitch loss calculation accounts for a number of voices, matching every predicted voice to the closest floor fact voice. This strategy permits for flexibility in voice ordering whereas nonetheless sustaining accuracy within the total pitch content material.

4. Instrument Loss

The instrument loss evaluates the accuracy of instrument choice for every voice.

This operate considers precise matches, devices from the identical household, and makes use of string similarity for extra nuanced comparisons. It gives a complete evaluation of how properly the mannequin identifies and assigns devices to every voice.

5. Combining the Losses

The ultimate loss is a weighted mixture of those particular person parts:

total_loss = (0.5 * pitch_loss +
0.15 * metre_loss +
0.15 * tempo_loss +
0.2 * instrument_loss)

This weighting scheme prioritises pitch accuracy whereas nonetheless contemplating different vital points of music transcription.

PPO coaching typically requires much more reminiscence than SFT for just a few causes:

1. A number of coverage evaluations — PPO wants to keep up each the present coverage (mannequin weights) and an “outdated” coverage to compute the chance ratio between them. This successfully doubles the mannequin parameters in reminiscence.
2. Expertise buffer — PPO shops a buffer of experiences (states, actions, rewards, and so forth.) to carry out updates in mini-batches. This buffer may be fairly giant and takes important reminiscence.
3. Benefit estimation — Computing benefits requires conserving observe of worth estimates and returns throughout trajectories, including one other layer of reminiscence overhead.
4. Extra optimisation aims — PPO tracks a number of loss parts (coverage loss, worth loss, entropy bonus) and their gradients, whereas SFT has a single loss.

Due to the above, we’re extra restricted than SFT within the dimension of the fashions we are able to prepare and the way a lot it prices. Whereas the above coaching I may do on an A100 40GB in Colab, for the PPO coaching I wanted extra reminiscence. I educated on an H100 80GB, which may prepare a LoRA with a rank of 128 and a batch dimension of 8.

My hyperparameter sweep was slender, I went with what appeared most intuitive utilizing batch sizes starting from 1 to 16 and studying charges from 2e-5 to 2e-4.

The mannequin made no enhancements to the duty. The textual content file with the outcomes is right here.

I tracked varied coaching metrics utilizing Weights & Biases (WandB). Key metrics included the coverage loss, worth loss, complete loss, KL divergence, and the reward mannequin’s rating.

For all hyperparameter runs, the logs no enchancment within the rewards and loss calculated over time. The KL divergence remained throughout the pre-defined threshold.