When LLMs Attempt to Cause: Experiments in Textual content and Imaginative and prescient-Primarily based Abstraction

fashions study to cause abstractly from just some examples? On this piece, I discover this query by testing each text-based (o3-mini) and image-capable (gpt-4.1) fashions on summary grid transformation duties. These experiments reveal the extent to which present fashions depend on sample matching, procedural heuristics, and symbolic shortcuts fairly than sturdy generalization. Even with multimodal inputs, reasoning usually breaks down within the face of delicate abstraction. The outcomes supply a window into the present capabilities and limitations of in-context meta-learning with LLMs.

Introduction

Meta-learning, the flexibility of a system to discover ways to study, has historically been explored by gradient-based optimization, memory-augmented networks, or express job embeddings. However with the rise of enormous language fashions (LLMs), notably the o3 household with superior reasoning capabilities, a brand new query emerges: can we use LLMs themselves as meta-learners in task-based domains like ARC? The Abstraction and Reasoning Corpus (ARC), launched by François Chollet, is a benchmark explicitly designed to check broad generalization. It gives input-output transformation puzzles with minimal supervision, few examples per job, and sometimes no shared surface-level construction throughout duties. In different phrases: a playground for meta-learning. To get an understanding of typical abstraction and reasoning duties, the reader can go to the ARC play web page.

Knowledge and Setup

To discover whether or not LLMs like o3-mini can carry out meta-learning on summary reasoning duties, I used information from the ARC Prize 2025 Kaggle competitors. The dataset repository will be discovered right here (Apache 2.0 license). The dataset consists of input-output grid transformations that problem fashions to deduce summary guidelines from just some examples.

Every job gives:

• A number of coaching examples (enter and output 2D grids)
• A single check enter grid for which the mannequin should predict the corresponding output

A second dataset gives the answer grids for every of the check enter grids. Right here’s a simplified instance of the information format:

# coaching examples - dictionary of dictionaries. 
# Right here is an extracted job
{'practice': [{'input': [[6, 6, 0], [6, 0, 0], [0, 6, 6]],
   'output': [[6, 6, 0, 6, 6, 0, 0, 0, 0],
    [6, 0, 0, 6, 0, 0, 0, 0, 0],
    [0, 6, 6, 0, 6, 6, 0, 0, 0],
    [6, 6, 0, 0, 0, 0, 0, 0, 0],
    [6, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 6, 6, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 6, 6, 0, 6, 6, 0],
    [0, 0, 0, 6, 0, 0, 6, 0, 0],
    [0, 0, 0, 0, 6, 6, 0, 6, 6]]},
  {'enter': [[4, 0, 4], [0, 0, 0], [0, 4, 0]],
   'output': [[4, 0, 4, 0, 0, 0, 4, 0, 4],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 4, 0, 0, 0, 0, 0, 4, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 4, 0, 4, 0, 0, 0],
    [0, 0, 0, 0, 0, 0, 0, 0, 0],
    [0, 0, 0, 0, 4, 0, 0, 0, 0]]},...,
   'check': [{'input': [[7, 0, 7], [7, 0, 7], [7, 7, 0]]}]
}

# instance of resolution to check enter grid - dictionary of lists
# Right here is the extracted resolution for the one check enter grid above
[[[3, 2, 3, 2, 3, 2],
  [7, 8, 7, 8, 7, 8],
  [2, 3, 2, 3, 2, 3],
  [8, 7, 8, 7, 8, 7],
  [3, 2, 3, 2, 3, 2],
  [7, 8, 7, 8, 7, 8]]]

Every grid is a 2D array of integers from 0–9, representing coloured pixels. Grids have various sizes and a grid transformation may additionally carry a measurement change from the enter to the output grid. To visualise the arrays, I used a customized colormap with matplotlib:

from matplotlib import colours
cmap = colours.ListedColormap([

    '#8B00FF',  # Violet
    '#4B0082',  # Indigo
    '#0000FF',  # Blue
    '#FFFF00',  # Yellow
    '#00FF00',  # Green
    '#FF7F00',  # Orange
    '#FF0000',  # Red
    '#964B00',  # Golden
    '#000000',  # Black
    '#FFFFFF',  # White
])
norm = colours.Normalize(vmin=0, vmax=9)

# Operate to visualise an array
def visualize_matrix(matrix, title='', cmap=cmap, norm=norm):
    plt.imshow(matrix, cmap=cmap, norm=norm)
    plt.title(title)
    plt.axis('off')  # Take away axes
    plt.present()

For mannequin interplay, I used OpenAI’s o3-mini mannequin through LangChain. In a while, we will even use gpt-4.1:

from langchain_openai import ChatOpenAI
import getpass
import os

# Immediate for a secret enter
openai_key = getpass.getpass("Enter your OpenAI API key: ")

os.environ["OPENAI_API_KEY"] = openai_key

AGENT_MODEL = "o3-mini"  # reasoning mannequin, https://platform.openai.com/docs/fashions
AGENT_LLM = ChatOpenAI(mannequin=AGENT_MODEL) 
# AGENT_LLM = ChatOpenAI(mannequin=AGENT_MODEL, reasoning_effort='low')

To deal with LLM responses, particularly when the mannequin returns a predicted output grid as Python code inside triple backticks, I wrote a utility:

import re, ast

def extract_python_code(response_string):
    match = re.search(r"```pythons*(.*?)```", response_string, re.DOTALL)
    if match:
        return ast.literal_eval(match.group(1).strip())
    return None

This setup allowed me to construction a full reasoning loop: immediate the mannequin with few-shot examples, extract and apply a generated algorithm, assess its efficiency on new check inputs and eventually use the evaluation to enhance the algorithm.

Testing Reasoning with o3-mini

To judge whether or not LLMs can “meta-learn” on summary reasoning duties, I examined the o3-mini mannequin utilizing a closed-loop reasoning setup impressed by how people would possibly strategy few-shot duties. For every ARC problem, I offered the mannequin with a handful of demonstration input-output grid pairs and requested it to derive a single reusable algorithm.

I outlined a sequence of prompts utilizing LangChain’s ChatPromptTemplate to simulate reasoning, software, evaluation, and refinement. The method mimics an interior coaching loop with restricted supervision:

• PROMPT_REASON: The mannequin is given coaching examples and requested to deduce a common algorithm in pseudocode.
• PROMPT_SOLVE: The generated algorithm is utilized to new inputs (each coaching and check).
• PROMPT_ASSESS: When the algorithm fails, the mannequin receives suggestions evaluating its predicted vs. anticipated outputs.
• PROMPT_SUMMARIZE_FEEDBACK: The mannequin summarizes cumulative suggestions from failed makes an attempt to iteratively refine its strategy.

from langchain_core.prompts import ChatPromptTemplate

PROMPT_REASON = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in solving abstract reasoning tasks. "
            "You will be given several demonstration input-output pairs of 2D arrays. "
            "Your goal is to develop a single algorithm that maps each input array to its corresponding output array.nn"
            
            "Each input and output is a 2-dimensional array of integers between 0 and 9. "
            "Solving the task involves:n"
            "- Analyzing the demonstration pairsn"
            "- Identifying abstract patterns or transformationsn"
            "- Formulating a general rule or algorithm that works across all examplesn"
            "- Producing pseudocode that implements the rulenn"
            
            "If prior attempts were made, you will also receive feedback summarizing what went wrong. "
            "Carefully use this feedback to improve your solution.nn"
            
            "Return only the updated algorithm as pseudocode. Do not describe or explain it.nn"
            "### Feedback (summary of previous attempts):n{attempt_history}nn"
            "### Demonstration Pairs:n{train_pairs}n"
        ),
        (
            "ai", 
            "Answer:"
        )
    ]
)

PROMPT_SOLVE = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in abstract reasoning. "
            "Previously, you analyzed demonstration input-output pairs and developed an algorithm "
            "to transform input arrays into output arrays.nn"
            
            "Now, use that algorithm to generate an output array for a new, unseen input array.nn"
            
            "Only return the output array, formatted as valid Python code within a code block. "
            "For example:n```pythonn[[2, 3], [5, 6]]n```n"
            
            "### Developed algorithm:n{reasoning_template}nn"
            "### New enter array:n{test_input}n"
        ),
        (
            "ai",
            "Reply:"
        )
    ]
)

PROMPT_ASSESS = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in abstract reasoning. "
            "A solution array was generated by applying the algorithm to the input array. "
            "Compare the generated solution to the actual target output. "
            "Analyze why the two arrays differ, and provide **clear and concise feedback** on how to improve the algorithm.nn"
            
            "Only return your feedback-do not repeat the arrays or algorithm.nn"
            
            "### Algorithm:n{reasoning_template}nn"
            "### Input array:n{test_input}nn"
            "### Solution array (generated by algorithm):n{solved_test_output}nn"
            "### Target output array:n{test_output}n"
        ),
        (
            "ai",
            "Answer:"
        )
    ]
)

PROMPT_SUMMARIZE_FEEDBACK = ChatPromptTemplate.from_messages(
    [
        (
            "system", 
            "You are an expert in summarizing feedback on algorithm development. "
            "You will be given a history of past attempts, each containing an algorithm and feedback about its performance.nn"
            
            "Your goal is to produce a **concise summary** of the most important lessons learned-"
            "focusing on how the algorithm should be improved and what mistakes should be avoided in future versions.nn"
            
            "Return only the feedback summary. Do not repeat the original attempts or feedback.nn"
            
            "### Attempt History:n{attempt_history}n"
        ),
        (
            "ai",
            "Answer:"
        )
    ]
)

These prompts are linked right into a easy LangChain pipeline:

reasoning_chain = PROMPT_REASON | AGENT_LLM
solve_chain = PROMPT_SOLVE | AGENT_LLM 
assess_chain = PROMPT_ASSESS | AGENT_LLM 
summarize_feedback_chain = PROMPT_SUMMARIZE_FEEDBACK | AGENT_LLM

For every ARC problem:

• The mannequin receives the demonstration pairs and prior suggestions;
• The mannequin generates a brand new algorithm in pseudocode (reasoning_template);
• The algorithm is examined on all of the demonstrations;
• If it fails, the mannequin: receives detailed suggestions on mismatched predictions; summarizes errors throughout makes an attempt; refines the following model of the algorithm;
• As soon as the mannequin will get all demonstrations appropriate, I check it on the unseen check enter.

This course of repeats for as much as a max variety of makes an attempt per problem. A profitable algorithm generalizes throughout the offered examples and applies appropriately to the withheld check case. This setup exams whether or not the mannequin can extract summary patterns, enhance its reasoning over time, and generalize from only a few examples.

reasoning_templates = {}

for i, id in enumerate(id_train_challenges):
    print(f"Coaching on problem {i} ID: {id}")
    train_pairs = train_challenges[id]['train']
    test_input = train_challenges[id]['test'][0]['input'] # solely choose the primary check enter 
    test_output = train_sols[id][0] # solely choose the primary check output
    train_pairs_str = ''
    for i, train_pair in enumerate(train_pairs):
        train_pairs_str += f"Demonstration pair {i+1}:n enter grid: {train_pair['input']} n output grid: {train_pair['output']}n"
    train_pairs_str = train_pairs_str.strip()

    # preserve attempting till you determine  clear up the problem
    right_wrong = "incorrect"
    # Begin with an empty reasoning template, which will probably be refined over time
    reasoning_template = '' 
    okay = 1
    max_attempts = 5
    attempt_history = []
    attempt_history_summary = ''
    whereas right_wrong == "incorrect":
        print(f"Try {okay} to unravel the problem...")

        # Construct the reasoning message with the present reasoning template and try historical past
        # This message will probably be used to generate a brand new reasoning template
        reason_message = {
            "train_pairs": train_pairs_str,
            "attempt_history": attempt_history_summary,  
        }
        res = reasoning_chain.invoke(reason_message)
        reasoning_template = res.content material

        # Assess reasoning template
        wrong_pairs = []
        for train_pair in train_pairs:
            demo_input = train_pair['input']
            demo_output = train_pair['output']
            # Take a look at the reasoning template on the demonstration pair
            test_message = {
                "test_input": demo_input,
                "reasoning_template": reasoning_template,
            }
            res = solve_chain.invoke(test_message)
            solved_demo_output = extract_python_code(res.content material)            
            # Evaluate the output with the demonstration output
            if solved_demo_output != demo_output:
                wrong_pairs.append((demo_input, demo_output, solved_demo_output))

        if len(wrong_pairs) > 0:
            right_wrong = 'incorrect'
            print(f"Reasoning template failed on {len(wrong_pairs)} demonstration pairs.")

            if okay >= max_attempts:
                print(f"Max makes an attempt reached ({max_attempts}). Stopping for problem {id}.")
                reasoning_templates[id] = ''
                break

            print("Assessing the reasoning template...")
            assessment_res = f'Algorithm failed on {len(wrong_pairs)} demonstration pairs. Right here is the suggestions:n'
            for demo_input, demo_output, solved_demo_output in wrong_pairs:
                assess_chain_message = {
                    "reasoning_template": reasoning_template,
                    "test_input": demo_input,
                    "solved_test_output": solved_demo_output,
                    "test_output": demo_output,
                }
                res = assess_chain.invoke(assess_chain_message)
                assessment_res += f" - From enter {demo_input} to output {demo_output}, your resolution was {solved_demo_output}: {res.content material.strip()}n"

            attempt_history.append({
                "try": okay,
                "reasoning_template": reasoning_template,
                "suggestions": assessment_res
            })

            summary_message = {
                "attempt_history": attempt_history,
            }
            summary_res = summarize_feedback_chain.invoke(summary_message)
            attempt_history_summary = summary_res.content material.strip()
        else:
            print("Resolution is appropriate!")
            right_wrong = "appropriate"
            reasoning_templates[id] = reasoning_template

            # check it towards the check enter/ output .... however don't give suggestions (that is alleged to be unknown)
            test_message = {
                "test_input": test_input,
                "reasoning_template": reasoning_template,
            }
            res = solve_chain.invoke(test_message)
            solved_test_output = extract_python_code(res.content material)
            if test_output != solved_test_output:
                print(f"Take a look at output doesn't match the true output for problem {id}.")
            else:
                print(f"Take a look at output matches the true output for problem {id}.")
                #visualize_matrix(test_input, "Enter grid")
                #visualize_matrix(test_output, "True output")
                #visualize_matrix(solved_test_output, "Take a look at Output")

            print("-" * 40)  # Separator between entries

        okay += 1

Outcomes: When Reasoning Works

In some instances, o3-mini was capable of appropriately infer a generalizable algorithm from just some input-output demonstrations. One such instance concerned producing a patterned tiling primarily based on a small 2×2 enter grid.

After only one try, the mannequin converged on the next pseudocode:

BEGIN  
  Let enter be a 2x2 grid, the place:
    enter[0] = [a, b]
    enter[1] = [c, d]
  
  Initialize output as an empty listing.

  FOR every row index r from 0 to five DO:
    Let original_row ← enter[r mod 2]
    
    IF (FLOOR(r / 2)) mod 2 = 1 THEN
      Let base_row ← REVERSE(original_row)
    ELSE
      Let base_row ← original_row
    ENDIF

    Initialize new_row as an empty listing.
    FOR repeat from 1 to three DO:
      Append all components of base_row to new_row.
    ENDFOR

    Append new_row to output.
  ENDFOR

  RETURN output
END

Right here is the anticipated resolution (true output) and the answer from the mannequin pseudocode (check output).

This algorithm demonstrates a number of notable reasoning capabilities:

• Sample abstraction: the mannequin inferred a repeating tiling sample from restricted information;
• Modulo logic: it launched modulo-based indexing (r mod 2 and (r // 2) mod 2) to alternate row conduct, mimicking visible mirroring;
• Grid development: the answer scales up the two×2 enter into a bigger 6×6 grid by repeated copying and reversal;

The truth that the mannequin found this construction with out hardcoded guidelines suggests it was participating in a type of algorithm synthesis, guided by few-shot abstraction.

Right here is one other profitable instance. 

Once more, after only one try, the mannequin converged on the next pseudocode:

BEGIN
  Let N = 3
  Create output as a 2D array of measurement (N×N) × (N×N), crammed with 0

  FOR every row r from 0 to N–1:
    FOR every column c from 0 to N–1:
      IF enter[r][c] ≠ 0 THEN
        FOR every i from 0 to N–1:
          FOR every j from 0 to N–1:
            Set output[(r * N) + i][(c * N) + j] = enter[i][j]
  RETURN output
END

Right here is the anticipated resolution and the answer from the mannequin pseudocode.

This algorithm successfully tiles the total enter grid into the output grid at each place the place the enter cell is non-zero. The tiling is aligned such that every copy of the unique grid is positioned at an offset decided by (r * N, c * N) —  the scaled-up coordinates of the non-zero enter cell.

What’s spectacular right here is that the mannequin:

• Learns conditional placement. It solely pastes the enter the place the enter worth is non-zero;
• Makes use of coordinate arithmetic to scale placement places, exhibiting spatial understanding;
• Treats the enter as each management logic and content material, combining format detection with repetition.

When Textual content-Primarily based Reasoning Isn’t Sufficient

Within the job beneath, the mannequin was given a small variety of demonstration input-output grid pairs and requested to deduce the transformation rule. 

The text-based mannequin (o3-mini) produced an in depth pseudocode resolution, structured, believable, and internally constant:

BEGIN
  Let unique ← enter grid  
  Let output ← deep copy of unique
  Let R ← variety of rows in unique  
  Let C ← variety of columns in unique

  // Compute ring index for each cell that's a part of a non-zero area.
  // A cell's ring index is outlined as:
  //   - 0 if the cell is on the boundary of the grid OR if a minimum of one in all its 4-neighbors is 0
  //   - In any other case, 1 + min(ring index of its 4-neighbors)

  Create grid ring of measurement R × C, crammed with −1

  For every cell (r, c) in unique:
    If unique[r][c] ≠ 0 then
      If r = 0 OR c = 0 OR r = R−1 OR c = C−1 then
        ring[r][c] ← 0
      Else if any 4-neighbor (up, down, left, proper) of (r, c) has worth 0 then
        ring[r][c] ← 0

  modified ← true
  Whereas modified is true:
    Set modified ← false
    For every cell (r, c) in unique such that unique[r][c] ≠ 0 and (ring[r][c] < 0 or not finalized):
      Let candidates ← listing of ring indices from any 4-neighbor (r′, c′) with a non-negative ring worth
      If candidates is non-empty then
        newRing ← min(candidates) + 1
        If ring[r][c] < 0 OR newRing < ring[r][c] then
          ring[r][c] ← newRing
          modified ← true

  // Course of propagation from outer ring inward.
  // For every cell with ring index ≥ 1, replace its worth primarily based on a neighbor with ring index one much less.
  // Choice is deterministic: "up, left, down, proper"

  Let maxRing ← most worth in ring

  For layer from 1 to maxRing:
    For every cell (r, c) such that ring[r][c] = layer:
      For route in [(−1,0), (0,−1), (1,0), (0,1)]:
        nr ← r + route.row
        nc ← c + route.col
        If (nr, nc) is in bounds AND ring[nr][nc] = layer − 1 then
          Let base ← output[nr][nc]
          If base = 2 then
            newValue ← 8
          Else if base mod 2 = 0 then
            newValue ← base + 2
          Else // base is odd
            newValue ← base + 1
          output[r][c] ← newValue
          Escape of neighbor loop

Return output
END

At first look, this appears like a classy technique: compute a hoop index for every non-zero area, then diffuse values inward utilizing deterministic guidelines. It even contains replace logic:

• If the “base” cell is 2 → assign 8
• If base is even → add 2
• If base is odd → add 1

However this complete algorithm, nonetheless coherent, is misguided. It utterly fails to match the true underlying transformation proven within the demonstrations.

As an alternative of reasoning about blue-bordered areas and their nested construction, the mannequin generated a generic flood-fill algorithm primarily based on distance from edge and adjacency to zeros. Even the suggestions retains refining the procedural strategy recognized earlier:

['Key lessons are to: • Precisely compute the ring index so that only',
 'true boundary (or external zero) cells get index 0, ensuring that',
 'inner cells receive higher indices for proper propagation. • Use a',
 'reliable, consistent method for selecting the "base" value for',
 'updates-ideally by considering all adjacent lower-ring cells or using',
 'a deterministic order-and use an immutable copy of the original grid',
 'for these lookups. • Apply the parity‐based update rules correctly so',
 'that cells with ring index ≥ 1 get the specified value increments',
 '(especially the special case when the base is 2) rather than remaining',
 'unchanged. • Ensure that the update logic cascades inward, allowing',
 'inner cells to correctly inherit and build upon values from outer',
 'rings.']

So what went incorrect?

• Topological, not visible. The mannequin centered on connectivity and edge proximity, ignoring the visually outlined areas.
• Procedural, not inferential. The logic was inflexible and hand-crafted, not derived from patterns within the examples.
• Demonstration-agnostic. There’s no signal the mannequin meaningfully included the few-shot examples. It doubtless defaulted to a well-known sample — spatial progress utilizing layers.

This isn’t shocking. Textual content-only LLMs haven’t any visible grounding. They tokenize the grid as symbolic enter — rows of digits, not enclosed figures or nested patterns. In consequence, their inductive biases lean towards symbolic or graph-like algorithms, not perceptual abstractions.

On this case, the mannequin fell into a typical entice: producing one thing plausible-sounding however incorrect. It produced a spatial propagation scheme which may work for a diffusion job however not the one at hand. This highlights a key weak point in text-based few-shot prompting for summary visible reasoning: the mannequin’s “reasoning” is disconnected from perceptual understanding. It invents algorithms primarily based on inner priors, not exterior cues.

When Reasoning Fails: Additionally Picture Fashions Aren’t Magic

To enhance generalization, I transitioned from purely text-based reasoning to image-based prompting, leveraging GPT-4.1’s multimodal capabilities by LangChain. This setup encoded input-output grid examples as base64 photos, which have been introduced alongside a pure language immediate describing the duty.

from langchain_core.messages import HumanMessage

import io
import base64

AGENT_MODEL = "gpt-4.1"

# Immediate for picture primarily based reasoning
PROMPT_REASON_IMG = """You're an skilled at fixing summary reasoning duties.

These are distinctive reasoning duties with restricted examples. You're given demonstration input-output 2D grids. 
The colormap used is as follows:

{{
    'Violet': 0,
    'Indigo': 1,
    'Blue': 2,
    'Yellow': 3,
    'Inexperienced': 4,
    'Orange': 5,
    'Purple': 6,
    'Golden': 7,
    'Black': 8,
    'White': 9
}}

Your aim is to develop a single algorithm that maps every enter grid to its corresponding output grid.

A profitable resolution entails:
- Analyzing the demonstration examples fastidiously
- Figuring out underlying visible or spatial patterns
- Formulating a common transformation rule
- Translating this rule into clear pseudocode

If this isn't your first try, a abstract of earlier suggestions can also be offered. Evaluation it fastidiously and incorporate it to enhance your resolution.

Take a look at your algorithm towards the demonstrations to make sure it really works.

Return **solely the algorithm pseudocode**, formatted as plain textual content. Don't clarify it or add further commentary.
"""

# In case your array is 10x10 and also you need every cell to be 20x20 pixels (cell_px), the picture will probably be 200x200 pixels.
# Convert matrix into picture
def visualize_grid_fig(matrix, cmap=cmap, norm=norm, cell_px=20, present=False):
    if kind(matrix) will not be np.ndarray:
        matrix = np.array(matrix)
    h, w = matrix.form[:2]
    figsize = (w * cell_px / 100, h * cell_px / 100)  # inches
    fig, ax = plt.subplots(figsize=figsize)
    ax.imshow(matrix, cmap=cmap, norm=norm)
    ax.axis('off')
    if present:
        plt.present()
    else:
        plt.shut(fig)
    return fig

# encode picture for mannequin
def fig_to_base64(fig, dpi=100):
    buf = io.BytesIO()
    fig.savefig(buf, format='png', dpi=dpi, bbox_inches='tight')
    buf.search(0)
    img_base64 = base64.b64encode(buf.learn()).decode('utf-8')
    buf.shut()
    return img_base64

# Within the loop exchange reasoning code with this
# reasoning with photos
reason_message = [{"type": "text", "text": PROMPT_REASON_IMG}]
for i, instance in enumerate(train_pairs):
    #fig_in = visualize_grid_fig(instance['input'], cmap, norm)
    #fig_out = visualize_grid_fig(instance['output'], cmap, norm)
    fig_in = visualize_grid_fig(instance['input'], )
    fig_out = visualize_grid_fig(instance['output'], )
    fig_in = fig_to_base64(fig_in)
    fig_out = fig_to_base64(fig_out)
    reason_message.append({"kind": "textual content", "textual content": f"### Enter grid {i+1}:"})
    reason_message.append({"kind": "image_url", "image_url": {"url": f"information:picture/jpeg;base64,{fig_in}"}})
    reason_message.append({"kind": "textual content", "textual content": f"### Output grid {i+1}:"})
    reason_message.append({"kind": "image_url", "image_url": {"url": f"information:picture/jpeg;base64,{fig_out}"}})
reason_message.append({"kind": "textual content", "textual content": f"### Suggestions (abstract of earlier makes an attempt): {attempt_history_summary}"})
reason_message = HumanMessage(content material=reason_message)
res = AGENT_LLM.invoke([reason_message])
reasoning_template = res.content material

The ensuing pseudocode marked a transparent step ahead in expressiveness. The mannequin was capable of:

• Detect blue-bordered squares utilizing visible options fairly than purely symbolic construction;
• Apply guidelines primarily based on sq. measurement and nesting depth to deduce inside fill colours;
• Type recognized squares by measurement earlier than filling, successfully avoiding overwrite conflicts.

Right here’s the generated pseudocode:

1. Let grid be the enter 2D array.
2. Create output_grid as a duplicate of grid.

3. Determine all blue-bordered squares within the grid:
   a. For every attainable top-left nook (i, j):
      i. For every attainable sq. measurement s (s ≥ 3, as much as min(grid top, grid width)):
         - Test if the sq. of measurement s beginning at (i, j) is absolutely inside bounds.
         - Test if all *border* cells of this sq. are Blue (worth = 2).
         - Test that the *inside* cells (not on the border) don't include any Blue (2).
         - If all situations are met, report the sq. as (i, j, s).

4. Type the listing of detected blue-bordered squares by measurement in ascending order (smallest first).

5. For every detected sq. (i, j, s), in sorted order:
   a. Decide the fill shade:
      - If the sq. is the smallest (no different blue-bordered sq. is absolutely inside it), set fill shade = Black (8).
      - If the sq. is the most important (no different blue-bordered sq. absolutely incorporates it), fill shade = 
        - If there are precisely 2 blue-bordered squares, set fill shade = Inexperienced (4).
        - If there are three blue-bordered squares within the grid, fill shade = Yellow (3).
      - If the sq. is nested (not smallest or largest), fill shade = Black (8).
      - (Extra complicated guidelines might generalize past these primarily based on demonstrations.)

   b. Fill the inside of the sq.:
      For every cell (x, y) strictly contained in the sq. (i+1 ≤ x < i+s−1) and (j+1 ≤ y < j+s−1):
         - If output_grid[x][y] will not be Blue (2), set it to the chosen fill shade.

6. Return output_grid.

Particular notes:
  - By no means overwrite Blue (2) border pixels. 
  - When filling, later (bigger) squares overwrite earlier (smaller) fills in overlapping areas. 
  - Solely course of legitimate blue-bordered squares (minimal measurement 3x3, full border). 
  - If there are a number of disjoint blue-bordered squares, deal with every independently for fill shade task as above matching the demonstration logic.

The mannequin clearly reveals structured reasoning. It develops an inner illustration of nested geometric varieties and makes an attempt to use rule-based transformations derived from examples.

Nevertheless, regardless of these advances, the mannequin nonetheless fails to generalize reliably. In novel configurations, it mis-assigns fill colours, falling again on brittle heuristics like size-based precedence or inflexible nesting assumptions. For example, it’d assume the most important sq. is at all times crammed with yellow, even when that logic now not holds in a brand new context. This failure reveals a deeper limitation: even with picture enter, the mannequin doesn’t “see” within the human sense. It doesn’t construct a holistic perceptual illustration of spatial relationships. As an alternative, it converts the picture into symbolic patterns and applies deterministic procedures like flood-fill, sorting, or positional indexing.

In follow, this implies the mannequin causes from inner abstractions, not perceptual grounding. It infers that “smaller squares get black,” or “fill primarily based on measurement rank,” with out absolutely understanding why these assignments occurred within the demonstrations. In consequence, any deviation from the anticipated format could cause it to misfire.

This means that whereas multimodal prompting extends the expressive vary of the mannequin, it doesn’t but present the type of versatile, generalizable visible reasoning that people show. These duties might in the end require stronger types of program induction, meta-learning, or hybrid techniques that combine perceptual grouping with discovered guidelines.

Conclusions

On this research, I explored whether or not giant language fashions — each text-based and multimodal — can carry out meta-learning from examples on summary reasoning duties. Particularly, I centered on a category of issues from the ARC dataset, the place options require figuring out visible patterns, studying transformations, and generalizing them to novel check inputs.

By direct prompting experiments, I discovered that:

• Textual content-based fashions (e.g., o3-mini) usually hallucinate believable algorithms which are topologically or procedurally sound however completely disconnected from the duty’s visible logic. These fashions depend on symbolic reasoning over tokenized grids, and default to acquainted heuristics like flood-fill, ring propagation, or rule-based updates, whatever the examples offered.
• Multimodal fashions (e.g., GPT-4 with imaginative and prescient) confirmed a clear enchancment in sample detection and relational reasoning. They efficiently recognized blue-bordered areas and tailored behaviors primarily based on relative measurement or nesting. Nevertheless, their generalization remained fragile: they nonetheless utilized brittle guidelines, reminiscent of fastened size-based assignments, and failed in novel layouts that deviated from the demonstrations.

These findings counsel that, even with visible enter, present LLMs don’t “see” as people do. They course of photos symbolically, not perceptually. Their reasoning is pushed by internally constructed guidelines, not a versatile, visible understanding of shapes, hierarchy, or affordance.

The constraints I noticed reinforce a central stress: few-shot prompting alone, even with photos, will not be enough for sturdy abstraction. True generalization doubtless requires:

• Program induction: inferring reusable, structured transformations from examples; 
• Perceptual grounding: creating architectures that parse and manipulate visible scenes compositionally;
• Meta-learning architectures: constructing fashions that adapt their reasoning methods dynamically fairly than making use of pre-learned heuristics;

In the present day’s LLMs are astonishing of their breadth, however they’re nonetheless guessing primarily based on priors, not studying to study within the human sense. They lack a powerful inductive bias for abstraction and transformation. ARC-style duties expose this hole clearly: success requires greater than sample recognition, it requires reasoning from examples in a structured, compositional manner. These outcomes are usually not discouraging, fairly clarifying. We now know the place the ceiling is. And the following technology of fashions, these with hybrid architectures, persistent reminiscence, and express meta-learning capabilities would possibly lastly break by it.