Interactive Knowledge Exploration for Laptop Imaginative and prescient Tasks with Rerun

instruments for information visualization and exploration have allowed me to iterate approach faster on my laptop imaginative and prescient initiatives, particularly when the issues I’m confronted with will not be simple and I have to make algorithm or design selections primarily based on dynamic time-varying alerts. These alerts can typically be difficult to research by simply quickly altering numbers plotted on the display or saved in a desk.

Whereas engaged on a few of these issues, I explored the built-in interactive parts from OpenCV, however other than some sliders the choices there are very restricted, particularly when making an attempt to combine some animated plots. There’s some hacky methods to get dynamic plots from matplotlib into OpenCV, which I explored on this submit.

I additionally explored totally different UI frameworks like tkinter, which I utilized in my final mission for a sentiment evaluation visualization. I constructed a customized part that allowed me to show dynamic frames. Nonetheless it nonetheless didn’t actually really feel like the suitable software for the duty, particularly when making an attempt to work with interactive plots.

After which I stumbled upon Rerun. Each every now and then I uncover a software or framework that actually excites me, and this was precisely a type of occasions. Rerun is an open supply software for visualizing information sometimes discovered within the robotics area starting from easy time collection information and static photos to complicated 3D level clouds, video streams or different varieties of multi-modal information. The demos look actually spectacular and the setup and code samples are so easy, I used to be instantly hooked.

So I made a decision to remodel my ball monitoring demo from a earlier mission and plot the info utilizing rerun. Let me present you the way simple it’s to make use of and create interactive purposes!

Quickstart

You may set up rerun in any of your python initiatives utilizing pip or uv:

pip set up rerun-sdk
uv add rerun-sdk

You can begin the viewer after putting in the sdk by merely working it from the command line:

rerun

The viewer will probably be your fundamental window the place your experiments will probably be proven. You may depart it open or shut it between your experiments.

To instantiate a rerun viewer from a python script, it’s good to spawn an occasion with an experiment identify:

import rerun as rr

rr.init("ball_tracking", spawn=True)

Ball Monitoring Demo

Rerun experiment recordings could be saved to and loaded from .rrd recording recordsdata. You may obtain the recording file for the ball monitoring demo from right here. Press Ctrl + O or choose Open... within the menu on the highest left of the rerun viewer and cargo the downloaded recording file.

You will note the ball monitoring demo playback as soon as after which the video stops. On the backside of the viewer, you have got the timeline. You may scrub by means of the timeline by clicking and dragging the deal with.

These recording recordsdata solely comprise the info of the experiment together with the video, its annotations and the time collection of the monitoring. The structure of the viewer is saved individually in a .rbl blueprint file. Obtain the blueprint file for the demo right here and open it on prime of the present information file.

Now now we have a barely higher overview with the place, velocity and acceleration plots separated and the video prominently centered.

Within the video body you’ll be able to click on on the annotations and within the left Blueprint panel you’ll be able to disguise or present them individually.

Time Collection Plots

To investigate a particular plot, you’ll be able to click on on the develop view button on the prime proper of any window, for instance the place plot.

This can be a TimeSeriesView. This view can be utilized to plot information in a 2D chart with the x-axis representing the time area, in our case the discrete body index of the video. On this ball monitoring demo, we iterate over the video frames in a loop, the place we will set the body index of our timeline explicitly.

frame_index = 0

whereas True:
    ret, body = cap.learn()
    if not ret:
        break

    frame_index += 1
    if frame_index >= num_frames:
        break

    rr.set_time("frame_idx", sequence=frame_index)

To create the plot for the place, it’s good to log a Scalar worth for the place of the tracked ball at each body index. On this case after we calculate the place we will merely log it to rerun:

rr.log("ball/position_y", rr.Scalars(pos))

To configure the type of this plot, we additionally have to log one thing to the identical entity path (ball/position_y), however for the reason that type doesn’t change we will log it as soon as earlier than the loop and provide a static argument.

rr.log(
    "ball/position_y",
    rr.SeriesLines(colours=[0, 128, 255], names="pos y"),
    static=True,
)

To outline the axis vary that’s seen per default, we have to specify a structure part.

view_pos = rrb.TimeSeriesView(
    origin="ball/position_y",
    axis_y=rrb.ScalarAxis(vary=(0, 700)),
)
structure = rrb.Blueprint(view_pos)
rr.send_blueprint(structure)

Video Stream

Equally we will create a view for the video frames by logging the picture to rerun. Since rerun expects an RGB photos however OpenCV ues BGR for its colour channel ordering, we have to convert the frames from BGR to RGB earlier than passing them to rerun.

frame_rgb: np.ndarray = cv2.cvtColor(body, cv2.COLOR_BGR2RGB)
rr.log("body", rr.Picture(frame_rgb))

So as to add annotations to the picture view we have to log spatial parts to a sub path of the desired entity path. For instance, we will draw the middle of the tracked ball:

rr.log(
    "body/factors",
    rr.Points2D(
        positions=[center],
        colours=[0, 0, 255],
        radii=4.0,
    ),
)

To incorporate the video body view within the structure, we will use a Spatial2DView node:

view_frame = rrb.Spatial2DView(origin="body")

Then we will stack the plot from earlier than vertically with the body view through the use of a Vertical structure part:

structure = rrb.Blueprint(
    rrb.Vertical(
        view_pos,
        view_frame,
        row_shares=[0.33],
    ),
)
rr.send_blueprint(structure)

The row shares specifies how a lot every of the rows occupies in percentages. We will omit the second row share entry for the body view for the reason that shares have so as to add as much as 1.

Limitations

Whereas engaged on this mission, I bumped into some limitations of Rerun. Within the authentic mission I visualized a prediction of the trajectory at each timestep, however that is at present not attainable in a time collection view. Additionally the layouts and configuration of the plotted information is restricted, for instance there’s no built-in approach to attract a circle with out fill. However for the reason that mission could be very actively being developed, there’s an excellent probability that a few of these may be attainable sooner or later.

Conclusion

The developer expertise with this software as a pc imaginative and prescient engineer is extraordinarily good, the person interface hundreds virtually immediately and the timeline scrubbing could be extremely useful for understanding or debugging alerts in time collection plots or in movies. I’ll positively maintain utilizing and exploring this mission and may solely suggest you to attempt it out for your self!

For extra particulars and the total implementation, checkout the supply code of this mission below src/ball_tracking/trajectory_rerun.py.

https://github.com/trflorian/ball-tracking-live-plot

All visualizations on this submit had been created by the writer.