Why Transformers Want Positional Encoding For Time Collection: A Visible Information

Whereas digging into basis fashions for time collection, I spotted that I may not likely perceive them with out first understanding transformers. I didn’t wish to use these fashions as black bins, so I began tracing the concepts backward, from basis fashions to transformers, and from transformers to self-attention. What made the transition fascinating is that though transformers had been initially constructed for language, the core thought carries naturally to time collection. The 2 modalities are very completely different, however they share one thing elementary: each are sequences, and in each instances, order adjustments which means.

In language, canine bites man could be very completely different from man bites canine.

Time collection are not any completely different. A temperature of $30^circ$ yesterday and $20^circ$ at this time tells a unique story from $20^circ$yesterday and $30^circ$ at this time. The values often is the similar, however their order adjustments the which means of the sequence.

The query is that if self-attention seems to be in any respect observations directly, how does a transformer know which statement got here first, which got here later, or how far aside two observations are?

That query led me to positional encoding.

What shocked me most was how such a easy mathematical thought may give a Transformer a way of order. The precise strategies have developed significantly since then, however the underlying downside stays the identical.

This text is my try and construct that instinct from the bottom up, beginning with a easy time collection and following the trail from uncooked observations to self-attention and at last to positional encoding.

From scalar observations to vector representations

Contemplate a easy time collection containing the temperature recorded over 5 weekdays:

Every statement $x_t$

A easy manner to do that is thru a discovered linear projection i.e. embedding:

giving us a sequence of vector representations: $e_1, e_2, e_3, e_4, e_5$

An embedding is a deep, summary illustration of the collection within the type of a multidimensional numerical vector that encodes its options and that the mannequin understands. [1]

Every $e_t$

The essential phrase right here is discovered. The mannequin just isn’t given a predefined vector illustration for a temperature akin to $18^circ$. The parameters $W_e$

At this level, $e_t$

How self-attention builds context?

Self-attention permits every statement to make use of info from the remainder of the sequence.

Suppose we wish to replace Friday’s illustration. The mannequin first creates three discovered projections from each $e_t$

The matrices $W_Q$

For Friday, its question $q_5$

Every comparability produces an consideration rating:

which measures how related statement ‘j’ is when updating Friday’s illustration. The scaling issue $sqrt{d_k}$

These scores are handed by means of a softmax perform to transform them into consideration weights:

Lastly, these weights are used to mix the worth vectors:

So $e_5$

Briefly:

Queries and keys be taught which observations are related to 1 one other. Values carry the data that’s mixed to kind the brand new illustration.

What occurs if we shuffle the sequence?

Now comes the essential query.

Suppose the identical 5 temperature observations are rearranged.

The values themselves haven’t modified; solely the order has. After the discovered projection, we nonetheless have the identical set of worth representations, simply rearranged.

Self-attention can nonetheless evaluate every illustration with all of the others. The identical question, key, and worth projections are utilized, and the identical sorts of pairwise relationships can nonetheless be computed.

What has disappeared is the temporal construction.

Nothing inside $e(27^circ)$ says that it initially got here from Thursday. Nothing inside $e(18^circ)$ says that it occurred after $e(27^circ)$. Additionally, if Wednesday and Friday have the identical temperature worth, the discovered projection will map them to the identical embedding vector. With out positional info, the mannequin subsequently has no option to distinguish which embedding got here from Wednesday and which got here from Friday.

That is the important thing limitation:

Self-attention can be taught which observations are associated, however with out an extra positional sign, it has no built-in option to know the place these observations occurred within the sequence.

What ought to positional info inform the mannequin?

If self-attention doesn’t know the order of the observations, then the following query is: what sort of positional info could be helpful?

At a minimal, we might need the mannequin to know:

• Which place an statement belongs to?
  Place 2 needs to be distinguishable from place 20.

• Which statement got here earlier than or after one other?
  The mannequin ought to have the ability to distinguish $t-1$

• How far aside are two observations?
  In time collection, the distinction between $t-1$

• That close by positions are associated in a structured manner.
  Place 10 and place 11 shouldn’t seem like two fully unrelated identifiers.

• That the illustration stays helpful over longer sequences.
  Ideally, the positional scheme ought to nonetheless present significant construction because the sequence grows.

For time collection, the third property is particularly helpful. A mannequin could care about an statement one step in the past due to short-term dependence, or seven steps in the past due to a weekly seasonal sample.

So positional info ought to do greater than merely assign a singular label to every timestep. It ought to give the mannequin a structured illustration of order and relative distance.

How can we symbolize place?

We now know what info is lacking. The subsequent query is methods to symbolize it.

A easy option to symbolize place could be to assign every timestep a quantity:

However feeding the uncooked place straight into the mannequin just isn’t superb. The values continue to grow with sequence size, and a single quantity doesn’t give the mannequin a wealthy illustration of positional relationships.

One of many unique Transformer’s options was sinusoidal positional encoding, the place every place is represented utilizing sine and cosine capabilities at completely different frequencies.

Why sine and cosine?

Begin with the best two-dimensional instance:

As $t$ adjustments, the positional vector strikes easily round a circle that enables close by positions to have completely different however nonetheless associated representations.

Extra importantly, shifting ahead by the identical variety of steps produces the identical form of change within the positional illustration. For instance, an offset of seven positions has the identical mathematical relationship whether or not we transfer from place 3 to 10 or from place 20 to 27. That’s helpful for time collection as a result of relative distance typically issues:

can symbolize very completely different temporal relationships.

The complete sinusoidal positional encoding extends this concept throughout many dimensions:

Completely different dimensions use completely different frequencies. Some change rapidly throughout close by positions, whereas others change far more slowly.

One helpful manner to consider that is as many clocks working at completely different speeds. Collectively, their readings give each place a structured positional signature.

So as an alternative of giving timestep (t) solely a quantity, we give it a vector:

that incorporates details about its place and its relationship to different positions.

Combining worth and place

After including positional info, every timestep is represented as:

the place $e_t$

Self-attention now builds its queries and keys from this mixed illustration:

So when the mannequin computes an consideration rating,

the comparability is now not primarily based solely on the noticed values. The question and key vectors had been created from representations that already include positional info.

In consequence, the mannequin can be taught relationships that depend upon each:

• what was noticed, and

• the place the observations occurred within the sequence.

For a time collection, this implies the mannequin can probably be taught that an statement one timestep in the past needs to be handled otherwise from one seven timesteps in the past, even when their values are related.

Positional encoding subsequently doesn’t inform the mannequin explicitly which lags are essential. It provides self-attention the data wanted to be taught which positional relationships matter for the duty.

Abstract

We began with a easy downside: self-attention can evaluate each statement with each different statement, however by itself it doesn’t know the order by which these observations occurred.

The answer is to complement every worth illustration $e_t$

Self-attention then builds its queries, keys, and values from $h_t$

For repeatedly sampled time collection, this makes relationships akin to t−1, t−7, or t−24 accessible to the mannequin. A lag of 1 step could seize short-term dependence, whereas a lag of seven or twenty-four steps could correspond to a seasonal sample.

Positional encoding, nevertheless, represents sequence place, not essentially real-world time. If observations are irregularly spaced, being one place aside doesn’t all the time imply being one hour or at some point aside. That is the place richer temporal encodings and time options change into essential.

Sinusoidal positional encoding is just one option to inject order right into a Transformer. Different approaches embody discovered positional embeddings, the place the place vectors themselves are discovered throughout coaching, and relative positional encodings, which focus extra straight on the space between two observations quite than their absolute positions.

For time collection, the issue can change into even richer. Sequence place will not be sufficient: the mannequin may must know the precise timestamp, calendar results, periodicity, or irregular gaps between observations.

So the broader query just isn’t merely:

How can we inform a Transformer that that is place 7?

however quite:

What notion of time does the mannequin really want for the duty?

···

And this brings me again to the place this exploration began: time-series basis fashions. Trendy architectures could use extra subtle approaches akin to rotary or discovered positional representations quite than the unique sinusoidal formulation. However understanding the straightforward sinusoidal building provides us the muse for understanding why these strategies exist within the first place.

We now have now constructed the muse of why positional info is required, how it’s represented, and the way self-attention makes use of it. With this instinct in place, extra superior positional encoding strategies ought to really feel a lot simpler to know after we encounter them in future.

···

Notice: The figures on this article had been conceptually designed by the creator and generated and refined with the help of an AI image-generation device.

References

[1] Peixeiro, Marco. Time Collection Forecasting Utilizing Basis Fashions: The way to Construct Excessive Accuracy Predictive Fashions. Manning, 2025.

[2] Davidson, Graeme, and Lei Ma. Time Collection with PyTorch: Trendy Deep Studying Toolkit for Actual-World Forecasting Challenges. Packt Publishing, 2026.