Dissecting “Reinforcement Studying” by Richard S. Sutton with customized Python implementations, Episode V
In our earlier put up, we wrapped up the introductory sequence on elementary reinforcement studying (RL) methods by exploring Temporal-Distinction (TD) studying. TD strategies merge the strengths of Dynamic Programming (DP) and Monte Carlo (MC) strategies, leveraging their greatest options to type a few of the most essential RL algorithms, reminiscent of Q-learning.
Constructing on that basis, this put up delves into n-step TD studying, a flexible strategy launched in Chapter 7 of Sutton’s e book [1]. This technique bridges the hole between classical TD and MC methods. Like TD, n-step strategies use bootstrapping (leveraging prior estimates), however in addition they incorporate the following n rewards, providing a novel mix of short-term and long-term studying. In a future put up, we’ll generalize this idea even additional with eligibility traces.
We’ll comply with a structured strategy, beginning with the prediction downside earlier than shifting to management. Alongside the best way, we’ll:
- Introduce n-step Sarsa,
- Prolong it to off-policy studying,
- Discover the n-step tree backup algorithm, and
- Current a unifying perspective with n-step Q(σ).
As all the time, you will discover all accompanying code on GitHub. Let’s dive in!
