TD or not TD: Analyzing the Role of Temporal Differencing in Deep Reinforcement Learning
By: Researcher
Disclaimer: The provided code links for this paper are external links. Science Nest has no responsibility for the accuracy, legality or content of these links. Also, by downloading this code(s), you agree to comply with the terms of use as set out by the author(s) of the code(s).
Please contact us in case of a broken link from here
| Authors | Thomas Brox, Vladlen Koltun, Alexey Dosovitskiy, Artemij Amiranashvili |
| Journal/Conference Name | ICLR 2018 1 |
| Paper Category | Artificial Intelligence |
| Paper Abstract | Our understanding of reinforcement learning (RL) has been shaped by theoretical and empirical results that were obtained decades ago using tabular representations and linear function approximators. These results suggest that RL methods that use temporal differencing (TD) are superior to direct Monte Carlo estimation (MC). How do these results hold up in deep RL, which deals with perceptually complex environments and deep nonlinear models? In this paper, we re-examine the role of TD in modern deep RL, using specially designed environments that control for specific factors that affect performance, such as reward sparsity, reward delay, and the perceptual complexity of the task. When comparing TD with infinite-horizon MC, we are able to reproduce classic results in modern settings. Yet we also find that finite-horizon MC is not inferior to TD, even when rewards are sparse or delayed. This makes MC a viable alternative to TD in deep RL. |
| Date of publication | 2018 |
| Code Programming Language | Python |
| Comment |
Copyright Researcher 2021