【5分钟 Paper】(TD3) Addressing Function Approximation Error in Actor-Critic Methods

简介: 【5分钟 Paper】(TD3) Addressing Function Approximation Error in Actor-Critic Methods
  • 论文题目:Addressing Function Approximation Error in Actor-Critic Methods

所解决的问题?

  value-base的强化学习值函数的近似估计会过估计值函数(DQN),作者将Double Q-Learning处理过拟合的思想引入actor critic算法中。(过估计的问题就在于累计误差会使得某些不好的statevalue变地很高(exploration 不充分所导致的))。还花了很大的心血在处理过估计问题修正后带来的方差过高的问题。

  作者将过估计的问题引入到continuous action space中,在continuous action space中处理过估计问题的难点在于policychange非常缓慢,导致currenttargetvalue差距不大, too similar to avoid maximization bias

背景

  以往的算法解决过估计问题的就是Double Q Learning那一套,但是这种方法虽然说会降低bias但是会引入高的variance(在选择下一个时刻s‘action的时候,不确定性变得更大才将以往DQNmax这一步变得不是那么max,与之带来的问题就是方差会变大),仍然会对policy的优化起负面作用。作者是用clipped double q learning来解决这个问题。

所采用的方法?

  作者所采用的很多components用于减少方差:

  1. DQN 中的 target network 用于variance reduction by reducing the accumulation of errors(不使用target network的使用是振荡更新的)。
  2. 为了解决valuepolicy耦合的关系,提出了延迟更新(delaying policy updates)的方式。(to address the coupling of value and policy, we propose delaying policy updates until the value estimate has converged)
  3. 提出了novel regularization的更新方式SARSA-style ( the variance reduction by averaging over valueestimates)。这种方法参考的是18Nachum的将值函数smooth能够减少方差的算法。
  • Nachum, O., Norouzi, M., Tucker, G., and Schuurmans, D. Smoothed action value functions for learning gaussian policies. arXiv preprint arXiv:1803.02348, 2018.

  当然multi-step return也能够去权衡方差与偏差之间的关系,还有一些放在文末扩展阅读里面了。

  作者将上述修正方法用于Deep Deterministic Policy Gradient算法中并将其命名为Twin Delayed Deep Deterministic policy gradient (TD3)算法中。一种考虑了在policyvalue 函数近似过程中所带来的一些误差对AC框架所带来的影响。

前人算法回顾

  首先回顾一下DPG算法的更新公式:


image.png

image.png

Clipped Double Q-Learning


  Double DQN中的target

image.png

Double Q-learning

image.png

  Clipped Double Q-learning


image.png

这里的image.png的是target actor(可参见伪代码,只用了一个actor)。这种方法会underestimation bias,由于underestimation bias 这种方法就需要加大探索度,不然算法的效率就会很低。

image.png


Addressing Variance


  设置target network用于减小policy更新所带的的方差,不然state value approx会很容易发散,不收敛。

  作者使用policy相比于value做延迟更新(Delayed Policy Updates),这样保证策略更新的时候,先将TD误差最小化,这样不会使得policy更新的时候受误差影响,导致其方差高。


Target Policy Smoothing Regularization


  作者认为similar actions should have similar value,所以对某个action周围加上少许噪声能够使得模型泛化能力更强。

image.png

  • Nachum, O., Norouzi, M., Tucker, G., and Schuurmans, D. Smoothed action value functions for learning gaussian policies. arXiv preprint arXiv:1803.02348, 2018.

算法伪代码:

取得的效果?

  作者与当前的sota算法对比,结果如下:

  作者还验证了target neteork对收敛性的影响:

  最终的实验:

所出版信息?作者信息?


  ICML2018上的一篇文章,Scott Fujimoto is a PhD student at McGill University and Mila. He is the author of TD3 as well as some of the recent developments in batch deep reinforcement learning.

  他还有俩篇论文比较有意思:Off-Policy Deep Reinforcement Learning without ExplorationBenchmarking Batch Deep Reinforcement Learning Algorithms


扩展阅读


  作者为了验证论文的复现性,参考了2017Henderson, P的文章实验了很多随机种子。

  • 参考文献:Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., and Meger, D. Deep Reinforcement Learning that Matters. arXiv preprint arXiv:1709.06560, 2017

  还有一些平衡biasvariance的方法,比如:

  1. importance sampling
  • Precup, D., Sutton, R. S., and Dasgupta, S. Off-policy temporal-difference learning with function approximation. In International Conference on Machine Learning, pp. 417–424, 2001.
  • Munos, R., Stepleton, T., Harutyunyan, A., and Bellemare, M. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1054–1062, 2016.
  1. distributed methods
  • Mnih, V., Badia, A. P., Mirza, M., Graves, A., Lillicrap, T., Harley, T., Silver, D., and Kavukcuoglu, K. Asynchronous methods for deep reinforcement learning. In Internationa lConference on Machine Learning, pp.1928– 1937, 2016.
  • Espeholt, L., Soyer, H., Munos, R., Simonyan, K., Mnih, V., Ward, T., Doron, Y., Firoiu, V., Harley, T., Dunning, I., et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint arXiv:1802.01561, 2018.
  1. approximate bounds
  • He, F. S., Liu, Y., Schwing, A. G., and Peng, J. Learning to play in a day: Faster deep reinforcement learning by optimality tightening. arXiv preprint arXiv:1611.01606, 2016.
  1. reduce discount factor to reduce the contribution of each error
  • Petrik, M. and Scherrer, B. Biasing approximate dynamic programming with a lower discount factor. In Advancesin Neural Information Processing Systems, pp. 1265–1272, 2009.

我的微信公众号名称:深度学习与先进智能决策

微信公众号ID:MultiAgent1024

公众号介绍:主要研究分享深度学习、机器博弈、强化学习等相关内容!期待您的关注,欢迎一起学习交流进步!


相关文章
|
Java 关系型数据库 数据库连接
java中调用postgis的函数找不到,报错[42883] ERROR: function st_area(public.geometry) does not exis,Navicat里面正常
今天在postgresql数据库中对一个图层表使用st_area()函数,获取图层中geometry字段的面积。
|
3月前
|
Python
【Azure 应用服务】Azure Function HTTP Trigger 遇见奇妙的500 Internal Server Error: Failed to forward request to http://169.254.130.x
【Azure 应用服务】Azure Function HTTP Trigger 遇见奇妙的500 Internal Server Error: Failed to forward request to http://169.254.130.x
|
3月前
【Azure 应用服务】Azure Function 启用 Managed Identity后, Powershell Funciton出现 ERROR: ManagedIdentityCredential authentication failed
【Azure 应用服务】Azure Function 启用 Managed Identity后, Powershell Funciton出现 ERROR: ManagedIdentityCredential authentication failed
|
4月前
|
前端开发
Error in created hook: “TypeError: _test.default is not a function
Error in created hook: “TypeError: _test.default is not a function
Fatal error: Call to undefined function openssl_pkey_get_private()
Fatal error: Call to undefined function openssl_pkey_get_private()
74 0
|
6月前
|
Go
Error: Package awesomeProject contains more than one main function Consider using File kind instead
Goland编辑器运行时出现“edit configuration”窗口,阻碍代码执行。解决方法:右键点击源文件运行。问题源于Go语言不支持函数重载,同一包内不能有两个同名函数,导致多入口冲突。初学者在main包中使用了多个Go源文件,应改为仅有一个源码文件来避免此问题。
61 0
error C2449: found ‘{‘ at file scope (missing function header?)和error C2059: syntax error : ‘}‘
error C2449: found ‘{‘ at file scope (missing function header?)和error C2059: syntax error : ‘}‘
122 0
|
6月前
|
计算机视觉 Python
error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor'
error: (-215:Assertion failed) !_src.empty() in function 'cv::cvtColor'
168 0
|
人工智能 自然语言处理 小程序
cloud function service error code -501000, error message 找不到对应的FunctionName.; at cloud.callFunction
cloud function service error code -501000, error message 找不到对应的FunctionName.; at cloud.callFunction
83 0
|
JavaScript Cloud Native Go
Error: Cannot find module ‘webpack/bin/config-yargs‘ at Function.Module._resolveFilename (intern
Error: Cannot find module ‘webpack/bin/config-yargs‘ at Function.Module._resolveFilename (intern
78 0

热门文章

最新文章