Soft Actor Critic | Hexo

Soft-Actor Critic(Max Entropy RL to encourage explore)

Reference

1. Reinforcement Learning with Deep Energy-Based Policies
2. Soft Actor-Critic

Annotations and Definitions

Entropy用于刻画分布的离散程度，Entropy越大分布集中程度越低，传统的RL算法(Actor-Critic及其变体)仅仅关注最大化累计奖励，这样的一个潜在问题是算法可能收敛于次优策略（不充分探索），为了解决这个问题，我们将每个state对应的策略Entropy作为奖励函数的一部分以鼓励探索

Definition: object of RL based on Entropy

$$J(\pi) = \sum_t E_{(s_t,a_T)|\rho_\pi}[r(s_t,a_t) + \alpha \mathcal H(\pi(\cdot|s_t))]$$

本质上是将原始来自环境的奖励$r(s_t,a_t)$增添了一项

Definition: State Value and State-Action Value

$$V_\pi(s_0) = E_{\tau |\pi}[\sum_{t=0}^\infty \gamma ^{t} (r(s_t,a_t)+\alpha \mathcal H^\pi(s_t))]\Rightarrow V^\pi(s_t) = E_{a_t|\pi}[Q^\pi(s_t,a_t) -\alpha \log \pi(a_t|s_t)] \\ Q^\pi(s_0,a_0) = E_{\tau |\pi}[r(s_t,a_t) + \sum_{t=1}^\infty \gamma^t (r(s_t,a_t) +\alpha \mathcal H^\pi(s_t))]\Rightarrow Q^{\pi}(s_t,a_t) = r(s_t,a_t) + \gamma E_{s_{t+1}|\rho}[V^\pi(s_{t+1})]$$

Policy Evaluation/Improvement and Iteration

1. Policy Evaluation 给定策略（state到分布的映射），计算V值（Temporal Difference,Single/Multi Steps）
2. Policy Improvement 将计算出的V值视为策略的函数，优化策略提升V值
3. Policy Iteration=Policy Evaluation Policy Improvement迭代进行若干次

Q学习借助TD Error进行Policy Evaluation，借助最大化Q值或最小化TD Error进行Improvement

Soft Policy Evaluation

Entropy框架实现Soft Policy Evaluation的思路和Q Learning相似

$$\begin{aligned} Q^\pi(s_t,a_t) &= r(s_t,a_t) +\gamma E_{s_{t+1}|\rho}[V^\pi(s_{t+1})]\\ &=r(s_t,a_t) + \gamma E_{s_{t+1}|\rho}[Q^\pi(s_{t+1},a_{t+1}) -\alpha \log \pi(a_{t+1}|s_{t+1})]\\ &=r(s_t,a_t) + \gamma E_{s_{t+1}|\rho}(\alpha \mathcal H(s_{t+1}) + E_{s_{t+1}|\rho,a_{t+1}|\pi}[Q^\pi(s_{t+1},a_{t+1})] \end{aligned}$$

Soft Policy Improvement

如果使用Max Q Value选择Action，算法可能退化为确定性算法

$$\pi^\prime(s_t) =\arg\max_{a_t}Q^\pi(s_t,a_t)\label{PI}\tag{Policy Improvement}$$

Max-Entropy RL的最大改变在添加Entropy到RL框架后，最优策略退化为正比于Q值指数的概率分布，可以证明采取$\ref{PI}$并添加Entropy到Q函数中则Policy Improvement的结果变为

$$\pi^\prime(a_t|s_t)=\frac{\exp{Q^\pi(s_t,a_t)}/\alpha}{Z_t}\\ Z_t= \sum_{a}\exp{\frac{Q^\pi(s_t,a_t)}{\alpha }}$$

并且按照这种方式进行策略更新Q函数满足

$$Q^\pi(s,a) \leq Q^{\pi^\prime}(s,a)$$

证明的关键是将$Q^\pi$拆分为当前策略$\pi$和最优策略$\pi^\prime$之间的KL Divergence

$$\begin{aligned} V^\pi(s)& = E_{a|\pi}[Q^\pi(s,a) - \alpha \log \pi(a|s)]\\ &=-E_{a|\pi}[\alpha \log \pi(a|s) -\alpha \log \exp \frac{Q^\pi(s,a)}{\alpha} +\alpha\log Z] +\alpha \log Z\\ &=-E_{a|\pi}[\alpha \log \pi(a|s) -\alpha \log \pi^\prime(a|s)]+\alpha \log Z\\ &=-\alpha D_{KL}(\pi(\cdot|s)\parallel \pi^\prime(\cdot|s)) +\alpha \log Z \end{aligned}$$

进行policy update时Q视为常数，因此$\pi =\pi^\prime$时完成一轮Policy Improvement

显然Maximize $V^\pi(s)$的策略满足

$$\pi =\pi^\prime$$

此时

$$V^\pi(s)\leq V^{\pi^\prime}(s)=E_{a|\pi^\prime}[Q^\pi(s,a) - \log \pi^\prime(a|s)] = \mathcal H^{\pi^\prime}(s) +E_{a|\pi^\prime}[Q^\pi(s,a)]$$

进一步借助Q-V关系

$$\begin{aligned} Q^\pi(s,a) & = E_{s_1|\rho}[r_0+\gamma V^\pi(s_1)]\\ &\leq r(s,a) + \gamma E_{s_1\in \rho}[\mathcal H^{\pi^\prime}(s_1) + E_{a|\pi^\prime}[Q^\pi(s,a)]]\leq \cdots \leq Q^{\pi^\prime}(s,a) \end{aligned}$$