[rllib] Revert [rllib] Port DDPG to the build_tf_policy pattern

[ericl]

This reverts #5242 to fix #5604

Two issues: (1) blowup of Q-values during training of MountainCar (and also Pendulum if you use higher tau values), (2) eager mode acts differently from graph mode somehow.

These are likely due to a subtle bug in the port to the build_tf_policy pattern. We should re-visit the port, but this PR reverts it quickly for 0.7.4.

[ericl]

@pcmoritz can we also cherry pick this into the 0.7.4. branch?

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16753/
Test FAILed.

[pcmoritz]

Sounds good, yeah! Shall we also merge it into master?

[ericl]

Yep

…

On Tue, Sep 3, 2019, 3:55 PM Philipp Moritz ***@***.***> wrote: Sounds good, yeah! Shall we also merge it into master? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#5626?email_source=notifications&email_token=AAADUSVQMRPW2J2LMUAZL3DQH3TPBA5CNFSM4ITK7DTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5Z2AKA#issuecomment-527671336>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAADUSWISPVBJEHOY5JVQ6LQH3TPBANCNFSM4ITK7DTA> .

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16765/
Test FAILed.

[pcmoritz]

There is another Jenkins error (it might involve copying a lot of code back, since it is shared by the SAC agent).

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16781/
Test FAILed.

[pcmoritz]

Jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16785/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16790/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16791/
Test FAILed.

[pcmoritz]

Jenkins retest this please

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16792/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/16793/
Test FAILed.