Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RLlib; Offline RL] Offline RL example that shows how to customize an offline data pipeline. #49046

Conversation

simonsays1980
Copy link
Collaborator

@simonsays1980 simonsays1980 commented Dec 3, 2024

Why are these changes needed?

RLlib's new Offline RL was built to be more powerful as its old stack one and competitive to the SOTA standards out in the market. Part of it is its highly customizable pipeline that can be used to literally read any data from
local or remote storage and transform data from raw format to SingleAgentEpisode instances to feed the learner connector pipeline and further to MultiAgentBatch format ready-to-train on the Learner.

This example is the first of multiple ones to show users the cacabilities of RLlib's new Offline RL API. It shows

  • how to read in raw image data from a cloud bucket
  • how to transform this data via byte streams to numpy arrays
  • how to further transform these arrays into SingleAgentEpisode format to be readable by ConnectorV2 pieces
  • how to convert further into MultiAgentBatch format to pass to a Learner's update method.

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

…eLearner._map_to_episodes' methods.

Signed-off-by: simonsays1980 <[email protected]>
… for it. In addition added keyword arguments to the 'OfflinePreLearner' mmethods such that overriding offers more customization.

Signed-off-by: simonsays1980 <[email protected]>
… how to customize an offline data pipeline.

Signed-off-by: simonsays1980 <[email protected]>
@simonsays1980 simonsays1980 marked this pull request as ready for review December 3, 2024 19:14
@sven1977 sven1977 changed the title [RLlib; Offline RL] - Offline RL example that shows how to customize an offline data pipeline. [RLlib; Offline RL] Offline RL example that shows how to customize an offline data pipeline. Dec 3, 2024
Copy link
Contributor

@sven1977 sven1977 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great! Thanks for the quick turnaround @simonsays1980 !

Just the one question on the commented-out line. ??

@sven1977 sven1977 enabled auto-merge (squash) December 3, 2024 20:57
@github-actions github-actions bot added the go add ONLY when ready to merge, run all tests label Dec 3, 2024
@sven1977 sven1977 self-assigned this Dec 3, 2024
@sven1977 sven1977 added rllib RLlib related issues rllib-offline-rl Offline RL problems rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-newstack labels Dec 3, 2024
@sven1977 sven1977 disabled auto-merge December 3, 2024 20:58
@sven1977 sven1977 enabled auto-merge (squash) December 4, 2024 12:23
@sven1977 sven1977 merged commit 10daade into ray-project:master Dec 4, 2024
6 checks passed
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this pull request Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
go add ONLY when ready to merge, run all tests rllib RLlib related issues rllib-docs-or-examples Issues related to RLlib documentation or rllib/examples rllib-newstack rllib-offline-rl Offline RL problems
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants