Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mkv #20

Merged
merged 81 commits into from
Sep 18, 2024
Merged

Mkv #20

Changes from 1 commit
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
bbb1644
basic structure
KeplerC Aug 19, 2024
9309045
Refactor Trajectory class to improve frame encoding and add support f…
KeplerC Aug 20, 2024
d394d67
fix loading
KeplerC Aug 20, 2024
daaaa57
doesnt work, consider migrate robot data loader over
KeplerC Aug 20, 2024
70dc889
static works
KeplerC Aug 20, 2024
5b0b462
feat: Improve frame encoding and add support for different encodings …
KeplerC Aug 21, 2024
9bf9eea
decode
KeplerC Aug 21, 2024
9d66f32
h5 cache
KeplerC Aug 21, 2024
c389804
figure out the issue of remuxing due to the context
KeplerC Aug 21, 2024
d875a0a
it works without h264
KeplerC Aug 21, 2024
e9f051e
fix the decoding bug and silient logs
KeplerC Aug 21, 2024
32d3dac
Refactor Trajectory class to improve frame encoding and add support f…
KeplerC Aug 21, 2024
945ddb0
feat: Add support for pre-initialized H.264 video streams in Trajecto…
KeplerC Aug 21, 2024
a3e2c34
Refactor Trajectory class to remove commented code and improve code r…
KeplerC Aug 21, 2024
c7c9284
Refactor Trajectory class to remove commented code and improve code r…
KeplerC Aug 21, 2024
b627e75
Refactor Trajectory class to improve code readability and remove comm…
KeplerC Aug 21, 2024
00d3d5e
init robot data loader structure
KeplerC Aug 22, 2024
5cfa061
convert from openx
KeplerC Aug 22, 2024
0e5226f
Refactor Trajectory class to improve frame encoding and add support f…
KeplerC Aug 22, 2024
e41675a
feat: Add HDF5Loader to support loading HDF5 files in fog_x/loader/__…
KeplerC Aug 22, 2024
c34a7c0
add h5 accessing
KeplerC Aug 22, 2024
239c230
code formatting
KeplerC Aug 22, 2024
2f10e68
Refactor Trajectory class to remove commented code and improve code r…
KeplerC Aug 25, 2024
8f40ff8
benchmark code, missing container loader info
KeplerC Aug 25, 2024
301f385
Refactor Trajectory class to remove commented code and improve code r…
KeplerC Aug 25, 2024
4545447
Refactor Trajectory class to improve code readability and remove comm…
KeplerC Aug 25, 2024
d594e39
Refactor Trajectory class to add optional cache path for storing cach…
KeplerC Aug 25, 2024
430c73b
Refactor Trajectory class to clear cache directory and improve code r…
KeplerC Aug 25, 2024
0220880
Refactor Trajectory class for improved code readability and removal o…
KeplerC Aug 25, 2024
e280615
Refactor Trajectory class to fix cache directory path
KeplerC Aug 25, 2024
87fecf1
Refactor Trajectory class to add HDF5Handler for converting data to H…
KeplerC Aug 25, 2024
b4254e8
save stream to a different file doesnt work yet
KeplerC Aug 25, 2024
058fd5b
Refactor VLALoader class to update file path for loading VLA data
KeplerC Aug 25, 2024
1a3bee4
Refactor DatasetHandler to clear OS cache after loading data
KeplerC Aug 26, 2024
a8a05ef
Refactor Trajectory class to improve code readability and add lazy lo…
KeplerC Aug 26, 2024
c866200
Refactor prepare function for improved code readability and consistency
KeplerC Aug 26, 2024
6e9c5bf
Refactor VLAHandler.measure_loading_time() to recursively load h5 dat…
KeplerC Aug 26, 2024
1cfcc27
Refactor Trajectory class to improve code readability and add lazy lo…
KeplerC Aug 26, 2024
975c4e5
Refactor Trajectory class to improve code readability and add lazy lo…
KeplerC Aug 26, 2024
e83e6da
Refactor RLDSLoader class to improve code readability and add lazy lo…
KeplerC Aug 26, 2024
4ba6453
fix tf record's benchmark to read the data
KeplerC Aug 26, 2024
2c4d797
support no cache baseline
KeplerC Aug 26, 2024
1cb9ee5
add visualization results
KeplerC Aug 26, 2024
d14c4ab
add DL dataset
KeplerC Aug 27, 2024
d430709
add basic support for octo integration
KeplerC Aug 27, 2024
3467b3f
fix a bug in loading
KeplerC Aug 27, 2024
b79068a
octo dataloader working
KeplerC Aug 27, 2024
5600985
Refactor RLDSLoader and Trajectory classes to improve code readabilit…
KeplerC Aug 27, 2024
1f445df
Refactor RLDSLoader and Trajectory classes to improve code readabilit…
KeplerC Aug 27, 2024
0333a09
open x dataset converter streamline
KeplerC Aug 30, 2024
d5c7332
support both lossy and lossless compression
KeplerC Aug 30, 2024
1567b44
chore: Update DEFAULT_DATASET_NAMES in openx.py and add lossy_compres…
KeplerC Aug 30, 2024
3953f7f
chore: Update vla_to_h5.py to process VLA data and convert it to HDF5…
KeplerC Aug 30, 2024
3fcfc43
Refactor RLDSLoader and Trajectory classes to improve code readabilit…
KeplerC Aug 30, 2024
c0a840f
Refactor RLDSLoader and Trajectory classes for improved code readabil…
KeplerC Aug 30, 2024
8e0b188
fix the logging; before randomly access
KeplerC Aug 30, 2024
a0e813a
add random loading
KeplerC Aug 30, 2024
143a4fe
async write to cache
KeplerC Aug 31, 2024
b13ae79
support dataloader, still has bugs
KeplerC Aug 31, 2024
305d8b6
support lerobot
KeplerC Aug 31, 2024
604486c
Refactor RLDSLoader and Trajectory classes to improve code readabilit…
KeplerC Aug 31, 2024
466c5cb
write as pytorch dataloader
KeplerC Aug 31, 2024
eccf7b2
multi proces to sped up
KeplerC Aug 31, 2024
e4913b1
chore: Refactor Trajectory class for improved code readability and ef…
KeplerC Aug 31, 2024
01841ee
Refactor Trajectory and VLAIterableDataset classes for improved code …
KeplerC Aug 31, 2024
5f5d328
fix rlds and add debug
KeplerC Sep 1, 2024
c4d7150
Refactor DatasetHandler class for improved code readability and perfo…
KeplerC Sep 1, 2024
fcd8f2d
Refactor evaluation script for improved code organization and perform…
KeplerC Sep 2, 2024
3516491
Refactor evaluation script for improved code organization and perform…
KeplerC Sep 2, 2024
d264839
Refactor VLA loader to use PyTorch dataloader for improved code reada…
KeplerC Sep 2, 2024
79117e6
Refactor loader modules to improve code organization and performance
KeplerC Sep 2, 2024
0670995
Refactor LeRobotLoader to load one episode at a time for improved per…
KeplerC Sep 2, 2024
e618571
Refactor LeRobotLoader to load one episode at a time for improved per…
KeplerC Sep 2, 2024
066abe9
Refactor RLDSLoader and HDF5Loader to use smaller shuffle buffer size…
KeplerC Sep 2, 2024
74d6c38
fix bugs that prevents rlds and lr to move forward after iterating th…
KeplerC Sep 2, 2024
47bce7e
fix size issue
KeplerC Sep 3, 2024
f14a943
backward support octo and rlds conversion
KeplerC Sep 3, 2024
04f3b4a
fix bug of resizing width
KeplerC Sep 7, 2024
0a0542d
data, etc.
KeplerC Sep 10, 2024
a35a695
submitted version
KeplerC Sep 18, 2024
3868a85
Merge branch 'mkv' of https://github.com/BerkeleyAutomation/fog_x int…
KeplerC Sep 18, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Refactor Trajectory class to fix cache directory path
  • Loading branch information
KeplerC committed Aug 25, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit e280615ea5b1375792b7c688fe707c73ae57b058
3 changes: 2 additions & 1 deletion benchmarks/openx.py
Original file line number Diff line number Diff line change
@@ -12,7 +12,7 @@
DEFAULT_EXP_DIR = "/tmp/fog_x"
DEFAULT_NUMBER_OF_TRAJECTORIES = 5
DEFAULT_DATASET_NAMES = ["berkeley_autolab_ur5"]
CACHE_DIR = "/tmp/fog_x/cache"
CACHE_DIR = "/tmp/fog_x/cache/"

class DatasetHandler:
"""Base class to handle dataset-related operations."""
@@ -40,6 +40,7 @@ def check_and_download_file(self, url, local_path):
subprocess.run(["gsutil", "-m", "cp", url, local_path], check=True)
else:
print(f"File {local_path} already exists. Skipping download.")

def check_and_download_trajectory(self, trajectory_index):
"""Checks if a trajectory and associated JSON files are already downloaded; if not, downloads them."""
os.makedirs(self.dataset_dir, exist_ok=True)