Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor flashloader #329

Merged
merged 80 commits into from
Jun 30, 2024
Merged
Changes from 1 commit
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
58dbcd7
major refactor to flash code
zain-sohail Nov 25, 2023
f376387
update dataframe class to be able to use index and dataset keys
zain-sohail Nov 25, 2023
08e8d9f
minor changes introduced
zain-sohail Nov 29, 2023
5c9a04c
change majorly the class with a new initialize method. now save parqu…
zain-sohail Nov 29, 2023
ff5dd07
now uses a simpler notation and save_parquet method after loading dat…
zain-sohail Nov 29, 2023
7852aaf
methods made more consistent and fixing the get_index_dataset_key
zain-sohail Nov 29, 2023
ac9abea
include steinn's proposed solution to pulse_id channel being empty
zain-sohail Nov 29, 2023
41fd70d
include unit tests and fixtures. still many to be done. needs to move…
zain-sohail Nov 29, 2023
da00635
add more tests, simplify logic on dataframe class
zain-sohail Dec 1, 2023
8b39bdb
remove the gmdTunnel channel because the datafile is not correct. Rep…
zain-sohail Dec 3, 2023
e1b9a9f
major structure changes
zain-sohail Dec 11, 2023
cd85dfd
docstrings etc
zain-sohail Dec 11, 2023
f6ca14e
updated buffer creation etc. tests won't work currently
zain-sohail Dec 12, 2023
c9f1fcc
fix linting errors and comment out tests for now
zain-sohail Dec 12, 2023
1398bf2
fix the error of getting wrong attribute in loader, and fix parquet l…
zain-sohail Dec 13, 2023
eb72230
fix lint error
zain-sohail Dec 13, 2023
4d950db
cleaning up the classes
zain-sohail Jan 6, 2024
b8bfdf0
add back easy access apis
zain-sohail Jan 6, 2024
1f95408
small fix
zain-sohail Jan 6, 2024
8f551d0
small fix
zain-sohail Jan 6, 2024
c85fdec
small fix
zain-sohail Jan 6, 2024
0a7e836
fix error with pickling
zain-sohail Jan 6, 2024
4a787eb
use old cfg
zain-sohail Jan 6, 2024
084f407
docstrings fixes
zain-sohail Jan 7, 2024
73802fa
fix tests
zain-sohail Jan 7, 2024
70a3c5b
fix certain problems with df_electron and add comphrehensive tests fo…
zain-sohail Jan 8, 2024
77bf46b
add tests
zain-sohail Jan 8, 2024
d8cc6f6
buffer handler tests
zain-sohail Jan 8, 2024
09cffec
ruff formated
zain-sohail Jan 8, 2024
0f23ddb
add parquethandler tests
zain-sohail Jan 8, 2024
ac4f8cd
further tests
zain-sohail Jan 8, 2024
1519752
fixes
zain-sohail Jan 8, 2024
d31e6b1
fix the lint error
zain-sohail Jan 8, 2024
ed18a5c
fix parse_metadata
zain-sohail Mar 27, 2024
ce8134f
put everything in one file
zain-sohail Mar 27, 2024
08a2adc
reoder
zain-sohail Mar 27, 2024
74b41dc
update interface from suggestions
zain-sohail Mar 27, 2024
b937db8
limit the cores used
zain-sohail Mar 27, 2024
9dc69aa
change interface of parquethandler to suggested
zain-sohail Mar 27, 2024
09a93d3
fix bug for df indexing
zain-sohail Mar 28, 2024
55cfa0c
merge main branch
zain-sohail Apr 18, 2024
4b3e6f7
Merge branch 'main' into refactor-flashloader
zain-sohail Apr 27, 2024
d316137
lint fix
zain-sohail Apr 27, 2024
c00207d
update dataframe saving and loading from parquet behavior
zain-sohail Apr 27, 2024
89130b0
remove saving/loading of parquets
zain-sohail May 19, 2024
dbef804
add instrument option
zain-sohail May 19, 2024
afd9772
fix tests
zain-sohail May 19, 2024
b9fce76
fix tests
zain-sohail May 19, 2024
6400878
fix tests
zain-sohail May 19, 2024
7129f57
fix tests
zain-sohail May 19, 2024
bc53214
Merge branch 'main' into refactor-flashloader
zain-sohail Jun 5, 2024
02aee6e
- added retrocompabtibility for older buffer files that have sectorID…
zain-sohail Jun 5, 2024
2142c11
fix ruff settings
zain-sohail Jun 5, 2024
79922ef
update tests
zain-sohail Jun 5, 2024
02ae74e
make small change to check actions status
zain-sohail Jun 5, 2024
f520310
bring back types
zain-sohail Jun 5, 2024
4c7d069
fix small error
zain-sohail Jun 5, 2024
9f6a31b
move utility func test to utility tests
zain-sohail Jun 6, 2024
f4a30e0
seperate to different modules
zain-sohail Jun 10, 2024
08f8f13
add time_elapsed method
zain-sohail Jun 10, 2024
fa68746
fix test issues
zain-sohail Jun 10, 2024
cb884dd
add tests for elapsed time
zain-sohail Jun 10, 2024
ae26555
fix main loader tests
zain-sohail Jun 10, 2024
04714bc
fix sxp loader tests
zain-sohail Jun 10, 2024
6589595
fix tests
zain-sohail Jun 10, 2024
1b73b76
fix minor issue with repr html
zain-sohail Jun 12, 2024
852a867
add available runs property
zain-sohail Jun 12, 2024
f010a2e
Merge branch 'main' into refactor-flashloader
zain-sohail Jun 13, 2024
8dd5e6a
Merge branch 'main' into refactor-flashloader
zain-sohail Jun 18, 2024
cd6fbf0
Merge branch 'v1_feature_branch' into refactor-flashloader
zain-sohail Jun 24, 2024
147e913
add back annotations
zain-sohail Jun 24, 2024
f2a26b9
use index and dataset keys
zain-sohail Jun 24, 2024
ebd2b32
Merge remote-tracking branch 'origin/v1_feature_branch' into refactor…
rettigl Jun 26, 2024
d131fe4
remove nans from all electron channels
zain-sohail Jun 28, 2024
194c874
use pd import, load h5 file inside df creator
zain-sohail Jun 28, 2024
af33740
update comments to explain the code
zain-sohail Jun 28, 2024
50f7ee1
make review changes
zain-sohail Jun 28, 2024
65d909d
fix tests with review comments
zain-sohail Jun 28, 2024
b7537a8
fix dropna
zain-sohail Jun 28, 2024
b0b090d
fix minor stuff and add test to see if exception handling works in pa…
zain-sohail Jun 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix tests
zain-sohail committed May 19, 2024
commit b9fce76ba96f97261395c742f6be46df4c24d1eb
4 changes: 2 additions & 2 deletions sed/loader/sxp/loader.py
Original file line number Diff line number Diff line change
@@ -53,7 +53,7 @@ def __init__(self, config: dict) -> None:
self.failed_files_error: List[str] = []
self.array_indices: List[List[slice]] = None

def initialize_paths(self) -> Tuple[List[Path], Path]:
def initialize_dirs(self) -> Tuple[List[Path], Path]:
"""
Initializes the paths based on the configuration.

@@ -940,7 +940,7 @@ def read_dataframe(
"""
t0 = time.time()

data_raw_dir, data_parquet_dir = self.initialize_paths()
data_raw_dir, data_parquet_dir = self.initialize_dirs()

# Prepare a list of names for the runs to read and parquets to write
if runs is not None:
14 changes: 7 additions & 7 deletions tests/loader/test_loaders.py
Original file line number Diff line number Diff line change
@@ -163,7 +163,7 @@ def test_has_correct_read_dataframe_func(loader: BaseLoader, read_type: str) ->

if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))

@@ -196,7 +196,7 @@ def test_timed_dataframe(loader: BaseLoader) -> None:
if loaded_timed_dataframe is None:
if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))
pytest.skip("Not implemented")
@@ -206,7 +206,7 @@ def test_timed_dataframe(loader: BaseLoader) -> None:

if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))

@@ -240,7 +240,7 @@ def test_get_count_rate(loader: BaseLoader) -> None:
if loaded_time is None and loaded_countrate is None:
if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))
pytest.skip("Not implemented")
@@ -251,7 +251,7 @@ def test_get_count_rate(loader: BaseLoader) -> None:

if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))

@@ -285,7 +285,7 @@ def test_get_elapsed_time(loader: BaseLoader) -> None:
if elapsed_time is None:
if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))
pytest.skip("Not implemented")
@@ -296,7 +296,7 @@ def test_get_elapsed_time(loader: BaseLoader) -> None:

if loader.__name__ in {"flash", "sxp"}:
loader = cast(FlashLoader, loader)
_, parquet_data_dir = loader.initialize_paths()
_, parquet_data_dir = loader.initialize_dirs()
for file in os.listdir(Path(parquet_data_dir, "buffer")):
os.remove(Path(parquet_data_dir, "buffer", file))