Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update clearpiperun to use raw SQL #775

Merged
merged 46 commits into from
Oct 29, 2024
Merged

Update clearpiperun to use raw SQL #775

merged 46 commits into from
Oct 29, 2024

Conversation

ddobie
Copy link
Contributor

@ddobie ddobie commented Oct 17, 2024

Fix #690.

shiblisaleheen and others added 25 commits July 26, 2024 09:56
@shiblisaleheen
timing and memory benchmark
8770187
@shiblisaleheen
delete raw initial
f97605c
@shiblisaleheen
adding profiler
03e3266
@shiblisaleheen
optimisation handling exceptions
bf4e91e
@ddobie
Added logging
3297d4f
@ddobie
Updated delete_run
4e75b8c
@ddobie
Fix syntax errors
dcbafe5
@ddobie
Disable triggers to see if that fixes speed issues
ee9de93
@ddobie
Remove memory profiling
98da705
@ddobie
Reenabled logging
9783865
@ddobie
Add end of loop logging, remove tqdm
bb49f84
@ddobie
Remove all tqdm, improve logging slightly
3c7c35b
@ddobie
Added timing
88738e7
@ddobie
Fixed tqdm missing
68b5f94
@ddobie
Fix logging
2c65e7d
@ddobie
Added units to logging
b9bec73
@ddobie
specify source id in logging
9fcb727
@ddobie
Toggle triggers
c7026ab
@ddobie
clean up clearpiperun
a781a8a
@ddobie
Other minor updates
2aacc61
@ddobie
Fix variable name
e4b7c8d
@ddobie
Correctly handle images and skyregions that are associated with multi…
ab25282 
…ple runs
@ddobie
PEP8
892c695
@ddobie
Merge branch 'dev' into adacs_delete_optimisation_ddobie
bfbc131
@ddobie
Updated changelog
6827654
@ddobie ddobie marked this pull request as ready for review October 18, 2024 04:44
@ddobie
Copy link
Contributor Author

ddobie commented Oct 18, 2024 •
edited
Loading

This now passes tests, but I'm not convinced that my method of checking whether an image/skyregion is associated with more than one run is actually efficient. It does work though.

@ddobie ddobie requested a review from mauch October 21, 2024 00:25
mauch
mauch approved these changes Oct 23, 2024
View reviewed changes
Copy link
Contributor

@mauch mauch left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly just lots of little f-string in log niggles :)
I tested and it works fine for me - you should test again to make sure my suggested changes are correct.

ddobie and others added 10 commits October 23, 2024 13:54
@ddobie @mauch
Update vast_pipeline/management/commands/clearpiperun.py
2f90ca6 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/utils/delete_run.py
8b9007d 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/utils/delete_run.py
4f8d4ae 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/utils/delete_run.py
860c44f 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/utils/delete_run.py
f90b53b 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/utils/delete_run.py
f904748 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/utils/delete_run.py
bf72de5 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/management/commands/clearpiperun.py
156c507 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/management/commands/clearpiperun.py
2e3a699 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie @mauch
Update vast_pipeline/management/commands/clearpiperun.py
000b92b 
Co-authored-by: Tom Mauch <[email protected]>
@ddobie
Copy link
Contributor Author

ddobie commented Oct 23, 2024

Thanks! Will give it another test locally before merging just to make sure :)

@ddobie ddobie merged commit 39d7c99 into dev Oct 29, 2024
5 checks passed
@ddobie ddobie deleted the adacs_delete_optimisation_ddobie branch October 29, 2024 00:19
ddobie added a commit that referenced this pull request Nov 7, 2024
@ddobie @shiblisaleheen @mauch
v1.2.0 release (#778)
a6a8d99 
* Organise v1.1.1-dev

* Fix changelog formatting and update changelog instructions (#772)

* Initial changelog formatting issues

* Update changelog + instructions

* Updated changelog

* Updated Code of conduct (#773)

* Updated Code of conduct

* Updated changelog

* Fixed grammar

* Fix zenodo DOI

* Fixed typo in README

* Shorten forced fit measurement names (#734)

* Shorten names

* Updated changelog

* Update clearpiperun to use raw SQL (#775)

* timing and memory benchmark

* delete raw initial

* adding profiler

* optimisation handling exceptions

* Added logging

* Updated delete_run

* Fix syntax errors

* Disable triggers to see if that fixes speed issues

* Remove memory profiling

* Reenabled logging

* Add end of loop logging, remove tqdm

* Remove all tqdm, improve logging slightly

* Added timing

* Fixed tqdm missing

* Fix logging

* Added units to logging

* specify source id in logging

* Toggle triggers

* clean up clearpiperun

* Other minor updates

* Fix variable name

* Correctly handle images and skyregions that are associated with multiple runs

* PEP8

* Updated changelog

* Remove commented code

* Remove whitespace - don't know why the linter didn't pick this up

* Update vast_pipeline/management/commands/clearpiperun.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/management/commands/clearpiperun.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/management/commands/clearpiperun.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/management/commands/clearpiperun.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Update vast_pipeline/utils/delete_run.py

Co-authored-by: Tom Mauch <[email protected]>

* Fix logging count

* Clean up logging statements

---------

Co-authored-by: Shibli Saleheen <[email protected]>
Co-authored-by: Tom Mauch <[email protected]>

* Quick memory optimisations (#776)

* Use itertuples over iterrows since iterrows is an enormous memory hog.

* Drop sources_df columns before renaming id column to avoid a copy of the while dataframe in memory.

* Decrease default partition size to 15MB

* Dont split (large-in-memory) list of DataFrames into dask bags (No performance hit).

* Don't write forced parquets in parallel (No perfomance hit for this).

* Dont overwrite input DataFrame when writing parquets.

* Update CHANGELOG.md

* Address review comments.

* Copy YAML objects before revalidation so the can be garbage collected.

* Appease flake8

* 750 configure workers (#777)

* Use itertuples over iterrows since iterrows is an enormous memory hog.

* Drop sources_df columns before renaming id column to avoid a copy of the while dataframe in memory.

* Decrease default partition size to 15MB

* Dont split (large-in-memory) list of DataFrames into dask bags (No performance hit).

* Don't write forced parquets in parallel (No perfomance hit for this).

* Initial configuration updates for processing options.

* Dont overwrite input DataFrame when writing parquets.

* Update CHANGELOG.md

* Address review comments.

* Copy YAML objects before revalidation so the can be garbage collected.

* Appease flake8

* Add processing options as optional with defaults.

* filter processing config to parallel association.

* Add a funtion to determine the number of workers and partitions for Dask.

* Use config values for num_workers and max_partition_size throughout pipeline.

* Correct working in config template.

* Update CHANGELOG.md

* Remove unused imports.

* Bump strictyaml to 1.6.2

* Use YAML 'null' to create Python None for all cores option.

* Make None the default in `calculate_workers_and_partitions` instead of 0

* Updated run config docs

* Allow null for num_workers_io and improve validation of processing parameters.

* Update num_workers_io default in docs.

---------

Co-authored-by: Dougal Dobie <[email protected]>

* Prepare v1.2.0 release

---------

Co-authored-by: Shibli Saleheen <[email protected]>
Co-authored-by: Tom Mauch <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mauch mauch mauch approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

clearpiperun consuming all available memory
3 participants
@ddobie @mauch @shiblisaleheen