Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Python 3.9/3.11 support, dropping Python 3.6 support, and moving to GitHub Actions #559

Merged
merged 14 commits into from
Dec 13, 2023

Conversation

sheriferson
Copy link
Contributor

@sheriferson sheriferson commented Nov 17, 2023

Overview:

This PR does the following:

  • Drops support for Python 3.6
  • Updates dependencies and code to run on Python 3.9/3.11
  • Moves IPFX tests to GitHub Actions (and runs onprem tests using self-hosted runner)

Type of Fix:

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing
    functionality to not work as expected)
  • Documentation Change

Unit Tests:

See here for the light and onprem test results from my fork:
https://github.com/sheriferson/ipfx/actions/runs/6898536678

Checklist

  • My code follows
    Allen Institute Contribution Guidelines
  • My code is unit tested and does not decrease test coverage
  • I have performed a self review of my own code
  • My code is well-documented, and the docstrings conform to
    Numpy Standards
  • I have updated the documentation of the repository where
    appropriate
  • The header on my commit includes the issue number
  • My code passes all tests
  • I have updated the CHANGELOG.md with the description of changes understandable by end users

Notes

I believe this addresses #558, #557, and #550. With the move away from Python 3.6 and the unpinning of most dependencies, pip install -e . succeeds on my M* Mac.

Action items

  • Deprecate CircleCI as well (?)
  • Nightly workflow not detected by GitHub Actions yet. Maybe after merging into the main branch.
  • Disable/deprecate outgoing internal CI/CD

@sheriferson sheriferson marked this pull request as ready for review November 17, 2023 22:28
README.md Outdated Show resolved Hide resolved
docker/py39/Dockerfile Outdated Show resolved Hide resolved
docker/py39/Dockerfile Outdated Show resolved Hide resolved
@@ -236,36 +235,3 @@ def serialize(self, targets: Optional[OneOrMany[Dict[str, Any]]] = None):
file_.write(self._data.getvalue())

self._reload_nwbfile()


def set_container_sources(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm going to invoke Chesterton's Fence here and ask for an explanation of what this existing code does? Why is it safe to remove this now but not previously?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree with the invocation.
But I've been looking through #361 and the commit making the changes (1aeb641) and I am yet to find a good source of documentation for why the method existed.
There are a couple of internal confluence pages that I'm trying to get access to in case they are enlightening.
If you have suggestions for where to look or who to check with, please let me know.

TEST_NWB_FILE_PATH = 'tests/data/nwb/Ctgf-T2A-dgCre;Ai14-495723.05.02.01.nwb'

def test_is_file_mies():
assert is_file_mies(TEST_NWB_FILE_PATH) == True
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add a test to test for the other type? I think it was HBG?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no specific test/check for HBG files. The usage is checking if the file is MIES, and if not it assumes it's HBG.
There is one nwb file in the repo that is not MIES, I could create a test that shows that the NWB data loads successfully when HBGNWBData class is used?

Copy link
Contributor

@njmei njmei Nov 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have this requirements.txt be as flexible as possible in terms of dependency pinning. But we should have have our tests use a pinned-requirements.txt so that future maintainers don't get constantly hit with dependency update related test failures.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't you want to know if the actual requirements.txt file customers are using will cause test failures either on install or on run? (likely because a dependency update caused an issue) I'm not following why it's a good idea to use different dependencies for tests vs. when pip installing.

@sheriferson sheriferson mentioned this pull request Dec 8, 2023
@njmei njmei self-requested a review December 12, 2023 17:07
After updating PyNWB, the pre-existing serialize method in the Nwb2Sink
class was throwing errors along the form of:

ValueError: Cannot change container_source once set: 'root' pynwb.file.NWBFile

This commit resolves this error by removing the use of set_container_sources()

After removal of the set_container_sources() method, tests pass and
testing done by scientists also did not reveal any issues, but because
the origin/reason for this code is not clear, we're not sure that
removing it doesn't break something else with NWB files, for example
with publishing to DANDI.

Should you run into issues with containers or other metadata in NWB
files, the work of the removed code might be the explanation. However
it's not clear how to modify that code to get around the original error
mentioned above.
Testing: re-created the images on the host for the self-hosted runner
and re-ran the onprem tests.
@sheriferson sheriferson force-pushed the feature/github-actions branch from c814d79 to 9643dc1 Compare December 13, 2023 01:51
@sheriferson sheriferson merged commit 622b8ff into AllenInstitute:master Dec 13, 2023
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants