Nwb insert config

.github/ISSUE_TEMPLATE/bug_report.md

@@ -12,11 +12,20 @@ A clear and concise description of what the bug is.
 
 **To Reproduce**
 Steps to reproduce the behavior:
+
 1. This error is on file '....' at file path '....'
 2. Click on '....'
 3. Scroll down to '....'
 4. See error
 
+<details><summary>Error Stack</summary>
+
+```python
+# Paste the error stack trace here
+```
+
+</details>
+
 **Expected behavior**
 A clear and concise description of what you expected to happen.
 
@@ -25,4 +34,3 @@ If applicable, add screenshots to help explain your problem.
 
 **Additional context**
 Add any other context about the problem here.
-

.github/ISSUE_TEMPLATE/feature_request.md

@@ -8,13 +8,15 @@ assignees: ''
 ---
 
 **Is your feature request related to a problem? Please describe.**
-A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+A clear and concise description of what the problem is.
+For example, I'm always frustrated when [...]
 
 **Describe the solution you'd like**
 A clear and concise description of what you want to happen.
 
 **Describe alternatives you've considered**
-A clear and concise description of any alternative solutions or features you've considered.
+A clear and concise description of any alternative solutions or features you've
+considered.
 
 **Additional context**
 Add any other context or screenshots about the feature request here.

.github/pull_request_template.md

@@ -5,15 +5,34 @@ include relevant motivation and context. Please list issues fixed or closed by
 This PR.
 
 - Fixes #000: How this PR fixes the issue
-  - `path/file.py`: Description of the change
-  - `path/file.py`: Description of the change
+    - `path/file.py`: Description of the change
+    - `path/file.py`: Description of the change
 - Fixes #000: How this PR fixes the issue
-  - `path/file.py`: Description of the change
-  - `path/file.py`: Description of the change
+    - `path/file.py`: Description of the change
+    - `path/file.py`: Description of the change
 
 # Checklist:
 
+<!--
+For items below with choices, select one (e.g., yes, no) 
+and check the box to indicate that a choice was selected.
+If not applicable, check the box and add `N/A` after the box.
+For example:
+- [X] This has a choice: no
+- [X] N/A. If choice, other item.
+This comment may be deleted on submission.
+
+Alter notes example:
+```python
+from spyglass.example import Table
+Table.alter() # Comment regarding the change
+```
+-->
+
 - [ ] This PR should be accompanied by a release: (yes/no/unsure)
-- [ ] (If release) I have updated the `CITATION.cff`
-- [ ] I have updated the `CHANGELOG.md`
+- [ ] If release, I have updated the `CITATION.cff`
+- [ ] This PR makes edits to table definitions: (yes/no)
+- [ ] If table edits, I have included an `alter` snippet for release notes.
+- [ ] If this PR makes changes to position, I ran the relevant tests locally.
+- [ ] I have updated the `CHANGELOG.md` with PR number and description.
 - [ ] I have added/edited docs/notebooks to reflect the changes

.github/workflows/lint.yml

@@ -1,6 +1,10 @@
 name: Lint
 
-on: [push, pull_request]
+on:
+  push:
+    branches: 
+      - '!documentation'
+  pull_request:
 
 jobs:
   black:

.github/workflows/publish-docs.yml

@@ -5,6 +5,7 @@ on:
       - "*.*.*" # For docs bump, use X.X.XaX
     branches:
       - test_branch
+  workflow_dispatch: # Manually trigger with 'Run workflow' button
 
 permissions:
   contents: write
@@ -40,7 +41,7 @@ jobs:
 
       - name: Deploy
         run: |
-          FULL_VERSION=${{ github.ref }}
+          FULL_VERSION=${{ github.ref_name }}
           export MAJOR_VERSION=${FULL_VERSION:0:3}
           echo "OWNER: ${REPO_OWNER}. BUILD: ${MAJOR_VERSION}"
           bash ./docs/build-docs.sh push $REPO_OWNER

.github/workflows/test-conda.yml

@@ -1,57 +1,80 @@
-name: Test conda env and run tests
+name: Tests
 
 on:
   push:
     branches: 
       - '!test_branch'
+      - '!documentation'
   schedule:  # once a day at midnight UTC
     - cron: '0 0 * * *'
+  workflow_dispatch: # Manually trigger with 'Run workflow' button
+
+concurrency: # Replace Cancel Workflow Action
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
 
 jobs:
   run-tests:
-    runs-on: ${{ matrix.os }}
+    runs-on: ubuntu-latest
     defaults:
      run:
        shell: bash -l {0}
-    strategy:
-      matrix:
-        os: [ubuntu-latest]  #, macos-latest, windows-latest]
     env:
-      OS: ${{ matrix.os }}
-      PYTHON: '3.8'
+      OS: ubuntu-latest
+      PYTHON: '3.9'
+      UCSF_BOX_TOKEN: ${{ secrets.UCSF_BOX_TOKEN }} # for download and testing
+      UCSF_BOX_USER: ${{ secrets.UCSF_BOX_USER }}
+    services:
+      mysql:
+        image: datajoint/mysql:8.0
+        env: # args: mysql -h 127.0.0.1 -P 3308 -uroot -ptutorial -e "CMD;"
+          MYSQL_DATABASE: localhost
+          MYSQL_ROOT_PASSWORD: tutorial
+        ports:
+          - 3308:3306
+        options: --health-cmd="mysqladmin ping" --health-interval=10s --health-timeout=5s --health-retries=3
     steps:
-    - name: Cancel Workflow Action
-      uses: styfle/[email protected]
-      with:
-        access_token: ${{ github.token }}
-        all_but_latest: true
     - name: Checkout
-      uses: actions/checkout@v3
+      uses: actions/checkout@v4
     - name: Set up Python
-      uses: actions/setup-python@v4
+      uses: actions/setup-python@v5
       with:
         python-version: ${{ env.PYTHON }}
-    - name: Set up conda environment
-      uses: conda-incubator/setup-miniconda@v2
+    - name: Set up conda
+      uses: conda-incubator/setup-miniconda@v3
       with:
         activate-environment: spyglass
         environment-file: environment.yml
         miniforge-variant: Mambaforge
         miniforge-version: latest
-    - name: Install spyglass
+        use-mamba: true
+    - name: Install apt dependencies
       run: |
-        pip install -e .[test]
+        sudo apt-get update # First mysql options
+        sudo apt-get install mysql-client libmysqlclient-dev libgirepository1.0-dev -y
+        sudo apt-get install ffmpeg libsm6 libxext6 -y # non-dlc position deps
+    - name: Run pip install for test deps
+      run: | 
+        pip install --quiet .[test]
     - name: Download data
       env:
-        UCSF_BOX_TOKEN: ${{ secrets.UCSF_BOX_TOKEN }}
-        UCSF_BOX_USER: ${{ secrets.UCSF_BOX_USER }}
-        WEBSITE: ftps://ftp.box.com/trodes_to_nwb_test_data/minirec20230622.nwb
+        BASEURL: ftps://ftp.box.com/trodes_to_nwb_test_data/
+        NWBFILE: minirec20230622.nwb # Relative to Base URL
+        VID_ONE: 20230622_sample_01_a1/20230622_sample_01_a1.1.h264
+        VID_TWO: 20230622_sample_02_a1/20230622_sample_02_a1.1.h264
         RAW_DIR: /home/runner/work/spyglass/spyglass/tests/_data/raw/ 
+        VID_DIR: /home/runner/work/spyglass/spyglass/tests/_data/video/ 
       run: |
-        mkdir -p $RAW_DIR
-        wget --recursive --no-verbose --no-host-directories --no-directories \
-        --user $UCSF_BOX_USER --password $UCSF_BOX_TOKEN \
-        -P $RAW_DIR $WEBSITE
+        mkdir -p $RAW_DIR $VID_DIR
+        wget_opts() { # Declare func with download options
+          wget \
+            --recursive --no-verbose --no-host-directories --no-directories \
+            --user "$UCSF_BOX_USER" --password "$UCSF_BOX_TOKEN" \
+            -P "$1" "$BASEURL""$2"
+        }
+        wget_opts $RAW_DIR $NWBFILE
+        wget_opts $VID_DIR $VID_ONE
+        wget_opts $VID_DIR $VID_TWO
     - name: Run tests
       run: |
-        pytest -rP  # env vars are set within certain tests
+        pytest --no-docker --no-dlc 

.github/workflows/test-package-build.yml

@@ -6,38 +6,42 @@ on:
       - master
       - maint/*
       - '!test_branch'
+      - '!documentation'
     tags:
       - "*"
   pull_request:
     branches:
       - master
       - maint/*
+  workflow_dispatch: # Manually trigger with 'Run workflow' button
 defaults:
   run:
     shell: bash
 jobs:
   build:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
         with:
           fetch-depth: 0
-      - uses: actions/setup-python@v4
+      - uses: actions/setup-python@v5
         with:
           python-version: 3.9
-      - run: pip install --upgrade build twine
+      - run: |
+          pip install --upgrade build twine
+          pip install importlib_metadata==7.2.1 # twine #977
       - name: Build sdist and wheel
         run: python -m build
       - run: twine check dist/*
       - name: Upload sdist and wheel artifacts
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: dist
           path: dist/
       - name: Build git archive
         run: mkdir archive && git archive -v -o archive/archive.tgz HEAD
       - name: Upload git archive artifact
-        uses: actions/upload-artifact@v3
+        uses: actions/upload-artifact@v4
         with:
           name: archive
           path: archive/
@@ -50,13 +54,13 @@ jobs:
     steps:
       - name: Download sdist and wheel artifacts
         if: matrix.package != 'archive'
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
           name: dist
           path: dist/
       - name: Download git archive artifact
         if: matrix.package == 'archive'
-        uses: actions/download-artifact@v3
+        uses: actions/download-artifact@v4
         with:
           name: archive
           path: archive/
@@ -73,13 +77,9 @@ jobs:
       - name: Install sdist
         if: matrix.package == 'sdist'
         run: pip install dist/*.tar.gz
-      - name: Install archive
-        if: matrix.package == 'archive'
+      - name: Install archive # requires tag
+        if: matrix.package == 'archive' && startsWith(github.ref, 'refs/tags/')
         run: pip install archive/archive.tgz
-      # - name: Install test extras
-      #   run: pip install spyglass[test]
-      # - name: Run tests
-      #   run: pytest --doctest-modules -v --pyargs spyglass
   publish:
     name: Upload release to PyPI
     runs-on: ubuntu-latest
@@ -89,9 +89,9 @@ jobs:
       url: https://pypi.org/p/spyglass-neuro
     permissions:
       id-token: write  # IMPORTANT: this permission is mandatory for trusted publishing
-    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/') && github.ref == 'refs/heads/master'
+    if: github.event_name == 'push' && startsWith(github.ref, 'refs/tags/')
     steps:
-      - uses: actions/download-artifact@v3
+      - uses: actions/download-artifact@v4
         with:
           name: dist
           path: dist/

.gitignore

@@ -59,6 +59,8 @@ coverage.xml
 *.cover
 .hypothesis/
 .pytest_cache/
+tests/_data/*
+wget-log*
 
 # Translations
 *.mo
@@ -127,6 +129,7 @@ dmypy.json
 .pyre/
 
 # Test Data Files
+tests/_data/*
 *.dat
 *.mda
 *.rec
@@ -136,6 +139,7 @@ dmypy.json
 *.videoTimeStamps
 *.cameraHWSync
 *.stateScriptLog
+tests/_data/*
 
 *.nwb
 *.DS_Store