Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add tutorial for DynamicTableRegion and Linked Tables #648

Closed
5 tasks done
oruebel opened this issue Jul 19, 2021 · 1 comment
Closed
5 tasks done

Add tutorial for DynamicTableRegion and Linked Tables #648

oruebel opened this issue Jul 19, 2021 · 1 comment
Assignees
@mavaylon1
Labels
category: enhancement improvements of code or code behavior help wanted: good first issue request for community contributions that are good for new contributors topic: docs Issues related to documentation
Milestone
Future

Comments

@oruebel
Copy link
Contributor

oruebel commented Jul 19, 2021

It would be useful to have a tutorial dedicated to DynamicTableRegion and how to use to: 1) link to metadata stored in tables, and 2) link between tables (similar to a foreign key in a database). This tutorial should cover:

  • Creating DynamicTableRegions
  • Using DynamicTableRegion as column in a DynamicTable
  • Interacting with tables that contain DynamicTableRegions
  • Constructing table hierarchies and the convenience functions available in hdmf.common.hierarichaltable

This issue has been identified as part of PR #645 and the functionality is being used in PyNWB as part of the ICEphys metadata tables defined in NeurodataWithoutBorders/pynwb#1383

Checklist

  • Have you ensured the feature or change was not already reported ?
  • Have you included a brief and descriptive title?
  • Have you included a clear description of the problem you are trying to solve?
  • Have you included a minimal code snippet that reproduces the issue you are encountering?
  • Have you checked our Contributing document?
@oruebel oruebel added category: enhancement improvements of code or code behavior help wanted: deep dive request for community contributions that will involve many parts of the code base help wanted: good first issue request for community contributions that are good for new contributors topic: docs Issues related to documentation labels Jul 19, 2021
@oruebel oruebel added this to the Future milestone Jul 19, 2021
@oruebel oruebel mentioned this issue Jul 19, 2021
Merged
30 tasks
oruebel added a commit that referenced this issue Jul 21, 2021
@oruebel
Add functionality for linked DynamicTables (#645)
c5f1bf7 
Smplify interacting with DynamicTables that reference other tables via DynamicTableRegion, creating a collection of linked tables. In the ICEphys case this is a "simple" linear hierarchy of tables, but in principle a table may contain any number of DynamicTableRegion columns. This PR adds several functions to simplify introspection of linked DynamicTables and conversion to pandas DataFrames. 

- [X] Fix #646 by adding ``AlignedDynamicTable.get``
- [X] Fix #651 by updating ``AlignedDynamicTable.get`` to support slicing with  ``[int, (str, str)]``, ``[int, str, str]``, and ``[int, str]`` to select a single cell or row of a category table, repectively
- [X] Add ``AlignedDynamicTable.get_colnames(...)`` functions to allow us to keep compliance of the ``colnames`` property with ``DynamicTable`` while providing an easy way to get the full list of column names.
- [X] Set name of DataFrame in ``DynamicTable.to_dataframe()`` and ``DynamicTable.get`` 
- [X] Add helper functions to ``DynamicTable`` to deal with foreign columns:
    - [X] ``DynamicTable.get_foreign_columns`` to identify if the table contains ``DynamicTableRegion`` columns
    - [X] ``DynamicTable.has_foreign_columns`` to  identify which columns are``DynamicTableRegion`` columns 
    - [X] ``DynamicTable.get_linked_tables`` to retrieve all tables linked to either directly or indirectly from
      the current table via ``DynamicTableRegion``
- [x] Implement the same helper functions also for ``AlignedDynamicTable``
    - [x] ``DynamicTable.get_foreign_columns`` to identify if the table contains ``DynamicTableRegion`` columns
    - [X] ``DynamicTable.has_foreign_columns`` to  identify which columns are``DynamicTableRegion`` columns 
    - [x] ``DynamicTable.get_linked_tables`` to retrieve all tables linked to either directly or indirectly from
      the current table via ``DynamicTableRegion``
- [X] Add new module ``hdmf.common.hierarchicaltable`` with helper functions to facilitate conversion of linked tables to a single Pandas dataframe. 
     - [X] ``to_hierarchical_dataframe`` to merge linked tables into a single consolidated pandas DataFrame.
     - [X] ``drop_id_columns`` to remove "id" columns from a DataFrame.
     - [X] ``flatten_column_index`` to replace a ``pandas.MultiIndex`` with a regular ``pandas.Index``
- [x]  Add test for DyanmicTableRegion pointing to AlignedDynamicTable to check that the all columns are used
- [x] Add tests for hierarchicaltable.py for 
    - [X] to_hierarchical_dataframe
    - [x] drop_id_columns
    - [x]  flatten_column_index functions
- [X] File issue tickets for open TODO items for future PRs 
    - [X] ``to_hierarchical_dataframe`` should be updated to support resolution of more than one DynamicTableRegion column. See #649
    - [x] Add tutorial for DynamicTableRegion and how to use for linking to tables and for creating linked tables. See #648
@oruebel oruebel removed the help wanted: deep dive request for community contributions that will involve many parts of the code base label Sep 11, 2021
@mavaylon1 mavaylon1 self-assigned this Dec 4, 2021
@mavaylon1
Copy link
Contributor

mavaylon1 commented Mar 13, 2023 •
edited
Loading

This exists in the tutorials. https://hdmf.readthedocs.io/en/dev/tutorials/plot_dynamictable_howto.html#sphx-glr-tutorials-plot-dynamictable-howto-py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mavaylon1 mavaylon1

Labels
category: enhancement improvements of code or code behavior help wanted: good first issue request for community contributions that are good for new contributors topic: docs Issues related to documentation
Projects
None yet
Milestone
Future
Development

No branches or pull requests
2 participants
@oruebel @mavaylon1