DM-48282: Refactor obscore to allow subclasses

doc/changes/DM-48282.feature.rst

@@ -0,0 +1,5 @@
+Modified the Obscore ``RecordFactory`` to support per-universe subclass discovery using entry points.
+
+* Added ``RecordFactory.get_record_type_from_universe`` to obtain the correct factory class.
+* Renamed ``ExposureRegionFactory`` to ``DerivedRegionFactory`` to make it clearer that this class is not solely used for exposures but the usage can change with universe.
+* Added ``RecordFactory.region_dimension`` to return the dimension that would be needed to obtain a region for this universe.

doc/lsst.daf.butler/dev/entryPoints.rst

@@ -0,0 +1,34 @@
+.. _lsst.daf.butler-dev_entry_points:
+
+.. py:currentmodule:: lsst.daf.butler
+
+Support Entry Points
+--------------------
+
+Some functionality is enabled by utilizing the Python entry points system.
+Entry points are managed by the ``project.entry_points`` section of the ``pyproject.toml`` file.
+
+
+Command Line Subcommands
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+The entry points that support command-line subcommands are documented in :ref:`daf_butler_cli-entry-points`.
+
+ObsCore RecordFactory
+^^^^^^^^^^^^^^^^^^^^^
+
+`lsst.daf.butler.registry.obscore.RecordFactory` is the class that generates ObsCore records from Butler datasets.
+This conversion generally requires decisions to be made that depend on the dimension universe for the butler.
+The default implementation only works with the ``daf_butler`` dimension universe namespace.
+Alternative implementations can be registered using the ``butler.obscore_factory`` entry point group.
+The label associated with the entry point should correspond to the universe namespace.
+
+For example, a hypothetical entry point for the default namespace could be written in the ``pyproject.toml`` file as:
+
+.. code-block:: TOML
+
+    [project.entry-points.'butler.obscore_factory']
+    daf_butler = "lsst.daf.butler.registry.obscore._records.DafButlerRecordFactory"
+
+The function listed for the entry point should return the Python type that should be used to generate records.
+It is required to be a subclass of `lsst.daf.butler.registry.obscore.RecordFactory`.

doc/lsst.daf.butler/index.rst

@@ -85,6 +85,7 @@ You can find Jira issues for this module under the `daf_butler <https://jira.lss
    :maxdepth: 1
 
    dev/dataCoordinate.rst
+   dev/entryPoints.rst
 
 Butler Command Line Interface Development
 -----------------------------------------

doc/lsst.daf.butler/writing-subcommands.rst

@@ -368,7 +368,10 @@ Adding Butler Subcommands
 Packages can add subcommands to the ``butler`` command using a plugin system. This section describes how to do that.
 To use the plugin system you should also read and understand the sections above about `the butler command`_.
 Then, write your subcommands and arrange them as described below in `Package Layout`_.
-Finally, declare them as ``butler`` command plugins as described in `Manifest`_.
+Finally, configure the package's entry points to make them known to the butler infrastructure, as described in `Entry Points`_.
+
+Older versions of the Butler plugin system supported a YAML resource file and environment variable to enable plugin discovery but this approach is now deprecated and will be removed in the future (this is documented in `Manifest`_).
+
 
 Package Layout
 --------------
@@ -391,11 +394,48 @@ The following conventions are recommended but not required:
    │    ├── arguments.py
    │    ├── options.py
    │    └── sharedOptions.py
-   ├── resources.yaml
    └── utils.yaml
 
-Manifest
---------
+Entry Points
+------------
+
+The butler subcommands use an entry point group named ``butler.cli``.
+The entry points should be declared in the package's ``pyproject.toml`` file in the standard manner.
+For example, in ``obs_base`` it looks like this:
+
+.. code-block:: toml
+
+    [project.entry-points.'butler.cli']
+    obs_base = "lsst.obs.base.cli:get_cli_subcommands"
+
+The name and location of the function does not matter, but by convention it is placed within the ``cli`` hierarchy of the package.
+The function mentioned should return all the registered click commands.
+An example implementation is:
+
+.. code-block:: python
+
+    import click
+
+    from . import cmd
+
+
+    def get_cli_subcommands() -> list[click.Command]:
+        """Return the location of the CLI command plugin definitions.
+
+        Returns
+        -------
+        commands : `list` [ `click.Command` ]
+            The command-line subcommands provided by this package.
+        """
+        return [getattr(cmd, c) for c in cmd.__all__]
+
+Which should be sufficient for most implementations where the commands are already stored in ``__all__``.
+
+Manifest (Deprecated)
+---------------------
+
+This section refers to the old approach to registering plugins.
+New code should not register subcommands this way.
 
 The ``butler`` command finds plugin commands by way of a resource manifest published in an environment variable.
 By convention it is usually in the ``cli`` folder and named ``resources.yaml``.
@@ -421,7 +461,7 @@ Publish the resource manifest in an environment variable: in the package's ``ups
 command to prepend ``DAF_BUTLER_PLUGINS`` with the location of the resource manifest. Make sure to use the
 environment variable for the location of the package.
 
-The settings for ``obs_base`` are like this:
+The settings for ``obs_base`` were like this:
 
 .. code-block:: text
 

python/lsst/daf/butler/registry/obscore/_manager.py

@@ -47,7 +47,7 @@
 from ..interfaces import ObsCoreTableManager, VersionTuple
 from ..queries import SqlQueryContext
 from ._config import ConfigCollectionType, ObsCoreManagerConfig
-from ._records import ExposureRegionFactory, Record, RecordFactory
+from ._records import DerivedRegionFactory, Record, RecordFactory
 from ._schema import ObsCoreSchema
 from ._spatial import RegionTypeError, RegionTypeWarning, SpatialObsCorePlugin
 
@@ -63,7 +63,7 @@
 _VERSION = VersionTuple(0, 0, 1)
 
 
-class _ExposureRegionFactory(ExposureRegionFactory):
+class _ExposureRegionFactory(DerivedRegionFactory):
     """Find exposure region from a matching visit dimensions records.
 
     Parameters
@@ -79,7 +79,7 @@ def __init__(self, dimensions: DimensionRecordStorageManager, context: SqlQueryC
         self.exposure_detector_dimensions = self.universe.conform(["exposure", "detector"])
         self._context = context
 
-    def exposure_region(self, dataId: DataCoordinate) -> Region | None:
+    def derived_region(self, dataId: DataCoordinate) -> Region | None:
         # Docstring is inherited from a base class.
         context = self._context
         # Make a relation that starts with visit_definition (mapping between
@@ -167,7 +167,7 @@ def __init__(
             dimensions,
             SqlQueryContext(self.db, column_type_info),
         )
-        self.record_factory = RecordFactory(
+        self.record_factory = RecordFactory.get_record_type_from_universe(universe)(
             config, schema, universe, spatial_plugins, exposure_region_factory
         )
         self.tagged_collection: str | None = None