Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes IPL-1 downloads with sdss_access #36

Merged
merged 4 commits into from
Nov 30, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,10 @@ Change Log

This document records the main changes to the sdss_access code.

2.0.5 (2022-11-30)
------------------
- Bug fix for downloading IPL-1 products with sdss_access

2.0.4 (2022-11-30)
------------------
- Fixing broken tests with MWM paths
Expand Down
18 changes: 18 additions & 0 deletions docs/sphinx/path_defs.rst
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,15 @@ SDSSWORK
:prog: sdsswork
:templates:

.. _dr18:

DR18
----

.. datamodel:: sdss_access.path.path:Path
:prog: DR18
:templates:

.. _dr17:

DR17
Expand Down Expand Up @@ -195,4 +204,13 @@ SDSS-V

.. datamodel:: sdss_access.path.path:Path
:prog: sdss5
:templates:

.. _ipl1:

IPL1
----

.. datamodel:: sdss_access.path.path:Path
:prog: IPL1
:templates:
12 changes: 12 additions & 0 deletions docs/sphinx/path_evolution.rst
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,10 @@ releases (DRs) and internal SDSS data releases, e.g. MaNGA Product Launches (MPL
Public Data Releases
--------------------

.. changelog:: sdss_access.path.changelog:compute_changelog
:prog: changes
:drs: dr18, dr17

.. changelog:: sdss_access.path.changelog:compute_changelog
:prog: changes
:drs: dr17, dr16
Expand Down Expand Up @@ -49,6 +53,14 @@ Public Data Releases
Internal Data Releases
----------------------

Internal Product Launches (IPLs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

.. changelog:: sdss_access.path.changelog:compute_changelog
:prog: changes
:drs: ipl1, ipl1


MaNGA Product Launches (MPLs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Expand Down
7 changes: 5 additions & 2 deletions docs/sphinx/paths.rst
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,8 @@ Current Working Paths

Public Data Release Paths

* :ref:`DR17 <dr17>` - paths defined for public data release 17
* :ref:`DR18 <dr18>` - paths defined for public data release 18
* :ref:`DR17 <dr17>`
* :ref:`DR16 <dr16>`
* :ref:`DR15 <dr15>`
* :ref:`DR14 <dr14>`
Expand All @@ -28,8 +29,10 @@ Public Data Release Paths

Internal SDSS Release Paths

* :ref:`IPL-1 <ipl1>` - Internal Product Launch 1

* :ref:`MPL-11<mpl11>` - MaNGA Product Launch 11
* :ref:`MPL-10<mpl10>`
* :ref:`MPL-10<mpl10>`
* :ref:`MPL-9<mpl9>`
* :ref:`MPL-8<mpl8>`
* :ref:`MPL-7<mpl7>`
Expand Down
5 changes: 3 additions & 2 deletions python/sdss_access/path/path.py
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,7 @@ class BasePath(object):

_netloc = {"dtn": "dtn.sdss.org", "sdss": "data.sdss.org", "sdss5": "data.sdss5.org",
"mirror": "data.mirror.sdss.org", "svn": "svn.sdss.org"}
_s5cfgs = ['sdss5', 'ipl1'] # list of collab-only SDSS-V releases/configs

def __init__(self, release=None, public=False, mirror=False, verbose=False,
force_modules=None, preserve_envvars=None):
Expand Down Expand Up @@ -832,7 +833,7 @@ def get_netloc(self, netloc=None, sdss=None, sdss5=None, dtn=None, svn=None, mir
elif svn:
return '{0}{1}'.format(self._netloc["svn"], "/public" if self.public else '')
else:
return self._netloc["sdss5"] if self.release == "sdss5" else self._netloc["sdss"]
return self._netloc["sdss5"] if self.release in self._s5cfgs else self._netloc["sdss"]

def set_netloc(self, netloc=None, sdss=None, sdss5=None, dtn=None, svn=None, mirror=None):
''' Set a url domain location
Expand Down Expand Up @@ -878,7 +879,7 @@ def get_remote_base(self, scheme="https", svn=None):
if self.public or scheme == "https":
remote_base = "{scheme}://{netloc}".format(scheme=scheme, netloc=netloc)
else:
user = "sdss5" if self.release == "sdss5" else "sdss"
user = "sdss5" if self.release in self._s5cfgs else "sdss"
remote_base = "{scheme}://{user}@{netloc}".format(scheme=scheme, user=user, netloc=netloc)
return remote_base

Expand Down
2 changes: 1 addition & 1 deletion python/sdss_access/sync/baseaccess.py
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ def remote(self, username=None, password=None, inquire=None):
use_dtn = self.remote_scheme == 'rsync'
# simplifies things to have a single sdss (or sdss5) machine in
# .netrc for SDSS-IV (or SDSS-V, respectively).
sdss5 = ( self.release == 'sdss5' )
sdss5 = ( self.release in self._s5cfgs )
self.set_netloc(sdss=not sdss5, sdss5=sdss5)
self.set_auth(username=username, password=password, inquire=inquire)
if use_dtn:
Expand Down
25 changes: 25 additions & 0 deletions tests/sync/test_access.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,3 +37,28 @@ def test_access(monkey_posix):
assert Access.access_mode == mode
assert issubclass(Access, core)

@pytest.mark.parametrize('cfg, exp',
[('sdss5', 'data.sdss5.org'),
('ipl1', 'data.sdss5.org'),
('dr17', 'data.sdss.org'),
('sdsswork', 'data.sdss.org'),
('mpl9', 'data.sdss.org')],
ids=['sdss5', 'ipl1', 'dr17', 'sdsswork', 'mpl9'])
def test_netloc(cfg, exp):
a = RsyncAccess(release=cfg)
assert a.netloc == exp
assert a.remote_base == f'https://{exp}'

@pytest.mark.parametrize('cfg, exp',
[('sdss5', 'sdss5'),
('ipl1', 'sdss5'),
('dr17', ''),
('sdsswork', 'sdss'),
('mpl9', 'sdss')],
ids=['sdss5', 'ipl1', 'dr17', 'sdsswork', 'mpl9'])
def test_remote_base(cfg, exp):
a = RsyncAccess(release=cfg)
a.remote()
assert a.netloc == 'dtn.sdss.org'
exp = exp if cfg == 'dr17' else f'{exp}@'
assert a.remote_base == f'rsync://{exp}dtn.sdss.org'