Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merge main #1624

Merged
merged 41 commits into from
Aug 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
41 commits
Select commit Hold shift + click to select a range
bade3ab
Only pass known keys to Server dataclass
pmrv Aug 20, 2024
3b27693
Backwards compatible access to executable name during instantiate
pmrv Aug 20, 2024
90f61c8
Add backwards compatibility when restoring server/executable
pmrv Aug 20, 2024
702e292
Fix project mv
jan-janssen Aug 22, 2024
0f5293d
Merge pull request #1615 from pyiron/project_mv
jan-janssen Aug 22, 2024
509ebae
Fix windows paths
jan-janssen Aug 22, 2024
9299ea4
Merge pull request #1616 from pyiron/fix_archive
jan-janssen Aug 22, 2024
8bfaf29
add old tar files
samwaseda Aug 23, 2024
9fb5a05
save current state
samwaseda Aug 23, 2024
9dd4157
stash changes to see whether the tests still fail
samwaseda Aug 23, 2024
fa80cd4
restore changes
samwaseda Aug 23, 2024
69820de
separate class
samwaseda Aug 23, 2024
5b5223a
add loading
samwaseda Aug 23, 2024
16b4e18
change order
samwaseda Aug 23, 2024
e9abb0b
Get executable keys from dataclass fields
jan-janssen Aug 23, 2024
6416c3a
Merge remote-tracking branch 'origin/server' into add_pack_bc_tests
samwaseda Aug 23, 2024
0c0c1eb
also check inputs
samwaseda Aug 23, 2024
6dad8c3
Merge pull request #1619 from pyiron/clean_up_executable
jan-janssen Aug 23, 2024
f6587f5
align things that I didn't change
samwaseda Aug 23, 2024
d72487f
Merge remote-tracking branch 'origin' into add_pack_bc_tests
samwaseda Aug 23, 2024
ee13e20
Merge remote-tracking branch 'origin' into server
samwaseda Aug 23, 2024
b5c3f6d
Merge remote-tracking branch 'origin/server' into add_pack_bc_tests
samwaseda Aug 23, 2024
7d37581
Make a backward change made by Jan temporarily, to see whether the Wi…
samwaseda Aug 23, 2024
5989c18
I don't understand this one change
samwaseda Aug 23, 2024
672d26e
add output check
samwaseda Aug 23, 2024
29fa89c
replace . by os.getcwd
samwaseda Aug 23, 2024
fe25269
Revert changes to DataContainer storage format
pmrv Aug 23, 2024
103302e
Revert "Make a backward change made by Jan temporarily, to see whethe…
samwaseda Aug 23, 2024
4394b7b
Merge pull request #1611 from pyiron/server
pmrv Aug 23, 2024
7ad654a
Merge pull request #1617 from pyiron/add_pack_bc_tests
pmrv Aug 23, 2024
7f694c6
Handle list in to_dict
pmrv Aug 23, 2024
dbd263a
Bump executorlib from 0.0.1 to 0.0.2
dependabot[bot] Aug 26, 2024
697ebd2
[dependabot skip] Update environment
github-actions[bot] Aug 26, 2024
2ca2d67
Merge pull request #1622 from pyiron/dependabot/pip/executorlib-0.0.2
jan-janssen Aug 26, 2024
d10d227
Add split_children_dict to undo join_children_dict
pmrv Aug 26, 2024
a017549
Save key order in DataContainer.to_dict
pmrv Aug 26, 2024
eadd9b0
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Aug 26, 2024
3fce519
Merge pull request #1623 from pyiron/main
jan-janssen Aug 26, 2024
c389069
Update unittests_old.yml
jan-janssen Aug 26, 2024
3ccfa33
Update pyproject.toml
jan-janssen Aug 26, 2024
959ad9d
Merge pull request #1620 from pyiron/server
jan-janssen Aug 26, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .ci_support/environment-docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ dependencies:
- psutil =6.0.0
- pyfileindex =0.0.27
- pyiron_snippets =0.1.4
- executorlib =0.0.1
- executorlib =0.0.2
- pysqa =0.1.21
- pytables =3.10.1
- sqlalchemy =2.0.32
Expand Down
2 changes: 1 addition & 1 deletion .ci_support/environment-mini.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ dependencies:
- psutil =6.0.0
- pyfileindex =0.0.27
- pyiron_snippets =0.1.3
- executorlib =0.0.1
- executorlib =0.0.2
- pysqa =0.1.21
- pytables =3.10.1
- sqlalchemy =2.0.32
Expand Down
2 changes: 1 addition & 1 deletion .ci_support/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ dependencies:
- psutil =6.0.0
- pyfileindex =0.0.27
- pyiron_snippets =0.1.4
- executorlib =0.0.1
- executorlib =0.0.2
- pysqa =0.1.21
- pytables =3.10.1
- sqlalchemy =2.0.32
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unittests_old.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ jobs:
- name: Setup Mambaforge
uses: conda-incubator/setup-miniconda@v3
with:
python-version: '3.9'
python-version: '3.10'
miniforge-variant: Mambaforge
channels: conda-forge
channel-priority: strict
Expand Down
2 changes: 1 addition & 1 deletion binder/environment.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ dependencies:
- psutil =6.0.0
- pyfileindex =0.0.27
- pyiron_snippets =0.1.4
- executorlib =0.0.1
- executorlib =0.0.2
- pysqa =0.1.21
- pytables =3.10.1
- sqlalchemy =2.0.32
Expand Down
27 changes: 25 additions & 2 deletions pyiron_base/interfaces/has_dict.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@
"""

from abc import ABC, abstractmethod
from collections import defaultdict
from typing import Any

from pyiron_base.interfaces.has_hdf import HasHDF
Expand Down Expand Up @@ -54,7 +55,7 @@ def create_from_dict(obj_dict):
type_field = obj_dict["TYPE"]
module_path, class_name = _extract_module_class_name(type_field)
class_object = _import_class(module_path, class_name)
version = obj_dict.get("VERSION", None)
version = obj_dict.get("DICT_VERSION", None)
obj = class_object.instantiate(obj_dict, version)
obj.from_dict(obj_dict, version)
return obj
Expand Down Expand Up @@ -122,6 +123,9 @@ def load(inner_dict):
return {k: load(v) for k, v in inner_dict.items()}
return create_from_dict(inner_dict)

obj_dict = self._split_children_dict(obj_dict)
if version is None:
version = obj_dict.get("DICT_VERSION", None)
self._from_dict({k: load(v) for k, v in obj_dict.items()}, version)

@abstractmethod
Expand Down Expand Up @@ -208,9 +212,28 @@ def _join_children_dict(children: dict[str, dict[str, Any]]) -> dict[str, Any]:
return {
"/".join((k1, k2)): v2
for k1, v1 in children.items()
for k2, v2 in v2.items()
for k2, v2 in v1.items()
}

@staticmethod
def _split_children_dict(
obj_dict: dict[str, Any],
) -> dict[str, Any | dict[str, Any]]:
"""
Undoes _join_children_dict.
"""
subs = defaultdict(dict)
plain = {}
for k, v in obj_dict.items():
if "/" not in k:
plain[k] = v
continue
root, k = k.split("/", maxsplit=1)
subs[root][k] = v
# using update keeps type stability, i.e. we always return a plain dict
plain.update(subs)
return plain


class HasHDFfromDict(HasHDF, HasDict):
"""
Expand Down
17 changes: 7 additions & 10 deletions pyiron_base/jobs/job/extension/executable.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
# Distributed under the terms of "New BSD License", see the LICENSE file.

import os
from dataclasses import asdict
from dataclasses import asdict, fields

from pyiron_snippets.resources import ExecutableResolver

Expand Down Expand Up @@ -208,20 +208,17 @@ def executable_path(self, new_path):

@classmethod
def instantiate(cls, obj_dict: dict, version: str = None) -> "Self":
return cls(codename=obj_dict["name"])
try:
codename = obj_dict["name"]
except KeyError:
codename = obj_dict["executable"]["name"]
return cls(codename=codename)

def _to_dict(self):
return asdict(self.storage)

def _from_dict(self, obj_dict, version=None):
data_container_keys = [
"version",
"name",
"operation_system_nt",
"executable",
"mpi",
"accepted_return_codes",
]
data_container_keys = tuple(f.name for f in fields(ExecutableDataClass))
executable_class_dict = {}
# Backwards compatibility; dict state used to be nested one level deeper
if "executable" in obj_dict.keys() and isinstance(obj_dict["executable"], dict):
Expand Down
11 changes: 6 additions & 5 deletions pyiron_base/jobs/job/extension/server/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import numbers
import socket
from concurrent.futures import Executor, Future
from dataclasses import asdict
from dataclasses import asdict, fields
from typing import Union

from pyiron_snippets.deprecate import deprecate
Expand Down Expand Up @@ -564,7 +564,6 @@ def view_queues():
def _to_dict(self):
self._data.run_mode = self._run_mode.mode
return asdict(self._data)
return server_dict

def _from_dict(self, obj_dict, version=None):
# backwards compatibility
Expand All @@ -578,9 +577,11 @@ def _from_dict(self, obj_dict, version=None):
if "additional_arguments" not in obj_dict.keys():
obj_dict["additional_arguments"] = {}

# Reload dataclass
for key in ["NAME", "TYPE", "OBJECT", "VERSION", "DICT_VERSION"]:
if key in obj_dict.keys():
# Reload dataclass and discard unknown keys
server_fields = tuple(f.name for f in fields(ServerDataClass))
# force tuple otherwise dict complains about changing size
for key in tuple(obj_dict):
if key not in server_fields:
del obj_dict[key]
self._data = ServerDataClass(**obj_dict)
self._run_mode = Runmode(mode=self._data.run_mode)
Expand Down
15 changes: 13 additions & 2 deletions pyiron_base/jobs/job/generic.py
Original file line number Diff line number Diff line change
Expand Up @@ -1178,9 +1178,20 @@ def _from_dict(self, obj_dict, version=None):
self._type_from_dict(type_dict=obj_dict)
if "import_directory" in obj_dict.keys():
self._import_directory = obj_dict["import_directory"]
self._server = obj_dict["server"]
# Backwards compatibility: Previously server and executable were stored
# as plain dicts, but now they are dicts with additional info so that
# HasDict can load them automatically.
# We need to check whether that was possible with the instance check
# below and if not, call from_dict ourselves.
if isinstance(server := obj_dict["server"], Server):
self._server = server
else:
self._server.from_dict(server)
if "executable" in obj_dict.keys() and obj_dict["executable"] is not None:
self._executable = obj_dict["executable"]
if isinstance(executable := obj_dict["executable"], Executable):
self._executable = executable
else:
self.executable.from_dict(executable)
input_dict = obj_dict["input"]
if "generic_dict" in input_dict.keys():
generic_dict = input_dict["generic_dict"]
Expand Down
4 changes: 2 additions & 2 deletions pyiron_base/jobs/job/util.py
Original file line number Diff line number Diff line change
Expand Up @@ -95,12 +95,12 @@ def _get_project_for_copy(job, project, new_job_name):
):
file_project = project.project
hdf5_project = project.project_hdf5.open(new_job_name)
elif isinstance(project, job.project.__class__):
elif isinstance(job.project, project.__class__):
file_project = project
hdf5_project = job.project_hdf5.__class__(
project=project, file_name=new_job_name, h5_path="/" + new_job_name
)
elif isinstance(project, job.project_hdf5.__class__):
elif isinstance(job.project_hdf5, project.__class__):
file_project = project.project
hdf5_project = project.open(new_job_name)
elif project is None:
Expand Down
9 changes: 5 additions & 4 deletions pyiron_base/project/archiving/import_archive.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import os
import posixpath
import tarfile
import tempfile
from shutil import copytree
Expand Down Expand Up @@ -57,8 +58,8 @@ def import_jobs(project_instance, archive_directory):

pr_import = project_instance.open(os.curdir)
df["project"] = [
os.path.normpath(
os.path.join(pr_import.project_path, os.path.relpath(p, common_path))
posixpath.normpath(
posixpath.join(pr_import.project_path, posixpath.relpath(p, common_path))
)
+ "/"
for p in df["project"].values
Expand Down Expand Up @@ -105,8 +106,8 @@ def transfer_files(origin_path: str, project_path: str):
str: Common path.
"""
df = get_dataframe(origin_path=origin_path)
common_path = os.path.commonpath(list(df["project"]))
copytree(os.path.join(origin_path, common_path), project_path, dirs_exist_ok=True)
common_path = posixpath.commonpath(list(df["project"]))
copytree(posixpath.join(origin_path, common_path), project_path, dirs_exist_ok=True)
return df, common_path


Expand Down
23 changes: 20 additions & 3 deletions pyiron_base/storage/datacontainer.py
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@
"HDF_VERSION",
"DICT_VERSION",
"READ_ONLY",
"KEY_ORDER",
]


Expand Down Expand Up @@ -827,6 +828,7 @@ def _on_unlock(self):


class DataContainer(DataContainerBase, HasHDF, HasDict):
__dict_version__ = "0.2.0"
__doc__ = f"""{DataContainerBase.__doc__}

If instantiated with the argument `lazy=True`, data read from HDF5 later via :method:`.from_hdf` are not actually
Expand Down Expand Up @@ -1027,13 +1029,28 @@ def to(v):
return data

def _to_dict(self):
return {"data": self.to_builtin(), "READ_ONLY": self.read_only}
# stringify keys in case we are acting like a list
data = {str(k): v for k, v in dict(self).items()}
order = list(data)
data["READ_ONLY"] = self.read_only
data["KEY_ORDER"] = order
return data

def _from_dict(self, obj_dict, version=None):
if version == "0.2.0":
order = obj_dict.pop("KEY_ORDER")
else:
order = None
self.read_only = obj_dict.pop("READ_ONLY", False)
for key in _internal_hdf_nodes:
obj_dict.pop(key, None)
with self.unlocked():
self.clear()
self.update(obj_dict["data"], wrap=True)
self.read_only = obj_dict.get("READ_ONLY", False)
if order is not None:
for key in order:
self[key] = obj_dict[key]
else:
self.update(obj_dict)


HDFStub.register(DataContainer, lambda h, g: h[g].to_object(lazy=True))
3 changes: 1 addition & 2 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -18,14 +18,13 @@ classifiers = [
"License :: OSI Approved :: BSD License",
"Intended Audience :: Science/Research",
"Operating System :: OS Independent",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
]
dependencies = [
"cloudpickle==3.0.0",
"executorlib==0.0.1",
"executorlib==0.0.2",
"h5io_browser==0.1.0",
"h5py==3.11.0",
"numpy==2.1.0",
Expand Down
2 changes: 2 additions & 0 deletions tests/static/pack/export.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
,id,status,chemicalformula,job,subjob,project,timestart,timestop,totalcputime,computer,hamilton,hamversion,parentid,masterid
0,0,finished,,toy,/toy,test_pack/my_project,2024-08-22 16:10:26.556984,,,pyiron@7720454e9ac5#1,ToyJob,0.4,,
Binary file added tests/static/pack/test_pack.tar.gz
Binary file not shown.
23 changes: 22 additions & 1 deletion tests/unit/archiving/test_import.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
from pyiron_base import Project
from pandas._testing import assert_frame_equal
from filecmp import dircmp
from shutil import rmtree
from shutil import rmtree, copytree
import tarfile
from pyiron_base._tests import PyironTestCase, ToyJob

Expand Down Expand Up @@ -181,5 +181,26 @@ def test_backwards_compatibility(self):
self.imp_pr.unpack(origin_path=self.arch_dir_comp, csv_file_name="ahoy.csv")


class TestUnpackingBackwardsCompatibility(PyironTestCase):
def test_import_old_tar(self):
copytree(
os.path.join(
os.path.dirname(os.path.abspath(__file__)),
"../../static/pack",
),
os.getcwd(),
dirs_exist_ok=True,
)
pr = Project("old_tar")
pr.unpack(origin_path="test_pack.tar.gz")
job = pr.load("toy")
self.assertEqual(job.job_name, "toy")
self.assertEqual(job.input.data_in, 100)
self.assertEqual(job.output.data_out, 101)
pr.remove(enable=True, enforce=True)
os.remove("test_pack.tar.gz")
os.remove("export.csv")


if __name__ == "__main__":
unittest.main()
Loading