Copy files in copy_to

[pmrv]

GenericJob could previously copy files from the working directory, but didn't by default. I exposed copy_files as an argument and change the default to True, same as in JobCore.

Fixes #348.

[sravankumarkb]

Thanks @pmrv for trying to fix this issue.

I tried to copy a project with some DFT calcs into another project using this branch. Below is the error I received.

Along with the below error, the ase_to_pyiron function is giving me some error when I switched to this branch. I am not sure if it has to do with the changes @pmrv made, but it could be because of some other pull request related to pyiron_atomistics. For now, I think I will stick to my original branch rather than updating my pyiron and breaking my notebooks.

Switching to this branch:

git fetch origin pull/349/head:copy-files
git checkout copy-files

Code in notebook:

pr['2_by_2'].create_group('rlx_bc_sym_slabs_vacancies')
pr['2_by_2']['sym_slabs_vacancies'].copy_to(pr['2_by_2']['rlx_bc_sym_slabs_vacancies'])

Error:

/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/h5io/_h5io.py:610: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array(array_restore)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-abe62d6dc4b1> in <module>
----> 1 pr['2_by_2']['sym_slabs_vacancies'].copy_to(pr['2_by_2']['rlx_bc_sym_slabs_vacancies'])

~/software/pyiron/pyiron/base/project/generic.py in copy_to(self, destination)
    166                     sub_project = self.open(sub_project_name)
    167                     destination_sub_project = destination.open(sub_project_name)
--> 168                     sub_project.copy_to(destination_sub_project)
    169             for job_id in self.get_job_ids(recursive=False):
    170                 ham = self.load(job_id)

~/software/pyiron/pyiron/base/project/generic.py in copy_to(self, destination)
    166                     sub_project = self.open(sub_project_name)
    167                     destination_sub_project = destination.open(sub_project_name)
--> 168                     sub_project.copy_to(destination_sub_project)
    169             for job_id in self.get_job_ids(recursive=False):
    170                 ham = self.load(job_id)

~/software/pyiron/pyiron/base/project/generic.py in copy_to(self, destination)
    168                     sub_project.copy_to(destination_sub_project)
    169             for job_id in self.get_job_ids(recursive=False):
--> 170                 ham = self.load(job_id)
    171                 ham.copy_to(destination)
    172             for file in self.list_files():

~/software/pyiron/pyiron/base/project/generic.py in load(self, job_specifier, convert_to_object)
    578             s.logger.warn('Job \'%s\' does not exist and cannot be loaded', job_specifier)
    579             return None
--> 580         return self.load_from_jobpath(job_id=job_id, convert_to_object=convert_to_object)
    581 
    582     def load_from_jobpath(self, job_id=None, db_entry=None, convert_to_object=True):

~/software/pyiron/pyiron/project.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    218         """
    219         job = super(Project, self).load_from_jobpath(job_id=job_id, db_entry=db_entry,
--> 220                                                      convert_to_object=convert_to_object)
    221         job.project_hdf5._project = Project(path=job.project_hdf5.file_path)
    222         return job

~/software/pyiron/pyiron/base/project/generic.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    597         if job_id:
    598             job = jobpath(db=self.db, job_id=job_id, user=self.user)
--> 599             job = job.load_object(convert_to_object=convert_to_object, project=job.project_hdf5.copy())
    600             job._job_id = job_id
    601             if convert_to_object:

~/software/pyiron/pyiron/base/job/core.py in load_object(self, convert_to_object, project)
    503             with project.open('..') as job_dir:
    504                 job_dir._mode = 'a'
--> 505                 return self.to_object(project=job_dir, job_name=self._name)
    506         return self
    507 

~/software/pyiron/pyiron/base/job/core.py in to_object(self, object_type, **qwargs)
    444             GenericJob: pyiron object
    445         """
--> 446         return self.project_hdf5.to_object(object_type, **qwargs)
    447 
    448     def get(self, name):

~/software/pyiron/pyiron/base/generic/hdfio.py in to_object(self, object_type, **qwargs)
   1188         if obj_type != str(type(new_obj)):  # Backwards compatibility
   1189             self["TYPE"] = str(type(new_obj))
-> 1190         new_obj.from_hdf()
   1191         return new_obj
   1192 

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf, group_name)
    548         self.input.from_hdf(self._hdf5)
    549         if "output" in self.project_hdf5.list_groups() and "structure" in self["output"].list_groups():
--> 550             self._output_parser.from_hdf(self._hdf5)
    551 
    552     def reset_output(self):

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf)
   1495                 self.structure = Atoms()
   1496             self.structure.from_hdf(hdf5_output)
-> 1497             self.generic_output.from_hdf(hdf5_output)
   1498             try:
   1499                 if "electrostatic_potential" in hdf5_output.list_groups():

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf)
   1569                         self.dft_log_dict[node] = hdf_dft[node]
   1570                     if 'bands' in hdf_dft.list_groups():
-> 1571                         self.bands.from_hdf(hdf_dft, "bands")
   1572 
   1573 

~/software/pyiron/pyiron/dft/waves/electronic.py in from_hdf(self, hdf, group_name)
    479                     if "resolved_densities" in nodes:
    480                         self.resolved_densities = h_dos["resolved_densities"]
--> 481             self.generate_from_matrices()
    482 
    483     def to_hdf_old(self, hdf, group_name="electronic_structure"):

~/software/pyiron/pyiron/dft/waves/electronic.py in generate_from_matrices(self)
    544         for i in range(len(self.kpoint_list)):
    545             self.add_kpoint(self.kpoint_list[i], self.kpoint_weights[i])
--> 546             _, length = np.shape(self.eigenvalue_matrix)
    547             for j in range(length):
    548                 val = self.eigenvalue_matrix[i][j]

ValueError: too many values to unpack (expected 2)

[pmrv]

I'm a bit confused, because this error occurs loading the job before the copying it. Can you check that you can load all the jobs in original (source) project?

[sravankumarkb]

I'm a bit confused, because this error occurs loading the job before the copying it. Can you check that you can load all the jobs in original (source) project?

Thanks @pmrv for going through the error. Actually, yes, you are right. There is an issue with loading jobs with this branch.

When I try with the convert_to_object=False option, I do not get an error.

temp_job = pr['2_by_2']['sym_slabs_vacancies'].load('Mg_Ca_0_25_0', convert_to_object=False)

temp_job['output/generic/energy_tot'][-1]

-41.4157936

However, I get the same error as above when I try to load a job with the default convert_to_object=True option

temp_job = pr['2_by_2']['sym_slabs_vacancies'].load('Mg_Ca_0_25_0')

Error:

/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/h5io/_h5io.py:610: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array(array_restore)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-13-2d4a8ad507b3> in <module>
----> 1 temp_job = pr['2_by_2']['sym_slabs_vacancies'].load('Mg_Ca_0_25_0')

~/software/pyiron/pyiron/base/project/generic.py in load(self, job_specifier, convert_to_object)
    578             s.logger.warn('Job \'%s\' does not exist and cannot be loaded', job_specifier)
    579             return None
--> 580         return self.load_from_jobpath(job_id=job_id, convert_to_object=convert_to_object)
    581 
    582     def load_from_jobpath(self, job_id=None, db_entry=None, convert_to_object=True):

~/software/pyiron/pyiron/project.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    218         """
    219         job = super(Project, self).load_from_jobpath(job_id=job_id, db_entry=db_entry,
--> 220                                                      convert_to_object=convert_to_object)
    221         job.project_hdf5._project = Project(path=job.project_hdf5.file_path)
    222         return job

~/software/pyiron/pyiron/base/project/generic.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    597         if job_id:
    598             job = jobpath(db=self.db, job_id=job_id, user=self.user)
--> 599             job = job.load_object(convert_to_object=convert_to_object, project=job.project_hdf5.copy())
    600             job._job_id = job_id
    601             if convert_to_object:

~/software/pyiron/pyiron/base/job/core.py in load_object(self, convert_to_object, project)
    503             with project.open('..') as job_dir:
    504                 job_dir._mode = 'a'
--> 505                 return self.to_object(project=job_dir, job_name=self._name)
    506         return self
    507 

~/software/pyiron/pyiron/base/job/core.py in to_object(self, object_type, **qwargs)
    444             GenericJob: pyiron object
    445         """
--> 446         return self.project_hdf5.to_object(object_type, **qwargs)
    447 
    448     def get(self, name):

~/software/pyiron/pyiron/base/generic/hdfio.py in to_object(self, object_type, **qwargs)
   1188         if obj_type != str(type(new_obj)):  # Backwards compatibility
   1189             self["TYPE"] = str(type(new_obj))
-> 1190         new_obj.from_hdf()
   1191         return new_obj
   1192 

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf, group_name)
    548         self.input.from_hdf(self._hdf5)
    549         if "output" in self.project_hdf5.list_groups() and "structure" in self["output"].list_groups():
--> 550             self._output_parser.from_hdf(self._hdf5)
    551 
    552     def reset_output(self):

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf)
   1495                 self.structure = Atoms()
   1496             self.structure.from_hdf(hdf5_output)
-> 1497             self.generic_output.from_hdf(hdf5_output)
   1498             try:
   1499                 if "electrostatic_potential" in hdf5_output.list_groups():

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf)
   1569                         self.dft_log_dict[node] = hdf_dft[node]
   1570                     if 'bands' in hdf_dft.list_groups():
-> 1571                         self.bands.from_hdf(hdf_dft, "bands")
   1572 
   1573 

~/software/pyiron/pyiron/dft/waves/electronic.py in from_hdf(self, hdf, group_name)
    479                     if "resolved_densities" in nodes:
    480                         self.resolved_densities = h_dos["resolved_densities"]
--> 481             self.generate_from_matrices()
    482 
    483     def to_hdf_old(self, hdf, group_name="electronic_structure"):

~/software/pyiron/pyiron/dft/waves/electronic.py in generate_from_matrices(self)
    544         for i in range(len(self.kpoint_list)):
    545             self.add_kpoint(self.kpoint_list[i], self.kpoint_weights[i])
--> 546             _, length = np.shape(self.eigenvalue_matrix)
    547             for j in range(length):
    548                 val = self.eigenvalue_matrix[i][j]

ValueError: too many values to unpack (expected 2)

PS: With the previous branch (my master) I was using, I did not get this error when loading jobs.
With the previous branch (my master),

temp_job = pr['2_by_2']['sym_slabs_vacancies'].load('Mg_Ca_0_25_0')

I get this warning, but everything works fine.

/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/h5io/_h5io.py:610: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array(array_restore)

[niklassiemer]

@sravankumarkb how does your master compare to the master branch? Since this PR does not touch anything on loading of jobs and, thus, I assume the master branch will have the same issue to load your jobs.

[pmrv]

I'm a bit confused, because this error occurs loading the job before the copying it. Can you check that you can load all the jobs in original (source) project?

Thanks @pmrv for going through the error. Actually, yes, you are right. There is an issue with loading jobs with this branch.

Good, then we're onto something already! ;)

Actually, while going through the code earlier I've noticed that if you call copy_to on a job that you got via inspect it'll go through a different code path that by default should copy the files as well. So if you're in hurry and need it working you can do

j = pr.inspect(...)
j.copy_to(...)

[niklassiemer]

I am ok with exposing the copy_files option. I just wonder why the default was False and if we should change the default here?

[liamhuber]

@raynol-dsouza does this have any bearing on our trouble loading protocol jobs? It's only been a week but already the details are hazy to me 😉 IIRC part of our child creation process got back a valid job object but the wrong/old HDF location.

[pmrv]

I am ok with exposing the copy_files option. I just wonder why the default was False and if we should change the default here?

I just think default True is vastly more useful. The only downside I can see is that old notebooks end up copying more than needed (if they didn't rely on the being copied before). It feels minor to me and I'd rather inconvenience some older users than all future users.

[raynol-dsouza]

does this have any bearing on our trouble loading protocol jobs? It's only been a week but already the details are hazy to me

@liamhuber no, loading protocol jobs had to do with the sequence in which the hdf content was being saved/loaded.

IIRC part of our child creation process got back a valid job object but the wrong/old HDF location.

the wrong hdf location, yes. But I believe this might totally be a Protocols related issue, and not a copy_to one. I will have to look into it.

[pmrv]

@sravankumarkb Did you check the loading again with the latest master branch? If it still won't work, I'll have another look.

[sravankumarkb]

@sravankumarkb Did you check the loading again with the latest master branch? If it still won't work, I'll have another look.

Hi @pmrv
I tried, but I get the same error again with this pull request branch. It might not be because of the changes you made, but rather something else.

pr.create_group('junk_copy_to_check')
temp_job = pr['2_by_2']['sym_slabs_vacancies'].load('Mg_Ca_0_25_0')
temp_job.copy_to(pr['junk_copy_to_check'])

/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/h5io/_h5io.py:610: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray.
  return np.array(array_restore)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-5-ea06e8f7cfaf> in <module>
      1 pr.create_group('junk_copy_to_check')
----> 2 temp_job = pr['2_by_2']['sym_slabs_vacancies'].load('Mg_Ca_0_25_0')
      3 temp_job.copy_to(pr['junk_copy_to_check'])

~/software/pyiron/pyiron/base/project/generic.py in load(self, job_specifier, convert_to_object)
    578             s.logger.warn('Job \'%s\' does not exist and cannot be loaded', job_specifier)
    579             return None
--> 580         return self.load_from_jobpath(job_id=job_id, convert_to_object=convert_to_object)
    581 
    582     def load_from_jobpath(self, job_id=None, db_entry=None, convert_to_object=True):

~/software/pyiron/pyiron/project.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    218         """
    219         job = super(Project, self).load_from_jobpath(job_id=job_id, db_entry=db_entry,
--> 220                                                      convert_to_object=convert_to_object)
    221         job.project_hdf5._project = Project(path=job.project_hdf5.file_path)
    222         return job

~/software/pyiron/pyiron/base/project/generic.py in load_from_jobpath(self, job_id, db_entry, convert_to_object)
    597         if job_id:
    598             job = jobpath(db=self.db, job_id=job_id, user=self.user)
--> 599             job = job.load_object(convert_to_object=convert_to_object, project=job.project_hdf5.copy())
    600             job._job_id = job_id
    601             if convert_to_object:

~/software/pyiron/pyiron/base/job/core.py in load_object(self, convert_to_object, project)
    503             with project.open('..') as job_dir:
    504                 job_dir._mode = 'a'
--> 505                 return self.to_object(project=job_dir, job_name=self._name)
    506         return self
    507 

~/software/pyiron/pyiron/base/job/core.py in to_object(self, object_type, **qwargs)
    444             GenericJob: pyiron object
    445         """
--> 446         return self.project_hdf5.to_object(object_type, **qwargs)
    447 
    448     def get(self, name):

~/software/pyiron/pyiron/base/generic/hdfio.py in to_object(self, object_type, **qwargs)
   1188         if obj_type != str(type(new_obj)):  # Backwards compatibility
   1189             self["TYPE"] = str(type(new_obj))
-> 1190         new_obj.from_hdf()
   1191         return new_obj
   1192 

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf, group_name)
    548         self.input.from_hdf(self._hdf5)
    549         if "output" in self.project_hdf5.list_groups() and "structure" in self["output"].list_groups():
--> 550             self._output_parser.from_hdf(self._hdf5)
    551 
    552     def reset_output(self):

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf)
   1495                 self.structure = Atoms()
   1496             self.structure.from_hdf(hdf5_output)
-> 1497             self.generic_output.from_hdf(hdf5_output)
   1498             try:
   1499                 if "electrostatic_potential" in hdf5_output.list_groups():

~/software/pyiron/pyiron/vasp/base.py in from_hdf(self, hdf)
   1569                         self.dft_log_dict[node] = hdf_dft[node]
   1570                     if 'bands' in hdf_dft.list_groups():
-> 1571                         self.bands.from_hdf(hdf_dft, "bands")
   1572 
   1573 

~/software/pyiron/pyiron/dft/waves/electronic.py in from_hdf(self, hdf, group_name)
    479                     if "resolved_densities" in nodes:
    480                         self.resolved_densities = h_dos["resolved_densities"]
--> 481             self.generate_from_matrices()
    482 
    483     def to_hdf_old(self, hdf, group_name="electronic_structure"):

~/software/pyiron/pyiron/dft/waves/electronic.py in generate_from_matrices(self)
    544         for i in range(len(self.kpoint_list)):
    545             self.add_kpoint(self.kpoint_list[i], self.kpoint_weights[i])
--> 546             _, length = np.shape(self.eigenvalue_matrix)
    547             for j in range(length):
    548                 val = self.eigenvalue_matrix[i][j]

ValueError: too many values to unpack (expected 2)

I tried loading a pyiron table, but I get the following error.

temp_job_2 = pr.load('tot_DFT_data_table')
temp_job_2.copy_to(pr['junk_copy_to_check'])

Traceback (most recent call last):

  File "/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3441, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)

  File "<ipython-input-7-711530b1a74a>", line 1, in <module>
    temp_job_2 = pr.load('tot_DFT_data_table')

  File "/u/skanchar/software/pyiron/pyiron/base/project/generic.py", line 580, in load
    return self.load_from_jobpath(job_id=job_id, convert_to_object=convert_to_object)

  File "/u/skanchar/software/pyiron/pyiron/project.py", line 220, in load_from_jobpath
    convert_to_object=convert_to_object)

  File "/u/skanchar/software/pyiron/pyiron/base/project/generic.py", line 599, in load_from_jobpath
    job = job.load_object(convert_to_object=convert_to_object, project=job.project_hdf5.copy())

  File "/u/skanchar/software/pyiron/pyiron/base/job/core.py", line 505, in load_object
    return self.to_object(project=job_dir, job_name=self._name)

  File "/u/skanchar/software/pyiron/pyiron/base/job/core.py", line 446, in to_object
    return self.project_hdf5.to_object(object_type, **qwargs)

  File "/u/skanchar/software/pyiron/pyiron/base/generic/hdfio.py", line 1190, in to_object
    new_obj.from_hdf()

  File "/u/skanchar/software/pyiron/pyiron/table/datamining.py", line 494, in from_hdf
    self.pyiron_table.filter_function = get_function_from_string(hdf5_input['filter'])

  File "/u/skanchar/software/pyiron/pyiron/base/master/generic.py", line 538, in get_function_from_string
    exec(function_dedent_str)

  File "<string>", line 6
    CE5OfXEMTnRxDVJxDi4=
                        ^
SyntaxError: invalid syntax

When I tried loading the pyiron table with convert_to_object=False and copy it, I get the following error.

temp_job_2 = pr.load('tot_DFT_data_table', convert_to_object=False)
temp_job_2.copy_to(pr['junk_copy_to_check'])

---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-8-034e5e3b6586> in <module>
      1 temp_job_2 = pr.load('tot_DFT_data_table', convert_to_object=False)
----> 2 temp_job_2.copy_to(pr['junk_copy_to_check'])

~/software/pyiron/pyiron/base/job/core.py in copy_to(self, project, new_database_entry, copy_files)
    633         new_job_core._master_id = self._master_id
    634         new_job_core._status = self._status
--> 635         self.project_hdf5.copy_to(project)
    636         if self.job_id:
    637             if new_database_entry:

~/software/pyiron/pyiron/base/generic/hdfio.py in copy_to(self, destination, file_name, maintain_name)
    286         if self.file_exists:
    287             with h5py.File(self.file_name, mode="r", libver='latest', swmr=True) as f_source:
--> 288                 with h5py.File(file_name, libver='latest', swmr=True) as f_target:
    289                     if destination.h5_path[0] == '/':
    290                         dest_path = destination.h5_path[1:]

/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py in __init__(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, fs_strategy, fs_persist, fs_threshold, **kwds)
    445                                fapl, fcpl=make_fcpl(track_order=track_order, fs_strategy=fs_strategy,
    446                                fs_persist=fs_persist, fs_threshold=fs_threshold),
--> 447                                swmr=swmr)
    448 
    449             if isinstance(libver, tuple):

/u/system/SLES12/soft/pyiron/dev/anaconda3/lib/python3.7/site-packages/h5py/_hl/files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
    197         if swmr and swmr_support:
    198             flags |= h5f.ACC_SWMR_READ
--> 199         fid = h5f.open(name, flags, fapl=fapl)
    200     elif mode == 'r+':
    201         fid = h5f.open(name, h5f.ACC_RDWR, fapl=fapl)

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/_objects.pyx in h5py._objects.with_phil.wrapper()

h5py/h5f.pyx in h5py.h5f.open()

FileNotFoundError: [Errno 2] Unable to open file (unable to open file: name = '/cmmc/u/skanchar/pyiron/projects/research/Mg_surfaces_solid_solns/Ca_solid_solns/Ca_in_Mg_0001_surface/all_configs/low_cov_sym/low_cov_sym_from_tot/junk_copy_to_check/tot_DFT_data_table.h5', errno = 2, error message = 'No such file or directory', flags = 40, o_flags = 0)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[niklassiemer]

What is the current status here?

[pmrv]

It regrettably dropped from my radar.

@sravankumarkb The first issue occurs before copy_to is called, so I think this is an issue with the job you're trying to load. The second issue is probably related to this; before tables were stored in a way that would break some tables after they're read from HDF. Check whether this table was created before this change was merged, if not so, this is a separate bug. The final issue seems to be that (sub-) directories are not created before the copying is called, I will look into that.

[sravankumarkb]

It regrettably dropped from my radar.

@sravankumarkb The first issue occurs before copy_to is called, so I think this is an issue with the job you're trying to load. The second issue is probably related to this; before tables were stored in a way that would break some tables after they're read from HDF. Check whether this table was created before this change was merged, if not so, this is a separate bug. The final issue seems to be that (sub-) directories are not created before the copying is called, I will look into that.

Hi Marvin @pmrv
Yes, the table was created before that change was merged May 19, 2021.
Yeah, I think the main issue is the (sub-) directories not getting created before calling copy. I think if the main issue is fixed, then this PR can be added to the main.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pmrv]

@sravankumarkb Can you give it another go with this version?

EDIT: Actually hold off until the tests pass.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pmrv]

@sravankumarkb This should be good to go now.