Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[config-system] Validator rewrite: Use jsonschema #576

Merged
merged 17 commits into from
Jun 2, 2016
Merged

[config-system] Validator rewrite: Use jsonschema #576

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
17 commits
Select commit Hold shift + click to select a range
1f22f03
Add a JSON schema for the convergence_strategy property.
ftsamis Apr 26, 2016
a072b78
Add an example json schema for the quantity_range_sampled type.
ftsamis May 7, 2016
b6467b5
Add the complete JSON schema equivalent to tardis_config_definition.yml
ftsamis May 16, 2016
1ae9996
Split the config schema to multiple sub-schemas.
ftsamis May 20, 2016
8ee81b4
Allow integers in exponential/decimal format. Disallow additional pro…
ftsamis May 24, 2016
1573fe4
Allow additional properties in model.abundances.uniform, so it is pos…
ftsamis May 24, 2016
58c9258
Add exponent as a property for exponential density (adapt #571).
ftsamis May 24, 2016
0519f4f
Add default empty values to non-mandatory objects
ftsamis May 31, 2016
1a1520b
setup.py: Add tardis/io/schemas in package data.
ftsamis May 31, 2016
da2d387
io/tests/test_config_reader.py: Remove test_custom_yaml_loader
ftsamis May 31, 2016
cd812e5
io/tests/test_configuration_namespace.py: Remove the unused config_va…
ftsamis May 31, 2016
2c8fc4c
io/config_validator.py: Rewrite the entire module to use jsonschema
ftsamis May 31, 2016
f2abb1c
Convert no_of_packets and last_no_of_packets to int
ftsamis Jun 1, 2016
2cd9832
Use ducktyping instead of checking if spectrum_list is a dict.
ftsamis Jun 1, 2016
001abe7
Make validate_dict/validate_yaml have a default argument value for sc…
ftsamis Jun 2, 2016
d59c357
Add 'damped' as the default value for convergence_strategy in the sch…
ftsamis Jun 2, 2016
60d1d1d
Remove 'tardis/data/tardis_config_definition.yml' and all the referen…
ftsamis Jun 2, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions conda-requirements
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ h5py==2.5
matplotlib==1.4.3
astropy==1.0.5
PyYAML==3.11
jsonschema==2.5.1
numexpr==2.4.4
Cython==0.21
networkx==1.10
Expand Down
38 changes: 19 additions & 19 deletions tardis/io/config_reader.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
import tardis
from tardis.io.model_reader import (
read_density_file, calculate_density_after_time, read_abundances_file)
from tardis.io.config_validator import ConfigurationValidator
from tardis.io import config_validator
from tardis.io.util import YAMLLoader, yaml_load_file
from tardis import atomic
from tardis.util import (species_string_to_tuple, parse_quantity,
Expand All @@ -27,6 +27,7 @@

default_config_definition_file = os.path.join(data_dir,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still need this line?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, removed all the references to tardis_config_definition.yml and the file itself.

'tardis_config_definition.yml')
config_schema_file = os.path.join(config_validator.schema_dir, 'base.yml')
#File parsers for different file formats:


Expand Down Expand Up @@ -486,7 +487,9 @@ def parse_spectrum_list2dict(spectrum_list):
"""
Parse the spectrum list [start, stop, num] to a list
"""

if isinstance(spectrum_list, dict):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how about we change to 'if 'start' in spectrum_list and 'stop' in spectrum list and 'num' in spectrum_list`: 😉

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was expecting a comment on this line.
I even thought about adding a "Dear Wolfgang, this is as temporary as it can get, please act like you didn't notice the missing ducks" comment. 😛

I'll do the change...

spectrum_list = [spectrum_list['start'], spectrum_list['stop'],
spectrum_list['num']]
if spectrum_list[0].unit.physical_type != 'length' and \
spectrum_list[1].unit.physical_type != 'length':
raise ValueError('start and end of spectrum need to be a length')
Expand Down Expand Up @@ -621,13 +624,8 @@ def from_config_dict(cls, config_dict, config_definition_file=None):

"""

if config_definition_file is None:
config_definition_file = default_config_definition_file

config_definition = yaml_load_file(config_definition_file)

return cls(ConfigurationValidator(config_definition,
config_dict).get_config())
return cls(config_validator.validate_dict(
config_dict, config_schema_file))

def __init__(self, value=None):
if value is None:
Expand Down Expand Up @@ -793,8 +791,8 @@ def from_config_dict(cls, config_dict, atom_data=None, test_parser=False,

config_definition = yaml_load_file(config_definition_file)
if validate:
validated_config_dict = ConfigurationValidator(config_definition,
config_dict).get_config()
validated_config_dict = config_validator.validate_dict(config_dict,
config_schema_file)
else:
validated_config_dict = config_dict

Expand Down Expand Up @@ -850,7 +848,9 @@ def from_config_dict(cls, config_dict, atom_data=None, test_parser=False,
structure_section = model_section['structure']

if structure_section['type'] == 'specific':
start, stop, num = model_section['structure']['velocity']
velocity = model_section['structure']['velocity']
start, stop, num = velocity['start'], velocity['stop'], \
velocity['num']
num += 1
velocities = quantity_linspace(start, stop, num)

Expand Down Expand Up @@ -1018,7 +1018,7 @@ def from_config_dict(cls, config_dict, atom_data=None, test_parser=False,



if montecarlo_section['convergence_strategy'] is None:
if 'convergence_stragegy' not in montecarlo_section:
logger.warning('No convergence criteria selected - '
'just damping by 0.5 for w, t_rad and t_inner')
montecarlo_section['convergence_strategy'] = (
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since we decided to use default values for now, I propose to move these default values to either the schema or the Simulation class

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good idea.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done, nice and clean in the schema!

Expand All @@ -1031,19 +1031,19 @@ def from_config_dict(cls, config_dict, atom_data=None, test_parser=False,
black_body_section = montecarlo_section['black_body_sampling']
montecarlo_section['black_body_sampling'] = {}
montecarlo_section['black_body_sampling']['start'] = \
black_body_section[0]
black_body_section['start']
montecarlo_section['black_body_sampling']['end'] = \
black_body_section[1]
black_body_section['stop']
montecarlo_section['black_body_sampling']['samples'] = \
black_body_section[2]
black_body_section['num']
virtual_spectrum_section = montecarlo_section['virtual_spectrum_range']
montecarlo_section['virtual_spectrum_range'] = {}
montecarlo_section['virtual_spectrum_range']['start'] = \
virtual_spectrum_section[0]
virtual_spectrum_section['start']
montecarlo_section['virtual_spectrum_range']['end'] = \
virtual_spectrum_section[1]
virtual_spectrum_section['stop']
montecarlo_section['virtual_spectrum_range']['samples'] = \
virtual_spectrum_section[2]
virtual_spectrum_section['num']
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What's the reason we keep this?
I propose to delete lines 1031-1046 and rename montecarlo.virtual_spectrum_range.samples to montecarlo.virtual_spectrum_range.num wherever it is accessed

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is part of the next and final big milestone/PR (configuration system restructure). My purpose in this PR was to only do changes related and absolutely necessary to the validator.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@yeganer that's dangerous as older configuration files need to work. I would flag it but leave it for now.

Copy link
Contributor

@yeganer yeganer Jun 1, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should have a function/module that provides the backwards compatibility.

if config.version < latest_config_version:
    config_dict = interpret_legacy_config(config_dict)

This function would then consecutively update from the config version the config is in to the latest verstion.


###### END of convergence section reading

Expand Down
Loading