param.py, various errors due to malformed csv file. Poor reporting coming from param.py

[tobeycarman]

Three problems encountered using param.py. There are not really bugs with param.py, but were issues with the csv file. However the error handling and reporting to the user is so bad that it is really hard to figure out what the errors with the csv file are. Really need to make the code more flexible and have better error reporting; not sure how to do that yet.

Here are the errors:

Problem with finding PFT data in csv file

develop@cd9dc82a3fe0:/work$ param.py --csv2fwt-v1 CMT52_parameter_CSV_file.csv /work/parameters/ /work/calibration/calibration_targets.py
Traceback (most recent call last):
  File "/work/scripts/util/param.py", line 2627, in <module>
    sys.exit(cmdline_entry())
  File "/work/scripts/util/param.py", line 2623, in cmdline_entry
    return cmdline_run(args)
  File "/work/scripts/util/param.py", line 2266, in cmdline_run
    csv2fwt_v1(csvfile, ref_directory=ref_params, ref_targets=ref_targets)
  File "/work/scripts/util/param.py", line 422, in csv2fwt_v1
    pft_data = csv_v1_read_section(data, bounds=sections['pft'])
KeyError: 'pft'

Solution

The script is very sensitive to the names of the columns. So in the beginning of the PFT section of data (~line 30), the header line must be exactly like this:

file,name,0,1,2,3,4,5,6,7,8,9,units,description,comment,refs

Basically make sure all columns are present

Problem with finding ref_targets

develop@cd9dc82a3fe0:/work$ param.py --csv2fwt-v1 CMT52_parameter_CSV_file.csv /work/parameters/ /work/calibration/calibration_targets.py
/work/calibration/calibration_targets.py
Traceback (most recent call last):
  File "/work/scripts/util/param.py", line 2627, in <module>
    sys.exit(cmdline_entry())
  File "/work/scripts/util/param.py", line 2623, in cmdline_entry
    return cmdline_run(args)
  File "/work/scripts/util/param.py", line 2266, in cmdline_run
    csv2fwt_v1(csvfile, ref_directory=ref_params, ref_targets=ref_targets)
  File "/work/scripts/util/param.py", line 463, in csv2fwt_v1
    raise RuntimeError("Invalid ref_targets file! Can't find appropriate data.")
RuntimeError: Invalid ref_targets file! Can't find appropriate data.

Solution

The script is very picky about the path to the reference targets being complete and correct. So if your reference targets is /work/calibration/calibration_targets.py (which is printed out right before the stack trace), then in your .csv file, you need to make sure that the full path is present. In other words your csv should look like this:

file,name,0,1,2,3,4,5,6,7,8,9,units,description,comment,refs
/work/calibration/calibration_targets.py,PFTNames,Lichen,...
/work/calibration/calibration_targets.py,GPPAllIgnoringNitrogen,115.55,...

Notice how rather than simply calibration_targets.py, the full path is specified, and that this path matches the line that is printed out immediately before the stack trace in the error.

Problem with invalid character

ValueError                                Traceback (most recent call last)
File /work/scripts/util/param.py:2628
   2624   return cmdline_run(args)
   2627 if __name__ == '__main__':
-> 2628   sys.exit(cmdline_entry())

File /work/scripts/util/param.py:2624, in cmdline_entry(argv)
   2622 def cmdline_entry(argv=None):
   2623   args = cmdline_parse(argv)
-> 2624   return cmdline_run(args)

File /work/scripts/util/param.py:2267, in cmdline_run(args)
   2265   ref_params = args.csv2fwt_v1[1]
   2266   ref_targets = args.csv2fwt_v1[2]
-> 2267   csv2fwt_v1(csvfile, ref_directory=ref_params, ref_targets=ref_targets)
   2268   return 0
   2270 if args.params2csv_v0:

File /work/scripts/util/param.py:549, in csv2fwt_v1(csv_file, ref_directory, overwrite_files, ref_targets)
    547 s = '  '
    548 for i in range(0,10):
--> 549   s += smart_format(p[str(i)])
    550 s += '// {}: {} // {} // {} // {}\n'.format(p['name'], p['units'], p['description'], p['comment'], p['refs'])
    551 full_string += s

File /work/scripts/util/param.py:601, in smart_format(x, n, basefmt, toolongfmt)
    574 '''
    575 Provides a flexible method for printing different number formats.
    576 
   (...)
    598 str : formatted version of x
    599 '''
    600 if type(x) == str:
--> 601   x = float(x)
    603 if len(str(x).strip()) > n:
    604   return toolongfmt.format(float(x))

ValueError: could not convert string to float: '`'

Solution

The script is very picky about having only numeric data in the fields that are expected to be numbers. In the above example, there was a backtick character as pointed out in the ValueError message. Searching for this in the csv file revealed it in one of the PFT slots for initvegnl. Fixing this (changing to a zero) let the script run.