Model crash when setting baseline_spending=True

[martinholmer]

I'm estimating just the steady-state equilibrium for a simple model specification that is not too different from the one in the run_examples/run_ogcore_example.py script.

With the following specification, I get the steady-state solution in about two minutes:

    og_spec = {
        'start_year': 2021,
        'frisch': 0.41,
        'cit_rate': [0.21],
        'alpha_T': [0.09],
        'alpha_G': [0.05],
        'r_gov_shift': 0.04,
        'PIA_maxpayment': 470,
        'baseline_spending': False,
        'debt_ratio_ss': 1.0,
        'tG1': 20, 'tG2': 256,
    }

But when I change baseline_spending to be True, OG-Core crashes with the following error traceback:

(ogcore-dev) run_examples% time python example_4.py
Number of workers =  4
making dir:  /Users/mrh/work/OG-Core-master/run_examples/OUTPUT_BASELINE/SS
making dir:  /Users/mrh/work/OG-Core-master/run_examples/OUTPUT_BASELINE/TPI
In runner, baseline is  True
r, w =  0.0648 1.1669904764984569
Traceback (most recent call last):
  File "/Users/mrh/work/OG-Core-master/run_examples/example_4.py", line 122, in <module>
    main()
  File "/Users/mrh/work/OG-Core-master/run_examples/example_4.py", line 68, in main
    runner(p, time_path=DO_TPI, client=client)
  File "/Users/mrh/work/OG-Core-master/ogcore/execute.py", line 45, in runner
    ss_outputs = SS.run_SS(p, client=client)
  File "/Users/mrh/work/OG-Core-master/ogcore/SS.py", line 595, in run_SS
    sol = opt.root(SS_fsolve, guesses, args=ss_params_baseline,
  File "/Users/mrh/anaconda3/envs/ogcore-dev/lib/python3.10/site-packages/scipy/optimize/_root.py", line 234, in root
    sol = _root_hybr(fun, x0, args=args, jac=jac, **options)
  File "/Users/mrh/anaconda3/envs/ogcore-dev/lib/python3.10/site-packages/scipy/optimize/_minpack_py.py", line 226, in _root_hybr
    shape, dtype = _check_func('fsolve', 'func', func, x0, args, n, (n,))
  File "/Users/mrh/anaconda3/envs/ogcore-dev/lib/python3.10/site-packages/scipy/optimize/_minpack_py.py", line 24, in _check_func
    res = atleast_1d(thefunc(*((x0[:numinputs],) + args)))
  File "/Users/mrh/work/OG-Core-master/ogcore/SS.py", line 532, in SS_fsolve
    inner_loop(outer_loop_vars, p, client)
  File "/Users/mrh/work/OG-Core-master/ogcore/SS.py", line 166, in inner_loop
    tr = household.get_tr(TR, None, p, 'SS')
  File "/Users/mrh/work/OG-Core-master/ogcore/household.py", line 192, in get_tr
    tr = ((p.eta[-1, :, :] * TR) /
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'
python example_4.py  10.20s user 1.48s system 219% cpu 5.333 total

Did I make a mistake in specifying the OG-Core example run?
Or is the logic associated with baseline_spending=True not being tested?
Or is something else going on that I don't understand?

[rickecon]

@martinholmer. I have replicated your error. We definitely test specifications of baseline_spending=True in test_fiscal.py, test_parameters.py, test_SS.py, and test_TPI.py. The steady-state in this baseline_spending=True case breaks pretty quickly, so my guess is that there is a weird error in the code early on that will be easy to fix. We'll work on this. Thanks.

[rickecon]

@jdebacker. I think @martinholmer has discovered a bug in the SS.py code for a case that is not tested (baseline=True and baseline_spending=True). His original error was solved by PR #785. But after that bug was fixed, the following bug was manifest. Every relevant test of baseline_spending=True in our tests (both local and GH Actions) is for the case when baseline=False.

test_fiscal.py: line 161

test_SS.py:     lines 50-51
                lines 148, 161
                lines 255, 264
                lines 460, 462, 479, 481

test_TPI.py:    line 36
                lines 151, 182

I think at least one place where the steady-state code is wrong is in lines 521-524 of SS.py. I think those two if statements should be indented under the else in line 517 as shown below such that they imply not p.baseline.

    if:
        ...
    else:
        BQ = guesses[3:-1]
        TR = guesses[-1]
        factor = factor_ss
        if p.baseline_spending:
            TR = TR_ss
        if not p.budget_balance and not p.baseline_spending:
            Y = TR / p.alpha_T[-1]

Otherwise lines 521-524 in SS.py undo anything from line 512 or line 515 in the case where p.baseline=True. I ran @martinholmer's experiment with this change and it got to the end of the steady-state computation with infinitesimally small GE loop errors. However, the aggregate resource constraint was not satisfied.

GE loop errors =  [-1.2479739464055228e-12, 2.0958790258873705e-12, 1.404654170755748e-12, -3.903127820947816e-14, 9.298117831235686e-16, 2.1989354781482007e-14, 2.2773449792623524e-14, 3.419486915845482e-14, 7.485331798839923e-14, 2.259954376415685e-14, 0.0, -5.5289106626332796e-14]
r, w =  0.01920549633655985 1.5371989072442491
K_d has negative elements. Setting them positive to prevent NAN.
Iteration: 01  Distance:  1.04652383382564
K_d has negative elements. Setting them positive to prevent NAN.
Foreign debt holdings =  2.5586551019363957
Foreign capital holdings =  0.23375987695461717
resource constraint:  -0.07336564288448541
Steady state government spending is negative to satisfy budget
Resource Constraint Difference: -0.07336564288448541
Traceback (most recent call last):
  File "/Users/richardevans/Documents/Economics/OSE/OG-Core/./run_examples/run_ogcore_martin.py", line 92, in <module>
    main()
  File "/Users/richardevans/Documents/Economics/OSE/OG-Core/./run_examples/run_ogcore_martin.py", line 84, in main
    runner(p, time_path=False, client=client)
  File "/Users/richardevans/Documents/Economics/OSE/OG-Core/ogcore/execute.py", line 45, in runner
    ss_outputs = SS.run_SS(p, client=client)
  File "/Users/richardevans/Documents/Economics/OSE/OG-Core/ogcore/SS.py", line 608, in run_SS
    output = SS_solver(b_guess, n_guess, rss, wss, Yss, BQss, TR_ss,
  File "/Users/richardevans/Documents/Economics/OSE/OG-Core/ogcore/SS.py", line 437, in SS_solver
    raise RuntimeError(err)
RuntimeError: Steady state aggregate resource constraint not satisfied

Given this error, I suspect that there is another place in the code where I have to make this change. Because it seems like the steady state is solving, but we have an accounting issue after the steady-state has solved. I am going to keep looking for this fix, but you know this case of baseline=True and baseline_spending=True better than I do.

And finally, after we find the guilty lines of code, we need to include some tests in the local set of tests that include the case in which baseline=True and baseline_spending=True.

[rickecon]

The steady-state now solves using the code updates currently in PR #786 with the experiment @martinholmer is proposing above. I will take the draft status off that PR once I have ensured that the TPI accounts for this case correctly and I have added tests for this case.

GE loop errors =  [1.740413368978011e-13, 1.411093464298574e-12, -5.784261958297066e-14, 4.152407584445683e-14, 5.1861293037802625e-14, 5.4366233737113134e-14, 3.2210345501937354e-14, 3.775452173115923e-14, 4.0826717007114155e-14, 7.645793720367777e-15, -5.2111093218343285e-15, 1.0827450047656839e-13]
r, w =  0.06016097186882638 1.1934295357832039
Iteration: 01  Distance:  2.1138133828432157e-12
Foreign debt holdings =  0.2523164486518481
Foreign capital holdings =  0.07991705205517102
resource constraint:  4.513247414683619e-13
Checking constraints on capital, labor, and consumption.
	There were no violations of the constraints on labor  supply.
	There were no violations of the constraints on  consumption.
Maximum error in labor FOC =  1.1013412404281553e-13
Maximum error in savings FOC =  1.0258460747536446e-13
JUST SAVED SS output to  /Users/richardevans/Documents/Economics/OSE/OG-Core/run_examples/OUTPUT_BASELINE/SS/SS_vars.pkl
It took 49.338387966156006 seconds to get that part done.
run time =  49.338425159454346

[jdebacker]

@rickecon and @martinholmer This isn't a bug. The combination of parameter values, baseline=True and baseline_spending=True does not make sense. baseline_spending=True means using the same level of spending as in the baseline run -- so it only applies to reform runs.

Note that in default_parameters.json we condition the validation of budget_balance on the value of baseline_spending, but we can't do the same for baseline since it's not a parameter defined in that file (not sure it should be), but we can put an assert statement somewhere that doesn't allow this combination of parameter values.

[martinholmer]

In issue #782, @jdebacker said:

@rickecon and @martinholmer This isn't a bug. The combination of parameter values, baseline=True and baseline_spending=True does not make sense. baseline_spending=True means using the same level of spending as in the baseline run -- so it only applies to reform runs.

Thanks for pointing this out. For the benefit of novice users of the model, it would be a good idea to tell users that the model considers this an illegal specification. The default_parameters.json does not clearly state that --- it says that baseline_spending is a "[f]lag to keep level of government spending and transfers constant between baseline and reform runs." And when a user uses this illegal specification, the model does not object and then crashes with no helpful error message.

Notice that in my original comment in this issue #782, I did not assert that this was a bug. I simply reported the details of the model crash and asked among other things: "Did I make a mistake in specifying the OG-Core example run?"

So, thank you for pointing out that the answer to that question is yes.

[rickecon]

Thanks @martinholmer and thanks @jdebacker. The rest of this discussion will take place in the thread for PR #786.