Addition totals calculated for dropq calculation

[talumbau]

• Instead of just a summary calculation of the delta between the policy
  and reform for total individual income tax, total payroll tax, and
  total combined tax for each budget year, we also compute these values
  for baseline and reform separately.

[talumbau]

Needed for TaxBrain:
ospc-org/ospc.org#380

[codecov-io]

Current coverage is 98.76% (diff: 100%)

Merging #1060 into master will increase coverage by <.01%

@@             master      #1060   diff @@
==========================================
  Files            38         38          
  Lines          2745       2761    +16   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits           2711       2727    +16   
  Misses           34         34          
  Partials          0          0          

Powered by Codecov. Last update 28fb74a...48c83c2

[martinholmer]

@talumbau, Can you eliminate the backslash continuations in pull request #1060? Doing so would harmonize the dropq code with the rest of the tax-calculator repo after #1036 was merged over a week ago.

[MattHJensen]

@talumbau, everything else looks good in this PR. I am going to delay releasing 0.7.1 until you merge this. Once you merge this, please go ahead and release 0.7.1 if I don't get there first.

[talumbau]

@MattHJensen @martinholmer I removed the backslash continuation lines.

[MattHJensen]

Thanks @talumbau. I'll merge now and release 0.7.1. Any additional changes can be included in 0.7.2

[martinholmer]

@talumbau, I don't understand your pull request #1060. The info on GitHub shows that it passed the code-coverage test but there is no record of it being subjected to the unit tests. And after I update my local version of the Tax-Calculator repo to include #1060, I get the following unit test failure.

=================================== FAILURES ===================================
_____________________________ test_full_dropq_puf ______________________________

puf_path = '/Users/mrh/work/OSPC/tax-calculator/taxcalc/tests/../../puf.csv'

    @pytest.mark.requires_pufcsv
    def test_full_dropq_puf(puf_path):

        myvars = {}
        myvars['_II_rt4'] = [0.39, 0.40, 0.41]
        myvars['_PT_rt4'] = [0.39, 0.40, 0.41]
        myvars['_II_rt3'] = [0.31, 0.32, 0.33]
        myvars['_PT_rt3'] = [0.31, 0.32, 0.33]
        first = 2016
        user_mods = {first: myvars}

        nyrs = 2
        # create a Policy object (clp) containing current-law policy parameters
        clp = Policy()
        clp.implement_reform(user_mods)
        # create a Records object (rec) containing all puf.csv input records
        rec = Records(data=puf_path)
        # create a Calculator object using clp policy and puf records
        calc = Calculator(policy=clp, records=rec)
        calc.increment_year()
        calc.increment_year()
        calc.increment_year()
        # create aggregate diagnostic table (adt) as a Pandas DataFrame object
        adt = multiyear_diagnostic_table(calc, nyrs)
        taxes_fullsample = adt.loc["Combined Liability ($b)"]

        assert taxes_fullsample is not None

        # Create a Public Use File object
        tax_data = pd.read_csv(puf_path)

        (mY_dec, mX_dec, df_dec, pdf_dec, cdf_dec, mY_bin, mX_bin, df_bin,
            pdf_bin, cdf_bin, fiscal_tots) = dropq.run_models(tax_data,
                                                              start_year=first,
                                                              user_mods=user_mods,
                                                              return_json=False,
                                                              num_years=2)

        pure_reform_revenue = taxes_fullsample.loc[first]
        dropq_reform_revenue = mY_dec['_combined_dec_0'].loc['sums']
        dropq_reform_revenue /= 1e9  # Round to billions of dollars
        diff = abs(pure_reform_revenue - dropq_reform_revenue)
        # Assert that dropq revenue is similar to the "pure" calculation
        assert diff / dropq_reform_revenue < 0.02

        # Assert that Reform - Baseline = Reported Delta
        delta_yr0 = fiscal_tots[0]
        baseline_yr0 = fiscal_tots[1]
        reform_yr0 = fiscal_tots[2]
        diff_yr0 = (reform_yr0.loc['combined_tax'] -
                    baseline_yr0.loc['combined_tax']).values
        delta_yr0 = delta_yr0.loc['combined_tax'].values
>       npt.assert_array_almost_equal(diff_yr0, delta_yr0, decimal=3)
E       AssertionError: 
E       Arrays are not almost equal to 3 decimals
E       
E       (mismatch 100.0%)
E        x: array([  1.081e+11])
E        y: array([  1.081e+11])

tests/test_dropq.py:114: AssertionError
----------------------------- Captured stdout call -----------------------------
You loaded data for 2009.
Your data have been extrapolated to 2013.
You loaded data for 2009.
Your data have been extrapolated to 2013.
You loaded data for 2009.
Your data have been extrapolated to 2013.
You loaded data for 2009.
Your data have been extrapolated to 2013.
elapsed time for this run:  24.0446178913
You loaded data for 2009.
Your data have been extrapolated to 2013.
You loaded data for 2009.
Your data have been extrapolated to 2013.
You loaded data for 2009.
Your data have been extrapolated to 2013.
elapsed time for this run:  24.5676641464
==================== 1 failed, 297 passed in 213.50 seconds ====================

What's going on?

@MattHJensen

[talumbau]

It seems like the unit tests did run, the above test is arequires_pufcsv test so it won't normally run with Travis CI though. The test is concerned with the combined tax (payroll and individual income tax). In dropq, we eventually calculate the difference in combined tax between the baseline and the policy for each tax record included in the calculation. That resulting column of data is summed. This test makes sure that the value of that calculation is within a tenth of a penny of the result of first summing the combined tax for baseline and summing the combined tax for the reform, and then finding the difference. It might be that this tolerance is too tight, since it passed on my machine but is not passing on yours right now. I will see if I can reproduce.

[talumbau]

hmm... I can't reproduce this failure. If I could get it to fail, I was going to attempt this change:

npt.assert_array_almost_equal(diff_yr0, delta_yr0, decimal=2)

(change the decimal argument from 3 to 2 to lower the precision level required)

Can you try that and see if it works?