Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add test that compares tbi and std behavioral responses #1843

Closed
wants to merge 2 commits into from
Closed

Add test that compares tbi and std behavioral responses #1843

wants to merge 2 commits into from

Conversation

GoFroggyRun
Copy link
Contributor

@GoFroggyRun GoFroggyRun commented Jan 24, 2018 •
edited
Loading

This PR is based on @martinholmer's PR #1840 that intends to identify a potential bug in tbi. The test added in #1840 passes. I modified the reform file in the test a bit, and the test fails to pass. I got the following information in particular:

(taxcalc-dev) Sean-Wangs-MBP:taxcalc seanwang$ py.test -m tbi_vs_std_behavior
============================================== test session starts ==============================================
platform darwin -- Python 2.7.14, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: /Users/seanwang/Documents/GIT/tax-calculator, inifile: setup.cfg
plugins: xdist-1.20.1, forked-0.2, hypothesis-3.38.5, pep8-1.0.6
collected 436 items                                                                                             

tests/test_tbi.py F                                                                                       [100%]

=================================================== FAILURES ====================================================
___________________________________________ test_behavioral_response ____________________________________________

puf_subsample =         DSI  e00200  e00300  e00400  e00600  e00650  e00700  e00800  e00900  \...     0  
76122         0        0        7       0  

[10776 rows x 87 columns]

    @pytest.mark.pre_release
    @pytest.mark.tbi_vs_std_behavior
    @pytest.mark.requires_pufcsv
    def test_behavioral_response(puf_subsample):
        """
        Test that behavioral-response results are the same
        when generated from standard Tax-Calculator calls and
        when generated from tbi.run_nth_year_tax_calc_model() calls
        """
        # specify reform and assumptions
        reform_json = """
        {
            "policy": {
                "_cpi_offset": {"2017": [0.0025]},
                "_II_rt1": {"2018": [0.10]},
                "_II_rt2": {"2018": [0.15]},
                "_II_rt3": {"2018": [0.25]},
                "_II_rt4": {"2018": [0.28]},
                "_II_rt5": {"2018": [0.33]},
                "_II_rt6": {"2018": [0.35]},
                "_II_rt7": {"2018": [0.396]},
                "_PT_rt1": {"2018": [0.10]},
                "_PT_rt2": {"2018": [0.15]},
                "_PT_rt3": {"2018": [0.25]},
                "_PT_rt4": {"2018": [0.28]},
                "_PT_rt5": {"2018": [0.33]},
                "_PT_rt6": {"2018": [0.35]},
                "_PT_rt7": {"2018": [0.396]},
                "_PT_excl_rt": {"2018": [0.0]},
                "_PT_excl_wagelim_rt": {"2018": [9e99]},
                "_PT_excl_wagelim_thd": {"2018": [[0.0, 0.0, 0.0, 0.0, 0.0]]},
                "_PT_excl_wagelim_prt": {"2018": [[0.0, 0.0, 0.0, 0.0, 0.0]]},
                "_CTC_c": {"2018": [1000.0]},
                "_CTC_ps": {"2018": [[75000.0, 110000.0, 55000.0, 75000.0, 75000.0]]},
                "_ACTC_Income_thd": {"2018": [3000.0]},
                "_DependentCredit_Child_c": {"2018": [0.0]},
                "_DependentCredit_Nonchild_c": {"2018": [0.0]},
                "_DependentCredit_Nonchild_c": {"2018": [0.0]},
                "_DependentCredit_before_CTC": {"2018": [0.0]},
                "_ALD_AlimonyPaid_hc": {"2019": [0.0]},
                "_ALD_AlimonyReceived_hc": {"2019": [1.0]},
                "_ALD_DomesticProduction_hc": {"2018": [0.0]},
                "_ID_prt": {"2018": [0.03]},
                "_ID_crt": {"2018": [0.8]},
                "_ID_Charity_crt_all": {"2018": [0.5]},
                "_ID_Casualty_hc": {"2018": [0.0]},
                "_ID_AllTaxes_c": {"2018": [[9e99, 9e99, 9e99, 9e99, 9e99]]},
                "_ID_Miscellaneous_hc": {"2018": [0.0]},
                "_ID_Medical_frt": {"2017": [0.1]}
            }
        }
        """
        assump_json = """
        {"behavior": {"_BE_sub": {"2018": [0.25]}},
         "growdiff_baseline": {},
         "growdiff_response": {},
         "consumption": {}
        }
        """
        params = Calculator.read_json_param_objects(reform_json, assump_json)
        # specify keyword arguments used in tbi function call
        kwargs = {
            'start_year': 2018,
            'year_n': 0,
            'use_puf_not_cps': True,
            'use_full_sample': False,
            'user_mods': {
                'policy': params['policy'],
                'behavior': params['behavior'],
                'growdiff_baseline': params['growdiff_baseline'],
                'growdiff_response': params['growdiff_response'],
                'consumption': params['consumption']
            },
            'return_dict': False
        }
        # generate aggregate results two ways: using tbi and standard calls
        num_years = 9
        std_res = dict()
        tbi_res = dict()
        for using_tbi in [True, False]:
            for year in range(0, num_years):
                cyr = year + kwargs['start_year']
                if using_tbi:
                    kwargs['year_n'] = year
                    tables = run_nth_year_tax_calc_model(**kwargs)
                    tbi_res[cyr] = dict()
                    for tbl in ['aggr_1', 'aggr_2', 'aggr_d']:
                        tbi_res[cyr][tbl] = tables[tbl]
                else:
                    rec = Records(data=puf_subsample)
                    pol = Policy()
                    calc1 = Calculator(policy=pol, records=rec)
                    pol.implement_reform(params['policy'])
                    assert not pol.reform_errors
                    beh = Behavior()
                    beh.update_behavior(params['behavior'])
                    calc2 = Calculator(policy=pol, records=rec, behavior=beh)
>                   assert calc2.behavior_has_response()
E                   assert False
E                    +  where False = <bound method Calculator.behavior_has_response of <taxcalc.calculate.Calculator object at 0x106f4b710>>()
E                    +    where <bound method Calculator.behavior_has_response of <taxcalc.calculate.Calculator object at 0x106f4b710>> = <taxcalc.calculate.Calculator object at 0x106f4b710>.behavior_has_response

tests/test_tbi.py:365: AssertionError
--------------------------------------------- Captured stdout call ----------------------------------------------
puf-read-time= 3.4
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 34.2
puf-read-time= 2.6
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 12.3
puf-read-time= 3.7
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 14.7
puf-read-time= 2.9
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 11.2
puf-read-time= 3.0
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 11.7
puf-read-time= 2.9
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 13.9
puf-read-time= 3.8
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 12.1
puf-read-time= 3.3
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 12.4
puf-read-time= 3.9
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
seed=3779082008
elapsed time for this run: 14.7
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
You loaded data for 2009.
Tax-Calculator startup automatically extrapolated your data to 2013.
--------------------------------------------- Captured stderr call ----------------------------------------------
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
ValueError: Buffer dtype mismatch, expected 'Python object' but got 'long'
Exception ValueError: "Buffer dtype mismatch, expected 'Python object' but got 'long'" in 'pandas._libs.lib.is_bool_array' ignored
============================================= 435 tests deselected ==============================================
================================== 1 failed, 435 deselected in 143.67 seconds ===================================

@martinholmer does this seem a bug to you?

cc @MattHJensen @hdoupe

@codecov-io
Copy link

Codecov Report

Merging #1843 into master will not change coverage.
The diff coverage is n/a.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #1843   +/-   ##
======================================
  Coverage     100%    100%           
======================================
  Files          37      37           
  Lines        3103    3103           
======================================
  Hits         3103    3103

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f69b78a...330fdab. Read the comment docs.

@martinholmer
Copy link
Collaborator

In pull request #1843, @GoFroggyRun asked:

does this seem a bug to you?

Since you asked me a direct question, I'll reply with a direct answer: No.

What we have here is a misuse of the Tax-Calculator library. Let me explain.

You mostly copied the test code from pull request #1840, but you did make some changes.

    assump_json = """
    {"behavior": {"_BE_sub": {"2018": [0.25]}},  # <======== YOU CHANGED "2013" TO "2018"
     "growdiff_baseline": {},
     "growdiff_response": {},
     "consumption": {}
    }
    """
    params = Calculator.read_json_param_objects(reform_json, assump_json)
    ... [snip] ...
                rec = Records(data=puf_subsample)
                pol = Policy()
                calc1 = Calculator(policy=pol, records=rec)
                pol.implement_reform(params['policy'])
                assert not pol.reform_errors
                beh = Behavior()
                beh.update_behavior(params['behavior'])
                calc2 = Calculator(policy=pol, records=rec, behavior=beh)
                assert calc2.behavior_has_response()  # <======== ASSERT STATEMENT FAILS
                calc1.advance_to_year(cyr)
                calc2.advance_to_year(cyr)
                calc2 = Behavior.response(calc1, calc2)

You knew when you called the assert calc2.behavior_has_response() statement that the calc2's current year was before 2018 because you advanced calc1 and calc2 to 2018 (or after) in the next two statements. You also knew that you had changed "behavior" so that the there was no behavioral response before 2018. So, I'm puzzled why you thought the assert calc2.behavior_has_response() statement would not make the test fail.

What you have done here is logically equivalent to the following situation:

A contributor prepares a pull request that adds the following test to the test_behavior.py file

def test_behavioral_response_logic():
   assert 1 == 2

and in the pull request the contributor says "I've found a bug in the behavioral response logic because my test fails".

Sean, I believe you can do better than this.

It may be true that there is a bug in the Behavior.response() logic, but the current version of your pull request #1843 has not found a bug.

@GoFroggyRun GoFroggyRun mentioned this pull request Jan 25, 2018
Closed
@GoFroggyRun
Copy link
Contributor Author

Silly me. Thanks for your logical equivalence example @martinholmer. Closing in favor of #1847.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@GoFroggyRun @codecov-io @martinholmer