Tour mode choice uses maz_to_maz_walk table, but trip mode choice uses matrix skim for same OD.

[stefancoe]

Describe the bug
The model crashes in trip_mode_choice because all probabilities are 0. Tour mode choice is 'walk' so 'walk' should be the only option in trip mode choice, but the OD distance for 'walk' is longer than the max allowed so all choices end up with 0 probability. Tracing the distances (scroll to bottom of attached trace files) in tour and trip mode choice shows different walk distances being used for the same OD. I confirmed that the distance used in tour mode choice is from our maz_to_maz_walk table and the distance used in trip mode choice is from the walk distance skim. The skim distance is quite large because the two MAZs belong to a very large TAZ and the skim value represents the intrazonal travel time. The MAZs are right next to each other so the distance is actually quite small.

We saw another case where the MAZs were not in the same zone and the distance values were different in the reverse direction according to the trip_mode_choice log file. So one direction was getting the maz_to_maz distance while the other was getting the skim distance.

To Reproduce
Steps to reproduce the behavior:

1. Use this mp settting, but change from 20 to 12 processors.
2. Use this for folder for all others settings, but change number of households to 100,000.
3. I'll provide a link once I get the input data uploaded.

Expected behavior
Tour and trip mode choice should use the same walk distance for the same OD.

tour_mode_choice_annotate_choosers_preprocessor-4c698d.csv

trip_mode_choice_annotate_trips_preprocessor-4c698d.csv

[jpn--]

I'm having trouble replicating this bug, possibly because I'm not matching your settings exactly. I am getting traced MAZ skim values that took correct for this household/trip. Does the problem persist for for you with other numbers of processors other than 12? Is changing from 20 to 12 processes required to produce the problem? Do you also see it if you run single-processes tracing that same household but with a much smaller sample size? In theory the problem should persist in all these scenarios as ActivitySim is supposed to be stable against changing the number of processes or sample sizes, but if it does not that might help diagnose the cause.

[stefancoe]

@jpn-- Thanks for looking into this! I will try to replicate this on a smaller sample and then provide the details and data if necessary.

[stefancoe]

@jpn-- I was able able to replicate the bug with 100 households. I am running with MP to get through accessibility, but let me know if you need to skip this step and I will upload the accessibility output table from this run. I uploaded the input data and the config files to to my google drive and shared them with you. The config files are in the folder psrc_activitysim, which is in the data_full folder.

https://drive.google.com/drive/folders/1L5R-v-mjTHYX4SM4S06PCRdhI5GCJ3nK?usp=sharing

The run args I am using:
activitysim run -c E:\asim_testing\psrc_activitysim\configs_mp -c E:\asim_testing\psrc_activitysim\configs_dev -d E:\asim_testing\data_full -o E:\asim_testing\asim_output_dirs\debug

Thanks!

*Edit to add that I am testing on the overflow-protection branch. Let me know if you want to me switch to generic-whale.

[jpn--]

@stefancoe yes please upload the accessibility table. If the same error happens in both single process and multi-process, debugging in SP is much easier than in MP.

[stefancoe]

@jpn-- Ok, working on that now. I will make sure the error still happens skipping accessibility in single process mode.

[stefancoe]

@jpn-- I uploaded the accessibility file and also updated the settings file in the configs_skip_accessibility folder to run with 100 HHs and trace the correct HHID. Running with these settings resulted in the same error. Thanks!

activitysim run -c E:\asim_testing\psrc_activitysim\configs_skip_accessibility -c E:\asim_testing\psrc_activitysim\configs_dev -d E:\asim_testing\data_full -o E:\asim_testing\asim_output_dirs\debug

[stefancoe]

@jpn-- Just wondering if you have been able to replicate this issue with single process/small sample?

[jpn--]

I am still not able to reproduce the issue you are showing. When I run the model I see the same household and the same walk tour, but I get the correct looking 0.8 mile walk distance. I get this using my "stable" branch of your repo from May, as well as using the most up-to-date version of your configs published in the "main" branch of the PSRC model repo. I've run with the now older overflow-protection branch of ActivitySim, as well as with the most current develop branch today.

@stefancoe you want to run this test one more time with your latest code and the latest develop-branch of ActivitySim? If it still fails, put literally everything: all the ActivitySim code you use, all the configs, all the log and trace and any other output files, all the input data files you're reading, all together in one place I can download and take one more stab at trying to reproduce.

[stefancoe]

@jpn-- I tried to run our setup using this branch:
https://github.com/ActivitySim/activitysim/tree/develop

It crashes immediately when using sharrow- see attached. Running a full population without sharrow results in an out of memory issue during trip destination, which does not occur when using the overflow-protection branch. Running with 100k households should cause the original issue.

The current config file on our repo has trip mode choice commented out. Can you test the config files I put in the google drive folder I shared with you on June 2nd:
#681 (comment)

If it does not crash using the traced HH, please try using 100k HHs.

[stefancoe]

@jpn-- Any luck on this?

[stefancoe]

@jpn-- I shared a folder with you on google drive that has everything needed to reproduce the bug including the run directory that is set to be restarted at trip mode choice. If you download the ‘Jeff’ folder to say c:, you can get the model running by using the following:

Activitysim run -c c:\Jeff\configs_skip_accessibility -c c:\Jeff\configs_dev -d c:\Jeff\data_full -o c:\Jeff\debug_small_sample

The run uses 10K HHs. The folder also includes the Activitysim code I have been using and an environment.yml file to create a conda environment if necessary. Ted Lin @tedlini from RSG tested it and was able to reproduce the bug.

Thanks!

https://drive.google.com/drive/folders/1WAyOIcfdBrs_2EMNFGEefZ1OrIdmVdtx?usp=drive_link

[jpn--]

Thanks! I will look at this soon.

[stefancoe]

@jpn-- Any updates on this? Thanks.