rsginc / daysim Goto Github PK

Get DVRPC testbed setup

SVN revisions 2582 through 2585

This configuration switch had been moved to Settings by John Mulholland, but we were still thinking that it was operational in configuration - it makes more sense as a configuration setting, so it was moved back.

Will create a separate issue on flagging invalid Configuration lines, which would have alerted us to this issue much sooner.

People who are self-employed and work at home are generally counted in the employment data, so their work location should be counted and compared against the employment data in shadow pricing. Without this change, the current code tends to drive the work at home share towards 0 when there are more jobs than workers in the region.

Was revision 2566 in SVN

Revisoin 2580 in SVN

Daysim.ChoiceModels.ChoiceModelUtility.DrawRandomTourTimePeriods, which is used to guess a tour arrival and departure time for WorkTourDestinationChoice, never returns PM peak for tour destination departure time. Since Commuter Rail only runs outbound in the PM peakin Nashville, we end up with no drive commuter rail trips. I traced (see attached) the work tour destination arrival time and work tour destination departure times, and we always only get 8pm or later for departure time. It looks like arrival times are ok though. This will impact every model to some degree, especially where there is different service by time-of-day, like commuter rail, BRT, etc.
out.zip

Adds three new configuration parameters
IntrazonalWalkMinutesPerMile_OverrideSkims
IntrazonaBikeMinutesPerMile_OverrideSkims
IntrazonalAutoMinutesPerMile_OverrideSkims

If they are set >0, are used instead of skims to determine the speed to apply to the short-distance adjustment for intra-zonals in ImpedanceRoster.SkimValue. Use when the intrazonal values for skims are not meaningful.

Added extra additive distance terms for over 20, over 30 and over 40 miles in the WorkLocationModel (new parameters 46,47,48) and a walk mode calibration constant (parameter 70) in all the TourMode models (this was the base mode in estimation, so no parameter was estimated).
If these parameters are not in the F12 file, DaySim will just ignore the variables, so it is backwards compatible.

Need to add work and other tour mode choice constants for person type university student in Nashville. This will help correct large differences in tour mode results for university students in the Vanderbilt area. School/univ tour mode choice already has person type segmentation.

@mabcal to identify some examples of good and bad practices in order to better define this task

The code to read the configuration file should pause and print an error if it reads a configuration line that is not valid (is not contained in Configuration.cs).
We still had the DestinationScale line in assuming that it was being used correctly, but it had been moved from Configuration to Settings, and there was never a warning or error to alert us that the DestinatlonScale line in the configuration file was no longer valid.

Was converted from minutes after 3 am to minutes after midnight twice

The DORP (driver or passenger) variable on the output trip file is used for this, since it is otherwise always 0 for transit trips, and there are no empty output fields available on the trip file, so this provides the information without changing the file format. The value is rounded to integer minutes of walk time, access and egress combined.

When making this change, I discovered an error in the path calculations for the new stop area version of the walk-to-transit path type model. It was not multiplying the access/egress walk times by 2 when run at the tour level for round-trip LOS, so that has been fixed. This does not affect cases where TAZ-based transit matrices are used - only DVRPC.

Bug found by John B

This was due to edits made to get it to also read SFCTA's HDF5 files.
Temporary fix was made so it will work for both.
Still need to make a more elegant fix to get rid of PSRC switch. (Idea is to use two separate file type labels in the Roster and use that instead of PSRC switch).

Changed to work like the other readers.

Daysim performance seems to max out at 8 threads....can we do better than that? We should profile the code and see where the bottlenecks are.

If a location is not specified on the command line the log is stored in the same directory as the .exe

I think we will/should be moving away from executables being kept in the model directories and therefore a better system would be for it to be stored in the same folder as the other default items such as the output folder are stored. I believe this is the same directory as the configuration file.

This came up for me as a developer because when I ran from inside Visual Studio the log file was difficult to find in subdirectory: bin\x64\Debug

Suzanne from PSRC had put in a a postbuild app that would write the SVN revision number to the log print file. This does not work for Git, so it was deleted from the post-build configuration

@mabcal requested

See RSGInc/DaysimDataTools#8

in order to diff output files from two identical runs to ensure there are no differences, we need to first sort output files by ID before writing them out.

Need to revise the transit path utility to:

1. use a weighted constant based on the IVT share by sub-mode
2. discount IVT by sub-mode

Will add Global.Configuration.Nashville == true

This was due to edits made to get it to also read SFCTA's HDF5 files.
Temporary fix was made so it will work for both.
Still need to make a more elegant fix to get rid of PSRC switch. (Idea is to use two separate file type labels in the Roster and use that instead of PSRC switch).

Nashville not working with recent changes, but issue due to something else.

Open matrix (OMX) is an HDF5-based matrix container for travel modeling. We need to add support for it to Daysim. Currently Daysim reads and writes HDF5-based matrices for SFCTA and PSRC. OMX also uses HDF5 as its container, but it uses it differently than SFCTA and PSRC. The OMX specification is here. PSRC wrote some C# for OMX I/O as well that we may be able to start from or simply use. We need to add OMX reader and writer classes as well as support to the impedance roster for OMX.

Add documentation to this Daysim project site on how to add a new model and a new test to our testing system. Currently we have Nashville as our test system with different bat files for running different tests.

We agreed our formal test system would look like:

1. a folder for each model
2. each model must run without a commercial travel modeling package license - so no TransCAD, Cube, or EMME matrices. Thus, need to use Daysim binary, VISUM binary, text, or OMX.
3. a DOS bat file for running the test
4. The test will run the model and an associated R/Python/etc script to process outputs
5. The script can either diff the same output from different runs or the user can after running two independent runs

This was done in a hurry for the Chattanooga project, and uses external R scripts to calculate the shadow prices and put them into the F12 files. Will create another issue to do this more elegantly within DaySim in the future.

PSRC and SFCTA are both using HDF5 for skims, but in slightly different ways. @mabcal is going to clean up the class so it works for both PSRC and SFCTA. Right now it is has a Global.Properties.PSRC toggle in it, which we'll continue to use for now. This will be removed when we do #37.

This can be especially useful for testing.

All the skim file readers were using short.maxint as a maximum storage value instead of ushort.maxint. I changed all readers to truncate at ushort.maxint -1 (the - 1 just to be safe). This will allow values to be twice as high before truncating (around 650 minutes, miles or dollars, instead of around 325)

The observation:

1. TransCAD skims with gaps between TAZ index.
2. The skims were read into memory using "transcad" format.
3. DaySim run would get into infinite loop (var random = randomUniform01.Uniform01() * originSegmentZone.TotalWeight in DestinationSampler.cs) in "Running choice models" stage for some households.
   a) Household 200 is one of these households, who lives in TAZ 889, internal zone number 690.
   b) There is no TAZ 690.
4. DaySim could run through without any problem when read the skims using "Text_IJ".
5. There is no households living in zone without connection.
6. Zone 773, 839, 855 are connected to network but have no households and employment. But aggregate log sum for these three zones are -92 when using "transcad" format while normal value when using "Text_IJ".

My guess: When using "transcad" format, the TAZ index from matrix is used as internal zone number while they should be treated as TAZ. In household 200, it is supposed to find aggegate logsum for TAZ 889, or internal zone 690. If program view the skim index as internal number, it would search for zone 690 aggregate logsum. But there is no TAZ 690 then no corresponding aggregate logsum.

Instead of using R scripts and F12 files, put shadow pricing code into Daysim

There is an issue with TransCAD matrix reader as described below.

If skims are in TransCAD format, an loading error might occur in following situations:

1. Matrix is saved in TransCAD 7; DaySim is built using TransCAD 6 DLL and local machine only has version 6 license;
2. Matrix is saved in TransCAD 7; DaySim is built using TransCAD 7 DLL but local machine only has version 6 license;
3. Matrix is saved in TransCAD 6; DaySim is built using TransCAD 7 DLL but local machine only has version 6 license;

Not sure whether this situation would work or not:

Matrix is saved in TransCAD 6; DaySim is built using TransCAD 6 DLL but local machine only has version 7 license;

There seems to be an error in “OtherTourDestinationCoefficients_nashville-v1.8.F12”, coefficient 24 is coded as 924 (row 27). I guess DaySim might ignore this coefficient if coded as 924.

Daysim is looking up the skims incorrectly for inbound versus outbound. They are transposed in some cases. We are investigating for Nashville.

I notice that during Engine.BeginCalculateAggregateLogsums(randomUtility) the automatic Garbage Collector dramatically reduces the amount of memory being used (by at least 50%) and it then stays low. I experimented by inserting GC.Collect() immediately after Engine.BeginLoadRoster() and saw that this forced an equivalent memory reduction at that (earlier) point in the process. Apparently there is something about the way that Engine.BeginLoadRoster is coded that causes memory to be hogged and retained until the entire roster is loaded and the corresponding impedance matrices are all created. This causes performance problems in cases where the ultimate size of the matrices is smaller than the available memory but Engine.BeginLoadRoster hogs more than the available memory. Paging grinds the process to a near standstill. Looked at another way, it requires more memory than should be necesary to support large rosters. I wonder if there is a way to recode Engine.BeginLoadRoster so that it frees memory up if possible after processing each entry.

We don't know what version / release each model is using. I suggest we create a Daysim.exe -version entry point that prints to the console the version / build date.

Changed it to work like the new code for the stop area version - uses transit travel time stored at the tour level. Needs to be tested on Nashville outputs.

This is called in 208 places. Haven't don't profiling but suspect it is called very often. The work is divvied into many batches, each run in a separate thread. As each thread does work it stores results. These results for each thread are stored separately in an array indexed by the batch id. The code to get the batch id from the thread is a linear search. This code is also called to look up information such as ModelIsInEstimationMode and GetChoiceProbabilityCalculator. It is true that the number of threads is small but there is no reason to do this -- each thread could simply store its associated batch id using Thread.SetData. Alternatively the name of each thread could be specified at creation. Even better the ChoiceModelHelper which is so often retrieved could be stored in the thread local data.

I do not know the limits to how much memory can be stored in each thread. It could be that each thread should just store all of its own data.