As a parcel-level simulation, I want to maintain a list of active development projects so that they can be completed, delayed, or cancelled depending on conditions in a simulation year.

Simulation runs sometimes error in this line, with the max_profit column not found. I think this is because there are no profitable developments passed into the module, but we need a better way of handling these edge cases systematically.

Approved new workflow for developer model

Latest in wiki.

This is a high-level design proposal meant for discussion and review. I've tagged this and other such issues with the label Type: Proposal. These are not meant to be treated as user stories or handled in the GitHub project workflow. Everything in here needs to be broken down into a user story for development.

Stories

• Parcels should be able to be split into smaller developable entities (let's call them "development sites") based on rules or user input
• Parcels should be able to be aggregated into larger development sites based on rules or user input
• If a parcel is split or aggregated, AND one of the resulting sites is developed, that site and the other sites resulting from the split/aggregation should be "remembered" in the next simulation year so as not to attempt a new, incompatible split or aggregation.
• It should be possible to manage one or multiple development sites as a project (let's call it a "development project") that should be analyzed as a group
• It should be possible to perform feasibility analysis on a development project (group of development sites)
• It should be possible to make development decisions on a development project
• There should exist a "pipeline" of projects that are ready for development
• Projects in the pipeline should be able to be "built" - added to the list of buildings that exist in the model - in the current simulation year, or scheduled with an expected completion date in the future
• Projects in the pipeline should be able to make progress on construction in a given year, without completion
• Projects in the pipeline should be able to be delayed or canceled based on certain conditions
• Individual development sites within projects should get "built" from the pipeline.

Summary

See "proposed workflow" below for details.

• A new Project Creator step generates development projects (via splitting, aggregation, or most likely a one-to-one match) from parcels.
• A new Pipeline Adder step appends results of the developer model to the pipeline table, which persists across model years.
• A new Constructor step builds projects from the pipeline table, site-by-site.

Current workflow

Here I'll try to summarize the current workflow that utilizes the proforma and developer models. Feel free to skip to the proposed workflow below. I'm using the model names from the urbansim_parcels repo's models.py.

1: Preliminary models

• The residential and non-residential hedonic models estimate land prices in the region for different uses
• The household and job location choice models move a subset of agents around the region
• The transition models add households and jobs to the region based on control totals

2: Feasibility

Generates a table of parcels with some metrics about development feasibility, like profitability.

• Inputs: parcels table
• Outputs: Table with development metrics (e.g. maximum profit), indexed by parcel_id

In this model, each parcel represents the possible development (or redevelopment) of one building, and each parcel's feasibility is analyzed as such.

3: Developer

Decides which buildings get developed, based on the outputs from the feasibility step.

• Inputs: feasibility, some demand-side parameters to determine how many units to build
• Outputs: DataFrame of buildings that are built in this year

Proposed workflow

Here are all of the tables involved:

• Parcels
• Buildings
• Development sites: can be subset or superset of parcels. Empty in first year.
• Development projects: comprised of one or more development sites. Empty in first year.
• Pipeline: comprised of development projects that are scheduled to be built
• Feasibility, Project Feasibility: created and deleted each year between the Project Creator/Feasiblity and Developer steps

1: Preliminary models, including scheduled developments, which are added directly into the pipeline.

1.1: Project Creator

A new ProjectCreator model takes in the parcel table and pipeline (empty in year 1), and generates development projects and development sites.

• Inputs: parcels, pipeline, configs
• Outputs: development_projects, development_sites, project_feasibility

The project creator performs these steps:

• Read in parcels table into a parcel_candidates table
• Read in pipeline table; remove parcels involved in an active pipeline project from parcel_candidates
• For user-configured projects:
  • Read in user-configured projects: these would be rules for how to split/aggregate/pass specific parcels into development sites, then combine development sites into a development project.
  • Create development project (row in a table that includes reference to parcels and development sites)
  • Create development sites (rows that include reference to development project and parcels)
  • Append to the development_projects and development_sites tables
  • Run ProjectProForma model (below) and append results to project_feasibility
  • Remove parcels involved in user-configured projects from parcel_candidates
• For everything else:
  • Run the rest of parcel_candidates through the Splitter function
  • If parcel is split:
    • Create development project and development sites
    • Append to development_projects and development_sites
    • ProjectProForma has been run. Append results to project_feasibility
  • If parcel is aggregated:
    • Need to handle this use case with an Aggregator function.
  • If parcel is not split or aggregated:
    • Create one development project, comprised of one development site
    • Append to development_projects and development_sites

The Splitter function should take in parcels and if they qualify under some criteria (probably it's larger than a maximum parcel size), run an algorithm to determine the "best" split. One possible algorithm is:

• Based on certain settings, generate a list of possible splits. This is a combination of numbers (e.g. split the parcel into X development sites) and uses. Most common use case will be splitting a large agricultural parcel into many (potentially hundreds) of single family homes. Need to consider computation time here.
• For each possible combination, run ProjectProForma.
• Compare profitability for each combination and select the best one.
• For the best combination, save the results of ProjectProForma so it can be appended to project_feasibility

The ProjectProForma model takes a development project as an input, runs sqftproforma on every development site in the project, and aggregates results up to the project level. This requires some thoughtfulness about aggregation rules (should we just add up max_profit?).

2: Feasibility

Runs sqftproforma model for projects with only one development site, and combines results with the project_feasibility table from the previous step.

• Inputs: development_projects, project_feasibility, configs
• Outputs: feasibility

Unlike the existing workflow, this step performs analysis on development sites as units, rather than parcels. Steps are:

• Read in development_projects into projects_to_analyze
• Remove projects in project_feasilibility (precalculated from previous step) from projects_to_analyze
• Run sqftproforma as usual on projects_to_analyze, output feasibility
• Append project_feasibility to feasibility

3: Developer

Decide which development projects get pushed into the pipeline, based on the outputs from the feasibility step.

• Inputs: feasibility, some demand-side parameters to determine how many units to build, configs
• Outputs: Index of projects to send to pipeline

There is not much change in the Developer model itself, except use development sites instead of parcels. It does this:

• Read in feasibility
• Do some filtering out of unprofitable projects and such
• Assign every project a probability based on rules from specific implementation
• Based on target numbers, randomly select a certain number of projects based on probability and output, output index of projects to build

3.1: Append to pipeline

A new PipelineAdder step should append development sites to the pipeline table. This may be a function we simply want the developer model to handle, but there may end up being some more complicated rules regarding how we add to the pipeline.

• Inputs: Index of projects from Developer step, pipeline, development_projects, development_sites
• Outputs: pipeline

Steps:

• Read inputs
• For each new project to add to the pipeline, calculate the following information:
  • Construction time (how long does it take to build), expected completion year
  • Velocity (derivation of above)
  • Some probability of completion, or of continued construction in the current simulation year
• Output new pipeline

4: Build from the pipeline

A new Constructor model "constructs" development sites in development projects in the pipeline. These are then added to the buildings table.

• Inputs: pipeline, buildings
• Outputs: pipeline, buildings

At this point, pipeline is a list of development projects, and each row has information on the status of development sites within the project. Steps:

• Read inputs
• For each project in pipeline:
  • If completed this year:
    • Add any remaining development site(s) to buildings
    • Remove from pipeline
  • Else:
    • Based on velocity, figure out if one of the development sites should be built this year
    • Add those development sites to buildings
    • Mark those development sites as built in the pipeline

We can add in logic to delay or cancel projects as well. pipeline and buildings persist across simulation years. Everything else is reset.

Add ability to use velocity function to control the speed at which large developments get developed in a parcel model, in order to reduce volatility related to large changes of local household counts in one year. See SEMCOG email.

As MAG, I want to input a pipeline of known developments (with uncertain information about start date) and integrate them into a simulation with controls or limits on how much of a project can develop in a single year.

Begin on a model class for market research. This should include functionality for:

• All "data collection" steps (i.e. registering status columns on parcel table)
• Actual market research calculations (e.g. absorption)
• Filtering input into feasibility step
• Adding information into table that gets passed to feasibility step

SqftProForma class has max_retail_height and max_industrial_height attributes; these should be generalized to a max_height attribute which is a dictionary, e.g.:

max_height = {"retail": 2, "industrial": 2}

(Suggestion by @hanase)

As a parcel-level simulation, I want to be able to split up large developments into sub-developments and evaluate the feasibility of these sub-developments on distinct timelines, to more realistically simulate large projects.

This has been requested several times. See Paul's explanation on Discourse:

Unlike the current spot pro forma, iIt will be multi-period in its design, enabling larger projects to be scheduled over multiple periods, and incorporating time-dependent costs and revenue expectations to be reflected in the ROI calculations, which will more appropriately reflect the risk - reward profile of large projects.

As a modeler, I want to be able to run parcel-level simulations in the same or less amount of time as with the previous version of the pro forma/developer modules.

My plan is to simply:

• Run a simulate.py from sanfran_urbansim
• Run a simulate.py from urbansim_parcels/sf_example

And compare the times. @conorhenley would like to hear any thoughts for other speed benchmarks that would be useful.

The continuous integration script for this repository includes running some code from urbansim/urbansim_parcels. That repo is private, and Travis is no longer able to install it now that the developer library has been moved to UDST.

This is the line that fails in Travis:
https://github.com/UDST/developer/blob/master/.travis.yml#L26

I'm not too familiar with how Travis works, but it looks like we might be able to fix this by generating a new deploy key. For now, I'm disabling that part of the continuous integration script so that we can continue running the unit tests on their own.

Reference:
https://docs.travis-ci.com/user/private-dependencies/

As SEMCOG, I want to make sure parcel level simulations don't develop single use buildings on land zoned for mixed use. We need to reproduce this behavior. See email.

As a modeler, I want to be able to generate a range of feasibility results for each parcel, based on random draws of distributions of input values, rather than a single estimate.

Uncertainty analysis will be added using Monte Carlo Simulation to provide a distribution of results rather than a single estimate.

1. Generate several combination of development sites (split into smaller pieces, leave it as it is).
2. Apply to each dev. site of every combination ProForma to calculate profitability.
3. Select combination with the best aggregated profitability.

Input

• Parcel

Output

• Development project with the best aggregated profitability for the received parcel
  

There is the same bug as in the public version of the model: When passing all forms (i.e. forms=None) into the developer model, the max profit is not computed. It's in _get_dataframe_of_buildings.

I submitted a PR fix for the previous version a while ago (still open):
UDST/urbansim#194

I probably cannot do a PR on a private repository without forking it to a public repository, so I won't.

Specifically, only_built, parcel_filter, residential_to_yearly, forms_to_test, pass_through, and simple_zoning seem to be missing from the class method. SqFtProForma.get_defaults() appears to be fully specified, so just making sure there's parity between these two functions should do the trick.

Method SqFtProForma.check_is_reasonable should be parameterizable or at least toggleable.

We should consider some use cases:

• need to execute proforma using values that are out of this fixed range
• need to set ad-hoc reasonable values

From Discourse:

Methods to speed up the search over feasible project configurations, and make this search 'smarter'.

This search has several parts:

0. Upon instantiating a SqftProForma object, generate a dictionary of DataFrames (one for each (form, parking configuration) pair. Each DataFrame is a lookup table that takes into accounts the various configurations and costs.
1. In the parcel model (this is our latest version), loop through forms and call the lookup method.
2. For each form, loop through each parking configuration to generate a DataFrame that provides information about profit, costs, revenue, etc.
3. Select the parking configuration with maximum profit

This is an open-ended issue. It seems like there may be some good ways to improve this search and reduce the brute-force complexity, but we haven't diagnosed specific problems, and we are hoping you can take a deeper look and consider possible solutions.

As a modeler, I should be able to use the developer model in a consistent manner with other urbansim models. Consistent with #10.

Overview

Current workflow is as follows:

dev = developer.Developer(feasibility.to_frame())

target_units = dev.\
    compute_units_to_build(len(agents),
                           buildings[supply_fname].sum(),
                           target_vacancy)

new_buildings = dev.pick(forms,
                         target_units,
                         parcel_size,
                         ave_unit_size,
                         total_units,
                         max_parcel_size=max_parcel_size,
                         drop_after_build=True,
                         residential=residential,
                         bldg_sqft_per_job=bldg_sqft_per_job)

if year is not None:
    new_buildings["year_built"] = year

if form_to_btype_callback is not None:
    new_buildings["building_type_id"] = new_buildings["form"].\
        apply(form_to_btype_callback)

all_buildings = dev.merge(buildings.to_frame(buildings.local_columns),
                          new_buildings[buildings.local_columns])

sim.add_table("buildings", all_buildings)

I would like to make this consistent with calls to other urbansim models in utils.py, such as hedonic_simulate. The function could look something like this:

cfg = misc.config(cfg)
feasibility = orca.get_table('feasibility').to_frame()
buildings = orca.get_table('buildings').to_frame()
new_buildings, all_buildings = developer.pick(feasibility, buildings, cfg)
orca.add_table('buildings', all_buildings)

Details

Very similar to #10. Essentially moving all these steps into the Developer class. A couple of specifics:

• Everything after the pick() method should be moved into a add_new_buildings() method that takes the results of pick and registers new buildings.
• Essentially all of the parameters for the run_developer function in utils.py should become YAML config variables.

We can add a very simple financing mechanism to the pro forma model. The pro forma model that Jessica Hitchcock shared with us and presented in class provides a good example.

Each of these can be added as config items (they would go in proforma.yaml, values for Santa Cruz example development in parentheses):

• Construction loan to cost ratio (70%)
• Loan fees (points) (2%)
• Interest rate (5.5%)
• Drawdown factor (60%)

And other variables can be calculated and factored into the profit calculation in the lookup method:

• Construction period (months) (21) - should be a function of development size
• Total construction cost
• Total loan amount
• Interest on construction loan
• Points on construction loan
• Total financing costs

The total financing cost is added to the total development costs, and then the calculation proceeds as usual. The way Jessica calculates profit is exactly the same as how pro forma calculates it, so I think this approach would work well.

I think this provides some helpful levers on the financing side, e.g. how do interest rates affect development?

@conorhenley would love any feedback.

Just like with #12, we should move to YAML configurations, like other urbansim models use. Specifically (these are copied from #12):

• Moving configuration parameters to explicit object attributes like in LCM objects
• Writing a from_yaml class method to instantiate Pro Forma objects from YAML, like this. Make sure the option to read directly from string (instead of file or buffer) is included.
• Writing a to_yaml class method, which depends on a to_dict method, like this.
• Writing a lookup_from_config method to wrap a lookup method for YAML, like this

As a modeler, I should be able to use the pro forma model in a consistent manner with other urbansim models.

Overview

Currently, the workflow looks like this:

pf = sqftproforma.SqFtProForma()

df = parcels.to_frame()

# add prices for each use
for use in pf.config.uses:
    df[use] = parcel_price_callback(use)

# convert from cost to yearly rent
if residential_to_yearly:
    df["residential"] *= pf.config.cap_rate

d = {}
for form in pf.config.forms:
    print "Computing feasibility for form %s" % form
    d[form] = pf.lookup(form, df[parcel_use_allowed_callback(form)])

far_predictions = pd.concat(d.values(), keys=d.keys(), axis=1)

sim.add_table("feasibility", far_predictions)

I would like to make this consistent with calls to other urbansim models in utils.py, such as hedonic_simulate. The function could look something like this:

cfg = misc.config(cfg)
df = parcels.to_frame()
feasibility = proforma.lookup(df, cfg)
orca.add_table('feasibility', feasibility)

This has the dual benefit of simplifying the function call to the core model, and making the workflow more consistent with other urbansim models.

Details

For now, this simply requires moving these steps from utils.py into the SqftProForma lookup method. New parameters to add to the configuration options are:

• parcel_filter
• residential_to_yearly
• simple_zoning

Refactoring will be considered in the v0.1.1 milestone.

Add ability update housing prices (or other housing market conditions) to REMI projected per capita income (or other indicators) over the course of a simulation, to more accurately simulate a rapidly changing housing market. (See SEMCOG email)

Read in user-configured projects: these would be rules for how to split/aggregate/pass specific parcels into development sites, then combine development sites into a development project.

I'm proposing changes to the Developer class similar to what was implemented in PR #26. This would split the pick() method out of the Developer class into its own class (called DevelopmentPicker or something like that). This would make the developer module consistent with the pro forma module, where:

• The main class (Developer or SqFtProForma) read the model config, contain methods for IO, and wrap the core functionality.
• Separate classes contain logic for the actual functionality (e.g. SqFtProFormaLookup or DevelopmentPicker).

As part of these changes I would also refactor the DevelopmentPicker.pick() method by breaking it up into smaller helped methods where appropriate.

Currently doesn't have a method similar to predict_from_cfg that distinguishes between filename and YAML string. Thus in urbansim_parcels, we're currently loading the YAML string which is a little inconsistent with the rest of the model system.

Similar to #20. We want to eventually build out the ability to return a range of results (rather than a point estimate) based on monte carlo simulation. See #14.

This is partly an issue here, partly in the parcel model, but we should reconsider the print functions throughout the pro forma and developer steps in both places. They need to provide the most useful feedback possible about what's happening and whether the models are properly responding to adjustments.

As SEMCOG, I want to be able to control the types of buildings that get developed in a parcel level simulation, so that building types are more predictable in later processes like household allocation.

From email:

Exiting proforma model assigns new development with random building types. For example, for residential development, SF, MF and Condo buildings are randomly assigned to the new constructed buildings. This could cause some problem in later stage of model run if building types are used in the household allocation or other process. Therefore, it would be preferable to have some logical way to assign building types instead of random. At this stage we are not quite sure what the best practice to assign building types and if it is a problem to assign building types based purely on new building characteristics.

Input

• Rules for user-configured development projects

Output

• Development projects composed of one or more development sites

Aggregate parcels and execute ProForma into this new (bigger) development site to look for increased profitability -in comparison to that of original parcellation-.

Input

• Parcels

Output

• Development site -from aggregated parcels- with the best profitability

I just finished another full run of our model. I got a lot of warnings of the form

A value is trying to be set on a copy of a slice from a DataFrame

Hear they all are each with some context. Hope we can get them fixed. :-)

. . .
Running step 'feasibility'
/home/da/udst/developer/developer/sqftproforma.py:1077: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.fars > self.max_industrial_height] = np.nan
/home/da/udst/developer/developer/sqftproforma.py:1074: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.fars > self.max_retail_height] = np.nan
. . .  
Running step 'residential_developer'
Number of agents: 1,875,553
Number of agent spaces: 2,047,607
Current vacancy = 0.08
Target vacancy = 0.20, target of new units = 296,834
235,185 feasible buildings before running developer
/home/da/udst/developer/developer/develop.py:312: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df["ave_unit_size"] = self.ave_unit_size
/home/da/udst/developer/developer/develop.py:313: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df["parcel_size"] = self.parcel_size
/home/da/udst/developer/developer/develop.py:314: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  df['current_units'] = self.current_units
Sum of net units that are profitable: 118,252
WARNING THERE WERE NOT ENOUGH PROFITABLE UNITS TO MATCH DEMAND
Adding 16,311 buildings with 119,857 residential_units
218,874 feasible buildings after running developer
/home/da/udst/urbansim_parcels/urbansim_parcels/utils.py:785: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  drop_buildings['year_demo'] = orca.get_injectable('year')
Unplaced households before: 75873
/home/da/udst/urbansim_parcels/urbansim_parcels/utils.py:805: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  agents.building_id[displaced_agents] = -1
Unplaced households after: 77144
Unplaced jobs before: 59063
Unplaced jobs after: 69705
. . .
Running step 'hlcm_simulate'
There are 2155293 total available units
    and 1875553 total choosers
    but there are 1 overfull buildings
    for a total of 365531 temporarily empty units
    in 201824 buildings total in the region
/home/da/udst/urbansim/urbansim/models/dcm.py:1846: VisibleDeprecationWarning: using a non-integer number instead of an integer will result in an error in the future
  replace=False)
Assigned 85292 choosers to new units
Total currently unplaced: 499
    and there are now 279740 empty units
    and 1 overfull buildings
Time to execute step 'hlcm_simulate': 10.08 s
Running step 'elcm_simulate'
There are 1489792 total available units
    and 592082 total choosers
    but there are 36135 overfull buildings
    for a total of 1264093 temporarily empty units
    in 15248 buildings total in the region
Assigned 51493 choosers to new units
/home/da/anaconda2/lib/python2.7/site-packages/pandas/core/indexing.py:465: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: http://pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy
  self.obj[item] = s
Total currently unplaced: 0
    and there are now 949203 empty units
    and 36135 overfull buildings

We are using
urbansim_parcels: 7078ca1d01c215778fe4e635f56c2e32b3c143ef
developer: b3bd8b3

From Discourse:

Capacity to evaluate trade-offs such as increasing density bonuses to offset inclusionary zoning requirements and their effect on project feasibility.

Goals we stated before are:

• Better parameterization (too many hard-coded variables and path dependencies)
• Better performance, if possible (probably mostly achieved by improving search in #13, reducing brute force complexity)
• More readable/maintainable code

I've made some progress on this refactoring already, but figured I'd write this up here as well to stick with the process. The first step for better maintainability/readability is to bring much of the core functionality out of the SqftProForma class altogether. I've created new classes for:

• The "reference table" that provides development possibilities for each combination of form and parking configuration (this was previously generated in the _generate_lookup method, but I wanted to distinguish names between this reference table and the actually "lookup" process that takes parcels are returns the most profitable configuration).
• The lookup process itself

I keep the original SqftProForma for its loading and saving methods, and use it as a facade pattern to call the other stuff. This is a pretty simple use case for a facade but I think it makes it easier to read.

See flowchart diagram.

Need more clarification.

From Discourse:

Market analysis to reflect initial phases of assessment of market feasibility within local market areas will be introduced as part of the real estate development model flow.

Idea is to filter parcels with some initial, crude metrics to scale the problem down before running it through the core calculations.

I changed my mind on some of the changes I incorporated in #25. I would like to implement the following changes:

• agents, buildings, supply_fname, and target_vacancy should not be attributes of the Developer object. I think these should be handled in the parcel model. I brought them into the object as part of #25 because they are arguments to the compute_units_to_build method. This brings me to...
• The compute_units_to_build method should be moved out of the developer module into the parcel model. It's not dependent on any data actually inside the Developer object, and is purely an operation on model data.
• For the same reason, the merge method should
  also be moved out of the developer module into the parcel model.
  • @conorhenley We should discuss the greater theory behind this separation of concerns. It's my opinion that the model code (e.g. urbansim_parcels) should handle all of the data operations, including the previous two methods. Maybe there's a good reason why Fletcher kept them in the Developer object? Maybe not.
• remove_developed_buildings and unplace_agents should also be moved out of the Developer object. They are only acting as pass-throughs to be called at a later point in the parcel model, and were moved into the Developer object so that they could be cleanly included in the YAML config. I think that they should be put back into the run_developer step in the parcel model as arguments, and taken out of the config altogether.

These changes also require updates to unit tests and helper methods.

Thus, Developer object attributes are distinguished from parameters passed to Developer.pick() as such:

• Variables that are configurations that live in a file like settings.yaml or res_developer.yaml, or are otherwise likely to be default values, are object attributes.
• Variables that are likely to change within the context of a simulation, or between regions' implementations, should be passed as parameters. This should go directly to the pick method rather than be saved as object attributes.

This method should receive a list of parcels and pipeline projects and remove all parcels who are part of an active project.

Input

• Parcels table
• Projects table

Output

• Subset of parcels not involved in any active project

I'm confused about why fars gets filtered by the heights variable in this line in the pro forma model.

• heights is a copy of the height column in the dev_info reference table. It's calculated from the FAR based on a parcel size of 10000 (the default value passed to parcel_sizes)
• but it's compared to the max_height value for the parcels actually being tested
• it seems like max_height should be compared to the height generated from the actual parcel size, not the default 10000.

This might be working correctly but I want to come back to this for clarification. This shows up in my example for class, in the 'surface' loop of _lookup_parking_cfg.

The current San Francisco example in urbansim_parcels doesn't yet have a pro forma YAML file to load configs from (it's loading using the from_defaults method of the SqFtProForma class). I need to create and test.

Currently, the developer workflow in the parcel model decides how many buildings to build based on numbers of target units and job spaces that are determined by vacancy rate. The user sets a desired vacancy rate, and the target number is generated from that. This isn't particularly realistic and we should eventually try to move toward ROI (return on investment) as the main control for how many buildings get built. This requires some feedback loops involving absorption rates, vacancy, etc.

This is a longer-term project that will hopefully get easier once we build out the new developer workflow.

Looking at parking_rates and sqft_per_rate among other params emerges the following question:
Why if parcel_sizes can use imperial and metric measure units the other parameters are fixed to the former? Maybe units could be considered imperial if not specified (default) and metric if some flag param exists.
Maybe not a top priority but could we consider this mainly to be able to use ProForma module outside countries that use imperial system.

The entire package should be compatible with at least Python 3.5, and we should add to Travis builds. We may want to further build out unit tests and add a code coverage tool before we do this.

Given that year_built will be assigned by the parcel model, I think it might make sense to remove the assignment in this line.

Input

• Development project

Output

• Insert development project into pipeline table

1. Calculate construction time
2. Derivate velocity
3. Probability (P) of completion in current year
4. Insert project into pipeline table

The types of parking configurations, as well as their square footage and cost, are configurable in the SqftProFormaConfig object. See docstrings for those pieces here.

As explained in the docstrings, these three set parking types are hardcoded in several places in the SqftProForma code. Examples here and here.

We should explore whether it makes sense to fully parameterize and expose these functionalities to be configurable, or to leave them as three rigid types that must be selected from. If the former, we need to develop a thoughtful specification for the parking configuration - perhaps a new config object.

DRAFT Model feedback loop: housing absorption to feasibility

User stories

• As a modeler, I want information about housing supply absorption in a given year to factor into the calculation of feasibility for potential developments in the same simulation year.
• As a modeler, I want to be able to control the extent to which housing supply affects the calculation of feasibility for potential developments in the same simulation year.

Basic mechanics

Current situation

The important output of this system is how many new developments are built in a given simulation year. Currently, this is determined through two models:

• The Pro Forma step calculates profitability for parcels. The primary endogenous inputs are 1) the current rents of the parcels, and 2) the typical rents for each use for each parcel. This does not currently use information about housing supply other than indirectly through prices.
• The developer step selects parcels for development. The way this currently works is:
  • Parcels are assigned a development probability based on results from pro forma
  • A target number of units to build is assigned. This is either an input, or calculated using the number of existing agents, number of existing units, and a target vacancy.
  • Enough buildings are built to match that unit target.

Problem with the current situation

In real life, how much development happens in a given year is more influenced by individual developers' decision making (i.e. profitability) than is reflected in the model. There are indeed development caps and other regulations that influence how much development happens, but a supply/demand equilibrium really should be the primary mechanism. And this shortcoming of the model is because there exists no feedback loop that provides information about building supply absorption back into the feasibility calculation.

Proposed solution

The basic steps we need are:

• Calculate absorption
• Add absorption to the feasibility step
• Revise developer process to select developments using a different metric

Detailed mechanics

Calculate absorption

What is absorption? (from Real Estate Market Analysis, page 20):

The pace at which the proposed project will be able to lease or sell space. Depending on the property type, the absorption rate could be expressed as:

• the number of apartments that will be leased or homes that will be sold each month;
• the length of time it will take to sell building sites in an industrial park; or
• the number of months until an office building or shopping center is fully leased.

A useful set of absorption related metrics for urbansim would be:

• Square footage absorbed for every simulation year, for every use. Save the absolute numbers (e.g. 50000 sq. ft. absorbed out of 100000 new and previously vacant) and the percentage (50% absorption).
• (Derived from above) Average % absorption for the last X (5?) years.
• (Derived from above) Rate of change in % or absolute absorption for the last X years.

As noted above, it will be useful to disaggregate absorption into uses: residential, office, industrial, etc. Residential and non-residential absorption will have to be calculated separately at the very least. The time period over which absorption is calculated is also important: this should be user-specified, but a good range would be 3-5 years.

Square footage is an easier unit than "units" or "job spaces" for now, since that's what the developer and pro forma models use.

Implementation, for every simulation year:

• After location choice step:
  • Calculate vacant square footage for this year (for each subgroup)
  • If last year's vacancy + new building information exists, calculate absorption
• After developer step: calculate new square footage for this year (for each subgroup)

Save absorption for every simulation year. Feasibility step can decide which records to use.

Add absorption to the feasibility step

Absorption for all previous simulation years is now an input. The current calculation of projected building revenue is as follows:

building_revenue = (building_bulks *
                    (1 - parking_sqft_ratio) *
                    self.building_efficiency *
                    df.weighted_rent.values /
                    self.cap_rate)

The simplest intervention is to subtract some amount from this:

building_revenue = (building_bulks *
                    (1 - parking_sqft_ratio) *
                    self.building_efficiency *
                    df.weighted_rent.values /
                    self.cap_rate)

# calculate absorption discount

building_revenue -= absorption_discount

How absorption_discount is generated from the absorption table can be an injected function, similar to the "profit to percentage" function in the developer model. One simple calculation could be:

• Treat the % not absorbed as the number of years it will take to fill the square footage. (absorption_years)
• Discount that number of years of revenue (requires using the cap rate):

absorption_discount = self.cap_rate * building_revenue * absorption_years

Revise developer process to select developments using a different metric

The number of developments that get built are determined in the developer model.
Currently, the rule is:

• If the number of target units (derived from target vacancy) is 0, don't build anything
• If the number of target units is greater than the number of profitable buildings, build everything
• If the number of target units is > 0 and < the number of profitable buildings, select buildings by probability (based on profitability)

New logic should look like this:

• If no development caps: build everything above a certain minimum profitability
• If development caps: for each cap,
  • Select buildings in relevant geography/subtype based on probability

Next steps

The absorption calculation should be built out into a more generalized market analysis module that includes other calculations, and which interfaces with feasibility in different ways.

The feasibility model expects users to set parcel_sizes in the yaml file (default is 10000). A comment in the code claims that the value does not matter because it cancels out in the pro-forma computation. This is true for some of the pro-forma measures. However, this settings influences the construction time and hence the profit. Here is more detail.

Probably an easy fix would be to keep the construction time on the per-sqft scale and then multiply it by the real parcel size in _lookup_parking_cfg().

Default settings for the Pro Forma model are currently set up as attributes of the SqftProFormaConfig object. Customization to these settings would involve passing a separate instance of the above object, like in SEMCOG's code here.

We should move to YAML configurations, like other urbansim models use. Specifically:

• Moving configuration parameters to explicit object attributes like in LCM objects
• Writing a from_yaml class method to instantiate Pro Forma objects from YAML, like this. Make sure the option to read directly from string (instead of file or buffer) is included.
• Writing a to_yaml class method, which depends on a to_dict method, like this.
• Writing a lookup_from_config method to wrap a lookup method for YAML, like this

As part of reengineering of core SqftProForma calculations, keep in mind that we would like to eventually extend functionality to include cash flow calculations. See #1 .

As an urbansim Developer object, I want to be able to make a decision on whether to build a project or not based on an internal rate of return or (given a discount rate) a net present value.

We should also expose the IRR/NPV figure to the end user.