The HMIS Warehouse project was initiated by the City of Boston's Department of Neighborhood Development to gather data from across various HMIS installations, produce aggregated reports, and supply de-duplicated client information to the Boston CAS system for Coordinated Access to housing.
The Warehouse is capable of ingesting standard HUD HMIS CSV files as well as data via the Social Solutions ETO API.
Copyright © 2017 Green River Data Analysis, LLC
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
A copy of the license is available in LICENSE.md
The City of Boston made a conscientious choice to release this project into the open source under a GPL. Our goal is to promote this opportunity, allowing Boston's investment to assist other municipalities and organizations, and realize the vision of a tool under continuous, collaborative improvement helping communities nationwide.
Looking ahead, we see the Warehouse codebase serving as a foundation for all communities that report to the department of Housing and Urban Development, or have a need to aggregate and de-duplicate homeless client data from across various systems. To our knowledge, this is the only open source, freely available implementation of many HUD reports.
The application is designed around the HUD Data Standards and the data structure is based on the HMIS Logical Model
The application is written primarily in Ruby on Rails and we use RVM to select a ruby version. Other ruby version managers should work fine, as would manually installing the ruby version mentioned in the .ruby-version
The application uses postgres for application data storage and Microsoft SQL Server or postgres for the warehouse data.
We've developed locally on OSX using homebrew and deployed to Ubuntu 16.04 using apt
for dependencies.
-
Amazon Web Services provides cloud hosting services. Configured via various env variables. Services used outside of hosting are SNS, SES, S3, and Glacier. Terms: https://aws.amazon.com/service-terms/
-
ETO API and QaaWS - HMIS and ancillary data can be fetched via various endpoints. OpenPath accesses Social Solutions ETO in a variety of ways, custom configured for each installation. [https://www.socialsolutions.com/software/eto/]
-
Nominatim - an API service for open street map data [https://nominatim.openstreetmap.org]. Only zip codes are sent to the service, used to display a map on client dashboards of last permanent destinations. Terms: https://operations.osmfoundation.org/policies/nominatim/
-
NOAA Weather - Weather for individual dates and one location per installation are fetched on-demand from NOAA [https://www.ncdc.noaa.gov/cdo-web/webservices/v2]
-
Exceptions can be sent to a slack channel. Configuration is done via env
EXCEPTION_WEBHOOK*
-
MassHealth - X12 data transfer via api, configurable per installation
-
OKTA - for single sign-on. [https://www.okta.com]
-
US Census - for comparing HMIS and population data. [https://www.census.gov/data/developers/about/terms-of-service.html]
-
Talent LMS - for training. [https://www.talentlms.com/terms]
If you are unfamiliar with contributing to open source projects on Github you may first want to read some of the guides at: https://guides.github.com/
There is a simple script to setup a development environment in bin/setup
. To make it run smoothly you should have:
- A running Ruby 2.3+ environment with bundler 1.11+ installed.
- A local install of postgresql 9.5+ allowing your user to create new databases.
- A local install of redis for caching. redis-server should be running on the default port
- libmagic (
brew install libmagic
) - freetds (
brew install freetds
)
Once these are in place, bin/setup
should:
- Install all ruby dependencies.
- Create initial copies of configuration files.
- Create an initial database and seed it with reference data and a randomly generated admin user.
If all goes well you should then be able to run bin/rails server
and open the Warehouse in your system at http://localhost:3000 using the email/password created during bin/setup
. If not, read bin/setup
to figure out what went wrong and fix it.
Hack on your version as you see fit and if you have questions or want to contribute open an issue on github.
We use the following common rails gems and conventions:
haml
for view templatingbootstrap
for base styles and layoutsass
for custom-csssimple_form
for formspagy
for paginationbrakeman
for basic security scanning.rack-mini-profiler
to make sure pages are fast. Ideally <200ms- helpers need to be explictly loaded in controllers. i.e. we have
config.action_controller.include_all_helpers = false
set bin/rake generate controller ...
doesn't make fixures and they are disabled in test_helper. We don't use them and instead seed data in test or let test create their own data however they need to.- it also doesn't make helper or asset stubs, make them by hand if you need one. See
config/application.rb
for details.
The project reads/writes from several different databases. We keep track of these different environments by setting up parallel db configs and structures for each database. Health care data is configured in config/database_health.yml and database resources are in db/health. Warehouse data is configured in config/database_warehouse.yml and resources are in db/warehouse. When running migrations, use the custom generators.
App migrations can be created with:
rails generate migration foo
and run with
rake db:migrate
Warehouse migrations can be created with:
rails generate warehouse_migration foo
and run with
rake warehouse:db:migrate
Health migrations can be created with
rails generate health_migration foo
and run with
rake health:db:migrate
Several tables in the warehouse database are modeled after HMIS CSV Format Specifications (2024 link). These tables can be identified by their casing, which matches the CSV Spec. For example Enrollment
, Client
, ProjectCoC
, etc.
The HMIS CSV-Structured tables share some common attributes:
data_source_id
: identifies which Data Source the HMIS data came fromEnrollmentID
: HUD Key referencing Enrollment tablePersonalID
: HUD Key referencing Client tableProjectID
: HUD Key referencing Project tableUserID
: HUD Key referencing User table
These "HUD Keys" are used to link HMIS records together. We always use composite keys for relationships between HMIS CSV tables. This is necessary because HUD Keys are not unique across data sources.
Enrollment.id
is that database ID for the enrollment. It is unique in the Enrollment table. It is an incrementing numeric id that is generated by the database.Enrollment.enrollment_id
is theEnrollmentID
column on the Enrollment table. We use aliasing the snake-case the HMIS fields, so you can either use EnrollmentID or enrollment_id (snake-case is preferred). This ID is a String (32 char max) that is defined by the HMIS system that generated the data. For the OP HMIS, we use uuids. Other HMIS's use other conventions. This value is NOT unique in the Enrollment Table, but it should be unique in combination with thedata_source_id
.- The above comments apply to ProjectID, PersonalID, OrganizationID, and HouseholdID.
Both models are backed by the same table. We have separate model classes for the Warehouse and HMIS to encapsulate different logic. Shared logic is present in concerns (app/models/concerns/hud_concerns
).
Database | Table name | Warehouse Model | HMIS Model | Usage |
---|---|---|---|---|
app |
users |
User |
Hmis::User |
This is the rails application user. Each record represents a user that can log into the application. These users have access controls, audit histories, and so on. |
warehouse |
User |
GrdaWarehouse::Hud::User |
Hmis::Hud::User |
These records are imported from external HMIS data sources, or generated from the OP HMIS. Imported records oftentimes do NOT correspond to any "real" application user. Table structure comes from User.csv specification. The UserID column from this table is present on all the other CSV-structured tables. In HUD's words, "UserID in this file is used to associate data in other CSV files with a specific user." |
There are some tables used by OP HMIS that look like CSV-structured tables, but they're not defined by the HUD spec. For example: CustomAssessments
, CustomClientAddress
, CustomClientName
, etc.
The CSV specs have a "Custom file transfer template" indicating that HMIS systems can export custom data in files Custom*.csv
as long as they follow the basic HUD-style format (using EnrollmentID, DateUpdated, DateCreated, etc.). Because of that, we decided to implement some of the custom HMIS data tables using this pseudo-HMIS-CSV structure.
Follow these steps to create a new custom report.
- Generate scaffolding for a new Rails Driver:
rails g driver custom_new_report_name
- Add the relevant routes, controller, and view to the new Driver. See other examples for how they should be namespaced.
- Add your new report to the
report_list
in theGrdaWarehouse::WarehouseReports::ReportDefinition
model. Make sure to wrap it in a check to see whether the driver is loaded. See ReportDefinition for examples. - Finally, run
rails db:seed
to populate the database with your new report definition. Alternatively, you can run the two relevant steps individually:# Generates a new ReportDefinition GrdaWarehouse::WarehouseReports::ReportDefinition.maintain_report_definitions # Adds the report to the "all reports" group AccessGroup.maintain_system_groups