stateofcalifornia / calcat

California Communicable diseases Assessment Tool

Introduction

Welcome to the California Communicable diseases Assessment Tool (CalCAT). This repository contains an application written in Shiny and for use with any US state to assist in assessing the many different models available for understanding COVID-19 transmission and spread. It brings together several data sources that are publicly available, and can be supplemented with your own data to improve the assessment.

Getting Started

ATTN: This application is designed to run with Rstudio. If you need to download and install Rstudio go here to get started.

0. Fork this Repository
1. Enter your state's name in line 50 of the cal_cat_data_routine.rmd where it says state_name <- "California" Ctrl+Alt+R This routine will download data for your state from a number of different modeling groups. You can find out more about them in the Technical notes app the application.
2. You may also modify the data path for the application (line 59), but it defaults to a subdirectory named for your state, for example data/CA.
3. Each data source feeding the CalCAT has one or more functions in the R folder which pull the data down when the RMarkdown file is run. You can run this daily to ensure your app has the freshest data.
4. Once the data routine markdown has completed, you can follow the output messages to see if any sources returned errors. If so, please submit an issue letting us know what problem you ran into.
5. Next, you can run the app by opening the global.R file and again modifying the state name, now on line 39 (for example state_name <- "Arizona").
6. Run and deploy the app.

Build and Test

The California version of the app, CalCAT, brings together even more publicly available data on cases, deaths, hospitalizations, and bed capacity. Many of these data likely exist in your region. You will need to update these inputs for maximal utility.

Contribute

We can't wait to see what you do with this. Please fork, edit and send us back as pull requests the changes you'd like to see. Maybe you'll incorporate your own estimator of R-Effective, or add functionality to import data; you could improve on our simple ensemble estimates. Go Wild!

A gift from California with love. “Together, all things are possible.” -- Cesar Chavez

Resources

• CalCAT California Version
• covid19.ca.gov

calcat's Issues

Format for submitted data

I'm a member of the ensemble contributors for the weekly CDC death projections and would like to add my model projections to this site. I'm looking for information on the format you want for the data but can't find a description. I've looked at some of the contributed data and there seems to be no consistency to the format of the data and none of the submissions seems similar to the format used for the CDC data. It looks like everyone is using whatever they want. I don't understand how you can create an ensemble forecast without a format for contributed data. Can you explain what you would like to see?

get_epi_forecasts

In the grab_epi_forecasts.R file and line 106 of the Rmd file, the get_epi_forecasts function was throwing an error, saying the URL was bad. I updated to https://raw.githubusercontent.com/epiforecasts/covid-regional/master/united-states/regional-summary/rt.csv and it seems to have resolved the issue

daily updates of can_full_reff_table.csv

Pardon me if this is a newbie question, and I just don't know where to look.

Will the list of county-level and CA-level statistics eventually be updated daily?
It looks like they have not been updated since June 10.
https://github.com/StateOfCalifornia/CalCAT/tree/master/data/CA

The data tables found there extend beyond that date, but it looks like those are projections.
can_state_reff_table.csv
can_full_reff_table.csv

Error when Running the Program to get Data

When I do run all for al_cat_data_routine.rmd, I get the following error on the MOBS section:

mobs <- grab_mobs()
Error: lexical error: invalid char in json text.
saved_old_versions/mobs_New Yor
(right here) ------^

Note: the arrow should be under the a in saved

Is this site still active?

I have not seen anything newer than 2 months ago on this site. Is it no longer in use?

CalCAT Community Meet up This week

this is a test @mccorvie

Completely modeling failure

There is currently a huge covid surge but looking at the graphs at https://skylab4.cdph.ca.gov/calcat/ shows nothing.

Is this a total failure of this model or am I missing something?

Invalid format in data for 5 counties of CA

When running the script for California, I found that the code expected to convert the timeseries JSON element into a dataframe but for 5 of the counties the timeseries element is empty and so the code fails. These counties are:
'06003', '06063', '06105', '06091', '06049'

Note: The actual error message I got was that mutate failed on as.Date. No surprise since it was trying to act on the "date" column and there was no such column!

P.S. It failed getting the OBSERVED intervention. The strong and weak intervention data might be OK.

File within the GA sub-directory do not exist or are non-readable.

CalCAT/global.R

Line 104 in ef1d04f

yu <- fread(paste0(data_path,"/ygg.csv")) %>% mutate(date = as.Date(date))

CalCAT/global.R

Line 130 in ef1d04f

yu.cnty <- fread(paste0(data_path, "ygg_county.csv")) %>% mutate(date = as.Date(date))

both ygg.csv and ygg_county.csv aren't available in the current data files, are these referencing yuganggu.csv and yuganggu_county.csv?

get_counties function error

Error in get_counties() : could not find function "get_counties"

Packagage Mangement?

Just tried to run global.R and need to install a few packages first. I recommend updating to something like the following for ease of use:

If a package is installed, it will be loaded. If any
are not, the missing package(s) will be installed
from CRAN and then loaded.

First specify the packages of interest
packages = c("tidyverse", "geomorph",
"phytools", "viridis")

Now load or install&load all

package.check <- lapply(
packages,
FUN = function(x) {
if (!require(x, character.only = TRUE)) {
install.packages(x, dependencies = TRUE)
library(x, character.only = TRUE)
}
}
)

https://vbaliga.github.io/verify-that-r-packages-are-installed-and-loaded/

Missing open source license

To fulfill the stated objectives of encouraging reuse in other jurisdictions, I recommend adding an explicit open source license to this repo. The MIT License is the most permissive to encourage collaboration and reuse, if so inclined.

https://opensource.org/licenses/MIT

Source for Johns Hopkins, Stanford, etc projections at the county level?

There is some code that contains helpful links to repos that contain the data used in visualization, but some are missing. Would anyone be able to point to where they are; e.g. Johns Hopkins, LEMMA, Stanford? Alternatively, having a source for the original data would be great. Thanks!

Getting error when trying to validate covidactnow URL

FYI I received this error message:

Error: Problem with `mutate()` input `date`.
x do not know how to convert 'date' to class “Date”
i Input `date` is `as.Date(date)`.

on line 74 of cal_cat_data_routine.Rmd:

can_county_observed <- grab_can_county_observed_intervention()

That seems to be because all the loop_can_cnty_observed() results for Massachusetts counties are coming back FALSE so I have no data. For example, valid <- RCurl::url.exists("25017") is returning FALSE on my system. However, if I run

cnty <- jsonlite::fromJSON(url)$timeseries %>% as.data.frame() %>% mutate(date = as.Date(date))
    cnty <- mutate(cnty, fips_char = fips_code, fips = as.numeric(fips_code))

I do get data. So I'm not sure why the valid check is coming back FALSE, although I suspect it has something to do with my system setup. I'm on Windows 10 (I know, I know), R 4.0.0, RCurl 1.98-1.2.

Intro page on phone - UX issue

You don't notice this on desktop, but on phone it's weird.
The , "nowcast" "forecast" "scenario" words all separated from icons. They are not clickable.

grab_can_county_observed_interventions Function

Working through the code to apply to Alaska and noting an issue with the grab_can_county_observed_intervention function. This appears to stem from the loop_can_cnty_observed function where the can.ca.co object returns a null. I can hard code this by specifying each fips code explicitly but the lapply function keeps failing.

Will work with it a bit to come up with my own solution but also reaching out to see if this is a simple oversight on my part.

upon further inspection it appears to be related to the RCurl::url.exists(url) code in the loop function returning false for all values.

I'm still having an issue even after it was closed.

#15

importing data via JSON

Hello, I recently discovered this tool (very cool!) and have just started working with it to see if I can run it with data from Washington state. I've created a fork in the branch and input "Washington" where instructed, but am running into an issue running the cal_cat_data_routine markdown file.

Specifically, in the grab_can_state_observed_intervention formula I'm unable to import the data from data.covidactnow.org (can.ca <- jsonlite::fromJSON(paste0("https://data.covidactnow.org/latest/us/states/",get_state_abbrv(State),".OBSERVED_INTERVENTION.timeseries.json"))$timeseries).
The result is an empty data.frame. Do I need to set up an API with them in order to access these data?

Also as a side question, is there code in your github to recreate your ensemble models for Flu?

Thanks,
Katrin from Public Health Seattle & King County

Highlight for nowcast map is red, which is confusing given the R-eff color scheme

Highlight for nowcast map is red, which is confusing given that R-eff color scheme for the map has red as the highest R-eff category. Shift to heavy outline to indicate county is selected?

Looking for data to run the Shiny App

I'm looking for the following datasets to run the Shiny App.

https://data.chhs.ca.gov/dataset/6882c390-b2d7-4b9a-aefa-2068cee63e47/resource/6cd8d424-dfaa-4bdd-9410-a3d656e1176e/download/covid19data.csv in read_hhs_covid.R

https://raw.githubusercontent.com/nytimes/covid-19-data/master/us-counties.csv and https://raw.githubusercontent.com/CDPHrusers/List-of-US-States/master/states.csv in get_counties.R

ICL duplication

two files read in for ICL but only one is now needed, causing odd display of the statewide nowcast map

Register for CalCAT Community Meetup

We want to gather the community of people working on CalCAT this week. Please register here bit.ly/calcatreg
@adamr-hub
@adannenberg
@AK-MCH-EPI
@anielsen001
@aschl
@baajur
@Bartolac
@briviere
@charada916
@chasemc
@CJE0113
@ClauGISCorona
@colearendt
@DaveParr
@DevenLu
@djd2164
@dnasko
@DrOlivia
@fersalme
@FloatingRootBeer
@GisKid
@guillermodeandajauregui
@hchawla-1
@jasonajones73
@jbayham
@jdh--
@jeffokamoto
@Jenniferha014
@joelrip
@jonnijohnson
@Josteniok
@jpflores-13
@jwilliamsholt
@karlanderson
@kjanevski
@kuanhulio
@lakikowolfe
@LaurenHarriman
@leetcode-notes
@leirbagflow
@MAYHEM-Lab
@mccorvie
@mgermino
@mshigekawa
@natedonnelly
@natekotila
@naterajj
@nayefahmad
@nerfherder987
@nkgithub01
@nturaga
@PaulSaam
@PrincessG
@Qberto
@qizongjun
@rashamad
@rdavidp
@rdavidparker
@restartus
@rkiddy
@sailing0505
@samedelstein
@scallionpancakes
@shalziekay
@skybett
@smach
@srravula1
@StevenLassen
@taqeric
@tdatu
@TJB8526
@tkleykamp
@unchainedbite
@vdtobias
@wehrley
@weikang9009
@WheresHJ

Date Parse Error

I get the following error:

Error: Problem with mutate() input date.
x do not know how to convert 'date' to class “Date”
ℹ Input date is as.Date(date).

The issue is happening in cal_cal_data_routine.rmd:

74 can_county_observed <- grab_can_county_observed_intervention()
fwrite(can_county_observed, paste0(data_path,"can_full_reff_table.csv"))

Testing

I highly recommend adding a code testing framework and tests to this package.

CSV filename is wrong in IHME data

The grab_ihme.R file refers to Hospitalization_all_locs.csv within the ihme-covid19.zip file. It looks like this should actually be Reference_hospitalization_all_locs.csv.

ihme <- grab_ihme(): File does not exist

When the initial script to download data is run: it returns the following:

ihme <- grab_ihme()
trying URL 'https://ihmecovid19storage.blob.core.windows.net/latest/ihme-covid19.zip'
Content type 'application/octet-stream' length 19866543 bytes (18.9 MB)
==================================================
downloaded 18.9 MB

Error in fread(file.path(temp2, paste0(folder, "/Hospitalization_all_locs.csv"))) :
File '/var/folders/kv/nt1xs5dd5637lbxxgg4z40n00000gn/T//RtmpXq2wsJ/file136c8464f30bd/2020_06_24.02/Hospitalization_all_locs.csv' does not exist or is non-readable. getwd()=='/Users/jasonpeskin/Desktop/CalCAT'