niehs / brassens

Bias-corrected Recordings from Air temperature Sensors to Support EnviroNmental Sciences

brassens

The development of this library is in progress.

A package to deal with Citizen Weather Stations temperature data

• calls to PurpleAir API from R session to recover timeseries of temperature on a given spatial area
• formatting data
• cleaning non-conventional measurements of temperature with statistical tools (CrowdQC+ library)
• calibration with reference stations (GHCNh dataset used by default)

Pipeline tutorial

Prepare all parameters

pa_file parameter can be NULL: in this case, you need to provide an API key to download data from PurpleAir API to load_pa(). wu_inv refers to an inventory of WeatherUnderground files in your local machine. Once you have WU files, you can create this inventory by using create_wu_inventory() function.

> config <- list(
  ts = as.POSIXct("2021-07-22 00:00:00", tz = "UTC"),
  te = as.POSIXct("2021-07-23 23:59:59", tz = "UTC"),
  area = your_polygon,
  wu_inv = your_wu_inventory,
  pa_file = path_to_your_pa_file
)

Load GHCNh data

This reference dataset is used for calibration step. You can also use another reference network for calibration.

ghcnh <- download_ghcnh(config$ts, config$te, config$area)

Open and process citizen weather stations from WeatherUnderground and PurpleAir.

You can tune maximum distance.

wu_list <- load_wu(config$ts, config$te, config$area, config$wu_inv) |>
  format_wu() |>
  clean_cws() |>
  calib_cws(ref = ghcnh, max_dist = 20000)

pa_list <- load_pa(
  ts = config$ts,
  te = config$te,
  area = config$area,
  storage_file = config$pa_file
) |>
  format_pa() |>
  clean_cws() |>
  calib_cws(ref = ghcnh, max_dist = 20000)

References

CrowdQC+ library
Fenner, D., Bechtel, B., Demuzere, M., Kittner, J. and Meier, F. (2021): CrowdQC+ – A quality-control for crowdsourced air-temperature observations enabling world-wide urban climate applications. Frontiers in Environmental Science 9: 720747. DOI: 10.3389/fenvs.2021.720747.

Global Historical Climatology Network - hourly
Menne, Matthew J.; Noone, Simon; Casey, Nancy W.; Dunn, Robert H.; McNeill, Shelley; Kantor, Diana; Thorne, Peter W.; Orcutt, Karen; Cunningham, Sam; Risavi, Nicholas. 2023. Global Historical Climatology Network-Hourly (GHCNh). NOAA National Centers for Environmental Information. [accessed on 2024/06/12]

PurpleAir data download functions adapted from
Callahan J, Martin H, Wilson K, Brasel T, Miller H (2023). AirSensor: Process and Display Data from Air Quality Sensors. R package version 1.1.1, https://CRAN.R-project.org/package=AirSensor.

PurpleAir: https://www2.purpleair.com
WeatherUnderground: https://www.wunderground.com

brassens's Issues

Credits

Mention credit to those libraries / data:

• script to download PA from r session?
• CrowdQC+
• PurpleAir
• WeatherUnderground
• GHCNh

Add "| MULTIPOLYGON" to find_ghcnh_polygon() line 8

format_area()

The goal of this function is to facilitate the function download_...(ts, te, area, ...) so that area...

• is a Polygon
• if not, return result of sf::st_bbox().

calib_cws()

Create a function calib_cws(x, ref) to remove systematic bias in the cws dataset.
Default ref = output from download_ghcnh() but users will be able to add its own network.
Note: systematic bias varies across the hours of the day, and also across the seasons. So it might be relevant to create a moving average to remove it.

load_wu()

• Rename from load_wu_period()
• It would be great to create a script (bash?) to generate a index table of files sent by Andrew (with station location + ts + te) so that it's gonna be easier to load data from ts, te, area. Transferred to issue #13.
• modify load_wu() so that it takes ts, te, area arguments only.

download_pa()

• Be sure download_pa() takes in parameters ts, te, area

Create clean vignettes

Clean plots functions

Create test data

• wu fake sample of 1 week on the triangle
• pa fake sample of 1 week on the triangle

WeatherUnderground inventory

Create a script to generate a index table of files sent by Andrew (with station location + ts + te) so that it's gonna be easier to load data from ts, te, area.

Write all roxygen

download_ghcnh()

• From existing functions, create a new one: download_ghcnh(ts, te, area).
• Rename load_...() functions in download_...()?

Test CrowdQC+ on tn and tx (instead of hourly temp)

Test all functions

• format_area
• convert_temp
• my_pal
• find_ghcnh_polygon
• find_nearest_ghcnh
• download_ghcnh_station
• download_ghcnh
• summarize_hourly_temp
• hourly_temp
• generate_site_id
• load_pa
• format_pa
• format_wu
• format_ghcnh
• manage_na
• clean_cws
• cut_area
• find_closest_ref
• calc_temp_error
• calib_cws

Hard to test

• find_sensors
• request_sensor_history
• request_sensors_history
• download_pa
• create_wu_inventory
• load_wu (need wu inventory but the testdata is fake, with fake stationID)
• clean_cws_large (but I tested the new function "cut_area"
• all plot functions

load_pa()

In order to avoid downloading the same data many time:

load_pa <- function(ts, te, area, file) {
- if is.null(file) then download_pa(ts, te, area) 
- else open and load corresponding data
}

merge()

Very easy function to return a merged dataset with clean and calibrated observations from both datasources.

est_error_prior()

• calc_error()
• fit a (loggamma?) distribution to error histogram
• estimate (alpha and beta) parameters

Keep only reliable GHCNH