doi-usgs / hasp

Hydrologic AnalySis Package. See official source code: https://doi-usgs.github.io/HASP/

HASP: Hydrologic AnalySis Package

Hydrologic AnalySis Package

The Hydrologic Analysis Package (HASP) can retrieve groundwater level and groundwater quality data, aggregate these data, plot them, and generate basic statistics. One of the benefits of HASP is its ability to aggregate two time-series of data into one record and generate statistics and graphics on that record. By merging two data sets together, users can view and manipulate a much longer record of data. Users can download the aggregated record and see basic statistics that have been calculated with these data. HASP also allows users to plot groundwater level trends in major aquifers as well. The explore_aquifers function allows users to pull data from wells classified in Principal Aquifers and synthesize water-level data to better understand trends.

Inspiration: https://fl.water.usgs.gov/mapper/

See https://rconnect.usgs.gov/HASP_docs/ for more information!

Installation of HASP

You can install the HASP package using the remotes package.

To install the remotes package, copy the following line into R or the “Console” window in RStudio:

install.packages("remotes")

To install the HASP package:

remotes::install_gitlab("water/stats/hasp",
                        host = "code.usgs.gov",
                        build_opts = c("--no-resave-data",
                                       "--no-manual"),
                        build_vignettes = TRUE, 
                        dependencies = TRUE)

During this installation, you may be prompted to update or install some packages. Press 3 to skip the updates, but don’t forget to update all of your packages later.

Sample workflow

Get groundwater data:

library(HASP)
library(dataRetrieval)
site <- "263819081585801"

#Field GWL data:
gwl_data <- dataRetrieval::readNWISgwl(site)

# Daily data:
parameterCd <- "62610"
statCd <- "00001"
dv <- dataRetrieval::readNWISdv(site,
                                parameterCd,
                                statCd = statCd)

Plot data:

y_axis_label <- readNWISpCode(parameterCd)$parameter_nm
title <- readNWISsite(site)$station_nm

monthly_frequency_plot(dv,
                       gwl_data,
                       parameter_cd = parameterCd,
                       plot_title = title,
                       y_axis_label = y_axis_label)

weekly_frequency_plot(dv, 
                      gwl_data,
                      parameter_cd = parameterCd, 
                      stat_cd = statCd,
                      plot_title = title,
                      y_axis_label = y_axis_label)

daily_gwl_plot(dv, 
               gwl_data,
               parameter_cd = parameterCd,
               plot_title = title,
               stat_cd = statCd,
               y_axis_label = y_axis_label)

gwl_plot_all(gw_level_dv = dv, 
             gwl_data = gwl_data, 
             parameter_cd = parameterCd,
             plot_title = title, 
             add_trend = TRUE)

Create Groundwater Report:

create_groundwater_report(siteID = site,
                          report_name = "L_2701",
                          report_folder = "reports",
                          output_type = "html")

Running this function will create an Rmarkdown file (Rmd). In RStudio, clicking the button “Knit” at the top of the file will render the Rmd file as either an HTML or Word document.

Water-quality plots

HASP also includes a few water-quality plot options.

# Water Quality data:

qw_data <- readWQPqw(paste0("USGS-", site),
                     c("Chloride",
                       "Specific conductance"))

qw_plot(qw_data, CharacteristicName = "Chloride",
        plot_title = title)

Sc_Cl_plot(qw_data, plot_title = title)

Composite workflows:

More information on the composite aquifer workflow can be found vignette("Priniciple_Aquifers", package = "HASP").

#included sample data:

aquifer_data <- aquifer_data
num_years <- 30

plot_composite_data(aquifer_data, num_years)

plot_normalized_data(aquifer_data, num_years)

Citing HASP

citation(package = "HASP")
#> To cite HASP in publications, please use:
#> 
#>   DeCicco, L.A., Prinos, S.T., Eslick-Huff, P.J., Hopkins, C.B., 2022,
#>   HASP: Hydrologic AnalySis Package, v1.0.0.,
#>   https://code.usgs.gov/water/stats/hasp, doi:10.5066/P9BUN5GV
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     author = {Laura DeCicco and Scott Prinos and Patrick Eslick-Huff and Candice Hopkins and Tara Root},
#>     title = {HASP: Hydrologic AnalySis Package},
#>     publisher = {U.S. Geological Survey},
#>     address = {Reston, VA},
#>     version = {1.0.0},
#>     institution = {U.S. Geological Survey},
#>     year = {2022},
#>     doi = {10.5066/P9BUN5GV},
#>     url = {https://code.usgs.gov/water/stats/hasp},
#>   }

Installation of R and RStudio

To use the HASP package, you will need to have R and RStudio installed on your computer. This installation will only need to be done once for each computer. This link has instructions for installing R and RStudio:

R and RStudio Installation Instructions

Useful links:

• Download R Windows
• Download R Mac
• Download RStudio

Running the apps

HASP includes two interactive applications as a way to explore the functionality of this package. One has functions for exploring data from a single site, and the other has functions for exploring data from an aquifer.

For more information see vignette("shinyApp", package = "HASP").

To run the single site application, use the following code:

HASP::explore_site()

To run the aquifer application use the following code:

HASP::explore_aquifers()

Disclaimer

This software is preliminary or provisional and is subject to revision. It is being provided to meet the need for timely best science. The software has not received final approval by the U.S. Geological Survey (USGS). No warranty, expressed or implied, is made by the USGS or the U.S. Government as to the functionality of the software and related material nor shall the fact of release constitute any such warranty. The software is provided on the condition that neither the USGS nor the U.S. Government shall be held liable for any damages resulting from the authorized or unauthorized use of the software.

Any use of trade, firm, or product names is for descriptive purposes only and does not imply endorsement by the U.S. Government.

hasp's Issues

Long load time for explore_site() for sites with no daily data

Reported by Virginia McGuire:

When trying to load data in explroe_site() for sites with no daily data, user experienced very long load time (10 minutes). Example site used that caused the delay is 375351097295901

Add message to Shiny App when retrieving data for a new site

The Shiny App is now successfully allowing users to pull data for a site and switch between sites. However, when a new site ID is pasted, it appears as if there is no data and/or stale data from the previous site. This may be an artifact of my network connection, but I don't want users to think the app isn't working when it is in fact just loading data.

Median in explanation is a diamond symbol, while a triangle shows on the plot

The explanation has a diamond symbol, while a triangle shows on he plot.

library(HASP)
library(dataRetrieval)
site <- "424520070562401"
name <- (readNWISsite(site))$station_nm
dv <- dataRetrieval::readNWISdv(site,
                                "72019",
                                statCd = "00003")
monthly_frequency_plot(dv,
                       parameter_cd = "72019",
                       plot_title = name,
                       y_axis_label = y_axis_label, flip_y = T)

From the above code, what do you expect the output to be?

Remove readNWISqw examples

Replace with readWQPqw

Error generated when trying to flip y-axis on "groundwater" hydrograph

In the explore_sites() package, when a user tries to set the "flip y-axis" to FALSE on the "Groundwater" (line) plot, they get an error message: DIscrete value supplied to continuous scale.

I was able to generate this error with data from the example site

Get rid of all sl_lev_va vs lev_va

As long as we introduce pcode to this app, we can get rid of all the complexities of what column the data is coming in.

Flip axis on hydrograph

For groundwater levels, we invert the axis of the hydrograph so that low values (like 0) are at the top of the graph and high values (like 250) are at the bottom. This is to help visualize "low" water conditions are actually high numbers. Therefore, I'd like to recommend flipping the default y-axis on the groundwater hydrograph plots.

Naming inconsistencies

We use two different names for discrete (hand-measured) data: field groundwater-level measurements (in the data type summary) and periodic data (in the data pull options). We need to make those names consistent one way or another so that users understand that these are the same measurements.

Code out - shiny app

The "code_out" stuff generally needs some work. We should be able to copy/paste the code below each plot and get the exact same output as what's shown in the app. I think in my latest PR the "gwl_plot.R" code for the "Daily" tab is working. We need to do a similar update for the other groundwater tabs.

Daily min and max repeating

On the Daily Plot, the Daily Min and Max are not in fact calculated min and max values for each day. They appear to be historical min and max values, as evidenced by the fact that they repeat each year. Either we need to change the label for what these values are, or figure out why we're seeing the wrong data as min and max.

Chloride and SC axes reversed in Sc_Cl_plot

The axis labels on the SC_Clplot seem to be reversed. This is noticed, because SC is typically > Cl, which is not the case for plots made with the function. This appears to be the case with other examples using the Shiny app. By this, I mean that the x axis should be specific conductance, and the y axis should be chloride.

library(HASP)
site <- "01193500"
parameterCd <- c("00095","90095","00940")
qw_data <- dataRetrieval::readNWISqw(site, parameterCd)
plot_title <- paste(attr(qw_data, "siteInfo")[["station_nm"]], ": Chloride")
trend_plot(qw_data, plot_title)

From the above code, what do you expect the output to be?

If possible, add labels to pcodes and stat codes in Shiny App

It would be helpful to add the definition of the pcode or stat code so that users understand what data they are pulling. The data for site 374004122092102 varies greatly based on what stat code is being displayed and it would be helpful for a user to understand what these codes mean.

Add y-axis labels to Weekly Frequency, Daily Plot, and Monthly Plot

Add text labels to y-axes for these plots so that the user understands plotted data

non-detects

With the switch to WQP, we're not plotting any values for non-detects. I think that's OK, but when there's NO detections, there's a weird empty plot.

We should probably add some text with the number of non-detects. We could also plot a bar or something like EGRET with the detection limits, but I don't consider that important for now (especially since this tends to be more of a groundwater level project...maybe move water quality plots to another project?)

Revisit dependencies

After we get more functions put in, let's revisit dependencies.

Dependencies in main functions:

• dplyr
• purrr
• tidyr
• lubridate
• EnvStats
• dataRetrieval
• ggplot2
• ggpmisc
• scales

Dependencies exclusively for shiny app(s) (could move to suggest in some fancy way):

• shiny
• shinyAce
• leaflet
• shiny
• shinycssloaders
• shinydashboard

Fix spelling

For aquifers, it's spelled "Principal" not "Principle"

Small bug, sorry!

Auto choose lev_va vs sl_lev_va

(copying over Issues from gitlab)
For the plots with discrete gwl data, there's probably a way to check which column to use automatically. Right now on the app...if the wrong one is picked, there's an error (see site 433641110441501)

Help define pcodes offered in qwdata plotting options

It appears that the "Get WQ Data" shows users any pcode that has non-daily values. This allows users to plot some strange non-numeric pcodes (such as Analyzing Agency), but that's a burden we can leave on the user for now.

It would be beneficial to somehow allow the pcodes to be defined so users understand what they are asking for. One option would be to add a link somewhere in the Shiny app with a link to the pcode definitions page: https://nwis.waterdata.usgs.gov/usa/nwis/pmcodes

Another option would be to spell them out, but this may be complicated for sites with lots of discrete data

Add Explanation/Legend to Map

A user requested that we add an explanation or legend to the map in the explore_aquifer() app. Hopefully this would be straightforward, and be as simple as "site used in composite hydrograph"

Normalized composite plot

https://groundwaterwatch.usgs.gov/composite/help/CompositeGroundwaterLevelHelpDocument.docx.html

As found by a user, I think the mean, max, min are using all the data instead of the annual medians in the normalized aquifer data function.
(also, check that that link is included in the help file)

Add slope to chloride trends

A user suggested that we add a non-parametric slope (Mann-Kendall and Sen) to the trend plots. That way, a user can see both the magnitude and significance of trends.

If there's no SC/Ch...put up a big message

Similarly, big messages if there's no dv

Flip the percentages

Percentages should be flipped:

NOTICE: upcoming default branch name change

The master branch of this repository will soon be renamed from master to main, as part of a coordinated change across the USGS-R and USGS-VIZLAB organizations. This is part of a broader effort across the git community to use more inclusive language. For instance, git, GitHub, and GitLab have all changed or are in the process of changing their default branch name.

We will make this change early in the week of February 6, 2022. The purpose of this issue is to give notification of the change and provide information on how to make it go smoothly.

When this change is made, forks and clones of the repository will need to be updated as well. Here are instructions for making the necessary changes.

• On GitHub, change the default branch name from master to main (<your repository> -> Settings -> Branches).
• Change the default branch on any forks you have, as well as local clones. (See details below)
• If you have collaborators on this repository, let them know that they will need to change their forked/local repos. Point them to this issue to facilitate the process!
• Search within your repository for "master" so that you can change references (e.g. URLs pointing to specific commit points of files) to point to "main" instead.
• When you are done, feel free to close this issue!

Changing default branches on forks and local clones

First, update your local git configuration so that new repositories created locally will have the correct default branch: git config --global init.defaultBranch main.

Now, for any forks, do the following:

1. Go to <your fork> -> Settings -> Branches and edit the default branch from master to main.
2. Update the settings for your local clone of this fork to match this change.

git branch -m master main
git fetch origin
git branch -u origin/main main
git remote set-head origin -a

Add message when map is loading

Add a message to allow users to understand that a map is loading in the explore_aquifer() function. There currently is no message, so it looks like the map is not loading until it suddenly pops up.

Site min/max identified in period of record

A user commented that they spend a lot of time in GWW manually checking pages to see if the site's minimum or maximum value is within a period of record (in this case, the last fiscal year).

Since this package is already calculating and displaying the min and max for a site, it would be nice to build in a query where a user could ask if the min or max is within a specified time period. Ideally, the user would specify the time period so they could see if a min/max for the site was recorded in the last year, 5 years, etc.

I imagine this would be done in R itself, not the Shiny application

Add daily water-level pcodes

In the shiny application, allow user to retrieve additional pcodes that often contain daily measurements. These are:
30210
72019
62600
62601
62611
62612
62613
72150
72227
72229
72230
72231
72232
72292
99019

Up vs down

However, in this example graph from the HASP GITHUB website the y-axis has been flipped upside down. In the case of water level elevation the y-Axis should go from lessor values on the bottom to greater values on the top. The Y-Axis scale for depth to water is flipped relative to water level because the larger the value the lower the water level.

In the graph for L-2701 from the existing groundwater watch website you can see that for groundwater level elevation the minimum y-value is at the bottom of the graph. However you can also see that groundwater watch never handled negative values right in regards to the bar graphs of percentiles.

Therefore I included below another example from groundwater watch for well L-738. You can see that for both of the L-738 graphs (below) higher water levels elevations are at the top of the graph, but a second y-axis has been added to the one of the graphs to also show depth to water.

Unlike Florida much of the nation only provides depth to water measurements so this graph the primary axis is the depth to water, and the secondary axis is water level elevation. I did not need to worry about that for Florida for which water level elevation is far more important than depth to water, but it is something that would have to be considered in a national app.

I think this is one thing that groundwater watch got right, up is up. Down is down. No matter how you cut it that is the way the graph should be shown, the only difference should be that the depth to water axis would increase downwards because greater depths to water mean lower water level elevations. I don't think it was this way in April when I looked at it. The last graph copied below from from April email.

Trouble loading data from sites with many pcodes

The Shiny App seems to crash with sites with many pcodes/an extensive QW sampling history. The app stops working and R Studio gives this error message:
Warning: Error in constructNWISURL: Maximum parameter codes allowed is 200, please adjust data request.

This issue was found at site 433910112550101

Chart titles in Shiny App display example data site number

It'd be great if the charts in the Shiny app would automatically display titles that indicate the site number they are displaying data for

If possible, allow axes on plots to auto-scale

Groundwater levels at some sites are very far from zero, so it would be ideal for the y-axis on plots to autoscale so that a user can see the full range of the hydrograph with more detail. An example site where the details of the hydrograph are masked a bit by the scaling is site 434307112382601.

Create documentation or instructions on how to walk through loading and exploring new sites

A user commented that it is confusing how to walk through the explore_site() application and doesn't understand when to load data and how to click through the various graphs.

Please create documentation showing how to load data from sites and use the various data loading options to then explore the plots generated. This package may be used broadly by the general public, so it would be nice to somehow document "first X, then Y" type steps to explore a site.

Change data point labels in explanation

On the monthly plot in the explore_site() application, the "Data Point" and "Monthly Median" seem to be flipped in the explanation. I believe the monthly median should be the black point, whereas the data point should be in red.

Weekly and Monthly plots not generating

For the site 313738110102901, the monthly and daily plots do not seem to be rendering (in the Shiny app). An error code appears "Error: replacement has 2 rows, data has 1"

This error does not make sense to me as a user. In R, there's another error "Total data time span is less than 20 years". If this is the reason for the lack of plots, we need to make that more clear for users.