usgs / earthquake-impact-utils

Utilities to support USGS earthquake shaking and impacts software, such as ShakeMap, PAGER, and groundfailure.

earthquake-impact-utils

Introduction

This project contains functionality that is common to at least six related repositories:

• ShakeMap: https://github.com/usgs/shakemap
• PAGER: https://github.com/usgs/pager
• groundfailure: https://github.com/usgs/groundfailure
• ShakeMap scenarios: https://github.com/usgs/shakemap-scenarios
• ps2ff: https://github.com/usgs/ps2ff
• shakelib: https://github.com/usgs/shakelib

Installation

The install.sh script installs this package and all dependencies. It is regularly tested on OSX and Ubuntu.

earthquake-impact-utils's Issues

Add support for arbitrary spectral periods to dataframe_to_xml()

ShakeMap will soon be getting other spectral periods from NSMP, so we need to be able to convert these to shakemap format.

pdlsender will fail if no product properties are set

Perhaps other senders...

Factor out openquake

Describe the bug
Openquake GMPEs are being used, but the dependency is too large for this base level library. Move use of openquake usage to a higher level repository.

To Reproduce
N.A.

Expected behavior
N.A.

Screenshots
N.A.

Environment (please complete the following information):
All

Additional context
Examine whether factoring out this dependency is even possible, since we are currently using GMPEs.

Add documentation

Is your process request related to testing, documentation, or general code management? Please describe.
Add documentation.

Describe the solution you'd like
Add a docs folder similar to: https://github.com/usgs/libcomcat/tree/master/docs

Describe alternatives you've considered
N.A.

Additional context
N.A.

Handle argument types in cancel() method of pdlsender

The send() method checks the input types and handles them accordingly, replicate that logic.

Future warning on pandas frame.append

Describe the bug
/Users/cbworden/miniconda/envs/shakemap/lib/python3.8/site-packages/impactutils/mapping/city.py:329: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
newdf = newdf.append(tdf)

Fix docstring for getTsunami method in query module

Sync repository

Is your process request related to testing, documentation, or general code management? Please describe.
earthquake-impact-utils should be backed up to GitLab.

Describe the solution you'd like
https://docs.gitlab.com/ee/user/project/repository/repository_mirroring.html

Describe alternatives you've considered
N.A.

Additional context
N.A.

Update string formatting

Is your feature request related to a problem? Please describe.
Upgrade string formatting.

Describe the solution you'd like
Change formating from %-formatting to f-string: https://www.python.org/dev/peps/pep-0498/

Describe alternatives you've considered
N.A.

Additional context
N.A.

Verify that versioneer is working correctly

Is your process request related to testing, documentation, or general code management? Please describe.
Verify versioneer.

Describe the solution you'd like
Add tests?

Describe alternatives you've considered
N.A.

Additional context
N.A.

Map scale bar

Cartopy doesn't have a scale bar, so we need a class for drawing scale bars.

Map scalebar has incorrect length

The scalebars, e.g. in Shakemap, are shown at incorrect lengths on the map if the location is not at the Equator. For a simple scenario in Alaska at latitude 66 deg, the scalebar shows 150 km when it really is about 60 km long based on the map extent and grid lines. For an event in Equador at latitude 0 deg, the scalebar has the correct length.

This seems to affect also the example images in the Shakemap Tutorial etc.

To reproduce it, I created some test Shakemaps with simple event.xml input files:

<earthquake id="scenario1" netid="TEST" network="TEST" lat="66.0" lon="-161.5" depth="4"
 mag="4.5" time="2020-08-27T12:00:00.000000Z" locstring="Alaska"/>

<earthquake id="scenario1" netid="TEST" network="TEST" lat="0.0" lon="-80.0" depth="4"
 mag="4.5" time="2020-08-27T12:00:00.000000Z" locstring="Equador"/>

See the images below. I've also drawn thick red lines to indicate the actually expected length for the given scalebar using GIMP.

Versions

• OS: Ubuntu 18.04
• Python 3.7
• Impact Utils 0.8.23
• Numpy 1.18.5
• Cartopy 0.18.0

Update testing

Is your process request related to testing, documentation, or general code management? Please describe.
Update testing form Travis to Azure.

Describe the solution you'd like
Should be similar to: https://github.com/usgs/libcomcat/blob/master/azure-pipelines.yml

Describe alternatives you've considered
N.A.

Additional context
N.A.

Increase test coverage

Is your process request related to testing, documentation, or general code management? Please describe.
Test coverage is too low.

Describe the solution you'd like
Increase test coverage with mock and pytest.

Describe alternatives you've considered
N.A.

Additional context
Anything network oriented should use mock or vcrpy.

Publish library to pypi

Is it possible to publish this library to pypi by adding requirements to setup.py ?

Move ComCatInfo to Pager

Describe the bug
ComCatInfo is being used by Pager. Since most of this functionality is already available in libcomcat, it should be removed. Temporarily move ComCatInfo into Pager while it is being removed from impact-utils.

Add deprecation warning.

To Reproduce
N.A.

Expected behavior
N.A.

Screenshots
N.A.

Environmnet (please complete the following information):
All

Additional context
N.A.

Remove duplicate code

Describe the bug
Example: The Cities class is in impact-utils and MapIO.

To Reproduce
N.A.

Expected behavior
Cities should only be in one location.

Screenshots
N.A.

Environment (please complete the following information):
All

Additional context
Check Pager, Groundfailure, and Shakemap to determine which dependency is used.

Refactor pdlsender class to not have duplicate code in send/cancel methods.

Describe the bug
Too much duplicated code between send and cancel methods.

set_num_precision bug

The set_num_precision method seems to have bugs. The following should give 230:

set_num_precision(231, 2, 'float')

but it gives 229.99999999999997. Similarly:

set_num_precision(221, 2, 'float')

gives 220.00000000000003. This is a problem for ground failure because we are using this method to round the aggregate hazard values. One thought might be that the solution would be to change the mode to "int":

set_num_precision(231, 2, 'int')

but that gives 229, which is different and less offensive in the number of digits reported but equally incorrect. Also, we originally set the mode to float because we'd like it to be able to handle something like:

set_num_precision(0.32113, 2, 'float')

but it gives 0.3 when it should be 0.32.

Cities missing on maps at left and bottom

City names plotted on maps closest to the left and lower map border can be missing if the number of plotted cities does not need to be thinned out, i.e. if few cities are within the mapped area.

The reason seem to be two ">" which should rather be ">=" in impactutils/mapping/city.py, lines 232, 233:

        newdf = newdf.loc[(newdf['lat'] > ymin) & (newdf['lat'] <= ymax) &
                          (newdf['lon'] > xmin) & (newdf['lon'] <= xmax)]

should be:

        newdf = newdf.loc[(newdf['lat'] >= ymin) & (newdf['lat'] <= ymax) &
                          (newdf['lon'] >= xmin) & (newdf['lon'] <= xmax)]

Using Shakemap for an example scenario (MMI threshold 4.0):

<earthquake id="scenario5" netid="TEST" network="TEST" lat="56.4" lon="162.2"
 depth="10" mag="5.0" time="2020-08-27T12:00:00.000000Z" locstring="Kamchatka"/>

Without the fix, no cities are shown even though I expected at least two:

With the fix, two small towns come up:

Check Dependencies

Is your process request related to testing, documentation, or general code management? Please describe.
The dependencies need to be checked to ensure that all those listed in the package_list are required

Describe the solution you'd like
Remove unused dependencies.

Describe alternatives you've considered
N.A.

Additional context
N.A.

GeoServe.getRegions uses "pop()" method

The pop() method removes the item, so subsequent calls to getRegions on the same object will not get anything. If self._regdict is a dictionary, I would suggest using the "get()" method, which should provide the same error handling (i.e., can return None if the key isn't there).

Update timezone functionality

Is your feature request related to a problem? Please describe.
Currently a shapefile is used to determine timezone offsets. This fiona as a dependency.

Describe the solution you'd like
Replace timezone shapefile with timezone raster (tif) file.

Describe alternatives you've considered
A geojson was considered, but the file size is too large for this repository.

Additional context
N.A.

Update Exceptions

Is your feature request related to a problem? Please describe.
Exceptions need to be updated.

Describe the solution you'd like
Exceptions should be similar to those setup in libcomcat: https://github.com/usgs/libcomcat/blob/9353bbd83d924bcb79c0b2cbe29ee72501514e8f/libcomcat/exceptions.py

Describe alternatives you've considered
N.A.

Additional context
N.A.

make sure pdl log is written someplace reasonable...

...instead of in the current directory.

Modify table format for dataframe_to_xml method

Here is a tabular (Excel, CSV) format for capturing both intensity data and instrumented ground motion data.

Column Name	Description	Required?
STATION	Station code	Y
NAME	Station name/description	N
SOURCE	Source of data	Y
NETID	Network ID contributing data	Y
LAT	Latitude of station or location of data	Y
ELEV	Elevation (m) of station or location of data	N
INTENSITY	MMI or equivalent scale data	N
INTENSITY_RESP	Number of intensity responses at that station	N (required if INTENSITY is present)
INTENSITY_SD	Standard deviation of intensity responses	N (required if INTENSITY is present)
[IMC]_[IMT]	Any number of columns consisting of data for a given IMC and IMT.	N

Table of IMC abbreviations:

IMC	Abbreviation
Arithmetic Mean	AM
Geometric Mean Rotation Degrees (GMRotd)	GMR
Quadratic Mean	QM
RotdXX	ROT
Geometric Mean	GM
Channel	(HNE, H1, Z, etc.)
Radial Transverse	RT
Greater of Two Horizontals	GTH

Make container class more bulletproof

setString(name,string) method in container class to check to see if string of name already exists in the container and raise an Exception if it does. Similar behavior should be implemented for all of the setter methods.

Examine where the Container classes are used

Check Shakemap, Pager, groundfailure, etc. for the container class as a dependency.

Make container class self-packing

After dropping a data set or group from HDFContainer, we should look into making the file smaller by using a repack tool, as described here:

https://stackoverflow.com/questions/11194927/deleting-information-from-an-hdf5-file