dbetchkal / nmsim-python

A Python-based wrapper for Noise Model Simulation (NMSIM). Eases the creation of input files and joining data from GIS databases.

NMSIM-Python

A Python-based wrapper for Noise Model Simulation (NMSIM). Eases the creation of input files and joining data from GIS databases. Improves overall flexibility of the tool for diverse applications.

• Purpose
• Files Associated With NMSIM
  • User-Provided Inputs
  • Intermediary Inputs
  • Outputs
• Site-Based Paradigm Example
  • 1. Initialize an NMSIM project from study area
  • 2. Create a site file (.sit) from metadata
  • 3. Create flight trajectory (.trj) from the overflights database
  • 4. Compile model to create site-specific analysis (.tis)
  • 5. Review the results by comparing to acoustic measurements
• Public domain

Purpose

“The Purpose of a System is What it Does.” -- Stafford Beer, 2001

This repository provides a scripting toolkit for the development of acoustic propagation models with Noise Model Simulation (NMSIM) software. At the request of the National Park Service, NMSIM was updated in 2014 to use the Nord2000 propagation algorithm, "an advanced ray-tracing noise model that includes refraction by the atmosphere and diffraction around terrain." The primary purpose of NMSIM-Python is to sidestep NMSIM's arcane Graphical User Interface (GUI) while continuing to leverage Nord2000's extensive functionality, standardization, speed, and physical accuracy.

The eventual goal of NMSIM-Python is to provide an overarching NMSIM class corresponding to the Control File (see Figure 1, .nms) which describes the parameters for an acoustic propagation model scenario. In this vision, a NMSIM object allows the user to gracefully implement inter-related modelling scenarios. By modifying input parameters, users can generate a specific sequence of acoustic propagation predictions. Such a tool enables the user to iterate over whichever dimension(s) of a civic problem are in question (e.g., properties of the noise source, trajectory, receiver location, etc.). Such focus could greatly expedite mitigation work, ground-truthing or experiments in perception.

For now, though, this repository exists only as:

(1) a set of loosely-connected python modules
(2) a few select Jupyter notebooks
(3) an arcpy toolbox meant for use with ArcGIS Pro

You can think of NMSIM-Python as an “exploded view” of the modelling process, with functions to actualize each input, intermediary, or output filetype used by the software (Figure 1). Current design favors scripters working in a project-driven, maximally-flexible environment. The Jupyter notebooks demonstrate this style of flexible use [note: they are only truly functional for National Park Service employees due to data-sharing issues. Please contact me if you would like to try them - I can figure out how to send you the appropriate data.] For a rich example of NMSIM-Python in use, see Kirby Heck's work with DENA-overflights.

Figure 1.) An "exploded view" the NMSIM modelling process as an information flow graph. The fundamental architecture is colored in blue. Useful output types are colored in amber, their respective raw NMSIM outputs in yellow, and input or intermediary file types in beige. Jupyter notebooks are green. Arcpy toolboxes are red.

The second purpose of NMSIM-Python is to promote records retention for models used in planning or compliance processes. Ideally, public models are publicly available alongside the documents that implement them. Thus, NMSIM-Python aims to support the storage of model objects for future reuse (i.e., as a pickle or other filetype).

Note: NMSIM-Python is a library in active development and therefore should be considered an unstable tool. For the same reason it would greatly benefit from the contributions of open-source programmers. It could also benefit from curious physicists/geographers/ecologists who have an interest in software testing and application. True batching (i.e., flexible compilation of batch files) should also be implemented, but currently isn’t. Batching isn’t purposeful for models containing sequences of isolated events, but as soon as overlapping events feature in a simulation batching is required. However, a batching routine would subsume the NMSIM Class, so writing it will have to wait until the class is available first!

---

Files Associated With NMSIM

NMSIM models represent three coupled systems:

(1) the landscape and atmosphere which
(2) noise sources move over/through
(3) apprehended by the ear that hears: be in animal, human, or metal

Considering the multiphysics involved -- and the RAM-limited era in which NMSIM was originally developed -- there are lots of files required to operate the software. This section attempts to provide a brief list of every file type shown in Figure 1. Such knowledge is key to scripting in NMSIM-Python as well as understanding limitations of the toolkit.

User-Provided Inputs

• NPS Unit Information: the 4-letter alpha code for the park unit in question can provide a quick reference for looking up larger rasters. It also provides a handy lookup for administrative metadata used in automatic labelling/titles/filenames.
• Site metadata: microphone/observer coordinates (WGS84), microphone/observer height (meters).
• Study Area polygon (.shp) usually rectangular, but can be any shape.
• Sound source data represented in the form of one-to-many sound hemispheres (.avg) and their operational metadata (.src) mapped along a control [read: ‘power’] parameter.
• Elevation rasters (16-bit .tif) large regional-scale rasters clipped to within 20 km of park boundaries.
• Impedance raster (16-bit .tif) not implemented!
• OPTIONAL GPS points for creation of flight (or ground) trajectories. The alternative is to use NMSIM’s built-in FlightTrackBuilder.exe module, which is geometrically arcane (and therefore extremely tedious.)

Intermediary Inputs [as facilitated by this library]

• Standardized project directory (or a blank project directory).
• Focused elevation raster (.flt, but also importantly .tif) clipped from wider extent. The grid float file (.flt) will be ingested by NMSIM. The geotiff raster (.tif) is used in scenarios where external GPS data or other covariates are included in the model. Regardless, it is always created for mapping convenience.
• Focused impedance raster (.flt) not implemented!
• Trajectory [read: ‘source positions'] (.trj) which can be from a previously-constructed file or generated from GPS coordinates + elevation raster.
• Site file [read: ‘receiver’] (.sit) microphone/observer position from coordinates and height.
• Source files [read: ‘source’ acoustic properties] (.avg, .src) hemispherical spectral sources and their associated metadata. NOTE: ISN'T SHOWN ON FIGURE 1.
• Control file (.nms) which integrates all NMSIM inputs.
• Batch file (.txt) which allows the model to be run from the Command Line Interface (CLI) program.
• OPTIONAL Weather file (.wea) overrides the standard weather parameters prescribed by ISO 9613-2:1996 which is useful for studying inversions, wind shear, turbulence or other atmospheric effects.

Outputs

• Site-based Model [read: 3D spectrogram representation] (.tis) resulting from control file. One use of these data could be time-aligning model spectrograms with GPS data or acoustic measurements. Further description is outside of the scope of this README, but site-based models provide a much-needed recourse for the validation of field studies or compliance efforts.

• Grid-based Model [read: 4D spectral raster representation] (.tig) resulting from control file. Reducing the dimensionality of the 4D spectral raster into a 2D metric raster is required for mapping (or basically any) purpose. Dimensionality reduction is identical to summarizing any one-third octave band acoustic time-series record with a simpler metric.

---

Site-Based Paradigm Example

The following section demonstrates use of NMSIM-Python to model within a site-based paradigm. In this example there is a single receiver which 'observes' the acoustic morphology of a propeller aircraft as it transits the landscape along a specific trajectory. The model results in a spectrogram (i.e., 2D representation of sound in time and frequency).

Figure 2.) NMSIM modelling via a site-based paradigm. Like Figure 1, the process is represented as an information flow graph. User-based inputs are shown as green arrows on the left margin. These lead to intermediary inputs, which are finally organized together into the fundamental NMSIM Control File for this model. In turn, the Control File is referenced by the Batch File, which is ultimately used by NMSIM.

1. Initialize an NMSIM project from study area

2. Create a site file (.sit) from metadata

3. Create flight trajectory (.trj) from the overflights database

4. Compile model to create site-specific analysis (.tis)

5. Review the results by comparing to acoustic measurements

---

Public domain

This project is in the worldwide public domain. As stated in CONTRIBUTING:

This project is in the public domain within the United States, and copyright and related rights in the work worldwide are waived through the CC0 1.0 Universal public domain dedication.

All contributions to this project will be released under the CC0 dedication. By submitting a pull request, you are agreeing to comply with this waiver of copyright interest.

nmsim-python's Issues

Linear interpolation will not work correctly for heading

In the GPS-to-Trajectory notebook: cell 1, line 281 is currently:
hi = np.linspace(row.heading, data.loc[next_ind, "heading"], interpSteps)[1:-1]

Which will fail, as heading is circular. np.interp accepts a period parameter, which looks like a fine solution.

Source file(s) not pictured in README information flow graph

the title says it all - neither the hemispheric source files (.avg) nor their corresponding metadata (.src) are included on the current information flow graph

Raise MemoryError proactively in certain cases

In NMSIM_InputOutputTools.tbx, it seems appropriate to anticipate that some users may try to create projects so large that NMSIM will crash. This should be handled in advance.

There are two prerequisites:

1. What is NMSIM's memory limit? (Presumably constant.) Convert to a maximum area.
2. How to extract the area of the Study Area Polygon from the Attribute Table using SearchCursor? Should be pretty easy.

A "NMSIM Primer" + "User's Guide" for this Repo

There are somewhat divergent, telescoping use cases for the code in this repository:

• (Simplest) NMSIM baselayer creation + project directory management
• (More Complex) NMSIM .sit from metadata... batch file creation... pairing NMSIM .tis to NVSPL measurements
• (Park-Specific, DENA) Loading GPS tracks; turning these into trajectory files

It is OK to postpone rigorous documentation and 'how-to' until the form of the toolkit has become more obvious. As-is there is obvious value for DENA flight tracks, but it would be nice to develop more a more modular approach where functions could be connected together easily.

Generic GPS Point to Ground Track Function

This isn't a feature of the current tool kit. Since the trajectory files are subtly different between flight and ground tracks, this function should be fairly similar to #6 .

Better parameter descriptions in Arcpy Toolbox

It would be nice to have richer descriptions of the input parameters. I've noticed some tools in ArcPro now have a blue "mouseover" information icon... but I'm not sure how to implement that on my own.

DEM smoothing

One issue with developing naive trajectory files from x,y points involves setting the appropriate z values. (In short: planes don't respect sudden dips in elevation from valleys or canyons.) A 90th percentile smooth of the raster allows more appropriate "above ground level" (AGL) estimations.

Two changes required:

• 1. Update NMSIM_Create_Base_Layers.py to also save a more typical elevation raster (as .tif)

• 2. Then simplify Convert ESRI shapefile (.shp) to NMSIM trajectory (.trj) to automatically smooth the .tif and compute altitudes AGL given only x, y positions. Enhance by allowing point-by-point override of altitude, but also velocity with column names from the .shp file.

Given ambient raster, extend metrics

For the United States, there are existing rasters that estimate summer daytime L50:

Mennitt D, Sherrill K, Fristrup K. A geospatial model of ambient sound pressure levels in the contiguous United States. J Acoust Soc Am. 2014 May;135(5):2746-64. doi: 10.1121/1.4870481. PMID: 24815258.

However, they are very inaccurate (read: ±12 dB) at the spatial scale a noise model is concerned with. Too inaccurate to be of use, unfortunately.

Still, it stands that if a user could supply a meaningful source of (ideally spectral) sound levels across the landscape it would be very interesting and useful to extend the functions in NMSIM grid results (.tig) to raster (.TIF). The following metrics could then be easily added:

• Time Audible (i.e., Duration)
• Increase above natural sound levels

Generic GPS point to Flight Trajectory Function

It would be convenient to have a tool to convert generic aviation-related (x, y, z) GPS points to NMSIM trajectory (.trj) files. Right now tracks_within() has a specific solution when provided with a geopandas.GeoDataFrame.

Shapefiles are no problem to convert using geopandas.read_file(), but feature classes would have to be saved as shapefiles, first. An important part of this work would be assessing whether the incoming points have all/partial/none of these fields:

• Elevation (MSL)
• Datetime
• Heading [could easily be computed]
• Velocity (knots) [could be computed]
• N-Number

Other changes for a general function include:

• Ability to override site parameters with (x, y) point in WGS84?
• NVSPL matching requirement into a boolean parameter.
• Handling UTC offset automatically

Function for creating Site Files (.sit) from Coordinates

Now that the elevation gridfloat files are saved with UTM Zone, converting coordinates to the correct UTM coordinates should be very straightforward.

The only nuance here will be writing in two ways to input the coordinates:

1. directly as parameter(s)
2. from the new NSNSD metadata database [anticipated 2021]

Documentation on all .ipynb functions

All functions need documentation.

Also: we may wish to package the functions in a Python file so they can be imported from other modules?

Streamline raster creation

To streamline NMSIM grid results (.tig) to raster (.TIF) :

• condense functions and import statements into python module (.py)
• provide a means to automate a "suite of metrics" to save out to raster

Computing from .shp, "time_elapsed" is physically incorrect

In Convert an ESRI shapefile (.shp) to a NMSIM trajectory (.trj).ipynb the line

time_elapsed = np.cumsum(velocity/np.array([np.linalg.norm(v) for v in V]))

should really be

time_elapsed = np.cumsum(np.array([np.linalg.norm(v) for v in V])/velocity)

`tracks_within` should handle multiple passes through study area

To avoid making a spectrogram without big gaps of NaN values, GPS points need to be fairly close together... so they are "densified" between successive points with linear (and circular) interpolation. However, the main use case also involves constraining them to the proximity of a study area.

Sometimes (especially at airstrips like Kantishna, Ruth Glacier, INR, etc) aircraft will depart and then return with the same flight_id - and tracks_within faces a big disjunction in time between points. In the best case scenario this will (1) slow down model runs, and (2) dilute time-dependent metrics like Percentile Leq, 1s and Percent Time Above. In a worst-case scenario, the airplane will (3) jump from perimeter point (x_t, y_t) to perimeter point (x_t+n, y_t+n) such that the resulting "imaginary" segment intersects the study area. In this case, even solely energy-based metrics (maximum LAeq, 1s) are contaminated.

Most solutions are awkward, so I'm just marking this as a bug for now. At the very least, densification over a certain time step should be eliminated!