Coder Social home page Coder Social logo

go-nexrad's Introduction

go-nexrad: NEXRAD Data Processing with Go

GoDoc Go Report Card Slack license

Go Tools for processing NEXRAD binary data.

Features

  • NEXRAD Level 2 (Archive II Format) Processing
    • Products
      • Reflectivity (REF)
      • Velocity (VEL)
      • Spectrum Width (SW)
      • Correlation Coefficient (RHO)
      • Differential Reflectivity (ZDR)

Sample Image

Reflectivity Radar for Hurricane Harvey after making landfall from the KCRP (Corpus Christy Radar site).

Hurricane Harvey

Resources

go-nexrad's People

Contributors

bwiggs avatar gitter-badger avatar kallsyms avatar phemmer avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

go-nexrad's Issues

Support old L2 format pre 6/1/2016

NOUS41 KWBC 141718
PNSWSH
Technical Implementation Notice 16-12
National Weather Service Headquarters Washington DC
118 PM EDT Thu Apr 14 2016
To: Subscribers
-NOAA Weather Wire Service
-Emergency Managers Weather Information Network
-NOAAPort
Other NWS Partners and Employees
From: Stephen Del Greco
Program Manager, NEXRAD Archive at NOAA’s National
Centers for Environmental Information (NCEI)

Subject: NEXRAD Level-II format at NCEI Archive Changes
Scheduled to Begin June 1, 2016

NCEI is changing the process and format used to ingest and
archive the NEXRAD Level-II Radar Data beginning June 1, 2016.
The new data format is now going to preserve the internal
compression used during the transmission of data to NCEI. This
format is common throughout the Radar community. A variety of
Radar tools and libraries, listed on the NCEI website, support
the format. Supported tools:
https://www.ncdc.noaa.gov/data-access/radar-data/radar-display-
tools

This change only affects Level-II data after June 1, 2016.
There will be no change to historical data.

Overview of upcoming changes:

The naming convention for the archived hourly tar files will
change.

Current example file name:
NWS_NEXRAD_NXL2DP_KILX_20160315020000_20160315025959.tar
New example file name:
NWS_NEXRAD_NXL2DPBL_KILX_20160315020000_20160315025959.tar

The additional "BL" in the product section of the file name
indicates that the volume scan files within the archived tar
files use custom Block BZip2 internal compression.

The naming convention of the volume scan files will also change.

Current example file name: KILX20160315_000554_V06.gz
New example file name: KILX20160315_000554_V06.ar2v

The .ar2v extension of the file name indicates that the volume
scan files within the tar file use custom Block BZip2 internal
compression.

The binary file format:

The internal binary file format will remain "Message 31" as
defined in the Interface Control Documents (ICDs) provided by
the NWS Radar Operations Center (ICD FOR RDA/RPG documents).
http://www.roc.noaa.gov/wsr88d/BuildInfo/Files.aspx
The upcoming change pertains to the collection and storage of
data at NCEI. NCEI receives these data in multiple messages,
each representing a portion of a sweep, and then concatenates
the messages to create the entire volume scan. Each message has
an uncompressed header, followed by bzip2-compressed content.
NCEI is currently decompressing these messages before combining
them into the volume scan file. The volume scan file is
compressed using gzip compression.

After the upcoming change, NCEI will no longer decompress these
messages and will instead concatenate the compressed messages
into the single volume scan file. The volume scan file will not
be recompressed, because the data inside is already received as
bzip2 compressed messages. The Radar Operations Center
documentation describes the transmission method. (ICD for
ARCHIVE II/USER documents).

https://www.weather.gov/media/notification/tins/tin16-12nexrad_format_change.pdf

DataMoment not observing data field size

Currently the values within a DataMoment are exposed as a byte array (DataMoment.Data). On top of this, the ScaledData() method iterates over this array as if it were a slice of uint8 (byte).

This usage is not correct as the data type is not guaranteed to be uint8. Instead the DataWordSize must be used to determine the type. If the value is 16, then DataMoment.Data needs to be unmarshalled as a vector of big endian uint16.

Possible web app, vector rendering, and different radar formats

I've been using this project for a bit and it is very useful. However, there are some changes that I propose and would love to hear feedback on.

First, these are the commands I use to enter the project:

git clone https://github.com/bwiggs/go-nexrad.git
cd go-nexrad && cd cmd && cd nexrad-render

# these two commands are just to download radar files
aws s3 cp --no-sign-request s3://noaa-nexrad-level2/2017/08/25/KCRP/KCRP20170825_235733_V06 .
aws s3 cp --no-sign-request s3://noaa-nexrad-level2/2022/04/18/KLWX/KLWX20220418_153931_V06 .

go get -u golang.org/x/sys

Then, there is an issue with velocity rendering, though reflectivity works fine. I fix it by doing:
in cmd/nexrad-render/main.go, around line 181, replace

// if product != "ref" {
// elv = 2 // uhhh, why did i do this again?
// }
label := fmt.Sprintf("%s %f %s VCP:%d %s %s", ar2.VolumeHeader.ICAO, ar2.ElevationScans[2][0].Header.ElevationAngle, strings.ToUpper(product), ar2.RadarStatus.VolumeCoveragePatternNum, ar2.VolumeHeader.FileName(), ar2.VolumeHeader.Date().Format(time.RFC3339))
render(out, ar2.ElevationScans[elv], label)

with

if product == "vel" {
	elv = 2 // uhhh, why did i do this again?
}
render(out, ar2.ElevationScans[elv], fmt.Sprintf("%s - %s", ar2.VolumeHeader.ICAO, ar2.VolumeHeader.Date()))

this is the commit.
I can then do

go run . -f KCRP20170825_235733_V06 -s 2048 -p vel
# OR
go build
./nexrad-render -f KCRP20170825_235733_V06 -s 2048 -p vel

which works awesome. The first change I propose is wondering what the issue is with the velocity rendering, and why I have to change the code as detailed above?

My second proposal is providing an option, perhaps another command flag, to render the output as a vector image, e.g. svg. You already use the draw2d library with draw2dimg.SaveToPngFile(), so i figured that using vector graphics would make the time loading the image on the user’s end much quicker. I figured you could do draw2dimg.SaveToSvgFile() to accomplish this.

My third proposal is porting this project to a web app. This would mean that one could upload a radar file, specify command flags (e.g. -s 2048 and -p vel), and then it will give you a png image to download, exactly how it runs on the command line.

My first thought was WebAssembly, but I am unsure how to interact with a CLI application such as this one using that framework. I found several guides online, but they are for simple applications where the program just takes user input and then logs the result to the console.

I then stumbled upon this where someone managed to do something very similar with PDF processing. The full repo for that article is found here.

I was wondering if it would be possible to make a rudimentary web application out of your program, using either these things that I have listed, or another way you know of. CSS / styling wouldn't be necessary as this would just be, as I said, rudimentary.

I love your program and will continue to use it, but in the meantime I hope you have time to read and consider these suggestions.

Cheers!
Andrew

Georeferenced CSV Output

@bwiggs Thanks so much for this repository!

I have a specific use case where I’d like to overlay the output of this tool on a street map, but I don’t believe this library does any georeferencing.

I’m considering writing an additional command that outputs a georeferenced CSV for a given product and tilt, I.E.: 39.00, -104.00, {{value}}.

I believe I can do this using existing code and the PROJ library.

I think CSV makes sense as it’s not specific to my use case, and can easily be processed by other tools like GDAL. I’ll add PROJ as an optional dependency and only fail if this specific command is run without PROJ.

Assuming I get this working, would you like me to open a pull request to get this into your repo?

Data Block - unknown type ''

Trying to render a recent radar scan from the same site as your Harvey example i.e. https://noaa-nexrad-level2.s3.amazonaws.com/2021/09/19/KCRP/KCRP20210919_000249_V06 and getting the same failure reading both as a library and via nexrad-render CLI:

go run main.go -f KCRP20210919_002125_V06

Output:

Generating ref from KCRP20210919_002125_V06 -> radar.png
PANI[0000] Data Block - unknown type ''              
panic: (*logrus.Entry) 0xc000146070

goroutine 1 [running]:
github.com/sirupsen/logrus.Entry.log(0xc0001467e0, 0xc0000120f0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

Debugging in, it looks like the m31 header is correct but the DataBlock is corrupt. Let me know if I can provide anything else.

Build 10 / Super Resolution Level II Support (all data pre-2008)

This is not an issue that is super important compared to the issue of supporting data before 2016, because I think it is much more likely people will want to use data before 2016 than they will want to use data before 2008. However, for the sake of thoroughness, I am opening this issue.

I was doing some more research on the history of NEXRAD data, and I came upon this article referenced by a wikipedia page. I realized that this means that there was yet another change in NEXRAD data formatting, this time in 2008 - although there was no set date, that wikipedia page said the changes happed from "March to August 2008".

From what I can gather, it looks like this update was when the MSG31 format was added - and although I know almost nothing about the actual binary structures of these radar files, I do know that message 31 is rather important in their decoding.

All that to say, these pre-2008 files do not work with go-nexrad. Again, this is not as big of an issue as the pre-2016 radar file incompatibility issue was, but I am opining this issue for the sake of thoroughness. Here are the radar files that I am using to test which work with the current go-nexrad project (all of these files are links to their direct downloads):

DOES NOT WORK

WORKS

I noticed that all the files that worked had a V03 on the end of their filename, but the one file with V04 did not work. All files without any V0 do not work, and currently it is V06. Is this relevant at all?

EDIT: I opened KABR20080604_201723_V04.gz in NOAA's Weather and Climate Toolkit and it appears to be either corrupted or a file that does not contain any radar data, if this is helpful at all.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.