unidata / ldm-alchemy

Python tools to handle LDM data feeds and convert to other formats.

Python LDM feed handler

Python code to facilitate on-the-fly processing of products coming from an LDM feed.

Currently, this project is only used to handle the NEXRAD Level 2 data feed coming across the LDM data feed, with options to dump the raw data or repackage as a bz2 or gz compressed file. There are also options to upload to Amazon S3.

The core of this is based around Python 3.5's new async support, using the asyncio library. As such, only Python 3.5 will ever work here. One of the core features, though, is that a single python process is able to handle and process (using multiple threads) the full level 2 stream.

There are no supported APIs here yet, as right now this is just a monotlithic script. Eventually, this same infrastructure will be used for other LDM data feeds, at which point useful API separation/abstraction will present itself.

Requirements

• Python >= 3.5
• netCDF4-python (for goes-restitch.py)
• psutil (for process-monitor.py)

ldm-alchemy's Issues

l2assemble backlog

l2assemble has no problem chugging through a backlog of data when it's writing it to disk. When it starts uploading things to S3, it falls behind pretty quickly. What I noticed was:

• It didn't seem like volumes were getting written to S3
• Chunks were falling behind
• Memory would exhaust an 8G machine

Seems like volumes are getting deprioritiezed somehow behind upload the chunks, so that the volumes stay in memory and eventually exhaust memory. Something to investigate...

l2assemble enhancements

• Handle re-issued volumes better
  • The loading from S3 problem
  • Validate the fix deployed to production is the right one
• Add logging for that case
• Start tracking latency of volumes (look at difference between current time and time of first chunk)
• Gracefully handle S3 write failure?
  • Maybe cache volume to disk?

Issue with GOES Full Disk

I am using the latest alchemy repo and am encountering an issue assembling the Full Disk tiles. I receive the following error in the goes restitch log.

[TIRS00 KNES] 2023-07-20 20:55:08,400 [<module>]: Exception raised: Traceback (most recent call last): File "/awips2/ldm/etc/goes-restitch.py", line 429, in <module> loop.run_until_complete(read_stream(loop, read_in, queues, timeout=args.timeout)) File "/home/awips/anaconda3/envs/alchemy/lib/python3.6/asyncio/base_events.py", line 488, in run_until_complete return future.result() File "/awips2/ldm/etc/ldm.py", line 118, in read_stream await sink.put(product) File "/awips2/ldm/etc/goes-restitch.py", line 234, in put await self[dataset_name(item, self.filename)].enqueue(item) File "/awips2/ldm/etc/goes-restitch.py", line 44, in dataset_name sat_id = dataset.satellite_id.replace('-', '') File "src/netCDF4/_netCDF4.pyx", line 2922, in netCDF4._netCDF4.Dataset.__getattr__ File "src/netCDF4/_netCDF4.pyx", line 2864, in netCDF4._netCDF4.Dataset.getncattr File "src/netCDF4/_netCDF4.pyx", line 1445, in netCDF4._netCDF4._get_att File "src/netCDF4/_netCDF4.pyx", line 1927, in netCDF4._netCDF4._ensure_nc_success AttributeError: NetCDF: Attribute not found
The issue seems to only occur with the Full Disk domain tiles - not CONUS.