mao-wfs / mao-merge-45m

Merge MAO datasets observed with NRO 45m telescope into a single Zarr file

mao-merge-45m

Merge MAO datasets observed with the Nobeyama 45m telescope

Installation

$ git clone https://github.com/mao-wfs/mao-merge-45m.git
$ cd mao-merge-45m
$ poetry install

Convert raw data to formatted (Zarr) data

Correlator output

from pathlib import Path
from mao_merge_45m import correlator


path_vdif = Path("/path/to/data.vdif")
path_raw_zarr = correlator.to_zarr(path_vdif)
path_fmt_zarr = correlator.convert(path_raw_zarr)

Accelerometer log

from pathlib import Path
from mao_merge_45m import accelerometer


path_gbd = Path("/path/to/data.gbd")
path_raw_zarr = accelerometer.to_zarr(path_gbd)
path_fmt_zarr = accelerometer.convert(path_raw_zarr)

Accelerometer log (CSV format)

from pathlib import Path
from mao_merge_45m import accelerometer_csv


path_csv = Path("/path/to/data.csv")
path_zarr = accelerometer_csv.convert(path_csv)

Weather log

from pathlib import Path
from mao_merge_45m import weather


path_csv = Path("/path/to/data.csv")
path_zarr = accelerometer.convert(path_csv)

Antenna log (normal 10-sps format)

from pathlib import Path
from mao_merge_45m import antenna


path_log = Path("/path/to/data.txt")
path_zarr = antenna.convert(path_log)

Antenna log (new 50-sps format)

from pathlib import Path
from mao_merge_45m import antenna_50_sps


path_log = Path("/path/to/data.txt")
path_zarr = antenna_50_sps.convert(path_log)

SAM45 log (text dumped by sldump)

from pathlib import Path
from mao_merge_45m import sam45


path_log = Path("/path/to/data.txt")
path_zarr = sam45.convert(path_log)

Merge formatted data into a single Zarr file

from pathlib import Path
from mao_merge_45m import merge


path_correlator = Path("/path/to/formatted/correlator.zarr")
path_accelerometer = Path("/path/to/formatted/accelerometer.zarr")
path_weather = Path("/path/to/formatted/weather.zarr")
path_antenna = Path("/path/to/formatted/antenna.zarr")
path_sam45 = Path("/path/to/formatted/sam45.zarr")
path_merged = Path("/path/to/merged.zarr")


merge.merge(
    path_correlator,
    path_merged,
    path_accelerometer_zarr=path_accelerometer,
    path_weather_zarr=path_weather,
    path_antenna_zarr=path_antenna,
    path_sam45_zarr=path_sam45,
)

mao-merge-45m's Issues

Rename functions

Add zip option to merge function

Add weather module

Replace real_Z1 and real_Z2.

Fix real_Z1 and real_Z2 in antenna_50_sps.py, lines 196-7, as they are reversed.

Rename project

• Rename package name (nro45_merge → mao_merge_45m)
• Rename repository name (nro45-merge → mao-merge-45m)

Do not use append method of DataFrame

Use pandas.concat instead of df.append to merge multiple DataFrame objects into one.

Update docs

Update log paths of merge function to be optional

Add command line interface

• Add command line interface, nro45-merge
• Add empty interface functions for dry-run
• Add logging feature

Make function options keyword-only arguments

Release v0.2.0

Add chunk length option to VDIF-to-Zarr conversion

Add accelerometer module

Add module to read GL900-APS GBD file and convert it to a xarray-compatible format.

Add support for SAM45 logs

Add support for Poetry >=1.2

Update pyproject.toml and poetry.lock to be compatible with Poetry >=1.2.

Any dependency declared in the legacy dev-dependencies section will automatically be added to a dev group.
As the dev-dependencies is now deprecated, projects should migrate to the new group syntax as soon as possible.
Keep in mind that the group syntax is a new feature of Poetry 1.2, and your project will not be buildable with Poetry 1.1 after migrating.
https://python-poetry.org/blog/announcing-poetry-1.2.0/#dependency-groups

Update calibration method of SAM45 logs

Update README

Add support for new 50-sps antenna logs

Add conversion from Zarr to VDIF

Add nro45_merge.correlator.to_vdif function to convert a Zarr file to a VDIF file.
(It is for identity check between a VDIF file and that converted as VDIF → Zarr → VDIF)

Fix incorrect assignment of accelerometer data as data variables

Publish the package to PyPI

Update test data and scripts

Update test data using real measurements so that test script for the merge module can be made.

Fix missing long_name of time coordinate

Add correlator module

Add module to read OCTAD VDIF file and convert it to a xarray-compatible format.

Remove unused modules

Rename prefix (path_) from path arguments

Eliminate duplicate timestamps.

Add a sentence to get_df that eliminates duplicate timestamps and interpolates the eliminated parts.

Update project dependencies

Read programmed values from new 50-sps antenna logs

Add merge module

Add accelerometer_csv module

Create a new module (accelerometer_csv) to read accelerometer logs as the CSV format.

Fix timezone of weather and accelerometer data

Weather and accelerometer data for distribution uses raw timestamps in JST, but correlator data are written in UTC.
Fix weather.to_dist_zarr and accelerometer.to_dist_zarr so that they uses UTC as the timezone of output Zarr files.

Allow VDIF to be passed to the convert function

Read Az/El values of the collimator from new 50-sps antenna logs

Fix wrong unit attributes of accelerometer data in Zarr

Add test for SAM45 logs

Support for writing merged dataset to zip storage

Add merge information to merged Zarr

Fix frequency range in correlator module

Update devcontainer

Fix wrong data_vars/coords assignment in merge module

Add test for new 50-sps antenna logs

Add options to shift timestamps of logs

• merge()関数の以下の行の間でcorrelator.timeの時刻にオフセットを与えられるようにする

  correlator = xr.open_zarr(path_correlator_zarr)
  # correlator.timeにオフセットを加える
  correlator.to_zarr(path_merged_zarr, mode="w")

• merge()関数に時刻オフセットの引数を追加する
• merge()関数のdocstringに時刻オフセットの記述を追加する

Fix timestamp to yymmdd

Add "date_parser = partial(pd.to_datetime, format=LOG_TIMEFMT)" into get_df.

Fix SAM45 timestamp to datetime64[ns]

Unify SAM45 timestamp types with other data.

Rename module (antenna_50sps.py -> antenna_50_sps.py)

init.py.にimport antenna_50_spsを加える。

Add antenna module

• Add antenna module
• Add antenna log to Zarr converter
• Add tests for the module

Add citation information

Add option to change integration time (5 or 10 ms)

The correlator module currently only supports for the integration time of 10 ms.
This issue will update the code to support for the integration time of 5 ms as well.

Note: In the current code, changing the integration time (10 → 5 ms) needs to change the following constants:

-N_UNITS_PER_SECOND = 6400
+N_UNITS_PER_SECOND = 12800
...
-N_SCANS_PER_SECOND = 100
+N_SCANS_PER_SECOND = 200
...
-N_SCANS_PER_MINUTE = 6000
+N_SCANS_PER_MINUTE = 12000
...
-UNIT_MILLISECOND = np.timedelta64(10, "ms")
+UNIT_MILLISECOND = np.timedelta64(5, "ms")

Add argument of SAM45 log path to merge function