realfastvla / realfast Goto Github PK

Nominal plan that needs to be finalized:

• 32 servers
  • can host at least 2 GPUs per server.
  • 32 GB
  • dual 12-core CPUs
    -~$5k
• 1 head node
  • 512 GB
  • dual 4-core CPUs
  • ~$12k

Open question:

• how does architecture affect latency, capacity, diversity of modes, etc.?

We will generate visibility snippets to be archived. What metadata and format should they have?

Baseline plan is to write SDMs with sdmpy without dedispersion correction.

Is it adequate to modify metadata only for new time range?

We have the ability to simulate transients, but do not regularly use it in real-time observing. It would be valuable to implement an end-to-end plan for adding, finding, and tracking mock transients as a regular check of data quality.

Martin Pokorny has ideas..

R6: A more granular schedule, with definite targets should be developed to track progress. This schedule shall be shared with affected aspects of NRAO to avoid priority collisions.

• Schedule needs to be integrated with other NRAO activities and priorities:
• Identify periods of conflict for key personnel
• More granular schedule for early identification of slippage

queue_monitor can accidentally parse the wrong mergepkl if two scans finish at the same time. In one case, the two scans were from different SBs, so the scans to archive were not correctly identified.
Offending code is:

for jobid in jobids:
...
if job == finishedjobs[-1]:

Need to finalize nominal plan for purchasing 64 GPUs:

• nVIdia Titan
• 6 GB global memory

Does choice of GPU affect/limit how we program for it?

Related q: is numba efficient for GPUs?

Currently the realfast system runs by setting the realfast@cbe-node-01 path to include @caseyjlaw software. Bad!

A group-controlled realfast user needs to be defined.

Work with the NRAO SSA group to create an ICD or requirements document to define the realfast / archive interface.

During a science run for 17A-396, we noticed a bdf remaining after archiving completed. It was the last scan and a calibrator scan. The SDM downloaded from the NRAO archive included the bdf, so that seems to have worked properly.
It is likely a bug related to bdf cleanup within the realfast code.

CBE will write SDMs for realfast to process. The metadata in those SDMs is not well defined yet, as they are a new data product.

The integration of realfast with the VLA observing system should be a prototype of a general system for third-party systems. As such, a general interface needs to be designed for systems like this.

R5: Identify and document (ICD) all external interfaces to the realfast system. Major subsystems within realfast should consider similar documentation.

Definition of data formats:

• Identification and enumeration of interfaces (e.g. XML documents TD consumes)
• Clearly identified interfaces in the processing pipeline (e.g. replacing the TD with another algorithm)

Can realfast operate commensally with VLA OTF mode?

Potential issues:

• phase stepping for VLASS is faster than DM sweep
• the slew rate is longer, but still fast!

We have working prototype that uses jupyterhub as a front end. Do we want to redevelop front end for internal portal or further develop notebooks for use with jupyterhub?

Development of algorithm and science behind concept of periodicity imaging. Open questions:

• what kinds of pulsars are accessible?
• is there a new science case in those pulsars?
• what algorithms are needed to access that science?
• what tests can we run to prove that case?

If all goes well, write it up!

Having "magic disk" and GPU hardware would make testing more effective and informative for eventual cluster order.

VOEventdb is a way of searching a history of VOEvents.
Could use realfast server for this, too.
See https://github.com/timstaley/voeventdb.

• Does 0.45 s phase stepping hurt realfast transient detection system?
• Does OTF hurt transient detection?

R2: Elaborate the scope, design and operation of the portal. Include all applicable interfaces (both software and hardware). Discuss the design with the NRAO SSA group.

Would be nice to get some rudimentary version of rfpipe (using CPUs via dask distributed scheduler) on CBE for NRAO review.

End of scheduling block process will trip up if a bdf never arrives. A few potential issues here:

1. sdm timeout not working
2. need to be able to know when bdfs will never arrive (correlator fail)
3. be sure that we can know state of sdm when bdfs are not around for any reason.

Prototypes and new features coming together:

• RDMA API
• new worker (GPUs, nvme)
• rfpipe
• new software environment

A test would motivate integration and answer rudimentary questions about performance before making big purchase.

Rudimentary co-observing working with LWA (from @xiggystardust), but how to manage this for:

• optical network
• general co-observing policy will require nrao ost integration

R4: Develop a commissioning plan and explicitly identify the tests which will mitigate the risks identified.

There are several areas of risk to the operations of VLA:

• Averaging in the CBE rather than Baseline cards
• Data rates through Correlator to CBE switch.
• Data rates through Infiniband switches

Many limits will be empirically determined, so a robust process to identify, monitor, and organize results must be defined.

R3: Work with the appropriate NRAO stakeholders define the policy for realfast data, proprietary periods, access controls etc.

Project plan graphically designed at: https://drive.google.com/open?id=0B7Md1PdoX8joQ2JGbTNGbmFHRFU. These times and issues need to be put into github.

September 2016 meeting notes at: https://docs.google.com/document/d/1sVUEs7_nO9kfFOml7mkK1W6e034KBGJu64Qsl_CxnZg.

Need detailed diagram of all systems and interfaces

• software
• hardware where software runs
• locations of machines (nrao, berkeley?)
• interfaces
• data rates
• responsible parties
• work to be done

First draft from years ago:
https://plus.google.com/photos/106898753996583382613/albums/5888271939578516929?authkey=CLjAy9PC38-HGg

Whiteboard exercise

Need to define path to get output visibility cutouts into archive.

Currently Paul's sdmpy library can create cutouts. Those products need to be ingested to archive system.

key questions:

• latency?
• GPU utilization?
• is memory buffer adequate for typical VLA observing patterns?

get new server with gpus on infiniband connection to cbe

NRAO PMD recommends some documents to support the realfast development process.

1. Requirements and technical architecture (two docs, actually)
2. Concept of operations ("how should system behave?")
3. Operations plan (list activities, set budgets, future)

Candidate detections will create visibility data to be archived. However, many candidates will be bad and should be cut before archiving.

Initially, this will require human feedback ("Astronomer on duty", AOD) to remove bad candidates and reduce the archived data rate. It is expected that this can be done on a ~day cadence.

• Observing control interface should be able to control candidate rate (e.g., set threshold for given SB)
• Need a plan to go from duty astronomer to NRAO (autonomous) system
  • Does this require 15 min a day? If so, NRAO can handle via data analyst.
  • If longer, we need to reduce output somehow. ML classifier?

Data volumes and rates are large and may be limiting either in the correlator or within the realfast cluster. Using compression/decompression may help trade compute for data rate limitations.
Prior art:

• https://github.com/aroffringa/dysco
  • Custom built for CASA in C, so hard to port.
  • Algorithm is well suited to low-SNR compression. Not tested yet.
• https://github.com/kiyo-masui/bitshuffle
  • Well designed and portable, esp if using hdf5
  • Requires high correlation between neighboring bytes (i.e., high SNR). Tests so far suggest this is not true for fast dump VLA.

The current implementation uses Jupyterhub and elasticsearch, both hosted at http://realfast.berkeley.edu.

• Does that work in the long term or can NRAO host?
• Could external services call on NRAO stored data?

Need to define and plan for electrical upgrades required to support realfast.

James Robnett knows details.