laszewsk / mlcommons

https://laszewsk.github.io/mlcommons/

MLCommons® Science

The main working group Web page is at:

• https://mlcommons.org/en/groups/research-science/

The working group github is located at

• https://github.com/mlcommons/science

The Policy document is located at

• https://github.com/mlcommons/science/blob/main/policy.adoc

The following descriptions and code to scientific benchmarks are avialable at:

• Cloudmask
• Earthqauke
• Stemdl
• Uno

Deveopment versions

Development versions of the codes prior to upload to the mlcommons directory are avalable as follows:

• Science Policy Document
• Benchmarks

mlcommons's Issues

cloudmask: atexit

why is atexit outcommented

atexit.register – should be working on most of the systems, I have tested it on Summit, dgx-2 and on Quadro GV100 graphics card attached to my workstation. The reason it was commented out because it did not work on the Rivanna machine in Virginia. Maybe different library for A100s. This needs to be investigated. I myself have not tested the code on any A100 cards.

greene team: cloudmask

• someone changed the slurm in script to use /c indicating they run it on windows, but don't you want to run it on green:

  sed -i "/log_file:/c\log_file: $(dirname $THEPATH)/cloudmask_run.log" $(dirname $THEPATH)/config.yaml
  sed -i "/mlperf_logfile:/c\mlperf_logfile: $(dirname $THEPATH)/mlperf_cloudmask.log" $(dirname $THEPATH)/config.yaml
  sed -i "/model_file:/c\model_file: $(dirname $THEPATH)/cloudModel.h5" $(dirname $THEPATH)/config.yaml
  

• we need a config.in.yaml that includes a USERTOREPLACE instead of the full username. than we can call make yaml to create the config.yaml file. The readme has to be updated. The makefile should use config.in.yaml and not config.yaml
• _load_data contains code that has absolute values that were replaced by UK team in main code for accuracy, should that be changed in data load?
• new log file library should be integrated: git+https://github.com/mlperf/[email protected]
• new numpy version should be tested: # numpy<1.24
• look at and update: laszewsk/mlcommons-nyu-uva#1
• look at and update: laszewsk/mlcommons-nyu-uva#2
• every week look at and update: https://docs.google.com/document/d/1LUuVSeLYv-717TJxtyikihLmuyf6at50FNBn57Doo_4/edit#heading=h.7p7jjuq4vbsm
• get rid of the presentations and move content into https://docs.google.com/document/d/1LUuVSeLYv-717TJxtyikihLmuyf6at50FNBn57Doo_4/edit#heading=h.7p7jjuq4vbsm
• create uniform accuracy measurement across training and inference
• Submitter: see #49
• setup_env seems to have ben replaced by Makefile: see Makefile and https://github.com/laszewsk/mlcommons/blob/main/benchmarks/cloudmask/experiments/greene/setup_env_and_yaml.py
• simple.slurm has a couple of comments that seem not needed: https://github.com/laszewsk/mlcommons/blob/main/benchmarks/cloudmask/experiments/greene/simple.slurm
• add other issues here

uva: cosmoflow tasks

• define and manage tasks here

uva: stemdl tasks

• define and manage tasks here

develop log file verification script

• directory structure
• log file analyser
  • eq
  • stemdl
  • uno
  • cloudmask

MLCommons submitter for Greene

The submitter in config file must be me or you need to go through an application process with MLCommons. You can find out how from the MLCOmmons web page. Till that has not been completed it must be me. This is not due to my rules, but the rules of MLCommons.

uva: uno tasks

• define and manage tasks here

Update the main mlcommons web site

the inference loop seems not to return the accuracy

Currently, we are loading the dataset in the inference function with 'data_loader = SLSTRDataLoader(args, file_paths, single_image=True, crop_size=CROP_SIZE); dataset = data_loader.to_dataset()' which seems to not be including the ground truth i.e msk when checking the function '_preprocess_images' in data_loader.py. This would mean calculating the accuracy is not possible without this ground truth and so we need to change a lot of changes to the cloud_inference to get accuracy?