toodef / neural-pipeline Goto Github PK

This need after 0.2 performance increase release.
Need to add to documentation guide about how to use Neural Pipeline more effective.

It's really useful to manage and customize console output by separated class.
Also, this makes possible to include some info to console:

• Real and model forward fps
• Some events, like it implemented in other monitors
• Manage metrics, that's show in progress bar

• Weights optimizing
• Easily add weights for monitoring
• Easily add weights for optimizing

The idea is cpu and gpu occupancy monitoring for analyse code pervormance

serialisation ---> serialization
initialisation ---> initialization

For solving this issue these steps needed:

• Add method enable_grads_acumulation(steps_num: int) to Trainer class
• Add gradients accumulating like described there
• Write tests for a simple network, there will be compared 2 losses values:
  1. Calculated without gradients accumulating
  2. Calculated with gradients accumulating

For do this test need to provide same data input to model and same weights in model (last can be done by flushing weights to file).

• [Optional] Explore how BatchNorm works with gradients accumulating. There says, that it's a problem (but disscussion from pre-relase of PyTorch 1.0 version)

Describe the bug

I was wondering when saw strange results of my metrics. I have custom metrics-wrappers around sklearn metrics:

from neural_pipeline.train_config import AbstractMetric, MetricsProcessor, MetricsGroup
from sklearn.metrics import precision_score, recall_score, accuracy_score

class Metric(AbstractMetric):
  def __init__(self, name, function):
    super().__init__(name)
    self.function = function

  def calc(self, output: torch.Tensor, target: torch.Tensor) -> np.ndarray or float:
    predicted = output.gt(0.5)
    return self.function(target, predicted)

class Metrics(MetricsProcessor):
  def __init__(self, stage_name: str):
    super().__init__()
    accuracy = Metric('accuracy', accuracy_score)
    precision = Metric('precision', precision_score)
    recall = Metric('recall', recall_score)
    self.add_metrics_group(MetricsGroup(stage_name).\
                           add(accuracy).\
                           add(precision).\
                           add(recall))

Configuration is following:

train_batch_size = 32
val_batch_size = len(X_test)

train_dataset = DataProducer([Dataset(X_train, y_train)], batch_size=train_batch_size)
validation_dataset = DataProducer([Dataset(X_test, y_test)], batch_size=val_batch_size)

train_stages = [TrainStage(train_dataset, Metrics('train')), 
                ValidationStage(validation_dataset, Metrics('validation'))]
loss = nn.BCELoss()
optimizer = optim.Adam(model.parameters(), lr=1e-3)
train_config = TrainConfig(train_stages, loss, optimizer)

fsm = FileStructManager(base_dir='data', is_continue=False)

epochs = 1
trainer = Trainer(model, train_config, fsm, device).set_epoch_num(epochs)
trainer.monitor_hub.add_monitor(TensorboardMonitor(fsm, is_continue=False))\
                   .add_monitor(LogMonitor(fsm))

trainer.train()

After training I load the metrics from data/monitors/metrics_log/metrics_log.json I got following results for validation data:

But I did same manually and got different result

I understand that if I used the batch_size that is not equal to the len of validation dataset I would get a different result. But here this is not the case. Another problem that I also found is result of last training step also differ from those computed manually no matter what value for epochs is used. I can't find error in the code, but it seems to me as magic

• Visual comparision (maybe by loading files from few logging monitors and passing it to another monitor)
• Console output
• Parsing the file for metrics comparision

Separate Predictor to Predictor and DataProducerPredictor. Predictor just init model and provide an interface to predict one item, DataProducerPredictor - provides an interface to predict all data from DataProducer object.

Also DataProducerPredictor can have interfaces:

• Connecting metrics to predict
• Select best and most worth predicts

Frequently we need to extract part of dataset for test network for example.
We need to implement this feature in DataProducer, cause it'll make possible to not change dataset class. Also, DataProsucer has flushing and loading indices interfaces.

Requirements:

• Possible of flushing indices for reproducible
• Different strategies for indices selection: (from begin, from range, all of it with step, e.t.c)

Need to add continue from best checkpoint option to Predictor

Environment

• OS: Ubuntu 18.04
• Python version: 3.6
• PyTorch version: 0.4.1
• Neural Pipeline version: 0.1.0

Describe the bug

There is traceback on screenshot, which happen after resuming train process with "resume" method of Trainer class in the end of last epoch.

• for DVC
• for datascience-cookiecutter

Now in np user do Trainer(..., device=torch.device('cuda)).
It's good but bad when we try to pass unexpected data to device.

Need to make possible use callback for this + create some predefinitions.

Need an example for multitask learning to show why train stages needed

• Make possible to share calculation between metrics
• Make plots grouping more intuitive
• Add function to restore metrics from log
• Make possible to disable/enable histogram

Environment

• OS: Ubuntu 18.04
• Python version: 3.6
• PyTorch version: 0.4.1
• Neural Pipeline version: 0.1.0

Describe the bug

TensorboardMonitor "is_continue" flag should be set True