deeptrackai / deeplay

Deeplay is a deep learning library in Python that extends PyTorch with additional functionalities focused on modularity and reusability. Deeplay seeks to address the common issue of rigid and non-reusable modules in PyTorch projects by offering a system that allows for easy customization and optimization of neural network components. Specifically, it facilitates the definition, training, and adjustment of neural networks by introducing dynamic modification capabilities for model components after their initial creation.

Core Philosophy

The core philosophy of Deeplay is to enhance flexibility in the construction and adaptation of neural networks. It is built on the observation that PyTorch modules often lack reusability across projects, leading to redundant implementations. Deeplay enables properties of neural network submodules to be changed post-creation, supporting seamless integration of these modifications. Its design is based on a hierarchy of abstractions from models down to layers, emphasizing compatibility and easy transformation of components. This can be summarized as follows:

• Enhance Flexibility: Neural networks defined using Deeplay should be fully adaptable by the user, allowing dynamic modifications to model components. This should be possible without the author of the model having to anticipate all potential changes in advance.
• Promote Reusability: Deeplay components should be immediately reusable across different projects and models. This reusability should extend to both the components themselves and the modifications made to them.
• Support Seamless Integration: Modifications to model blocks and components should be possible without the user worrying about breaking the model's compatibility with other parts of the network. Deeplay should handle these integrations automatically as far as possible.
• Hierarchy of Abstractions: Neural networks and deep learning are fundamentally hierarchical, with each level of abstraction being mostly agnostic to the details of the levels below it. An application should be agnostic to which model it uses, a model should be agnostic to the specifics of the components it uses, a component should be agnostic to the specifics of the blocks it uses, and a block should be agnostic to the specifics of the layers it uses . Deeplay reflects this hierarchy in its design.

Deeplay Compared to Torch

Deeplay is designed as a superset of PyTorch, retaining compatibility with PyTorch code while introducing features aimed at improving modularity and customization. Unlike PyTorch's fixed module implementations, Deeplay provides a framework that supports dynamic adjustments to model architectures. This includes capabilities for on-the-fly property changes and a style registry for component customization. Users can easily transition between PyTorch and Deeplay, taking advantage of Deeplay's additional features without losing the familiarity and functionality of PyTorch.

Deeplay Compared to Lightning

While Deeplay utilizes PyTorch Lightning for simplifying the training loop process, it goes further by offering enhanced modularity for the architectural design of models. PyTorch Lightning focuses on streamlining and optimizing training operations, whereas Deeplay extends this convenience to the model construction phase. This integration offers users a comprehensive toolset for both designing flexible neural network architectures and efficiently managing their training, positioning Deeplay as a solution for more adaptive and intuitive neural network development.

Quick Start Guide

The following quick start guide is intended for complete beginners to understand how to use Deeplay, from installation to training your first model. Let's get started!

Installation

You can install Deeplay using pip:

pip install deeplay

or

python -m pip install deeplay

This will automatically install the required dependencies, including PyTorch and PyTorch Lightning. If a specific version of PyTorch is desired, it can be installed separately.

Getting Started

Here you find a series of notebooks that give you an overview of the core features of Deeplay and how to use them:

• GS101 Understanding the Core Objects in Deeplay

  Layers, Blocks, Components, Models, Applications.

• GS111 Training Your First Model

  Creating, training, saving and using a deep learning model with Deeplay.

• GS121 Working with Deeplay Modules

  Differences between Deeplay and PyTorch modules. How to create, build, and configure Deeplay modules.

• GS131 Using the Main Deeplay Methods

  Application.fit(), Application.test(), DeeplayModule.predict(), Trainer.fit().

• GS141 Using Deeplay Applications

  Main Deeplay applications. Controlling loss functions, optimizers, and metrics. Training history. Callback.

• GS151 Using Deeplay Models

  Main Deeplay models. Making a model. Weight initialization.

• GS161 Using Deeplay Components

  Main Deeplay components.

• GS171 Using Deeplay Blocks

  Main Deeplay blocks. Adding, ordering, and removing layers. Operations.

• GC181 Configuring Deeplay Objects

  DeeplayModule.configure() and selectors.

• GC191 Using Styles

  Styles.

Advanced Topics

• AT201 Using Mappings as Inputs

Developer Tutorials

Here you find a series of notebooks tailored for Deeplay's developers:

• DT101 Deeplay File Structure

• DT111 Style Guide

• DT121 Overview of Deeplay Classes

• DT131 Deeplay Applications

• DT141 Deeplay Models

• DT151 Deeplay Components

• DT161 Deeplay Operations

• DT171 Deeplay Blocks

• DT181 Overview of Deeplay Internal Structure

deeplay's Issues

Problems with RNN

There seem to be some problems with the RNN:

• during training the log messages appear empty as "...."
• the dropout as parameter seems not to be working correctly

See the example rnn.py in DeepTrackAI/DeepLearningCrashCourse#25

Cannot access DeeplayModule attributes inside DeeplayModules

It is not possible to configure the layers of a DeeplayModule inside the class definition of another DeeplayModule, i.e. this works in general:

rnn = RecurrentNeuralNetwork(in_features=10,hidden_features=[20,30,40],out_features=50)
rnn.blocks.layer.configure(torch.nn.LSTM)

But this does not:

class RNNModel(DeeplayModule):
    def __init__(self):
        super().__init__()
        rnn=RecurrentNeuralNetwork(in_features=10,hidden_features=[20,30,40],out_features=50)
        rnn.blocks.layer.configure(torch.nn.LSTM)

No properties or attributes of the internal DeeplayModule, i.e. blocks, layers, etc, can be accessed.

Logs not populating after inference when model is initialized before an application

I've encountered a problem where, if I create a model prior to starting a deeplay application, application.logs remains empty following inference. Logs are instead stored in model.logs. For example:

import torch
import torch.nn as nn

import deeplay as dl

model = dl.LayerActivation(
    layer=dl.Layer(nn.Linear, 1, 10),
    activation=dl.Layer(nn.Sigmoid),
)
model.layer.log_output("x_transformed")
model = model.create()

classifier = dl.BinaryClassifier(
    model=model,
    optimizer=dl.Adam(lr=0.01),
).create()

x = torch.randn(2, 1)
y = classifier(x)

print(classifier.logs) # same for classifier.model.logs
# {}

print(model.logs["x_transformed"].shape)
# (2, 10)

if model = model.create() is skipped, then classifier.logs is populated while model.logs does not exist. Is this an expected behavior?

Update requirements.txt

After some recent PRs, the requirements file is out of date. Looks like some of the packages that are used for lodestar application / model are not listed here.

.configure unexpected behavior

Consider a sequential model composed of a CNN and an MLP:

from deeplay import ConvolutionalNeuralNetwork, MultiLayerPerceptron, Sequential

cnn = ConvolutionalNeuralNetwork(
    in_channels=1, 
    hidden_channels=[32, 64], 
    out_channels=96, 
    out_activation=nn.ReLU
)

mlp = MultiLayerPerceptron(
    in_features=None, 
    hidden_features=[64], 
    out_features=96, 
    out_activation=nn.ReLU
)

model = Sequential(cnn, mlp)

This default model works correctly. The problem arises when configuring the MLP.

For example, when expanding the number of hidden features:

model[1].configure(hidden_features=[32, 64])

Until this point, everything works correctly:

MultiLayerPerceptron(
  (blocks): LayerList(
    (0): LinearBlock(
      (layer): Layer[LazyLinear](out_features=32, bias=True)
      (activation): Layer[ReLU]()
    )
    (1): LinearBlock(
      (layer): Layer[Linear](in_features=32, out_features=64, bias=True)
      (activation): Layer[ReLU]()
    )
    (2): LinearBlock(
      (layer): Layer[Linear](in_features=64, out_features=96, bias=True)
      (activation): Layer[ReLU]()
    )
  )
)

But when creating (or building) the model, it breaks:

(1): MultiLayerPerceptron(
    (blocks): LayerList(
      (0): LinearBlock(
        (layer): LazyLinear(in_features=0, out_features=32, bias=True)
        (activation): ReLU()
      )
      (1): LinearBlock(
        (layer): Linear(in_features=32, out_features=64, bias=True)
        (activation): ReLU()
      )
      (2): LinearBlock(
        (layer): Linear(in_features=64, out_features=96, bias=True)
        (activation): ReLU()
      )
      (3): LinearBlock(
        (layer): Linear(in_features=32, out_features=64, bias=True)
        (activation): ReLU()
      )
      (4): LinearBlock(
        (layer): Linear(in_features=64, out_features=96, bias=True)
        (activation): ReLU()
      )
    )
  )

This issue was tested on a brand-new Linux machine with a fresh installation of deeplay

Need for a more "standard" way to define final models in Deeplay?

After discussing with some of you, we agreed that there is a need for a simpler and more standard way to define final models in Deeplay. This would allow users to directly specify model attributes or layers during model definition.

The proposed solution is to create a folder within Deeplay called "models." Here, we would define functions that implement standard models. This way, users, especially newcomers, can easily call and use models without needing to delve into more complex Deeplay functionalities.

I believe this approach makes sense for the book and improves user ease. It also helps to clarify the distinction between "components" and "models" in Deeplay, which is currently somewhat ambiguous.

Here's an example for an RNN:

def RecurrentNN(
    in_features,
    out_features,
    hidden_sizes=[...],
    bidirectional=False,
    layer_type="LSTM",
   # more paremeters
):
    if layer_type == "LSTM":
        layer = nn.LSTM
    else:
        # Handle other layer types if needed

    rnn = dl.components.RecurrentNeuralNetwork(
        in_features,
        hidden_sizes
        # other parameters
    )
    rnn.blocks.layer.configure(layer, bidirectional=bidirectional, #other parameters)

    dense_top = dl.components.MultiLayerPerceptron(#parameters)

    return dl.sequential(rnn, dense_top)

The function would return a DeeplayModule that remains customizable for more advanced users!

What do you think @giovannivolpe @Henrik-KM @cmanzo @BenjaminMidtvedt @HarshithBachimanchi ?

Can't show the central position of cells by following the notebook "Cell_Localization"

As I follow the notebook "Cell_Localization" which is in the tutorial, I find I could not generate and show the centroid. It is empty, which should have values.

Lowercase Module Names

According to PEP 8, module names should be lowercase. Shall we change this consistently in deeplay?

Cannot fetch attributes from LSTM,GRU,RNN

When wrapping torch.nn.GRU, LSTM or RNN layers in DL Layers, the attributes are fetched as "args" and "kwargs" and the DLModule cannot be built. I.e. the following layerList:

blocks = LayerList()
rnn = Layer(torch.nn.GRU,1,1,1,0)
act = Layer(torch.nn.Identity)
blocks.append(LayerActivation(rnn,act))
print(rnn)

(blocks): LayerList(
(0): LayerActivation(
(layer): Layer[GRU](args=1, kwargs=1)
(activation): LayerIdentity

When built, returns the following TypeError:

rnn.build()

TypeError: RNNBase.init() got an unexpected keyword argument 'args'

Instability in configurations when nesting `multi`.

There is a bug in nested calls to multi in blocks. Existing configurations cause errors, stating that some DeeplayModule is not attached to the root. Currently, we've attempted to fix this by removing all such configurations. This is a hotfix and should be properly fixed to address the root cause of the detached modules

Combining Tensor and Dict inputs

In our latest pull request #31, we've added the ability to take dictionaries as inputs for the network. Now, we're figuring out how to smoothly blend layers that usually work with tensors and those set up for dictionaries.

So, here's the question: should we make it a rule that if just one layer in a Sequential model expects dictionaries, all layers roll with it and accept dictionary inputs automatically? Your thoughts on this would be super helpful!

create vs build, configure, replace

The method module.create() does't overwrite module and requires mymodule = module.create() whereas module.build(), module.configure() and module.replace() do.
It makes sense, since create must return a new instance but I believe it can easily produce mistakes.

Side effects when loading in lodeSTAR checkpoint

Hello!
I just tried out the multi cell tracking with the deeplay lodeSTAR implementation following the notebook. Everything worked fine and I got the following satisfying result when pushing an image through after training:

Afterwards I wanted to try out saving and loading a checkpoint and performing the same plot. Since you use a lightning trainer these lines should be fine to load in the checkpoint after training:

loaded = dl.LodeSTAR.load_from_checkpoint(<.cpkt-path>, n_transforms=4,  optimizer=dl.Adam(lr=1e-4))

loaded.model.eval()

When I now run the same plot after loading in the model I get this result. I think it's still right but some form of translation is happening. I think loading in the model this way might be wrong on my side.

Thanks a lot for your help!

Syntax for slice selector for direct vs child.

I'm about to introduce a powerful new system for selecting which submodules to configure. It's based on __getitem__. The idea is to introduce powerful syntax to quickly select all items one wants to configure. I will first introduce the issue and then explain how the syntax works.

What do I need input on?

Consider the following two statements:

selection_1 = model[..., "blocks", 0:2]
selection_2 = model[..., "pool", 0:2]

Intuitively one would imagine that the first syntax would select the first two items of the list blocks. This means that slice should select the children of the previous selection.

However, that doesn't work for the second syntax. One does not want to select the first two children of every "pool" layer. One likely wants to select the first 2 "pool" layers.

I think the best solution is to have slices work select the children of the previous selection and introduce an alternative syntax for slicing the current selection directly. However, I'm not sure what that alternative syntax would be. This is what I need input on.

One idea is to have a string syntax like: model[..., "pool>0:2"]. But I'm very open to other suggestions and ideas.

What can be done

Consider the cnn. It has a child blocks with several children. Each child has the attributes pool, layer, activation, normalization.

• cnn["blocks"] would select the list of blocks. Equivalent to cnn.blocks
• cnn["blocks", 0] would select the first block of blocks. Equivalent to cnn.blocks[0].
• cnn[..., "layer"] would select every module named layer. Equivalent to block.layer for block in cnn.blocks
• cnn["blocks", 0, ...] would select every child of the first block. Equivalent to [cnn.blocks[0].pool, cnn.blocks[0].layer, cnn.blocks[0].activation, cnn.blocks[0].normalization]

you can of course mix and match the syntaxes. cnn[:, :, "layer"], cnn[..., "layer", ...] etc.

RNNs are not compatible with deeplay Layers

The problem is that argspec = self.get_argspec() in dl.Layers, when applied on the torch recurrent layer, does not correctly access the init input parameters. The reason is that the init method of this family of layers is taged as @overload which makes it difficult to access positional arguments correctly.

Intermediate inputs for sequential blocks

Continuation from #13

My suggestion would to have a subclassed

class GraphSequentialBlock(SequentialBlock):
....

where the interleaving of inputs is handled in the forward call. @JesusPinedaC thoughts?