wavefrontshaping / complexpytorch Goto Github PK

hi
I am a beginner to complex valued networks, My program uses real-valued loss functions and the network does not converge and I have a question: What kind of loss function should guide the network for complex valued networks? Is real or complex loss more appropriate? This must have something to do with how torch backspreads now. My understanding in this area is insufficient, although torch gives the specific calculation procedure.

I am looking forward to your reply. Thank you very much.

Hi,
About the naive batch norm, I believe it's more effective to normalize the tensor with the absolute value rather than normalizing each part individually:

What do you think?

Here is your implyment of complex dropout
https://github.com/wavefrontshaping/complexPyTorch/blob/master/complexFunctions.py#L21-L23
we know that dropout process is disabling some cell in a certain pdf., so for the complex dropout, real part and imag part shoud share the same "diabale cell index", but How can I guarantee that their indices are the same in above implyment?

Dear author, the current version can not compute in parallel, could you pls revise it ?

return (fr(input.real)-fi(input.imag)).type(dtype)
RuntimeError: imag is not implemented for tensors with non-complex dtypes.

Hi,
'I realized that the configuration is common for all input channels, is there any way to make a separate configuration for each channel? For example, change the padding size for each?

in the complexLayer.py, there is something wrong
class ComplexPReLU(Module):
def init(self):
super().init()
self.r_prelu = PReLU()
self.i_prelu = PReLU()

@staticmethod
def forward(self, inp):
    return self.r_prelu(inp.real) + 1j*self.i_prelu(inp.imag)

you need to decline the @staticmethod so that the ComplexPReLU would not report the bug

Hello and thanks a lot for your repo.
I am trying to have a complex model which uses complex 1D signals The size of the signals is 1024 for each real and complex signal separately. I separate real and imaginary and reshape the signals to [1,1,1,1024] but naturally I cannot use kernel size of size larger than 1. Normally I used conv1d for this signal (with the normal pytorch repository) but this repo does not yet support conv1d.
My request is: I wanted to see if you have any suggested solutions or simple changes that I can make to accommodate 1d convolutions or somehow use the current 2d convolutions to do the trick .
Thank you

Hi,

Thanks for the great work. I'm having a problem with ComplexBatchNorm1d layer when doing forward propagation. After passing a tensor of size [-1, 128] to this layer, this returned an error:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-54-45169697ed1b> in <module>
     34 
     35             # Forward + backward + optimizer
---> 36             output = model(input_data) # fw
     37             loss = mse(output, reference_data)
     38 

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

<ipython-input-53-40fbbf908ada> in forward(self, x)
     37         x = complex_relu(x)
     38         print(x.size())
---> 39         x = self.bn1d(x)
     40 
     41         x = self.out(x)

~/.local/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

~/.local/lib/python3.8/site-packages/complexPyTorch/complexLayers.py in forward(self, input)
    313 
    314         if self.training and self.track_running_stats:
--> 315                 self.running_covar[:,0] = exponential_average_factor * Crr * n / (n - 1)\
    316                     + (1 - exponential_average_factor) * self.running_covar[:,0]
    317 

RuntimeError: expand(torch.cuda.FloatTensor{[128, 64, 128]}, size=[128]): the number of sizes provided (1) must be greater or equal to the number of dimensions in the tensor (3)

I'm not sure how to interpret this error, and what would cause it to happen. Can you help? Thanks!

Hello, thanks for your work. I am trying to use it for image to image translation but I got the following error while using the ConvTranspose2d error : RuntimeError: Input type (CUDAComplexFloatType) and weight type (torch.cuda.FloatTensor) should be the same
Have you been able to successfully use ConvTranspose2d?

Thanks

According to the discussion in the original article, the real and imaginary parts should be strictly positive or negative at the same time so that Crelu can satisfy the Coccy-Riemann equation, but neither the convolutional layer nor the BN layer can make the output satisfy the strictly positive or negative at the same time, please tell me how everyone solved this problem so that Crelu can satisfy the Coccy-Riemann equation.
My current predictions all converge to the average of all labeled values， I guess that it's because of the unsatisfied Cauchy Riemann equation leading to negative non-conductivity, which in turn affects the gradient solution and transfer.

Similar to torch.nn.MSELoss().

I guess the function is pretty obvious as seen in pytorch/pytorch#46642

def complex_mse_loss(output, target):
    return (0.5*(output - target)**2).mean(dtype=torch.complex64)

https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/Convolution.cpp#L812

Complex Convolution
conv(W, x, b) = conv(Wr, xr, br) - conv(Wi, xi, 0) + i(conv(Wi, xr, bi) + conv(Wr, xi, 0))
where W, x and b are all complex inputs.
With Gauss Trick:
a = conv(Wr, xr, br),
b = conv(Wi, xi, 0),
c = conv(Wr + Wi, xr + xi, bi + br)
conv(W, x, b) = a - b + i(c - a - b)

from typing import List, Optional

import torch
import torch.nn.functional as F
from torch import Tensor, nn


class ComplexConvTranspose1dn(nn.ConvTranspose1d):

    def forward(self, input: Tensor, output_size: Optional[List[int]] = None) -> Tensor:
        if self.padding_mode != 'zeros':
            raise ValueError('Only `zeros` padding mode is supported for ConvTranspose1d')

        assert isinstance(self.padding, tuple)
        # One cannot replace List by Tuple or Sequence in "_output_padding" because
        # TorchScript does not support `Sequence[T]` or `Tuple[T, ...]`.
        num_spatial_dims = 1
        output_padding = self._output_padding(
            input, output_size, self.stride, self.padding, self.kernel_size,  # type: ignore[arg-type]
            num_spatial_dims, self.dilation)  # type: ignore[arg-type]

        i_r = input.real
        i_i = input.imag
        w_r = self.weight.real
        w_i = self.weight.imag
        b_r = self.bias.real
        b_i = self.bias.imag

        a = F.conv_transpose1d(i_r, w_r, b_r, self.stride, self.padding, output_padding, self.groups, self.dilation)
        b = F.conv_transpose1d(i_i, w_i, None, self.stride, self.padding, output_padding, self.groups, self.dilation)
        c = F.conv_transpose1d(i_r + i_i, w_r + w_i, b_r + b_i, self.stride, self.padding, output_padding, self.groups, self.dilation)

        return torch.complex(a - b, c - a - b)


class ComplexConvTranspose2dn(nn.ConvTranspose2d):

    def forward(self, input: Tensor, output_size: Optional[List[int]] = None) -> Tensor:
        if self.padding_mode != 'zeros':
            raise ValueError('Only `zeros` padding mode is supported for ConvTranspose2d')

        assert isinstance(self.padding, tuple)
        # One cannot replace List by Tuple or Sequence in "_output_padding" because
        # TorchScript does not support `Sequence[T]` or `Tuple[T, ...]`.
        num_spatial_dims = 2
        output_padding = self._output_padding(
            input, output_size, self.stride, self.padding, self.kernel_size,  # type: ignore[arg-type]
            num_spatial_dims, self.dilation)  # type: ignore[arg-type]

        i_r = input.real
        i_i = input.imag
        w_r = self.weight.real
        w_i = self.weight.imag
        b_r = self.bias.real
        b_i = self.bias.imag

        a = F.conv_transpose2d(i_r, w_r, b_r, self.stride, self.padding, output_padding, self.groups, self.dilation)
        b = F.conv_transpose2d(i_i, w_i, None, self.stride, self.padding, output_padding, self.groups, self.dilation)
        c = F.conv_transpose2d(i_r + i_i, w_r + w_i, b_r + b_i, self.stride, self.padding, output_padding, self.groups, self.dilation)

        return torch.complex(a - b, c - a - b)


class ComplexConvTranspose3dn(nn.ConvTranspose3d):

    def forward(self, input: Tensor, output_size: Optional[List[int]] = None) -> Tensor:
        if self.padding_mode != 'zeros':
            raise ValueError('Only `zeros` padding mode is supported for ConvTranspose3d')

        assert isinstance(self.padding, tuple)
        # One cannot replace List by Tuple or Sequence in "_output_padding" because
        # TorchScript does not support `Sequence[T]` or `Tuple[T, ...]`.
        num_spatial_dims = 3
        output_padding = self._output_padding(
            input, output_size, self.stride, self.padding, self.kernel_size,  # type: ignore[arg-type]
            num_spatial_dims, self.dilation)  # type: ignore[arg-type]

        i_r = input.real
        i_i = input.imag
        w_r = self.weight.real
        w_i = self.weight.imag
        b_r = self.bias.real
        b_i = self.bias.imag

        a = F.conv_transpose3d(i_r, w_r, b_r, self.stride, self.padding, output_padding, self.groups, self.dilation)
        b = F.conv_transpose3d(i_i, w_i, None, self.stride, self.padding, output_padding, self.groups, self.dilation)
        c = F.conv_transpose3d(i_r + i_i, w_r + w_i, b_r + b_i, self.stride, self.padding, output_padding, self.groups, self.dilation)

        return torch.complex(a - b, c - a - b)

Hi,
I am a beginner of CNN, and I want to feed my datasets (wave datasets) to the model, and I get the error:
"RuntimeError: stack expects each tensor to be equal size, but got [117792, 8] at entry 0 and [117792, 2, 128] at entry 1"

I noticed that you said "The only usage difference is that the forward function takes two tensors, corresponding to real and imaginary parts, and returns two ones too.", but i dont know how to take two tensors.

I don't know if my question is clear, maybe you can give me more guidance or example.

Thank you.

(1 - exponential_average_factor) * self.running_mean
TypeError: unsupported operand type(s) for *: 'float' and 'NoneType'

There is a bug in batch normalization, when we initialize with track_running_stats=False , the parameters arent initialized(as expected) but in training it tries to update running mean and running covariance.

error occurs at this line
self.running_mean = exponential_average_factor * mean
+ (1 - exponential_average_factor) * self.running_mean
I think we need to keep this inside the loop checking the status of track_running_stats. And might have to change while self.training is false as well(havent checked this though).

Hello. I'm implementing a Deep Complex NN based model.
I appreciate that I could quickly implement the model, thanks to you.

However, I think I found an improvement point in your code.
I think ComplexBatchNorm2d (or possibly 1d, but I didn't check it) might cause RAM leaks.

Memory leaks are not critical. It continuously and slightly grows in RAM after each epoch. However, I encountered an error: python kernel is going to die if there is no space.

You might want to see this link.

After small modification, it works. I just added some .detach()s in your code. I'm not sure this is correct theoretically, but there are no longer memory leaks after the modification. I'll send you a pull request!

Thank you.

from complexPyTorch.complexFunctions import complex_relu, complex_max_pool2d, complex_sigmoid

ImportError: cannot import name 'complex_sigmoid' from 'complexPyTorch.complexFunctions' (/usr/local/lib/python3.10/dist-packages/complexPyTorch/complexFunctions.py)

In line 301 of ComplexLayers.py, it should be "input = input - mean[None, :]" instead of "input = input - mean[None, :, None, None]" since the input of ComplexBatchNorm1d only has 2 or 3 dimensions.

File "D:/Pycharm/coplexcnn/train.py", line 124, in
y_hat = net(X)
File "C:\Users\MyPC\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:/Pycharm/complexcnn/train.py", line 79, in forward
x = self.bn1(x)
File "C:\Users\MyPC\AppData\Local\Programs\Python\Python38\lib\site-packages\torch\nn\modules\module.py", line 1051, in _call_impl
return forward_call(*input, **kwargs)
File "D:\Pycharm\complexcnn\complexLayers.py", line 294, in forward
self.running_mean = exponential_average_factor * mean
RuntimeError: The size of tensor a (253) must match the size of tensor b (32) at non-singleton dimension 1

How to solve it?Thank

Hi,
When I am running the example code, it always raises the complex value warning. I believe that it effectively affect the final performance of the network because only half of the complex data were considered during the training. Is there anyway that we could fix the issue?

Thank you.

hey and thanks for the great work,

Just a question on the grad computations (though this could be my lack of understanding on autograd complex analysis).

For the forward pass you have correctly, for example in

class ComplexLinear(Module):

def __init__(self, in_features, out_features):
    super(ComplexLinear, self).__init__()
    self.fc_r = Linear(in_features, out_features)
    self.fc_i = Linear(in_features, out_features)

def forward(self,input_r, input_i):
    return self.fc_r(input_r)-self.fc_i(input_i), \
           self.fc_r(input_i)+self.fc_i(input_r)

That the forward pass computes Re(x) Re(w) - Im(x) Im(w) + i ( Re(x) Im(w) + Im(x) Re(w) ), which is good... Then we want to compute the gradients of the weights.

Via complex differentiation we find
f(w) = w*x w,x \in Complex
df / dw = x

(which holds under cauchy-riemann) but when I find the derivatives of the real/imaginary components of the network I get different answers to the soln above (code below)

`
import torch as tc
import numpy as np
from torch import nn

class ComplexLinear(nn.Module):

    def __init__(self, in_features, out_features):
        super(ComplexLinear, self).__init__()
        self.fc_r = nn.Linear(in_features, out_features)
        self.fc_i = nn.Linear(in_features, out_features)

    def forward(self, input_r, input_i):
        return self.fc_r(input_r)-self.fc_i(input_i), \
               self.fc_r(input_i)+self.fc_i(input_r)

class Net(nn.Module):
    def __init__(self, n_input, n_output):
        super(Net, self).__init__()

        self.layer = ComplexLinear(n_input, n_output)

    def forward(self, x):
        xr = x[:,:,0]
        xi = x[:,:,1]
        out = self.layer(xr, xi)
        return out

X = tc.tensor([[[-0.16,0.51]]], requires_grad=True)

# First dim # datapoints, second dim # variables, third dim complex component
net = Net(X.shape[1],1)

Y = net(X)
Y[0].backward()
Y[1].backward()
print(net.layer.fc_r.weight.grad)
print(net.layer.fc_i.weight.grad)
`

returns
tensor([[0.3500]])
tensor([[-0.6700]])

whereas I think the answer should be X, which is
tensor([[-0.16]])
tensor([[0.51]])

... Really looking forward to your answer and happy to contribute, I'm going to be working on complex graph neural networks and there doesn't seem to be anything else online about this so could integrate this with your package?

Max

Hi I noticed that you have custom matmul (

Hi, thank you for the nice library.

There seems to be a small mistake in the complexPyTorch.complexLayers.ComplexDropout2d layer, which gives a device mismatch error (torch version 2.0.1+cu118):

""" .... line 106, in complex_dropout
return mask*input
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
"""

I managed to solve it by simply moving the mask on the right device in complexPyTorch.complexFunctions.complex_dropout2d as follows

`
def complex_dropout2d(input, p=0.5, training=True):

# need to have the same dropout mask for real and imaginary part,

# this not a clean solution!

device = input.device

mask = torch.ones(*input.shape, dtype = torch.float32, device = device)

mask = torch.nn.functional.dropout2d(mask, p, training)*1/(1-p)

mask.type(input.dtype)

mask = mask.to(device) # Line added

return mask*input`

Best!

I found some problems when using ComplexGRUCell
a. the complex-valued operator runs much lower than the real-valued operator, using such a complex-valued structure is much slower
b. the convergence may not be easily controlled, is there any paper supporting the point like complex-valued RNN is better than real-valued RNN in specific cases?

Hello,

I have some issues during testing or model.eval() with ComplexBatchNorm2d (the training is fine). The problem line is:

input = input - mean[None, :, None, None]

The problem seems to be the shape difference between mean and input.
In this case, the mean shape is [1000, 64, 32, 32] and input shape is [64, 64].
Note that, I used VGG16 like model with CIFAR10 dataset.

With the same model, NaiveComplexBatchNorm2d is working fine.