delzac / cntkx Goto Github PK

Hello, sorry to bother you again. I see you added the Linear Attention in the cntkx, and I want to know how to implement the causal transformer mentioned in the related paper. Thank you very much!

Hi,
Have you looked at LayerNormalization in CNTK? I think the original implementation is wrong in scale and bias parameter tensor dimensionality.
I also need Instance Normalization - I have my own implementation. I will also implement other normalization methods as well. It would be great to discuss with someone knowledgeable such as yourself. Would you be okay to exchange email with me?

Traceback (most recent call last):
File "train.py", line 4, in
import cntkx as Cx
File "C:\Anaconda3\lib\site-packages\cntkx_init_.py", line 1, in
from . import layers
File "C:\Anaconda3\lib\site-packages\cntkx\layers_init_.py", line 1, in
from .layers import *
File "C:\Anaconda3\lib\site-packages\cntkx\layers\layers.py", line 316
assert x.shape[0] > 0, f"input tensor must have a defined shape, input shape is {x.shape}"
^
SyntaxError: invalid syntax

D:\GIT\Segmentation\ImageSegmentation>python unettest.py
C:\Anaconda3\lib\site-packages\cntk\cntk_py_init.py:32: UserWarning: Unsupported Windows version (7). CNTK supports Windows 10 and above, only.
warnings.warn('Unsupported Windows version (%s). CNTK supports Windows 10 and above, only.' % my_distro_ver)
Traceback (most recent call last):
File "unettest.py", line 2, in
import cntkx as CX
File "C:\Anaconda3\lib\site-packages\cntkx_init_.py", line 1, in
from . import layers
File "C:\Anaconda3\lib\site-packages\cntkx\layers_init_.py", line 1, in
from .layers import *
File "C:\Anaconda3\lib\site-packages\cntkx\layers\layers.py", line 316
assert x.shape[0] > 0, f"input tensor must have a defined shape, input shape is {x.shape}"
^
SyntaxError: invalid syntax

Hello, there

I am very grateful for the cntkx you are developing.

Now I'm trying to use scaled_dot_product_attention, but it doesn't seem like you apply C.sqrt in the follow part, is that a missing or is it not a problem?

Hi @delzac,

Maybe you saw my work. I also develop a library that is similar to cntk in terms of syntax with dynamic graphs. In the beginning, the idea came meaningful to me, creating a deep learning library where users get the most flexible use scenarios. I have developed the library for 9 months almost. I had to change the design of the library 2-3 times. I got tired of those implementation changes. Redesigning is very hard. The cost of mistakes is high. After a while, the idea of creating a library became less interesting and more meaningless. I am about to lose my motivation, to be honest.

How do you keep developing the library that somehow is not popular maybe it will not be ever? How does it make sense to you? And why do you keep going further?

I am solo developing. I contacted with someone but not sure if he will help me with gpu implementations. He says his schedule is so busy and also he wants to help.

Do you happen to know where we can get some support?

Hi Delzac,
Do you know how to apply a sliding window, for example on image, to extract patches? I want to implement custom pooling op on each patch.

thx

Hi delzac,

Do you have any idea how to create a layer wrapper for upsampling? I want to create a layer wrapper to use inside Sequential([]), but a naive implementation like this does not work:

def UpsamplingLayer(name=''):
    @C.BlockFunction('Upsampling', name=name)
    def inner(x):
        y = ops.upsample(x)
       return y
    return inner

This will raise exception because ops.upsample() calls reshape(), which does check x when the layer is created, but x is undefined at that time.

Hello, delzac! Thank you very much for your job in cntkx. For the new SIREN layer, I found the code in the author's project is very complex, but your code is very simple, have you test it?