Normalization Layer for Amplitude Modulation Schemes about autoencoder-based-communication-system

[foryouforever]

based on your code, i plot a figure which is similar to the figure4c, i modify the encoded2, this is the modified code, def norm(x): n_power=tf.cast(tf.shape(x)[0],tf.float32) n_channel=tf.cast(tf.shape(x)[1],tf.float32) s=tf.norm(x) return (tf.sqrt(n_power/n_channel)/s)*x encoded2 = Lambda(lambda x: norm(x))(encoded1) instead of normalizing the power of every symbol to be 1,I normalize the power of n symbols to be n,which means the average power of every symbol is still unit however,in this way,I get a terrible ber SNR: -4 BER: 0.8243 SNR: -3.5 BER: 0.8124 SNR: -3.0 BER: 0.80428 SNR: -2.5 BER: 0.79442 SNR: -2.0 BER: 0.78052 SNR: -1.5 BER: 0.77344 SNR: -1.0 BER: 0.7591 SNR: -0.5 BER: 0.74424 SNR: 0.0 BER: 0.72774 SNR: 0.5 BER: 0.7165 SNR: 1.0 BER: 0.69768 SNR: 1.5 BER: 0.67856 SNR: 2.0 BER: 0.65764 SNR: 2.5 BER: 0.63698 SNR: 3.0 BER: 0.61506 SNR: 3.5 BER: 0.59394 SNR: 4.0 BER: 0.56804 SNR: 4.5 BER: 0.54096 SNR: 5.0 BER: 0.51614 SNR: 5.5 BER: 0.4864 SNR: 6.0 BER: 0.4553 SNR: 6.5 BER: 0.42904 SNR: 7.0 BER: 0.39406 SNR: 7.5 BER: 0.36084 SNR: 8.0 BER: 0.33036

[immortal3]

no! normalization is used to normalize the energy of signal.So, it will make sending signal power to 1 unit.

[v-i-s-h]

Okay.. So how to get amplitude modulation schemes? Like, how to reproduce Fig 4.c?? In the paper, it says

Fig.4c for the same parameters but with an average power normalization
instead of a fixed energy constraint

[immortal3]

I has same question to author but i never emailed to author about this.So, I never implemented figure 4.3.

[v-i-s-h]

Okay.. I mailed author a couple of days back. But no reply yet!!
Even though, they mention in paper that source code is available at https://github.com/radioml/introdlphy, it seems such a repo doesn't exist. :/

[immortal3]

I mailed author for asking about normalization layer but didn't get an answer and I also didn't mention all details in an email.So, somehow i know that I will not be getting a reply.Again That was my mistake.

But this paper was on arxiv for a long time and then published to IEEE still GitHub repo has not been shared publicly.I think there are only two implementation is available on github of this paper.One is mine and other is on pytorch.

[lintongtong123]

hi, i also have same question about the average power constraint, i think batch normalization may solve this question. but i am not sure about this and i have some problems in using batch normalization in this question

[immortal3]

no , batch normalization is something different.batch normalization is trying to keep normal gaussian distribution which will have zero mean and 1 stddev.but here we are trying to normalize power.

[lintongtong123]

However, i try to implement batch norm(you should set gamma_constraint=max_norm(1.4142136)) in your code, it can produce fig4(c). It seems that it can achieve the average power constraint. But when i introduce batch norm into the code, the loss is large(about 0.17).

@lintongtong123 I've tried using batch norm the results are too good to be true, I have tried implementing the norm layer as mentioned in the paper table-1, but I was not able to replicate the results. I am pretty sure this has something to with the norm layer.

[lintongtong123]

@Vsag Do you get fig4(c) by using batch norm? how do you implement batch norm in the code?I am interested in it.

@lintongtong123 sorry I am not implementing figure 4 (c) I was talking about 3 (a) figure . I wasn't able to replicate even that, did you get the results as seen in figure 3? If you want to implement batch norm these links are helpful. I use pytorch as we are using dense networks link-2 will be helpfu.

https://discuss.pytorch.org/t/example-on-how-to-use-batch-norm/216
https://www.programcreek.com/python/example/107653/torch.nn.BatchNorm1d

[lintongtong123]

@Vsag you should increase N if you want to get the results as seen in figure 3

@lintongtong123 what is the size of N for which you were able to replicate the results as seen in fig 3? I tried so many things but finally gave up tbf. Were you able to replicate the results?

[lintongtong123]

about 1000000

are you sure? I mean should the normalisation layer be the one mentioned in table one i.e list of layers?

[lintongtong123]

if you want to get fig3, you can try increase N in the code, about 1000000.

I've done what you said but at snr -4db the decoder is only a 50% accurate i.e it gets wrong results 50% of the time. but the hamming decoders start at about 78-80% accuracy at -4 snr. I am not sure about increasing the data size.

[immortal3]

here increasing data size want help unless we are dealing with more bits in msg. because after all if we use small number of bit let's assume n = 4 which means total 16 possibilities so trying data with 1000 or 5000 will be much more enough for small number of bits.

@lintongtong123 can you share result(constellation diagram) of batch norm layer.

@immortal3 can you please tell me accuracy of Hamming 7,4 hard and MLD at -4db? Thanks!

[immortal3]

i think bler is visible in readme file so you can calculate accuracy.so bler is 0.5. which is same as 50% accuracy.

@immortal3 Can you share your hamming codes' matlab code, because I swear when I implemented MLD I got accuracy close to 70-75% and in no way getting 50% accuracy. Code for reference.

`clear;
n=7;
k=4;
for_one=3.5;
for_zero=1.5;
%Generates Random Integers
x=randi([0,15],[500000,1]);
C=zeros(2^k,n);
transmit_vals=zeros(size(x,1),7);

temp=[];
parity_check=zeros(1,n-k);
%converts 2^k sequences into binary numbers
for values=1:2^k,
    temp=de2bi(values-1,4);
    temp=temp(:,end:-1:1);
    C(values,1:4)=temp;
    parity_check(1)=xor(xor(C(values,1),C(values,2)),C(values,3));
    parity_check(2)=xor(xor(C(values,1),C(values,3)),C(values,4));
    parity_check(3)=xor(xor(C(values,2),C(values,3)),C(values,4));
    C(values,5:end)=parity_check;

end

for values=1:size(x,1),
    temp=de2bi(x(values),4);
    temp=temp(:,end:-1:1);
    transmit_vals(values,1:4)=temp;
    parity_check(1)=xor(xor(transmit_vals(values,1),transmit_vals(values,2)),transmit_vals(values,3));
    parity_check(2)=xor(xor(transmit_vals(values,1),transmit_vals(values,3)),transmit_vals(values,4));
    parity_check(3)=xor(xor(transmit_vals(values,2),transmit_vals(values,3)),transmit_vals(values,4));
    transmit_vals(values,5:end)=parity_check;
    zero_indx=find(transmit_vals(values,:)==0);
    one_indx=find(transmit_vals(values,:)==1);
    transmit_vals(values,zero_indx)=for_zero;
    transmit_vals(values,one_indx)=for_one;

end
EbNo=[-4:0.5:8];

rate=k/n;
vals=zeros(1000,1);
predicted=zeros(size(x));
for values=1:size(EbNo,2),
    i=0;
    %total=0;
    std_dev=(1/(2*rate*10^(EbNo(1,values)/10)))^0.5;
    %Noise=normrnd(0,std_dev,size(transmit_vals,1),size(transmit_vals,2));

    while(i<size(transmit_vals,1)),
        total=0;
        Noise=normrnd(0,std_dev,1,size(transmit_vals,2));
        temp=transmit_vals(i+1,:)+Noise;
        %vals(i+1)=sum(temp>1 & temp<4);
        if (sum(temp>for_zero & temp<for_one))==0
            dist=9;
            indx=0;
            zero_indx=find(temp<=for_zero);
            one_indx=find(temp>=for_one);
            temp(1,zero_indx)=0;
            temp(1,one_indx)=1;

            for j=1:size(C,1),
               if dist>=sum(xor(temp,C(j,:)))
                    indx=j-1;
                    dist=sum(xor(temp,C(j,:)));
               elseif dist==sum(xor(temp,C(j,:))),
                        total=total+1;
               end
            end
            %total

            predicted(i+1)=indx;
            i=i+1;




        end

    end
    %plot(vals);
    values
    err(values)=(sum(predicted~=x)/size(x,1));
    %(sum(predicted~=x)/1000)

    %pause;

end
plot(EbNo,err);

`

[immortal3]

@Vsag oh sorry,you're talking about real hamming code i give accuracy of autoencoder.again it was also wrong.This code i used for plotting graph.bler is 0.37038 in autoencoder at -4.so,you are right than accuracy will be 70-75% as error is 37-32%.

SNR_db = -4:0.5:8;
SNR = 10.^(SNR_db./10); 
k = 4;
ber_7_4_hamming_hard = bercoding(SNR_db,'block','hard',n,k,4);
M = 4;
ber_4_4 = berawgn( SNR_db ,'psk',M,'nondiff');
ber_7_4_autoencoder = [ 0.37038  0.32722  0.29286  0.25436  0.21936  0.18242  0.1502  0.11876  0.09346  0.06972  0.04974  0.03586  0.02388  0.0169  0.00966  0.00574  0.00294  0.00154  0.00094  0.00034  0.00016  6e-05  3e-05  1e-05  8e-06 ];
%ber_7_4_fit = berfit(SNR_db,ber_7_4_autoencoder);
semilogy(SNR_db,ber_7_4_hamming_hard.*2,'b-.','linewidth',1),grid on,hold on;
semilogy(SNR_db,ber_7_4_autoencoder,'ro','linewidth',0.25,'MarkerFaceColor',[ 1 0 0]);

BER_SOFT=(7.*qfunc(sqrt(3*SNR)))+(7.*qfunc(sqrt(4*SNR)))+(qfunc(sqrt(7*SNR)));
semilogy(SNR_db,BER_SOFT./7,'b-','linewidth',1);
semilogy(SNR_db,ber_4_4.*4,'k','linewidth',1)
legend('Hamming (7,4) Hard Decision','Autoencoder(7,4)','Hamming (7,4) MLD','UnCoded BPSK(4,4)');
legend('Location','southwest');
ylim([1E-5 1]);```

[immortal3]

I also added code of both figure in case anyone need it.

@immortal3 sorry to be picky but in the code why is the first value in ber_7_4_autoencoder 37%, that corresponds to the ML part I suppose? Did you change the values for the project or did you actually get 37% for -4db? Thanks

[immortal3]

if you can see last two line generate matlab like array which i copy paste into .m file for plotting.I didn't change any value i just copy pasted generate result and also you can check difference between research paper plotting and my plotting.In initial area(-4db to -2db) it's not even same.

I ran your code a 30-40 times using different hyper paramters, I even wrote my own pytorch implementation, I never got 37% at -4db can you tell me how to do it? What all do I have to change in the code you've written. I am sorry if this sounds rude I am asking you in the most humble way possible so don't take it the wrong way. I am talking about the (7,4) auto encoder so there are no confusions.

[immortal3]

Wait I upload that to repo also.

Why is the norm layer batch normalised? You've claimed that it was something from the table's layer. I have given up man whatever you say.

[immortal3]

check repo i uploaded code!

Hey @immortal3 I have implemented batchnorm, you were claiming it's not batchnorm that was the reason I was pestering/bugging you. No hard feelings!! But is the norm layer used by the authors batch norm? The point with using batch norm I got this result, it give way better results, I think we are missing something, it's somewhere in between. I got a whopping 18% bler at -4db. So yeah thats what I have to say Thank you for bearing with me.

[immortal3]

i got also same result using batch norm but it's just illusion created by batch norm when you plot constellation diagram you get real picture what is going on.reality is signal we are send is no longer unit power so our ber calculation is wrong.we calculate ber assuming our signal is unit power and then draw it but here signal is no longer unit power.

[immortal3]

when you plot constellation diagram using batch norm then instead of lying on circle between (-1,1) it will go beyond like (-4,4) and give better result.When I first implemented this thing i got same result and thought it was improvement but after seeing what encoder is sending i get idea why it's not actual improvement.

[immortal3]

@Vsag any updates ?

Hey none as of now, my final year project is ml in communication systems, I have mailed my guide saying the problem and mailed the author's if anyone tells me anything you'll be the first to know. If I plot the batch norm what you say happens. One thing I have to try is, all the DL libraries implement a batch norm where the update the mean, std like they update the weights. I'll have to try where they make the just normalize the features to just Gaussian where the signal is normalised between (0,1). I'll have to check. You'll be the first to know :) I promise

[immortal3]

@Vsag Any updates ? 😅 😅

[lalapo]

have you ever tried to simulate the paper title deep learning based MIMO Communication?

[immortal3]

no, i seen that paper but many details are not clearly mentioned to really re-produce results.
And also i don't have much knowledge in that field.
It would be really cool to see that someone reproduce that MIMO paper's results.

[foryouforever]

sorry to disturb you,have you read the paper "deep learning based communication over the air",the channel in that paper is more complicated

[immortal3]

Yes, I tried. I was not able to understand many detail.

[liangchaowei]

Excuse me sir, did you draw fig3a fig3b with python? I am stuck with it for several weeks. Could you please share me with your code? Thank you so much!

[immortal3]

no,fig3a and fig3b are done with Matlab.

[MarkHan1001]

@foryouforever I normalize the average power of every symbol as you do, but the diagram is different from yours. It still is similar to the figure4(b). what detail does I miss?
Thanks.

[MarkHan1001]

@lintongtong123 I use BatchNormalization layer, and the diagram is similar to the figure4(c). the amplitude is [-4,4 on the x- and y- axes. When I set gamma_constraint=max_norm(), the amplitude can be changed. But the shape of constellation is changed, and the average power can't be controlled. How do you set gamma_constraint?
Thanks.

[liangchaowei]

@immortal3

semilogy(SNR_db,ber_4_4.*4,'k','linewidth',1)

I am sorry to trouble you. I am not sure whether the bit error rate of (4,4) system times 4 equals to the block error rate of (4,4)system. I mean that we should set 4 bit as a block at first rather than times 4 at last. Thank you!

[immortal3]

Did that. But then BER curve is different than author's Curve. So, I ended up doing 4 times.

[liangchaowei]

@immortal3
How did you set 4 bit as a block at first? I didn't come up with that.

Did you draw the constellation of traditional QPSK 16PSK and 16QAM? I am a beginner and I didn't find any material related to this in our library or on the Internet. Would you please show me how to realize that? Thank you so much!

[LitterBrother]

I am also researching the O'Shea's paper on the physical layer design based on DL. I have some issues bothering me on the Fig.4 (c). Could someone tell me how to implement the average energy normalization (i.e., QAM modulation scheme, not PSK one)?

[v-i-s-h]

@LitterBrother Did you check this: #2 (comment)

[LitterBrother]

@v-i-s-h Don't you think the value of ber is too terrible to be convincible?

[immortal3]

@LitterBrother Here the point is not ber. point is that how autoencoder is being able to learn when we impose some constraints.

[LitterBrother]

@immortal3 Thanks for your kind. However, when I change the normalization method offered by @foryouforever, I still get a 16-PSK constellation.

[v-i-s-h]

@LitterBrother
This is my working version.

# ---------------------------------- Auto encoder model ------------------------------------
with tf.name_scope( "Input_symbol" ):
    tx_in = tf.placeholder( dtype = tf.float32, shape = (None,noOfSymbols), name = "tx_symbol" )

with tf.name_scope( "Transmitter"):
    with tf.name_scope( "TX_1" ):
        l1_wgt  = tf.Variable( tf.random_normal([noOfSymbols,noOfSymbols],
                                                stddev=np.sqrt(3.0/(noOfSymbols+noOfSymbols))), 
                                name = "l1_wgt" )
        l1_bias = tf.Variable( tf.zeros([noOfSymbols]), name = "l2_bias" )
        l1_out  = tf.nn.relu( tf.matmul(tx_in,l1_wgt) + l1_bias )
    
    with tf.name_scope( "TX_2" ):
        l2_wgt  = tf.Variable( tf.random_normal([noOfSymbols,noOfChannels],
                                                stddev=np.sqrt(3.0/(noOfSymbols+noOfChannels))), 
                                name = "l2_wgt" )
        l2_bias = tf.Variable( tf.zeros([noOfChannels]), name = "l2_bias" )
        l2_out  = tf.matmul(l1_out,l2_wgt) + l2_bias

    with tf.name_scope( "TX_normalization" ):
        if scheme == 'PSK' :
            # For PSK
            l3_out      = np.sqrt(0.5*noOfChannels) * tf.multiply( 
                                l2_out,
                                tf.rsqrt( tf.maximum(
                                    tf.reduce_sum( tf.square(l2_out), axis = 1, keepdims = True ),
                                    1e-12 )
                                ),
                                name = "l3_out"
                            )
        # For XAM
        elif scheme == 'XAM' :
            thisBatchSize = tf.cast( tf.shape(l2_out)[0], tf.float32 ) 
            s = tf.norm( l2_out ) 
            l3_out  =   tf.multiply( tf.sqrt(thisBatchSize)/s, l2_out, name = "l3_out" )
        else:
            print( "Invalid scheme {}".format(scheme) )
    
    tx_out  = tf.identity( l3_out, name = "tx_out" )
    
with tf.name_scope( "Channel" ):
    # Placeholder for additive noise
    ch_noise    = tf.placeholder( dtype = tf.float32, shape = (None,noOfChannels), name = "ch_noise" )
    
    # Add noise
    ch_out      = tf.add( tx_out, ch_noise, name = "ch_out" )
    

with tf.name_scope( "Reciever" ):
    rx_in   = tf.identity( ch_out, name = "rx_in" )
    with tf.name_scope( "RX_1" ):
        # rx1_out     = tf.layers.dense( inputs = rx_in, units = noOfSymbols, activation = tf.nn.relu )
        rx1_wgt     = tf.Variable( tf.random_normal([noOfChannels,noOfSymbols],
                                                    stddev=np.sqrt(3.0/(noOfChannels+noOfSymbols))), 
                                    name = "rx1_wgt" )
        rx1_bias    = tf.Variable( tf.zeros([noOfSymbols]), name = "rx1_bias")
        rx1_out     = tf.nn.relu( tf.matmul(rx_in,rx1_wgt) + rx1_bias )

    with tf.name_scope( "RX_2" ):
        # rx2_out     = tf.layers.dense( inputs = rx1_out, units = noOfSymbols, activation = tf.nn.softmax )
        rx2_wgt     = tf.Variable( tf.random_normal([noOfSymbols,noOfSymbols],
                                                    stddev=np.sqrt(3.0/(noOfSymbols+noOfSymbols))), 
                                    name = "rx2_wgt" )
        rx2_bias    = tf.Variable( tf.zeros([noOfSymbols]), name = "rx2_bias" )
        rx2_out     = tf.nn.softmax( tf.matmul(rx1_out,rx2_wgt) + rx2_bias, name = "rx2_out" )

    rx_out  = tf.identity( rx2_out, name = "rx_out" )

[pachpandepriti]

batch norm(you should set gamma_constraint=max_norm(1.4142136)) in your code, it can produce fig4(c).

did u implement this?

[flysky6688]

how do they produce to fig4(c)?I still don't understand it.Could you help me to explain this?Thanks a lot.

[tinyxuyan]

Did that. But then BER curve is different than author's Curve. So, I ended up doing 4 times.

I think the right answer is that ber_8_8 = 1 - (1 - ser_2_2).^3 which agrees with the fig in article. Reason maybe that the block is constructed in dimensions which are orthogonal to each other.

[Urvashi-Patel]

What's a probability of Autoencoder output? I need maximum probability, how to calculate?

[yhalihal]

Hi!
Did anyone got an answer? I tried to email him too a week ago but sadly no response...
@immortal3 have you maybe tried to implement "Deep learning based MIMO communication" since? Tnx a lot

[Judithcodes]

wow. I joined the conversation late, but I have learnt a whole lot. Thanks for sharing guys.