porting to a ZCU102 and ZCU111? about verilog-ethernet

[alexforencich]

The only signals you need for data are the TX/RX_P/N connections. The rest of the signals are module control and status pins. You can ignore the status pins, but you may need to tie some of the control pins to specific levels so the transceiver will work correctly. For example, you have to tie tx_disable correctly so that the laser turns on in an optical module. You may also have to tie the rs pins. If you're using a direct attach copper cable, the levels on those pins might not make any difference.

You only need to drive what signals are present on your board. The VCU118 has dual QSFP28 connectors, so it has control signals for those. You can think of the VCU118 has having 8x 10G ethernet interfaces. Main thing to do is regenerate the GTH/GTY core with the appropriate configuration for your device, and then connect that to the PHY cores.

I think you can also use the gigabit port, you'll just have to figure out how to access that from the PL side and then connect it to the appropriate gigabit MAC core, and possibly a gigabit PCS/PMA core if it's an SGMII PHY chip.

Also, my standing offer for anyone who wants a design ported to a new board: donate a copy of the hardware, and I'll port it and maintain it.

[sirjansel0t]

Thank you for the swift response. I started porting the project onto the ZCU102. I will let you know if i was successful.

Also, the qsfp1_gt_txdata_# have bit widths of 128 but only 64 of them are actually useful? I looked at the GTH/GTY core and it seems like they take an input of 128 per channel but your core is specified to have a user data width of 64. is it just being 0 padded and only 64 is actually useful?

I'm referring to these signals:

https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/rtl/fpga.v#L470

wire [127:0] qsfp1_gt_txdata_1;
wire [127:0] qsfp1_gt_txdata_2;
wire [127:0] qsfp1_gt_txdata_3;
wire [127:0] qsfp1_gt_txdata_4;

wire [127:0] qsfp2_gt_txdata_1;
wire [127:0] qsfp2_gt_txdata_2;
wire [127:0] qsfp2_gt_txdata_3;
wire [127:0] qsfp2_gt_txdata_4;

I'm wondering because I am modifying the GTH/GTY core and it has a different userdata_tx_data_in size where it's 64 bits wide per GTH channel. Yours generate 128 per GTY channel.

Also, does comment: // Divide by 8 to get output frequency of 125 MHz
https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/rtl/fpga.v#L172

What does that exactly mean? You take in a 125 MHz clk and divide by 8 to get 125?

Sorry for the noobie questions. I'm new to all this stuff.

[alexforencich]

So the native GTY primitive has a bit width of 128, but not all of those bits are used. So there is an option to generate the core with or without the width change. I generated it without that, so I have to concatenate 128 bit signals and then use only the lower 64 bits. If you enable that option, then everything will match.

The divide by 8 is the MMCM output divider setting, there is also a milultiplier, which is also set to 8. So it multiplies and then divides the clock by 8. You could remove that if you want, but it's useful in case you need to generate some other clocks.

[sirjansel0t]

Do you have SFP and QSFP transceiver recommendations that you use to test this with? I just realized I only have 1000BASE-T so I have to go out and buy something.

I'm still working on this but I am getting close. For now I'm just going to implement 10G on the ZCU-102, not the 1G but I should prepare to somehow test it. Also, would I be able to test the 10G mac with just a 1000BASE-T to see if i can get 1G with it? Let me know. Thank you!

[alexforencich]

I usually use direct attach copper cables unless I have a need for a transceiver. I can't recommend any specific transceivers, there are a wide variety depending on your requirements.

You won't be able to test the 10G MAC by connecting it to a 1G interface. However, some of the example designs can switch the 1G port into the 10G path so you can test the 10G design with a loopback module and 1G NIC.

[sirjansel0t]

Thanks for the direct copper cable advice. I didn't think of that. Looks cheaper than a transceiver.

I was looking at the timing constraints on your code between lines 4 to 11 and I see those in the VCU118 and VCU 108 manual but not in the ZCU102 manual. Are those necessary? I'm wondering bc they make it clear that if you don't put those on the VCU118/108 you can potentially damage the board. I'm not sure if I need something similar on the ZCU102

[sirjansel0t]

These are the timing constraints I am referring to.

https://github.com/alexforencich/verilog-ethernet/blob/master/example/VCU118/fpga_10g/fpga.xdc#L4

FYI, which example are you referring to that switches the 1G interface?

Thank you.

[alexforencich]

Any board that has both a 1G and a 10G interface (VCU108 and VCU118 at least) should be set up that way - gigabit MAC, 10G MAC, 10G UDP stack, AXI stream switch.

The first two of those general parameters are required at least for non Zynq parts, not sure if you need them for Zynq. I think the tools will complain if you leave them out. Compress is generally a good idea, it makes the output bit file smaller and speeds up the configuration sequence. But it's not required. The other parameters are for loading the design from the flash on the dev board, these will need to match the board configuration. They are not required when loading via JTAG. They may not be applicable for Zynq designs, again not sure about that.

[sirjansel0t]

This is my only final critical warning. It won't route the first 2 SFP ports and I don't know what is going on.

[Vivado 12-1411] Cannot set LOC property of ports, Terminal sfp0_rx_p cannot be placed on E4 (GTHE4_CHANNEL_X1Y12) because the pad is already occupied by terminal sfp1_rx_p possibly due to user constraint ["/home/serf203/verilog-ethernet/example/ZCU102/fpga_10g/fpga.xdc":60]

[Vivado 12-1411] Cannot set LOC property of ports, Instance sfp_gty_inst/inst/gen_gtwizard_gthe4_top.gtwizard_ultrascale_0_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[27].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[0].GTHE4_CHANNEL_PRIM_INST can not be placed in GTHE4_CHANNEL of site GTHE4_CHANNEL_X1Y13 because the bel is occupied by sfp_gty_inst/inst/gen_gtwizard_gthe4_top.gtwizard_ultrascale_0_gtwizard_gthe4_inst/gen_gtwizard_gthe4.gen_channel_container[27].gen_enabled_channel.gthe4_channel_wrapper_inst/channel_inst/gthe4_channel_gen.gen_gthe4_channel_inst[1].GTHE4_CHANNEL_PRIM_INST(port:). This could be caused by bel constraint conflict ["/home/serf203/verilog-ethernet/example/ZCU102/fpga_10g/fpga.xdc":61]

There seems to be a conflict between the xdc files that I made and the one generated by the transceiver core.

The only constraints i have on those ports are:

set_property PACKAGE_PIN D2 [get_ports "sfp0_rx_p"] ;# Bank 230 - MGTHRXP0_230
set_property PACKAGE_PIN C4 [get_ports "sfp1_rx_p"] ;# Bank 230 - MGTHRXP1_230
set_property PACKAGE_PIN B2 [get_ports "sfp2_rx_p"] ;# Bank 230 - MGTHRXP2_230
set_property PACKAGE_PIN A4 [get_ports "sfp3_rx_p"] ;# Bank 230 - MGTHRXP3_230

set_property PACKAGE_PIN E4 [get_ports "sfp0_tx_p"] ;# Bank 230 - MGTHTXP0_230
set_property PACKAGE_PIN D6 [get_ports "sfp1_tx_p"] ;# Bank 230 - MGTHTXP1_230
set_property PACKAGE_PIN B6 [get_ports "sfp2_tx_p"] ;# Bank 230 - MGTHTXP2_230
set_property PACKAGE_PIN A8 [get_ports "sfp3_tx_p"] ;# Bank 230 - MGTHTXP3_230

any idea what should I do?

[sirjansel0t]

i figured it out. I had some mislabels but now it's fixed. Unfortunately i cannot test anything since I don't have sfp connectors =(

[sirjansel0t]

Can I use an sfp connector to test a 10G MAC with a 1G interface? I have 1000BASE-T. I would just like to make sure. Sorry for all the dumb questions.

[alexforencich]

No, you cannot use a 1G SFP module to test a 10G connection, the line protocols are totally different. You could set up the interface in gigabit mode, though - use a gigabit MAC and gigabit PCS/PMA from Xilinx - but I'm not sure if this would show you anything useful.

[sirjansel0t]

Yes I was wondering if I can setup one of the SFP connectors in a 1 gigabit configuration. I didn't see that in your examples. I know you set up all boards using a dedicated 1 gigabit ethernet port. Right now I have all SFP ports set up for 10G but I can change them to 1 gigabit. I don't need all 4 SFP ports in a 10G config. Do I just need to use the PCS/PMA? Do i need to use the transceiver wizard at all? Thanks a lot.

[alexforencich]

I think all you need to do is create a gigabit PCS/PMA core instance, and that will contain the transceiver. You may run in to problems with sharing a quad between a transceiver wizard instance and a PCS/PMA core. Not completely sure about that, though. You'll also need to swap out the MAC for a gigabit MAC and then either swap the stack or for the 8 bit version or add width converters. Or I guess I recently updated the MAC modules to use the FIFO and width converter combo modules, so you should be able to set the 1G MAC to have a 64 bit AXI stream interface.

[sirjansel0t]

i was able to start testing today. My computer can recognize that it is connected to my board but it does not ping back or when i try netcat, nothing echos back.

The design mimics the ExaNIC X10 which also uses the GTH transceivers. I noticed in this port you created 2 different instances of BUFG_GT clocks.

https://github.com/alexforencich/verilog-ethernet/blob/master/example/ExaNIC_X10/fpga/rtl/fpga.v#L279

So you have gt_txusrclk[n],gt_txusrclk2[n], gt_rxusrclk[n],gt_rxusrclk2[n]. gt_txusrclk2 and gt_rxusrclk2 also divided by 1 and what is the reason for that? what's the different between both BUFGT_GT instances? None of the other boards have 2 instances of BUFGT_GT.

The other question I have is that the ExaNIC X10 has a 161.1328125 MHz reference clock but the ZCU102 has a 156 MHz. I am wondering if that affects the BUFGT_GT divider for gt_rxusrclk2 and gt_txusrclk2.

Please let me know.

Thank you!

[alexforencich]

GTH and GTY transceivers are different; all of the other boards use GTY transceivers - the GTY transceivers support an internal data width of 64 and a user data width of 128, while GTH transceivers support an internal data width of 32 and a user data width of 64. Since the 10G stack uses a 64 bit datapath, the transceiver needs to be configured with a user data width of 64 bits. With the GTY transceivers, the internal and user data widths match and hence they run from the same clock. With the GTH transceivers, the internal data width is 32, so it runs at double the clock frequency. The two buffers have different DIV settings, one is a divide by one, the other is a divide by two. The reference clock will not affect the BUFG_GT settings.

You won't be able to ping the design; it currently does not support ICMP. I recommend setting up wireshark to check for ARP traffic while you try netcat (or ping, ping will also trigger an ARP request). If you don't see an ARP response from the board, then you won't be able to exchange any data. Next step would be to add an ILA somewhere in the receive path to examine the incoming traffic to make sure the ARP request is reaching the board in tact.

[sirjansel0t]

Ok thanks. I think got it. I think I was just testing the wrong SFP port. How do I check if I am getting the full 10G line rate? Let me know.

Thank you!

[alexforencich]

FYI: there is now a ZCU102 reference design in the repository here: https://github.com/alexforencich/verilog-ethernet/tree/master/example/ZCU102/fpga