uwcms / cms-calo-layer1 Goto Github PK

With the current FE oRSC bitfile, we can't make the LEDs light up via direct XMD writes. See the Makefile [1] for the xmd commands, or the instructions below.

Once should be able to (on a machine connected to oRSC via XMD):

# power cycle oRSC - lights will be blinky from Tom's BE bitfile
cd cms-calo-layer1/rct2orsc_byhand
# upload latest FE bitfile - lights are off
make orscfebits 
# write LED control registers via XMD
make lightson
# turn off LEDs
make lightsoff

Expected behavior: lights go on and off
Actual behavior: lights stay off

[1] https://github.com/uwcms/cms-calo-layer1/blob/master/rct2orsc_byhand/Makefile#L42

Adding @ekfriis , @nwoods ....

I followed the instructions in issue #4 .

This is where I am stuck and don't know what is going on:

Both 60001 and 60002 ports are open. Both /tmp/softipbus and /tmp/softipbus-forward are running on the server:

Now, I do a capture from ctp6commander using the following command:

python cli.py --connection ctp6_connections.xml --verbose capture

Then check the following prompt:

~ # /tmp/softipbus-forward 
1970-01-01 00:03:00 | INFO  | ipbus2mem serving memory @ 0000000000000000 on port 60001
1970-01-01 00:03:07 | INFO  | Connecting client #0
1970-01-01 00:03:07 | DEBUG | Detected data from client #0
1970-01-01 00:03:07 | DEBUG | Processing 40 bytes of data from client #0
1970-01-01 00:03:07 | DEBUG | Processing 10 words of data
1970-01-01 00:03:07 | DEBUG | Processing full transaction
1970-01-01 00:03:07 | INFO  | Forwarding transations with TX: /dev/ttyUL1 and RX: /dev/ttyUL1
1970-01-01 00:03:07 | DEBUG | Forwarding 1 transaction
1970-01-01 00:03:07 | DEBUG | Waiting for result
1970-01-01 00:03:07 | DEBUG | Read back transactions
1970-01-01 00:03:07 | DEBUG | Got new header packet f0000020
1970-01-01 00:03:07 | DEBUG | Processing full transaction
1970-01-01 00:03:07 | DEBUG | Forwarding 1 transaction
1970-01-01 00:03:07 | DEBUG | Forwarding 1 transaction
1970-01-01 00:03:07 | DEBUG | Waiting for result
1970-01-01 00:03:07 | DEBUG | Read back transactions
1970-01-01 00:03:07 | DEBUG | Partial transaction
1970-01-01 00:03:07 | DEBUG | Detected data from client #0
1970-01-01 00:03:07 | INFO  | Disconnected client #0

Of course, this will not do anything because no data has been sent.... Now I do

python cli.py --connection ctp6_connections.xml --verbose status

and the following lines goes on for eternity until CTRL-C is pressed:

1970-01-01 00:06:03 | DEBUG | Read back transactions
1970-01-01 00:06:03 | DEBUG | Processing full transaction
1970-01-01 00:06:03 | DEBUG | Waiting for result
1970-01-01 00:06:03 | DEBUG | Read back transactions
1970-01-01 00:06:03 | DEBUG | Processing full transaction
1970-01-01 00:06:03 | DEBUG | Waiting for result
1970-01-01 00:06:03 | DEBUG | Read back transactions

Then, I go to the xmd mode and did this on ayinger:

connect mb mdm -debugdevice deviceNr 2
dow /scratch/tapas/cms-calo-layer1/ctp6_fe_uart_ipbus/payload.elf
run
read_uart

and run python cli.py --connection ctp6_connections.xml --verbose status
This command crashes everytime it runs ( I will open a seprate issue for this), but
in the mean time at the xmd prompt, these few lines shows up:

Partial t1970-01-01 00:00:00 | DEBUG | recv 4 
ransaction
1970-01-01 00:00:16 | DEBUG | Processing full transaction
1970-01-01 22:11:12 | DEBUG | Processing transaction 000
1970-01-01 22:14:04 | DEBUG | ==> IPBUS_NIREAD   nwords: 0 @ addr: 0
1970-01-01 00:00:16 | DEBUG | Partial transaction
1970-01-01 00:00:00 | DEBUG | sent 4

Nothing changes at the /tmp/softipbus-forward prompt. It stays

1970-01-01 00:10:00 | DEBUG | Forwarding 1 transaction
1970-01-01 00:10:00 | DEBUG | Waiting for result

Here is my question:
-) What am I missing ?
-) How do correctly know that I send data and read it back ?

@tsarangi @nwoods
The VMEStream executable vme2fd needs to be tested with the hardware. A description of the usage of vm2fd is in the Wiki: https://github.com/uwcms/cms-calo-layer1/wiki/The-oRSC#vmestream.

The vmestream_echo_test runs on the oRSC back-end: https://github.com/dabelknap/cms-calo-layer1/tree/master/orsc_be_vmestream_echo_test. It takes data from VME (sent from vme2fd) and resends it back to vme2fd. From the VMEPC, a sample session should look like this:

$ ./vme2fd orsc_send orsc_recv

In a separate terminal,

$ echo "put that in your pipe and smoke it" > orsc_send

The string should be read by vme2fd, sent to the oRSC via VME, then sent back to vme2fd. The string should then appear in the output named pipe.

$ cat osrc_recv
put that in your pipe and smoke

Notice that the string gets truncated. This is because vme2fd assumes 32-bit words (4 characters), and it does not transmit a word until it is completed.

Without Linux on oRSC-BE, we need standalone code to receive uart signals from FE. First, we are trying to replicate orsc_fe_spi_echo_test but with uart.

Hi guys (@jtikalsky @mathiasblake @ekfriis @tsarangi @dabelknap),
I spent a few hours debugging this today and have come up with no solutions. I am following twiki instructions and previous steps for building the backend server binary and starting a new server appear to be working successfully. I have tried reverting to an old version of previously working bitfiles/code that tapas had saved in a tarball to no success.
Any suggestions on other things I can do to debug would be fantastic. I have read through most of the issue-logs on github at this point... and nothing jumped out at me to check. (Error message below.)

(from twiki)
/usr/bin/python cli.py --connection ctp6_connections.xml --verbose status 1 5 12

And get this error:
/usr/bin/python cli.py --connection ctp6_connections.xml --verbose status 1 5 12
INFO:main:Setting up connection: file://ctp6_connections.xml
19-02-14 15:28:58.372356 [0x6002290] ERROR - ASIO reported a Timeout in TCP callback
Traceback (most recent call last):
File "cli.py", line 170, in ?
commands[args.command](hw, args)
File "cli.py", line 49, in do_status
status_flags = api.status(hw, args.links)
File "/afs/hep.wisc.edu/cms/ojalvo/tapasCTP6/onsoft_23Oct2013/ctp6commander/api.py", line 156, in status
hw.dispatch()
RuntimeError: St9exception

For the record, ctp6_connections.xml is:

I am trying to replicate what Maria did before Evan left. It is described here https://github.com/uwcms/cms-calo-layer1/wiki/The-CTP6#loading-the-front-end-payload-with-xmd

I cannot get the make payload to work, I get:

XMD% ERROR: Failed to Open JTAG Cable
Cable target is not connected to the host

The driver is fine, the green light is on on the USB-JTAG box. Everything looks plugged in. The switch is down (I tried both positions) on the JTAG input. I traced
the USB cable back to l1ts-rct-01, that seems fine, all the flying leads seem to be plugged in to the ribbon cable. Any suggestions?

Am I missing some critical step?

Thanks,
Pam

Should be doable with orsc_fe_spi_echo_test and orsc_be_spi_echo_test.

How to get the FPGAs to configure, with XMD:

xmd
fpga -f /afs/hep/ecad/mathias/orsc_bits/top_fe.bit -debugdevice deviceNr 2
rst
fpga -f /afs/hep/ecad/mathias/orsc_bits/top_be.bit -debugdevice deviceNr 1
rst

More XMD stuff to follow.

This is the analog of @nwoods 's task in #4, but should be done by @dabelknap at Previssin 904.

Austin, please give it a shot! We need to figure out exactly what bitfiles we need to load onto the FPGAs - see my out-of-band mail to Tom & Mathias.

I don't think there is a need for it to be in a separate repo, since it will never be used outside the context of the oRSC. We can move it to a subdirectory in cms-calo-layer1, like spistream [1]

[1] https://github.com/uwcms/cms-calo-layer1/tree/master/spistream

@dabelknap

Mathias and I have managed to make the oRSC echo test work at Chamberlin. I have committed all my changes to the master branch.
See 070d4b6

Look at the README file to see how to configure and test it in 904.
https://github.com/uwcms/cms-calo-layer1/blob/master/orsc_be_uart_echo_test/README.md

cc @dabelknap

This should sit on the oRSC FE, read a stream via VMEStream and echo it back. Will allow validation of VMEStream.

We need to update the oRSC XMLs and BSPs to whatever the latest development is from Mathias.

@tsarangi @dabelknap @nwoods

Please make sure you can compile the as described in the README

The orsc_(fe|be)_spi_echo_tests should be able to do this. We first need to update the oRSC BSPs and HW xmls - see #5.

I'm not sure if the oRSC FE firmware has the SPI already in it, we need to double check with Mathias.

I think it got killed by a Quattor update. I can't find it with yum in SLC - I'll keep looking, but any tips are appreciated @jtikalsky

We (@dabelknap and @pklabbers and @mcepeda) are trying to write to a single address from the VME PC and read it out of the oRSC backend via XMD.

We use a very old tool called vmedia which gives you a console which lets you peek/poke into vmedia. Steps to do this, following [1]:

1. connect to ayinger (this has the VME PC)
2. get the latest cms-calo-layer1 (maybe create a fresh area)
3. "make orscbebits" - uploads the latest bitfile in cms-calo-layer1/bitfiles to the ORSC backend (double check this is the correct one)
4. the lights are no longer blinky on the front
5. export CAEN_LINK=1 to use lower crate (double check this is what you want)
6. source /cms/cmslab/trigger_upgrade/cactusprojects/rct/environment.sh to get vmedia binary in PATH
7. run vmedia, then "A32" (32-bit addressing) then "lpoke 1 0000 DEADBEEF" (poke 1 long word [32-bits] into address 0000 with value DEADBEEF)
8. vmedia returns an error - expected behavior: it should not
9. Make sure the XMD debugger is hooked up between ayinger and the oRSC. Then connect via XMD to the ORSC backend connect mb mdm -debugDevice deviceNr 1;
10. expected behavior: see DEADBEEF when you mrd 0x00000000

NB that after flashing the bitfile with XMD, the CAEN controller DTK (data transfer ack) LED is lit, when it shouldn't be according to our thinking. Perhaps this is a clue.

cc @mathiasblake @nwoods @tsarangi

[1] https://twiki.cern.ch/twiki/bin/view/CMS/RCTAtChamberlin

This problem was discovered in an attempt to run orsc_be_vmestream_echo_test on the oRSC BE (Issue #8).

The software was loaded onto the oRSC BE using the following steps (assuming everything is compiled):

$ cd cms-calo-layer1/orsc_be_vmestream_echo_test
$ xmd
XMD$ connect mb mdm
XMD$ dow payload.elf
XMD$ read_uart
XMD$ run

There is a xil_printf statement just after the start of main() (https://github.com/uwcms/cms-calo-layer1/blob/master/orsc_be_vmestream_echo_test/src/vmestream_echo.c#L27), and it is never reached.

Stepping through the Assembly (payload.S) with the debugger, I found what appears to be the problem. Before main is called, it enters a function called __init, which in turn calls frame_dummy. When end of frame_dummy is reached, instead of returning to __init, it jumps to an address outside of the address range of the program.

XMD% stp
     19C:   B60F0008  rtsd     r15, 8
     1A0:   3021001C  addik    r1 , r1 , 28   # end of <frame_dummy>

XMD% stp
    FEF4:   00000000  add      r0 , r0 , r0   # jumps outside the address range of the program

XMD% stp
    FEF8:   00000000  add      r0 , r0 , r0 
...

When nothing is found at the address 0xfef4, it move to the next address. It finds nothing here so it continues in this way to address 0xfffc.

...
XMD% stp
    FFFC:   00000000  add      r0 , r0 , r0

XMD% stp
   10000:   B0000000  imm      0
   10004:   B8080050  brai     80

XMD% stp
      50:   B0001000  imm      4096            # back to the beginning
      54:   31A009A0  addik    r13, r0 , 2464

At this point, the program returns to the beginning, and we are stuck in an infinite loop.

See https://svnweb.cern.ch/trac/cactus/ticket/376

We need to make the corresponding memory map XML to the CTP example [1] for the oRSC. Here is the relevant oRSC FE HDL from Matthias:

 when x"10008000" => fpled1grnena_b_reg  <= BRAM_WRDATA_A(0)   ;

means that a register that drives the LED green is at address 0x10008000.

        when x"10008000" => fpled1grnena_b_reg  <= BRAM_WRDATA_A(0)   ;
        when x"10008004" => fpled1redena_b_reg  <= BRAM_WRDATA_A(0)   ;
        when x"10008008" => fpled2grnena_b_reg  <= BRAM_WRDATA_A(0)   ;
        when x"1000800C" => fpled2redena_b_reg  <= BRAM_WRDATA_A(0)   ;
        when x"10008010" => fpled3grnena_b_reg  <= BRAM_WRDATA_A(0)   ;
        when x"10008014" => fpled3redena_b_reg  <= BRAM_WRDATA_A(0)   ;
        when x"10008018" => s6yellowled_reg     <= BRAM_WRDATA_A(0)   ;
        when x"1000801C" => s6orangeled_reg     <= BRAM_WRDATA_A(0)   ;
        when x"10008020" => s6greenled_reg      <= BRAM_WRDATA_A(0)   ;
        when x"10008024" => lvshftregdata_reg   <= BRAM_WRDATA_A(0)   ;
        when x"10008028" => lvmuxregclk_reg     <= BRAM_WRDATA_A(0)   ;
        when x"1000802C" => lvdelayregclk_reg   <= BRAM_WRDATA_A(0)   ;
        when x"10008030" => lvshftregrst_reg    <= BRAM_WRDATA_A(0)   ;
        when x"10008034" => lv2eclspare1_reg    <= BRAM_WRDATA_A(0)   ;
        when x"10008038" => lv2eclspare2_reg    <= BRAM_WRDATA_A(0)   ;
        when x"1000803C" => lvwstb_b_reg        <= BRAM_WRDATA_A(7 downto 0) ;
        when x"10008040" => avagorst_b_reg      <= BRAM_WRDATA_A(0)   ;
        when x"10008044" => ckarst_reg          <= BRAM_WRDATA_A(0)   ;
        when x"10008048" => ckcrst_reg          <= BRAM_WRDATA_A(0)   ;
        when x"1000804C" => k7_program_b_reg    <= BRAM_WRDATA_A(0)   ;

[1] https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/etc/ctp6_fe.xml

This standalone package should live on the oRSC backend, receive data over VMEstream and forward it via SPIStream to the backend.

Will be 1/2 based on the validation code in #8, and half on orsc_be_spi_echo_test.

orsc_fe_spi_echo_test is failing when it reaches XIntc_Connect in spiecho.c. While stepping through payload.S with the debugger, I get in main:

XMD% stp
    65A0:   B0000000  imm      0
    65A4:   30E03970  addik    r7 , r0 , 14704 // 3970 <XSpi_InterruptHandler>

XMD% stp
    65A8:   B0000000  imm      0
    65AC:   31007324  addik    r8 , r0 , 29476 // 7324 <SpiInstance>

XMD% stp
    65B0:   B000FFFF  imm      -1
    65B4:   B9F4A2BC  brlid    r15, -23876 // 870 <XIntc_Connect>
    65B8:   30C00001  addik    r6 , r0 , 1

XMD% stp
     870:   DEADBEEF

XMD% stp
     874:   DEADBEEF

870 is supposed to read 870: 3021ffe0 addik r1, r1, -32 in <XIntc_Connect>.

Using commit dcaef28

Hi @jtikalsky, can you put on the wiki [1] instructions on how to recover from the JTAG cable driver getting wedged on ayinger/spaten so Nate/Tapas can fix it when you aren't around?

Do you know what causes the problem?

cc @nwoods @tsarangi

[1] https://github.com/uwcms/cms-calo-layer1/wiki

We may need to install Petalinux, Xilinx, etc.

See instructions here:

https://github.com/uwcms/cms-calo-layer1/wiki/UsingXilinxGUI

@nwoods or @tsarangi, please give a shot at importing the projects into the workspace - this is the only thing Evan-specific. Once they are imported Mathias' wizardry is unleashed.

This is low priority.

You need to:

1. Compile the softipbus TCP server for Petalinux. See the README here [1] for details.
2. Upload the softipbus-forward server to /tmp on the CTP6 backend. Get the appropriate IP address and ssh password (user is root) from Jes.
3. SSH and run /tmp/soft-ipbus on the BE (leave it running)
4. Compile ctp6_fe_uart_ipbus and upload it (make upload in its directory)
5. Use ctp6commander [2] CLI to check the link status
6. Document these instructions in the README.md

[1] https://svnweb.cern.ch/trac/cactus/browser/trunk/cactuscore/softipbus/README.md
[2] https://github.com/uwcms/ctp6commander

It is only used for writing a single word. We need to implement something which calls the block write [1], and update all calls to read/write.

Also, OrscEmulator [2] does not emulate the behavior or read/write correctly and should be changed (which will break the tests).

        CAENVME_BLTWriteCycle
        -----------------------------------------------------------------------------
        Parameters:
                [in]  Handle    : The handle that identifies the device.
                [in]  Address   : The VME bus address.
                [in]  Buffer    : The data to be written to the VME bus.
                [in]  Size      : The size of the transfer in bytes.
                [in]  AM        : The address modifier (see CVAddressModifier enum).
                [in]  DW        : The data width.(see CVDataWidth enum).
                [out] count     : The number of bytes transferred.
        -----------------------------------------------------------------------------
        Returns:
                An error code about the execution of the function.
        -----------------------------------------------------------------------------
        Description:
                The function performs a VME block transfer write cycle.

[1] http://jazz.physik.unibas.ch/~tikorost/work/TAPSelectronics/Software/vmelib/CAENVMElib.h
[2] https://github.com/uwcms/cms-calo-layer1/blob/master/VMEStream/src/vmestream/OrscEmulator.cc#L67