Coder Social home page Coder Social logo

optohybridv3's Introduction

Optohybrid v3 Firmware

doc/images/block_diagram.svg

Directory Hierarchy:

├── optohybrid_registers.xml -- XML Address Table
├── Hog                      -- Hog build system submodule
├── doc                      -- documentation/images
├── tools                    -- Python scripts
├── src                      -- Source code, constraints, IP cores
│   ├── optohybrid_top.vhd
│   ├── cluster_building     -- Cluster building
│   ├── control              -- Control
│   ├── gbt                  -- GBTx Tx/Rx
│   ├── ip_cores             -- Virtex-6 IP Cores
│   ├── ip_cores_a7          -- Artix-7 IP Cores
│   ├── mgt                  -- Transceivers
│   ├── pkg                  -- VHDL Packages
│   ├── trigger              -- S-bits
│   ├── ucf                  -- Constraints
│   ├── utils                -- Shared/misc utility modules
│   └── wb                   -- Wishbone
├── Top                      -- Hog top levels
│   ├── oh11
│   └── oh21
└── README.org               -- This readme

Building the firmware

This firmware is using the HOG framework as a build system:

A Makefile is provided that should make firmware building more straightforward. You can run for example:

make update      # update from the XML file
make oh21-200    # builds the 200t version of the  ge21 firmware
make oh21-75     # builds the 75t version of the ge21 firmware
make oh21 -j2    # builds both versions of the firmware in parallel with 2 jobs
make oh11        # builds long and short versions of the GE11 firmware
make all -j8     # update from the XML file and build all firmware flavors, do it all in parallel :)

n.b. Support for GE1/1 through a planAhead wrapper is in development (but still experimental)

Alternatively, you can build the firmware manually, for example, the firmware can be built as:

git clone https://github.com/cms-gem-daq-project/OptoHybridv3
cd OptoHybridv3
git submodule update --init --recursive
Hog/CreateProject.sh oh21
Hog/LaunchSynthesis.sh oh21
Hog/LaunchImplementation.sh oh21

Updating the address table or firmware version

The firmware can be updated from the address table by running make update from the root of the repository.

If you want to increment the version, just run make with an argument, e.g.

make update version=03.04.01

Versioning

Releases are labeled as version_major.version_minor.version_patch.hardware_id

i.e. in HARDWARE_VERSION XY

x=chamber type (0=GE11 short, 1=GE11 long, 2=GE21) y=hardware generation

Specifically:

StationOH VersionChamber VariantTag
GE1/1v3BShort0B
GE1/1v3CShort0C
GE1/1v3CLong1C
GE2/1v12A
GE2/1v22B

TMR

TMR for different modules can be configured on and off to allow for customization between GE2/1 and GE1/1.

It is controlled by the platform specific hardware package, viz. hardware_pkg_ge11.vhd and hardware_pkg_ge21.vhd

For example:

constant EN_TMR : integer := 1;

constant EN_TMR_TRIG_FORMATTER     : integer := 1*EN_TMR;
constant EN_TMR_GBT_DRU            : integer := 1*EN_TMR;
constant EN_TMR_SBIT_DRU           : integer := 1*EN_TMR;
constant EN_TMR_SOT_DRU            : integer := 1*EN_TMR;
constant EN_TMR_CLUSTER_PACKER     : integer := 0*EN_TMR;

You can change a 1 to a 0 to to disable TMR for that module, or EN_TMR from 1 to 0 to disable TMR entirely.

LED Indicators

Optohybrid v3 LED Assignments
led[15] = (rsvrd, always on)led[7] = Cluster Rate >= 10,000,000 Hz
led[14] = Clock, dividedled[6] = Cluster Rate >= 1,000,000 Hz
led[13] = MGTS ready, blinkingled[5] = Cluster Rate >= 100,000 Hz
led[12] = GBT Ready, blinkingled[4] = Cluster Rate >= 10,000 Hz
led[11] = GBT Request flashled[3] = Cluster Rate >= 1,000 Hz
led[10] = L1A Flashled[2] = Cluster Rate >= 100 Hz
led[9] = Resync Flashled[1] = Cluster Rate >= 10 Hz
led[8] = BC0 Flashled[0] = Cluster Rate >= 1 Hz

The “right side” LED indicators provide a logarithmic progress bar indicator which provides a convenient way to monitor the rate of incoming clusters received by the Optohybrid.

Prior to the receipt of the first S-bit (after a reset or resync), the progress bar will show a strobing “cylon” pattern to indicate an idle state.

HDMI Output

The Optohybrid has an HDMI connector connected to differential pairs of the FPGA.

The functionality is programmable by changing firmware, but in the current configuration they provide a programmable way to output S-bits for use in the cosmic ray test stand.

Each conductor can be individually programmed into one of 4 modes:

ModeDescription
Mode 0Each signal was a single VFAT (set by selN)
Mode 1Each signal was the OR of three VFATs in an ieta row (row is set by selN)
Mode 2Each signal was the OR of four VFATs in an iphi half column
(e.g. 0-3, 4-7, 8-11, 12 15, 16-19, 20-23) (phi half is set by selN)
Mode 3Disabled (all outputs driven to zero)

The registers you will want to use to control the HDMI output are:

  • GEM_AMC.OH.OH{X}.FPGA.CONTROL.HDMI.SBIT_SEL{0-7}
  • GEM_AMC.OH.OH{X}.FPGA.CONTROL.HDMI.SBIT_MODE{0-7}

The assignments of these signals in the HDMI cable are:

NET "ext_sbits_o[0]" LOC = H17; # tmds clk p
NET "ext_sbits_o[1]" LOC = G17; # tmds clk n
NET "ext_sbits_o[2]" LOC = J16; # tmds d2  p
NET "ext_sbits_o[3]" LOC = J17; # tmds d2  n
NET "ext_sbits_o[4]" LOC = L14; # tmds d1  p
NET "ext_sbits_o[5]" LOC = L15; # tmds d1  n
NET "ext_sbits_o[6]" LOC = M17; # tmds d0  p
NET "ext_sbits_o[7]" LOC = M18; # tmds d0  n

Control

In the version 3 firmware, all Optohybrid registers are accessible through a Wishbone bus with an address table defined in single XML file, optohybrid_registers.xml (accessible in Github at ./optohybrid_registers.xml):

This file can be used in the same way as the backend (CTP-7) registers file and with the same tools, e.g. rw_reg.py. The contents of this file should be embedded into the CTP-7 firmware at a sub-address which is allocated to the Optohybrid.

16 bits are allocated for the Optohybrid address space. Within this 16-bit address space, the bits are subdivided according to the scheme:

– highest 4 are used as the module ID (wishbone slave #)

– lowest 12 bits are used by the wishbone splitters as individual register addresses

Thus we can have up to 16 slaves, and 4096 endpoints per slave. Presently we use only 6 slaves with a firmware that is nearly finalized, so it is expected that 16 slaves will not be a limiting factor.

TTC Synchronization

The Optohybrid has a local accumulator which increments bunch crossing number (bxn) and produces a locally generated bx0 flag.

In order for the two flags, (1) locally generated and (2) received from GBTx, to be in sync a programmable parameter (bxn offset) is provided which allows you to set an “offset” which is the value which will be assumed by the bxn counter after ttc resync.

The synchronization of these two flags can be monitored by checking the status of bxn_sync_error in the OH status register (n.b. that bx0_sync_error is NOT suitable for this, as it is a 1bx wide pulse useful for internal counters when the bx0 is received, while bxn_sync_error will persist for the entire orbit).

When the Optohybrid is correctly timed in to the TTC system, bxn_sync_error should be 0. A software routine in the CTP-7 to find the correct bxn offset should be easily designed.

A counter in the counters module counts the number of bx0 sync errors since the last resync or hard-reset.

The anticipated synchronization process is:

  1. From the backend electronics, send a BC0 is to all Optohybrids
  2. Inside of the Optohybrid, adjust the BXN offset until bxn_sync_error is read to be 0
    • This indicates that the local OH bxn counter matches the remote counter
  3. Inside of the backend electronics, adjust the delays of incoming trigger data until the 0x50 (BC0) frame markers are aligned between all Optohybrids

optohybridv3's People

Contributors

andrewpeck avatar thomaslenzi avatar jsturdy avatar evka85 avatar lpetre-ulb avatar

Stargazers

 avatar

Watchers

James Cloos avatar  avatar  avatar  avatar  avatar

optohybridv3's Issues

Node names mismatch

After the 3.1.5.1C release the node names for registers GEM_AMC.OH.OHX.FPGA.TRIG.CNT.VFATY_SBITS has been changed to
GEM_AMC.OH.OHX.FPGA.TRIG.CNT.VFATY.
Please revert this change

Feature Request: Optohybrid Firmware Release Register

Please change OHv3 firmware release info registers to match CTP7 registers. e.g. for the CTP7 we can see easily the release vX.Y.Z where X is the Major, Y is the Minor, and Z is the build, and the date in four dedicated registers:

eagle60 > kw RELEASE
0x6640000c None         GEM_AMC.GEM_SYSTEM.RELEASE
0x6640000c r    GEM_AMC.GEM_SYSTEM.RELEASE.MAJOR                        0x00000003
0x6640000c r    GEM_AMC.GEM_SYSTEM.RELEASE.MINOR                        0x00000002
0x6640000c r    GEM_AMC.GEM_SYSTEM.RELEASE.BUILD                        0x00000007
0x66400010 r    GEM_AMC.GEM_SYSTEM.RELEASE.DATE                         0x20171211

However the OH FW gives just the date as three separate registers:

0x65000004 None         GEM_AMC.OH.OH0.FPGA.CONTROL.RELEASE
0x65000008 r    GEM_AMC.OH.OH0.FPGA.CONTROL.RELEASE.YEAR                0x00002018
0x65000008 r    GEM_AMC.OH.OH0.FPGA.CONTROL.RELEASE.MONTH               0x00000001
0x65000008 r    GEM_AMC.OH.OH0.FPGA.CONTROL.RELEASE.DAY                 0x00000016

Definitely prefer the approach used by the CTP7. Could we change the OH to have this layout?

[feature request] uhal compatible address table

In order to have the uhal address tables dynamically select the appropriate OH version (see evka85/GEM_AMC#39), we need to implement the FPGA block in the uhal_amc_<card>>_link<xx>.xml file as a module, as understood by the uhal xml parser.

For this to work, all attributes that are not uhal understood (id, address, mask, permission, description, mode, parameters, tags) should be dropped, and the FPGA node should be renamed top
This would make most sense to have a parser to dump the uhal address table from the standard address table.

Thus, the new optohybrid_registers_uhal.xml would be like:

<?xml version="1.0" encoding="ISO-8859-1"?>

<!-- START: GLOBALS DO NOT EDIT -->
<!DOCTYPE node [
<!ENTITY NUM_VFATS_PER_OH "24">
<!ENTITY VFAT_BITMASK  "0xffffff">
]>
<!-- END: GLOBALS DO NOT EDIT -->

<!--  The tags attribute is a free test attribute which meaning is defined by the uHAL user -->
<node id="top">
    <!--Control module -->
    <node id="CONTROL"  address="0x0000"
...
    </node>
</node>

Then in the GEM_AMC uhal, xml file, we'd have:

...
<node id="OH" address="0x400000" >
  <node id="OH0" address="0x0" >
    <node id="FPGA" address="0x0" module="file://optohybrid_registers_uhal.xml"/>
    <node id="GEB" address="0x100000">
    ...
  </node>
</node>
...

If a rework were in the cards, I'd suggest to parse all of the special FW generating attributes using the uhal supported attributes, but also understand inertia

Trigger links fail to initialize properly

from Brian:

Evaldas and I noticed some interesting behavior last night. It appears to be an intermittent problem. The trigger module in the OHv3 FW does not seem to always initialize properly after the FW is loaded.

The steps to repeat the problem are:

  1. Program the GBTs
  2. Reset the SCA
  3. Program the FPGA with the PROMless functionality
  4. Issue a Trigger Module Reset
  5. Read “kw GEM_AMC.TRIGGER.OHX.LINK” for OH X and check the missed comma counters

We saw that in several cases the missed comma counter registers are always overflow.

http://cmsonline.cern.ch/cms-elog/1055454

This happened maybe 2-3 times in a row and then we started trying to count the occurrence at this it was “well behaved” for a little bit only occurring 2 times in 30 tries of the above procedure. But it happened 2-3 times beforehand also.

Maybe this is something worth investigating further?

We found that overnight the link once stable, remained stable (e.g. missed comma counters were 0x0 after 8-12 hours):

http://cmsonline.cern.ch/cms-elog/1055566

So it really seems to be bad either “good” or always bad after the FW is loaded.

GE2/1 FW communication

GE2/1 FW communication gradually deteriorates with each new release. The last release which actually can be used is 3.2.4.2A. With this release when programming an OH, the communication is possible in most of the cases (although not always, a powercycle or another programming attempt might be required). The FPGA however gets the program properly. @evka85 can provide the chipscope logs which shows bitslippling in the GBT<->OH communication.

Sbit partitions order has changed

After upgrading the firmwares to the latest versions (CTP7 - 3.8.3 ; OH - 3.2.3C), the Sbits started to be incorrectly mapped, but only for the VFAT part. I traced the change to the sbit_cluster_packer where invert_partitions is no longer defined :

      `ifdef invert_partitions  // need to make a choice about whether strip-0 is in partition 0 or 7
        partition[0] <= {vfat_s1[23], vfat_s1[15],  vfat_s1[7]};
        partition[1] <= {vfat_s1[22], vfat_s1[14],  vfat_s1[6]};
        partition[2] <= {vfat_s1[21], vfat_s1[13],  vfat_s1[5]};
        partition[3] <= {vfat_s1[20], vfat_s1[12],  vfat_s1[4]};
        partition[4] <= {vfat_s1[19], vfat_s1[11],  vfat_s1[3]};
        partition[5] <= {vfat_s1[18], vfat_s1[10],  vfat_s1[2]};
        partition[6] <= {vfat_s1[17], vfat_s1[9 ],  vfat_s1[1]};
        partition[7] <= {vfat_s1[16], vfat_s1[8 ],  vfat_s1[0]};
      `else
        partition[0] <= {vfat_s1[16], vfat_s1[8 ],  vfat_s1[0]};

As a consequence, the formula used in the ctp7_modules is no longer correct :

int vfatObserved = 7-int(sbitAddress/192)+int((sbitAddress%192)/64)*8;

There are two possible fixes :

  • Change the formula to follow the new mapping :
int vfatObserved = int(sbitAddress/192)+int((sbitAddress%192)/64)*8;
  • Compile the firmware with the previous invert_partitions option set.

Tag 3.1.5.1C on master branch

While rebasing the ULB TDC on the master branch, I noticed that the tag 3.1.5.1C (commit 6457816) was not pushed on master branch (currently commit 5b2160b).

Is it the expected behavior ? On which commit should I rebase the TDC to be up to date ?

Incorrect VFAT mapping

In S-bits.vhd, need to correct the mapping of VFATs into the cluster packer to reflect the "electronics" slot numbering, to the "logical" slot numbering (not the same as the "software" numbering)

GE1/1 Clock Phase

Right now GE1/1 uses a 320MHz clock that gets divided down to 40MHz... this will introduce some variable latency into the trigger path. Either a 40MHz clock should be used instead (which requires switching to the phase-shiftable clock output, which was seen to have worse jitter performance) or some mechanism to align the clocks in firmware.

Discussion on VFAT Reset Behavior

Brief summary of issue

Discussion started by @andrewpeck via email, migrating it here. We are discussing the behavior of GEM_AMC.OH.OH0.FPGA.CONTROL.VFAT.RESET in the FW and how it could be used.

Types of issue

  • Bug report (report an issue with the code)
  • Feature request (request for change which adds functionality)
  • Discussion

Proposed Behavior

There is a register that already does a "pulse" reset of the VFATs: GEM_AMC.OH.OH0.FPGA.GBT.FORCE_RESET_VFATS.

So I guess writing 0x1 (anything?) to this register causes a pulse reset to be sent to all lines that lasts just one clock cycle so the VFAT's will not be held in hard reset?

@andrewpeck proposes the following (paraphrasing):

Change the register you have been using, GEM_AMC.OH.OH0.FPGA.CONTROL.VFAT.RESET, into a 12 bit mask such that writing 0x1 to the n^th bit it asserts and holds that reset line in the hard reset state (voltage high?) until 0x0 is written to that bit. This gives the functionality to individually reset a given reset line and keep the VFATs on that line in the reset state. Then we keep FORCE_RESET_VFATS as the global reset which is pulsed.

@andrewpeck also mentions supporting that this pulsed reset is part of the GBT node because it emulates the different TTC commands which can be received from the GBT:

eagle60 > readKW GEM_AMC.OH.OH0.FPGA.GBT.FORCE
0x65010014 w GEM_AMC.OH.OH0.FPGA.GBT.FORCE_L1A
0x65010018 w GEM_AMC.OH.OH0.FPGA.GBT.FORCE_BC0
0x6501001c w GEM_AMC.OH.OH0.FPGA.GBT.FORCE_RESYNC
0x65010020 w GEM_AMC.OH.OH0.FPGA.GBT.FORCE_RESET_VFATS

Current Behavior

Right now writing 0x1 to GEM_AMC.OH.OH0.FPGA.CONTROL.VFAT.RESET will raise a voltage level and trigger the hard reset for all vfat3 chips (well reset lines) on the detector.

The chips will be held in hard reset until 0x0 is written to GEM_AMC.OH.OH0.FPGA.CONTROL.VFAT.RESET.

Context (for feature requests)

Deciding how the hard reset from the OH should be interpreted.

@andrewpeck @evka85 @mexanick @jsturdy

HDMI mode/sel question

@andrewpeck do I read correctly from the documentation that there are 8 HDMI s-bit output mode and 8 HDMI s-bit output select settings?
Previously mode was common, and sel would be chosen based on the mode, however if I understand correctly, I can now configure the HDMI output to have each of the 8 signals independently, i.e., signal 1 could be a specific VFAT, while signal 2 could be the iEta row containing that VFAT.
Can you confirm if this is correct (and desired @bdorney)?

Magically uninverted S-bit pair on one OH pair

On one pair (VFAT 7, pair 2 -- in electronics numbering counting from 1-24, 0-7) appears by all accounts to be polarity swapped.

However, when unswapping in the firmware we see that at least on the only test stand (904 cosmic stand) where this can be checked at the moment, the pair is inverted in logic.

To correctly function, this inverted pair has to be marked as normal and NOT inverted by firmware. It is completely known what is causing this problem.

Fixed in 0fcfc6a

Iterating on OH FW Version Registers

Follow-up to #3

In OH FW version 3.0.8.A the FW version is sort of shown as a single register:

0x65000004 None         GEM_AMC.OH.OH0.CONTROL.RELEASE
0x65000008 r    GEM_AMC.OH.OH0.CONTROL.RELEASE.DATE                     0x20180207
0x6500000c r    GEM_AMC.OH.OH0.CONTROL.RELEASE.VERSION                  0x0300080a

However from the SW point of view it would be easier if the OH reported the firmware version identically to how the CTP7 does:

0x6640000c r    GEM_AMC.GEM_SYSTEM.RELEASE.MAJOR                        0x00000003
0x6640000c r    GEM_AMC.GEM_SYSTEM.RELEASE.MINOR                        0x00000003
0x6640000c r    GEM_AMC.GEM_SYSTEM.RELEASE.BUILD                        0x00000006
0x66400010 r    GEM_AMC.GEM_SYSTEM.RELEASE.DATE                         0x20180120

So for example for v3.0.8.a request that the OH reports:

0x65000008 r    GEM_AMC.OH.OH0.CONTROL.RELEASE.DATE                     0x20180207
0x6500000c r    GEM_AMC.OH.OH0.CONTROL.RELEASE.VERSION                  0x0300080a
0x6500000c r    GEM_AMC.OH.OH0.CONTROL.RELEASE.MAJOR                     0x00000003
0x6500000c r    GEM_AMC.OH.OH0.CONTROL.RELEASE.MINOR                     0x00000000
0x6500000c r    GEM_AMC.OH.OH0.CONTROL.RELEASE.BUILD                      0x00000008
0x6500000c r    GEM_AMC.OH.OH0.CONTROL.RELEASE.GENERATION          0x0000000a

Where for OHv3b the GENERATION will report 0xb instead.

This will make getting the FW version and checking for functionality easier in SW (e.g. CNT.VFATN_SBITS doesn't exist before FW v3.0.7.a)

@andrewpeck

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.