sixtrack / sixtrack

SixTrack – 6D Tracking Code

SixTrack

SixTrack is a single particle 6D symplectic tracking code optimized for long term tracking in high energy rings. It is mainly used for the LHC for dynamic aperture studies, tune optimization, and collimation studies.

Authors

F. Schmidt (DESY, CERN), J.D. Andersson, R. Assman, J. Barranco, V.K. Berglyd Olsen, C. Bracco, R. Bruce, R. De Maria, M. D'Andrea, M. Fiascaris, M. Fjellstrom, H. Grote, K. Heinemann, F. James, K. Koelbig, R. Kwee-Hinzmann, Y. Levinsen, E. Mcintosh, A. Mereghetti, D. Mirarchi, K. Paraschou, T. Persson, V. Previtali, E. Quaranta, H. Ranshall, S. Redaelli, A. Rossi, A. Santamaria, K. Sjobak, Y. Sun, C. Tambasco, M. Vaenttinen, J.F. Wagner, T. Weiler, J. Wretborn (CERN), M. Fitterer (FNAL, CERN), V. Gupta (Google Summer of Code), S. Kostoglou (NTUA, CERN), J. Molson (UMAN, LAL, CERN), A. Patapenka (NIU, CERN), G. Robert-Demolaize (BNL, CERN)

Core Devs: V.K. Berglyd Olsen, R. De Maria, A. Mereghetti, J. Molson, T. Persson, K. Sjobak.

License

Copyright 2019 CERN. This software is distributed under the terms of the GNU Lesser General Public License version 2.1, copied verbatim in the file LICENSE.md.

In applying this licence, CERN does not waive the privileges and immunities granted to it by virtue of its status as an Intergovernmental Organization or submit itself to any jurisdiction.

Reference

If you use SixTrack 5 in your publication, please cite:

• De Maria, R., et al. ‘SixTrack Version 5: Status and New Developments’. In Proceedings of IPAC 2019, 3200–3203. Melbourne, Australia: JACoW, 2019. DOI:10.18429/JACoW-IPAC2019-WEPTS043.

Quick Build

To build the standard release version of SixTrack, run the script cmake_six with no additional options. This will build the SixTrack 64 bit executable with the singletrackfile, zlib support, as well as the crlibm math library.

Resources

• Main Website: sixtrack.web.cern.ch/SixTrack
• Documentation: PDF | HTML

Source Code

• ChangeLog
• Development Cheat Sheet

Building SixTrack

• General Requirements and Build Options
• Further details covered in the build manual

Developer Tools

• A MAD-X/SixTrack output testing and comparison framework

sixtrack's Issues

New Wire model

DAFOR: Arguments for input/output files

For making things easier in the CMAKE

Implement generic rf-multipoles

Should fix also #5

RF multipole longitudinal kick problem

Issue:

RF multipoles (such as crab cavities) work on dpsv = deltaP/P0 = delta, RF cavities work on ejf = total energy

Why is this a problem:

There is a pile of auxiliary variables for the energy, updating them in different order causes accuracy problems
We may not actually be conserving quantities which should be conserved, by missing a factor beta_0 \approx 1-10{^-9}

Physics impact:

Long term tracking
Crab emittance growth studies?

Resources:

https://cds.cern.ch/record/1491234/files/RF-Note_final_1.pdf
http://ab-dep-abp.web.cern.ch/ab-dep-abp/LCU/LCU_meetings/2011/20110419/workoutline.pdf

Hints from Riccardo:

Pay attention to the definition of sigma - it is not really z!
sigma = s - beta0 c t
Something else in MadX
pz = something unreadable = \sqrt{(1+\delta)^2 + p_x^2 + p_y^2} should be conserved
Normalization of the voltage — a factor beta0 missing?
Hamiltonial / multipole expansion / assumptions in Panofsky-Wenzel theorem

My notes while reading the code and documentation:

It appears that the hamiltonian is of the form using delta and Delta s
However, the code uses sigmv for Delta s
According to TWIKI, sigmv is sigma = s-beta0_c_t, while Delta s = s - beta c t

Quicktest

Quicktest: An easy-to-use testing system.

Contents of the QuickTest folder:

• Script to run QuickTest: Optional arguments: List of path /reference to computational unit. Will make(_six) binaries using the current working copy, create DateTime-SHA1 folders, copy binary+input+canonical there, then run run_test (or default test if this is not in place: Just run the binary+full diff vs. provided canonicals). Do not accept if there are uncommited changes.
• Folders for tests: Contents:
  • input folder:
    • Contains the fort.2, fort.3, CollDB etc.
    • CR/Testing script (standard name: "run_test") to return no error when run and output files are copied
  • Reference folders: named as sorted list of options such as reference_option1_option2_etc. Contents
    • Canonicals: Containing a full copy of all (relevant) outputs
  • DateTime-SHA1: Binary+output from running the binary produce my test system.
  • Readme? Describe scenario few lines + author + which SHA1 was used for canonicals.

Collimation emittance is read from beam-beam block

The collimation block defines the opening of the collimators with respect to a certain emittance. This emittance is read from the beam-beam block, and the variables in the collimation block to specify the emittance are completely ignored.

Since the collimator openings are usually given with respect to an emittance of 3.5 um, and the beam-beam block should be run with the correct emittance (2.5 um for HL-LHC), it is currently impossible to run beam-beam and collimation at the same time without completely changing the input of the collimation block. Furthermore, this fact is not documented, which can lead to many errors.

I suggest that we bring back to life the mute variables in the collimation block for emittances.

Collimation block is not called for lattices without RF elements.

If I try and enable the collimation code using lattices without RF cavities (for example, the current FCC-hh) then only thin4d is called.

thin6d contains the calls to enable the collimation tracking.

Putting RF elements into the lattice enables thin6d to be called and thus collimation works.
If collimation is enabled in fort.3 would it be possible to force thin6d to be used?

Cleanup CRLIBM

Move the "liberic" files out of crlibm folder, remove fpucontrol.h, possibly get new CRLIBM. Merge before changing CRLIBM version!

typo in eq 99 physics manual (octupole)

reported from Adrian

from sympy import *
var('x,y',real=True)

x,y=var('x,y',real=True)
px=-re((x+I_y)__3)
py=im((x+I_y)**3)

DIST(ribution block)

Implement generic particle distribution generation:

• define examples for the input and we iterate on the specification
• the module will fill the default tracking arrays
• the hugenpart&stf&datamod will become default

Cleanup of random number handling and PARPRO

• Disentangle the meaning of "nran"
• Out-of-sync use of zfz RNG stream:
  • LINOPT, RESEX, SUBRE, SUBSEA skips crabs and BB blocks
  • ORD, PHASAD and MAINCR/"initialize_kicks" does not skip these elements
• In block PARPRO, consolidate the astuce if/else's for setting size parameters
  • Collimat's mmul=11, otherwise=20
  • nele/nblz sometimes different
  • beamgas...

Collimation block input checking: Number of packages

Problem: The collimation block maximally accepts 200 (?) packages of 64 particles each. However, if the user tries to ask for more packages, it tries to do so and ends up corrupting the memory.

Solution: Check that the input is valid, stop with an error message if it is not.

Update particle population through a pipe

In order to make it possible to run (for example) single-cavity collective effects simulations with multiple bunches, it is necessary to (1) send the current particle population to an external program, and (2) receive an updated particle population, possibly replacing the current population if a second bunch is to be loaded.

I am however not sure how this would interact with collimation...

The primary idea for implementation is to communicate the particles via a set of pipes. It would be best if the tracking variables particles are loaded/stored without converting to ASCII.

This module would be configured from a new block in fort.3 (PIPE maybe?), giving the names of the input/output FIFOs and possibly the element to use (or else just use tracking starting point).

subroutine for readin with round_near

for reading in floating point numbers, round_near is used by the DYNK and ELEN block. fround already presents a subroutine which only takes the field length and field as input, but it is limited by the maximum number of fields:

      double precision function fround(errno,fields,f)
      implicit none
      integer maxf
      parameter (maxf=30)

If a field is then longer than 30 characters, this would result in a non-zero terminated string. As round_near assumes a zero-terminated string the readin becomes incorrect.

Task: rewrite fround without field limitation.

Exit with non-zero status at error

In prror / abend, call stop with status other than 0

FMA implementation

Floating point div0, tbetax = 0 for some elements

Description as reported by Andrea Santamaria:
https://www.evernote.com/l/AHNAr9ej1jlK_rD0-TvprNn_bjbXfq7hJMY

Most likely, this is a bug in LINOPT.

SixTest run_pro should check that the test exists

If one by mistake puts a wrong folder name in the TESTS variable before running run_pro, this is not discovered. I then copies nothing, then attempts to run SixTrack which predictably crashes.

If run_pro (and run_kill?) would check that the folder actually exist before trying to run a test, this would be better.

SixTrack CR version has problems working with BTRFS file system

It seems that when running the SixTest checks for the CR version on top of the BTRFS file system, something goes wrong. After running several tests, mkdir, touch etc. fails claiming that the disk is full until the machine is rebooted. Afterwards, files may be missing from the SixTest folder.

We should run a test on a different machine to see if this is a machine specific problem or a file system bug/limitation...

Test thick6ddynk check_extras fails for ifort

As pointed out by Miriam in #81 , when running the test thick6ddynk with SixTrack compiled with ifort, the tests fail. The DUMP files show some difference in the last digit compared to the canonical (from gfortran).

CRLIBM issues: Compile with SSE2 support, drop x87

The function cos_rn() returns different results for different the same and large argument when compiled with different versions of GCC.

This can be avoided by using SSE2 math instead of x87, as this produces consistent results (consistent across compiler versions, and for gcc version 4.4.7 x87 and SSE agrees).

For gcc (crlibm), this is done by using the compilation flags "-mfpmath=sse -msse2".

dynk_parseFUN function type selection based on first three chars ONLY

This means it can't separate e.g. "LIN" and "LINSEG". Possible solution: Use trim and dynk_stringzerotrim.

Fort.3 ZIPIT block for BOINC

The block should contain a list of files to put in Sixout.zip. This would enable collimation etc. to run on BOINC. If no ZIPIT block present, we may want to have a "default list" of files? In case of no BOINC, the block shall not be valid (In DATEN, detect no BOINC flag and exit if ZIPIT block is present).

See: http://indico.cern.ch/event/476291/contribution/0/attachments/1209033/1762820/SixTrack_20160108.pptx

Assignments:

• @kyrsjo : DATEN modification and new global arrays
• @ericmcintosh / @kyrsjo : Update code to select the files to zip

Future ideas for FMA

Ideas which were thrown around but never actually implemented in the PLATO FMA analysis merged in version 4.5.35:

• Use binary DUMP files ( #15 ) to increase accuracy / decrease filesize of readin/readback
• Implement Laskar method as is done in Lifetrack https://cdcvs.fnal.gov/redmine/projects/lifetrac/wiki/FMA_Analysis
• Use double precision in PLATO
• Optional extra input arguments for fine-tuning analyses

Maintain a new executable with large number of element for fcc

make_six: collimat and crlibm

make_six allows both the crlibm and collimat options to be combined.

This causes a segfault when running sixtrack.

#0  0x838D3E3 in _gfortran_backtrace
#1  0x838C76C in .L3 at compile_options.o:?
#2  0xF7FFDB6F
#3  0x8380A1D in do_sub at addition_scs.c:575 (discriminator 3)
#4  0x8383C49 in sine at sine.c:52
#5  0x8383CC2 in scs_sin_rn at sine.c:80
#6  0xFFFFFFFF

Program received signal SIGSEGV, Segmentation fault.
do_sub (result=0xffff9240, x=0xffff9240, y=0xffff9160) at addition_scs.c:575
575       for(j=0; i<SCS_NB_WORDS; i++,j++)    R_HW[j] = res[i];

dipedge element

Not document, nor tested. Not clear if non-linear effect is present.

Initial distributions

Generating initial distributions in the standard SixTrack like is now done in collimation version.

Necessary distributions:

• Load file
• Load file (normalized coordinates)
• Gaussian
• Pencil beam
• ...

Please use this issue to keep track of which distributions should be implemented + describe what it is and how it should be implemented.

The idea would be to have a new block in fort.3, which if present supersede/overwrite the distribution generated by the standard TRACK/INIT block.

Build failure with debug flag enabled

On fedora 23

$ gfortran --version
GNU Fortran (GCC) 5.1.1 20150618 (Red Hat 5.1.1-4)

Using:
./make_six gfortran debug

eventually gives:

Setting up for Windows (as well)

gfortran -c -frecord-marker=4 -O4 -m32 -g -fno-second-underscore -funroll-loops -mfpmath=sse -msse2 track.f
gfortran -c -frecord-marker=4 -O4 -m32 -g -fno-second-underscore -funroll-loops -mfpmath=sse -msse2 sixve.f
sixve.f:34798:31:

   endfile (100,iostat=ierro)
                           1

Error: Symbol ‘ierro’ at (1) has no IMPLICIT type
Makefile:17: recipe for target 'sixve.o' failed
make: *** [sixve.o] Error 1

./make_six :
tried to build SixTrack_4529_cernlib_crlibm_gfortran_debug_O4 in the directory with the same name.

The following selections are ON : tilt tracking fast crlibm cernlib gfortran debug
The following selections are OFF : api naglib da collimat cpss boinc cr nagfor pgf90 g95 bpm beamgas bnlelens bignblz hdf5 ifort fio SSE lf95

It appears to have failed!!!

Without "debug" sixtrack builds correctly.

Duplicate

duplicate

New beam-beam user interface

Beam-beam lens parameter will be always read from input and never inferred.

Doubtfull implementation of solenoid

The variable ejf0v is used, but not updated, in the solenoid kick blocks kickvso1 and kickvso2

Combined function curvature effects

Update multipole block with hk1 part of the potential

DUMP improvements: FRONT and BLOCK

Dump should (1) support outputting it's data at the front of elements (FRONT keyword, similar to HIGH?) and (2) should work with BLOCK (drift) elements.

windows binary incompatibilities

Test the latest windows building environment

DUMP bug: ghost particles

Looking at the output file of the DUMP block, we can observe that there are some lines with the same particle ID, s position and turn, but displaying different coordinates (see image attached). This looks like a bug, since the position of a certain particle at a certain moment should be unique.

The input files to recreate the simulation can be found here: https://drive.google.com/folderview?id=0B87GPcovg_nROHpXMjFwckVXaVk&usp=sharing

Binary output from DUMP

Today, DUMP can only produce ASCII files of varying format. Which format to use is controlled by the format flag, set on a per-file basis.

In order to be able to produce smaller files, which can be read and written faster, a format type giving binary output would be very useful.

Output unit 54 used two places

It seems that the output unit number 54 is used two places: In the collimation version (controlled by parameter outlun) for the file colltrack.out, and also for the file beambeamdist.dat in subroutine bnlrdis. There is also a fort.54 mentioned in call boincrf('fort.54',filename) followed by an open(54, ... statement in +cd open - however this seems to be wrapped in +if bnlelens (to be confirmed). There is also a close(54) in subroutine closeUnits, inside a +if bnlelens/+ei (confirmed).

Error messages in dynk_parseFUN should strip \0s before WRITE

In dynk_parseFUN, there are many error messages like

 DYNK> dynk_parseFUN():FILELIN
 DYNK> Error opening file 'input_h^@^@^@^@^@^@^@^@^@^@^@^@^@'

where ^@ is the representation of a NULL character in less.

This is written by code like this:

            write(lout,*) "DYNK> Error opening file '",
     &           cexpr_dynk(ncexpr_dynk), "'"

The problem is that cexpr_dynk is always zero-terminated. It should therefore be passed through trim(stringzerotrim( ... )) before use in write.

comnul: Several DYNK arrays missing

The arrays funcs_dynk, iexpr_dynk,fexpr_dynk, csets_unique_dynk, fsets_origvalue_dynk, are NOT initialized in subroutine COMNUL.

Rank mismatch complilation warnings

Problem:
gfortran gives a lot of warnings when compiling "dabnews.f" and "lielib.f". This may not be a real problem, but the excessive output may mask real problems by drowning them.

gfortran -c -frecord-marker=4 -O4 -m32 -g -fno-second-underscore -funroll-loops dabnews.f
dabnews.f:376.17:

  call daall(iall,1,'$$UNPACK$$',nomax,nvmax)                       
             1

Warning: Rank mismatch in argument 'ic' at (1) (rank-1 and scalar)
dabnews.f:383.17:

  call daall(iall,1,aa,nomax,nvmax)                                 
             1

(etc. -- the output is here cropped)

The code for these files are found in dabnew.s and lielib.s, and are preprocesses through the astuce masks lielib.ast and dabnews.ast.

Number of particles per tracking limited to 64

The number of particles per tracking round is currently limited to 64 particles only.

Problems:

• Output files fort.91-i, where i=1..32
• Big matrices al, as, hv, and bl1v

The first issue to tackle, then merge, is the binary output files. These are written in program maincr and subroutine writebin. They are read in subroutine join, subroutine crcheck, and subroutine postpr. Additionally, they are used in the external program SUSSIX.

Auto-assignment of output unit numbers

In many parts of SixTrack, output to (or input from) named files, their number and filename specified directly or implicitly through input files, are required. This means that it is often up to the user to specify the fortran file unit number.

This could be avoided if

1. There was a "known safe" range of unit numbers to use
2. There was a mechanism in sixtrack to automatically allocate one of these safe unit numbers, keeping track of which units have already been allocated, and which units are currently free.

No collimation examples new fix

See af71c39

plain makefile not fully compatible with make_six

No STOP when detecting wrong collimator material

In the subroutine "collimate2", a list of materials is accepted:

      if (c_material.eq.'BE') then
         mat = 1
      elseif (c_material.eq.'AL') then
         mat = 2
      elseif (c_material.eq.'CU') then
         mat = 3
      elseif (c_material.eq.'W') then
         mat = 4
      elseif (c_material.eq.'PB') then
         mat = 5
      elseif (c_material.eq.'C') then
         mat = 6
      elseif (c_material.eq.'C2') then
         mat = 7
!02/2008 TW added vacuum and black absorber (was missing) 
      elseif (c_material.eq.'VA') then
         mat = 11
      elseif (c_material.eq.'BL') then
         mat = 12
      else
+if cr
         write(lout,*) 'ERR>  Material not found. STOP', c_material
+ei
+if .not.cr
         write(*,*) 'ERR>  Material not found. STOP', c_material
+ei
!        STOP
      endif

However, if the material is not recognized, the program just prints a warning and the continues, since the STOP command has been commented out.

Are there any reason to not reinstate this STOP? Quite a few CPU hours were recently wasted due to a bad CollDB which contained an unrecognized material...

Remove "Comments as version control"

• Delete all lines with "!hrXX" - these are just new parenthesis, and the old expressions are often wrong; this should remove about 3000 unnecessary lines

More?

To be done in a separate branch...

bignblz and collimation

In order to use the FCC lattice, one needs to enable the bignblz option in order to increase nblz and friends.

Currently there is no way to increase nblz and have collimation enabled in the configuration.
There are no if statements to increase nblz if both the collimation and bignblz flags are enabled. One must manually edit the source code each time a new build is made.

Typo in the single diffraction cross section for collimation

See: http://cds.cern.ch/record/447077
Equation 3.11

The middle equation has b = 1/36 (106 - 17 M^2) b_el

The code in sixtrack has 1/26 instead of 1/36. This creates an unphysical break in the differential cross section.

The offending bit of code is currently at line 63542 in sixtrack.s
elseif (( xm2 .ge. 2.d0 ).and. ( xm2 .le. 5.d0 )) then
!hr09 bsd = (106.d0-17.d0*xm2) * bpp / 26.d0
bsd = ((106.d0-17.d0*xm2) * bpp )/ 26.d0 !hr09

The bsd = ((106.d0-17.d0*xm2) * bpp )/ 26.d0 !hr09
should be

bsd = ((106.d0-17.d0*xm2) * bpp )/ 36.d0 !hr09

Make trombone matrix symplectic

Trombone matrix is not a symplect map in 6D tracking (even if the matrix is symplectic) because it is applied to non-canonical variables x',y',delta instead of px,py,pt.