madx code and markdown about cpymad

[ydutheil]

Hi,
How would you see that implemented?
I think markdown processors generate html code. Do you mean to implement a MADX syntax highlight color rules to a markdown processor?

In order to use it like

TWISS, SEQUENCE=mysequence, FILE=myfile;

[sterbini]

Hi,
I would see it in a simple (perhaps naive) way.
The user could convert a MADX input file in a markdown one using a function madx2md that comes with cpymad as in https://codimd.web.cern.ch/s/r1WHKg5iS. This use case is very common for me and my colleagues since we have a lot of raw-MADX files (raw in the sense of "not-pythonic").
On the other hand, this is also compatible with the pythonic approach: the MADX input file can be produced by cpymad in a pythonic way using his logging capability. So one converts, if needed, the logging (commented) string in a markdown string that can be copied in a file and put on github/lab to be read as a light and formatted documentation.

[coldfix]

What do you mean by converting to markdown?

Do you mean converting e.g. the text:

CALL, FILE="sequence.madx";
TWISS;

to the string:

```
CALL, FILE="sequence.madx";
TWISS;
```

i.e. adding markdown code segment markers?

If so, it seems pretty trivial to do by oneself and not in any way specific to madx.

Or do you mean adding a pygments lexer for the MAD-X language, which would then be able to generate HTML from a language=madx denoted code segment when rendering HTML using docutils or sphinx? Edit: i.e. adding a converter from madx to html, not to markdown.

Or do you mean something else entirely?

[ydutheil]

@sterbini could you upload your codiMD doc somewhere public, at least for the output?
Reading it I understand it transforms a MADX code

! # THE header
! and some text
a=12;
TWISS();
! ## some hother header
! more text about it
exec, mymacro ;
show, myelement;

into an MD-version with the comments out of the codeblock

THE header

and some text

a=12;
TWISS();

some hother header

more text about it

exec, mymacro ;
show, myelement;

Note that @sterbini and here the syntax highlight language is set to fortran in the code blocks.

[sterbini]

@ydutheil
Exactly.
I past here the codi MD

madx2md

The main idea is to produce agile documentation from MADX files (main configuration or macros files) using markdown approach.

Each line starting with "!" will be interpreted as a markdown input.

In order to comment MADX code w/o printing it as as markdown one can use '//' for commenting.

One can use 🚧, ❓, ⚠️ to get easy-to-track information in the markdown file. Not to break compatibility with the codi/gitlab/git/mkdocs markdown dialect I suggest to keep only basic markdown features.

Example

Let us imaging that your document your mask as

! ## Introduction 
! This is a simple example how to use the markdown to comment an ascii file.
! We assume that all comments start with "!", that is no '/* ..*/' or '//' comments.
! We assume that the first line of the *MADX* input is a comment.
! To know more click [here](https://www.google.com).
! We assume $\beta\approx1$ (yes, only for test $\LaTeX$).
! We use the *fortran* typesettings, we can improve on that.
! One should use plain markdown.

! ## Header and links
! **S. Fartoukh**,  March 2009
! **M. Giovannozzi**, November 2010

option, warn, info;
! [?] should we remove the temp?
system, "rm -rf temp";
system, "mkdir temp";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/V6.503 db5";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/runIII lhc";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/HLLHCV1.4 slhc";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/errors/0705 wise";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/V6.503/WISE/After_sector_3-4_repair fidel";
option, -echo,-info;

! ## Calling macros
! The good man will comment his code.
call,file="slhc/toolkit/macro.madx";        !Call some macro facilities

mylhcbeam=1; ! LHC beam 1 (clockwise), LHC beam 2 (clockwise), LHC beam 2 (counterclockwise)

! Set this flag if the file is not used as a mask file (sets 1 to 1)
not_a_mask=0;
! Set this flag to correct the errors of D2 in the NLC (warning: for now only correcting b3 of D2, still in development)
correct_for_D2=0;
! Set this flag to correct the errors of MCBXF in the NLC (warning: this might be less reproducible in reality, use with care)
correct_for_MCBX=0;
!TODO: it is flat orbit or optics?
choose_flat_optics=0;

! ## Manual Thin Sequence building

option, -echo,-warn,-info;
if (mylhcbeam==4){
  call,file="lhc/lhcb4.seq";
} else {
  call,file="lhc/lhc.seq";
};
option, -echo,warn,-info;

! ## Install HL-LHC
call,file="slhc/hllhc_sequence.madx";

! ## Slice nominal sequence
exec, myslice;

Then you can do something like

# %%
import pandas as pd

def madx2df(inputFile):
    with open(inputFile) as fid:
            mask_content = fid.read()
    # %% split the mask
    # I am assuming that the start of the file is '! ## '
    assert(mask_content[0:5]=="! ## ")
    aux=mask_content.split("! ## ") 
    title=[]
    body=[]
    for i in aux:
        title.append(i.split('\n')[0])   
        body.append("".join([a+'\n' for a in i.split('\n')[1:]])) 
    # I remove the first (empty line, see assertion above)
    title=title[1:]
    body=body[1:]
    # %% built a pandas df
    return pd.DataFrame(body,index=title, columns=['Code string'])

def madx2md(inputFile, outputFile):
    df= madx2df(inputFile)
    myString=''
    for row in df.iterrows():   
        print(row[0])        
        myString=myString+'## ' +row[0]+ '\n'
        bodyString=row[1]['Code string'][0:-1]
        commentBefore=True
        for lines in bodyString.split('\n'):
            if len(lines.strip())>0:
                if lines.strip()[0]=='!':
                    if commentBefore:
                        myString=myString+lines.strip()[1:]+ '\n'
                    else:
                        myString=myString+'```\n'
                        myString=myString+lines.strip()[1:]+ '\n'
                    commentBefore=True
                else:
                    if commentBefore:
                        myString=myString+'``` fortran\n'
                        myString=myString+lines+ '\n'
                    else:
                        myString=myString+lines+ '\n'
                    commentBefore=False
        if commentBefore==False:
            myString=myString+'```\n'

    with open(outputFile, 'w') as fid:
        fid.write(myString)

# %%
madx2md('/afs/cern.ch/work/s/sterbini/beambeam_macros/macro_bb.madx','/afs/cern.ch/work/s/sterbini/beambeam_macros/macro_bb.md')

that you get your md file that I copied in the following. This may help documenting and refactoring the MADX code.

---

Introduction

This is a simple example how to use the markdown to comment an ascii file.
We assume that all comments start with "!", that is no '/* ..*/' or '//' comments.
We assume that the first line of the MADX input is a comment.
To know more click here.
We assume $\beta\approx1$ (yes, only for test $\LaTeX$).
We use the fortran typesettings, we can improve on that.
One should use plain markdown.

Header and links

S. Fartoukh, March 2009
M. Giovannozzi, November 2010

option, warn, info;

[?] should we remove the temp?

system, "rm -rf temp";
system, "mkdir temp";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/V6.503 db5";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/runIII lhc";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/HLLHCV1.4 slhc";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/errors/0705 wise";
system, "ln -fns /afs/cern.ch/eng/lhc/optics/V6.503/WISE/After_sector_3-4_repair fidel";
option, -echo,-info;

Calling macros

The good man will comment his code.

call,file="slhc/toolkit/macro.madx";        !Call some macro facilities
mylhcbeam=1; ! LHC beam 1 (clockwise), LHC beam 2 (clockwise), LHC beam 2 (counterclockwise)

Set this flag if the file is not used as a mask file (sets 1 to 1)

not_a_mask=0;

Set this flag to correct the errors of D2 in the NLC (warning: for now only correcting b3 of D2, still in development)

correct_for_D2=0;

Set this flag to correct the errors of MCBXF in the NLC (warning: this might be less reproducible in reality, use with care)

correct_for_MCBX=0;

TODO: it is flat orbit or optics?

choose_flat_optics=0;

Manual Thin Sequence building

option, -echo,-warn,-info;
if (mylhcbeam==4){
  call,file="lhc/lhcb4.seq";
} else {
  call,file="lhc/lhc.seq";
};
option, -echo,warn,-info;

Install HL-LHC

call,file="slhc/hllhc_sequence.madx";

Slice nominal sequence

exec, myslice;

[sterbini]

What do you mean by converting to markdown?

@coldfix Sorry perhaps there is a bit of confusion since you cannot access https://codimd.web.cern.ch/s/r1WHKg5iS

I copied above the codimd page.
Indeed, I meant adding markdown code segment markers by parsing the madx input file.
Each line starting with "!" will be interpreted as a markdown input, the rest as code and enclosed in the markdown close environment.

[coldfix]

Hi,

Each line starting with "!" will be interpreted as a markdown input.

In order to comment MADX code w/o printing it as as markdown one can use '//' for commenting.

One can use construction, question, warning to get easy-to-track information in the markdown file. Not to break compatibility with the codi/gitlab/git/mkdocs markdown dialect I suggest to keep only basic markdown features.

I think this would make a good idea for your own module that you can share with your colleagues or upload to PyPI. For now, I don't like it to be included in cpymad because I don't think it's in any way canonic: I wouldn't even have had the idea to use ! to denote markdown content, and then split madx file in this specific way.

In fact, I don't too much like the idea to clutter madx files with comments containing markdown syntax. Writing dedicated markdown files for documentation, and dedicated code files for code, will usually get you more readable results in both departments. I don't think a sensible madx file has comparable structure to documentation (sections etc) and in the end both the documentation and the madx code will uglier by trying to achieve both at the same time. If you have a madx file as in the example, it was probably written specifically for documentation, so why not write it specifically as markdown in the first place? In your position, if you need to validate your code examples inside the documenation, I would probably even go the other way round, and use a tool that runs the code examples inside the markdown file, similar to what doctest does.

I copied above the codimd page.

Please always do that when reporting issues. It's usually bad practice to reference external resources because they might not be accessible to everyone, their content or url may change with time, or go down entirely. By posting inline, you conserve the content for others that view the issue in months or years from now.

Best, Thomas

[sterbini]

Thanks for sharing your opinion and for the suggestions.
Cheers,
Guido