szaghi / fobis

FoBiS.py, Fortran projects Building System for poor people

FoBiS.py

FoBiS.py, Fortran Building System for poor people

A KISS tool for automatic building modern Fortran projects.

Status

Issues

Python support

Why?

GNU Make, CMake, SCons & Co. are fantastic tools, even too much for poor-fortran-people.

However, the support for modern Fortran project is still poor: in particular, it is quite difficult (and boring) to track the inter-module-dependency hierarchy of project using many module files.

Modern Fortran programs can take great advantage of using modules (e.g. encapsulation), however their compilations can quickly become a nightmare as the number of modules grows. As a consequence, an automatic build system able to track (on the fly) any changes on the inter-module-dependency hierarchy can save the life of poor-fortran-people.

Why not use an auto-make-like tool?

There are a lot of alternatives for deal with inter-module-dependency hierarchy, but they can be viewed as a pre-processor for the actual building system (such as auto-make tools or even the Fortran compiler itself that, in most cases, can generate a dependency list of a processed file), thus they introduce another level of complexity... but poor-fortran-people always love the KISS (Keep It Simple, Stupid) things!

FoBiS.py is designed to do just one thing: build a modern Fortran program without boring you to specify a particular compilation hierarchy.

OK, what can FoBiS.py do? We are poor-fortran-people, we do not understand you...

Let us consider the following project tree

└── src
    ├── cumbersome.f90
    └── nested-1
        ├── first_dep.f90
        └── nested-2
            └── second_dep.inc

The main program contained into cumbersome.f90 depends on first_dep.f90 via the use statement use NesteD_1, thus it actually depends on the module nested_1. This module depends on second_dep.inc via the include statement include 'second_dep.inc'. Note that the dependency files are stored in a cumbersome nested tree. Write a makefile for this very simple example could waste many minutes... when the modules number increases the time wasted blows up!

It would be very nice to have a tool that automatically track the actual dependency-hierarchy and build the project on the fly, without the necessity to track the dependency-hierarchy changes. FoBiS.py just makes this... and few more things!

Suppose your goal is to build some (all) of the main programs contained into the project tree. In this case FoBiS.py can save your life: just type

FoBiS.py build

in the root of your project and FoBis.py will build all the main programs nested into the current root directory. Obviously, FoBiS.py will not (re-)compile unnecessary objects if they are up-to-date (like the "magic" of a makefile).

FoBiS.py has many (ok... some) others interesting features: if I have convinced you, please read the following.

Go to Top

Main features

• ⚡ Automatic parsing of files for dependency-hierarchy creation in case of use and include statements;
• ⚡ automatic building of all programs found into the root directory parsed or only a specific selected target;
• avoid unnecessary re-compilation (algorithm based on file-timestamp value);
• simple command line interface (CLI);
• friendly support for external libraries linking:
  • direct linking with full libraries paths;
  • relative linking via extending linker search paths;
• support for widely used compilers:
  • GNU Fortran Compiler;
  • Intel Fortran Compiler Classic (ifort);
  • Intel Fortran Compiler (ifx);
  • g95 Fortran Compiler;
  • IBM XL Fortran Compiler;
  • NAG Fortran Compiler;
  • PGI Fortran Compiler;
• custom compiler support;
• ⚡ configuration-files-free;
• ... but also configuration-file driven building for complex buildings;
• ⚡ parallel compiling enabled by means of concurrent multiprocessing jobs;
• ⚡ support submodules;
• advanced automatic (re-)building algorithms:
  • ⚡ automatic (re-)building when compiling flags change with respect the previous building;
  • ⚡ automatic (re-)building when linked library change;
  • ⚡ automatic building of projects upon which the actual project depends;
• ⚡ advanced automatic micro-atomic introspective unittest support by means of doctests;
• ⚡ generation of GNU Make makefile with rules fully supporting dependency-hierarchy for make-irreducible users;
• easy-extensible;
• well integrate with a flexible pythonic pre-processor, PreForM.py.

Go to Top

Documentation

FoBiS.py documentations are hosted on GitHub. The wiki and the README are the main documentation resources. Other sources of documentation are the examples.

Here is a non-comprehensive list of the main topics

Install	Usage
Manual Install	Getting Started
PyPi Install	A Taste of FoBiS.py
	Uncommon usage
	fobos: the FoBiS.py makefile
	FoBiS.py in action

Go to Top

Copyrights

FoBiS.py is an open source project, it is distributed under the GPL v3 license. A copy of the license should be distributed within FoBiS.py. Anyone interested to use, develop or to contribute to FoBiS.py is welcome. Take a look at the contributing guidelines for starting to contribute to the project.

Go to Top

A screencast of a very cumbersome example

fobis's Issues

Correct politically incorrect references to "Fortran poor men"...

into "Fortran poor people", the gender neutrally is important!

Add "rules" to fobos for performing "minor" tasks, e.g. create documentation, create project archive, etc...

Feature request:

Add the possibility to write "rules" into fobos file for performing "minor" tasks (e.g. to create documentation, to create project archive, etc...). One of the nice features of GNU Make is the ability to perform heterogeneous tasks other than the code building. It would be nice to have a similar feature into fobos file.

Presently, I am thinking at a peculiar set of fobos sections (for both single/many building modes versions) that should start with the prefix "rule-rule_name", namely

...
[rule-makedoc]
rule = doxygen ...
...
[rule-tar-prj]
rule = tar cf myprj.tar *
...

These rules should be invoked by means of a third execution-mode of FoBiS.py other than "clean" and "build", maybe "run" or "execute". Obviously, this execution mode requires a fobos file. The invocation should be similar to

FoBiS.py run makedoc

and the rule "rule-makedoc" (namely doxygen ...) is executed. Such a feature should come with a "list rules" switch for listing the rules defined into the fobos file.

FoBis.py -h command

There should be some note about additional help from
FoBiS.py build -h
FoBiS.py clean -h
FoBiS.py rule -h
commands in the FoBis.py -h output because it is not fully obvious/intuitive that they exist?

preprocessor flags in config files

In the documentation there isn't any option for setting preprocessor flags in the config file, but I see this is now a command line option. It would be nice to have this also for configuration files.

(I'm mostly going to use the config file mode of FoBiS because I like to be certain that when I build something new it will be with the same settings as the last time I built the program, and I don't have a personal computer at school.)

Make fobos case sensitive/insensitive selectable

By default configparser is case insensitive: all options are stored lower case. Setting "fobos.optionxform = str" make it case sensitive. It should become a user option to select sensitive/insensitive case.

Makefile export

It seems that the function save_makefile which I am mostly interested in, has not been invoked in the source code and export to Makefile doesn't work

Non-module source files

Can FoBiS handle old style .f Fortran files that are just subroutines or functions (not modules)? There could be a flag where you pass in the file names or directories and tell it to compile them even though they are not part of the set of files that were analyzed for module dependency. In this case, the order is not important.

support ibm xl fortran

It is time to support ibm xl fortran compiler: I will try on fermi cluster

Modes

Hello Stefano!
I'm very happy that you included the "modes" feature! Now I can have only one config file for both release and debug, which is very nice!

However, you seem to have neglected to document it in the main readme, which is a shame since then users will discover this feature when old fobos files fail to work.

pip install with Python3

pip install FoBiS.py doesn't work for Python 3. It seems to be trying to install the Python 2 version of the "multiprocessing" package? The error I get is:

C:\Users\USERNAME>pip install FoBiS.py
Downloading/unpacking FoBiS.py
  Downloading FoBiS.py-1.2.9.tar.gz
  Running setup.py (path:C:\Users\USERNAME\AppData\Local\Temp\pip_build_USERNAME
\FoBiS.py\setup.py) egg_info for package FoBiS.py

Downloading/unpacking multiprocessing (from FoBiS.py)
  Running setup.py (path:C:\Users\USERNAME\AppData\Local\Temp\pip_build_USERNAME
\multiprocessing\setup.py) egg_info for package multiprocessing
    Traceback (most recent call last):
      File "<string>", line 17, in <module>
      File "C:\Users\USERNAME\AppData\Local\Temp\pip_build_USERNAME\multiprocess
ing\setup.py", line 94
        print 'Macros:'
                      ^
    SyntaxError: invalid syntax
    Complete output from command python setup.py egg_info:
    Traceback (most recent call last):

  File "<string>", line 17, in <module>

  File "C:\Users\USERNAME\AppData\Local\Temp\pip_build_USERNAME\multiprocessing\
setup.py", line 94

    print 'Macros:'

                  ^

SyntaxError: invalid syntax

----------------------------------------

This is using the Anaconda 3-2.1.0 Python package on Windows (don't know if that makes a difference or not).

FoBiS tries to compile emacs backups

I’m getting some weird errors if there are Emacs backup files in the source directories. These files have the formats:

• file_name.f90~
• #file_name.f90#

It would be great if FoBiS just ignored these… are at least provided a way to tell it to ignore files matching *.f90~ and \#*.f90\#

Modes error

Hello Stefano,
when I try to build and make a typo in the mode name, there is no proper error handling, I'm just handed an stack trace and told that the section is missing.

It seems like it would be good to have something that catches this error and tells the user they entered the wrong mode name instead.

Thank you for your nice builder!

FoBiS.py no longer working after upgrade!

I just upgraded FoBiS.py using:

$ sudo pip install --upgrade FoBiS.py

The upgrade seemed to proceed without incident.

Then, I tried to build something and got the following errors:

Traceback (most recent call last):
  File "/usr/local/bin/FoBiS.py", line 5, in <module>
    main()
  File "/usr/local/lib/python2.7/site-packages/fobis/fobis.py", line 142, in main
    build()
  File "/usr/local/lib/python2.7/site-packages/fobis/fobis.py", line 85, in build
    builder.build(file_to_build=pfile, output=__config__.cliargs.output, nomodlibs=nomodlibs, mklib=__config__.cliargs.mklib)
  File "/usr/local/lib/python2.7/site-packages/fobis/Builder.py", line 414, in build
    result = syswork(link_cmd)
  File "/usr/local/lib/python2.7/site-packages/fobis/utils.py", line 20, in syswork
    return [error, str(output.decode("utf-8"))]
UnicodeEncodeError: ‘ascii’ codec can't encode character u'\u2019' in position 25: ordinal not in range(128)

It seems that \u2019 is the unicode character “RIGHT SINGLE QUOTATION MARK” Any thoughts on what is going wrong? I’m wondering if I should try downgrading to get rid of this error

Interdependencies among different projects

Say I was using FoBiS to compile libraries which are in turn used by multiple programs (an example can be found here). I believe currently, if I make a change to a module within the library, the main programs are not recompiled (since they are only linking to the .a library file that is produced by a previous FoBiS call, and have no knowledge of how they are interdependent). Is there currently any way to tell FoBiS that one project depends on another? (Note: I'm getting this idea from foraytool which has this feature).

Problems with intrinsic modules

Hi Stefano,
I’m running into this error:

Attention: the file ‘src/tests/test1.f90’ depends on 'iso_c_binding' that is unreachable

And the build is failing. However, iso_c_binding is an intrinsic module and does not require any source file to be built, same with iso_fortran_env and ieee_…

Is there a way to tell FoBiS.py not to try to build the intrinsic module, and not to fail if it can’t seem to find it?

Avoid Recursion

In view of parallel-enabling recursive functions must be converted into iterative ones.

Using previously compiled modules

Does FoBiS support compiling a project that depends on a pre-compiled mod file (say one that goes with a third-party library)? In this case, the source for this module is not visible to the script and I'm getting the error about unreachable modules.

Documentation improvement

Improve documentation: exploit wiki of GitHub

target/file specific cflags heritage

Hi Stefano,
Would it be possible to make the .cflags.heritage file so that it will keep track of the flags on a per-source and target basis? It may already do this when using a fobos file, but when run without a fobos each time FoBiS.py is run to build different targets, it overwrites the cflags used for the old target with those used for the new target. That means, that if two targets have different cflags, building either of them will cause FoBiS.py to think that the other one needs to be rebuilt due to stale cflags.

One idea is you could make files like this:
.cflags.heritage.<target_name> where <target_name> is the name of the thing you’re building.

Another idea is to have one line per target in .cflags.heritage with some sort of key at the beginning of the line. Then FoBiS.py can look up the cflags associated with the target it is working on, if they need updating, then rebuild the target and insert the new flags after the key. You could even just delete the whole line the stale key/flags combo is on if a rebuild is needed due to the flags, and insert the same key with updated flags at the beginning or end of the file.

These lines might look like:

libjsonfortran.a : “-O3 -Wextra -Wall -pedantic”

where the key is just the target name, followed by a space or some special separator character and then the cflags.

Basically, I think you need a dictionary structure to store cflags at least on a per target basis. It’s possible that you may want to compile different sources for a given target with different flags too, but I’m not sure if FoBiS.py currently supports this, and if it does, I don’t know how to do it.

Multiprocessing FoBiS.py abort issue

With multiprocessing activated, if the building is aborted the exit from FoBiS.py is trouble: it can happen that some processes remain append

improve compiler command check

The supported compiler have default compiler command (gfortran, ifort, g95...) that must be more accurately checked, e.g. in many OS gfortran is installed as gfortran-4.x. This can cause problems.

Library search path

Might be nice if FoBoS supported the gfortran -L flag for the library search path. I think currently, you have to include the full path to the libraries used with the -libs flag, since gfortran doesn't use the include directory when searching for libraries.

Refactor CLI

The number of CLAs grows too much... it is time to group them into "class" for the sake of clarity

Define file extension to be preprocessed

Hi Guys,

Is there a way to define the file extensions that will be preprocessed?
That way only the files with those extensions will be passed to PreForM and located into the PreForM folder.
And the files not preprocessed will be compiled normally.

for example only the files with extension *.pf and *.ph will be preprocessed and converted into *.f and *.h files and copied into the pfm_dir folder

If the selection cannot be done based on file's extension, could it be file name based?
The idea would be not to preprocess all files.

Thank you

heritage flags switch causes FoBiS.py to fail if output directories missing

See the log here: https://travis-ci.org/zbeekman/json-fortran/jobs/53563827

FoBiS.py will happily create a new directory to put compiled programs and libraries in, BUT this directory is also where the .cflags.heritage file is stored. When compiling with -ch turned on, FoBiS.py should be able to treat this case of a missing directory and .cflags.heritage file as if the .cflags.heritage file had been deleted.

This is the error I see:

File "/usr/local/bin/FoBiS.py", line 5, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/fobis/fobis.py", line 14, in main
    __config__.run_fobis()
  File "/usr/local/lib/python2.7/dist-packages/fobis/config.py", line 466, in run_fobis
    self._get_cli(fake_args=fake_args)
  File "/usr/local/lib/python2.7/dist-packages/fobis/config.py", line 436, in _get_cli
    self._postinit()
  File "/usr/local/lib/python2.7/dist-packages/fobis/config.py", line 130, in _postinit
    self._check_cflags_heritage()
  File "/usr/local/lib/python2.7/dist-packages/fobis/config.py", line 119, in _check_cflags_heritage
    with open(self.cliargs.build_dir + '.cflags.heritage', 'w') as chf:
IOError: [Errno 2] No such file or directory: ‘lib/.cflags.heritage’

OpenMP

I'm trying to use FoBiS to compile some code that has OpenMP directives using gfortran. If I add the -fopenmp flag to the cflags, it doesn't work. I get errors on link like this:

Undefined symbols for architecture x86_64:
  "_GOMP_parallel", referenced from:
  ...
  "_omp_get_num_threads", referenced from:
  ...
  "_omp_get_thread_num", referenced from:
  ...
ld: symbol(s) not found for architecture x86_64

If I call gfortran manually with the -fopenmp flag, it works fine. Any ideas?

Code Refactoring

Code refactoring for:

• support pybuilder;
• support PyPI;
• support TravisCI.

output settings ignored

Hello Stefano,
I just realized that the latest FoBis names the final executable with the name of the source file being compiled, whereas previously you could specify the name of the executable in the configuration file and have that respected.

Now I would have to rename the builds manually if I want several different versions built with different preprocessing flags.

Indicate clearly that building was successful

Currently building the example (using compiler=GNU), results in two lines being printed, the last one being a call to the compiler typeset in red. It is not obvious that this indicates a successful build.

Build seems to ignore compiler error

I just tried the newest version, and built my project that has a few modules with it. It compiled as before. However, I then introduced a syntax error to test FoBiS, and the compilation fails as expected, but FoBiS reports that the build succeeded! My guess is that since I had a binary from last build FoBiS somehow thinks everything went well?

I attach a snippet of the output, the compiler error is in swedish.

Building ./src/SPH1D.f95
Compiling ./src/SPH1D.f95 serially 
./src/SPH1D.f95:9.4:

    E
    1
Fel: Oklassificerbar sats vid (1)
Linking ./SPH1D
Target ./src/SPH1D.f95 has been successfully built

A release

I think FoBiS deserves at least one version number by now! We are three users, that should be enough to release 1.0.3 or something. Version numbers make you sound more important and reliable, proven* fact!

*There is a marvellous proof, but alas this text field is too small to contain it.

Documentation of mod_dir

Hello Stefano!
I was making a new fobos file and wondering if I should change the "mod_dir" setting, when I realized I was unsure of what it does. So I went to read your nice documentation, but the docs are misldeading :(

They say that "mod_dir" and "obj_dir" set the directories where fobis will look for obj and mod files, but they don't mention that these paths are relative to the "build_dir". And especially the syntax used in the example file makes it seem like they are could be arbitrary. At least that is how it works on the latest FoBiS.

Another Python3 Issue

If the source file has a syntax error, then I'm getting an error when running FoBiS with Python 3.4.2.

Traceback (most recent call last):
  File "/Users/anaconda/bin/FoBiS.py", line 5, in <module>
    main()
  File "/Users/anaconda/lib/python3.4/site-packages/fobis/fobis.py", line 140, in main
    build()
  File "/Users/anaconda/lib/python3.4/site-packages/fobis/fobis.py", line 90, in build
    builder.build(file_to_build=pfile, output=__config__.cliargs.output, nomodlibs=nomodlibs)
  File "/Users/anaconda/lib/python3.4/site-packages/fobis/Builder.py", line 302, in build
    print(self.colors.red + result[1] + self.colors.end)
TypeError: Can't convert 'bytes' object to str implicitly

This can be seen when trying to compile the following file (src/test.f90):

program blah

abc

end program blah

with the command:

FoBiS.py build -s ./src -compiler gnu

Python 3 issue

There seems to be an issue when using FoBiS with Python 3.4.2.

Traceback (most recent call last):
  File "./FoBiS.py", line 1129, in <module>
    __builder__.build(file_to_build=pfile,output=__cliargs__.output,nomodlibs=__nomodlibs__,mklib=__cliargs__.mklib)
  File "./FoBiS.py", line 399, in build
    print(self.colors.red+result[1]+self.colors.end)
TypeError: Can't convert 'bytes' object to str implicitly

I get this when using the -libs flag with a library that doesn't exist.

Meaning of dexe

Hello Stefano!
What does the "dexe" option in the example fobos files do?

Pythonic preprocessor

Fortran has not its own standard preprocessor (yes fpp has been discussed during 2003/2008 standard definition, but as far as I know it is not a standard and it not widely used). Presently, cpp is the most used preprocessor. However, cpp has many limitations. It would be very useful to have a modern, features-rich, python preprocessor. Even more nice, this preprocessor should be integrated within FoBiS.py.

Add gcov Parser

In order to generate a report of uncovered procedures it is necessary a parser of gcov output as suggested by Izaak

Building libraries with multiple source files

Hey Stefano,

I think there is a bug when building a library that consists of more than one source file. If the library is only one file, then it works, but if there are multiple files then it doesn't. It seems like the object code for anything other than the main file is not getting archived in the library.

Example (for my fortran-astrodynamics-toolkit project):

FoBiS.py build -compiler gnu -colors -cflags "-c -O2" -dbld ./lib/ -s ./src/ -dmod ./ -dobj ./ -t fortran_astrodynamics_toolkit.f90 -o libfat.a -mklib static

All the modules are compiled, but the resulting .a file doesn't contain any of the code from any module other than the main one. What should happen is that all the .o files that were compiled should be added to the library (using something like ar -rcs libfat.a ./lib/*.o).

program and module in the same file get compiled but not linked

Is there any way to get FoBiS to compile and link a program and module contained in the same file?

Cleaning issue

If I want to put the obj and mod code in the build directory (not in a subdirectory), it seems to work using -dmod ./ -dobj ./. However, if I try to clean this, I get an error.
Command:

FoBiS.py clean -dmod ./ -dobj ./ -dbld ./lib/

Error is:

OSError: [Errno 22] Invalid argument: 'lib/./'

pypandoc

When I try to update FoBiS.py using sudo pip install FoBiS.py --upgrade, I get the error:

  File "<string>", line 17, in <module>

  File "/private/tmp/pip_build_root/FoBiS.py/setup.py", line 3, in <module>

    import pypandoc

ImportError: No module named pypandoc

FoBiS.py build: error: argument -lflags: expected one argument

I’m confused… I’m trying to pass more than one link flag but I get the following error:

FoBiS.py build: error: argument -lflags: expected one argument

it seems that the space between -lflags and the stuff that follows it causes this error.

Standard file format for fobis config files

I would suggest using a standard format for the fobis files. For example, JSON, which is easily read/written using Python and other languages. An example might look something like this:

{
    "modes": [
        {
            "name": "shared",
            "compiler": "Gnu",
            "cflags": "-c -fPIC",
            "lflags": "-shared",
            "mod_dir": "./mod/",
            "obj_dir": "./obj/",
            "build_dir": "./build/",
            "src": "./src/",
            "colors": true,
            "quiet": false,
            "target": "library.f90",
            "mklib": "shared",
            "log": true
        },
        {
            "name": "static",
            "compiler": "Gnu",
            "cflags": "-c",
            "mod_dir": "./mod/",
            "obj_dir": "./obj/",
            "build_dir": "./build/",
            "src": "./src/",
            "colors": true,
            "quiet": false,
            "target": "library.f90",
            "output": "Staticlib.a",
            "mklib": "static",
            "log": true
        }
    ]
}

It might also provide more flexibility in the long run if these files get more complex.

integrate with ford

Try to improve the integration with ford

Need -c flag when compiling

Trying out FoBiS on a simple project:

./FoBiS.py build -s ./src -compiler gnu -o ./bin/test

It fails on the first file because the cmd being called by syswork(cmd) is:

gfortran  -J ././mod/  ./src/blah.f90 -o ././obj/blah.o

It's missing the -c flag.

It works if I add the '-c' to this line in the code:

buildparser.add_argument('-cflags',                required=False,action='store',               default='-c',            help='Compile flags')

I don't know if this is the proper solution, or if I'm just using it wrong.

Also, I can't pass -c to FoBiS using the -cflags command line argument, because it thinks it's a new argument.

I'm using Python 2.7 on a Mac, if that helps.

Feature request: better/more verbose output

It would be nice if there was a flag to increase the verbosity of FoBiS.py to help trouble shoot problems. There should be a flag that makes FoBiS.py output what commands it is running as it runs them. Right now, when something goes wrong it is too opaque as to whether the problem was in FoBiS.py (a bug) or if the user did something wrong. For example, see jacobwilliams/json-fortran#53 where the problem was an extra space was introduced into the build script. If there had been some sort of step by step debugging output it would have been easier to see that flags weren’t passed into FoBiS.py due to an error in the shell script.

Idea: Recompile if -cflags have changed

It would be interesting if FoBiS could detect if you have changed the -cflags since the last time you compiled, and then realize that it needs to recompile everything again. This could be a useful feature.

fobos local variables

Add feature: allow definition of local variables inside fobos file that can be shared between different modes/options

Some command switch to predefine output exe name?

Currently the name of executable is defined by the name of the file with "program/end program"? Can it be predefined manually using some option switch?

Problmen with utf-8

Hello Stefano,
sometimes when I accidentally give a incorrect flag to the compiler, fobis will crash with a message ending like this:

File "FoBiS/src/main/python/fobis/Builder.py", line 457, in _compile
result = syswork(cmd)
File "FoBiS/src/main/python/fobis/utils.py", line 27, in syswork
return [error, str(output.decode("utf-8"))]
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2018' in position 50: ordinal not in range(128)

Today I put a print statement in utils.syswork to see what the compiler was returning and saw:
gfortran: error: unrecognized command line option ‘-fdefault-integer-4’

So these ‘marks’ that gfortran sets around the offending flag crashes FoBiS and I never get to know what my error was :(