Interdependencies among different projects about fobis

[szaghi]

Presently, fobis has not this feature. How foraytool triggers the
re-compilation of the main program? I have an idea that I would like to
try... I would like to introduce an md5sum (or similar) dependecy between
sources and compiled objects. Anyhow I will studg foraytool and i will try
to implement this feature tomorrow. Se you soon.

Il giorno 18:49 dom 15/feb/2015 Jacob Williams [email protected]
ha scritto:

Say I was using FoBiS to compile libraries which are in turn used by
multiple programs (an example can be found here
https://github.com/jacobwilliams/Fortran-Astrodynamics-Toolkit/blob/master/build.sh).
I believe currently, if I make a change to a module within the library, the
main programs are not recompiled (since they are only linking to the .a
library file that is produced by a previous FoBiS call, and have no
knowledge of how they are interdependent). Is there currently any way to
tell FoBiS that one project depends on another? (Note: I'm getting this
idea from foraytool https://code.google.com/p/foraytool/ which has this
feature).

—
Reply to this email directly or view it on GitHub
#35.

[szaghi]

Hi Jacob,

I have just read some docs of foraytool (and its source) and I understand that I do not understand you...

Let us consider as our prototype scenario the following circumstances:

• a (main) target that depends on other (sub-)target;
• the dependency is accomplished by means of only compiled object (.o or .a or .so) link;
• when the sub-target compiled object is not update with respect its own sources, also the main target must be recompiled.

Is the above scenario plausible? In this case the solution is not simple because: how the sub-target compiled object must track its own sources? Presently, the easiest way is to add an option like before-build target check the build status of sub-target where I have to find a simple way to set the sub-target build settings...

With foraytool you specify that the target dependson subtarget... but they are very similar to a fobos file modes... Adding a before-build hook should be simple into fobos, but it is more complex to do (easily and KISS) into the command line.

[jacobwilliams]

Hey Stefano,

I don't quite understand your 3rd bullet point. :)

Here's my scenario:

The sub-target is a library that contains various modules. So, say I change a subroutine in one of the modules. Then, I go to recompile the main program. The main program is linking to the library (and also using the module within it that was changed). It is clear that what should happen is the main program now needs to be recompiled, since something it is using has changed. This is exactly what FoBiS currently does within a given project. But, now, it would need to somehow do this among multiple projects.

It could be as easy as checking if the .a library file for the sub-target has changed, and if so, everything in the main program is recompiled? Maybe this check could just be done on all the -libs that the project is using?

Or, perhaps you could define different projects or modes within one fobos file and specify that one is dependent on another (like the dependson foray feature). I haven't tried to use a fobos file yet, so I don't know what is possible there.

Also, if you've ever used Visual Studio and Intel Fortran, there is a feature like this to specify that one project depends on another. Also note that if you have multiple projects that depend on each other, this also means the projects themselves have to be compiled in a certain order. Foray also has this, as was mentioned.

I only suggest this since it is something that I use, and it is quite handy for projects that are building multiple libraries. However, I recognize that you may not choose to add this feature, since it might get FoBiS away from a KISS philosophy. But, it is something to consider.

[szaghi]

Dear Jacob,

this does not broke KISSness (it is an optional feature that "advanced" user can exploit, but it is not mandatory to use), and I am actually working on that, not just "consider" it.

What we have to decide is how implement it. My third point is just the main issue. Let me to be more clear. How do you like to track the modifications on libfoo.a? The modifications of libfoo.a will trigger the rebuild of main program, but presently if the libfoo.a is simply linked you cannot be advised of any changes of libfoo.a sources... With foraytool you have 2 targets one depending on the other, not just one target with a linked library...

Now, I can view 2 different approaches (and we can try implement both of them, not just one):

• create in FoBiS.py a mechanism similar to foraytool to create 2 (or more) targets with one depending on the other; this approach, I think, is simpler with fobos rather than the command line;
• add another mechanism to track libfoo.a modifications; presently the only (simple) way I see for this approach is to the check a hash (e.g. md5sum) of the libfoo.a:
  • build your main and concurrently save the hash of libfoo.a aside the library;
  • modify libfoo, thus create a new libfoo.a;
  • FoBiS.py build main_project: now FoBiS.py checks ALSO the hash of libfoo.a and recompile the main because the hash differs from the old previously stored.

Do you agree? Any other suggestions are welcome.

See you soon.

[Tobychev]

How would you store the hashes? As far as I'm aware FoBiS doesn't create any extra files beyond the intermediate build results, so I'm guessing saving the hashes would mean creating an entirely new metadata file...

I can think of a third way: whenever you build something put a copy of everything the linker needs in the bld/ directory, including any external libraries. FoBiS would detect that the library binaries haven't changed and reuse them with its current method of checking files, and an flag can tell FoBiS to compare its own versions of binaries against something elsewhere on the file system.

A version of this would be to copy "volatile" libraries (those you expect to change often) only, to save some space.

In the extreme KISS limit, just making local versions of the libraries and only linking against those would combine with the existing "clean" command to produce the desired behaviour, with the clean command forcing a complete rebuild and fetching of fresh copies of the libraries.

A final remark is that a configuration file can support more complexity than the command line while still being "simple", and I'm fine with restricting the most advanced functionality to the configuration file.

[szaghi]

Hi Tomas, thank you for your contribute.

• yes, I was thinking to create a new family of intermediate building output (the hashes) that must be stored aside the compiled object (.o, .a or .so) that the user flag to be volatile as you said;
• storing the original linked objects instead of their hashes is an option, but the hashes files are very lightweight... I was planning to store just the md5sum hashes in plain ascii text and compare these hashes with the actual library ones computed on-the-fly by FoBiS.py;
• I agree with you: complex usage is easier to be implemented into fobos rather than the command line.

I am happy that restricting such an advance usage to fobos is fine for you, but my first trial will be to support also the command line.

My main concern is about the mechanism:

• interdependent targets/projects as in foraytool or
• still one target having special linked dependencies?

The first approach is more complex, but potentially more powerful, I suppose.

I aspect the response of Jacob and your and then I will try one of them (or a new from you).

See you soon.

[Tobychev]

Hello Stefano,
I'm not sure what you mean, Interdependent is a big word, implying some complex behaviour. But it is not clear exactly what complex behaviour this would mean.

The current use case is (as I understand it) to ensure FoBiS detects changes to binary dependencies whenever a user rebuilds a project. Interdependent evokes images of something grander, like building a library resulting in automatic rebuilding of everything that depends on it.

[szaghi]

Yes,

this is exactly the point...

• add a simple new building trigger for changes in binary dependencies or
• add a more powerful mechanism into which many targets/projects depends each other, they being interdependent (e.g. auto rebuilding all of them if necessary)

I think that foraytool has (at least partially) the second behavior, but I do not completely understand what Jacob needs.

Anyhow, I am going to implement the simple new binary trigger...

[szaghi]

Hi all,

the new version 1.4.0 (both on PyPi and github) has the light "volatile" libraries feature...

https://github.com/szaghi/FoBiS/tree/master/examples/cumbersome_dependency_program_volatile_libs

https://github.com/szaghi/FoBiS/wiki/Autorebuild-when-external-linked-libraries-change

Please, let me know what you think about this.

See you soon.

[jacobwilliams]

I'll try to test the "volatile" feature on my projects (probably won't have time until later in the week). Looks like it will be a good feature! Question: does this mean that FoBiS has to figure out the location of the libraries? For example, if you pass a bunch of paths using the -L option, and then add a library using the-lBLAH option, the compiler will use some logic to search for the libBLAH.a library. Will FoBiS do this also? And is there any danger that it will pick the wrong library (say, a library with the same name is in two different directories)? Just wondering about edge cases...

The more complex option (defining multiple projects with dependencies) would still be good to have, I think. (Probably there is no way to do it from command line, but it is OK for this to be an "advanced" feature). This has the advantage of actually being simpler to use, since if one project is building a library, and another is using it, FoBiS can handing the linking and paths automatically, without the user having to specify them.

[szaghi]

"Looks like a good feature..." it is your feature :-)

Hi Jacob,

The volatile attribute is intended to be applied at only the libraries linked with the full path without the linker intrusion (-L and -l switches). The reason is that the libraries linked with their full path are more likely a poor library rather than a system library: track the change of a system library is not a FoBiS.py aim, when this happens just type FoBiS.py clean ; FoBiS.py build, whereas a volatile library can change suddendly and it is nice that FoBiS.py check them for us. Anyhow, introducing a volatile system library should be simple, if you want I will do it later in the morning.

The interdependency projects (I am really thinking to a fobos files dependency, the notion of projects is currently unknown for FoBiS.py) is in my radar, but I cannot say when it will be ready. However, we can start to discuss some details. I am thinking to something like this:

• consider 2 fobos (project) files placed elsewhere (not necessarly in the same directory;
• fobos_lib is the project file for building a library;
• fobos_prog is the fobos of the main program using that library;
• we want to build the main program and let FoBiS to care about all dependncies in between to keep uptodate the library progect...

In this scenario, I thinking to add a special option into the fobos like the following (I am using the default section just as an example, but it can be used into any defined modes)

[default]
dependon = ./relative_path_to/fobos_lib:static
compiler = ...
....

Note that I have specified not only the fobos file but also the actual mode (section) that in this case is static but it can be any modes defined into fobos_lib (in case it is omitted the default or the first one is used). Now, if we do FoBiS.py build -f fobos_prog the first thing it does will be FoBiS.py build -f fobos_lib -mode static into the correct directory and then it continues with the building of the program. In this scenario this feature can be used also from the command line avoiding the fobos_prog, but the fobos_lib is still necessary.

What do you think about this feature (that is your :-)?

@Tobychev What do you think?

See you later.

P.S. @jacobwilliams I have just realized that you use MS Visual Studio... you are not so poor Fortran man :-) because I do not use such a tool (I cannot live without vim), please feel free to request any useful features of Visual Studio that FoBiS do not have (yet)

[szaghi]

Dear all,

in v1.4.3 the volatile has been extended to also library linked via -L -l switches... some notes

• I had to "simulate" the linker searching algorithm thus presently only .a and .so libraries are searched (in this order)...
• I do not know how the linker resolve library collisions (e.g. if you have in the same directory libfoo.a and libfoo.so which library is linked?), thus I assumed that .a wins on .so :-)

For MS windows users I needs help...

[szaghi]

Dear all,

in v1.5.0 also the more complex inter-dependency projects feature, see

• https://github.com/szaghi/FoBiS/wiki/Autorebuild-with-interdependent-projects
• https://github.com/szaghi/FoBiS/tree/master/examples/cumbersome_dependency_program_interdepent

I hope both of you agree with this implementation, anyhow I will happy to accept any suggestions.

See you soon.

[jacobwilliams]

Terrific! Will try and test it next week when I get a chance. I like the implementation of referring to the fobos file of another project. I think that is better for some situations than the way foray does it (where everything is in one monolithic file).

Question: can I have multiple projects in the dependon flag? So, if it depends on multiple libraries, they will all be compiled?

Visual Studio is actually not a bad editor. The Intel Fortran integration does the dependency analysis automatically among multiple projects, so I don't have to worry about it.

[szaghi]

Dear Jacob, in the new version 1.5.1 (besides some bug-fixes) your feature request should be accomplished: now dependon is a list of dependencies able to handle multiple projects.

For more details see

• https://github.com/szaghi/FoBiS/tree/master/examples/cumbersome_dependency_program_interdepent
• https://github.com/szaghi/FoBiS/wiki/Autorebuild-with-interdependent-projects

Please, let me know your opinions.

See you soon.

[szaghi]

I have added a screencast for you

https://github.com/szaghi/FoBiS/tree/master/examples/cumbersome_dependency_program_interdepent#screencast

[jacobwilliams]

Very nice!

I have another suggestion: In your example, the main fobos file has this:

ext_libs  = dep1 dep2
lib_dir   = ./dependency_lib_1/build/ ./dependency_lib_2/build/
include   = ./dependency_lib_1/build/mod/ ./dependency_lib_2/build/mod/
dependon  = ./dependency_lib_1/fobos_lib:static ./dependency_lib_2/fobos_lib:static

It would be great if only the dependon variable was necessary, and the others were accounted for automatically for FoBiS. So, all the user has to do is specify the dependent projects, and the include paths, lib names, etc. are automatically handled by FoBiS (these should be known since they are specified in the dependent fobos files). What do you think?

[szaghi]

I think you are a very smart man... nice suggestion: the user likes KISS
things! Monday your feature request should be accomplished.

Have a good weekend
Il 20/feb/2015 23:07 "Jacob Williams" [email protected] ha scritto:

Very nice!

I have another suggestion: In your example, the main fobos file has this:

ext_libs = dep1 dep2
lib_dir = ./dependency_lib_1/build/ ./dependency_lib_2/build/
include = ./dependency_lib_1/build/mod/ ./dependency_lib_2/build/mod/
dependon = ./dependency_lib_1/fobos_lib:static ./dependency_lib_2/fobos_lib:static

It would be great if only the dependon variable was necessary, and the
others were accounted for automatically for FoBiS. So, all the user has to
do is specify the dependent projects, and the include paths, lib names,
etc. are automatically handled by FoBiS (these should be known since they
are specified in the dependent fobos files). What do you think?

—
Reply to this email directly or view it on GitHub
#35 (comment).

[szaghi]

Hi Jacob,

You must give me more details for your new feature request. When dependon is used the only other options that I can almost safely activate is include, but the others, namely ext_libs and lib_dir, depend on the user choices. As matter fact, the user can link the library both with full path, libs option, and relative path, ext_libs option. As a consequence I can safely remove the necessity for include option, but the others rely on user choices. Do you agree or do you any other suggestions?

Maybe, we can agree that when dependon is activated the include and lib_dir are automatically activated also, but ext_libs or libs must rely on user choices...

Thank you for your help.

[szaghi]

Dear all,

I have just uploaded a new version (both github and PyPi), having the requested features, namely automatically add include and lib_dir options when dependon is used. As aforementioned, the option ext_libs or ext_vlibs or libs or vlibs must still set: I like to allow the user to select the linking mode he prefers without forcing one mode. From the fobos of the example you can see that now it enough to set

ext_libs  = dep1 dep2
dependon  = ./dependency_lib_1/fobos_lib:static ./dependency_lib_2/fobos_lib:static

I close this issue, but I will reopen it if anyone ask me more.

See you soon.

[jacobwilliams]

Terrific!
I still think that it would be nice not to have to duplicate the library names in the main fobis file. It would require forcing one of the linking options, but I don't see the harm in that. I would say use the one with the full path to the library. This will be known, since all the information is in the dependent fobos file. What do you think? I don't see the harm. It would be transparent to the user (it would "just work", like a poor Fortran man would expect). :)

[szaghi]

Hi Jacob,
Indeed I have also thinked to force the direct linking just 2 seconds after pushed the new version... however, I am now oriented to another compromise: I think it would be nice to force automatically the direct linking, but allowing the user to specify also the other mechanism in the case it is mandatory (as you, I presently do not see this case, but the devil hides away into the details...), somenthing like

dependon = lib_foo/foo_fobos:static(indirect)

This is just a paranoiac precaution... Anyhow tomorrow I will push the new version with the direct link forcing.
Thank you for your help.
See you son.

[szaghi]

Hi all,

the new version v1.5.3 (upload on both github and PyPi) the Jacob request should be accomplished (with my paranoiac compromise...): now defining dependon = ... option will automatically set all other library options that are not longer required to be explicitly defined. For more details see:

• https://github.com/szaghi/FoBiS/wiki/Autorebuild-with-interdependent-projects
• https://github.com/szaghi/FoBiS/tree/master/examples/cumbersome_dependency_program_interdepent

See you soon.