Add missing dependencies to Fiji build system about fiji

[larrylindsey]

Sorry, this got lost in the noise that is my inbox.

Yeah, I think it'd be awesome to have that be part of core Fiji.

On Wed, Feb 4, 2015 at 2:51 PM, Curtis Rueden [email protected]
wrote:

Somehow my analysis above missed the components which are part of fiji.git
but not currently present on the Fiji update site, such as
Fiji_Archipelago. @larrylindsey https://github.com/larrylindsey: Are
you interested in those components becoming part of core Fiji? Or would you
rather keep them on a separate update site?

Also, for checking what has changed in a Fiji installation, the following
invocation is really useful:

Contents/MacOS/ImageJ-macosx --update upload-complete-site --simulate Fiji

—
Reply to this email directly or view it on GitHub
#37 (comment).

[ctrueden]

There now exists a script bin/populate-app.sh that uses Maven to populate a Fiji.app with the current Fiji's dependencies (at least the JAR files—it still does not support Fiji's non-JAR resources; see scijava/scijava-maven-plugin#17). I ran this script on an empty directory, and diffed the result against a fully up-to-date Fiji.app. For JAR files, excluding version differences caused by manual uploads, there are three sorts of remaining discrepancies:

Remaining JAR discrepancies

0.0.0-STUB files

These files mark obsolete JARs, working around a limitation in the ImageJ Updater. The Maven-dependency version of Fiji does not and should not have these dependencies.

List of 0.0.0-STUB.jar files

• jars/bio-formats/jai_imageio-0.0.0-STUB.jar
• jars/fiji-compat-0.0.0-STUB.jar
• jars/gentyref-0.0.0-STUB.jar
• jars/imglib2-ui-0.0.0-STUB.jar
• jars/imglib2-ui-0.0.0-STUB.jar
• jars/jcommon-1.0.24.jar
• jars/jgrapht-0.0.0-STUB.jar
• jars/jython-shaded-0.0.0-STUB.jar
• jars/log4j-0.0.0-STUB.jar
• jars/mapdb-0.0.0-STUB.jar
• jars/netcdf-0.0.0-STUB.jar
• jars/yecht-0.0.0-STUB.jar
• plugins/Sholl_Analysis-0.0.0-STUB.jar
• plugins/Simple_Neurite_Tracer-0.0.0-STUB.jar
• plugins/TrackMate_-0.0.0-STUB.jar

Retroweaver JARs

• retro/backport-util-concurrent-3.1.jar
• retro/retrotranslator-runtime-1.2.9.jar
• retro/retrotranslator-transformer-1.2.9.jar

These are still shipped for theoretical backwards compatibility, but should not be part of Fiji's dependency tree. In practice it is likely they could be removed with zero impact on users, but there is also little reason to do so, since they are tucked away in their own folder.

XML functionality

There are two JARs shipped with Fiji, which are not part of Maven Fiji's dependency tree, but which are required for Bio-Formats to behave properly:

• jars/serializer-2.7.2.jar
• jars/xalan-2.7.2.jar

This is really a problem with pom-scijava, not this toplevel fiji component, because those dependencies are in fact necessary at runtime and should be inherited from Fiji's dependencies. I.e.: the correct solution is not to add explicit dependencies on them here at the fiji level. I have filed scijava/pom-scijava#219 to track this issue.

Conclusion

There is not really anything meaningfully left to be done from a Maven dependency standpoint with respect to Fiji versus the Fiji.app distribution from the Java-8 update site. There are still things that could maybe be improved, such as wrapping the non-JAR dependencies into JAR files which can be depended upon via Maven, such that e.g. jgo sc.fiji:fiji still has Fiji's extra LUTs and scripts and whatnot. But it's a bonus, and I don't feel it's mission-critical for how Fiji is used. If someone requests it, we can file a dedicated issue and pursue it. But in the meantime, I'm happy to call this issue here thoroughly resolved!

[ctrueden]

To clarify: anything we want to bundle with Fiji, but do not want to "inflict" on downstream projects that add sc.fiji:fiji as a dependency, should be marked with <optional>true</optional> in the dependency declaration. See e.g. 8c80ec1.

[ctrueden]

Here is a diff of the latest Fiji distribution compared to a Fiji whose JAR files are populated from scratch via a fiji.git source checkout using mvn -Dimagej.app.directory=.../Fiji.app:

diff --git a/update-site.txt b/fiji-git.txt
index a720c6d..768a1ac 100644
--- a/update-site.txt
+++ b/fiji-git.txt
@@ -3,22 +3,18 @@
 ./jars/T2-TreelineGraph-1.0.2.jar
 ./jars/VIB-lib-2.0.1.jar
 ./jars/VectorString-1.0.2.jar
-./jars/ant-junit.jar
-./jars/ant-launcher.jar
-./jars/ant-nodeps.jar
-./jars/ant.jar
 ./jars/args4j-2.0.25.jar
 ./jars/asm-4.0.jar
 ./jars/asm-analysis-4.0.jar
 ./jars/asm-commons-4.0.jar
 ./jars/asm-tree-4.0.jar
 ./jars/asm-util-4.0.jar
-./jars/autocomplete.jar
 ./jars/base64-2.3.8.jar
 ./jars/batik-1.8.jar
 ./jars/bcmail-jdk14-138.jar
 ./jars/bcprov-jdk14-138.jar
 ./jars/bigdataviewer-core-1.0.0.jar
+./jars/bio-formats/bio-formats_plugins-5.0.6.jar
 ./jars/bio-formats/formats-api-5.0.6.jar
 ./jars/bio-formats/formats-bsd-5.0.6.jar
 ./jars/bio-formats/formats-common-5.0.6.jar
@@ -36,12 +32,9 @@
 ./jars/commons-lang3-3.1.jar
 ./jars/commons-logging-1.1.1.jar
 ./jars/commons-math3-3.2.jar
-./jars/edu_mines_jtk.jar
 ./jars/eventbus-1.4.jar
-./jars/fake-2.0.0-SNAPSHOT.jar
 ./jars/fiji-compat-2.0.1.jar
 ./jars/fiji-lib-2.1.0.jar
-./jars/fiji-scripting-2.0.0-SNAPSHOT.jar
 ./jars/gentyref-1.1.0.jar
 ./jars/groovy-2.3.6.jar
 ./jars/gson-2.2.4.jar
@@ -49,7 +42,6 @@
 ./jars/ij1-patcher-0.11.1.jar
 ./jars/imagej-2.0.0-rc-23.jar
 ./jars/imagej-common-0.12.1.jar
-./jars/imagej-launcher-3.1.6.jar
 ./jars/imagej-legacy-0.10.2.jar
 ./jars/imagej-ops-0.8.0.jar
 ./jars/imagej-plugins-commands-0.4.1.jar
@@ -66,25 +58,21 @@
 ./jars/imglib2-algorithm-fft-0.1.1.jar
 ./jars/imglib2-algorithm-gpl-0.1.2.jar
 ./jars/imglib2-ij-2.0.0-beta-29.jar
-./jars/imglib2-ops-2.0.0-beta-26.jar
 ./jars/imglib2-realtransform-2.0.0-beta-28.jar
 ./jars/imglib2-roi-0.2.0.jar
 ./jars/imglib2-ui-2.0.0-beta-28.jar
 ./jars/invokebinder-1.2.jar
 ./jars/itext-2.1.5.jar
 ./jars/itextpdf-5.1.1.jar
-./jars/jacl.jar
 ./jars/jai-codec-1.1.3.jar
 ./jars/jai-core-1.1.3.jar
-./jars/jama-1.0.3.jar
+./jars/jama-1.0.2.jar
 ./jars/java-cup-0.11a.jar
 ./jars/javac-1.6.0.24-ubuntu-fiji2.jar
 ./jars/javassist-3.16.1-GA.jar
 ./jars/jcodings-1.0.10.jar
 ./jars/jcommon-1.0.17.jar
 ./jars/jdatepicker-1.3.2.jar
-./jars/jdom-2.0.2.jar
-./jars/jdom-contrib.jar
 ./jars/jdom2-2.0.5.jar
 ./jars/jep-2.4.2.jar
 ./jars/jffi-1.2.7-native.jar
@@ -104,18 +92,16 @@
 ./jars/jnr-posix-3.0.1.jar
 ./jars/jnr-unixsocket-0.3.jar
 ./jars/jnr-x86asm-1.0.2.jar
-./jars/joda-time-2.3.jar
+./jars/joda-time-2.2.jar
 ./jars/joni-2.1.1.jar
 ./jars/jpedalSTD-2.80b11.jar
 ./jars/jruby-core-1.7.12.jar
 ./jars/jruby-stdlib-1.7.12.jar
-./jars/js.jar
 ./jars/jsch-0.1.49.jar
 ./jars/jython-shaded-2.5.3.jar
 ./jars/jzlib-1.1.2.jar
 ./jars/kryo-2.21-shaded.jar
 ./jars/legacy-imglib1-1.0.0-DEPRECATED.jar
-./jars/log4j-1.2.17.jar
 ./jars/logback-classic-1.1.1.jar
 ./jars/logback-core-1.1.1.jar
 ./jars/mapdb-1.0.3.jar
@@ -162,10 +148,8 @@
 ./jars/snakeyaml-1.13.jar
 ./jars/spim_data-1.0.0-beta-3.jar
 ./jars/swing-checkbox-tree-1.0.0.jar
-./jars/tools-1.4.2.jar
 ./jars/udunits-4.3.18.jar
 ./jars/wavelets-2.0.0.jar
-./jars/weave_jy2java-2.0.0-SNAPSHOT.jar
 ./jars/weka-dev-3.7.11.jar
 ./jars/xml-apis-1.3.04.jar
 ./jars/yecht-1.0.jar
@@ -182,7 +166,6 @@
 ./plugins/3D_Blob_Segmentation-2.0.2.jar
 ./plugins/3D_Objects_Counter-2.0.0.jar
 ./plugins/3D_Viewer-3.0.1.jar
-./plugins/Algorithm_Launcher.jar
 ./plugins/AnalyzeSkeleton_-2.0.3.jar
 ./plugins/Analyze_Reader_Writer-2.0.0.jar
 ./plugins/Anisotropic_Diffusion_2D-2.0.0.jar
@@ -190,18 +173,15 @@
 ./plugins/Arrow_-2.0.1.jar
 ./plugins/Auto_Threshold-1.16.0.jar
 ./plugins/BalloonSegmentation_-2.0.0.jar
-./plugins/BeanShell_Interpreter-2.0.0-SNAPSHOT.jar
 ./plugins/Bug_Submitter-2.0.1.jar
-./plugins/CLI_-2.0.0-SNAPSHOT.jar
 ./plugins/CPU_Meter-2.0.0.jar
 ./plugins/Calculator_Plus-2.0.0.jar
-./plugins/Clojure_Interpreter-2.0.0-SNAPSHOT.jar
 ./plugins/Colocalisation_Analysis-2.0.1.jar
 ./plugins/Color_Histogram-2.0.6.jar
 ./plugins/Color_Inspector_3D-2.3.1.jar
 ./plugins/Colour_Deconvolution-2.0.0.jar
 ./plugins/CorrectBleach_-2.0.1.jar
-./plugins/Descriptor_based_registration-2.0.7.jar
+./plugins/Descriptor_based_registration-2.0.0.jar
 ./plugins/Dichromacy_-2.0.0.jar
 ./plugins/Differentials_-2.0.0.jar
 ./plugins/Directionality_-2.0.1.jar
@@ -224,12 +204,7 @@
 ./plugins/Image_Expression_Parser-3.0.0.jar
 ./plugins/Interactive_3D_Surface_Plot-2.31.1.jar
 ./plugins/IsoData_Classifier-2.0.0.jar
-./plugins/JRuby_Interpreter-2.0.0-SNAPSHOT.jar
-./plugins/Javascript_-2.0.0-SNAPSHOT.jar
-./plugins/Jython_Interpreter-2.0.0-SNAPSHOT.jar
 ./plugins/Kuwahara_Filter-2.0.0.jar
-./plugins/LSM_Reader.jar
-./plugins/LSM_Toolbox.jar
 ./plugins/Lasso_and_Blow_Tool-2.0.1.jar
 ./plugins/Linear_Kuwahara-2.0.0.jar
 ./plugins/LocalThickness_-3.1.1.jar
@@ -245,9 +220,8 @@
 ./plugins/RATS_-2.0.0.jar
 ./plugins/RandomJ_-1.5.2.jar
 ./plugins/Reconstruct_Reader-2.0.1.jar
-./plugins/Refresh_Javas-2.0.0-SNAPSHOT.jar
 ./plugins/SPIM_Opener-2.0.1.jar
-./plugins/SPIM_Registration-2.0.8.jar
+./plugins/SPIM_Registration-2.0.0.jar
 ./plugins/Samples_-2.0.0.jar
 ./plugins/Series_Labeler-2.0.0.jar
 ./plugins/SheppLogan_-2.0.0.jar
@@ -268,8 +242,8 @@
 ./plugins/Time_Stamper-2.0.0.jar
 ./plugins/ToAST_-25.0.1.jar
 ./plugins/TopoJ_-2.0.0.jar
-./plugins/TrackMate_-2.7.1.jar
-./plugins/Trainable_Segmentation-2.1.8.jar
+./plugins/TrackMate_-2.7.0.jar
+./plugins/Trainable_Segmentation-2.1.6.jar
 ./plugins/TrakEM2_-1.0d.jar
 ./plugins/TrakEM2_Archipelago-2.0.0.jar
 ./plugins/TransformJ_-2.8.2.jar
@@ -283,10 +257,8 @@
 ./plugins/Volume_Calculator-1.0.1.jar
 ./plugins/Volume_Viewer-2.01.1.jar
 ./plugins/bUnwarpJ_-2.6.1.jar
-./plugins/bigdataviewer_fiji-1.0.2.jar
-./plugins/bio-formats_plugins-5.0.6.jar
+./plugins/bigdataviewer_fiji-1.0.0.jar
 ./plugins/blockmatching_-2.0.3.jar
-./plugins/ij-ImageIO_.jar
 ./plugins/level_sets-1.0.1.jar
 ./plugins/mpicbg_-0.6.1.jar
 ./plugins/panorama_-2.0.0.jar

[ctrueden]

The discrepancies fall into the following categories:

1. Bio-Formats special case:

+./jars/bio-formats/bio-formats_plugins-5.0.6.jar
-./plugins/bio-formats_plugins-5.0.6.jar

The bio-formats_plugins library should be filed into /plugins, not /jars/bio-formats. The cause of the erroneous behavior is special case handling for Bio-Formats in the ImageJ Maven plugin, which we need to fix.
2. The ImageJ launcher:

-./jars/imagej-launcher-3.1.6.jar

This library is missing because it is actually a renamed NAR archive. We need to somehow improve the way that ImageJ and NAR work together.
3. Libraries with duplicate classes:

-./jars/imglib2-ops-2.0.0-beta-26.jar
-./jars/jdom-2.0.2.jar
-./jars/edu_mines_jtk.jar

These JAR files were still hanging around, even though they were at the same time replaced with newer JARs: the imglib2-ops classes moved into imagej-ops; the classes from jdom-2.0.2 are also present in jdom2-2.0.5; and the classes of edu_mines_jtk are exactly the same as those in mines-jtk-20100113. I have now removed the old libraries. (Additionally: mines-jtk-20100113 has an incompatible license, and will be removed from the ImageJ update site in the future; see also imglib/imglib2#38, imglib/imglib2#61, fiji/registration_3d#1, imglib/imglib2-algorithm-fft#3.)
4. Third party libraries with conflicting versions:

-./jars/jama-1.0.3.jar
+./jars/jama-1.0.2.jar
-./jars/joda-time-2.3.jar
+./jars/joda-time-2.2.jar

Somehow, older versions of these libraries are being selected by Maven. Somewhere in the dependency hierarchy are components which depend on the old versions. The solution is to manage the version of these libraries in parent POMs, and use the managed version consistently throughout the entire SciJava software stack.
5. Fiji libraries with out-of-date versions:

-./plugins/Descriptor_based_registration-2.0.7.jar
+./plugins/Descriptor_based_registration-2.0.0.jar
-./plugins/SPIM_Registration-2.0.8.jar
+./plugins/SPIM_Registration-2.0.0.jar
-./plugins/TrackMate_-2.7.1.jar
+./plugins/TrackMate_-2.7.0.jar
-./plugins/Trainable_Segmentation-2.1.8.jar
+./plugins/Trainable_Segmentation-2.1.6.jar
-./plugins/bigdataviewer_fiji-1.0.2.jar
+./plugins/bigdataviewer_fiji-1.0.0.jar

These discrepancies are simply caused by plugin developers uploading new releases of their components without a corresponding bump in the relevant POM parent (i.e., pom-fiji, pom-trakem2 and pom-bigdataviewer). As we migrate toward a fully automated Jenkins-based solution, these issues will disappear since the update site will be fully driven by the versions declared in the parent POMs. For now, I fixed most of these discrepancies by hand: fiji/pom-fiji@820c172, bigdataviewer/pom-bigdataviewer@91606fb.
6. Obsolete components:

-./jars/fake-2.0.0-SNAPSHOT.jar
-./jars/fiji-scripting-2.0.0-SNAPSHOT.jar
-./jars/weave_jy2java-2.0.0-SNAPSHOT.jar
-./plugins/Algorithm_Launcher.jar
-./plugins/BeanShell_Interpreter-2.0.0-SNAPSHOT.jar
-./plugins/CLI_-2.0.0-SNAPSHOT.jar
-./plugins/Clojure_Interpreter-2.0.0-SNAPSHOT.jar
-./plugins/JRuby_Interpreter-2.0.0-SNAPSHOT.jar
-./plugins/Javascript_-2.0.0-SNAPSHOT.jar
-./plugins/Jython_Interpreter-2.0.0-SNAPSHOT.jar
-./plugins/Refresh_Javas-2.0.0-SNAPSHOT.jar
-./plugins/ij-ImageIO_.jar

These components are all obsolete and unmaintained, with no further updates planned. In particular, the various scripting interpreter components have been replaced by a unified Script Interpreter component included with the ImageJ2 Swing user interface.
7. Legitimately undeclared third party libraries:

-./jars/ant-junit.jar
-./jars/ant-launcher.jar
-./jars/ant-nodeps.jar
-./jars/ant.jar
-./jars/autocomplete.jar
-./jars/jacl.jar
-./jars/jdom-contrib.jar
-./jars/js.jar
-./jars/log4j-1.2.17.jar

We could either stop shipping these libraries with Fiji altogether, or else declare them as optional runtime dependencies in fiji/fiji.
8. LSM Toolbox plugin:

-./plugins/LSM_Reader.jar
-./plugins/LSM_Toolbox.jar

Fiji has shipped the LSM Toolbox since basically the beginning, and we even Mavenized it, but it was not declared in the toplevel fiji component for some reason. I have now done that: ddafd6f
9. Java tools library:

-./jars/tools-1.4.2.jar

We need to stop shipping this. It is actually a licensing violation to redistribute it. The scripting-java component uses it to compile Java code, but we can likely get away with using the Java 6 javax.tools.JavaCompiler API instead. Further research needed; for details see imagej/ImageJ#105.

[ctrueden]

Somehow my analysis above missed the components which are part of fiji.git but not currently present on the Fiji update site, such as Fiji_Archipelago. @larrylindsey: Are you interested in those components becoming part of core Fiji? Or would you rather keep them on a separate update site?

Also, for checking what has changed in a Fiji installation, the following invocation is really useful:

Contents/MacOS/ImageJ-macosx --update upload-complete-site --simulate Fiji

[ctrueden]

@larrylindsey Cool, thanks for the reply. I uploaded the following plugins to the core Fiji update site:

• Archipelago_Plugins
• FS_Align_TrakEM2
• Fiji_Archipelago
• TrakEM2_Archipelago

[ctrueden]

It seems I never actually removed jdom and jdom-contrib from Fiji—they were still hanging around, and finally caused some mischief. So I marked them obsolete today, after verifying that nothing needs them. (Everything in Fiji now uses jdom2.)