Support source bundles about p2-maven-plugin

[tombujok]

Hi Filip,

I have read the discussion from the mailing list. So, now you can manipulate the MANIFEST.MF in cases an artifact is not a proper OSGi bundle. You could use the option to re-generate the MANIFEST.MF for a jar.
But, I have never used a source-bundle so it's a bit difficult for me too understand your requirement. Could you elaborate a bit more on that? What's your use case?

Cheers,
Tom

[fikovnik]

Thanks a lot for such a quick response!

I will make an analogy to regular maven development with eclipse IDE. When you add a dependency in a pom.xml and then you run mvn eclipse:eclipse -DdownloadSources=true maven will automatically configure the project so you can easily reference sources. I would like to have the same in an eclipse plugin projects where the dependencies are configured using target platform. When a source bundle is part of a target platform it will be recognized by eclipse PDE and corresponding classes can be easily accessed.

It is just for developer's convenience.

[tombujok]

OK. No problem. But it's more a task for a eclipse plugin, don't you think?
My plugin simply generates a target site that may be consumed by the definition of the target platform.
Do you know how it works with these source bundles? Does it work like this that if I include these source jars in the p2 update site eclipse will fetch these jars automatically (do they have to have a specific name, MANIFEST.MF entries)?

[fikovnik]

The basis of the source bundles are described in [1]. I guess the good example might also be the tycho source plugin [2].

The idea is that something like this:

• for each artifact (that has includeSourceBundle set)
  • and for each dependency that is being fetched, also a corresponding -sources.jar is downloaded as well and unpacked
  • generate correct MANIFEST.MF file with (Eclipse-SourceBundle entry properly set)
  • pack as jar
  • generate <artifact-bundle-name>.source feature and include the corresponding plugin

Finally, in the p2 repo you can make two categories - runtime (original jars) and source (or even SDK that includes both).

Right now I have plenty of other work, but I can help with the implementation (especially if it seems I'm the only one who uses it).

[1] http://help.eclipse.org/indigo/index.jsp?topic=%2Forg.eclipse.pde.doc.user%2Ftasks%2Fpde_individual_source.htm
[2] http://git.eclipse.org/c/tycho/org.eclipse.tycho.git/tree/tycho-source-plugin/src/main/java/org/eclipse/tycho/source/OsgiSourceMojo.java

[tombujok]

Hi Filip,

Done. Could you please test and report feedback?

Tom

[tombujok]

BTW, if you have already generated sites the artifacts are stored in the WORKSPACE/.metadata/.plugins/org.eclipse.pde.core/.bundle_pool folder
Please remove this folder. Then refresh the site by double clicking on the address in the target file browser. Then go to the Preferences -> PLugin Development -> Target Platform -> then select the site, reload it, and reselect all dependencies.

[fikovnik]

Hi,

Good job! Looks great.

It will take me a bit more time to do a proper test, but it seems that it does exactly what it should.

There is maybe one thing that I have not really test yet. It seems that you only support transitive dependencies as first level bundles. If I have dependency A that itself depends on B you generate two bundles A,B. There might be cases when you want to actually embed some transitive dependencies inside one another (B.jar will be contained in A.jar). In this case it might be a bit trickier to generate the correct source bundle - there will be multiple roots. Also in this case you would want to be able to exclude certain dependencies.

/cheers
Filip

[tombujok]

Thanks Filip.

Don't worry, submit feedback whenever you have time to test.

When it comes to nested jars I do not support it for now. p2-maven-plugin works entirely with the maven model of dependencies (dependecies that are defined in the pom.xml). There are no nested jars in maven, so I did not really pay attention to the nested source jars. Transitive dependencies work like in Maven too, A->B->C will fetch all dependencies (if all specified in the pom.xml). The plugin's main goal is to simplify the generation of the p2 sites with 3rd party dependencies residing in the maven dependencies, that are (or aren't) OSGi bundles. When it comes to pure Eclipse-RCP dependencies (features, plugins) I assume that once you create them you expose an update site yourself. p2-maven-plugin does not understand eclipse-plugin/eclipse-feature archetypes though, so it will not parse the MAINFEST.MF to check the dependencies and to fetch sources. It's main goal was to work with maven-based dependecies and to enable the user to use them in the Eclipse-RCP environment.

Hope it helps.
Tom

[fikovnik]

I guess I did not express myself correctly. In maven you define the dependencies in pom.xml. A dependency can have a transitive dependency. In such a case this dependency is also downloaded and added to the current project with an option to be explicitly excluded. In the end you have a following structure:

P (your project)

• A (declared dependency)
  • B (transitive dependency from A)

Your plugin, on the other hand, creates A and B as top level bundles, it does not embed B inside A. For most of the cases this is surely the preferred behavior and the way OSGi bundles should work. However, in some cases (like with antlr/stringtemplate), you really want to embed the transitive dependency into a bundle and keep it all private (no export). In the end you will ended up with only one bundle A with (B inside its jar and with correct Bundle-Classpath entry).

[tombujok]

Thanks for the clarification. I might have misunderstood your point. I get now what you mean, but the only thing that I have is the definition of the transitive dependencies in maven - and maven knows nothing about embedding of jars.
Your example (antlr/stringtemplate) in the pom.xml is describe as follows (http://search.maven.org/#artifactdetails|org.antlr|antlr|3.1|jar):

<dependencies>
        <dependency>
            <groupId>org.antlr</groupId>
            <artifactId>stringtemplate</artifactId>
            <version>3.2</version>
        </dependency>
        <dependency>
            <groupId>antlr</groupId>
            <artifactId>antlr</artifactId>
            <version>2.7.7</version>
        </dependency>
    </dependencies>

which means that stringtemplate is a dependency of antlr. I cannot distunguish the case when you would like to embed it and when not, since in maven the declaration looks the same everywhere.
Right now, the plugin will generate bundle antrl with imports of stringtemplate's API and bundle stringtemplate with the exports of its API. Why would you bother to nest it? What's the added value?

[fikovnik]

Hi,

Right now, the plugin will generate bundle antrl with imports of stringtemplate's API and bundle stringtemplate with > the exports of its API. Why would you bother to nest it? What's the added value?

The problem is when you have a multitude of projects that uses different version of antlr. Now, one of the killing feature of OSGi is that it allows to form a better dependency hierarchy over the standard flat one offered by Java. On the other hand there are corner cases it does not work - wrongly assembled manifests. I know that you can easily argue that it should be solved somewhere else and I agree. However, the practice shows that it is not always that easy :-). I though it is easy to walk the dependency of maven artifacts so you could make the bundles even more customized.

I'm in the process of converting my "custom" target platform assembly into this plugin and so far it works well! Thanks a lot! It does simplify the process of adding a new dependency a lot while it leaves the possibility of not using m2e. The source bundle feature works great so I will close this issue.

btw: the execution/id has to be always default-cli?

[tombujok]

Thanks for the retest. I get your point, but it would be a bit complex to implement it right now, so maybe in the future :)

I am happy that you enjoyed using p2-maven-plugin. When the execution id is set to default-cli you can invoke the plugin directly from the command line typing "mvn p2:site" - default-cli is the defaut command-line config for the plugin (one plugin in the pom.xml can have more than one config). If you name the execution/id differently you have to bind it to a mvn phase, such as compile, process-resources, etc.

BTW, as I understand you do not use m2e. What is the reason?
p2-maven-plugin in general is not bound to m2e since it generates the p2 site. If you use m2e there's also no easy way to include a 3rd party dependency, so I think p2-maven-plugin is beneficial for both cases (with or without m2e).
Even if you add a dependency to pom.xml using m2e it will not be added to your target platform.
Do you have some other observations on this?

[fikovnik]

If you name the execution/id differently you have to bind it to a mvn phase, such as compile, process-resources, etc.

OK, this is just my misunderstanding of Maven stuff. The error message it printed (when default-cli was missing) was not completely clear to me at the beginning.

BTW, as I understand you do not use m2e. What is the reason?

So far to me (and I might be wrong), the m2e only raises the level of complexity of the development environment configuration. I was hitting the problem that there are not the appropriate connectors for the plugins I was using. Most of the work is done in Scala and that did not help either - the m2e connector for Scala plugin is work in progress. Eclipse with EMF and Scala is already a resource glutton so with m2e things got only worse. In the end I had a hard time to justify this accidental complexity to my colleagues who are mostly academics.
Right now the setup is simple and works well. Adding a new plugin to the project is easy as well as adding a new dependency (thanks to your plugin).
I might be missing something, but at this time I do not see what is the advantage of using m2e.

If you use m2e there's also no easy way to include a 3rd party dependency, so I think p2-maven-plugin is beneficial for both cases (with or without m2e).

I might be wrong, I guess you can use the consider pom dependencies and it is recognized in eclipse, no? Otherwise, I would be really puzzled how ppl are do ing it? I asked at the tycho-user forum, but I understand that the question is irrelevant if I do not consider using m2e.

[tombujok]

Yes, I agree that m2e does not give you much, and the whole environment is heavyweight. For me, the whole maven / tycho setup is a nightmare, with pom.xml files, build.properties, MANIFEST.MF and a horrible duplication of information.
I use m2e to import the project quicker, as well as if I add a new maven module it's recognized automatically. I guess you use mvn eclipse:eclipse and reimport? That's also reasonable, I have to test it as well.

When it comes to m2e and dependencies it does not help you at all! When you set pom dependencies to consider the dependencies will be fetched and included only in the command-line. When you import the project to the IDE/PDE, you do not see them, since they do not reside in the target platform - and target platform can contain update-sites only... Tat is the reason why the project will not compile. I don't know what people do, but it's a mess, that's why I developed my plugin. Whenever somebody on our project wanted to add a 3rd party dependency it was an epiphany. The project used to stop compiling or people had problems to import it to an IDE/PDE.

I tried to discuss this issue on the tycho-dev list: http://dev.eclipse.org/mhonarc/lists/tycho-dev/msg00756.html
You can have a look at the whole thread. Tycho developers, however, completely ignored the topic, as if everything was fine. Maybe for them it's fine, but for me it's really problematic and over-engineered (I talked to many people, and 3rd party dependencies cause problems in most of the companies that use RCP and tycho). Maybe some day I will fork tycho and apply these changes myself, but without the support of the community it does not make sense at all, so I will probably invest my very limited open-source time somewhere else.

[fikovnik]

Interesting! I cannot agree with you more about the pain of setting up the external dependencies (neither can I express the happiness when I found your plugin! :-) ).

I have a very similar experience with tycho mailing list wrt the same subject. Even though I was only asking for best practices. I did not really wanted to spend more time on it (I have already spent too much time on maven which is hard to justify) and my solution as complex as it was worked fine. Then I found your plugin that simplified the whole setup drastically. However, it really makes me wonder how the ppl go around the dependencies on 3rd party libraries that are not available at any existing p2 repos?

One last thing, perhaps you could also add to your (already very solid) documentation how do you integrate your plugin to the development process. I mean, what is the best way how to have both tycho and maven aware of the repository that has been built. Also it would be really handy that you could put a link to your plugin to tycho website. There must be more ppl with similar needs :-)