microsoft / mvnfeed-cli

Tool to transfer Maven artifacts between repositories

MvnFeed CLI

The MvnFeed CLI is a command line interface to transfer artifacts from one Maven repository into another.

The tool has been developed to help teams use the Azure Artifacts infrastructure until the upstream source for Maven feeds is available.

The CLI downloads artifacts from one repository into a stage directory before uploading them onto the target repository.

Installation

To install:

1. Ensure prerequisites are installed:

   • Python2 or 3

2. Check out the Git repository;

3. Run the install script:

   # python 2
   python scripts/dev_setup.py
   
   # python 3
   python3 scripts/dev_setup.py

4. Verify that the CLI has been installed correctly:

   mvnfeed -h

Configuration

The configuration file for the MvnFeed CLI is stored in the file ~/.mvnfeed/mvnfeed.ini and can be edited manually, but the CLI also provides methods to add/remove configuration values.

1. set the directory where artifacts will be saved once downloaded:

   mvnfeed config stage_dir set --path /tmp/artifacts

2. add a repository:

   mvnfeed config repo add \
       --name my_devops_feed \
       --username my_username
       --url https://{organization}.pkgs.visualstudio.com/_packaging/{feed}/maven/v1

   About authorization: currently only basic auth is supported but if your repository supports other authentication means, edit the configuration file and directly modify the authorization value. This value will be passed unmodified in the Authorization HTTP header.

3. list the configured repositories:

   mvnfeed config repo list
   central
     url : http://central.maven.org/maven2
   jcenter
     url : http://jcenter.bintray.com
   jboss
     url : https://repository.jboss.org/nexus/content/repositories/releases
   clojars
     url : https://repo.clojars.org
   atlassian
     url : https://packages.atlassian.com/maven/public
   google
     url : https://maven.google.com
   my_devops_feed
     url : https://{organization}.pkgs.visualstudio.com/_packaging{feed}/maven/v1

Usage

If only a couple of dependencies need to be transferred, you can choose to transfer them one by one:

mvnfeed artifact transfer \
    --from=[REPO_NAME] \
    --to=[REPO_NAME] \
    --name=[GROUP_ID]:[ARTIFACT_ID]:[VERSION] \
    --transfer_deps

where:

• [REPO_NAME] is the name of one of the configured repositories;
• transfer_deps: when defined, dependencies defined in POM files will also be transferred.

For more than a handful of dependencies, it is best to define them in a file and use the bulk-transfer command:

artifacts artifact bulk-transfer \
    --from=[REPO_NAME] \
    --to=[REPO_NAME] \
    --filename=[FILENAME] \
    --transfer_deps

where:

• [FILENAME] is the name of a file listing the artifacts in the following format:

  {GROUP_ID}:{ARTIFACT_ID}:{TYPE}[:{ARCHITECTURE}]:{VERSION}
  

• transfer_deps: when defined, dependencies defined in POM files will also be transferred.

For example:

org.projectlombok:lombok:1.18.2
com.microsoft.azure:azure-dependencies-bom:pom:1.0.0.M4
org.elasticsearch:elasticsearch:test-jar:tests:2.4.4
io.netty:netty-transport-native-epoll:jar:linux-x86_64:4.1.28.Final

Important notes

• detail information about the transfer can be found in the mvnfeed.log logfile created in the current directory;

• because it is not possible to delete uploaded artifacts, mvnfeed tries not to be too clever when parsing the POM files (like deducting versions from dependencyManagement block and such.) As a result, you may see following output on the console:

  missing explicit version for jmock:jmock in hamcrest-parent-1.1.pom. Skipping

  this means that you will have to look at the given POM file and manually upload the skipped dependency, if necessary.

• use the :pom type to explicitely transfer a BOM file (e.g. com.microsoft.azure:azure-dependencies-bom:pom:1.0.0.M4)

Recommanded workflow

Because you may encounter a very large number of missing versions, it is advised to first list the exact dependencies that must be transferred.

1. create a Gradle/Maven project that only includes the desired dependencies, with the only caveat that the Gradle/Maven is not using their respective caches;

2. get the list of all transitive dependencies:

   # gradle
   ./gradlew --gradle-user-home . --debug dependencies > dependencies.txt
   
   # maven
   mvn dependency:list -DoutputFile=dependencies.txt

3. use mvnfeed to do some cleanup:

   # gradle
   mvnfeed input cleanup \
       --type=gradle \
       --input_file=dependencies.txt \
       --ouput_file=dependencies_cleanup.txt
   
   # maven
   mvnfeed input cleanup \
       --type=maven \
       --input_file=dependencies.txt \
       --ouput_file=dependencies_cleanup.txt

4. transfer the dependencies (without the --transfer_deps flag):

   artifacts artifact bulk-transfer \
       --from=[REPO_NAME] \
       --to=[REPO_NAME] \
       --filename=dependencies_cleanup.txt

Feedback

If you have any issues, questions, comments, or feature requests regarding this tool, please file an issue within this Github repo using our contribution guidelines.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

mvnfeed-cli's Issues

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

transfer fails with No module named 'requests'

it has installed no worries, added repos to the config, and then tried the transfer. it's between 2 az artifact feeds. Wondering if it's the version of python I'm using? (Python 3.9.0)

Thanks for your time :)

mvnfeed artifact transfer
--from=[REPO_NAME]
--to=[REPO_NAME]
--name=[GROUP_ID]:[ARTIFACT_ID]:[VERSION]
--transfer_deps

No module named 'requests'
Traceback (most recent call last):
File "C:\Python39\lib\site-packages\knack\cli.py", line 206, in invoke
cmd_result = self.invocation.execute(args)
File "C:\Python39\lib\site-packages\knack\invocation.py", line 134, in execute
self.commands_loader.load_arguments(command)
File "c:\dev\projects\mvnfeed-cli\src\mvnfeed-cli\mvnfeed\cli\mvnfeed_commands.py", line 13, in load_arguments
super(MvnFeedCommandsLoader, self).load_arguments(command)
File "C:\Python39\lib\site-packages\knack\commands.py", line 181, in load_arguments
self.command_table[command].load_arguments()
File "C:\Python39\lib\site-packages\knack\commands.py", line 76, in load_arguments
cmd_args = self.arguments_loader()
File "C:\Python39\lib\site-packages\knack\commands.py", line 216, in arguments_loader
return list(extract_args_from_signature(CLICommandsLoader._get_op_handler(operation),
File "C:\Python39\lib\site-packages\knack\commands.py", line 233, in get_op_handler
op = import_module(mod_to_import)
File "C:\Python39\lib\importlib_init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1030, in _gcd_import
File "", line 1007, in _find_and_load
File "", line 986, in _find_and_load_unlocked
File "", line 680, in _load_unlocked
File "", line 790, in exec_module
File "", line 228, in _call_with_frames_removed
File "c:\dev\projects\mvnfeed-cli\src\mvnfeed_modules\mvnfeed-cli-transfer\mvnfeed\cli\transfer\transfer.py", line 8, in
import requests
ModuleNotFoundError: No module named 'requests'

gradle metadata transfer

Hi.
Pls check if there is a gradle's *.module metadata file present along with *.pom.
And transfer it too

Doesn't transfer WAR files only JAR files

I have a package that publishes a .war file:

When transferred, it doesn't transfer the .war file and throws a warning that the .jar file it is expecting cannot be found

myshuttle-1.0.5.jar was not found in the repository

Here is what gets transferred:

Can't seem to run it :(

I use rtx to manage my python environments, and I have the latest python3 installed, and yet, for some reason it doesn't run:

❯ mvnfeed -h
Traceback (most recent call last):
  File "/Users/Me/.local/share/rtx/installs/python/3.11.5/bin/mvnfeed", line 5, in <module>
    from mvnfeed.cli.__main__ import main
  File "/Users/Me/gitwork/mvnfeed-cli/src/mvnfeed-cli/mvnfeed/cli/__main__.py", line 8, in <module>
    from .mvnfeed_cli import MvnFeedCLI
  File "/Users/Me/gitwork/mvnfeed-cli/src/mvnfeed-cli/mvnfeed/cli/mvnfeed_cli.py", line 4, in <module>
    from mvnfeed.cli.common.config import GLOBAL_CONFIG_DIR, CLI_ENV_VARIABLE_PREFIX, load_config
ModuleNotFoundError: No module named 'mvnfeed.cli.common'

I don't use a lot of python things by hand, so I don't know what I'd need to do to make it work.

Pom transfer error

Hi,

I have a problem when i transfer a jar+pom from a nexus repo to another one nexus (3.15.2-01 OSS).

The content of the pom is modified during the upload, for exemple the content below after upload, some request informations have been added before and after the pom content :

--d92f595d39c0f2a21fbefc29acf43172
Content-Disposition: form-data; name="/tmp/artifacts/dolmenUtils-2.4.1.pom"; filename="dolmenUtils-2.4.1.pom"

....

--d92f595d39c0f2a21fbefc29acf43172--

For information, the download on the local machine is good, the pom file in /tmp/artifacts/ has a good content.

I use python 3.7.