030 / n3dr

Nexus3 Disaster Recovery (N3DR) is a tool that is capable of downloading all artifacts from a Nexus3 server and to migrate them to another one.

N3DR

Nexus3 Disaster Recovery (N3DR).

Backup or Migrate an entire Nexus Artifact Server

Download all artifacts at once or migrate automatically from Nexus to Nexus.

Although the Nexus backup and restore documentation indicates that one could backup and restore Nexus, the data seems not to be restored completely as 500 errors occur when an artifact is downloaded from the UI after restore. It could also be possible that some steps were not issued as they have should been. Apart from that, the restore is capable of restoring the Nexus configuration.

N3DR excludes the backup of group repositories and is able to backup various repositories, configure them, restore artifacts or migrate them to another Nexus3 server. Table 1 indicates what is supported by N3DR.

Note: uploads to proxy repositories are not supported by Nexus3 itself. As a workaround one could create a hosted repository in Nexus and upload the backed up proxy content to it.

The aims of the n3dr tool are:

• to backup artifacts from a certain Nexus3 repository.
• to migrate and/or restore them to another Nexus3 server.
• configuration-as-code (cac).

type	backup	upload	label	cac
apt	x	x	^	x
bower
cocoapods
composer
conan
conda
cpan
docker	x	x	``	x
elpa
gitlfs
go
helm
maven2	x	x	+	x
npm	x	x	*	x
nuget	x	x	$	x
p2
pypi
r
raw	x	x	%	x
rubygems	x	x	-	x
yum	x	x	#	x
unknown	n/a	n/a	?

Table 1: Overview of Nexus3 types that can be downloaded, uploaded and/or configured by N3DR.

Quickstarts

• docker
• maven2
• rubygems

Instructions

• snap

Stargazers over time

n3dr's Issues

vulnerability docker scanning

https://github.com/knqyf263/trivy

Create CSV report of downloaded files.

https://stackoverflow.com/q/41063108/2777965
https://stackoverflow.com/q/43843273/2777965

Explain how this tool is different to other existing Nexus-CLIs

• https://github.com/RiotGamesMinions/nexus_cli
• https://github.com/packagemgmt/repositorytools
• https://github.com/thiagofigueiro/nexus3-cli

validate url

if !govalidator.IsURL(n.URL) {
	log.Fatalf("%v not an URL", n.URL)
}

https://medium.com/@nate510/don-t-use-go-s-default-http-client-4804cb19f779

var netClient = &http.Client{
Timeout: time.Second * 10,
}

response, _ := netClient.Get(url)

CII Best Practices Badge Program

https://bestpractices.coreinfrastructure.org/en

go routines and channels

Improve download speed by using go routines and channels.

how to upload?

docker-compose example

tool example

The docker-compose mount has issues

Refactor Nexus3.repositories()

I've selected Nexus3.repositories() for refactoring, which is a unit of 26 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

Upload logging stops

Although the logging stops, the uploading seems to continue. The progress of the upload is unclear.

stats

• run command that returns number of artifacts (total)
• return list of repositories

Refactor Nexus3.StoreArtifactsOnDisk()

I've selected Nexus3.StoreArtifactsOnDisk() for refactoring, which is a unit of 19 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

sonarcloud build badge

https://sonarcloud.io/api/project_badges/measure?project=030_nexus3-cli&metric=alert_status

https://stackoverflow.com/q/43161401/2777965

https://medium.com/@proustibat/using-sonarcloud-with-travis-ci-to-measure-and-continuously-improve-code-quality-5ea7fb5a587b

CI

https://chadmayfield.com/2018/09/01/pulling-artifacts-from-nexus-in-less-than-25-lines-of-bash/

#!/bin/bash

url="http://fileserver/service/rest/beta/search/assets?repository=maven-releases&name=dot-files"

artifact=( $(curl -s -X GET --header 'Accept: application/json' \
    "$url" | grep -Po '"downloadUrl" : ".*?[^\\]*.(zip.sha1|zip)",' | \
    awk -F '"' '{print $4}' | sort -Vr | head -n2) )

((${#artifact[@]})) || echo "ERROR! No artifacts found at provided url!"

for i in "${artifact[@]}"; do
    if [[ $i =~ (sha1) ]]; then
        checksum=$(curl -s "$i" | awk '{print $1}')
    else
        file="$(echo "$i" | awk -F "/" '{print $NF}')"
        curl -sO "$i" || { echo "ERROR: Download failed!"; exit 1; }

        if [ "$(sha1sum "$file" | awk '{print $1}')" != "$checksum" ]; then
            echo "ERROR: Checksum validation on $file failed!"; exit 1;
        else
            printf "Downloaded : %s\nChecksum   : %s\n" "$file" "$checksum"
        fi
    fi
done
#EOF

or

https://www.waltercedric.com/index.php/development/129-java/framework/maven-build-system/2302-fetching-artifact-programmatically-through-rest-api-in-nexus-3-x

#!/bin/sh

repo="https://nexus.url.com"
groupId=$1
artifactId=$2
version=$3

# optional
classifier=$4
type=$5

if [[ $type == "" ]]; then
  type="jar"
fi
if [[ $classifier != "" ]]; then
  classifier="-${classifier}"
fi

groupIdUrl="${groupId//.//}"
filename="${artifactId}-${version}${classifier}.${type}"

if [[ ${version} == *"SNAPSHOT"* ]]; then repo_type="snapshots"; else repo_type="releases"; fi

if [[ $repo_type == "releases" ]]
 then
   wget --no-check-certificate "${repo}/repository/releases/${groupIdUrl}/${artifactId}/${version}/${artifactId}-${version}${classifier}.${type}" -O ${filename} -k
 else
   versionTimestamped=$(wget -q -O- --no-check-certificate "${repo}/repository/snapshots/${groupIdUrl}/${artifactId}/${version}/maven-metadata.xml" | grep -m 1 \<value\> | sed -e 's/<value>\(.*\)<\/value>/\1/' | sed -e 's/ //g')

   wget --no-check-certificate "${repo}/repository/snapshots/${groupIdUrl}/${artifactId}/${version}/${artifactId}-${versionTimestamped}${classifier}.${type}" -O ${filename}
 fi

docker image

https://github.com/ory/dockertest

create zip option

Upload all artifacts that reside in a folder to a specific Nexus3 repository

• read all files from folder
• send pom, jar and sources

download all artifacts

Refactor Nexus3.artifactName(url string)

I've selected Nexus3.artifactName(url string) for refactoring, which is a unit of 16 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

github templates

.github

issue template etc.

see kubernetes .github as example

go-report

unhappy flow

go run main.go

results in:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x40 pc=0x65a198]

goroutine 1 [running]:
nexus3-cli/cli.continuationToken(0x6ed79f, 0x4, 0x0, 0x0)
        /home/ben/dev/nexus3-cli/cli/cli.go:35 +0x198
nexus3-cli/cli.ContinuationTokenRecursion(0x6ed79f, 0x4, 0x0, 0xc0000a9f40, 0x40712f)
        /home/ben/dev/nexus3-cli/cli/cli.go:63 +0x39
main.downloadURLs()
        /home/ben/dev/nexus3-cli/main.go:14 +0x48
main.main()
        /home/ben/dev/nexus3-cli/main.go:77 +0x20
exit status 2

if nexus is not running. The suggestion is to return a clear error message.

No artifacts in releases section

retrieve all artifacts from all repositories

curl -X GET "http://localhost:9999/service/rest/v1/repositories" -H  "accept: application/json"

3.17 support

• Increase version nexus in travis to 3.17
• fix issues
• update documentation

bettercodehub.com

https://bettercodehub.com/results/030/n3dr

codecov

https://github.com/codecov/example-go

language: go

go:
  - 1.8.x
  - tip

before_install:
  - go get -t -v ./...

script:
  - go test -race -coverprofile=coverage.txt -covermode=atomic

after_success:
  - bash <(curl -s https://codecov.io/bash)

docker-compose

What would you like to be added:

• docker-compose

Why is this needed:

• some variables are hard to set on Windows

Refactor initConfig()

I've selected initConfig() for refactoring, which is a unit of 17 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

cli

implement cobra

Refactor root.go

I've selected root.go for refactoring, which is a module of 59 lines of code that couples the top-level components tightly. Addressing this will make our codebase more maintainable and improve Better Code Hub's Couple Architecture Components Loosely guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Minimize the amount of interface code; that is, code in modules that are both called from and call modules of other components ("throughput"), and code in modules that are called from modules of other components ("incoming").
• Why❓
  Having loose coupling between top-level components makes it easier to maintain them in isolation.
• How 🔧
  You can hide a component's implementation details through various means, e.g. using the "abstract factory" design pattern.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

CHANGELOG.md

https://keepachangelog.com/en/1.0.0/

Refactor Nexus3.downloadURL(token string)

I've selected Nexus3.downloadURL(token string) for refactoring, which is a unit of 33 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

upload of standalone jars

Check whether an artifact already exists in Nexus. If true, then do not upload the artifact.

TestDownloadArtifact

func TestDownloadArtifact(t *testing.T) {
	actualError := n.downloadArtifact("http://releasesoftwaremoreoften.com")
	expectedError := "HTTP exist?"

	if actualError.Error() != expectedError {
		t.Errorf("Error incorrect. Expected: %v. Actual: %v", expectedError, actualError)
	}
}

returns

panic: runtime error: index out of range [recovered]
        panic: runtime error: index out of range

goroutine 47 [running]:
testing.tRunner.func1(0xc00026d500)
        /usr/local/go/src/testing/testing.go:830 +0x392
panic(0x70c7c0, 0x9ef6c0)
        /usr/local/go/src/runtime/panic.go:522 +0x1b5
nexus3-cli/cli.Nexus3.artifactName(0x76e168, 0x15, 0x767e86, 0x5, 0x768c6f, 0x8, 0x76b409, 0xe, 0x774405, 0x23, ...)
        /home/ben/dev/nexus3-cli/cli/cli.go:132 +0x347
nexus3-cli/cli.Nexus3.downloadArtifact(0x76e168, 0x15, 0x767e86, 0x5, 0x768c6f, 0x8, 0x76b409, 0xe, 0x774405, 0x23, ...)
        /home/ben/dev/nexus3-cli/cli/cli.go:140 +0x99
nexus3-cli/cli.TestDownloadArtifact(0xc00026d500)
        /home/ben/dev/nexus3-cli/cli/cli_test.go:140 +0x78
testing.tRunner(0xc00026d500, 0x781a20)
        /usr/local/go/src/testing/testing.go:865 +0xc0
created by testing.(*T).Run
        /usr/local/go/src/testing/testing.go:916 +0x35a
FAIL    nexus3-cli/cli  63.468s

godoc

see godoc badge at https://github.com/kubernetes/kubernetes

Refactor Nexus3.downloadArtifact(url string)

I've selected Nexus3.downloadArtifact(url string) for refactoring, which is a unit of 20 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

solve all tech debt reported by sonar

continue on error option or something

I restored everything, but then it turned out that a forgot to add a number of artifacts or new artifacts were added to the old Nexus. I tried to run the tool again, but then the following was returned:

FATA[0017] HTTPStatusCode: '400'; ResponseMessage: 'Repository does not allow updating assets: releases'; ErrorMessage: '<nil>'

I think that this is a bug as I do not want to wait for several hours again because just a couple of artifacts were missing.

Explain how to download all artifacts from all repositories at once

set admin pass

~/.n3dr.yaml

---
somePass

download all artifacts

./n3dr-linux repositories -n NEXUS_FQDN -u admin -b

download all repositories does not seem to work on windows

What happened:

• run 2.2.0 on windows
  What you expected to happen:
• download of artifacts
  How to reproduce it (as minimally and precisely as possible):
• run 2.2.0
  Anything else we need to know?:
• some logging would be helpfull
  Environment:
• windows

maven-metadata does not get stored anymore

time="2019-05-02T00:50:10+02:00" level=info msg=file1/file1 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:130"
time="2019-05-02T00:50:10+02:00" level=info msg=maven-metadata.xml func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:131"
time="2019-05-02T00:50:10+02:00" level=info msg=file2/file2/1.0.0 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:130"
time="2019-05-02T00:50:10+02:00" level=info msg=file2-1.0.0.pom.sha1 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:131"
time="2019-05-02T00:50:10+02:00" level=info msg=file2/file2/1.0.0 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:130"
time="2019-05-02T00:50:10+02:00" level=info msg=file2-1.0.0.jar.md5 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:131"
time="2019-05-02T00:50:10+02:00" level=info msg=file3/file3/1.0.0 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:130"
time="2019-05-02T00:50:10+02:00" level=info msg=file3-1.0.0.pom func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:131"
time="2019-05-02T00:50:10+02:00" level=info msg=file3/file3/1.0.0 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.go:130"
time="2019-05-02T00:50:10+02:00" level=info msg=file3-1.0.0.jar.sha1 func=nexus3-cli/cli.Nexus3.artifactName file="/home/ben/dev/nexus3-cli/cli/cli.g

This is caused by after the introduction of defining the path based on the downloadURL

Refactor Nexus3.continuationToken(token string)

I've selected Nexus3.continuationToken(token string) for refactoring, which is a unit of 20 lines of code. Addressing this will make our codebase more maintainable and improve Better Code Hub's Write Short Units of Code guideline rating! 👍

Here's the gist of this guideline:

• Definition 📖
  Limit the length of code units to 15 lines of code.
• Why❓
  Small units are easier to analyse, test and reuse.
• How 🔧
  When writing new units, don't let them grow above 15 lines of code. When a unit grows beyond this, split it in smaller units of no longer than 15 lines.

You can find more info about this guideline in Building Maintainable Software. 📖

---

ℹ️ To know how many other refactoring candidates need addressing to get a guideline compliant, select some by clicking on the 🔲 next to them. The risk profile below the candidates signals (✅) when it's enough! 🏁

---

Good luck and happy coding! ✨ 💯

solve all code-smells reported by sonar

LICENSE.md

use url library to construct urls

possibility to change nexus hostname

[BUG]: Index out of range when downloading

Tried to download entire repo, got index out of range:

panic: runtime error: index out of range

goroutine 1 [running]:
n3dr/cli.Nexus3.artifactName(0x7ffe55ce7f37, 0x24, 0x7ffe55ce7f66, 0x5, 0x7ffe55ce7f04, 0xa, 0x7ffe55ce7f19, 0x14, 0xc0004081e0, 0x52, ...)
	/go/n3dr/cli/cli.go:132 +0x329
n3dr/cli.Nexus3.downloadArtifact(0x7ffe55ce7f37, 0x24, 0x7ffe55ce7f66, 0x5, 0x7ffe55ce7f04, 0xa, 0x7ffe55ce7f19, 0x14, 0xc0004081e0, 0x52, ...)
	/go/n3dr/cli/cli.go:140 +0x8f
n3dr/cli.Nexus3.StoreArtifactsOnDisk(0x7ffe55ce7f37, 0x24, 0x7ffe55ce7f66, 0x5, 0x7ffe55ce7f04, 0xa, 0x7ffe55ce7f19, 0x14, 0xc0000000d4, 0xc0001ec000)
	/go/n3dr/cli/cli.go:196 +0x13a
main.main()
	/go/n3dr/main.go:26 +0x551

Instead of showing plain urls when downloading artifacts, consider showing a progressbar

retrieve nexus password from file

or env variable by using docker-compose and .env file

Investigate some tools

• https://codeclimate.com/github/marcells/node-build-monitor
• https://david-dm.org/marcells/node-build-monitor
• https://snyk.io/test/github/marcells/node-build-monitor