artifactory-cleanup is an extended and flexible cleanup tool for JFrog Artifactory.
The tool has simple YAML-defined cleanup configuration and can be extended with your own rules on Python. Everything must be as a code, even cleanup policies!
As simple as one command!
# docker
docker pull devopshq/artifactory-cleanup
docker run --rm devopshq/artifactory-cleanup artifactory-cleanup --help
# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup
artifactory-cleanup --helpSuppose you want to remove all artifacts older than N days from reponame repository. You should take the following steps:
- Install
artifactory-cleanup(see above) - Create a configuration file
artifactory-cleanup.yaml. variables.
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
# $VAR is auto populated from environment variables
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD
policies:
- name: Remove all files from repo-name-here older than 7 days
rules:
- rule: Repo
name: "reponame"
- rule: DeleteOlderThan
days: 7- Run the command TO SHOW (not remove) artifacts that will be deleted. By default
artifactory-cleanupuses "dry mode".
# Set the credentials with delete permissions
export ARTIFACTORY_USERNAME=usernamehere
export ARTIFACTORY_PASSWORD=password
# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup
# cli
artifactory-cleanup- Verify that right artifacts will be removed and add
--destroyflag TO REMOVE artifacts:
# docker
docker run --rm -v "$(pwd)":/app -e ARTIFACTORY_USERNAME -e ARTIFACTORY_PASSWORD devopshq/artifactory-cleanup artifactory-cleanup --destroy
# cli
artifactory-cleanup --destroyLooking for more examples? Check examples folder!
- Always specify version of
artifactory-cleanupwhen using it in the production.1.0.0is just an example, find the latest version in pypi: https://pypi.org/project/artifactory-cleanup/
# docker
docker pull devopshq/artifactory-cleanup:1.0.0
docker run --rm devopshq/artifactory-cleanup:1.0.0 artifactory-cleanup --version
# python (later we call it 'cli')
python3 -mpip install artifactory-cleanup==1.0.0
artifactory-cleanup --help- Use CI servers or cron-like utilities to run
artifactory-cleanupevery day (or every hour). TeamCity and GitHub have built-in support and show additional logs format - Do not save credentials in the configuration file, use environment variables.
- Use
--ignore-not-foundflag to ignore errors when the repository is not found. It's useful when you have a configuration for multiple repositories and some of them are not found. - Use
--worker-count=<WORKER_NUM>to increase the number of workers. By default, it's 1. It's useful when you have a lot of artifacts and you want to speed up the process.
# Debug - "dry run" mode by default
# debug run - only print artifacts. it does not delete any artifacts
artifactory-cleanup
# Debug run only for policytestname.
artifactory-cleanup --policy-name policytestname
# REMOVE
# For remove artifacts use --destroy
artifactory-cleanup --destroy
# For remove artifacts use environment variable
export ARTIFACTORY_CLEANUP_DESTROY=True
artifactory-cleanup
# Specify config filename
artifactory-cleanup --config artifactory-cleanup.yaml
# Specify config filename using environment variable
export ARTIFACTORY_CLEANUP_CONFIG_FILE=artifactory-cleanup.yaml
artifactory-cleanup --config artifactory-cleanup.yaml
# Look in the future - shows what the tool WILL remove after 10 days
artifactory-cleanup --days-in-future=10
# Not satisfied with built-in rules? Write your own rules in python and connect them!
artifactory-cleanup --load-rules=myrule.py
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py
# Save the table summary in a file
artifactory-cleanup --output=myfile.txt
# Save the summary in a Json file
artifactory-cleanup --output=myfile.txt --output-format=jsonRepo- Apply the rule to one repository. If no name is specified, it is taken from the rule name (inCleanupPolicydefinition)
- rule: Repo
name: reponame# OR - if you have a single policy for the repo - you can name the policy as reponame
# Both configurations are equal
policies:
- name: reponame
rules:
- rule: RepoRepoList- Apply the policy to list of repositories.
- rule: RepoList
repos:
- repo1
- repo2
- repo3RepoByMask- Apply rule to repositories matching by mask
- rule: RepoByMask
mask: "*.banned"PropertyEq- Delete repository artifacts only with a specific property value (property_key is the name of the parameter, property_value is the value)
- rule: PropertyEq
property_key: key-name
property_value: 1PropertyNeq- Delete repository artifacts only if the value != specified. If there is no value, delete it anyway. Allows you to specify the deletion flagdo_not_delete = 1
- rule: PropertyNeq
property_key: key-name
property_value: 1DeleteOlderThan- deletes artifacts that are older than N days
- rule: DeleteOlderThan
days: 1DeleteWithoutDownloads- deletes artifacts that have never been downloaded (DownloadCount=0). Better to use withDeleteOlderThanrule
- rule: DeleteWithoutDownloadsDeleteOlderThanNDaysWithoutDownloads- deletes artifacts that are older than N days and have not been downloaded
- rule: DeleteOlderThanNDaysWithoutDownloads
days: 1DeleteNotUsedSince- delete artifacts that were downloaded, but for a long time. N days passed. Or not downloaded at all from the moment of creation and it's been N days
- rule: DeleteNotUsedSince
days: 1DeleteEmptyFolders- Clean up empty folders in given repository list
- rule: DeleteEmptyFoldersDeleteByRegexpName- delete artifacts whose name matches the specified regexp
- rule: DeleteByRegexpName
regex_pattern: "\d"KeepLatestNFiles- Leaves the last (by creation time) files in the amount of N pieces. WITHOUT accounting subfolders
- rule: KeepLatestNFiles
count: 1KeepLatestNFilesInFolder- Leaves the last (by creation time) files in the number of N pieces in each folder
- rule: KeepLatestNFilesInFolder
count: 1KeepLatestVersionNFilesInFolder- Leaves the latest N (by version) files in each folder. The definition of the version is using regexp. By default it parses semver using the regex -([\d]+\.[\d]+\.[\d]+)")
- rule: KeepLatestVersionNFilesInFolder
count: 1
custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"KeepLatestNupkgNVersions- Leaves N nupkg (adds*.nupkgfilter) in release feature builds
- rule: KeepLatestNupkgNVersions
count: 1DeleteDockerImagesOlderThan- Delete docker images that are older than N days
- rule: DeleteDockerImagesOlderThan
days: 1DeleteDockerImagesOlderThanNDaysWithoutDownloads- Deletes docker images that are older than N days and have not been downloaded
- rule: DeleteDockerImagesOlderThanNDaysWithoutDownloads
days: 1DeleteDockerImagesNotUsed- Removes Docker image not downloaded since N days
- rule: DeleteDockerImagesNotUsed
days: 1IncludeDockerImages- Apply to docker images with the specified names and tags
- rule: IncludeDockerImages
masks: "*singlemask*"
- rule: IncludeDockerImages
masks:
- "*production*"
- "*release*"ExcludeDockerImages- Exclude Docker images by name and tags.
- rule: ExcludeDockerImages
masks:
- "*production*"
- "*release*"KeepLatestNVersionImagesByProperty(count=N, custom_regexp='some-regexp', number_of_digits_in_version=X)- Leaves N Docker images with the same major.(^\d+\.\d+\.\d+$)is the default regexp how to determine version which matches semver1.1.1. If you need to add minor then setnumber_of_digits_in_versionto 2 or if patch then set to 3 (by default we match major, which 1). Semver tags prefixed withvare supported by updating the regexp to include (an optional)vin the expression (e.g.,(^v?\d+\.\d+\.\d+$)).
- rule: KeepLatestNVersionImagesByProperty
count: 1
custom_regexp: "[^\\d][\\._]((\\d+\\.)+\\d+)"-
DeleteDockerImageIfNotContainedInProperties(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)- Remove Docker image, if it is not found in the properties of the artifact repository. -
DeleteDockerImageIfNotContainedInPropertiesValue(docker_repo='docker-local', properties_prefix='my-prop', image_prefix=None, full_docker_repo_name=None)- Remove Docker image, if it is not found in the properties of the artifact repository.
IncludePath- Apply to artifacts by path / mask.
- rule: IncludePath
masks: "*production*"
- rule: IncludePath
masks:
- "*production*"
- "*develop*"IncludeFilename- Apply to artifacts by name/mask
- rule: IncludeFilename
masks:
- "*production*"
- "*develop*"ExcludePath- Exclude artifacts by path/mask
- rule: ExcludePath
masks:
- "*production*"
- "*develop*"ExcludeFilename- Exclude artifacts by name/mask
- rule: ExcludeFilename
masks:
- "*.tag.gz"
- "*.zip"If you want to create your own rule, you can do it!
The basic flow how the tool calls Rules:
Rule.check(*args, **kwargs)- verify that the Rule configured right. Call other services to get more information.Rule.aql_add_filter(filters)- add Artifactory Query Language expressionsRule.aql_add_text(aql)- add text to the result aql queryartifactory-cleanupcalls Artifactory with AQL and pass the result to the next stepRule.filter(artifacts)- filter out artifacts. The method returns artifacts that will be removed!.- To keep artifacts use
artifacts.keep(artifact)method
- To keep artifacts use
Create myrule.py file at the same folder as artifactory-cleanup.yaml:
# myrule.py
from typing import List
from artifactory_cleanup import register
from artifactory_cleanup.rules import Rule, ArtifactsList
class MySimpleRule(Rule):
"""
This doc string is used as rule title
For more methods look at Rule source code
"""
def __init__(self, my_param: str, value: int):
self.my_param = my_param
self.value = value
def aql_add_filter(self, filters: List) -> List:
print(f"Today is {self.today}")
print(self.my_param)
print(self.value)
return filters
def filter(self, artifacts: ArtifactsList) -> ArtifactsList:
"""I'm here just to print the list"""
print(self.my_param)
print(self.value)
# You can make requests to artifactory by using self.session:
# url = f"/api/storage/{self.repo}"
# r = self.session.get(url)
# r.raise_for_status()
return artifacts
# Register your rule in the system
register(MySimpleRule)Use rule: MySimpleRule in configuration:
# artifactory-cleanup.yaml
- rule: MySimpleRule
my_param: "Hello, world!"
value: 42Specify --load-rules to the command:
# docker
docker run -v "$(pwd)":/app devopshq/artifactory-cleanup artifactory-cleanup --load-rules=myrule.py
# cli
artifactory-cleanup --load-rules=myrule.pyIn case you have set up your Artifactory self-signed certificates, place all certificates of the chain of trust into
the certificates folder and add additional argument to the command:
docker run -v "$(pwd)":/app -v "$(pwd)/certificates":/mnt/self-signed-certs/ devopshq/artifactory-cleanup artifactory-cleanupWe can handle conan's metadata by creating two policies:
- First one removes files but keep all metadata.
- Second one look at folders and if it contains only medata files - removes it (because there's no associated with metadata files)
The idea came from #47
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD
policies:
- name: Conan - delete files older than 60 days
rules:
- rule: Repo
name: "conan-testing"
- rule: DeleteNotUsedSince
days: 60
- rule: ExcludeFilename
masks:
- ".timestamp"
- "index.json"
- name: Conan - delete empty folders (to fix the index)
rules:
- rule: Repo
name: "conan-testing"
- rule: DeleteEmptyFolders
- rule: ExcludeFilename
masks:
- ".timestamp"
- "index.json"We can combine docker rules with usual "files" rules!
The idea came from #61
# artifactory-cleanup.yaml
artifactory-cleanup:
server: https://repo.example.com/artifactory
user: $ARTIFACTORY_USERNAME
password: $ARTIFACTORY_PASSWORD
policies:
- name: Remove docker images, but keep last 3
rules:
# Select repo
- rule: Repo
name: docker-demo
# Delete docker images older than 30 days
- rule: DeleteDockerImagesOlderThan
days: 30
# Keep these tags for all images
- rule: ExcludeDockerImages
masks:
- "*:latest"
- "*:release*"
# Exclude these docker tags
- rule: ExcludePath
masks: "*base-tools*"
# Keep 3 docker tags for all images
- rule: KeepLatestNFilesInFolder
count: 3In order to provide a new release of artifactory-cleanup, there are two steps involved.
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot]")
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent