seagate / cortx-re Goto Github PK
View Code? Open in Web Editor NEWCORTX Release Engineering Scripts and Dockerfiles.
Home Page: https://github.com/Seagate/cortx
License: GNU Affero General Public License v3.0
CORTX Release Engineering Scripts and Dockerfiles.
Home Page: https://github.com/Seagate/cortx
License: GNU Affero General Public License v3.0
GitHub Packages provides feature to store docker images. We should move all RE related docker images to GitHub Packages of cortx-re repository.
Latest build RELEASE.INFO shows OS: "Red Hat Enterprise Linux Server release 7.8 (Maipo)"
is should be Red Hat Enterprise Linux Server release 7.7 (Maipo)
AFAIK we are using Dockerfile to build Base Image and rhel7-init:7.7
has 7.7 version.
[root@ssc-vm-0245 ~]# docker run -it registry.access.redhat.com/rhel7-init:7.7 cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.7 (Maipo)
[root@ssc-vm-0245 ~]#
Please verify and correct.
custom build are not supported by HW deployment due to following reasons
Currently repositories in Docker images are pointed to http://ci-storage.mero.colo.seagate.com/releases/eos/github/master/rhel-7.7.1908/
after branch name change need them to update to http://ci-storage.mero.colo.seagate.com/releases/eos/github/release/rhel-7.7.1908/
Also now CentOS repository links are installing centos-release for CentOS 7.8 version. Hence need to exclude it duringyum upgrade all
We should implement below for regular OVA releases.
OVA Build frequency - Every Sprint End ( 2 Weeks)
OVA Versioning - As mentioned in Seagate/cortx#441 (comment) We should use semantic versioning for OVA . Currently we have released cortx-va-1.0.*.zip file at https://github.com/Seagate/cortx/releases/tag/VA . We should create new GitHub releases names cortx-va-1.0. and upload cortx-va-1.0..zip into it.
Release Notes - We can add Jira query or GitHub log query output to list down changes landed into main branch in sprint. For internal releases we used to send release notes with Jira issues fixed within sprint along with Provisioning guide. @Venkataraman has added Release Notes template at https://github.com/Seagate/cortx/blob/main/doc/RNTemplate.rst . We can improve and add it as description to GitHub release.
Refer sample LDR Release notes from Release Build #2714 for reference. Jira query used is Sprint = EES_Sprint-33 AND resolutiondate >= '2020-12-17 00:00' AND resolutiondate <= '2020-12-29 00:00'
. Refer detailed output at - Sprint end Jira query
You can use get-jira-stats.sh and changelog.sh to fetch Git and Jiras stats for sprint automatically.
Document Location :
Gitlab
Due to security issues we need to disable multi-branch job's for components. We have created have Jenkins pipelines with manual trigger for validation at http://eos-jenkins.colo.seagate.com/job/Cortx-PR-Build/
However this will be additional step for Seagate developers also for PR. Currently Jenkins pipelines take repo url and branch names as input to trigger job.
As discussed with @gowthamchinna . We can improve it further to make it easy.
Observed that during cortx-manager build for cortx-1.0
branch cortx-provisioner is installed from main
branch.
16:43:16 + yum install -y cortx-prvsnr
16:43:16 Loaded plugins: fastestmirror, ovl, priorities
16:43:17 Determining fastest mirrors
16:43:25 Resolving Dependencies
16:43:25 --> Running transaction check
16:43:25 ---> Package cortx-prvsnr.x86_64 0:1.0.0-417_gitbe88d99_el7 will be installed
16:43:25 --> Processing Dependency: python36-PyYAML for package: cortx-prvsnr-1.0.0-417_gitbe88d99_el7.x86_64
16:43:25 --> Running transaction check
16:43:25 ---> Package python36-PyYAML.x86_64 0:3.13-1.el7 will be installed
16:43:25 --> Finished Dependency Resolution
16:43:25
16:43:25 Dependencies Resolved
16:43:25
16:43:25 ================================================================================
16:43:25 Package Arch Version Repository Size
16:43:25 ================================================================================
16:43:25 Installing:
16:43:25 cortx-prvsnr x86_64 1.0.0-417_gitbe88d99_el7 cortx-C7.7.1908 292 k
16:43:25 Installing for dependencies:
16:43:25 python36-PyYAML x86_64 3.13-1.el7 epel 149 k
16:43:25
16:43:25 Transaction Summary
16:43:25 ================================================================================
16:43:25 Install 1 Package (+1 Dependent package)
We need to update our base docker image with correct versioning to support future activities. These docker images are acts as a build environment for cortx components.
Consider
Suggested Docker image naming pattern
If we plan to use single docker image for the build process
If we plan to use multiple docker image
Note : docker image name cortx, cortx_base, cortx_core will be reserved for future cortx docker workflow
Example
cortx_build_env:centos-7.7.1908
cortx_build_env:centos-7.8.2003
cortx_build_env:centos-8.0.1905
cortx_build_env:rhel-7.7
cortx_build_env:rhel-7.8
...
How others doing
Build environment docker images mostly not available in public domain anyhow added some application image naming patterns here
minio/minio:RELEASE.2020-09-26T03-44-56Z-s390x
postgres:13-alpine
maven:3.6.3-ibmjava-alpine
sonarqube:8.4.2-community
telegraf:1.14.5-alpine
jenkins:2.60.2-alpine
kapacitor:1.5.5-alpine
@shailesh-vaidya please provide your suggestion on this
(venv_cortx) [root@localhost ~]# provisioner setup_provisioner srvnode-1:$(hostname -f) --logfile --logfile-filename /var/log/seagate/provisioner/setup.log --source rpm --config-path ~/config.ini --dist-type bundle --target-build ${CORTX_RELEASE_REPO}
2021-02-09 06:20:32,858 - INFO - Starting to build setup 'srvnode-1_root@localhost_22'
2021-02-09 06:20:32,863 - INFO - Profile location '/root/.provisioner/srvnode-1_root@localhost_22'
2021-02-09 06:20:32,863 - INFO - Preparing setup pillar
2021-02-09 06:20:32,864 - INFO - Generating setup keys [skipped]
2021-02-09 06:20:32,864 - INFO - Generating a roster file
2021-02-09 06:20:32,865 - INFO - Copying config.ini to file root
2021-02-09 06:20:32,871 - INFO - Preparing CORTX repos pillar
2021-02-09 06:20:32,887 - INFO - Ensuring 'srvnode-1' is ready to accept commands
2021-02-09 06:20:59,600 - INFO - Resolving node grains
2021-02-09 06:21:01,621 - INFO - Preparing salt-masters / minions configuration
2021-02-09 06:21:01,628 - INFO - Preparing factory profile
2021-02-09 06:21:01,668 - INFO - Installing Cortx yum repositories
2021-02-09 06:21:08,409 - INFO - Setting up custom python repository
2021-02-09 06:21:12,883 - INFO - Checking paswordless ssh
2021-02-09 06:21:17,125 - INFO - Configuring the firewall
2021-02-09 06:21:21,339 - INFO - Installing SaltStack
2021-02-09 06:21:31,000 - INFO - Installing provisioner from a 'rpm' source
2021-02-09 06:21:41,766 - INFO - Configuring salt minions
2021-02-09 06:21:46,698 - INFO - Configuring salt-masters
2021-02-09 06:22:11,553 - INFO - Copying factory data
2021-02-09 06:22:26,895 - INFO - Installing provisioner API
2021-02-09 06:22:32,678 - INFO - Starting salt minions
2021-02-09 06:22:37,222 - INFO - Ensuring salt-masters are ready
2021-02-09 06:22:51,941 - INFO - Ensuring salt minions are ready
2021-02-09 06:23:05,813 - INFO - Updating release distribution type
2021-02-09 06:23:17,736 - ERROR - salt command failed, reason {'srvnode-1': 'Traceback (most recent call last):\n File "/usr/bin/provisioner", line 10, in \n from importlib.metadata import distribution\nModuleNotFoundError: No module named 'importlib.metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 13, in \n from importlib_metadata import distribution\nModuleNotFoundError: No module named 'importlib_metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 570, in _build_master\n ws.require(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 888, in require\n needed = self.resolve(parse_requirements(requirements))\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 779, in resolve\n raise VersionConflict(dist, req).with_context(dependent_req)\npkg_resources.ContextualVersionConflict: (salt 3001.1 (/usr/lib/python3.6/site-packages), Requirement.parse('salt>=3002.2'), {'cortx-prvsnr'})\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 15, in \n from pkg_resources import load_entry_point\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3095, in \n @_call_aside\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3079, in _call_aside\n f(*args, **kwargs)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3108, in _initialize_master_working_set\n working_set = WorkingSet._build_master()\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 572, in _build_master\n return cls._build_from_requirements(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 585, in _build_from_requirements\n dists = ws.resolve(reqs, Environment())\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 774, in resolve\n raise DistributionNotFound(req, requirers)\npkg_resources.DistributionNotFound: The 'salt>=3002.2' distribution was not found and is required by cortx-prvsnr'}, args {'targets': 'srvnode-1', 'fun': 'cmd.run', 'fun_args': ['provisioner pillar_set --fpath release.sls release/type '"bundle"''], 'fun_kwargs': {}, 'kw': {'roster_file': '/root/.provisioner/srvnode-1_root@localhost_22/srv/config/roster', 'ssh_options': ['UserKnownHostsFile=/dev/null', 'StrictHostKeyChecking=no']}, 'secure': False}
2021-02-09 06:23:17,737 - ERROR - provisioner failed
{'srvnode-1': 'Traceback (most recent call last):\n File "/usr/bin/provisioner", line 10, in \n from importlib.metadata import distribution\nModuleNotFoundError: No module named 'importlib.metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 13, in \n from importlib_metadata import distribution\nModuleNotFoundError: No module named 'importlib_metadata'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 570, in _build_master\n ws.require(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 888, in require\n needed = self.resolve(parse_requirements(requirements))\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 779, in resolve\n raise VersionConflict(dist, req).with_context(dependent_req)\npkg_resources.ContextualVersionConflict: (salt 3001.1 (/usr/lib/python3.6/site-packages), Requirement.parse('salt>=3002.2'), {'cortx-prvsnr'})\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File "/usr/bin/provisioner", line 15, in \n from pkg_resources import load_entry_point\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3095, in \n @_call_aside\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3079, in _call_aside\n f(*args, **kwargs)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 3108, in _initialize_master_working_set\n working_set = WorkingSet._build_master()\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 572, in _build_master\n return cls._build_from_requirements(requires)\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 585, in _build_from_requirements\n dists = ws.resolve(reqs, Environment())\n File "/usr/lib/python3.6/site-packages/pkg_resources/init.py", line 774, in resolve\n raise DistributionNotFound(req, requirers)\npkg_resources.DistributionNotFound: The 'salt>=3002.2' distribution was not found and is required by cortx-prvsnr'}
We need to create mechanism to build integrated build for all CORTX components. Same is requested in Seagate/cortx#204 . This ticket is to track development for this.
Plan -
docker run -v /var/artifacts:/var/artifacts -v /root/cortx:/cortx-workspace cortx_build:7.7.1908 make clean cortx-prvsnr
/var/artifacts/
We need to take everyday backup of all Seagate GitHub repositories including PR, wiki, pull issues...etc
https://docs.github.com/en/enterprise/2.20/user/github/creating-cloning-and-archiving-repositories/backing-up-a-repository
https://docs.github.com/en/github/creating-cloning-and-archiving-repositories/backing-up-a-repository
Note : if this already in-place please provide backup strategy we are following to ensure a perfect disaster recovery.
If this repo contains the scripts that we use to build CORTX, then shouldn't we share it with the community? They need to know how to build CORTX also. Any private stuff in here?
Tagging @shailesh-vaidya and @amolkongre
As of now we are only doing workspace check in the build targets, in future if we require more validations then its suggested to create a generic validate target and reuse the same on each component build.
Build Makefile : https://github.com/seagate/cortx-re/blob/main/docker/cortx-build/Makefile
Ref : #245
We should use generic workflow to build component rpm's . I have created one for Provisioner at provisioner-build.yml . It will also need changes in Dockerfile's .
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: ubuntu-latest
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout@v2
- name: Checkout
uses: actions/checkout@v2
with:
# Repository name with owner. For example, actions/checkout
repository: Seagate/cortx-prvsnr
token: ${{ secrets.TOKEN }}
path: provisioner
# Runs a set of commands using the runners shell
- name: Build Provisioner packages
run: |
echo "${{ secrets.GITHUB_TOKEN }}" | docker login docker.pkg.github.com -u ${{ github.actor }} --password-stdin
docker pull docker.pkg.github.com/seagate/cortx-re/cortx_centos:7.7.1908
docker run -i -v $PWD:/build docker.pkg.github.com/seagate/cortx-re/cortx_centos:7.7.1908 'sh /build/build_scripts/build_provisoiner.sh'
- name: Upload files to a GitHub release
uses: svenstaro/[email protected]
with:
repo_token: ${{ secrets.TOKEN }}
release_name: provisioner
file: provisioner/release/*.rpm
tag: 1.0.0
file_glob: true
body: "Provisioner Release"
There are some better actions available to upload artifacts to GitHub releases. We can try out them. Do review and provide suggestions.
We can use base-docker-image workflow to generate Docker Images. It takes most of the parameters as variables so can be easily scaled to build Docker images for other components.
Workflow details are as below.
steps:
- uses: actions/checkout@v2
- name: Set ENV
run: |
echo ::set-env name=INPUT_REPOSITORY::$( echo $GITHUB_REPOSITORY | tr '[:upper:]' '[:lower:]')
echo ::set-env name=IMAGE_NAME::cortx_centos
echo ::set-env name=TAG::7.7.1908
- name: Build and push Docker images
uses: docker/[email protected]
env:
DEFAULT_PASSWORD: ${{ secrets.DEFAULT_PASSWORD }}
REPO_NAME: ${{ github.event.repository.name }}
with:
# Path to the Dockerfile (Default is '{path}/Dockerfile')
dockerfile: docker/cortx_components/centos/Dockerfile
# Build Arguments for docker image build.
build_args: 'CENTOS_RELEASE=7.7.1908,USER_PASS=root:"$DEFAULT_PASSWORD"'
path: docker/cortx_components/centos/
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
registry: docker.pkg.github.com
repository: ${{ env.INPUT_REPOSITORY }}/${{ env.IMAGE_NAME }}
tags: ${{ env.TAG }}
We should below issues in Release Automation Job pipeline
If release_note_url is not provided as input then email notification should not contain Release_Note: details . Currently it adds row like
Release_Note: | EOS Build Release Notes Cortx-1.0.0-230
Build location should be tagged build location and not actual location. Currently it shows
Build_location: | http://cortx-storage.colo.seagate.com/releases/eos/github/Cortx-v1.0.0_Beta/rhel-7.7.1908/Cortx-1.0.0-230/prod/
RELEASE is getting added multiple times in RELEASE.INFO file
RELEASE: "Cortx-1.0.0-35-rc14"
RELEASE: Cortx-1.0.0-100-rc13
Metadata rpm is generated but not added in Release Build.
Refer - http://eos-jenkins.mero.colo.seagate.com/job/Release_Engineering/job/Release-automation/68/console . From console
logs Beta-230-1.x86_64.rpm is generated however it is not added into http://cortx-storage.colo.seagate.com/releases/eos/github/Cortx-v1.0.0_Beta/rhel-7.7.1908/Cortx-1.0.0-230/prod/
We should fetch system interface names dynamically instead of hardcoding them. As we observed that interface names change per OS template used.
Refer - #228 (comment)
Seagate Github Repo Branch protection rules are not up to date with DCO/ SLA and other mandatory checks. Please create Seagate standard for branch protection rule and apply it for all component repo branch's. also please document this.
Github Repo's:
Branch's:
Document:
We should be able to provision and deploy Cortx Stack from ISO file. We have generated two iso files.
Release ISO - This contains packages from all them component similar as release build.
Pre-requisite ISO - This contains required third party packages.
ISO files are placed at http://cortx-storage.colo.seagate.com/releases/cortx-iso/
Release ISO - http://cortx-storage.colo.seagate.com/releases/cortx-iso/releases/Cortx-1.0.0-Rel-2795.iso
Pre-requisite ISO - http://cortx-storage.colo.seagate.com/releases/cortx-iso/prereqs/Cortx-1.0.2-prereqs.iso
Release ISO content -
.
├── CHANGESET.txt
├── cortx-csm_agent-1.0.0-700_ebe8cb6.x86_64.rpm
├── cortx-csm_web-1.0.0-700_bf9d05f.x86_64.rpm
├── cortx-dsal-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-dsal-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-ganesha-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-fs-ganesha-test-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-ha-1.0.0-30_ba5d410.x86_64.rpm
├── cortx-hare-1.0.0-744_git02842fa.el7.x86_64.rpm
├── cortx-libsspl_sec-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-libsspl_sec-devel-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-libsspl_sec-method_none-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-libsspl_sec-method_pki-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── cortx-motr-1.0.0-430_git6d9ef31_3.10.0_1062.el7.x86_64.rpm
├── cortx-motr-devel-1.0.0-430_git6d9ef31_3.10.0_1062.el7.x86_64.rpm
├── cortx-motr-tests-ut-1.0.0-430_git6d9ef31_3.10.0_1062.el7.x86_64.rpm
├── cortx-nsal-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-nsal-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-prvsnr-1.0.0-380_git0d7a721_el7.x86_64.rpm
├── cortx-prvsnr-cli-1.0.0-380_git0d7a721_el7.x86_64.rpm
├── cortx-s3iamcli-1.0.0-1012_git138b0f4.noarch.rpm
├── cortx-s3iamcli-devel-1.0.0-1012_git138b0f4.noarch.rpm
├── cortx-s3server-1.0.0-1012_git138b0f4_el7.x86_64.rpm
├── cortx-sspl-1.0.0-520_gitf110da8.el7.noarch.rpm
├── cortx-sspl-cli-1.0.0-520_gitf110da8.el7.noarch.rpm
├── cortx-sspl-test-1.0.0-520_gitf110da8.el7.noarch.rpm
├── cortx-utils-1.0.0-148_2045f17.el7.x86_64.rpm
├── cortx-utils-devel-1.0.0-148_2045f17.el7.x86_64.rpm
├── python36-cortx-prvsnr-0.31.0-1.0.0.x86_64.rpm
├── RELEASE.INFO
├── RPM-GPG-KEY-Seagate
├── systemd-python36-1.0.0-520_gitf110da8.el7.x86_64.rpm
├── uds-pyi-1.0.0-1.r2.el7.x86_64.rpm
└── udx-discovery-0.1.2-3.el7.x86_64.rpm
0 directories, 36 files
Pre-requisite ISO content -
.
├── commons
│ ├── consul
│ │ └── consul_1.6.2
│ │ ├── consul
│ │ ├── consul_1.6.2_linux_amd64.zip
│ │ └── consul_1.6.2_SHA256SUMS
│ ├── elasticsearch
│ │ ├── elasticsearch-6.8.8.rpm
│ │ └── elasticsearch-oss-6.8.8.rpm
│ ├── ha
│ ├── kibana
│ │ ├── kibana-6.8.8-x86_64.rpm
│ │ └── kibana-oss-6.8.8-x86_64.rpm
│ ├── node
│ │ ├── node-v12.13.0-linux-x64.tar.xz
│ │ └── SHASUMS256.txt.asc
│ ├── rsyslog-8.40
│ │ ├── libestr-0.1.11-1.el7.x86_64.rpm
│ │ ├── libfastjson4-0.99.8-1.el7.centos.x86_64.rpm
│ │ ├── liblognorm5-2.0.6-1.el7.x86_64.rpm
│ │ ├── rsyslog-8.40.0-1.el7.x86_64.rpm
│ │ ├── rsyslog-elasticsearch-8.40.0-1.el7.x86_64.rpm
│ │ └── rsyslog-mmjsonparse-8.40.0-1.el7.x86_64.rpm
│ └── saltstack-2019.2.0
│ ├── m2crypto-0.31.0-3.el7.x86_64.rpm
│ ├── m2crypto-debuginfo-0.31.0-3.el7.x86_64.rpm
│ ├── python2-ioflo-1.3.8-4.el7.noarch.rpm
│ ├── python2-libcloud-2.0.0-3.el7.noarch.rpm
│ ├── python2-mock-1.0.1-12.el7.noarch.rpm
│ ├── python2-raet-0.6.6-6.el7.noarch.rpm
│ ├── python2-timelib-0.2.4-4.el7.noarch.rpm
│ ├── python2-zmq-15.3.0-6.el7.x86_64.rpm
│ ├── python2-zmq-tests-15.3.0-6.el7.x86_64.rpm
│ ├── python36-cherrypy-5.6.0-5.el7.noarch.rpm
│ ├── python36-ioflo-1.3.8-4.el7.noarch.rpm
│ ├── python36-libcloud-2.0.0-3.el7.noarch.rpm
│ ├── python36-libnacl-1.6.1-2.el7.noarch.rpm
│ ├── python36-m2crypto-0.31.0-3.el7.x86_64.rpm
│ ├── python36-mock-1.0.1-12.el7.noarch.rpm
│ ├── python36-raet-0.6.6-6.el7.noarch.rpm
│ ├── python36-timelib-0.2.4-4.el7.noarch.rpm
│ ├── python36-zmq-15.3.0-6.el7.x86_64.rpm
│ ├── python36-zmq-tests-15.3.0-6.el7.x86_64.rpm
│ ├── python-cherrypy-5.6.0-5.el7.noarch.rpm
│ ├── python-libnacl-1.6.1-2.el7.noarch.rpm
│ ├── python-zmq-debuginfo-15.3.0-6.el7.x86_64.rpm
│ ├── repodata
│ │ ├── 13ef8c6573196e779ae69d0b3618a12c65da5b4c922ccecf243bcf87ac364ba2-other.sqlite.bz2
│ │ ├── 188abf84ba145252326260190e9148696ea793d3ef6e5add2b22b5a539078ea2-other.xml.gz
│ │ ├── 78f19de94fbf1a2f20c73e4bfe44369c33bf49a55c6b4ac37a793c302a8c413b-filelists.sqlite.bz2
│ │ ├── 8256c9f089a94bf31f424655dd4096078464e09eeb11f907a41cf7fa89c0fbd5-primary.xml.gz
│ │ ├── 91416d25ec47b60e4e3dc05158a238447b91af44a937b4fda5e559153a9f504e-primary.sqlite.bz2
│ │ ├── d3a1ab9e67a811555f7dda62038573a69eed0e38b9be3ffb95e442dd390a5f03-filelists.xml.gz
│ │ └── repomd.xml
│ ├── salt-2019.2.0-2.el7.noarch.rpm
│ ├── salt-api-2019.2.0-2.el7.noarch.rpm
│ ├── salt-cloud-2019.2.0-2.el7.noarch.rpm
│ ├── salt-master-2019.2.0-2.el7.noarch.rpm
│ ├── salt-minion-2019.2.0-2.el7.noarch.rpm
│ ├── salt-ssh-2019.2.0-2.el7.noarch.rpm
│ ├── SALTSTACK-GPG-KEY.pub
│ ├── salt-syndic-2019.2.0-2.el7.noarch.rpm
│ └── SRPMS
│ ├── m2crypto-0.31.0-3.el7.src.rpm
│ ├── python-cherrypy-5.6.0-5.el7.src.rpm
│ ├── python-ioflo-1.3.8-4.el7.src.rpm
│ ├── python-libcloud-2.0.0-3.el7.src.rpm
│ ├── python-libnacl-1.6.1-2.el7.src.rpm
│ ├── python-mock-1.0.1-12.el7.src.rpm
│ ├── python-raet-0.6.6-6.el7.src.rpm
│ ├── python-timelib-0.2.4-4.el7.src.rpm
│ ├── python-zmq-15.3.0-6.el7.src.rpm
│ └── salt-2019.2.0-2.el7.src.rpm
├── csm_uploads
│ ├── eos-py-utils-1.0.0-14_0a04f73.noarch.rpm
│ ├── eos-py-utils-1.0.0-21_ec2f7ce.noarch.rpm
│ ├── eos-py-utils-1.0.0-26_56e5515.noarch.rpm
│ └── stats_utils-1.0.0-12_12726bf.x86_64.rpm
├── EPEL-7
│ ├── csnappy-0-6.20150729gitd7bc683.el7.x86_64.rpm
│ ├── erlang-appmon-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-asn1-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-common_test-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-compiler-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosEventDomain-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosEvent-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosFileTransfer-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosNotification-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosProperty-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosTime-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-cosTransactions-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-crypto-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-debugger-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-dialyzer-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-diameter-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-edoc-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-eldap-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-erl_docgen-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-erl_interface-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-erts-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-et-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-eunit-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-examples-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-gs-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-hipe-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-ic-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-inets-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-jinterface-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-kernel-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-megaco-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-mnesia-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-observer-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-odbc-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-orber-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-os_mon-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-otp_mibs-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-parsetools-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-percept-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-pman-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-public_key-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-reltool-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-runtime_tools-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-sasl-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-sd_notify-0.1-1.el7.x86_64.rpm
│ ├── erlang-snmp-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-ssh-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-ssl-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-stdlib-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-syntax_tools-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-test_server-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-toolbar-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-tools-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-tv-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-typer-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-webtool-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-wx-R16B-03.18.el7.x86_64.rpm
│ ├── erlang-xmerl-R16B-03.18.el7.x86_64.rpm
│ ├── genders-1.22-2.el7.x86_64.rpm
│ ├── gflags-2.1.1-6.el7.x86_64.rpm
│ ├── glog-0.3.3-8.el7.x86_64.rpm
│ ├── hiredis-0.12.1-1.el7.x86_64.rpm
│ ├── jq-1.6-1.el7.x86_64.rpm
│ ├── libgenders-1.22-2.el7.x86_64.rpm
│ ├── libsodium-1.0.18-1.el7.x86_64.rpm
│ ├── libuv-1.34.0-1.el7.x86_64.rpm
│ ├── miniz-1.15-4.r4.el7.x86_64.rpm
│ ├── nodejs-6.17.1-1.el7.x86_64.rpm
│ ├── npm-3.10.10-1.6.17.1.1.el7.x86_64.rpm
│ ├── oniguruma-5.9.5-3.el7.x86_64.rpm
│ ├── openpgm-5.2.122-2.el7.x86_64.rpm
│ ├── perl-common-sense-3.6-4.el7.noarch.rpm
│ ├── perl-Config-Any-0.24-1.el7.noarch.rpm
│ ├── perl-Config-General-2.61-1.el7.noarch.rpm
│ ├── perl-JSON-XS-3.01-2.el7.x86_64.rpm
│ ├── perl-MCE-1.862-1.el7.noarch.rpm
│ ├── perl-Sereal-3.015-1.el7.noarch.rpm
│ ├── perl-Sereal-Decoder-3.015-1.el7.x86_64.rpm
│ ├── perl-Sereal-Encoder-3.015-1.el7.x86_64.rpm
│ ├── perl-Types-Serialiser-1.0-1.el7.noarch.rpm
│ ├── perl-YAML-LibYAML-0.54-1.el7.x86_64.rpm
│ ├── python2-keyring-5.0-4.el7.noarch.rpm
│ ├── python2-pip-8.1.2-10.el7.noarch.rpm
│ ├── python2-qpid-proton-0.29.0-1.el7.x86_64.rpm
│ ├── python36-dateutil-2.4.2-5.el7.noarch.rpm
│ ├── python36-docutils-0.14-1.el7.noarch.rpm
│ ├── python36-jinja2-2.8.1-2.el7.noarch.rpm
│ ├── python36-m2crypto-0.35.2-5.el7.x86_64.rpm
│ ├── python36-pika-0.10.0-10.el7.noarch.rpm
│ ├── python36-ply-3.9-2.el7.noarch.rpm
│ ├── python36-psutil-2.2.1-5.el7.x86_64.rpm
│ ├── python36-pysocks-1.6.8-6.el7.noarch.rpm
│ ├── python36-PyYAML-3.12-1.el7.x86_64.rpm
│ ├── python36-requests-2.14.2-2.el7.noarch.rpm
│ ├── python36-six-1.11.0-3.el7.noarch.rpm
│ ├── python36-urllib3-1.25.1-1.el7.noarch.rpm
│ ├── qpid-proton-c-0.29.0-1.el7.x86_64.rpm
│ ├── qpid-proton-c-docs-0.29.0-1.el7.noarch.rpm
│ ├── rabbitmq-server-3.3.5-34.el7.noarch.rpm
│ ├── rubygem-net-ssh-2.9.1-1.el7.noarch.rpm
│ ├── s3cmd-2.0.2-1.el7.noarch.rpm
│ ├── statsd-0.7.2-10.el7.noarch.rpm
│ ├── wxBase-2.8.12-20.el7.x86_64.rpm
│ ├── wxGTK-2.8.12-20.el7.x86_64.rpm
│ ├── wxGTK-gl-2.8.12-20.el7.x86_64.rpm
│ └── yaml-cpp-0.5.1-2.el7.x86_64.rpm
├── lustre
│ └── custom
│ ├── o2ib
│ │ ├── kmod-lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ ├── kmod-lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.src.rpm
│ │ ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ ├── lustre-client-debuginfo-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ ├── lustre-client-devel-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ ├── lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ ├── lustre-iokit-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ │ └── repodata
│ │ ├── 2aece93708f688cecddd729693c030222c4fd46497c0ddcd34d78126a013332c-other.sqlite.bz2
│ │ ├── 7aac29b1ff0985942a940c00e7a7c7407dffad61ed70c60cc146ecae5e8c1f5e-other.xml.gz
│ │ ├── 8b8cf2981380893327a9d665265751706bf5f1e8c66a25a779ee8563014dc6a6-filelists.xml.gz
│ │ ├── 9fa844fb4c1604ec3a3c70f8b1cf716993df5708be176520e2eb654bdbef2bff-primary.sqlite.bz2
│ │ ├── d19ed9631312aa2aac7e51497a02a06cdad613b5305ed573e41f6df934a92e5b-primary.xml.gz
│ │ ├── db99ef7b255331dfde6d30998cd60eb0391fe182795c66661db0a4deb0466be6-filelists.sqlite.bz2
│ │ └── repomd.xml
│ └── tcp
│ ├── kmod-lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ ├── kmod-lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.src.rpm
│ ├── lustre-client-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ ├── lustre-client-debuginfo-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ ├── lustre-client-devel-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ ├── lustre-client-tests-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ ├── lustre-iokit-2.12.0.5_357_g0c95a37-1.el7.x86_64.rpm
│ └── repodata
│ ├── 4cb5f2fc4cc0a94bd5bdb90e1c3ade5af36790a9ba9829505dcfa80e638693a8-primary.xml.gz
│ ├── 5e0fd7e1d5025ebd2b834f12b8660459d2549b670ae298437410324e6957bf84-other.sqlite.bz2
│ ├── 75f6f7ddd1c4bb648ad43c63010ae59d975fcdb4baab72d9a0c9024e76c23d01-primary.sqlite.bz2
│ ├── 854b716d80c8b561ee8e8503ba931df84c442fa6039b06857262fe8e28012c6f-filelists.xml.gz
│ ├── c63312811e1e964e6379b5db7180bad8fefd6634d674341cbeef969002fa0e71-other.xml.gz
│ ├── cdab7080b901697c8599d7d57ed577b79e2ad03e33d8388a0696e3456b078d31-filelists.sqlite.bz2
│ └── repomd.xml
├── misc
│ └── scsi-network-relay-1.1.0-47.x86_64.rpm
├── performance
│ ├── kmod-lustre-client-2.12.3-1.el7.x86_64.rpm
│ ├── linux.mellanox.com
│ │ └── public
│ │ └── repo
│ │ └── mlnx_ofed
│ │ └── 4.7-3.2.9.0
│ │ └── rhel7.7
│ │ └── x86_64
│ │ └── MLNX_LIBS
│ │ ├── ar_mgr-1.0-0.47.MLNX20191120.gf30d03a.47329.x86_64.rpm
│ │ ├── cc_mgr-1.0-0.46.MLNX20191120.gf30d03a.47329.x86_64.rpm
│ │ ├── dapl-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│ │ ├── dapl-devel-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│ │ ├── dapl-devel-static-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│ │ ├── dapl-utils-2.1.10mlnx-OFED.3.4.2.1.0.47329.x86_64.rpm
│ │ ├── dump_pr-1.0-0.42.MLNX20191120.gf30d03a.47329.x86_64.rpm
│ │ ├── fabric-collector-1.1.0.MLNX20170103.89bb2aa-0.1.47329.x86_64.rpm
│ │ ├── hcoll-4.4.2938-1.47329.x86_64.rpm
│ │ ├── ibacm-41mlnx1-OFED.4.3.3.0.0.47329.x86_64.rpm
│ │ ├── ibacm-devel-41mlnx1-OFED.4.3.3.0.0.47329.x86_64.rpm
│ │ ├── ibdump-5.0.0-3.47329.x86_64.rpm
│ │ ├── ibsim-0.7mlnx1-0.11.g85c342b.47329.x86_64.rpm
│ │ ├── ibutils-1.5.7.1-0.12.gdcaeae2.47329.x86_64.rpm
│ │ ├── ibutils2-2.1.1-0.113.MLNX20191121.g1c29603.47329.x86_64.rpm
│ │ ├── infiniband-diags-5.4.0.MLNX20190908.5f40e4f-0.1.47329.x86_64.rpm
│ │ ├── infiniband-diags-compat-5.4.0.MLNX20190908.5f40e4f-0.1.47329.x86_64.rpm
│ │ ├── infiniband-diags-guest-5.4.0.MLNX20190908.5f40e4f-0.1.47329.x86_64.rpm
│ │ ├── kmod-iser-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── kmod-isert-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── kmod-kernel-mft-mlnx-4.13.3-1.rhel7u7.x86_64.rpm
│ │ ├── kmod-knem-1.1.3.90mlnx1-OFED.4.7.2.0.7.1.gec1f2f8.rhel7u7.x86_64.rpm
│ │ ├── kmod-mlnx-en-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│ │ ├── kmod-mlnx-nvme-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── kmod-mlnx-ofa_kernel-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── kmod-rshim-1.16-0.ga7ad4e6.rhel7u7.x86_64.rpm
│ │ ├── kmod-srp-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── knem-1.1.3.90mlnx1-OFED.4.7.2.0.7.1.gec1f2f8.rhel7u7.x86_64.rpm
│ │ ├── libdisni-1.9-OFED.4.5.0.0.6.47329.x86_64.rpm
│ │ ├── libdisni-devel-1.9-OFED.4.5.0.0.6.47329.x86_64.rpm
│ │ ├── libibcm-41mlnx1-OFED.4.1.0.1.0.47329.i686.rpm
│ │ ├── libibcm-41mlnx1-OFED.4.1.0.1.0.47329.x86_64.rpm
│ │ ├── libibcm-devel-41mlnx1-OFED.4.1.0.1.0.47329.i686.rpm
│ │ ├── libibcm-devel-41mlnx1-OFED.4.1.0.1.0.47329.x86_64.rpm
│ │ ├── libibmad-5.4.0.MLNX20190423.1d917ae-0.1.47329.i686.rpm
│ │ ├── libibmad-5.4.0.MLNX20190423.1d917ae-0.1.47329.x86_64.rpm
│ │ ├── libibmad-devel-5.4.0.MLNX20190423.1d917ae-0.1.47329.i686.rpm
│ │ ├── libibmad-devel-5.4.0.MLNX20190423.1d917ae-0.1.47329.x86_64.rpm
│ │ ├── libibmad-static-5.4.0.MLNX20190423.1d917ae-0.1.47329.i686.rpm
│ │ ├── libibmad-static-5.4.0.MLNX20190423.1d917ae-0.1.47329.x86_64.rpm
│ │ ├── libibumad-43.1.1.MLNX20190905.1080879-0.1.47329.i686.rpm
│ │ ├── libibumad-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64.rpm
│ │ ├── libibumad-devel-43.1.1.MLNX20190905.1080879-0.1.47329.i686.rpm
│ │ ├── libibumad-devel-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64.rpm
│ │ ├── libibumad-static-43.1.1.MLNX20190905.1080879-0.1.47329.i686.rpm
│ │ ├── libibumad-static-43.1.1.MLNX20190905.1080879-0.1.47329.x86_64.rpm
│ │ ├── libibverbs-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│ │ ├── libibverbs-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│ │ ├── libibverbs-devel-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│ │ ├── libibverbs-devel-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│ │ ├── libibverbs-devel-static-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│ │ ├── libibverbs-devel-static-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│ │ ├── libibverbs-utils-41mlnx1-OFED.4.7.0.0.2.47329.i686.rpm
│ │ ├── libibverbs-utils-41mlnx1-OFED.4.7.0.0.2.47329.x86_64.rpm
│ │ ├── libmlx4-41mlnx1-OFED.4.7.3.0.3.47329.i686.rpm
│ │ ├── libmlx4-41mlnx1-OFED.4.7.3.0.3.47329.x86_64.rpm
│ │ ├── libmlx4-devel-41mlnx1-OFED.4.7.3.0.3.47329.i686.rpm
│ │ ├── libmlx4-devel-41mlnx1-OFED.4.7.3.0.3.47329.x86_64.rpm
│ │ ├── libmlx5-41mlnx1-OFED.4.7.0.3.3.47329.i686.rpm
│ │ ├── libmlx5-41mlnx1-OFED.4.7.0.3.3.47329.x86_64.rpm
│ │ ├── libmlx5-devel-41mlnx1-OFED.4.7.0.3.3.47329.i686.rpm
│ │ ├── libmlx5-devel-41mlnx1-OFED.4.7.0.3.3.47329.x86_64.rpm
│ │ ├── librdmacm-41mlnx1-OFED.4.7.3.0.6.47329.i686.rpm
│ │ ├── librdmacm-41mlnx1-OFED.4.7.3.0.6.47329.x86_64.rpm
│ │ ├── librdmacm-devel-41mlnx1-OFED.4.7.3.0.6.47329.i686.rpm
│ │ ├── librdmacm-devel-41mlnx1-OFED.4.7.3.0.6.47329.x86_64.rpm
│ │ ├── librdmacm-utils-41mlnx1-OFED.4.7.3.0.6.47329.i686.rpm
│ │ ├── librdmacm-utils-41mlnx1-OFED.4.7.3.0.6.47329.x86_64.rpm
│ │ ├── librxe-41mlnx1-OFED.4.4.2.4.6.47329.i686.rpm
│ │ ├── librxe-41mlnx1-OFED.4.4.2.4.6.47329.x86_64.rpm
│ │ ├── librxe-devel-static-41mlnx1-OFED.4.4.2.4.6.47329.i686.rpm
│ │ ├── librxe-devel-static-41mlnx1-OFED.4.4.2.4.6.47329.x86_64.rpm
│ │ ├── libvma-8.9.5-1.x86_64.rpm
│ │ ├── libvma-devel-8.9.5-1.x86_64.rpm
│ │ ├── libvma-utils-8.9.5-1.x86_64.rpm
│ │ ├── mft-4.13.3-6.x86_64.rpm
│ │ ├── mlnx-en-doc-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│ │ ├── mlnx-en-sources-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│ │ ├── mlnx-en-utils-4.7-3.2.9.0.g457f064.rhel7u7.x86_64.rpm
│ │ ├── mlnx-ethtool-5.1-1.47329.x86_64.rpm
│ │ ├── mlnx-fw-updater-4.7-3.2.9.0.x86_64.rpm
│ │ ├── mlnx-iproute2-5.2.0-1.47329.x86_64.rpm
│ │ ├── mlnx-ofa_kernel-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── mlnx-ofa_kernel-devel-4.7-OFED.4.7.3.2.9.1.g457f064.rhel7u7.x86_64.rpm
│ │ ├── mlnx-ofed-all-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-all-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-basic-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-basic-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-bluefield-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-bluefield-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnxofed-docs-4.7-3.2.9.0.noarch.rpm
│ │ ├── mlnx-ofed-dpdk-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-dpdk-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-eth-only-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-guest-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-guest-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-hpc-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-hpc-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-hypervisor-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-hypervisor-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-kernel-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-vma-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-vma-eth-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-vma-eth-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-vma-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-vma-vpi-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mlnx-ofed-vma-vpi-user-only-4.7-3.2.9.0.rhel7.7.noarch.rpm
│ │ ├── mpi-selector-1.0.3-1.47329.x86_64.rpm
│ │ ├── mpitests_openmpi-3.2.20-e1a0676.47329.x86_64.rpm
│ │ ├── mstflint-4.13.0-1.41.g4e8819c.47329.x86_64.rpm
│ │ ├── mxm-3.7.3112-1.47329.x86_64.rpm
│ │ ├── neohost-backend-1.4.0-14.x86_64.rpm
│ │ ├── ofed-scripts-4.7-OFED.4.7.3.2.9.x86_64.rpm
│ │ ├── openmpi-4.0.2rc3-1.47329.x86_64.rpm
│ │ ├── opensm-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│ │ ├── opensm-devel-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│ │ ├── opensm-libs-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│ │ ├── opensm-static-5.5.1.MLNX20191120.0c8dde0-0.1.47329.x86_64.rpm
│ │ ├── perftest-4.4-0.11.gd240b65.47329.x86_64.rpm
│ │ ├── qperf-0.4.9-9.47329.x86_64.rpm
│ │ ├── repodata
│ │ ├── sharp-2.0.0.MLNX20190922.a9ebf22-1.47329.x86_64.rpm
│ │ ├── sockperf-3.6-0.git737d1e3e5576.47329.x86_64.rpm
│ │ ├── srptools-41mlnx1-5.47329.x86_64.rpm
│ │ ├── ucx-1.7.0-1.47329.x86_64.rpm
│ │ ├── ucx-cma-1.7.0-1.47329.x86_64.rpm
│ │ ├── ucx-devel-1.7.0-1.47329.x86_64.rpm
│ │ ├── ucx-ib-1.7.0-1.47329.x86_64.rpm
│ │ ├── ucx-ib-cm-1.7.0-1.47329.x86_64.rpm
│ │ ├── ucx-knem-1.7.0-1.47329.x86_64.rpm
│ │ └── ucx-rdmacm-1.7.0-1.47329.x86_64.rpm
│ └── lustre-client-2.12.3-1.el7.x86_64.rpm
├── prvsnr_uploads
│ ├── eos-install-prereqs-centos.sh
│ ├── eos-install-prereqs.sh
│ └── eos-install-prereqs-vm.sh
├── repodata
│ ├── 0d4b318f4891f3bbc5d4b44815e89e909b36d2a1bef3e55b5a1834fd647abc73-other.sqlite.bz2
│ ├── 28d5d775c6a0f993022052b1e24fc6cc7092690f33e198945dac6a1ed1da0c86-filelists.xml.gz
│ ├── 4dd28b99973b8c5089b5134aa9d166948949038d73a9f88ca4d404eaa0eab5a6-primary.sqlite.bz2
│ ├── 7c9fbba7753a97ba8485f88c011678e2645ae60bfa9f535e0275c310cda5a6c1-other.xml.gz
│ ├── babd8342b235d1964de2c3e431bf809eb5d0918f22ce158b6bd53233871e1f4d-primary.xml.gz
│ ├── f6f37d665ad6eb9a52b32bb3c667d1c7bbcf6e087076c32feed9b333a8afa412-filelists.sqlite.bz2
│ └── repomd.xml
└── s3server_uploads
├── log4cxx_eos-0.10.0-1.x86_64.rpm
├── log4cxx_eos-devel-0.10.0-1.x86_64.rpm
├── python2-jmespath-0.9.0-1.el7.noarch.rpm
├── python2-s3transfer-0.1.10-1.el7.noarch.rpm
├── python36-boto3-1.4.6-1.el7.noarch.rpm
├── python36-botocore-1.6.0-1.el7.noarch.rpm
├── python36-s3transfer-0.1.10-1.el7.noarch.rpm
├── python36-xmltodict-0.9.0-1.el7.noarch.rpm
└── s3cmd-2.0.2-1.el7.noarch.rpm
33 directories, 355 files
We are using polling method to trigger Jenkins jobs on code commit. Seems we are exceeding GitHub API limit and that's causing delay in multi-branch pipelines execution.
Please refer - HA Multi-branch job.
Logs are as below,
Started by timer
[Mon Sep 14 05:25:00 UTC 2020] Starting branch indexing...
05:25:00 Connecting to https://api.github.com using cortx-admin/****** (RE Github Access)
05:25:00 GitHub API Usage: Current quota has 2936 remaining (483 over budget). Next quota of 5000 in 50 min. Sleeping for 8 min 8 sec.
05:28:00 GitHub API Usage: Still sleeping, now only 5 min 8 sec remaining.
05:31:00 GitHub API Usage: Still sleeping, now only 2 min 8 sec remaining.
05:33:09 GitHub API Usage: Current quota has 2320 remaining (590 over budget). Next quota of 5000 in 42 min. Sleeping for 9 min 51 sec.
05:36:09 GitHub API Usage: Still sleeping, now only 6 min 51 sec remaining.
05:39:09 GitHub API Usage: Still sleeping, now only 3 min 51 sec remaining.
05:42:09 GitHub API Usage: Still sleeping, now only 51 sec remaining.
05:43:00 GitHub API Usage: Current quota has 100 remaining (2194 over budget). Next quota of 5000 in 32 min. Sleeping until reset.
Prod Jenkins logs are showing,
2020-12-08 06:13:00.982+0000 [id=54] WARNING c.c.h.p.f.c.PeriodicFolderTrigger#run: Queue refused to schedule org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject@67267635[Release_Engineering/github-work/GitHub Multibranch Jobs/nfs-multibranch]
2020-12-08 06:13:00.985+0000 [id=54] WARNING c.c.h.p.f.c.PeriodicFolderTrigger#run: Queue refused to schedule org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject@754b1c5c[Archive/Cortx-PreMerge-Test/PV-PreMerge-ST]
[12/08/20 06:13:02] SSH Launch of qa-nfs-node on 10.237.65.146 failed in 4,726 ms
[12/08/20 06:13:02] SSH Launch of qa-perf-node on 10.237.65.111 failed in 4,803 ms
2020-12-08 06:14:00.983+0000 [id=48] WARNING c.c.h.p.f.c.PeriodicFolderTrigger#run: Queue refused to schedule org.jenkinsci.plugins.workflow.multibranch.WorkflowMultiBranchProject@754b1c5c[Archive/Cortx-PreMerge-Test/PV-PreMerge-ST]
2020-12-08 06:14:22.483+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.484+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.485+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.486+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
2020-12-08 06:14:22.486+0000 [id=35] WARNING c.c.jenkins.GitHubRepositoryName#create: Could not match URL http://gitlab.mero.colo.seagate.com/eos/fs/nfs-ganesha.git
Dependabot couldn't find a .yml for this project.
Dependabot requires a .yml to evaluate your project's current Github_actions dependencies. It had expected to find one at the path: /.github/workflows/<anything>.yml
.
If this isn't a Github_actions project, or if it is a library, you may wish to disable updates for it from within Dependabot.
Due to VM infra Issue 2 of our Jenkins Node causes build failures. please find more info below
Affected Node
Dev Build Node = ssc-vm-c-0266.colo.seagate.com
Release Build Node = ssc-vm-0497.colo.seagate.com
Jenkins Build Failure Log
subscription-manager status
[root@ssc-vm-0497 ~]# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux Server
Product ID: 69
Version: 7.7
Arch: x86_64
Status: Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:
Ends:
[root@ssc-vm-c-0266 736373]# subscription-manager list
+-------------------------------------------+
Installed Product Status
+-------------------------------------------+
Product Name: Red Hat Enterprise Linux Server
Product ID: 69
Version: 7.7
Arch: x86_64
Status: Not Subscribed
Status Details: Not supported by a valid subscription.
Starts:
Ends:
We have submodules configured for all component repositories in cortx repo. All other components are configured for main branch however for cortx-prvsnr it's pre-cortx-1,0 branch.
This may be because pre-cortx-1,0 is default branch for cortx-prvsnr repository.
Git checkout takes too long ( ~15 min ) in cloudform VM.
Priority : High
Observation
Repo : cortx-csm
Timetaken : 0.23 m
Network Speed
curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Seagate Technology LLC (192.55.14.36)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Singtel (Singapore) [6.13 km]: 5.158 ms
Testing download speed................................................................................
Download: 642.32 Mbit/s
Testing upload speed......................................................................................................
Upload: 767.29 Mbit/s
Git Checkout Log
$ time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-cs
m.git
07:55:44.667551 exec-cmd.c:238 trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
07:55:44.670553 git.c:439 trace: built-in: git clone https://<PAT>@github.com/Seagate/cortx-csm.git
Cloning into 'cortx-csm'...
07:55:45.094549 run-command.c:663 trace: run_command: git remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.134548 exec-cmd.c:238 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:45.141547 git.c:704 trace: exec: git-remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.141547 run-command.c:663 trace: run_command: git-remote-https origin https://<PAT>@github.com/Seagate/cortx-csm.git
07:55:45.204547 exec-cmd.c:238 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
* Couldn't find host github.com in the _netrc file; using defaults
* Trying 52.74.223.119:443...
* TCP_NODELAY set
* Connected to github.com (52.74.223.119) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
CApath: none
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=github.com
* start date: May 5 00:00:00 2020 GMT
* expire date: May 10 12:00:00 2022 GMT
* subjectAltName: host "github.com" matched cert's "github.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert SHA2 High Assurance Server CA
* SSL certificate verify ok.
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
User-Agent: git/2.22.0.windows.1
Accept: */*
Accept-Encoding: deflate, gzip
Pragma: no-cache
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF87A:3EB5973:5F39EAA9
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x2dee370 [serially]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (52.74.223.119) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Host: github.com
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.22.0.windows.1
Accept: */*
Accept-Encoding: deflate, gzip
Pragma: no-cache
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF8E0:3EB59EF:5F39EAAA
<
* Connection #0 to host github.com left intact
07:55:47.600703 run-command.c:663 trace: run_command: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://<PAT>@github.com/Seagate/cortx-csm.git/
07:55:47.659702 exec-cmd.c:238 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:47.668701 git.c:439 trace: built-in: git fetch-pack --stateless-rpc --stdin --lock-pack --thin --check-self-contained-and-connected --cloning https://<PAT>@github.com/Seagate/cortx-csm.git/
* Couldn't find host github.com in the _netrc file; using defaults
* Found bundle for host github.com: 0x2dee370 [serially]
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (52.74.223.119) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Host: github.com
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.22.0.windows.1
Accept-Encoding: deflate, gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6224
* upload completely sent off: 6224 out of 6224 bytes
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-Frame-Options: DENY
< X-GitHub-Request-Id: 5824:63DC:2DDF9FA:3EB5AF9:5F39EAAB
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
07:55:49.360686 run-command.c:663 trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 22760 on sgspictvw2228' --check-self-contained-and-connected --pack_header=2,56013
07:55:49.435685 exec-cmd.c:238 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:49.472199 git.c:439 trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 22760 on sgspictvw2228' --check-self-contained-and-connected --pack_header=2,56013
* Connection #0 to host github.com left intactiB | 8.96 MiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722 eceiving objects: 100% (56013/56013), 40.56 MiB | 8Receiving objects: 100% (56013/56013), 41.69 MiB | 8.09 MiB/s, done.
Resolving deltas: 100% (25949/25949), done.
07:55:56.126192 run-command.c:663 trace: run_command: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
07:55:56.173192 exec-cmd.c:238 trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
07:55:56.183192 git.c:439 trace: built-in: git rev-list --objects --stdin --not --all --quiet '--progress=Checking connectivity'
Checking out files: 100% (686/686), done.
real 0m23.450s
user 0m0.000s
sys 0m0.031s
Repo : cortx-csm
Timetaken : 14.17 m
Network Speed
[root@ssc-vm-0034 736373]# curl -s https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py | python -
Retrieving speedtest.net configuration...
Testing from Seagate Technology LLC (134.204.222.36)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Nitel (Denver, CO) [48.60 km]: 35.627 ms
Testing download speed................................................................................
Download: 198.21 Mbit/s
Testing upload speed................................................................................................
Upload: 310.32 Mbit/s
Git Checkout Log
[root@4be3b7e549a9 /]# time GIT_CURL_VERBOSE=1 GIT_TRACE=1 git clone https://<PAT>@github.com/Seagate/cortx-csm.git
02:32:13.129992 git.c:371 trace: built-in: git 'clone' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
Cloning into 'cortx-csm'...
02:32:13.133696 run-command.c:369 trace: run_command: 'git-remote-https' 'origin' 'https://<PAT>@github.com/Seagate/cortx-csm.git'
* Couldn't find host github.com in the .netrc file; using defaults
* About to connect() to github.com port 443 (#0)
* Trying 140.82.114.4...
* Connected to github.com (140.82.114.4) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=github.com,O="GitHub, Inc.",L=San Francisco,ST=California,C=US
* start date: May 05 00:00:00 2020 GMT
* expire date: May 10 12:00:00 2022 GMT
* common name: github.com
* issuer: CN=DigiCert SHA2 High Assurance Server CA,OU=www.digicert.com,O=DigiCert Inc,C=US
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache
< HTTP/1.1 401 Authorization Required
< Server: GitHub Babel 2.0
< Content-Type: text/plain
< Content-Length: 21
< WWW-Authenticate: Basic realm="GitHub"
< X-GitHub-Request-Id: 2A98:2BC3:3987949:54AED2F:5F39EC2D
< X-Frame-Options: DENY
<
* Ignoring the response-body
* Connection #0 to host github.com left intact
* Issue another request to this URL: 'https://<PAT>@github.com/Seagate/cortx-csm.git/info/refs?service=git-upload-pack'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> GET /Seagate/cortx-csm.git/info/refs?service=git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept: */*
Accept-Encoding: gzip
Accept-Language: en-US, *;q=0.9
Pragma: no-cache
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:39879CA:54AED5C:5F39EC2D
< X-Frame-Options: DENY
<
* Connection #0 to host github.com left intact
02:32:15.583453 run-command.c:369 trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.584471 exec_cmd.c:118 trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
02:32:15.586117 git.c:371 trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--thin' '--check-self-contained-and-connected' '--cloning' 'https://<PAT>@github.com/Seagate/cortx-csm.git/'
* Couldn't find host github.com in the .netrc file; using defaults
* Found bundle for host github.com: 0xad4a60
* Re-using existing connection! (#0) with host github.com
* Connected to github.com (140.82.114.4) port 443 (#0)
* Server auth using Basic with user '<PAT>'
> POST /Seagate/cortx-csm.git/git-upload-pack HTTP/1.1
Authorization: Basic ZTM5ZDI0MGIyYmI1NWE3MzkxOTMxOTg1ZDAyMDFmNzQ5ODk2NzdmNTo=
User-Agent: git/2.12.2
Host: github.com
Accept-Encoding: gzip
Content-Type: application/x-git-upload-pack-request
Accept: application/x-git-upload-pack-result
Content-Encoding: gzip
Content-Length: 6214
* upload completely sent off: 6214 out of 6214 bytes
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-result
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
< X-GitHub-Request-Id: 2A98:2BC3:3987A2E:54AEE1E:5F39EC2E
< X-Frame-Options: DENY
<
remote: Enumerating objects: 291, done.
remote: Counting objects: 100% (291/291), done.
remote: Compressing objects: 100% (213/213), done.
02:32:16.435083 run-command.c:369 trace: run_command: 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.436121 exec_cmd.c:118 trace: exec: 'git' 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
02:32:16.437700 git.c:371 trace: built-in: git 'index-pack' '--stdin' '-v' '--fix-thin' '--keep=fetch-pack 53 on 4be3b7e549a9' '--check-self-contained-and-connected' '--pack_header=2,56013'
* Connection #0 to host github.com left intactiB | 74.00 KiB/s
remote: Total 56013 (delta 132), reused 143 (delta 72), pack-reused 55722
Receiving objects: 100% (56013/56013), 41.69 MiB | 68.00 KiB/s, done.
Resolving deltas: 100% (25949/25949), done.
02:46:30.154135 run-command.c:369 trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.154769 exec_cmd.c:118 trace: exec: 'git' 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
02:46:30.156481 git.c:371 trace: built-in: git 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet' '--progress=Checking connectivity'
real 14m17.110s
user 0m3.559s
sys 0m1.760s
We have builds from Jul at http://cortx-storage.colo.seagate.com/releases/cortx/github/dev/rhel-7.7.1908/ . We should remove old Dev builds because its consuming lots of space on server.
Beta build at http://eos-jenkins.colo.seagate.com/job/GitHub-Cortx-v1.0.0_Beta-release-jobs/job/s3-release-pipeline-centos7.7.1908/146/console failing with below error,
00:01:56.416
00:01:56.416 Transaction check error:
00:01:56.416 file /usr/lib64/liblog4cxx.a from install of log4cxx_eos-devel-0.10.0-1.x86_64 conflicts with file from package log4cxx_cortx-devel-0.10.0-1.x86_64
00:01:56.416
00:01:56.416 Error Summary
00:01:56.416 -------------
00:01:56.416
Currently Codacy Ranking is B for cortx-re repo. Most of issues are reported for code style . We should pick and keep fixing issues gradually from Code Style issue list
We should enable user to build CORTX on CentOS 7.9.2003 and test deployment.
Reference - https://cortxcommunity.slack.com/archives/C014WFC5KEF/p1613150930026100
Executed http://eos-jenkins.colo.seagate.com/job/Release_Engineering/job/Cortx-Automation/job/cortx-build-cleanup/ for release
environment. It has removed some tagged releases also from /mnt/bigstorage/releases/cortx/github
To migrate CI workflow from Jenkins to GitHub actions we should discuss and finalize our approach on below tasks. Please add if anything is missed. I have added possible solutions for few tasks. Please review and add your suggestions.
Motr, S3 and Hare pre-merge jobs -
Motr pre-merge job runs on hardware. S3, Hare pre-merge jobs are executed on Cent-OS/RedHat virtual machines. Currently GitHub only supports Ubuntu from Linux distribution family https://docs.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners
Possible solution
Integration Build Generation and location -
Cortx need to be deployed as full stack for all the features. Currently we build all components individually and generate integrated release build. Release build contains rpm's from all the components. We should define strategy to build component level rpm's and generate release build from them. We should also finalize location (GitHub Release nomenclature) to place them.
Possible solution
We can create GitHub actions in every component repository and place artifact's (generated rpm's) at component repository. Component rpm's will be placed in component level GitHub releases. Release engineering repository will have GitHub action to collect all components rpm's, perform other steps (rpm-signing, dependency validation etc.) and place release build at Release Engineering repository.
Branch wise Release build generation and location -
We are generating branch wise release builds and placing them in different directories of ci-storage. We need to plan for Release build structure to support branch wise releases.
Possible solution
Finalize Release name syntax and adopt it in GitHub releases. So that branch wise release's can be maintained.
Third party packages -
We are using couple of third party packages which are not built by Seagate team. We are storing them in ci-storage uploads directory. We will need to store them in GitHub release.
Possible solution
Create release named uploads and use it in GitHub actions to download and install third party packages.
Generic workflow to build component Docker images and rpm's
We should develop and use standard workflow's to build component Docker images and rpm's. This will help in debugging issues faster and support multiple OS and Kernel versions.
Created #39 and #27 for workflow related discussion. Please add your suggestions/reviews there.
Automated notifications, validation and checks -
Currently Jenkins has configured many automated notifications e.g. Email notification, Add release build details as Jira comment,. We have also configured couple of validation checks to avoid deployment failures e.g. RPM validation and dependency check. Motr CI customization etc. We will need to plan them in GitHub Actions.
CC: @amolkongre @johnbent @mukul-seagate11 @gauravchaudhari02 @pujamudaliar @venkuppu-chn @balajiramachandran-seagate
IPv4/IPv6 manipulation library
Library home page: https://files.pythonhosted.org/packages/c2/f8/49697181b1651d8347d24c095ce46c7346c37335ddc7d255833e7cde674d/ipaddress-1.0.23-py2.py3-none-any.whl
Path to dependency file: cortx-re/scripts/release_support/ova-release-notes/requirements.txt
Path to vulnerable library: cortx-re/scripts/release_support/ova-release-notes/requirements.txt
Dependency Hierarchy:
Found in HEAD commit: 07ac8101f14a36d963102c94ebaea9129a7e3c55
Lib/ipaddress.py in Python through 3.8.3 improperly computes hash values in the IPv4Interface and IPv6Interface classes, which might allow a remote attacker to cause a denial of service if an application is affected by the performance of a dictionary containing IPv4Interface or IPv6Interface objects, and this attacker can cause many dictionary entries to be created. This is fixed in: v3.5.10, v3.5.10rc1; v3.6.12; v3.7.9; v3.8.4, v3.8.4rc1, v3.8.5, v3.8.6, v3.8.6rc1; v3.9.0, v3.9.0b4, v3.9.0b5, v3.9.0rc1, v3.9.0rc2.
Publish Date: 2020-06-18
URL: CVE-2020-14422
Base Score Metrics:
Type: Upgrade version
Origin: https://security-tracker.debian.org/tracker/CVE-2020-14422
Release Date: 2020-06-18
Fix Resolution: 3.5.3-1+deb9u2, 3.7.3-2+deb10u2, 3.8.4~rc1-1
Today we faced Disk space issue on Prod Jenkins. Jenkins log reported error as,
2020-10-15 08:41:07.188+0000 [id=50] WARNING hudson.model.Queue#save: Failed to write out the queue file /var/jenkins_home/queue.xml
java.io.IOException: Disk quota exceeded
Jenkins Home is mounted over nfs at /mnt/data1/releases/jenkins/jenkins-prod
We found that there is enough disk space available on cortx-storage server.
[root@ssc-vm-0125 jenkins_workspace]# df -Th | grep cortx-storage
cortx-storage.colo.seagate.com:/mnt/data1 nfs4 4.9T 3.1T 1.9T 62% /mnt/data1
[root@ssc-vm-0125 jenkins_workspace]#
While debugging this issue along with @gowthamchinna found that jenkins_home directory is consuming 100Gb space.
[root@ssc-vm-0279 jenkins]# du -sh *
32G jenkins-dev
100G jenkins-prod
We are placing all our component artifacts at nfs mount and jenkins_home should be containing only configuration files. It should not consume this much space. On investigation found that space is consumed to Jenkins job log files. I am cleaning up old log files.
To avoid this in future for every Jenkins job we should enable log rotation,
Can we share the scripts and documentations to allow community members to build their own OVA images from the source code? If so, this would help the community a LOT because developers could have an easy way to test their changes.
Due to Github intermittent failure, Release Job failed but component job passed with bad rpm url. this causes
Bad Build (No GPG key) : http://cortx-storage.colo.seagate.com/releases/eos/github/release/rhel-7.7.1908/2773/dev/
To maintain clean build in release branch we need to ensure build quality before new changes merged into release branch and this can be achieved by introducing CT (Continuous Testing) in the release branch pre merge pipeline (Multibranch pipeline)
Existing Multibranch Workflow :
Disadvantage
Suggested Workflow
Perform VM deployment on each pull request targeted to release branch. later sanity test and other functionality related tests can be added to this.
Configure GitHub Pull request status check (required) for the above. This allows merges can be performed only when VM deployment is success.
Hello,
Is it possible to use github releases (as @mukul-seagate11 suggested it maybe be) to host VM images for the opensource community? For Seagate-only people, here is an OVA image that we would like to test whether it can be hosted by github: LINK.
Thanks!
The release VM (OVA) that we have currently is awesome for quick deployment and testing.
However we also need a release VM (OVA) that is setup for development. Such a VM should come bundled with all the build dependencies.
Anything else @andriytk ?
We should reduce Component Docker images sizes particularly for RHEL builds. Observed that docker images size for S3Server is 8.18 GB . This seems to be causing increased build time for S3Server. For RHEL builds, S3Server takes around 45 minutes which was around 20 mins in Cent-OS.
[root@ssc-vm-0497 ~]# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
rhel_csm 7.7.1908 d47dd4dc784f 3 days ago 3.27 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_csm 7.7.1908 d47dd4dc784f 3 days ago 3.27 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_hare 7.7.1908 48c93d80b1d1 6 weeks ago 7.83 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_s3 7.7.1908 cc3a485e6d7a 7 weeks ago 8.18 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_nfs 7.7.1908 0f709a90d6ae 7 weeks ago 8.01 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_mero 7.7.1908 e8d510ef2f5b 7 weeks ago 6.62 GB
ssc-vm-0279.colo.seagate.com:5000/rhel_eos 7.7.1908 ba9f80c644b5 7 weeks ago 2.03 GB
registry.access.redhat.com/rhel7-init 7.7 fe040a67fd36 4 months ago 205 MB
GitHub packages showing Docker icon greyed out and mouseover popup says 'The service is deprecated'
Would it make sense to have a Daily OVA release?
There are many unused and disconnected Jenkins Agents in Production Jenkins. We need to remove them to free up infra resources. In order to do that we need the following info for each node
We should create a Jenkins job to list down the above info for offline nodes.
Refernce -
Observed that FAILED STAGE is not set correctly in email sent for failure notification. For Motr jobs failure in Build#414 , email notification shows FAILED STAGE as 'build Hare'
However from console logs build is failed in building S3server phase.
For release build we are creating metarpm which will add all rpms mentioned in RELEASE.INFO file in cortx.spec file and after executing rpmbuild command on cortx.spec file we are getting one metarpm generated.
Refer cortx.spec file at -> https://github.com/Seagate/cortx-re/blob/metarpm-script/scripts/cortx_metarpm/cortx.spec
This cortx.spec file is generating rpm with release type name for example: beta,ga etc.
As per comments mentioned on https://jts.seagate.com/browse/EOS-8766 we need to change rpm name and also there is echo command in %build section of cortx.spec file which is expected to be removed.
But after removing %build and %install section, only source rpm is created and not rpm in /rpmbuild/RPMS folder.
Priyank reported a issue that starting from build 519, cortx-1.0.0-XXX-single.iso size increased from 1.7G to 3.2G.
We need to perform below 2 things to address the issue
Need to update Deployment Guide, Known Issues links and Product name in README.txt
Email address in GPG key need to be updated.
Luster version is empty in RELEASE.info file
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.