Coder Social home page Coder Social logo

pulp / pulp_installer Goto Github PK

View Code? Open in Web Editor NEW
53.0 17.0 81.0 3.16 MB

Ansible roles to install & configure Pulp 3 from PyPI

Home Page: https://docs.pulpproject.org/pulp_installer/

License: GNU General Public License v2.0

Ruby 4.45% Shell 7.23% Python 14.71% Makefile 5.63% Dockerfile 1.68% Jinja 66.23% Vim Script 0.07%

pulp_installer's Introduction

DEPRECATED! This repo is deprecated. You can read about why, and the migration path, here.

Pulp CI

Pulp 3 Ansible Installer

A collection of roles to install or upgrade Pulp 3. Can also reconfigure or add plugins to an existing install.

See https://docs.pulpproject.org/pulp_installer for full documentation.

Get Help

Documentation: https://docs.pulpproject.org/pulp_installer/

Issue Tracker: https://github.com/pulp/pulp_installer/issues

Forum: https://discourse.pulpproject.org/.

Join pulp on Matrix for User support

Join #pulp-dev on Matrix for Developer discussion.

pulp_installer's People

Contributors

abraverm avatar asmacdo avatar bmbouter avatar codeheeler avatar daviddavis avatar dkliban avatar dralley avatar ehelms avatar evgeni avatar fao89 avatar git-hyagi avatar goosemania avatar ichimonji10 avatar ipanova avatar jlsherrill avatar lubosmj avatar lyrch avatar mdellweg avatar mikedep333 avatar newswangerd avatar pavelpicka avatar pieterlexis avatar pulpbot avatar quba42 avatar rochacbruno avatar simaishi avatar spredzy avatar wbclark avatar werwty avatar ziegenberg avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

pulp_installer's Issues

CI is failing for ansible core main branch

During the "prepare" molecule phase for the test "core (git+https://github.com/ansible/ansible.git)":

  TASK [Gathering Facts] *********************************************************
  fatal: [debian-11]: FAILED! => {
      "ansible_facts": {},
      "changed": false,
      "failed_modules": {
          "ansible.legacy.setup": {
              "failed": true,
              "module_stderr": "Error: No such container: inventory_hostname\n",
              "module_stdout": "",
              "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
              "rc": 1
          }
      }
  }
  
  MSG:
  
  The following modules failed to execute: ansible.legacy.setup

(This occurs for all the distros/containers.

pulp_installer fails to install rubygems on Debian 11 using Ansible main branch

From the CI test core (git+https://github.com/ansible/ansible.git)

  TASK [pulp.pulp_installer.pulp_devel : Install several useful packages (distro-agnostic)] ***
  FAILED - RETRYING: [debian-11]: Install several useful packages (distro-agnostic) (5 retries left).
  FAILED - RETRYING: [debian-11]: Install several useful packages (distro-agnostic) (4 retries left).
  changed: [centos-7]
  FAILED - RETRYING: [debian-11]: Install several useful packages (distro-agnostic) (3 retries left).
  FAILED - RETRYING: [debian-11]: Install several useful packages (distro-agnostic) (2 retries left).
  FAILED - RETRYING: [debian-11]: Install several useful packages (distro-agnostic) (1 retries left).
  fatal: [debian-11]: FAILED! => {
      "attempts": 5,
      "cache_update_time": 1642609638,
      "cache_updated": false,
      "changed": false,
      "rc": 100
  }
  
  STDOUT:
  
  Reading package lists...
  Building dependency tree...
  Reading state information...
  
  
  
  STDERR:
  
  E: Version '1:2.7+2' for 'rubygems' was not found
  
  
  
  MSG:
  
  '/usr/bin/apt-get -y -o "Dpkg::Options::=--force-confdef" -o "Dpkg::Options::=--force-confold"       install 'dstat=0.7.4-6.1' 'htop=3.0.5-7' 'iotop=0.6-24-g733f3f8-1.1' 'jq=1.6-2.1' 'ncdu=1.15.1-1' 'tmux=3.1c-1+deb11u1' 'tree=1.8.0-1+b1' 'wget=1.21-1+deb11u1' 'curl=7.74.0-1.3+deb11u1' 'rubygems=1:2.7+2' 'npm=7.5.2+ds-2' 'rsyslog=8.2102.0-2'' failed: E: Version '1:2.7+2' for 'rubygems' was not found

pulp_installer 3.17+ is unable to install pulp 3.15+ from katello's rpm repos.

Version
Please provide the versions of the pulp_installer, pulpcore and plugin packages in use.

pulp_installer 3.17.2
pulpcore 3.17.3
pulp-rpm 3.17.3

Describe the bug
A clear and concise description of what the bug is.

pulp-rpm rpm package fails to install, due to the exclusion of libcomps from this commit: bc30ba9

To Reproduce
Steps to reproduce the behavior:
I was working to debug pulpcore rpm pipelines, so steps I took to create the issue:

git clone https://github.com/theforeman/forklift/
cd forklift
ansible-playbook pipelines/pulpcore.yml -e pipeline_version=3.17 -e pipeline_os=centos7 -e pipeline_type=pulpcore -e foreman_expected_version=

I would imagine, given the specific issue, any "pulp-installer" install of pulp-rpm will exhibit the same failure, due to the exclusion of the libcomps build from install.

Expected behavior
A clear and concise description of what you expected to happen.

A successful clean install of pulp-rpm from rpm packages.

Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.

I am not fully aware of the whole context, as it appears there is an issue with libcomps not providing a python2 library, and the commit referenced above attempts to fix that. I am not aware how important the referenced python2 library is to any other parts of the system.

If a missing libcomps python2 library is not an issue for new installs, I would think a more robust solution for pulp_installer would be the way to go. If the missing library is an issue for new installs, I would imagine a more robust fix in pulpcore-packaging would be needed instead..

pulp installer is trying to create /var/lib/pulp

This is a copy of an issue filed https://pulp.plan.io/issues/9548

On fresh debian 11 install I get this error

RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static content] ************************************************************
[WARNING]: Using world-readable permissions for temporary files Ansible needs to create when becoming an unprivileged user. This may
be insecure. For information on securing this, see https://docs.ansible.com/ansible/user_guide/become.html#risks-of-becoming-an-
unprivileged-user
fatal: [repo-mirror.wasd.tv]: FAILED! => {"changed": true, "cmd": ["/opt/pulp/lib/bin/pulpcore-manager", "collectstatic", "--clear", "--noinput", "--link"], "delta": "0:00:02.437759", "end": "2021-11-02 14:30:43.872728", "msg": "non-zero return code", "rc": 1, "start": "2021-11-02 14:30:41.434969", "stderr": "Traceback (most recent call last):
  File \"/opt/pulp/lib/bin/pulpcore-manager\", line 8, in <module>
    sys.exit(manage())
  File \"/opt/pulp/lib/lib/python3.9/site-packages/pulpcore/app/manage.py\", line 11, in manage
    execute_from_command_line(sys.argv)
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/__init__.py\", line 419, in execute_from_command_line
    utility.execute()
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/__init__.py\", line 413, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/base.py\", line 354, in run_from_argv
    self.execute(*args, **cmd_options)  
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute
    output = self.handle(*args, **options)
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py\", line 187, in handle
    collected = self.collect()
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py\", line 114, in collect
    handler(path, prefixed_path, storage)
  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py\", line 312, in link_file
    os.makedirs(os.path.dirname(full_path), exist_ok=True)
  File \"/usr/lib/python3.9/os.py\", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File \"/usr/lib/python3.9/os.py\", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File \"/usr/lib/python3.9/os.py\", line 215, in makedirs
    makedirs(head, exist_ok=exist_ok)
  File \"/usr/lib/python3.9/os.py\", line 225, in makedirs
    mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/var/lib/pulp'", "stderr_lines": ["Traceback (most recent call last):", "  File \"/opt/pulp/lib/bin/pulpcore-manager\", line 8, in <module>", "    sys.exit(manage())", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/pulpcore/app/manage.py\", line 11, in manage", "    execute_from_command_line(sys.argv)", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/__init__.py\", line 419, in execute_from_command_line", "    utility.execute()", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/__init__.py\", line 413, in execute", "    self.fetch_command(subcommand).run_from_argv(self.argv)", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/base.py\", line 354, in run_from_argv", "    self.execute(*args, **cmd_options)", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/core/management/base.py\", line 398, in execute", "    output = self.handle(*args, **options)", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py\", line 187, in handle", "    collected = self.collect()", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py\", line 114, in collect", "    handler(path, prefixed_path, storage)", "  File \"/opt/pulp/lib/lib/python3.9/site-packages/django/contrib/staticfiles/management/commands/collectstatic.py\", line 312, in link_file", "    os.makedirs(os.path.dirname(full_path), exist_ok=True)", "  File \"/usr/lib/python3.9/os.py\", line 215, in makedirs", "    makedirs(head, exist_ok=exist_ok)", "  File \"/usr/lib/python3.9/os.py\", line 215, in makedirs", "    makedirs(head, exist_ok=exist_ok)", "  File \"/usr/lib/python3.9/os.py\", line 215, in makedirs", "    makedirs(head, exist_ok=exist_ok)", "  File \"/usr/lib/python3.9/os.py\", line 225, in makedirs", "    mkdir(name, mode)", "PermissionError: [Errno 13] Permission denied: '/var/lib/pulp'"], "stdout": "", "stdout_lines": []}

my playbook

- name: pulp
  hosts:
    - "pulp"
  roles:
    - pulp.pulp_installer.pulp_all_services
  vars:
    ansible_shell_allow_world_readable_temp: true
    ansible_python_interpreter: /usr/bin/python3
    pulp_media_root: /opt/pulp/media
    pulp_cache_dir: /opt/pulp/cache
    pulp_user_home: /opt/pulp/home
    pulp_install_dir: /opt/pulp/lib
    pulp_config_dir: /opt/pulp/etc
    pulp_settings:
      secret_key: "secret"
      content_origin: https://repo-mirror
    pulp_default_admin_password: "xxx"
    pulp_install_plugins:
      # galaxy-ng: {}
      # pulp-2to3-migration: {}
      pulp-ansible: {}
      # pulp-certguard: {}
      pulp-container: {}
      # pulp-cookbook: {}
      pulp-deb: {}
      pulp-file: {}
      # pulp-gem: {}
      # pulp-maven: {}
      pulp-npm:
        source_dir: /tmp/pulp-src/pulp-npm
        git_url: https://github.com/pulp/pulp_npm
        git_revision: b91ad5e7972a02321b90337dc04f4a7afc185849
      pulp-python: {}
      pulp-rpm: {}
    ### webserver
    pulp_webserver_server: nginx
    pulp_configure_firewall: none
    pulp_webserver_https_port: 4444
    pulp_client_max_body_size: 1024M
    pulp_webserver_tls_files_remote: true
    pulp_webserver_tls_cert: /etc/ssl/crt
    pulp_webserver_tls_key: /etc/ssl/key
    pulp_webserver_httpd_servername: repo-mirror
  environment:
    DJANGO_SETTINGS_MODULE: pulpcore.app.settings

refactor pulp_database for better performance

If you run profile_tasks at the end of molecule (release-static):

  ===============================================================================
  pulp_common ----------------------------------------------------------- 425.33s
  geerlingguy.postgresql ------------------------------------------------ 133.23s
  pulp_webserver --------------------------------------------------------- 79.17s
  pulp_workers ----------------------------------------------------------- 64.30s
  pulp_repos ------------------------------------------------------------- 60.44s
  pulp_database ---------------------------------------------------------- 59.78s
  pulp_database_config --------------------------------------------------- 57.27s
  pulp_rpm_prerequisites ------------------------------------------------- 52.33s
  pulp_content ----------------------------------------------------------- 44.89s
  pulp_api --------------------------------------------------------------- 43.84s
  pulp_health_check ------------------------------------------------------ 28.78s
  pulp_redis ------------------------------------------------------------- 17.10s
  gather_facts ------------------------------------------------------------ 4.79s
  stat -------------------------------------------------------------------- 1.01s
  assert ------------------------------------------------------------------ 0.44s
  lineinfile -------------------------------------------------------------- 0.08s
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  total ---------------------------------------------------------------- 1072.78s

The decently long time to run geerlingguy.postgresql and pulp_database can be partially explained by an unnecessary task (the separate task to "Install scl-utils") and the fact that the following tasks are duplicated with geerlingguy.postgresql:

  • Check if PostgreSQL database is initialized.
  • Ensure PostgreSQL database is initialized.

Revert pip 22.0 workaround for pip-tools

Now that pip-tools 6.5.0 is out with compatibility with pip 22.0, we no longer need the workaround for #858.

In fact, the workaround will become a potential problem once a Linux distro's bundled pip gets upgraded to pip 22.0, and we'd have to downgrade it in the venv (which may or may not be an issue.)

Add a `ptest` to the developer role

As a developer I am always trying to run the tests, but I always have to go look in the CI to remember the commands to do it.

For the functional tests it's usually something like:

    pytest -v -r sx --color=yes --suppress-no-test-exit-code --pyargs pulp_ansible.tests.functional -m parallel -n 8
    pytest -v -r sx --color=yes --pyargs pulp_ansible.tests.functional -m "not parallel"

It would be great to have a ptest pulp_ansible that produces these ^ commands. Later we can also add the unit tests, but the way those are getting invoked is changing so let's just do the functional tests for now.

Building the collection fails with an install command error on MacOS

As reported by @newswangerd:

TASK [Building pulp.pulp_installer] ********************************************
changed: [hub -> localhost] => (item=make vendor)
failed: [hub -> localhost] (item=make dist) => {"ansible_loop_var": "item", "changed": true, "cmd": ["make", "dist"], "delta": "0:00:01.276042", "end": "2022-01-12 12:15:18.946259", "item": "make dist", "msg": "non-zero return code", "rc": 2, "start": "2022-01-12 12:15:17.670217", "stderr": "install: illegal option -- D\nusage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]\n               [-o owner] file1 file2\n       install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]\n               [-o owner] file1 ... fileN directory\n       install -d [-v] [-g group] [-m mode] [-o owner] directory ...\nmake: *** [build/src/galaxy.yml] Error 64", "stderr_lines": ["install: illegal option -- D", "usage: install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]", "               [-o owner] file1 file2", "       install [-bCcpSsv] [-B suffix] [-f flags] [-g group] [-m mode]", "               [-o owner] file1 ... fileN directory", "       install -d [-v] [-g group] [-m mode] [-o owner] directory ...", "make: *** [build/src/galaxy.yml] Error 64"], "stdout": "install -m 644 -DT galaxy.yml build/src/galaxy.yml", "stdout_lines": ["install -m 644 -DT galaxy.yml build/src/galaxy.yml"]}

The error appears to be that MacOS install doesn't support install -D, whereas GNU install does.

improvements and expectation management pulpcon21

Following up on the discussion Pulp Installer - Community Perspective & Discussion Nov 10th we decided to have a ticket here to collect on the findings.

Basically I challenged the poor installer team that pulp_installer, by it's nature as ansible collection, will always be orchestration and that it could never satisfy a users expectation of an installer package. As the discussion progressed it became clear that not all was lost and that once it was sorted what expectations need to be explained away and what gaps can be bridged, pulp_installer could be in a better place and know better on what to focus.

pulp_installer takes idempotency in it's hands

  • the good:
  • You can put differnts parts on different systems and pulp_installer can manage upgrades and the consistency of the configuration.
  • the bad:
    • typical operation system methods don't apply:
    • Your pakage manager won't do security updates or tell You what version of pulp You have
    • Your system won't know when maintanance tasks restart services.
    • Your configuration changes currently will be reset, next pulp_installer run. This can also affect redis, apache, nginx and postgres.
  • what could be improved:
    • "services status playbooks" : used withe the same vars, they produce host/service-> status in short runs. Then the admin understands the service landscape.
    • "config backup --extra-var": produce tar file host/dir/[before/now]/conffile. Now the admin can track the config and is able to diff/change and restart services to revert.

what could be improved - section is a first suggestion.

calling pip_package_info after roles fails on Ansible 2.9 / Python 2.7

This fix for #831 introduced a bizarre regression

In the release-upgrade test's converge.yml, we call the pip_package_info module in a task after calling the roles. pip_package_info now fails to load on Ansible 2.9 / Python 2.7:

    TASK [Obtain list of packages & versions in the venv after install] ************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ImportError: Error while finding module specification for 'ansible.modules.packaging.language.pip_package_info' (ModuleNotFoundError: No module named 'ansible.modules')
fatal: [fedora-33]: FAILED! => {
    "changed": false, 
    "rc": 1
}

MSG:

MODULE FAILURE
See stdout/stderr for the exact error


MODULE_STDERR:

Traceback (most recent call last):
  File "/usr/lib64/python3.9/runpy.py", line 130, in _get_module_details
    spec = importlib.util.find_spec(mod_name)
  File "/usr/lib64/python3.9/importlib/util.py", line 94, in find_spec
    parent = __import__(parent_name, fromlist=['__path__'])
ModuleNotFoundError: No module named 'ansible.modules'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 102, in <module>
  File "<stdin>", line 94, in _ansiballz_main
  File "<stdin>", line 40, in invoke_module
  File "/usr/lib64/python3.9/runpy.py", line 206, in run_module
    mod_name, mod_spec, code = _get_module_details(mod_name)
  File "/usr/lib64/python3.9/runpy.py", line 139, in _get_module_details
    raise error(msg.format(mod_name, type(ex).__name__, ex)) from ex
ImportError: Error while finding module specification for 'ansible.modules.packaging.language.pip_package_info' (ModuleNotFoundError: No module named 'ansible.modules')

This error makes little sense. ansible.modules cannot be found, but it is presumably part of the python packages that ansible transfers over to the managed node in order to run tasks on it.

become_user: root creates incompatibilities in some environments

In some environments, doing sudo su - is not allowed. Wheres, its preferrable allowed to run sudo CMD. Whenever combining the become_user: root and become: true this creates a problem.

For example,. if the sudoers file allows "/bin/sh -c *", it is sufficient for "become: true", but fails for tasks which are "become_user: root".

pulp_installer should flush handlers on certain tasks that fail

For some tasks, we should flush handlers even if the task fails.

For example, the SELinux policy task task that copies the policy files to /usr, but triggers a handler to actually install them via a command.

This change will leave the system in a more correct state.

The change will also ensure that if a user runs the installer a 2nd time, it will perform the job of the handler once, rather than never. (And thus never actually installing the policy.)

See "flush_handlers" in this docs page:
https://docs.ansible.com/ansible/latest/user_guide/playbooks_blocks.html

CentOS 8 is EOL

We should officially drop support for it, and use CentOS 8 Stream instead in our CI tests.

However, CentOS 8.5 will probably continue to work unofficially in the meantime (if users adjust their repo URLs manually to use vault.centos.org .)

pulp_installer fails on "Run pip-compile to check pulpcore/plugin compatibility" with pip 22.0

pip 22.0 is out, and it is breaking pip-tools 6.4.0.

This is breaking new installs. In CI, only F35 is unaffected because it has new enough pip to satisfy our check.

    TASK [pulp.pulp_installer.pulp_common : Run pip-compile to check pulpcore/plugin compatibility] ***
Warning: 35m[WARNING]: conditional statements should not include jinja2 templating
delimiters such as {{ }} or {% %}. Found: {{ failed_condition |
default("compatibility.rc != 0") }}
fatal: [debian-11]: FAILED! => {
    "changed": false, 
    "cmd": [
        "/usr/local/lib/pulp/bin/pip-compile"
    ], 
    "delta": "0:00:00.461577", 
    "end": "2022-01-31 18:07:23.270010", 
    "failed_when_result": true, 
    "rc": 1, 
    "start": "2022-01-31 18:07:22.808433"
}

STDERR:

Traceback (most recent call last):
  File "/usr/local/lib/pulp/bin/pip-compile", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/click/decorators.py", line 26, in new_func
    return f(get_current_context(), *args, **kwargs)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/piptools/scripts/compile.py", line 342, in cli
    repository = PyPIRepository(pip_args, cache_dir=cache_dir)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 106, in __init__
    self._setup_logging()
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging
    assert isinstance(handler, logging.StreamHandler)
AssertionError


MSG:

non-zero return code

Sporadic failure of `Set state of pulpcore app`

This task (which ensures the pulp service is started and enabled) is sporadically ("once in a blue moon") failing:
pulp_common: Set state of pulpcore app

TASK [pulp.pulp_installer.pulp_common : Set state of pulpcore app] *************
fatal: [debian-11]: FAILED! => {
    "changed": false
}

MSG:

Unable to start service pulpcore.service: Job for pulpcore.service failed because the control process exited with error code.
See "systemctl status pulpcore.service" and "journalctl -xe" for details.

changed: [centos-8]
changed: [fedora-35]
changed: [centos-7]

This message states that systemd detected the service started but then stopped with an error code prematurely. It may have been started and failed during this task, or possibly during the restart handler task before it. (Not 100% sure if this is how the systemd/ansible integration works.)

https://github.com/pulp/pulp_installer/runs/5040349479?check_suite_focus=true#step:7:1913

Upgrades fail on `pulp_common : Collect static content` due to `ModuleNotFoundError: No module named 'packaging'`

Some of our release-upgrade tests (debian-11) and all of our source-upgrade tests are failing due to the error below.

It appears that recent versions of "django-lifecycle" introduced the dependency on "packaging", but do not declare it.
rsinger86/django-lifecycle#101

RUNNING HANDLER [pulp_common : Collect static content] *************************
changed: [centos-7]
fatal: [debian-11]: FAILED! => {
    "changed": true,
    "cmd": [
        "/usr/local/lib/pulp/bin/pulpcore-manager",
        "collectstatic",
        "--clear",
        "--noinput",
        "--link"
    ],
    "delta": "0:00:00.906203",
    "end": "2022-02-14 21:07:25.851834",
    "rc": 1,
    "start": "2022-02-14 21:07:24.945631"
}

STDERR:

Traceback (most recent call last):
  File "/usr/local/lib/pulp/bin/pulpcore-manager", line 8, in <module>
    sys.exit(manage())
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/app/manage.py", line 11, in manage
    execute_from_command_line(sys.argv)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
    utility.execute()
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django/core/management/__init__.py", line 395, in execute
    django.setup()
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django/apps/config.py", line 301, in import_models
    self.models_module = import_module(models_module_name)
  File "/usr/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/app/models/__init__.py", line 4, in <module>
    from .base import (  # noqa
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/pulpcore/app/models/base.py", line 9, in <module>
    from django_lifecycle import LifecycleModel
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django_lifecycle/__init__.py", line 1, in <module>
    from .django_info import IS_GTE_1_POINT_9
  File "/usr/local/lib/pulp/lib/python3.9/site-packages/django_lifecycle/django_info.py", line 1, in <module>
    from packaging.version import Version
ModuleNotFoundError: No module named 'packaging'


MSG:

non-zero return code

Task `pulp.pulp_installer.pulp_common : Restart pulpcore service` sporadically fails

This task (which ensures the pulp service is started and enabled) is sporadically ("once in a blue moon") failing:
pulp_common: Set state of pulpcore app

RUNNING HANDLER [pulp.pulp_installer.pulp_common : Restart pulpcore service] ***
changed: [centos-7]
fatal: [centos-8]: FAILED! => {
    "changed": false
}

MSG:

Unable to restart service pulpcore.service: Job for pulpcore.service failed because the control process exited with error code.
See "systemctl status pulpcore.service" and "journalctl -xe" for details.

changed: [debian-11]
changed: [fedora-35]

This message states that systemd detected the service started but then stopped with an error code prematurely. It may not have finished starting up yet in the previous task (Set states of new pulp workers).

https://github.com/pulp/pulp_installer/runs/5145081381?check_suite_focus=true#step:7:2164

pulpcore-selinux rpm fails to install for new rpm installs using pulp-installer 3.17+

Version
Please provide the versions of the pulp_installer, pulpcore and plugin packages in use.

pulp_installer 3.17.2
pulpcore 3.17.3
pulp-rpm 3.17.3

Describe the bug
A clear and concise description of what the bug is.

Since the introduction of the fix for #798 pulpcore-selinux doesn't get installed for new installs. Leading to a failure in the install steps that execute selinux commands.

To Reproduce
Steps to reproduce the behavior:

I was working to debug pulpcore rpm pipelines, so steps I took to create the issue:

git clone https://github.com/theforeman/forklift/
cd forklift
ansible-playbook pipelines/pulpcore.yml -e pipeline_version=3.17 -e pipeline_os=centos7 -e pipeline_type=pulpcore -e foreman_expected_version=

I would imagine, given the specific issue, any "pulp-installer" install of pulp-rpm will exhibit the same failure, due to the removal of the pulpcore-selinux package.

Expected behavior
A clear and concise description of what you expected to happen.

For rpm based installations that use selinux, a way for the pulpcore-selinux to be installed from rpm.

Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.

Due to the nature of the dependency chain, (pulpcore-selinux needing to depend on pulpcore), the old way the installer was handling the package led to premature upgrade of core pulpcore rpm packages. The fix at the time was to remove pulpcore-selinux from the undelcared_deps list. This has led to pulpcore-selinux not being installed at all instead. Ideally, a way for pulp installer to handle this package for both installs and upgrades would be needed.

"Collect static content" fails sometimes when run on a shared filesystem

This error was reported when using NFS for /var/lib/pulp:

https://gist.github.com/rlopez133/2f7ce4ef45556a3d446898cdce66e71c

RUNNING HANDLER [pulp.pulp_installer.pulp_common : Collect static content] *****
[WARNING]: Module remote_tmp /var/lib/pulp/.ansible/tmp did not exist and was
created with a mode of 0700, this may cause issues when running as another
user. To avoid this, create the remote_tmp dir with the correct permissions
manually
fatal: [192.168.1.141]: FAILED! => {"changed": true, "cmd": ["/usr/bin/pulpcore-manager", "collectstatic", "--clear", "--noinput", "--link", "--ignore", "galaxy_ng"], "delta": "0:00:02.873603", "end": "2021-10-27 00:47:16.973164", "msg": "non-zero return code", "rc": 1, "start": "2021-10-27 00:47:14.099561", "stderr": "CommandError: [Errno 17] File exists: '/usr/lib/python3.8/site-packages/django/contrib/admin/static/admin/fonts/LICENSE.txt' -> '/var/lib/pulp/assets/admin/fonts/LICENSE.txt'", "stderr_lines": ["CommandError: [Errno 17] File exists: '/usr/lib/python3.8/site-packages/django/contrib/admin/static/admin/fonts/LICENSE.txt' -> '/var/lib/pulp/assets/admin/fonts/LICENSE.txt'"], "stdout": "Deleting 'admin/css/autocomplete.css'", "stdout_lines": ["Deleting 'admin/css/autocomplete.css'"]}
fatal: [192.168.1.251]: FAILED! => {"changed": true, "cmd": ["/usr/bin/pulpcore-manager", "collectstatic", "--clear", "--noinput", "--link", "--ignore", "galaxy_ng"], "delta": "0:00:03.061705", "end": "2021-10-27 00:47:17.279792", "msg": "non-zero return code", "rc": 1, "start": "2021-10-27 00:47:14.218087", "stderr": "CommandError: [Errno 17] File exists: '/usr/lib/python3.8/site-packages/django/contrib/admin/static/admin/img/icon-yes.svg' -> '/var/lib/pulp/assets/admin/img/icon-yes.svg'", "stderr_lines": ["CommandError: [Errno 17] File exists: '/usr/lib/python3.8/site-packages/django/contrib/admin/static/admin/img/icon-yes.svg' -> '/var/lib/pulp/assets/admin/img/icon-yes.svg'"], "stdout": "Deleting 'admin/css/login.css'\nDeleting 'admin/css/changelists.css'\nDeleting 'admin/css/base.css'\nDeleting 'admin/css/forms.css'\nDeleting 'admin/css/fonts.css'\nDeleting 'admin/css/autocomplete.css'\nDeleting 'admin/css/responsive.css'\nDeleting 'admin/css/nav_sidebar.css'\nDeleting 'admin/css/dashboard.css'", "stdout_lines": ["Deleting 'admin/css/login.css'", "Deleting 'admin/css/changelists.css'", "Deleting 'admin/css/base.css'", "Deleting 'admin/css/forms.css'", "Deleting 'admin/css/fonts.css'", "Deleting 'admin/css/autocomplete.css'", "Deleting 'admin/css/responsive.css'", "Deleting 'admin/css/nav_sidebar.css'", "Deleting 'admin/css/dashboard.css'"]}
changed: [192.168.1.201] => {"changed": true, "cmd": ["/usr/bin/pulpcore-manager", "collectstatic", "--clear", "--noinput", "--link", "--ignore", "galaxy_ng"], "delta": "0:00:04.723707", "end": "2021-10-27 00:47:18.840259", "msg": "", "rc": 0, "start": "2021-10-27 00:47:14.116552", "stderr": "", "stderr_lines": [], "stdout": "\n165 static files symlinked to '/var/lib/pulp/assets'.", "stdout_lines": ["", "165 static files symlinked to '/var/lib/pulp/assets'."]}

It happened only once for @rlopez133 , out of repeated runs.

The most logical conclusion is that the 3 nodes were running the task at the same time (against the same filesystem), and they conflicted.

packages mode: pulpcore gets upgraded prematurely during "pulp_common: Install the Pulp undeclared yum package dependencies"

This seems to be causing Ci breakage when upgrading on EL7. Since other packages need to be upgraded at the same time to successfully upgrade:
https://github.com/pulp/pulp_installer/runs/4085353507?check_suite_focus=true#step:7:1108

I recall someone else reporitng a bug caused by this as well.

This underlying reason is that pulpcore-selinux gets installed via that task. But pulpcore-selinux actually depends on pulpcore, not the other way around (as of 3.15.) This was an old workaround, or false assumption that I made when I implemented packages mode support.

Unexpected download failures.

While we use a nice approach with try..until, to avoid unexpected download failures/problems when we trying to download something, in some extreme cases it does not seem to be enough on its own. Some of these cases :

Version:

  • pulp_installer: Latest

To Reproduce
It is difficult to reproduce it because this problem is sporadic and depends on the remote server. But the following happens:

TASK [Get latest CentOS 8 GPG keys and its temporary dependency] ***************
    skipping: [centos-7]
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (5 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (4 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (3 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (2 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (1 retries left).
fatal: [centos-8]: FAILED! => {
    "attempts": 5, 
    "changed": false, 
    "results": []
}

MSG:

Failure downloading https://vault.centos.org/8.5.2111/BaseOS/x86_64/os/Packages/centos-linux-repos-8-3.el8.noarch.rpm, Request failed: <urlopen error _ssl.c:880: The handshake operation timed out>
    TASK [Get latest CentOS 8 GPG keys and its temporary dependency] ***************
    skipping: [centos-7]
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (5 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (4 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (3 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (2 retries left).
    FAILED - RETRYING: Get latest CentOS 8 GPG keys and its temporary dependency (1 retries left).
fatal: [centos-8]: FAILED! => {
    "attempts": 5, 
    "changed": false, 
    "results": []
}

MSG:

Failure downloading https://vault.centos.org/8.5.2111/BaseOS/x86_64/os/Packages/centos-gpg-keys-8-3.el8.noarch.rpm, Request failed: <urlopen error [Errno 99] Cannot assign requested address>

Expected behavior
We want to try to download something again, until it is fully downloaded without any problems.

Possible code for re-evaluation

- name: Migrate CentOS 8 to CentOS Stream 8
block:
- name: Get latest CentOS 8 GPG keys and its temporary dependency
dnf:
name:
- https://vault.centos.org/8.5.2111/BaseOS/x86_64/os/Packages/centos-gpg-keys-8-3.el8.noarch.rpm
- https://vault.centos.org/8.5.2111/BaseOS/x86_64/os/Packages/centos-linux-repos-8-3.el8.noarch.rpm
disablerepo:
- baseos
- appstream
- extras
retries: 5
delay: 12
register: result
until: result is succeeded
when:
- ansible_distribution == "CentOS"
- ansible_distribution_version in ["8.3","8.4"]


References

Update molecule images for upgrades

These are the images used for release-upgrade

platforms:
  - <<: *platform_base
    name: centos-7
    # molecule often fails to pull, so we pull all images in .travis.yml
    image: quay.io/pulp/pulp_rpm-ci-c7:3.1.0
    command: /sbin/init
  - <<: *platform_base
    name: debian-11
    image: quay.io/pulp/pulp-ci-dbullseye:3.14.4
    command: /sbin/init
  - <<: *platform_base
    name: fedora-33
    image: quay.io/pulp/pulp_rpm-ci-f33:3.9.0
    command: /usr/sbin/init

And these are the images used for source-upgrade:

platforms:
  - <<: *platform_base
    name: debian-11
    # molecule often fails to pull, so we pull all images in .travis.yml
    image: quay.io/pulp/pulp-ci-dbullseye:3.14.4
    command: /sbin/init
  - <<: *platform_base
    name: fedora-33
    image: quay.io/pulp/pulp_rpm-ci-f33:3.9.0
    command: /usr/sbin/init

There are 3 shortcomings:

  1. centos-7 is not tested for source-upgrade
  2. centos-8 is not tested at all (this was an omission I made due to being a hurry a few years ago). It should be created for a version inbetween 3.14 and 3.18.
  3. fedora 33 is EOL should be replaced with Fedora 35. The images should be created for the latest version, 3.18.

We should also document how we create them. The most important step (docker commit) is listed here:
https://github.com/pulp/pulp_installer/blob/main/docs/contributing.md

Rough list of steps:

  1. Identify the version of pulpcore/pulp_installer you want to create the image from.
  2. If the version is old, put the variables for the plugin versions in molecule/release-static/group_vars/all . You'll need to research compatible versions. This will be an ad-hoc modification to the file.
  3. Ad-hoc modify molecule/release-static/group_vars/all for any differences vs the intended test, such as the webserver.
  4. run molecule create && molecule converge (defaults to release-static)
  5. Run docker commit
  6. Modify molecule.yml
  7. Run the upgrade test.
  8. Upload the images to quay.io (or whatever new repo we are using.)

Set pulp_pkg_repo by default

Previously, discussions with the Foreman project team were that pulp_installer should not set a default value of pulp_pkg_repo.

Recently, a conversation said otherwise: We can default it, we just have to document the limitations.

This effectively means that if a user sets pulp_install_source==packages, they do not need to manually set pulp_pkg_repo, or try to determine what version is compatible (by reading our CI variables.)

Installation of Galaxy fails due to symlink no such file or directory

The installer fails at step:

TASK [pulp.pulp_installer.pulp_webserver : Symlink nginx snippets] *************
failed: [autohub.ewl.example.com] (item={'changed': False, 'rc': 0, 'stdout': '\r\n/usr/lib/python3.8/site-packages/pulp_container/app/webserver_snippets/nginx.conf\r\n\x1b[0m', 'stdout_lines': ['', '/usr/lib/python3.8/site-packages/pulp_container/app/webserver_snippets/nginx.conf', '\x1b[0m'], 'stderr': 'Shared connection to autohub.ewl.example.com closed.\r\n', 'stderr_lines': ['Shared connection to autohub.ewl.example.com closed.'], 'failed': False, 'failed_when_result': False, '{}pulp_webserver_plugin': {'key': 'pulp-container', 'value': {}}, 'ansible_loop_var': '{}pulp_webserver_plugin'}) => {"{}pulp_webserver_snippet": {"{}pulp_webserver_plugin": {"key": "pulp-container", "value": {}}, "ansible_loop_var": "{}pulp_webserver_plugin", "changed": false, "failed": false, "failed_when_result": false, "rc": 0, "stderr": "Shared connection to autohub.ewl.example.com closed.\r\n", "stderr_lines": ["Shared connection to autohub.ewl.example.com closed."], "stdout": "\r\n/usr/lib/python3.8/site-packages/pulp_container/app/webserver_snippets/nginx.conf\r\n\u001b[0m", "stdout_lines": ["", "/usr/lib/python3.8/site-packages/pulp_container/app/webserver_snippets/nginx.conf", "\u001b[0m"]}, "ansible_loop_var": "{_}_pulp_webserver_snippet", "changed": false, "msg": "Error while linking: [Errno 2] No such file or directory: b'' -> b'/etc/nginx/pulp/pulp_container.conf'", "path": "/etc/nginx/pulp/pulp_container.conf"}
failed: [autohub.ewl.example.com] (item={'changed': False, 'rc': 0, 'stdout': '\r\n/usr/lib/python3.8/site-packages/pulp_ansible/app/webserver_snippets/nginx.conf\r\n\x1b[0m', 'stdout_lines': ['', '/usr/lib/python3.8/site-packages/pulp_ansible/app/webserver_snippets/nginx.conf', '\x1b[0m'], 'stderr': 'Shared connection to autohub.ewl.example.com closed.\r\n', 'stderr_lines': ['Shared connection to autohub.ewl.example.com closed.'], 'failed': False, 'failed_when_result': False, '{}pulp_webserver_plugin': {'key': 'pulp-ansible', 'value': {}}, 'ansible_loop_var': '{}pulp_webserver_plugin'}) => {"{}pulp_webserver_snippet": {"{}pulp_webserver_plugin": {"key": "pulp-ansible", "value": {}}, "ansible_loop_var": "{}pulp_webserver_plugin", "changed": false, "failed": false, "failed_when_result": false, "rc": 0, "stderr": "Shared connection to autohub.ewl.example.com closed.\r\n", "stderr_lines": ["Shared connection to autohub.ewl.example.com closed."], "stdout": "\r\n/usr/lib/python3.8/site-packages/pulp_ansible/app/webserver_snippets/nginx.conf\r\n\u001b[0m", "stdout_lines": ["", "/usr/lib/python3.8/site-packages/pulp_ansible/app/webserver_snippets/nginx.conf", "\u001b[0m"]}, "ansible_loop_var": "{_}_pulp_webserver_snippet", "changed": false, "msg": "Error while linking: [Errno 2] No such file or directory: b'' -> b'/etc/nginx/pulp/pulp_ansible.conf'", "path": "/etc/nginx/pulp/pulp_ansible.conf"}
failed: [autohub.ewl.example.com] (item={'changed': False, 'rc': 0, 'stdout': '\r\n/usr/lib/python3.8/site-packages/galaxy_ng/app/webserver_snippets/nginx.conf\r\n\x1b[0m', 'stdout_lines': ['', '/usr/lib/python3.8/site-packages/galaxy_ng/app/webserver_snippets/nginx.conf', '\x1b[0m'], 'stderr': 'Shared connection to autohub.ewl.example.com closed.\r\n', 'stderr_lines': ['Shared connection to autohub.ewl.example.com closed.'], 'failed': False, 'failed_when_result': False, '{}pulp_webserver_plugin': {'key': 'galaxy-ng', 'value': {'collectstatic': False, 'version': '4.4.0'}}, 'ansible_loop_var': '{}pulp_webserver_plugin'}) => {"{}pulp_webserver_snippet": {"{}pulp_webserver_plugin": {"key": "galaxy-ng", "value": {"collectstatic": false, "version": "4.4.0"}}, "ansible_loop_var": "{}pulp_webserver_plugin", "changed": false, "failed": false, "failed_when_result": false, "rc": 0, "stderr": "Shared connection to autohub.ewl.example.com closed.\r\n", "stderr_lines": ["Shared connection to autohub.ewl.example.com closed."], "stdout": "\r\n/usr/lib/python3.8/site-packages/galaxy_ng/app/webserver_snippets/nginx.conf\r\n\u001b[0m", "stdout_lines": ["", "/usr/lib/python3.8/site-packages/galaxy_ng/app/webserver_snippets/nginx.conf", "\u001b[0m"]}, "ansible_loop_var": "{_}_pulp_webserver_snippet", "changed": false, "msg": "Error while linking: [Errno 2] No such file or directory: b'' -> b'/etc/nginx/pulp/galaxy_ng.conf'", "path": "/etc/nginx/pulp/galaxy_ng.conf"}

This is actually a late consequence of the following task:

TASK [pulp.pulp_installer.pulp_webserver : Check installed plugins for nginx snippets] ***
ok: [autohub.ewl.example.com] => (item={'key': 'pulp-container', 'value': {}}) => {"{}pulp_webserver_plugin": {"key": "pulp-container", "value": {}}, "ansible_loop_var": "{_}_pulp_webserver_plugin", "changed": false, "failed_when_result": false, "rc": 0, "stderr": "Shared connection to autohub.ewl.example.com closed.\r\n", "stderr_lines": ["Shared connection to autohub.ewl.example.com closed."], "stdout": "\r\n/usr/lib/python3.8/site-packages/pulp_container/app/webserver_snippets/nginx.conf\r\n\u001b[0m", "stdout_lines": ["", "/usr/lib/python3.8/site-packages/pulp_container/app/webserver_snippets/nginx.conf", "\u001b[0m"]}
ok: [autohub.ewl.example.com] => (item={'key': 'pulp-ansible', 'value': {}}) => {"{}pulp_webserver_plugin": {"key": "pulp-ansible", "value": {}}, "ansible_loop_var": "{_}_pulp_webserver_plugin", "changed": false, "failed_when_result": false, "rc": 0, "stderr": "Shared connection to autohub.ewl.example.com closed.\r\n", "stderr_lines": ["Shared connection to autohub.ewl.example.com closed."], "stdout": "\r\n/usr/lib/python3.8/site-packages/pulp_ansible/app/webserver_snippets/nginx.conf\r\n\u001b[0m", "stdout_lines": ["", "/usr/lib/python3.8/site-packages/pulp_ansible/app/webserver_snippets/nginx.conf", "\u001b[0m"]}
ok: [autohub.ewl.example.com] => (item={'key': 'galaxy-ng', 'value': {'collectstatic': False, 'version': '4.4.0'}}) => {"{}pulp_webserver_plugin": {"key": "galaxy-ng", "value": {"collectstatic": false, "version": "4.4.0"}}, "ansible_loop_var": "{_}_pulp_webserver_plugin", "changed": false, "failed_when_result": false, "rc": 0, "stderr": "Shared connection to autohub.ewl.example.com closed.\r\n", "stderr_lines": ["Shared connection to autohub.ewl.example.com closed."], "stdout": "\r\n/usr/lib/python3.8/site-packages/galaxy_ng/app/webserver_snippets/nginx.conf\r\n\u001b[0m", "stdout_lines": ["", "/usr/lib/python3.8/site-packages/galaxy_ng/app/webserver_snippets/nginx.conf", "\u001b[0m"]}

The stdout of the script check_snippet.py registered in the variable snippets in collections/ansible_collections/pulp/pulp_installer/roles/pulp_webserver/tasks/main.yml has an unexpected empty line which is taken as source for the symlinking with the | first pipe in collections/ansible_collections/pulp/pulp_installer/roles/pulp_webserver/tasks/nginx.yml instead of the filename on the 2nd line.

https://issues.redhat.com/browse/AAP-1413

Setting pulp_user_home should set the entirety of /var/lib/pulp

At initial install time, users should be able to chose a new location for all the data under /var/lib/pulp by setting pulp_user_home.

The follow variables are independent of it by default:

  • roles/pulp_common/defaults/main.yml:pulp_cache_dir: '/var/lib/pulp/tmp'
  • roles/pulp_common/defaults/main.yml:pulp_media_root: '/var/lib/pulp/media'

The following variables are defined relative to it and require no further updates

  • roles/pulp_webserver/defaults/main.yml:pulp_webserver_static_dir: "{{ pulp_user_home | regex_replace('\/$', '') }}/pulpcore_static"
  • roles/pulp_common/templates/settings.py.j2:STATIC_ROOT = "{{ pulp_user_home }}/assets"

We probably need to set them all with the regex_replace. I remember us having a discussion over the regex_replace when pulp_webserver_static_dir was added.

Also, the docs roles/pulp_common/README.md should state what the PULP_USER_HOME's default is, and that the other variables are defined relative to it (effectively /var/lib/pulp/foo).

PostgreSQL failed to initialize

TASK [pulp.pulp_installer.pulp_database : Ensure PostgreSQL database is initialized.] ***
�[1;30mtask path: /root/.ansible/collections/ansible_collections/pulp/pulp_installer/roles/pulp_database/tasks/install_postgres.yml:91�[0m
�[0;31mfatal: [pulprepo001.idx.way.blu]: FAILED! => {"changed": false, "module_stderr": "/etc/profile.d/lang.sh: line 19: warning: setlocale: LC_CTYPE: cannot change locale (C.UTF-8)\n", "module_stdout": "", "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error", "rc": 1}�[0m

"risky-file-permissions File permissions unset or incorrect"

ansible lint is complaining about a bunch of file permissions.
You can see output with any PR. e.g. https://github.com/pulp/pulp_installer/pull/797/files


Check failure on line 13 in roles/pulp_common/tasks/ambiguously-named-repo.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/ambiguously-named-repo.yml#L13

risky-file-permissions File permissions unset or incorrect

Check failure on line 13 in roles/pulp_common/tasks/ambiguously-named-repo.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/ambiguously-named-repo.yml#L13

risky-file-permissions File permissions unset or incorrect

  ini_file:
    path: "{{ repo_file.stdout }}"
    section: "{{ __ambiguously_named_repo }}"
roles/pulp_common/tasks/install_pip.yml
  become: true
  become_user: '{{ pulp_user }}'

- name: Create pulp install dir

Check failure on line 79 in roles/pulp_common/tasks/install_pip.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/install_pip.yml#L79

risky-file-permissions File permissions unset or incorrect

Check failure on line 79 in roles/pulp_common/tasks/install_pip.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/install_pip.yml#L79

risky-file-permissions File permissions unset or incorrect

  file:
    path: '{{ pulp_install_dir }}'
    state: directory
        clients: "{{ pulp_install_dir }}/bin/pip"
      register: pip_pkgs

    - name: Create requirements.in file to check pulpcore/plugin compatibility

Check failure on line 125 in roles/pulp_common/tasks/install_pip.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/install_pip.yml#L125

risky-file-permissions File permissions unset or incorrect

Check failure on line 125 in roles/pulp_common/tasks/install_pip.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/install_pip.yml#L125

risky-file-permissions File permissions unset or incorrect

      template:
        src: templates/requirements.in.j2
        dest: "{{ pulp_install_dir }}/requirements.in"
        virtualenv_command: '{{ pulp_python_interpreter }} -m venv'
      when: pip_pkgs.packages[pulp_install_dir + '/bin/pip'].dynaconf[0].version is version("3.1.1", "<")

    - name: Create constraints file to lock the django and pulpcore version when plugins are installed

Check failure on line 293 in roles/pulp_common/tasks/install_pip.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/install_pip.yml#L293

risky-file-permissions File permissions unset or incorrect

Check failure on line 293 in roles/pulp_common/tasks/install_pip.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/install_pip.yml#L293

risky-file-permissions File permissions unset or incorrect

      template:
        src: "pip_constraints_for_plugins.txt.j2"
        dest: "{{ pulp_install_dir }}/pip_constraints_for_plugins.txt"
roles/pulp_common/tasks/preflight_function.yml
  become: true
  become_user: '{{ pulp_user }}'

- name: Backup currently installed packages for any potential troubleshooting purposes

Check failure on line 25 in roles/pulp_common/tasks/preflight_function.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/preflight_function.yml#L25

risky-file-permissions File permissions unset or incorrect

Check failure on line 25 in roles/pulp_common/tasks/preflight_function.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/preflight_function.yml#L25

risky-file-permissions File permissions unset or incorrect

  copy:
    src: '{{ pulp_install_dir }}/requirements.txt'
    dest: '{{ pulp_install_dir }}/requirements.txt.orig'
roles/pulp_common/tasks/repos.yml
      failed_when: false
      check_mode: False

    - name: Enable the CentOS PowerTools repo

Check failure on line 98 in roles/pulp_common/tasks/repos.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/repos.yml#L98

risky-file-permissions File permissions unset or incorrect

Check failure on line 98 in roles/pulp_common/tasks/repos.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/repos.yml#L98

risky-file-permissions File permissions unset or incorrect

      ini_file:
        path: "{{ repo_file.stdout }}"
        section: "PowerTools"
      failed_when: false
      check_mode: False

    - name: Enable the CentOS powertools repo

Check failure on line 115 in roles/pulp_common/tasks/repos.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/repos.yml#L115

risky-file-permissions File permissions unset or incorrect

Check failure on line 115 in roles/pulp_common/tasks/repos.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/repos.yml#L115

risky-file-permissions File permissions unset or incorrect

      ini_file:
        path: "{{ repo_file.stdout }}"
        section: "powertools"
    - ansible_facts.distribution == "CentOS"
    - ansible_facts.distribution_major_version|int >= 8

- name: "Configure {{ __pulp_pkg_repo_name }} RPM repository"

Check failure on line 129 in roles/pulp_common/tasks/repos.yml
GitHub Actions / core (ansible-core)

roles/pulp_common/tasks/repos.yml#L129

risky-file-permissions File permissions unset or incorrect

Check failure on line 129 in roles/pulp_common/tasks/repos.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_common/tasks/repos.yml#L129

risky-file-permissions File permissions unset or incorrect

  template:
    src: pulpcore.repo.j2
    dest: /etc/yum.repos.d/pulpcore.repo
roles/pulp_database/tasks/ambiguously-named-repo.yml
  failed_when: false
  check_mode: False

- name: "Enable the {{ __ambiguously_named_repo }} repo"

Check failure on line 13 in roles/pulp_database/tasks/ambiguously-named-repo.yml
GitHub Actions / core (ansible-core)

roles/pulp_database/tasks/ambiguously-named-repo.yml#L13

risky-file-permissions File permissions unset or incorrect

Check failure on line 13 in roles/pulp_database/tasks/ambiguously-named-repo.yml
GitHub Actions / core (git+https://github.com/ansible/ansible.git)

roles/pulp_database/tasks/ambiguously-named-repo.yml#L13

risky-file-permissions File permissions unset or incorrect

  ini_file:
    path: "{{ repo_file.stdout }}"
    section: "{{ __ambiguously_named_repo }}"

pulp_installer is unable to upgrade from very-old RPMs due to python3-drf-yasg

Pulp 3.8 replaced drf-yasg with drf-spectacular, but on EL8, upgrading directly from pulp versions < 3.8, to 3.15 (where the python 3.8 change happened), causes the install to fail.

It cannot upgrade or remove python3-drf-yasg, but the old version of python3-django depends on python3-drf-yasg. So python38-django conflicts with python3-django.

pulp_configure_firewall should not install a firewall if set to "auto"

pulp_configure_firewall is currently explained (in pulp_webserver's README.md):

* `pulp_configure_firewall` Install and configure a firewall. Valid values are `auto`, `firewalld`,
  and `none`. Defaults to `auto` (which is the same as `firewalld`, but may change in the future).

This does not make much sense for "auto".

"auto" should detect if firewalld is already installed, and configure firewalld if it is installed.

Only a value of "firewalld" should result in it being installed.

Vagrant devel installs have SELinux errors

Transferred from https://pulp.plan.io/issues/9211

TASK [pulp_devel : SELinux status] *********************************************
ok: [pulp3-source-fedora34] => {
    "selinux_analyze.stdout_lines": [
        "SELinux is preventing gunicorn from search access on the directory vagrant.",
        "SELinux is preventing gunicorn from search access on the directory /.",
        "SELinux is preventing gunicorn from getattr access on the directory /home/vagrant/devel/pulpcore.",
        "SELinux is preventing gunicorn from read access on the directory models.",
        "SELinux is preventing gunicorn from open access on the directory /home/vagrant/devel/pulpcore/pulpcore/app/models.",
        "SELinux is preventing gunicorn from getattr access on the directory /home/vagrant.",
        "SELinux is preventing gunicorn from getattr access on the file /home/vagrant/devel/pulp_ansible/pulp_ansible/app/settings.py.",
        "SELinux is preventing gunicorn from read access on the file settings.py.",
        "SELinux is preventing gunicorn from open access on the file /home/vagrant/devel/pulpcore/pulpcore/app/settings.py.",
        "SELinux is preventing gunicorn from ioctl access on the file /home/vagrant/devel/pulp_ansible/pulp_ansible/app/settings.py.",
        "SELinux is preventing pulpcore-worker from read access on the file __init__.cpython-39.pyc.",
        "SELinux is preventing pulpcore-worker from open access on the file /home/vagrant/devel/pulp_ansible/pulp_ansible/__pycache__/__init__.cpython-39.pyc.",
        "SELinux is preventing pulpcore-worker from ioctl access on the file /home/vagrant/devel/pulp_ansible/pulp_ansible/__pycache__/__init__.cpython-39.pyc.",
        "SELinux is preventing pulpcore-worker from name_connect access on the tcp_socket port 5432.",
        "SELinux is preventing pulpcore-worker from add_name access on the directory [email protected].",
        "SELinux is preventing pulpcore-worker from remove_name access on the directory [email protected].",
        "SELinux is preventing pulpcore-worker from rmdir access on the directory [email protected].",
        "SELinux is preventing nginx from read access on the file nginx.conf.",
        "SELinux is preventing nginx from open access on the file /home/vagrant/devel/pulp_ansible/pulp_ansible/app/webserver_snippets/nginx.conf.",
        "SELinux is preventing nginx from getattr access on the file /home/vagrant/devel/pulp_ansible/pulp_ansible/app/webserver_snippets/nginx.conf."
    ]
}

This appears to be due to /home/vagrant being on sshfs.

The policies could be updated to allow access to sshfs (this may not be appropriate security-wise unless made optional via a boolean, or a separate policy file), or perhaps vagrant's sshfs can accept an SELinux label.

Error during pulp_installer if ran as user-data

Running pulp_installer as user-data at the end of an EC2 fresh install fails during one of the last steps with:

TASK [pulp.pulp_installer.pulp_webserver : Accept HTTP connections on port 80] ***
ERROR:dbus.proxies:Introspect error on :1.57:/org/fedoraproject/FirewallD1: dbus.exceptions.DBusException: org.freedesktop.DBus.Error.NoReply: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: DBusException: org.freedesktop.DBus.Error.Disconnected: Connection was disconnected before a reply was received
fatal: [127.0.0.1]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}

It seems a problem with firewalld hanging up if instance is freshly installed and still not rebooted:
https://forums.centos.org/viewtopic.php?f=47&t=52162&p=220915#p220915
but I cannot be sure.

I can provide my user-data (shell script), if it can be useful.

From: https://pulp.plan.io/issues/9528

pulp.pulp_installer.pulp_common : Run pip-compile to check pulpcore/plugin compatibility]

When running the an initial install, using the ansible installer version 3.17.0 I get the following error:

fatal: [gazza-boy]: FAILED! => {"changed": false, "cmd": ["/usr/local/lib/pulp/bin/pip-compile"], "delta": "0:00:00.641376", "end": "2022-02-01 13:52:47.438926", "failed_when_result": true, "msg": "non-zero return code", "rc": 1, "start": "2022-02-01 13:52:46.797550", "stderr": "Traceback (most recent call last):\n File "/usr/local/lib/pulp/bin/pip-compile", line 8, in \n sys.exit(cli())\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 1128, in call\n return self.main(*args, **kwargs)\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 1053, in main\n rv = self.invoke(ctx)\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 1395, in invoke\n return ctx.invoke(self.callback, **ctx.params)\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 754, in invoke\n return __callback(*args, **kwargs)\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/decorators.py", line 26, in new_func\n return f(get_current_context(), *args, **kwargs)\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/piptools/scripts/compile.py", line 342, in cli\n repository = PyPIRepository(pip_args, cache_dir=cache_dir)\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/piptools/repositories/pypi.py", line 106, in init\n self._setup_logging()\n File "/usr/local/lib/pulp/lib64/python3.8/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging\n assert isinstance(handler, logging.StreamHandler)\nAssertionError", "stderr_lines": ["Traceback (most recent call last):", " File "/usr/local/lib/pulp/bin/pip-compile", line 8, in ", " sys.exit(cli())", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 1128, in call", " return self.main(*args, **kwargs)", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 1053, in main", " rv = self.invoke(ctx)", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 1395, in invoke", " return ctx.invoke(self.callback, **ctx.params)", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/core.py", line 754, in invoke", " return __callback(*args, **kwargs)", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/click/decorators.py", line 26, in new_func", " return f(get_current_context(), *args, **kwargs)", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/piptools/scripts/compile.py", line 342, in cli", " repository = PyPIRepository(pip_args, cache_dir=cache_dir)", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/piptools/repositories/pypi.py", line 106, in init", " self._setup_logging()", " File "/usr/local/lib/pulp/lib64/python3.8/site-packages/piptools/repositories/pypi.py", line 455, in _setup_logging", " assert isinstance(handler, logging.StreamHandler)", "AssertionError"], "stdout": "", "stdout_lines": []}

pulp_install_selinux_policies is not used anywhere

$ grep version  MANIFEST.json
  "version": "3.18.0",
/home/user1/.ansible/collections/ansible_collections/pulp/pulp_installer/roles/pulp_common
(polemarch) [polemarch@s-msk-p-plm-as2 pulp_common]$ grep -riH pulp_install_selinux_policies *
    - name: Clone SELinux policy from Git
      git:
        # TEMP: Temporary repo and branch for el8 and related set of features.
        repo: '{{ __pulp_selinux_repo }}'
        version: '{{ __pulp_selinux_version }}'
        dest: '{{ pulp_user_home }}/pulpcore-selinux'
      become: true
      become_user: '{{ pulp_user }}'

    - name: Compile the SELinux policies
      make:
        file: /usr/share/selinux/devel/Makefile
        chdir: '{{ pulp_user_home }}/pulpcore-selinux'
        target: '{{ item }}.pp'
      loop: '{{ __pulp_selinux_policy_pkgs }}'
      become: true
      become_user: '{{ pulp_user }}'

    - name: Install the SELinux policy packages on disk
      copy:
        src: '{{ pulp_user_home }}/pulpcore-selinux/{{ item }}.pp'
        dest: '/usr/local/share/selinux/{{ ansible_facts.selinux.type }}/{{ item }}.pp'
        remote_src: true
        mode: 0644
        owner: root
        group: root
      loop: '{{ __pulp_selinux_policy_pkgs }}'
      notify:
        - Load the SELinux policy packages
        - Restore SELinux contexts on Pulp dirs that must exist
        - Restore SELinux contexts on Pulp dirs that may exist

  become: true
  when:
    - ansible_facts.os_family == 'RedHat'
    # when permissive or enforcing. That would be stored in .mode & .config_mode
    - ansible_facts.selinux.status == "enabled"

There is no any condition in 'when' block on pulp_install_selinux_policies variable based

"symlink has no referent" error during vagrant up

When trying to "vagrant up pulp3-source-fedora35" on fedora35 (from git master branch) I get an error:

There was an error when attempting to rsync a synced folder.
Please inspect the error message below for more info.

Host path: /home/<user>/devel/pulp_installer/
Guest path: /vagrant
Command: "rsync" "--verbose" "--archive" "--delete" "-z" "--copy-links" "--no-owner" "--no-group" "--rsync-path" "sudo rsync" "-e" "ssh -p 2222 -o LogLevel=FATAL   -o ControlMaster=auto -o ControlPath=/tmp/vagrant-rsync-20211130-117348-txfbyq -o ControlPersist=10m  -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -i '/home/<user>/devel/pulp_installer/.vagrant/machines/pulp3-source-fedora35/virtualbox/private_key'" "--exclude" ".vagrant/" "/home/<user>/devel/pulp_installer/" "[email protected]:/vagrant"
Error: symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/default/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/packages-dynamic/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/packages-static/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/packages-upgrade/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/release-dynamic/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/release-static/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/release-upgrade/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/source-dynamic/debian-redis-server.service"
symlink has no referent: "/home/<user>/devel/pulp_installer/molecule/source-upgrade/debian-redis-server.service"
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1330) [sender=3.2.3]

Simple workaround is to remove those files and installation end up successfully.

pulp-installer getting started cannot be followed.

Following https://docs.pulpproject.org/pulp_installer/quickstart/#getting-started
leads to an experience of "computer-says-no".

  • missing dependency: sudo
    sudo
  • missing dependency: role gerlingguy.postgresql install gerlingguy.postgresql
  • should reference hardware requirements hardware requirements
  • should tell user to install ansible so ansible-playbook and ansible-galaxy commands are accessible.
  • should tell user to look where path of example-use is. whereis_playbook_path
  • should tell user to add localhost to inventory. giving up no target host

And probably the missing role could be depended on.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.