pulp / pulp_ostree Goto Github PK

Is your feature request related to a problem? Please describe.
While importing a commit to pulp ostree repository needs to mention the ref which can be determined automatically.
If it's not mentioned, getting the error The parent commit could not be loaded.

Describe the solution you'd like
If the ref parameter wasn't mentioned determine it automatically to avoid confusion.

Describe alternatives you've considered
Another option is changing ref to be a mandatory parameter.

Steps to reproduce the issue

Instantiate a new repository locally and publish it:

# on the host machine
ostree --repo=repo1 init --mode=archive
mkdir test
echo "test1" > test/test.txt
ostree --repo=repo1 commit --branch foo test
ostree summary -u --repo=repo1

# on the host machine
python3 -m http.server 8001

# on the pulp machine
pulp ostree repository create --name foo
pulp ostree remote create --name foo --url http://0.0.0.0:8001/repo1/
pulp ostree repository sync --name foo --remote foo
pulp ostree distribution create --name foo --base-path foo --repository foo
http http://localhost:5001/pulp/content/foo/summary

Add a new commit to the repository and re-sync it:

# on the host machine
echo "test2" > test/test2.txt
ostree --repo=repo1 commit --branch foo test
ostree summary -u --repo=repo1

# on the pulp machine
pulp ostree repository sync --name foo --remote foo
http http://localhost:5001/pulp/content/foo/summary

pulp [None]: pulpcore.content.handler:ERROR: Multiple (pass-through) matches for {b}/{p}
[2023-05-24 17:05:33 +0000] [1141] [ERROR] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request
    resp = await request_handler(request)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_app.py", line 504, in _handle
    resp = await handler(request)
  File "/usr/local/lib64/python3.8/site-packages/aiohttp/web_middlewares.py", line 117, in impl
    return await handler(request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/authentication.py", line 48, in authenticate
    return await handler(request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/handler.py", line 245, in stream_content
    return await self._match_and_stream(path, request)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/handler.py", line 673, in _match_and_stream
    ca = await sync_to_async(get_contentartifact_blocking)()
  File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 448, in __call__
    ret = await asyncio.wait_for(future, timeout=None)
  File "/usr/lib64/python3.8/asyncio/tasks.py", line 455, in wait_for
    return await fut
  File "/usr/lib64/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/asgiref/sync.py", line 490, in thread_handler
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pulpcore/content/handler.py", line 668, in get_contentartifact_blocking
    ca = ContentArtifact.objects.select_related(
  File "/usr/local/lib/python3.8/site-packages/django/db/models/query.py", line 439, in get
    raise self.model.MultipleObjectsReturned(
pulpcore.app.models.content.ContentArtifact.MultipleObjectsReturned: get() returned more than one ContentArtifact -- it returned 2!
127.0.0.1 [24/May/2023:17:05:33 +0000] "GET /pulp/content/foo/summary HTTP/1.0" 500 225 "-" "HTTPie/3.2.2"

The errors are raised for config and summary files.

Enabling tests to run in s3 or azure storage backend results in raising the following error:

Internal Server Error: /rerouted/djnd/api/v3/repositories/ostree/ostree/c61a3eaa-2cd3-420a-91fc-a0c0e7060f4a/import_commits/
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/exception.py", line 47, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.8/site-packages/django/core/handlers/base.py", line 181, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.8/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/viewsets.py", line 125, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 509, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 469, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 480, in raise_uncaught_exception
    raise exc
  File "/usr/local/lib/python3.8/site-packages/rest_framework/views.py", line 506, in dispatch
    response = handler(request, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pulp_ostree/app/viewsets.py", line 75, in import_commits
    serializer.is_valid(raise_exception=True)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 227, in is_valid
    self._validated_data = self.run_validation(self.initial_data)
  File "/usr/local/lib/python3.8/site-packages/rest_framework/serializers.py", line 429, in run_validation
    value = self.validate(value)
  File "/usr/local/lib/python3.8/site-packages/pulp_ostree/app/serializers.py", line 38, in validate
    self.validate_tarball(new_data)
  File "/usr/local/lib/python3.8/site-packages/pulp_ostree/app/serializers.py", line 45, in validate_tarball
    if not is_tarfile(artifact.file.path):
  File "/usr/local/lib/python3.8/site-packages/django/db/models/fields/files.py", line 59, in path
    return self.storage.path(self.name)
  File "/usr/local/lib/python3.8/site-packages/django/core/files/storage.py", line 128, in path
    raise NotImplementedError("This backend doesn't support absolute paths.")

When importing a commit into a repository a user is required to specify a parent commit if he want to create a new ref. The parent commit should be optional and Pulp should try to get it from the commit.

[vagrant@centos7-hammer-devel ~]$ hammer repository upload-content --path ~/projects/2efad613-fb5e-4485-b67c-10f8ebed27e8-commit.tar  --id 1
Could not upload the content:
  OSTree commit ref uploads require a repository name.

The pulp logs:

Nov 01 13:26:21 pulp [842c773a-fff6-4ed4-840c-2c33e805b87a]:  - - [01/Nov/2021:13:26:21 +0000] "GET /pulp/api/v3/status HTTP/1.1" 301 0 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.3p183"
Nov 01 13:26:21 pulp [842c773a-fff6-4ed4-840c-2c33e805b87a]:  - - [01/Nov/2021:13:26:21 +0000] "GET /pulp/api/v3/status/ HTTP/1.1" 200 2401 "-" "rest-client/2.1.0 (linux x86_64) ruby/2.7.3p183"
Nov 01 13:26:22 pulp [842c773a-fff6-4ed4-840c-2c33e805b87a]:  - - [01/Nov/2021:13:26:22 +0000] "POST /pulp/api/v3/remotes/ostree/ostree/ HTTP/1.1" 201 566 "-" "OpenAPI-Generator/2.0.0a2/ruby"
Nov 01 13:26:22 pulp [842c773a-fff6-4ed4-840c-2c33e805b87a]:  - - [01/Nov/2021:13:26:22 +0000] "POST /pulp/api/v3/repositories/ostree/ostree/ HTTP/1.1" 201 464 "-" "OpenAPI-Generator/2.0.0a2/ruby"

{"displayMessage":"OSTree commit ref uploads require a repository name.","errors":["OSTree commit ref uploads require a repository name."]}

When syncing an ostree remote on in one pulp instance, that is pointing to a 2nd pulp instance, i seem to get a 404:

    description: 404, message='Not Found', url=URL('https://devel8.markarth.example.com/pulp/content/Default_Organization/Library/custom/ostree/test/config')

The base path of the distribution in the upstream satellite is: Default_Organization/Library/custom/ostree/test (and content is served from /pulp/content)

The change https://github.com/pulp/pulp_ostree/pull/175/files introduced the concept of including/excluding particular refs from a remote repository. The change should be documented with pulp-cli-ostree examples.

After deleting all my ostree repos, i ran orphan cleanup, but all my ostree content was still in pulp. It doesn't seem to be getting cleaned up properly as i still see artifacts/content units/refs/commits in the db

I was uploading a ~2gb ostree tar to katello and I hit this error: Too many open files: '/var/lib/pulp/media/upload/dc61a682-91aa-4733-b52e-4f5e7b22e9ce'. I used the fedora cockpit image builder UI to create the image.

Related part of log:

Dec 10 14:57:23 centos8-katello-devel-nfs pulpcore-api[675]: pulp [3c5cdf9e-9b0a-4c6b-8805-2450b3df4b00]: - - [10/Dec/2021:14:57:23 +0000] "GET /pulp/api/v3/tasks/e6a9ae4e-6a83-419d-bffa-be4ecd6c6e78/ HTTP/1.1" 200 565 "-" "OpenAPI-Gener
ator/3.16.1/ruby"
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: pulp [3c5cdf9e-9b0a-4c6b-8805-2450b3df4b00]: pulpcore.tasking.pulpcore_worker:INFO: Task e6a9ae4e-6a83-419d-bffa-be4ecd6c6e78 failed ([Errno 24] Too many open files: '/var/
lib/pulp/media/upload/dc61a682-91aa-4733-b52e-4f5e7b22e9ce')
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: pulp [3c5cdf9e-9b0a-4c6b-8805-2450b3df4b00]: pulpcore.tasking.pulpcore_worker:INFO: File "/usr/lib/python3.8/site-packages/pulpcore/tasking/pulpcore_worker.py", line 370,
in _perform_task
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: result = func(*args, **kwargs)
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: File "/usr/lib/python3.8/site-packages/pulpcore/app/tasks/upload.py", line 31, in commit
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: temp_file.write(chunk.file.read())
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: File "/usr/lib/python3.8/site-packages/django/core/files/utils.py", line 42, in
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: read = property(lambda self: self.file.read)
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: File "/usr/lib/python3.8/site-packages/django/db/models/fields/files.py", line 45, in _get_file
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: self._file = self.storage.open(self.name, 'rb')
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: File "/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 38, in open
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: return self._open(name, mode)
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: File "/usr/lib/python3.8/site-packages/django/core/files/storage.py", line 238, in _open
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-worker-2[848]: return File(open(self.path(name), mode))
Dec 10 14:57:24 centos8-katello-devel-nfs pulpcore-api[675]: pulp [3c5cdf9e-9b0a-4c6b-8805-2450b3df4b00]: - - [10/Dec/2021:14:57:24 +0000] "GET /pulp/api/v3/tasks/e6a9ae4e-6a83-419d-bffa-be4ecd6c6e78/ HTTP/1.1" 200 1559 "-" "OpenAPI-Gene
rator/3.16.1/ruby"
Dec 10 14:57:25 centos8-katello-devel-nfs pulpcore-api[675]: pulp [3c5cdf9e-9b0a-4c6b-8805-2450b3df4b00]: - - [10/Dec/2021:14:57:25 +0000] "GET /pulp/api/v3/distributions/ostree/ostree/?base_path=Default_Organization%2FLibrary%2Fcustom%2
F1%2F1 HTTP/1.1" 200 553 "-" "OpenAPI-Generator/2.0.0a3.dev1639106532/ruby"

I tried increasing the open file limit following these instructions:

1. In /etc/sysctl.conf, fs.file-max = 262144
2. In /etc/security/limits.conf

• soft nproc 65535
• hard nproc 65535
• soft nofile 65535
• hard nofile 65535

3. Reloaded environment

os.path.join:

Join one or more path components intelligently. The return value is the concatenation of path and any members of *paths with exactly one directory separator (os.sep) following each non-empty part except the last, meaning that the result will only end in a separator if the last part is empty. If a component is an absolute path, all previous components are thrown away and joining continues from the absolute path component.

os.path.join(/task_cwd/, /tmp/repo_name) == /tmp/repo_name

It is not necessary to store the checksum of a commit or a couple of lines for a configuration in a separate file. Use TextField instead.

The sync and import pipelines are very slow when downloading or associating content.

The root cause of the problem might be the use of built-in ostree capabilities provided by the gi module.

• Apply migrations from pulpcore
• Remove redundant sync_to_async calls
• Bump dependencies

Version
Please provide the versions of the pulpcore and pulp_ostree packages in use, and how they are installed. If you are using Pulp via Katello, please provide the Katello version.

Describe the bug
A clear and concise description of what the bug is.

To Reproduce
Steps to reproduce the behavior:

Expected behavior
A clear and concise description of what you expected to happen.

Additional context
Add any other context about the problem here. Please provide links to any previous discussions via Discourse or Bugzilla.

First, a new content unit(s) for static deltas must be established. Furthermore, Pulp will provide a facility for computing the static deltas.

The calculation should be pretty straightforward for repositories which do not contain static deltas -- a single call to https://lazka.github.io/pgi-docs/#OSTree-1.0/classes/Repo.html#OSTree.Repo.static_delta_generate. If the static deltas are present in the repository, there is nothing to do.

A Pulp repository will contain an option (e.g., deltas_upto=1) that will tell Pulp how many commits deeper should the deltas be computed if not present. The procedure should be reusable for both synced (remote repositories) and imported (uploaded tarballs) repositories.

Before creating a new repository version (after every import/sync), Pulp computes the static delta between recognized commits. The process will be completely hidden from a user.

This might be useful when we would want to benefit from the domains feature. We should follow the same approach as we sketched in pulp_rpm, where we first rewrote the tests and then added the domains support.

Users should be allowed to specify which refs they want to be synced. Syncing a whole remote repository may be cumbersome because there can be dozens of refs. Syncing a single ref from the official Fedora IoT repository takes approximately 15 minutes.

Users will be able to specify the refs they want to sync from like this:

pulp ostree remote create --name bla --url bla --include-refs "fedora/stable/x86_64/iot, fedora/devel/x86_64/iot"