flatpak / flat-manager

Manager for flatpak repositories

Flatpak is a system for building, distributing, and running sandboxed desktop applications on Linux.

See https://flatpak.org/ for more information.

Flatpak is available in the package repositories of most Linux distributions and can be installed from there. See https://flatpak.org/setup/ for quick setup instructions for many distributions.

Community discussion happens in #flatpak:matrix.org, on the mailing list, and on the Flathub Discourse.

Read documentation for Flatpak here.

Contributing

Flatpak welcomes contributions from anyone! Here are some ways you can help:

• Fix one of the issues and submit a PR
• Update flatpak's translations and submit a PR
• Improve flatpak's documentation, hosted at http://docs.flatpak.org and developed over in flatpak-docs
• Find a bug and submit a detailed report including your OS, flatpak version, and the steps to reproduce
• Add your favorite application to Flathub by writing a flatpak-builder manifest and submitting it
• Improve the Flatpak support in your favorite Linux distribution

Hacking

See CONTRIBUTING.md

Related Projects

Here are some notable projects in the Flatpak ecosystem:

• Flatseal: An app for managing permissions of Flatpak apps without using the CLI
• Flat-manager: A tool for managing Flatpak repositories

flat-manager's Issues

flatpak-manager-client keeps failing with ValueError when building org.libreoffice.LibreOffice on flathub

Caused latest two build attempts https://flathub.org/builds/#/builders/32/builds/770 and https://flathub.org/builds/#/builders/32/builds/1098 to fail, see https://flathub.org/builds/#/builders/32/builds/1098/steps/11/logs/stdio:

Traceback (most recent call last):
  File "/srv/buildbot-master/sandbox/lib/python3.4/site-packages/buildbot/scripts/flat-manager-client", line 539, in <module>
    res = args.func(args)
  File "/srv/buildbot-master/sandbox/lib/python3.4/site-packages/buildbot/scripts/flat-manager-client", line 434, in commit_command
    commit_build(args.build_url, args.wait, args.token)
  File "/srv/buildbot-master/sandbox/lib/python3.4/site-packages/buildbot/scripts/flat-manager-client", line 291, in commit_build
    job = wait_for_job (job_url, token);
  File "/srv/buildbot-master/sandbox/lib/python3.4/site-packages/buildbot/scripts/flat-manager-client", line 241, in wait_for_job
    raise ApiError(resp, resp.json())
  File "/usr/lib/python3.4/site-packages/requests/models.py", line 850, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib64/python3.4/json/__init__.py", line 318, in loads
    return _default_decoder.decode(s)
  File "/usr/lib64/python3.4/json/decoder.py", line 343, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib64/python3.4/json/decoder.py", line 361, in raw_decode
    raise ValueError(errmsg("Expecting value", s, err.value)) from None
ValueError: Expecting value: line 1 column 1 (char 0)

Documentation uses argument that Flatpak 0.8.9 doesn't recognize

In this document: https://github.com/flatpak/flat-manager in the README it states;

flatpak-builder --install-deps-from=flathub --repo=local-repo builddir org.gnome.eog.yml

but when I run this command flatpak-builder states "Option parsing failed: Unknown option --install-deps-from=flathub"

Do I need a newer flatpak-builder?

Investigate replacing ostree custom deserialization code with zvariant

Currently, in ostree.rs there is manual code to deserialize the GVariant format used by OSTree which could ideally be replaced with zvariant using the gvariant format.

upstream zbus has currently a test case of a ostree repo to test the gvariant feature. So normally it should work without issues

Generate .flatpakref for all applications and addons

We would like to make addons easily installable too, so having these created by flat-manager would be nice.

I think this area is one of the conditions, that needs change

flat-manager/src/jobs.rs

Line 456 in 7bfca06

let unwanted_exts = [".Debug", ".Locale", ".Sources", ".Docs"];

And maybe

flat-manager/src/jobs.rs

Line 629 in 7bfca06

if build_ref.ref_name.starts_with("app/") || build_ref.ref_name.starts_with("runtime/")

Generate static deltas

I'm using flat-manager to host self-made applications. This works fine, but I noticed downloads can be slow if the application is big.

Digging the docs, it turned out that static deltas are the way to go! And indeed, running

$ flatpak build-update-repo  --gpg-sign=$KEY  --generate-static-deltas /srv/flatpak/myrepo

solves the issue.

While flat-manager actually runs flatpak build-update-repo, no static deltas are generated (=--generate-static-deltas is not set).

What is the recommended way to generate provide deltas? (i.e. for first-time app installations or updating via the flatpak command).

If it is --generate-static-deltas, would you accept a patch to enable the feature per repo on-demand?

support url paths for flat-manager-client?

./flat-manager-client push --commit --publish $(./flat-manager-client create "https://example.com/flatpak/api/" main) ~/test-build/local-repo
Api call to https://example.com/api/v1/build failed with status 404, details: {'status': 404, 'error-type': 'no-error', 'message': 'No json error details from server'}

Better errors for push

Currently some operations are giving internal server error without further context when server runs out of disk space. It would be nice to get this into an error. See eg https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/320974756

flat-manager fails to build on Debian 9.9

Along with some other errors, I think this is the relevant info;

  = note: /usr/bin/ld: cannot find -lpq
          collect2: error: ld returned 1 exit status

It points (after some googling) to the fact that 'ld' cannot find the postgresql development libraries that rust needs to link in to make flat-manager able to read the database. Doing an apt install libpq-dev solves the problem.

I'll send a documentation patch to mitigate this issue but in the meantime I'm documenting this for posterity.

[flat-manager-client] Unexpected TypeError exception in publish

When publishing a build, I get the following error:

$ flat-manager-client publish http://flatpak/api/v1/build/77
Publishing build http://flatpak/api/v1/build/77
Unexpected TypeError exception in publish: the JSON object must be str, not 'dict'
Traceback (most recent call last):
  File "flat-manager-client.1", line 680, in <module>
    "result": args.func(args),
  File "flat-manager-client.1", line 545, in publish_command
    job = publish_build(args.build_url, args.wait, args.token)
  File "flat-manager-client.1", line 374, in publish_build
    reparse_job_results(job)
  File "flat-manager-client.1", line 260, in reparse_job_results
    job["results"] = json.loads(job.get("results", "{}"))
  File "/usr/lib/python3.5/json/__init__.py", line 312, in loads
    s.__class__.__name__))

I get it all the time, since I set up the server. The publishing, however, is successful, so I ignored the error 'till today. Nevertheless, I'm wondering if I'm the only one to get the exception?

Adding a print(job) before the crash, I get the following:

{
   'location':'http://flatpak/api/v1/build/77/publish',
   'log':'',
   'repo':'stable',
   'status':0,
   'start_after':None,
   'contents':'{"build":77}',
   'kind':1,
   'results':{},
   'id':124
}

So it would seem, that "results" is already valid json?

Podman/Docker compose support for flat-manager

I noticed that there is a Dockerfile, but the README describes how to install postgres to the host...

I think there should be an example.container-compose.yml file at least, since its best practice to run databases in a container.

I made an example already however I'm having issues with it.

version: "3.8"
networks:
  flat:

services:
  db:
    image: postgres
    ports:
     - "${POSTGRES_PORTS}"
    volumes:
      - ./db:/data
    env_file:
      - .env
    networks:
      - flat
    environment:
      POSTGRES_PASSWORD: "${POSTGRES_PASSWORD}"
      POSTGRES_DB: "${POSTGRES_DB}"

  backend:
    build: .
    ports:
      -  "${REPO_PORTS}"
    command: ["/usr/local/bin/flat-manager"]
    env_file:
      - .env
    depends_on:
      - db
      # restart: always
    networks:
      - flat
    volumes:
        - ./config.json:/config.json

# We want this for the diesel CLI app
DATABASE_URL=postgres://postgres:PSWD@db:5432/repo
# config file location
REPO_CONFIG=/config.json

POSTGRES_PASSWORD=PSWD
POSTGRES_PORTS=5432:5432
POSTGRES_DB="repo"




REPO_PORTS=8082:8082

Support subsets in the storefront info

Flatpak allows us to add commits to a subset, which allows clients to broadly filter the repository: flatpak/flatpak#3955. For example, in Flathub, we want to have a "FOSS-only" repository--it's the same OSTree repo, but a separate summary file that only contains commits marked as free software.

I'd like to implement this in a generic way by adding a subsets array field to the storefront info. flat-manager will add the app's commits to all of the given subsets.

[Feature Request] use rsync to host flatpak repository

A link to flatpak/flatpak#4383

I am a user from China and I am affected by flathub/flathub#813, as GFW in China completely censored Fastly CDN.
Currently flatpak repo are only intended to used by a CDN, but it bascially restricted the mirroring method to becoming a reverse proxy, which is proved very toxic to mirror sites: for tuna and sjtug

Rsync rationale

I don't know flatpak very well, but I think flatpak is a repo manager for an ostree repo, which itself seems to be easy for rsync to mirror: https://github.com/ostreedev/ostree-releng-scripts/blob/master/rsync-repos.
I suggest flatpak should support an additional remote storage backend using rsync in parallel with the current CDN-like approach.

Rsync advantages

• Commonly accepted protocol for mirroring
• Incremental updates
• Packed transfering of files
• Safe resume from failed sync
• Support both unauthorized client and authorized client

Rsync problems

• Pressure on server resource
• Still more costly than plain deb/rpm packages (but better than making large number of small http requests)

CDN advantage

• Doesn't require a strong server
• Approachable to many (but not more than mirror sites since CDN providers are not approachable everywhere) areas using a relatively low price

CDN-only problems

• Effectively limits mirror methods to reverse proxy only, which is unacceptable for some mirror sites and irritating for the rest
• Too many delta file increases the risk of failure because of increased number of http request round trip
• No safe and incremental way of recovering from failed download

These are not flatpak-specific issues; many FOSS projects are affected by them.

Many mirror sites can support rsync in addition to plain http; for these sites rsync are a proved working way of mirroring and are way easier to manage. If remotes like flathub doesn't have much server resource, they can limit the bandwidth to unauthorized clients and reserve more resource to mirror sites.

Better support for build tokens

As part of the Flathub donations & payments project, we would like to make it easier for developers to use build tokens to upload their apps. Currently, build tokens are given out on an ad-hoc basis, but we want to have a self-service UI accessible to all developers.

In order to do this, some changes are needed to the way builds and build tokens work in flat-manager. Most significantly, builds are currently not associated with an app. Build tokens can only be scoped to "all builds" or "a specific build by ID", and separately can be scoped to "build"/"upload"/"publish". This means we can't create tokens that allow developers to create builds for their app, then manage exactly that set of builds.

To fix this, I'd like to associate each build with an app prefix. There will be an endpoint to query builds by app, and tokens will have a field to restrict their scope to one or more app prefixes.

Another issue is that there is no authentication on downloading from build repos. This needs to be changed so that builds can be marked as public, and if they are not public, a build token is required to access them.

Delta generator does not support SSL

Setting up the elementary flatpak repository with flat-manager I came across an error with publishing.

Our Setup

We have a simple server running flat-manager. It is setup with very similar ansible playbooks. We then have delta generator running on another server also setup from the ansible playbooks.

The flatpak repository itself is published with nginx with regular http and https enabled (both work, no redirects).

The relevant flat-manager config looks like this:

{
  "repos": {
    "elementary": {
      "path": "/mnt/flatpak_elementary_io/repository/repo",
      "base-url": "https://flatpak.elementary.io/repo",
      "collection-id": "io.elementary",
      "suggested-repo-name": "elementary",
      "runtime-repo-url": "https://flatpak.elementary.io/repo.flatpakrepo",
      "subsets": {},
      "deltas": [
        { "id": ["*.Platform", "*.Sdk", "*.Platform.Compat.*" ], "depth": 3},
        { "id": ["*.Sdk.Locale", "*.Platform.Locale"], "depth": 2},
        { "id": ["*.Locale"], "depth": 0},
        { "id": ["*.Sdk.Debug", "*.Platform.Compat.Debug"], "depth": 2},
        { "id": ["*.Debug"], "arch": ["x86_64"], "depth": 2},
        { "id": ["*.Debug"], "depth": 0},
        { "id": ["*.Sources"], "depth": 0},
        { "id": ["*"], "arch": ["x86_64"], "depth": 2},
        { "id": ["*"], "depth": 1}
      ],
      "appstream-delta-depth": 5
    }
  }
}

Console log

When flat-manager runs a publish job, it gets assigned to the delta generator worker successfully, but then errors out pretty fast with this:

Apr 27 05:23:54 flatpak-delta delta-generator-client[34803]:  WARN 2021-04-27T05:23:54Z: flatmanager::ostree: Pull error, retrying commit 0c8541999d2800ce48fb5579bfcc6f92e60da12f55e20a4bf5e40c5517913635: Command ostree pull exited unsucessfully with stderr: error: Server returned status 403: Forbidden
Apr 27 05:23:54 flatpak-delta delta-generator-client[34803]:  INFO 2021-04-27T05:23:54Z: flatmanager::ostree: Pulling commit 0c8541999d2800ce48fb5579bfcc6f92e60da12f55e20a4bf5e40c5517913635
Apr 27 05:23:54 flatpak-delta ostree[35688]: libostree HTTP error from remote upstream for <https://flatpak.elementary.io/repo/objects/7f/bd8d6aed46d8ef91f40a881c37e92e11f89526d0df1f7ab2a47f1e7db2366f.filez>: Server returned status 403: Forbidden
Apr 27 05:23:54 flatpak-delta ostree[35688]: libostree HTTP error from remote upstream for <https://flatpak.elementary.io/repo/objects/3e/36eb4b4105a5f323a1a913f5deaa066ef5cf47239958aa955b153e7a065e2c.filez>: Server returned status 403: Forbidden
Apr 27 05:23:54 flatpak-delta ostree[35688]: libostree HTTP error from remote upstream for <https://flatpak.elementary.io/repo/objects/1f/330e24596aa43c10d6a388da8674d78724e391026e8e8db56153974d5b8fa1.filez>: Server returned status 403: Forbidden
Apr 27 05:23:54 flatpak-delta ostree[35688]: libostree HTTP error from remote upstream for <https://flatpak.elementary.io/repo/objects/67/c7f1829adec5cc1dc0e95be1441b53034f643a15913b9c8a58d13a6704587c.filez>: Server returned status 403: Forbidden
Apr 27 05:23:54 flatpak-delta ostree[35688]: libostree HTTP error from remote upstream for <https://flatpak.elementary.io/repo/objects/db/3bacca2f45754090bb7fb4485edf9cdef0479773f687a5d403655d3d959b88.filez>: Server returned status 403: Forbidden
Apr 27 05:23:54 flatpak-delta ostree[35688]: libostree HTTP error from remote upstream for <https://flatpak.elementary.io/repo/objects/cf/8a7355154f5a9655a94efc443a5c5375346fd264b5b1204d97ff4b152f1a2a.filez>: Server returned status 403: Forbidden

All of those files do exist, and from what I can tell, it seems the http client used for delta generator is trying to access https://flatpak.elementary.io without ssl configured / enabled, and therefor getting an error.

As a sanity check, this is from the delta-generator server:

root@flatpak-delta:/home/btkostner# curl https://flatpak.elementary.io/repo/config
[core]
repo_version=1
mode=archive-z2
indexed-deltas=true

Show version in status page

Handle pyc files

Pyc files need to be removed and added to gitignore.

flat-manager-client exits with code 0 even if job fails

See https://buildbot.flathub.org/#/builders/2/builds/16576. The publish job failed, but flat-manager exited with code 0. It would make more sense to get a non-zero exit code if the job fails.

flat-manager-client should reattach to an in-progress job rather than failing

After being killed (for instance because of #2) running flat-manager-client with --wait complains the the build is already committed even though the commit is still in progress.

It would be nice if it could still wait for the running job to finish (and if possible, print its output).

Feature Request: Add IPFS support for both storage and client pulling

Support for IPFS could help both mitigate the burden of storage, and distribution, while also potentially benefiting users that are more remote (by allowing local peers to help share images). An example of IPFS support for OCI images can be found in the nerdctl project.

The expected result would be native backend support for IPFS/IPNS links to images and layers to be served by the http server. The other feature would be for IPFS/IPNS enabled clients to allow for direct passing of links to the client for them to manage locally.

Related issue in the flatpak cli tool repo

Failed to install applications from flat-manager: "GET /repo/summary.idx HTTP/1.1" - 400 68 Flatpak 1.12.7 0.000590

I have followed the commands in README.md, the flat-manager is installed.

When i "upload" the build application:

./flat-manager-client push --commit $(./flat-manager-client create http://192.168.2.151:8280 stable) /home/berg/flatpak/localrepo/

The flat-manager server "accept" the application.

root@netsrv:/home/berg/flatpak/flat-manager/repo# ls
config  extensions  objects  refs  state  tmp

root@netsrv:/home/berg/flatpak/flat-manager/build-repo/4# ls
appstream  config  extensions  objects  org.flatpak.Hello.flatpakref  parent  refs  state  summaries  summary  summary.idx  tmp

Then, i created a remote hub

flatpak remote-add --if-not-exists myflathub http://192.168.2.151:8280/repo

and try to install

flatpak install myflathub org.flatpak.Hello

both client and server complain:

flatpak install myflathub org.flatpak.Hello
Looking for matches…
error: Unable to load summary from remote myflathub: Server returned status 400: Bad Request


2024-03-29T02:57:53Z INFO  flatmanager::logger] 192.168.2.151:41358 "GET /repo/summary.idx HTTP/1.1" - 400 68 Flatpak 1.12.7 0.00059

however, when i changed the url of myflathub,

flatpak remote-modify myflathub --url=http://192.168.2.151:8280/build-repo/4

it works...

Now, my question is, what is the URL of my selfhosted repo in flat-manager.
the version 4 in "build-repo/4" will change by "flat-manager-client push..."

--url=http://192.168.2.151:8280/build-repo/4

Json decode error during upload stage

Since switching to flat-manager we have had some builds failed with:

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This happens during upload.

There has also been some failed builds at Flathub.

https://gitlab.com/freedesktop-sdk/freedesktop-sdk/-/jobs/168340169

https://flathub.org/builds/#/builders/20/builds/28

Allow purging refs from repository

Right now there is no nice way to delete an app from a flatpak repository. It would be awesome if there was an endpoint for this and a cli command with flat-manager-client.

Add a license

This projects has currently no license

Generated .flatpakref Title field uses app_id instead of more friendly name

Given that the title field is supposed to be a human readable name for the application it seems a bit strange to use the RDNN when the .flatpakref file is generated. I appreciate the correct information from the appdata file may be difficult to get at this stage though.

This is something I'd be interested in working on if anyone was able to give me any pointers in terms of the best approach.

Support for user-based access control?

Is it planned at some point to be able to control (deny or allow) which applications a user can download?
Obviously this would require implementing a certain level of Login

Allow repo token verification to use public keys

Currently flat-manager only uses the repo secret to verify the JWT used for authenticated requests. The token itself is created by the authorization server. It would be better if flat-manager did not require a secret for validation then since it just increases the risk that the secret could be leaked.

Since JWT supports public key signatures such as RSA and EDCSA, it would be nice if flat-manager supported verification with public keys rather then HMAC secrets. Then the private key could just live with the authorization server. Since jsonwebtoken supports many algorithms, this shouldn't be too hard to add.

Getting a 400 error when using the Github deploy action

Hey there,

I set up a flatpak repo, but I just get a 400 error when trying to deploy via a github action using the actions at https://github.com/bilelmoussaoui/flatpak-github-actions

Api call to redacted/api/v1/build failed with status 400, details: {'status': 400, 'error-type': 'no-error', 'message': 'No json error details from server'}
Error: Failed to publish the build: Error: The process '/usr/bin/flat-manager-client' failed with exit code 1

Mar 21 11:35:43 flatpak flat-manager[172593]: [2022-03-21T10:35:43Z INFO  actix_server::builder] Starting 1 workers
Mar 21 11:35:43 flatpak flat-manager[172593]: [2022-03-21T10:35:43Z INFO  actix_server::builder] Starting server on 0.0.0.0:8080
Mar 21 11:35:43 flatpak flat-manager[172593]: [2022-03-21T10:35:43Z INFO  flatmanager::app] Started http server: 0.0.0.0:8080
Mar 21 11:45:15 flatpak flat-manager[172593]: [2022-03-21T10:45:15Z INFO  flatmanager::logger] 52.190.196.115 "POST /api/v1/build HTTP/1.1" github 400 24 Python/3.10 aiohttp/3.7.4 0.004536

Attaching via strace didn't reveal more information. What am I doing wrong? This is the relevant action code:

      - uses: actions/checkout@v2
      - uses: bilelmoussaoui/flatpak-github-actions/flatpak-builder@v4
        name: "Build"
        with:
          bundle: recordwatcher.flatpak
          manifest-path: de.termitel.recordwatcher.json
          cache-key: flatpak-builder-tt-record-watcher-${{ github.sha }}

      - uses: bilelmoussaoui/flatpak-github-actions/flat-manager@v3
        name: "Deploy"
        with:
          repository: termitel
          flat-manager-url: redacted
          token: ${{ secrets.FLATPAK_KEY }}

This is the config.json:

{
  "repos": {
    "stable": {
      "path": "repo",
      "collection-id": "de.termitel.Stable",
      "suggested-repo-name": "Termitel",
      "runtime-repo-url": "redacted/repo/flathub.flatpakrepo",
      "gpg-key": null,
      "base-url": null,
      "subsets": {
        "all": {
          "collection-id": "de.termitel.Stable",
          "base-url": null
        }
      }
    }
  },
  "host": "0.0.0.0",
  "port": 8080,
  "delay-update-secs": 10,
  "database-url": "postgres://%2Fvar%2Frun%2Fpostgresql/repo",
  "build-repo-base": "/home/flatpak/build-repo",
  "build-gpg-key": null,
  "gpg-homedir": null,
  "secret": "redacted"
}

I compiled flat-manager from HEAD on a Ubuntu box. Deploying manually via flat-manager-client works:

$ ./flat-manager-client push --commit $(./flat-manager-client create redacted stable) RecordWatcher/termitel
Uploading refs to redacted/api/v1/build/9: ['app/de.termitel.recordwatcher/aarch64/master', 'runtime/de.termitel.recordwatcher.Debug/aarch64/master']
Refs contain 33 metadata objects
Remote missing 33 of those
Has 16 file objects for those
Remote missing 16 of those
Uploading file objects
Uploading 1 files (214 bytes)
Uploading 1 files (39982483 bytes)
Uploading 3 files (1731 bytes)
Uploading 1 files (4946843 bytes)
Uploading 10 files (11266 bytes)
Uploading metadata objects
Uploading 33 files (4885 bytes)
Uploading deltas
Creating ref app/de.termitel.recordwatcher/aarch64/master with commit b308577a76134cf0d826ded174982d8e04b03569ef7a95b99b865da57a26ffb5
Creating ref runtime/de.termitel.recordwatcher.Debug/aarch64/master with commit ac2dfde8e35f9233eb0e29786cca79aae4be61e597bed000372964c7fb6be8c6
Committing build redacted/api/v1/build/9

What am I doing wrong?

Best regards,
CK

Question: Does the flat-manager-client have a size limitations?

Dear developers,

The flatpak-image of the texlive distribution org.freedesktop.Sdk.Extension.texlive seems to fail not in the building phase but in the uploading phase, see also Log. The flatpak package should be pretty big (download size around 1.5 GB, installation size around 3.2 GB).
So could this be a problem for the flat-manager?

Regards

Jakob

Information on access to flatpak apps over HTTP

Hi team, I have no clue till now how flatpak works nor how flatpak apps are installed? More interested in making a client hub for apps but could someone tell me how to access this apps? Just an overview would be fine

GPG not working in docker image

The official dockerfile uses ubuntu:22.04 as a base, which only has the command gpg and not gpg2.
load_gpg_key uses the gpg2 command and therefore can't load a gpg key with that setup.
You get a generic "Failed to read config file" error message once you try to add a build-gpg-key (weirdly gpg-key on repos doesn't throw an error).

I symlinked it to fix the issue for me (ln -s /usr/bin/gpg /usr/bin/gpg2), but it should get fixed in the dockerfile.

Side note:
The ubuntu image uses LC_ALL=C, which can make gpg throw a "A locale function failed" error if the key includes non-ASCII characters (for example in the uid name). Exporting keys still works, so it's not a problem.