johannesjh / req2flatpak Goto Github PK
View Code? Open in Web Editor NEWConvert Python package requirements to Flatpak build manifests
Home Page: https://johannesjh.github.io/req2flatpak/
License: MIT License
Convert Python package requirements to Flatpak build manifests
Home Page: https://johannesjh.github.io/req2flatpak/
License: MIT License
I'm on Fedora. I did:
dnf install poetry
(never used this before, I know its probably better, etc etc but I'm slow to adopt new things)dnf install pre-commit
.pre-commit run --all
and after recent commits on main
, this works:pre-commit run --all
poetry-check.............................................................Passed
poetry-lock..............................................................Passed
Sync with Poetry.........................................................Passed
isort (python)...........................................................Passed
black....................................................................Passed
prettier.................................................................Passed
rstcheck.................................................................Passed
pylama...................................................................Failed
- hook id: pylama
- exit code: 1
Executable `pylama` not found
bandit...................................................................Passed
Nice!
Except pylama
is missing. So I pip install pylama
. This gave me 8.4.1.
Try again:
pre-commit run --all
poetry-check.............................................................Passed
poetry-lock..............................................................Passed
Sync with Poetry.........................................................Passed
isort (python)...........................................................Passed
black....................................................................Passed
prettier.................................................................Passed
rstcheck.................................................................Passed
pylama...................................................................Failed
- hook id: pylama
- exit code: 1
Traceback (most recent call last):
File "/home/cbm/.local/bin/pylama", line 8, in <module>
sys.exit(shell())
^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/main.py", line 115, in shell
errors = check_paths(
^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/main.py", line 68, in check_paths
errors += run(path=path, code=code, rootdir=rootdir, options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/core.py", line 36, in run
linter.run_check(ctx)
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/lint/pylama_pydocstyle.py", line 33, in run_check
for err in PyDocChecker().check_source(
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/checker.py", line 158, in check_source
error = this_check(
^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/checker.py", line 527, in check_imperative_mood
and not function.is_property(self.property_decorators)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/parser.py", line 223, in is_property
return any(
^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/parser.py", line 224, in <genexpr>
decorator.name in property_decorator_names
TypeError: argument of type 'bool' is not iterable
Traceback (most recent call last):
File "/home/cbm/.local/bin/pylama", line 8, in <module>
sys.exit(shell())
^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/main.py", line 115, in shell
errors = check_paths(
^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/main.py", line 68, in check_paths
errors += run(path=path, code=code, rootdir=rootdir, options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/core.py", line 36, in run
linter.run_check(ctx)
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/lint/pylama_pydocstyle.py", line 33, in run_check
for err in PyDocChecker().check_source(
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/checker.py", line 158, in check_source
error = this_check(
^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/checker.py", line 527, in check_imperative_mood
and not function.is_property(self.property_decorators)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/parser.py", line 223, in is_property
return any(
^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/parser.py", line 224, in <genexpr>
decorator.name in property_decorator_names
TypeError: argument of type 'bool' is not iterable
bandit...................................................................Passed
I'm not sure what pylama
is but I tried the running it by itself without the pre-commit
"stuff", and I have a similar error:
$ pylama --version
pylama 8.4.1
$ pylama
Traceback (most recent call last):
File "/home/cbm/.local/bin/pylama", line 8, in <module>
sys.exit(shell())
^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/main.py", line 115, in shell
errors = check_paths(
^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/main.py", line 68, in check_paths
errors += run(path=path, code=code, rootdir=rootdir, options=options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/core.py", line 36, in run
linter.run_check(ctx)
File "/home/cbm/.local/lib/python3.11/site-packages/pylama/lint/pylama_pydocstyle.py", line 33, in run_check
for err in PyDocChecker().check_source(
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/checker.py", line 158, in check_source
error = this_check(
^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/checker.py", line 527, in check_imperative_mood
and not function.is_property(self.property_decorators)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/parser.py", line 223, in is_property
return any(
^^^^
File "/home/cbm/.local/lib/python3.11/site-packages/pydocstyle/parser.py", line 224, in <genexpr>
decorator.name in property_decorator_names
TypeError: argument of type 'bool' is not iterable
There have been many improvements during the past few months, thanks to @cbm755 and @real-yfprojects, which means we could release a new version.
Release Notes.rst
file that we include into the documentation. It would be nice if these release notes could show up in pypi as well. See #54poetry update
, just to keep things up to date. See #55One would get started faster with this script if the format was explained in README.md
.
For example, some other tool does this:
# Generated with flatpak-pip-generator requests-toolbelt==0.10.1 --yaml
name: python3-requests-toolbelt
buildsystem: simple
build-commands:
That's pretty helpful when you review an old file as it includes exactly the command line used to build it.
Even without the full command line, it would still be helpful to note where the file came from
Low priority and I guess I don't have a strong argument for yaml, except that flatpak-pip-generator
has it (also via --yaml
) and I prefer reading yaml over JSON.
(Thanks for this project, works great and will save me a lot of time!)
Build and publish docs in github CI.
This is a follow up ticket from #4 .
Tooling
consider adding a python linter to the pre-commit configuration.
(ideally something that solves problems and does not just complain about them?, if such a linter exists?)
# Generated by /home/cbm/.local/bin/req2flatpak --requirements Pillow==9.4.0 --target-platforms 310-x86_64 310-aarch64 --outfile python3-Pillow.yaml
That /home/cbm/.local/bin/
is correct but not very professional and not really relevant to almost everyone reading the file.
Some nice way to get just the exe name from argv
? or we have to do something like pathlib.Path(argv[...]).name
?
I remember also seeing ./
like:
# Generated by ./req2flatpak --requirements
which is probably the same fix...
Documentation enhancements: The following two documentation enhancements would be nice to have. Both require setting up of some sort of documentation generator:
Tooling: Both enhancements require some sort of documentation generator such as sphinx. The generator should run locally and in github ci. Updates to the master branch should automatically update the documentation. Documentation for tagged commits should also be available. Documentation generators such as sphinx are probably able to do these jobs nicely out of the box... we just need to set things up.
poetry run sphinx-build ...
or tox run -e docs sphinx-build ...
should do the jobSuggested enhancement: Automated tests could be run in a matrix for different architectures and python versions:
Tooling, possible solutions:
test runtime on github ci:
skip_missing_interpreters
option.test runtime for local testing:
skip_missing_interpreters
optionmotivation: the platform-info
command allows commandline users to save platform information to a file. but the commandline interface currently provides no method for using this platform information later on when generating a flatpak build module. (users would have to write their own script to do this).
hence the suggestion to add an additional commandline argument, allowing users to specify a target platform by referencing a .json file.
I used poetry when setting up this project, but I never really used/tested/documented the building and installation of packages.
Hence, the following todos:
pip install git+https://github.com/johannesjh/req2flatpak.git@main
CONTRIBUTING.md
). No need for lengthy descriptions, but simply noting the commands could be useful.Hi!
Thank you @real-yfprojects for your contribution in #3 and #18
Would you like to co-author and co-maintain this project?, you'd be more than welcome!
...then I would add you on github.
thx
Johannes
I'm on Fedora. I did:
dnf install poetry
(never used this before, I know its probably better, etc etc but I'm slow to adopt new things)dnf install pre-commit
.pip install pylama
When I try make test
, I get:
poetry run python3 -m unittest
RuntimeError
The Poetry configuration is invalid:
- Additional properties are not allowed ('group' was unexpected)
at /usr/lib/python3.11/site-packages/poetry/core/factory.py:43 in create_poetry
39│ message = ""
40│ for error in check_result["errors"]:
41│ message += " - {}\n".format(error)
42│
→ 43│ raise RuntimeError("The Poetry configuration is invalid:\n" + message)
44│
45│ # Load package
46│ name = local_config["name"]
47│ version = local_config["version"]
make: *** [Makefile:11: test] Error 1
AFAICT, the actual command line tool doesn't run build_module_as_str
. Nor does pytest
.
(I tried putting an assert False
in there and nothing has triggered it yet).
In #41, I'm adding build_module_as_yaml_str
, which is similarly untested.
I guess the command line tool should run these classmethods intead, except they are designed to return strings and the command-line tool is based on streams.
TLDR: I think we can bump the max version number of manylinux to whatever happens to be the max supported version in the operating system where req2flatpak is run.
In detail:
The problem, failing tests: I added additional testdata by running req2flatpak --platform-info > tests/cp311-x86_64.platforminfo.json
and found that tests.test_platform_factory.RegressionTest.test
fails now. The reason is that my python installation supports newer versions of manylinux, including 2.36 and 2.37, whereas req2flatpak's req2flatpak.PlatformFactory._cp3_linux_tags
method only knows of manylinux up to version 2.35
Background research: According to https://github.com/pypa/manylinux and https://packaging.python.org/en/latest/specifications/platform-compatibility-tags/, the (newer) manylinux versions follow PEP 600, which states that a wheel tagged manylinux_x_y
shall work on any distro based on glibc>=x.y
. (And indeed, running ldd --version
confirms that my system comes with glibc 3.37).
There is really no way for req2flatpak.PlatformFactory._cp3_linux_tags
to guess the glibc version that might be installed on a system. Because, e.g., an old system with an old version of glibc may nonetheless have a new version of python installed, and an old version of python might be installed on a new system; I don't think there is a strict correlation between glibc versions and python versions.
Implementation options, with regard to req2flatpak, the following options come to my mind, in order of increasing complexity:
So, which specific max version number should req2flatpak assume to be supported by a platform? I'd argue to assume a version that is widespread amongst stable linux distributions. It is better for req2flatpak to assume a low version of manylinux and download compatible packages than the other way round. The current debian stable (version 12, "bookworm") ships with glibc v2.36, i.e., manylinux 2.36. I'd say we should use this.
Review code and docs, make wording consistent:
manifest or build module?
The last officially supported python version is 3.7
. Currently the script only supports 3.10
which is a major flaw.
Add test automation:
Example Usage Tests: Tests to ensure that code examples in the documentation keep working.
End-to-End Regression Test: to ensure that the overall program output does not change for given inputs if we mock/control/fixate all other aspects.
Test parameters:
PypiClient.cache
)PyPi Future compatibility Test: To safeguard against changes in pypi's API.
Test steps:
Vendored Functionality Tests: tests to make sure that vendored / re-implemented functionality is kept up to date.
parse_wheel_filename
method should behave in the exact same way as the same function in pypa's packaging module. This test should fail in the unlikely scenario that pypa changes the behavior of the parse_wheel_filename
method.PlatformFactory.from_python_version_and_arch
produces the same system tags as a real system.Unit tests for req2flatpak operations
Not a top priority since the above tests already cover a lot of the functionality.
Follow-up ticket: run tests in a matrix, see #30
Hi @cbm755
I would like to thank you for contributing to this project in your past pull requests.
Would you like to co-author and co-maintain this project?, you'd be more than welcome!
...then I would add you on github as a collaborator.
thx
Johannes
enhancement:
actual behavior: for some reason, if I invoke req2flatpak without any commandline arguments, it waits for stdin input from the commandline.
suggestion for improvement: it would probably be better (less confusing) to display usage help and exit.
bug, probably related:
it seems that req2flatpak also waits for commandline input if invoked with the --requirements
option.
Let's disable the wiki page in the settings of this repo. We don't need it.
Let's also add tags to the repo to improved discoverability. E.g. when searching flatpak generate
or flatpak generate manifest
this repo won't show up.
I tried to build python library that need rust.
req2flatpak --requirements jellyfish==1.0.1 --target-platforms 310-x86_64 310-aarch64 --outfile req.json
Upon adding req.json to the manifest file and run the build process, this error occurred.
========================================================================
Building module python3-package-installation in /home/kevin/Desktop/Flatpak/org.spyder_ide.spyder/.flatpak-builder/build/python3-package-installation-3
========================================================================
Running: pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}" --prefix=${FLATPAK_DEST} --no-build-isolation jellyfish
Using pip 23.3.1 from /usr/lib/python3.11/site-packages/pip (python 3.11)
Looking in links: file:///run/build/python3-package-installation
ERROR: Could not find a version that satisfies the requirement jellyfish (from versions: none)
ERROR: No matching distribution found for jellyfish
req.json file
{
"name": "python3-package-installation",
"buildsystem": "simple",
"build-commands": [
"pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} --no-build-isolation jellyfish"
],
"sources": [
{
"type": "file",
"url": "https://files.pythonhosted.org/packages/20/9b/16744275fb3d47c1982a16d460e33e5f8bc897b570aa2f8cbfba8bd72217/jellyfish-1.0.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
"sha256": "57b2e7c452581450c3550373393714ebcf5c2dc445d5744ec74846648c140aee",
"only-arches": [
"aarch64"
]
},
{
"type": "file",
"url": "https://files.pythonhosted.org/packages/39/c1/186025168b8e4b0d9cd6fdfcd2c44844621c390387b085981af781434b8c/jellyfish-1.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
"sha256": "69f6b5f89383705834877b86100cf5a49c985a200076524b1c24ceb9e25916e5",
"only-arches": [
"x86_64"
]
}
]
}
Package Version
------------------------- ------------
altair 5.0.1
anyio 3.6.2
anymarkup-core 0.8.1
anytree 2.9.0
apipkg 3.0.2
apparmor 3.1.6
appdirs 1.4.4
argcomplete 3.1.2
asteval 0.9.30
asttokens 2.4.1
attrs 23.1.0
Automat 22.10.0
Babel 2.12.1
backcall 0.2.0
beautifulsoup4 4.12.2
Bottleneck 1.3.7
breezy 3.3.4
Brotli 1.1.0
bs4 0.0.1
build 1.0.3
CacheControl 0.13.1
certifi 2023.7.22
cffi 1.16.0
chardet 5.2.0
charset-normalizer 3.2.0
ChemSpiPy 2.0.0
cleo 2.0.1
click 8.1.7
cmdln 2.0.0
coloredlogs 15.0.1
comm 0.1.4
configobj 5.0.8
constantly 15.1.0
contourpy 1.0.7
crashtest 0.4.1
createrepo-c 0.20.1
crit 3.18
cryptography 41.0.4
cssselect 1.2.0
cupshelpers 1.0
cycler 0.11.0
debugpy 1.6.6
decorator 5.1.1
defusedxml 0.7.1
dill 0.3.6
distlib 0.3.7
distro 1.8.0
dnspython 2.4.2
docopt 0.6.2
dulwich 0.21.6
emcee 3.1.4
entrypoints 0.4
et-xmlfile 1.0.1
executing 1.2.0
fastbencode 0.2
fastimport 0.9.14
fastjsonschema 2.18.0
filelock 3.12.3
fonttools 4.39.4
fs 2.4.16
future 0.18.3
Genshi 0.7.7
gevent 23.9.0
gmpy2 2.1.5
gpg 1.23.1
graphviz 0.20.1
greenlet 3.0.0
h11 0.14.0
h2 4.1.0
h5py 3.9.0
hierplace 1.0.0
hpack 4.0.0
html5lib 1.1
httpcore 0.18.0
httpx 0.25.0
humanfriendly 10.0
hyperframe 6.0.1
hyperlink 21.0.0
idna 3.4
importlib-metadata 6.8.0
incremental 22.10.0
iniconfig 2.0.0
installer 0.7.0
ipaddr 2.2.0
ipykernel 6.25.2
ipython 8.16.1
ipywidgets 8.1.1
jaraco.classes 3.2.3
jdcal 1.4.1
jedi 0.19.1
jeepney 0.8.0
Jinja2 3.1.2
jsonschema 4.17.3
jsonschema-specifications 2023.7.1
jupyter_client 8.4.0
jupyter_core 5.4.0
jupyterlab-widgets 3.0.8
keyring 24.2.0
keyring-keyutils 0.1.1
keyrings.alt 0.0.0
kinet2pcb 1.1.0
kinparse 1.2.0
kiwi 9.25.17
kiwi-keg 2.1.1
kiwisolver 1.4.4
LibAppArmor 3.1.6
libarchive-c 4.0
libcomps 0.1.19
llvmlite 0.40.0
lmfit 1.2.2
lmfitxps 1.3.0
lockfile 0.12.2
lxml 4.9.3
markdown-it-py 2.2.0
MarkupSafe 2.1.3
matplotlib 3.6.3
matplotlib-inline 0.1.6
mdurl 0.1.2
mercurial 6.5.2
merge3 0.0.12
more-itertools 9.1.0
mpmath 1.3.0
msgpack 1.0.5
munkres 1.1.4
nest-asyncio 1.5.6
nftables 0.1
notify2 0.3.1
numba 0.57.1
numexpr 2.8.4
numpy 1.25.2
objgraph 3.5.0
odfpy 1.4.2
olefile 0.46
openpyxl 3.0.10
osc 1.4.3
packaging 23.1
pandas 2.1.1
parso 0.8.3
patiencediff 0.2.12
patsy 0.5.3
pexpect 4.8.0
pickleshare 0.7.5
Pillow 10.0.1
pip 23.2.1
pipgrip 0.10.4
pipx 1.2.0
pkginfo 1.7.1
platformdirs 3.5.1
poetry 1.6.1
poetry-core 1.7.0
poetry-plugin-export 1.5.0
progressbar 2.5
prompt-toolkit 3.0.38
protobuf 4.23.4
psutil 5.9.5
ptyprocess 0.7.0
PubChemPy 1.0.4
pure-eval 0.2.2
py 1.11.0
pyasn1 0.5.0
pyasn1-modules 0.3.0
pybind11 2.11.1
pycairo 1.25.1
pycares 4.3.0
pycparser 2.21
pycryptodomex 3.18.0
pycups 2.0.1
pycurl 7.45.2
pyftpdlib 1.5.7
Pygments 2.16.1
PyGObject 3.46.0
pylev 1.4.0
pyOpenSSL 23.2.0
pyparsing 3.1.0
pyproject_hooks 1.0.0
PyQt5 5.15.10
PyQt5-sip 12.13.0
pyrsistent 0.19.3
pysendfile 2.0.1
PySide2 5.15.11
PySocks 1.7.1
python-dateutil 2.8.2
python-linux-procfs 0.7.0
python-xlib 0.33
pytz 2023.3.post1
pyudev 0.24.1
pyxattr 0.8.1
PyYAML 6.0.1
pyzmq 25.1.1
rapidfuzz 2.13.7
referencing 0.30.2
reportlab 4.0.5
req2flatpak 0.2.0
requests 2.31.0
requests-toolbelt 1.0.0
requirements-parser 0.2.0
rich 13.5.2
rpds-py 0.7.1
rpm 4.18.0
schema 0.7.5
scipy 1.11.1
SecretStorage 3.3.3
service-identity 23.1.0
setuptools 68.1.2
shellingham 1.5.0
shiboken2 5.15.11
shiboken2-generator 5.15.11
shiboken6 6.6.0
simplejson 3.19.1
six 1.16.0
sniffio 1.3.0
soupsieve 2.4.1
spec-cleaner 1.2.1
spyder-terminal 1.2.2
stack-data 0.6.2
sympy 1.12
TBB 2021.10.0
terminado 0.17.1
toml 0.10.2
tomli 2.0.1
tomlkit 0.12.1
toolz 0.12.0
tornado 6.3.3
traitlets 5.11.2
trove-classifiers 2023.10.18
Twisted 22.10.0
typing_extensions 4.8.0
uncertainties 3.1.7
unicodedata2 15.0.0
urllib3 2.0.7
userpath 1.9.0
virtualenv 20.24.5
wcwidth 0.2.6
webencodings 0.5.1
wheel 0.41.2
widgetsnbextension 4.0.9
zipp 3.15.0
zope.event 5.0
zope.interface 6.0
zopfli 0.2.3
zstandard 0.21.0
Operating System: openSUSE Tumbleweed 20231103
KDE Plasma Version: 5.27.9
KDE Frameworks Version: 5.111.0
Qt Version: 5.15.11
Kernel Version: 6.5.9-1-default (64-bit)
Graphics Platform: X11
Processors: 12 × AMD Ryzen 5 4600H with Radeon Graphics
Memory: 7.6 GiB of RAM
Graphics Processor: NVIDIA GeForce GTX 1650 Ti/PCIe/SSE2
Manufacturer: LENOVO
Product Name: 82B5
System Version: Lenovo Legion 5 15ARH05
Flatpak-builder: 1.2.3
I noticed these aren't rendered as bullet lists since they are missing trailing whitespace (for some reason github's renderer doesn't care):
req2flatpak/docs/source/development.rst
Lines 29 to 33 in 847406f
Originally posted by @real-yfprojects in #55 (comment)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.