apertium / packaging Goto Github PK
View Code? Open in Web Editor NEWDebian, Fedora, Windows, macOS packaging scripts for Apertium, HFST, CG-3, and related techs.
Home Page: https://wiki.apertium.org/wiki/Packaging
Debian, Fedora, Windows, macOS packaging scripts for Apertium, HFST, CG-3, and related techs.
Home Page: https://wiki.apertium.org/wiki/Packaging
At time of writing:
But if it becomes an issue, one can manually build everything (automake, CMake).
Ubuntu 23.10 was released in early October with the codename mantic
.
Needs to be added to the packaging system.
Currently each package is built in turn, by a single host. Need to rewrite build system to be master-slave, which means turning the whole current way of doing things on its head.
Master should merely prepare source packages and react to build results. This also means each target must be decoupled - a failure for Debian Sid should not prevent successful builds on Ubuntu Focal from being published.
Why not just use OBS? Because OBS is RPM-centric and reconstructs the whole build environment every build, which is just not needed. One could probably customize OBS for our needs, but that would be equal amount of work. I actually also want to move our RPM builds from OBS to our own service, because OBS doesn't do what I want.
But I fully concede that what I want to do is reimplement OBS, just for our needs. OBS's fundamental architecture is exactly what we want.
Why not just use Github/Travis? Some of the builds require 8+ GiB RAM and/or takes over an hour. We would quickly run into limits.
Automatically generate and maintain Gentoo ebuild files for all packages.
Guaranteed usable source tarballs are in the respective Apt repos:
For language and pair packages, provide both source and binary ebuilds.
I'm looking to the page https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/config/Apertium.yaml where there is the list of translators used by Wikimedia's Content Translation. As far as I see, there is one (and only one) recent Apertium translators, not listed in issue #18, that is missing: Polish → Silesian (pol-szl). Is there a problem in packaging?
Excludes should work like rsync so that we can include single files and folders, but exclude everything else in the tree.
NAME="Kali GNU/Linux"
ID=kali
VERSION="2020.4"
VERSION_ID="2020.4"
VERSION_CODENAME="kali-rolling"
ID_LIKE=debian
When using apt-install-nightly.sh to install apertium core. My kali linux is detected as Xenial[You can also find in attached screenshot]. Hence getting dependency errors
replaced "deb http://apertium.projectjj.com/apt/nightly xenial main" with "deb http://apertium.projectjj.com/apt/nightly bullseye main" in apertium.list file and it worked 😁 😁
Don't retry builds that failed last time if the tarball is identical, unless some other reason applies.
While trying to install from the nightly Debian / Ubuntu repository, I came across this error:
Err:1 https://apertium.projectjj.com/apt/nightly jammy InRelease
Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown. Could not handshake: Error in the certificate verification. [IP: 144.217.254.26 443]
The contents of my apertium.list
in /etc/apt/sources.list.d/
is deb [arch=amd64 signed-by=/usr/share/keyrings/apertium-packaging.public.gpg] https://apertium.projectjj.com/apt/nightly jammy main
I understand this isn't top priority, since the hashes are still verified, but the SSL certificate might have expired or is failing somewhere else.
apertium-all-dev
should be separated into its own package so that we don't have to build all of apertium
just to update dependency versions.
$ mkdir foo
$ cd foo
$ wget https://apertium.projectjj.com/apt/nightly/pool/main/l/lttoolbox/lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb
--2020-02-10 09:30:36-- https://apertium.projectjj.com/apt/nightly/pool/main/l/lttoolbox/lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb
Resolving apertium.projectjj.com (apertium.projectjj.com)... 192.99.34.59, 2607:5300:60:493b::1
Koplar til apertium.projectjj.com (apertium.projectjj.com)|192.99.34.59|:443... tilkopla.
HTTP request sent, awaiting response... 200 OK
Length: 17028 (17K) [application/x-debian-package]
Saving to: ‘lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb’
lttoolbox_3.5.1+g463~f32d294a-1~e 100%[=============================================================>] 16,63K 80,7KB/s in 0,2s
2020-02-10 09:30:37 (80,7 KB/s) - ‘lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb’ saved [17028/17028]
$ dpkg-deb -X lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb .
./
./usr/
./usr/bin/
./usr/bin/lt-proc
./usr/bin/lt-tmxcomp
./usr/bin/lt-tmxproc
./usr/share/
./usr/share/doc/
./usr/share/doc/lttoolbox/
./usr/share/doc/lttoolbox/AUTHORS
./usr/share/doc/lttoolbox/NEWS.gz
./usr/share/doc/lttoolbox/README
./usr/share/doc/lttoolbox/changelog.Debian.gz
./usr/share/doc/lttoolbox/copyright
./usr/share/lintian/
./usr/share/lintian/overrides/
./usr/share/lintian/overrides/lttoolbox
./usr/share/man/
./usr/share/man/man1/
./usr/share/man/man1/lt-proc.1.gz
./usr/share/man/man1/lt-tmxcomp.1.gz
./usr/share/man/man1/lt-tmxproc.1.gz
$ zcat ./usr/share/man/man1/*.gz
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.
To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.
To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.
To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.
$
expected actual man-pages in that .deb
Currently -Zgzip
is forced upon everything, because there was a time where newer distros defaulted to building xz debs that older distros could not use. Every distro now supports xz.
Need to check if 3rd party tools expect data.tar.gz to come out of the .deb files, fix them if so, and then switch to xz.
Set up https://developer.android.com/ndk/ builds for the core tools
A big chunk of the build time is actually installing build-dependency packages. It is possible to share the chroots or Docker images on a per set of build-dependencies basis, so do that.
All languages and pairs should execute all installed modes to ensure they at least run.
Builds for https://hub.docker.com/_/amazonlinux/ both 1 and 2
Create https://github.com/microsoft/winget-pkgs packages
As requested by @hectoralos in the mailing list
https://github.com/goavki/apertium-apy uses submodules, which the Github Subversion bridge do not handle. Add support for this, probably via a combination of svn + git.
For commits where none of the files used in the build change, don't bother triggering a rebuild. Easily accomplished by comparing the checksum of new tarball with existing one, and bailing out if they're the same.
Hello apertiumers,
I tried with dnf install apertium-all-dev, but it shows the error with "Unable to find a match: apertium-all-dev"
The CentOS / RHEL replacement: https://rockylinux.org/
Some things that use APIs that are not reflected by MinGW have to be built on native Windows (e.g. spellers). This should be easily scriptable with WSL or Cygwin.
Similarly, odd bugs crop up for macOS cross-builds, so make them native.
Hello,
When installing in Ubuntu Jammy following the instruccions in this page https://wiki.apertium.org/wiki/Install_Apertium_core_using_packaging, I get the following error:
curl -sS https://apertium.projectjj.com/apt/install-nightly.sh | sudo bash
Cleaning up old install, if any...
s’ha eliminat '/etc/apt/trusted.gpg.d/apertium.gpg'
s’ha eliminat '/etc/apt/preferences.d/apertium.pref'
s’ha eliminat '/etc/apt/sources.list.d/apertium.list'
Determining Debian/Ubuntu codename...
Found evidence of impish...
Found evidence of jammy...
Settling for jammy - enabling the Apertium nightly repo...
Installing Apertium GnuPG key to /etc/apt/trusted.gpg.d/apertium.gpg
Installing package override to /etc/apt/preferences.d/apertium.pref
Creating /etc/apt/sources.list.d/apertium.list
Running apt-get update...
All done - enjoy the packages! If you just want all core tools, do: sudo apt-get install apertium-all-dev
jordi@jordi-SVF1531B4E:~/sc/apertium$ sudo apt-get install apertium-all-dev
S'està llegint la llista de paquets… Fet
S'està construint l'arbre de dependències… Fet
S'està llegint la informació de l'estat… Fet
No s'han pogut instal·lar alguns paquets. És possible que hàgiu
demanat una situació impossible o que estigueu utilitzant la
distribució «unstable» i alguns paquets requerits encara no s'hagin
creat o bé encara no els hagin introduït des d'«Incoming».
La informació següent pot ajudar-vos a resoldre la situació:
Els següents paquets tenen dependències sense satisfer:
apertium-all-dev : Depèn: apertium-anaphora (>= 1.1.0) però s'instal·larà 1.0.2+g112503c54cc-1groovy1
Depèn: apertium-dev (>= 3.8.1) però no s'instal·larà
Depèn: apertium-eval-translator però no és instal·lable
Depèn: apertium-get (>= 1.0.0) però no s'instal·larà
Depèn: apertium-lex-tools-dev (>= 0.3.1) però no s'instal·larà
Depèn: apertium-recursive (>= 1.1.0) però no s'instal·larà
Depèn: apertium-regtest (>= 0.9.1) però no és instal·lable
Depèn: apertium-separable (>= 0.5.0) però s'instal·larà 0.3.6+g2545b130bd6-1groovy1
Depèn: lexd (>= 1.1.0) però no s'instal·larà
Depèn: lttoolbox-dev (>= 3.6.1) però s'instal·larà 3.5.3+g499e6702cca-1groovy1
E: No es poden corregir els problemes, teniu paquets retinguts que estan trencats.
Let me know if you need any other detail
I tried to upgrade my manual installation to ubuntu packaging as instructed in README:
$ wget -qO- https://apertium.projectjj.com/apt/install-nightly.sh | bash
$ apt-get install apertium-apy
The resulting server doesn't seem to be functional:
flammie@flammie:~$ sudo systemctl start apertium-apy
flammie@flammie:~$ sudo systemctl status apertium-apy
● apertium-apy.service - Translation server and API for Apertium
Loaded: loaded (/lib/systemd/system/apertium-apy.service; enabled; vendor preset: enabled)
Active: inactive (dead) (Result: exit-code) since pe 2019-08-16 08:53:43 EDT; 5s ago
Docs: http://wiki.apertium.org/wiki/Apertium-apy
Process: 28320 ExecStart=/usr/bin/apertium-apy /usr/share/apertium/modes (code=exited, status=1/FAILURE)
Main PID: 28320 (code=exited, status=1/FAILURE)
elo 16 08:53:42 flammie.name systemd[1]: apertium-apy.service: Unit entered failed state.
elo 16 08:53:42 flammie.name systemd[1]: apertium-apy.service: Failed with result 'exit-code'.
elo 16 08:53:43 flammie.name systemd[1]: apertium-apy.service: Service hold-off time over, scheduling restart.
elo 16 08:53:43 flammie.name systemd[1]: Stopped Translation server and API for Apertium.
elo 16 08:53:43 flammie.name systemd[1]: apertium-apy.service: Start request repeated too quickly.
elo 16 08:53:43 flammie.name systemd[1]: Failed to start Translation server and API for Apertium.
The problem seems to be with the locale but I think the startup scripts should handle setting the correct locale?
It seems that the startup script tries to set C.UTF8 locale but somehow doesn't work, using /etc/default/apertium-apy fixed it:
flammie@flammie:~$ cat /etc/default/apertium-apy
LC_ALL=C.utf8
Installing aperture using the instructions in the wiki (https://wiki.apertium.org/wiki/Install_Apertium_core_using_packaging) does not work:
$ sudo dnf install apertium-all-devel
Viimeisin metatiedon vanhenemistarkistus: 0:00:21 sitten, to 12. elokuuta 2021 12.45.17.
Virhe:
Ongelma: ristiriitaiset pyynnöt
- 'nothing provides' apertium-regtest >= 0.0.1 apertium-all-dev-3.7.0.g4.5c9ba1e0-3.1.noarch:n tarvitsemana
(yritä lisätä '--skip-broken' ohittaaksesi asennuskelvottomat paketit)
The package apertium-regtest
is missing.
As per hfst/hfst#507, add export LD_PRELOAD=tcmalloc.so
for data-only builds.
The giella-*
packages need to be split so one can get spellers separately from other linguistic tools.
Since Apertium has now moved wholly to git, this project needs an overhaul to support git better.
While it works nicely to send emails to the people directly responsible for breaking a build, a second notification channel should be added. Some way each package can notify either on IRC or a shared email address.
The packages apertium-apy depends on python3-tornado (>= 3.1).
Current versions in Debian testing/sid are
apertium-apy:
*** 0.10.0+g388~13c407db-1~sid1 9000
9000 http://apertium.projectjj.com/apt/nightly sid/main amd64 Packages
python3-tornado:
5.0.0-2 700
700 http://ftp.debian.org/debian unstable/main amd64 Packages
*** 5.0.0-1 800
800 http://ftp.debian.org/debian testing/main amd64 Packages
Trying to "systemctl start apertium-apy" currently causes
systemd[1]: Started Translation server and API for Apertium.
python3[1663]: [W 180323 23:46:52 servlet:1466] Unable to import CLD2, continuing using naive method of language detection
python3[1663]: [I 180323 23:46:52 servlet:1316] 135 pair modes found
[...]
python3[1663]: [I 180323 23:46:52 servlet:1528] Serving at http://localhost:2737
python3[1663]: [I 180323 23:46:52 servlet:1540] Initialised systemd watchdog, pinging every 5000.0s
python3[1663]: Traceback (most recent call last):
python3[1663]: File "servlet.py", line 1541, in <module>
python3[1663]: tornado.ioloop.PeriodicCallback(wd.watchdog_ping, 1000 * wd.period, loop).start()
python3[1663]: TypeError: __init__() takes 3 positional arguments but 4 were given
systemd[1]: apertium-apy.service: Main process exited, code=exited, status=1/FAILURE
The help page for PeriodicCallback.init() states
| .. versionchanged:: 5.0
| The ``io_loop`` argument (deprecated since version 4.1) has been removed.
which I guess explains the issue.
This is probably more of a bug with APy itself, but I guess as long as the Debian packages don't depend strictly on <=4.1, it should be patched here (or else made incompatible with >4.1, temporarily breaking the packages for testing/unstable users).
Create https://github.com/Microsoft/vcpkg packages for the libraries
Updated previously:
hfst 3.16.0
hfst/hfst@3a99b73
https://github.com/hfst/hfst/releases/tag/v3.16.0
https://salsa.debian.org/science-team/hfst
foma 0.10.0
mhulden/foma@6184b5a
https://apertium.projectjj.com/apt/release/source/foma/
https://salsa.debian.org/science-team/foma
New releases below:
We've reached the point where an update to a single language may break multiple translation pairs, so will have to bundle matching versions.
Linked with #12
In order to have fully automatic releases from git tags, we need the dependencies explicitly encoded in the tagged releases somehow. Possible heuristics where things released within the same 24 hours counts as bundled.
Currently, each build checks if any dependencies can be updated, and updates if so. Unfortunately, each such update leaves behind an image in history that will never be used again. The current workaround is to detect when there's too much history and rebuild the image from scratch when needed.
Instead, create a base image on the first build and use multi-stage builds to update. This is the best balance between no history and full history - it would allow similar builds to reuse the cache to get to the base image, but not keep around the useless pre-update ones.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.