apertium / packaging Goto Github PK

All languages and pairs should execute all installed modes to ensure they at least run.

apertium/apertium-eng-spa#3

Since Apertium has now moved wholly to git, this project needs an overhaul to support git better.

Hello apertiumers,
I tried with dnf install apertium-all-dev, but it shows the error with "Unable to find a match: apertium-all-dev"

apertium-all-dev should be separated into its own package so that we don't have to build all of apertium just to update dependency versions.

Currently each package is built in turn, by a single host. Need to rewrite build system to be master-slave, which means turning the whole current way of doing things on its head.

Master should merely prepare source packages and react to build results. This also means each target must be decoupled - a failure for Debian Sid should not prevent successful builds on Ubuntu Focal from being published.

Why not just use OBS? Because OBS is RPM-centric and reconstructs the whole build environment every build, which is just not needed. One could probably customize OBS for our needs, but that would be equal amount of work. I actually also want to move our RPM builds from OBS to our own service, because OBS doesn't do what I want.

But I fully concede that what I want to do is reimplement OBS, just for our needs. OBS's fundamental architecture is exactly what we want.

Why not just use Github/Travis? Some of the builds require 8+ GiB RAM and/or takes over an hour. We would quickly run into limits.

Currently -Zgzip is forced upon everything, because there was a time where newer distros defaulted to building xz debs that older distros could not use. Every distro now supports xz.

Need to check if 3rd party tools expect data.tar.gz to come out of the .deb files, fix them if so, and then switch to xz.

Installing aperture using the instructions in the wiki (https://wiki.apertium.org/wiki/Install_Apertium_core_using_packaging) does not work:

$ sudo dnf install apertium-all-devel
Viimeisin metatiedon vanhenemistarkistus: 0:00:21 sitten, to 12. elokuuta 2021 12.45.17.
Virhe: 
 Ongelma: ristiriitaiset pyynnöt
  - 'nothing provides' apertium-regtest >= 0.0.1 apertium-all-dev-3.7.0.g4.5c9ba1e0-3.1.noarch:n tarvitsemana
(yritä lisätä '--skip-broken' ohittaaksesi asennuskelvottomat paketit)

The package apertium-regtest is missing.

The packages apertium-apy depends on python3-tornado (>= 3.1).

Current versions in Debian testing/sid are

apertium-apy:
 *** 0.10.0+g388~13c407db-1~sid1 9000
       9000 http://apertium.projectjj.com/apt/nightly sid/main amd64 Packages
python3-tornado:
     5.0.0-2 700
        700 http://ftp.debian.org/debian unstable/main amd64 Packages
 *** 5.0.0-1 800
        800 http://ftp.debian.org/debian testing/main amd64 Packages

Trying to "systemctl start apertium-apy" currently causes

systemd[1]: Started Translation server and API for Apertium.
python3[1663]: [W 180323 23:46:52 servlet:1466] Unable to import CLD2, continuing using naive method of language detection
python3[1663]: [I 180323 23:46:52 servlet:1316] 135 pair modes found
[...]
python3[1663]: [I 180323 23:46:52 servlet:1528] Serving at http://localhost:2737
python3[1663]: [I 180323 23:46:52 servlet:1540] Initialised systemd watchdog, pinging every 5000.0s
python3[1663]: Traceback (most recent call last):
python3[1663]:   File "servlet.py", line 1541, in <module>
python3[1663]:     tornado.ioloop.PeriodicCallback(wd.watchdog_ping, 1000 * wd.period, loop).start()
python3[1663]: TypeError: __init__() takes 3 positional arguments but 4 were given
systemd[1]: apertium-apy.service: Main process exited, code=exited, status=1/FAILURE

The help page for PeriodicCallback.init() states

 |  .. versionchanged:: 5.0
 |     The ``io_loop`` argument (deprecated since version 4.1) has been removed.

which I guess explains the issue.
This is probably more of a bug with APy itself, but I guess as long as the Debian packages don't depend strictly on <=4.1, it should be patched here (or else made incompatible with >4.1, temporarily breaking the packages for testing/unstable users).

As per hfst/hfst#507, add export LD_PRELOAD=tcmalloc.so for data-only builds.

Updated previously:

hfst 3.16.0
hfst/hfst@3a99b73
https://github.com/hfst/hfst/releases/tag/v3.16.0
https://salsa.debian.org/science-team/hfst

foma 0.10.0
mhulden/foma@6184b5a
https://apertium.projectjj.com/apt/release/source/foma/
https://salsa.debian.org/science-team/foma

New releases below:

Builds for https://hub.docker.com/_/amazonlinux/ both 1 and 2

Linked with #12

In order to have fully automatic releases from git tags, we need the dependencies explicitly encoded in the tagged releases somehow. Possible heuristics where things released within the same 24 hours counts as bundled.

https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap
https://docs.brew.sh/Taps

The giella-* packages need to be split so one can get spellers separately from other linguistic tools.

At time of writing:

• MacPorts: https://trac.macports.org/wiki/howto/buildUniversal exists, but many ports don't build for arm64 yet.
• Homebrew: https://discourse.brew.sh/t/timeline-for-homebrew-on-arm-macs/8191/11 - not supported yet either.

But if it becomes an issue, one can manually build everything (automake, CMake).

Ubuntu 23.10 was released in early October with the codename mantic.

Needs to be added to the packaging system.

We've reached the point where an update to a single language may break multiple translation pairs, so will have to bundle matching versions.

Currently, each build checks if any dependencies can be updated, and updates if so. Unfortunately, each such update leaves behind an image in history that will never be used again. The current workaround is to detect when there's too much history and rebuild the image from scratch when needed.

Instead, create a base image on the first build and use multi-stage builds to update. This is the best balance between no history and full history - it would allow similar builds to reuse the cache to get to the base image, but not keep around the useless pre-update ones.

• https://github.com/apertium/apertium/milestone/1
• apertium/apertium#4
• apertium/apertium#16
• apertium/apertium#36

The CentOS / RHEL replacement: https://rockylinux.org/

Create https://github.com/Microsoft/vcpkg packages for the libraries

https://github.com/goavki/apertium-apy uses submodules, which the Github Subversion bridge do not handle. Add support for this, probably via a combination of svn + git.

Don't retry builds that failed last time if the tarball is identical, unless some other reason applies.

I'm looking to the page https://phabricator.wikimedia.org/diffusion/GCXS/browse/master/config/Apertium.yaml where there is the list of translators used by Wikimedia's Content Translation. As far as I see, there is one (and only one) recent Apertium translators, not listed in issue #18, that is missing: Polish → Silesian (pol-szl). Is there a problem in packaging?

OS details:

NAME="Kali GNU/Linux"
ID=kali
VERSION="2020.4"
VERSION_ID="2020.4"
VERSION_CODENAME="kali-rolling"
ID_LIKE=debian
When using apt-install-nightly.sh to install apertium core. My kali linux is detected as Xenial[You can also find in attached screenshot]. Hence getting dependency errors

Workaround

replaced "deb http://apertium.projectjj.com/apt/nightly xenial main" with "deb http://apertium.projectjj.com/apt/nightly bullseye main" in apertium.list file and it worked 😁 😁

$ mkdir foo
$ cd foo
$ wget https://apertium.projectjj.com/apt/nightly/pool/main/l/lttoolbox/lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb
--2020-02-10 09:30:36--  https://apertium.projectjj.com/apt/nightly/pool/main/l/lttoolbox/lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb
Resolving apertium.projectjj.com (apertium.projectjj.com)... 192.99.34.59, 2607:5300:60:493b::1
Koplar til apertium.projectjj.com (apertium.projectjj.com)|192.99.34.59|:443... tilkopla.
HTTP request sent, awaiting response... 200 OK
Length: 17028 (17K) [application/x-debian-package]
Saving to: ‘lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb’

lttoolbox_3.5.1+g463~f32d294a-1~e 100%[=============================================================>]  16,63K  80,7KB/s    in 0,2s

2020-02-10 09:30:37 (80,7 KB/s) - ‘lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb’ saved [17028/17028]

$ dpkg-deb -X lttoolbox_3.5.1+g463~f32d294a-1~eoan1_amd64.deb .
./
./usr/
./usr/bin/
./usr/bin/lt-proc
./usr/bin/lt-tmxcomp
./usr/bin/lt-tmxproc
./usr/share/
./usr/share/doc/
./usr/share/doc/lttoolbox/
./usr/share/doc/lttoolbox/AUTHORS
./usr/share/doc/lttoolbox/NEWS.gz
./usr/share/doc/lttoolbox/README
./usr/share/doc/lttoolbox/changelog.Debian.gz
./usr/share/doc/lttoolbox/copyright
./usr/share/lintian/
./usr/share/lintian/overrides/
./usr/share/lintian/overrides/lttoolbox
./usr/share/man/
./usr/share/man/man1/
./usr/share/man/man1/lt-proc.1.gz
./usr/share/man/man1/lt-tmxcomp.1.gz
./usr/share/man/man1/lt-tmxproc.1.gz
$ zcat ./usr/share/man/man1/*.gz
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.
This system has been minimized by removing packages and content that are
not required on a system that users do not log into.

To restore this content, including manpages, you can run the 'unminimize'
command. You will still need to ensure the 'man-db' package is installed.
$

expected actual man-pages in that .deb

Some things that use APIs that are not reflected by MinGW have to be built on native Windows (e.g. spellers). This should be easily scriptable with WSL or Cygwin.

Similarly, odd bugs crop up for macOS cross-builds, so make them native.

Hello,

When installing in Ubuntu Jammy following the instruccions in this page https://wiki.apertium.org/wiki/Install_Apertium_core_using_packaging, I get the following error:

curl -sS https://apertium.projectjj.com/apt/install-nightly.sh | sudo bash
Cleaning up old install, if any...
s’ha eliminat '/etc/apt/trusted.gpg.d/apertium.gpg'
s’ha eliminat '/etc/apt/preferences.d/apertium.pref'
s’ha eliminat '/etc/apt/sources.list.d/apertium.list'
Determining Debian/Ubuntu codename...
Found evidence of impish...
Found evidence of jammy...
Settling for jammy - enabling the Apertium nightly repo...
Installing Apertium GnuPG key to /etc/apt/trusted.gpg.d/apertium.gpg
Installing package override to /etc/apt/preferences.d/apertium.pref
Creating /etc/apt/sources.list.d/apertium.list
Running apt-get update...
All done - enjoy the packages! If you just want all core tools, do: sudo apt-get install apertium-all-dev
jordi@jordi-SVF1531B4E:~/sc/apertium$ sudo apt-get install apertium-all-dev
S'està llegint la llista de paquets… Fet
S'està construint l'arbre de dependències… Fet
S'està llegint la informació de l'estat… Fet
No s'han pogut instal·lar alguns paquets. És possible que hàgiu
demanat una situació impossible o que estigueu utilitzant la
distribució «unstable» i alguns paquets requerits encara no s'hagin
creat o bé encara no els hagin introduït des d'«Incoming».
La informació següent pot ajudar-vos a resoldre la situació:

Els següents paquets tenen dependències sense satisfer:
apertium-all-dev : Depèn: apertium-anaphora (>= 1.1.0) però s'instal·larà 1.0.2+g112503c54cc-1groovy1
Depèn: apertium-dev (>= 3.8.1) però no s'instal·larà
Depèn: apertium-eval-translator però no és instal·lable
Depèn: apertium-get (>= 1.0.0) però no s'instal·larà
Depèn: apertium-lex-tools-dev (>= 0.3.1) però no s'instal·larà
Depèn: apertium-recursive (>= 1.1.0) però no s'instal·larà
Depèn: apertium-regtest (>= 0.9.1) però no és instal·lable
Depèn: apertium-separable (>= 0.5.0) però s'instal·larà 0.3.6+g2545b130bd6-1groovy1
Depèn: lexd (>= 1.1.0) però no s'instal·larà
Depèn: lttoolbox-dev (>= 3.6.1) però s'instal·larà 3.5.3+g499e6702cca-1groovy1
E: No es poden corregir els problemes, teniu paquets retinguts que estan trencats.

Let me know if you need any other detail

Excludes should work like rsync so that we can include single files and folders, but exclude everything else in the tree.

I tried to upgrade my manual installation to ubuntu packaging as instructed in README:

$ wget -qO- https://apertium.projectjj.com/apt/install-nightly.sh | bash
$ apt-get install apertium-apy

The resulting server doesn't seem to be functional:

flammie@flammie:~$ sudo systemctl start apertium-apy
flammie@flammie:~$ sudo systemctl status apertium-apy
● apertium-apy.service - Translation server and API for Apertium
   Loaded: loaded (/lib/systemd/system/apertium-apy.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since pe 2019-08-16 08:53:43 EDT; 5s ago
     Docs: http://wiki.apertium.org/wiki/Apertium-apy
  Process: 28320 ExecStart=/usr/bin/apertium-apy /usr/share/apertium/modes (code=exited, status=1/FAILURE)
 Main PID: 28320 (code=exited, status=1/FAILURE)

elo 16 08:53:42 flammie.name systemd[1]: apertium-apy.service: Unit entered failed state.
elo 16 08:53:42 flammie.name systemd[1]: apertium-apy.service: Failed with result 'exit-code'.
elo 16 08:53:43 flammie.name systemd[1]: apertium-apy.service: Service hold-off time over, scheduling restart.
elo 16 08:53:43 flammie.name systemd[1]: Stopped Translation server and API for Apertium.
elo 16 08:53:43 flammie.name systemd[1]: apertium-apy.service: Start request repeated too quickly.
elo 16 08:53:43 flammie.name systemd[1]: Failed to start Translation server and API for Apertium.

The problem seems to be with the locale but I think the startup scripts should handle setting the correct locale?

It seems that the startup script tries to set C.UTF8 locale but somehow doesn't work, using /etc/default/apertium-apy fixed it:

flammie@flammie:~$ cat /etc/default/apertium-apy 
LC_ALL=C.utf8

A big chunk of the build time is actually installing build-dependency packages. It is possible to share the chroots or Docker images on a per set of build-dependencies basis, so do that.

Automatically generate and maintain Gentoo ebuild files for all packages.

Guaranteed usable source tarballs are in the respective Apt repos:

• https://apertium.projectjj.com/apt/nightly/source/
• https://apertium.projectjj.com/apt/release/source/

For language and pair packages, provide both source and binary ebuilds.

While trying to install from the nightly Debian / Ubuntu repository, I came across this error:

Err:1 https://apertium.projectjj.com/apt/nightly jammy InRelease
  Certificate verification failed: The certificate is NOT trusted. The certificate issuer is unknown.  Could not handshake: Error in the certificate verification. [IP: 144.217.254.26 443]

The contents of my apertium.list in /etc/apt/sources.list.d/ is deb [arch=amd64 signed-by=/usr/share/keyrings/apertium-packaging.public.gpg] https://apertium.projectjj.com/apt/nightly jammy main

I understand this isn't top priority, since the hashes are still verified, but the SSL certificate might have expired or is failing somewhere else.

Create https://github.com/microsoft/winget-pkgs packages

• dan
• nno
• nob
• swe
• swe-dan
• swe-nor
• dan-nor
• nno-nob

• ron
• cat
• ita
• por
• spa
• eng
• oci
• fra
• afr
• nld
• ind
• zlm
• ron-cat
• eng-cat
• ita-cat
• por-cat
• spa-cat
• oci-fra
• fra-cat
• afr-nld
• ind-zlm

For commits where none of the files used in the build change, don't bother triggering a rebuild. Easily accomplished by comparing the checksum of new tarball with existing one, and bailing out if they're the same.

Set up https://developer.android.com/ndk/ builds for the core tools

(SF ref: https://sourceforge.net/p/apertium/tickets/123/)

While it works nicely to send emails to the people directly responsible for breaking a build, a second notification channel should be added. Some way each package can notify either on IRC or a shared email address.

As requested by @hectoralos in the mailing list

• cat
• fra
• por
• fra-cat
• por-cat