container-images's People
container-images's Issues
[URGENT] Download server with a load > 3
Something goes wrong and I strongly suspect something is wrong around Matomo import_logs.py
... or something is wrong with kiwix-serve
which is really high on the IO too, maybe a consequence that the library.kiwix.org throttling does not work well?
`robots.txt` has not being migrated in library image and more
In the original Nginx vhost:
location /robots.txt {
alias /var/www/library.kiwix.org/robots.txt;
}
with robots.txt
is
User-agent: *
Crawl-delay: 3
On the top it is also really important to threshold the number of request to library.kiwix.org. Our old Nginx configuration had:
limit_req_zone $binary_remote_addr zone=limit:10m rate=3r/s;
This should be migrated too
Wrong description for wp1-0.8 nopic
The current description is the same as for wp1-0.8: "Wikipedia 45.000 best articles with pictures"
It should be "Wikipedia 45.000 best articles without pictures"
[URGENT] library.kiwix.org SSL certificate wrong
$ curl https://library.kiwix.org/
curl: (60) SSL certificate problem: self signed certificate
More details here: https://curl.haxx.se/docs/sslcerts.html
curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn't adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you'd like to turn off curl's verification of the certificate, use
the -k (or --insecure) option.
And all of this should work automatically, so I don't understand why we have this kind of problem.
Nighlies should be removed periodicaly
Here are the cron entries on the old download server:
0 0 * * * find /var/www/download.kiwix.org/nightly/ -mindepth 1 -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;
0 0 * * * find /var/www/download.openzim.org/nightly/ -mindepth 1 -maxdepth 1 -type d -mtime +30 -exec rm -rf {} \;
How is mirrorbrain database initialized
I do not see anything about that in the Docker images. The SQL is there mirrors-postgresql.sql
but not run anywere looks like?
purpose of /data/vhost.d/ ?
It seems to be some kind of temporary date, but I think it should not be here.
manageContentRepository.pl wiki user/password should not be in the image
They should be container ENV variable
Are the mwoffliner scripts really the scripts used to generate Kiwix files?
Underlying goal: When debugging, I sometimes want to determine the real commands used for a failing build.
I ask because mwmatrixoffliner has a miscapitalized parameter --mwURL, while mwoffliner and these scripts use --mwUrl.
I'm about to change mwmatrixoffliner to match.
But as I understand it, these commands should not even work with mwmatrixoffliner due to the difference. It doesn't work on my command line when I try to use --mwUrl as these scripts do.
Are these the real scripts used?
wiki password is in clear text in docker-compose.yaml
Weaknesses in library-docker
library.kiwix.org.xml
is not generated. This is how it is on the old serveur:
/var/www/kiwix/tools/tools/scripts/manageLibraryKiwixOrg.pl --source=/var/www/download.kiwix.org/library/library_zim.xml > library.kiwix.org.xml
This should be synced one time a day automatically
On the top of that, the kiwix-serve should be run in a while true loop, to avoid the whole service to dies if there is a bug in kiwix-serve causing a crash. Currently we run:
while [ 42 ] ; do kiwix-serve --port=4242 --library --interface=lo --threads=16 --verbose library.kiwix.org.xml ; sleep 1 ; done
Scanning mirrors with mirrorbrain is too long
Should take a few dozens of minutes and it seems to take days.
tmp.kiwix.org cert seems not to be created properly
Same as #4 but for tmp.kiwix.org
Prepare letsencrypt and reverse-proxy to welcome stats.kiwix.org
Unfortunately we can not really host that in the cloud, would be to expensive. So we will have to put it on that box two, that should be able to work without problem. I will create the docker-compose and take care of the migration, but we should also update the other dockers.
HTTP should be a redirect to HTTPS
We need a zimquarantine-docker
ZIM files which are uploaded to download.kiwix.org
should be checked before being made fully available to download. We have a tool called zimcheck
(part of the zim-tools
) aimed to do that checks. This task is about making a Docker container able to use that tool.
This Docker image should:
- Have an input directory (volume)
- Have an output directory (volume)
- Monitor the input directory for ZIM files, check them if
zim-check
and if OK them move them to output directory (keeping the subdirectory hierarchy). - How the system should behave if an error occurs is a bit unclear so far to me: ideally it should inform the zimfarm if possible (transmit the log), but would prefer to keep that Docker image AFAP independent of the zimfarm. Maybe both requirements are mutually exclusive!?!
Remarks:
- We should check that the move operation (if successful is immediate), otherwise the rest of the toolchain might have problem with incomplete ZIM files
This task depends on openzim/zim-tools#26
Visual editor does not work at all on Kiwix welcome page
I don't know if other pages are impacted, but I suspect yes. This is not a problem with the RAM.
Kiwix & openZIM wikis are very slow
Need more RAM :
$ sloppy stats openzim
CONTAINER CPU % MEM / LIMIT MEM % NET I/O Extern NET I/O Intern
openzim/wiki-ad4ada 0.4% 960 MiB / 960 MiB 100.0% 410 B / 2.39 KiB 10.9 B / 0 B
Watcherbot should allow to sync Slack and IRC
In Both directions.
[URGENT] library.kiwix.org is down
http://library.kiwix.org/ gives an HTTP error 502
Deux containers bizarres sur la machine download.kiwix.org
4c260cc527d6 4c7964895535 "/bin/sh -c start.sh" 22 hours ago Up 22 hours 80/tcp friendly_torvalds
ec07f3270e21 4c7964895535 "/bin/sh -c start.sh" 22 hours ago Up 22 hours 80/tcp confident_brown
@florentk Tu sais ce que c'est?
http://warehouse.cardshop.hotspot.kiwix.org/ opens download.openzim.org
If no virtualhost is configured for a domain, it should returns 404.
Create a super-seeder docker image/container
Looks like still not all BitTorrent clients can deal properly with our Web seeds. Having a complete and always running super-seeder would help to solve that problem. We could run it on a mirror (files already there). Additionally this Docker image might be interesting to a few Kiwix supporters who have that way a solution to support the project by easily sharing a bit of there bandwidth.
Using rsync, see https://download.kiwix.org/README, and rtorrent, that should not be too complicated.
Publish an update rate on wiki.kiwix.org
To know if we are good at updating ZIM content, see https://meta.wikimedia.org/wiki/Kiwix/Plan_2018-2019#Targets
`manageContentRepository.pl` is called in none of the docker images
It should run one time a day, here is how the cron looks like on the old server:
30 2 * * * cd /var/www/kiwix/tools/tools/scripts ; ./manageContentRepository.pl --wikiPassword=42 --writeWiki --writeHtaccess --writeLibrary --deleteOutdatedFiles
This should be introduced in the library Dockerfile.
watcherbot does not report about wiki changes
watcherbot complains that it is not able to connect to the openzim/Kiwix wikis.
docker rsync container keeps restarting
# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
89884059fa7f openzim/zimfarm-warehouse "python3 app/main.py" 2 minutes ago Up 2 minutes 0.0.0.0:1522->22/tcp zimfarm-warehouse
0f16e183f6b8 kiwix/cardshop-warehouse "python3 /app/main.py" 3 days ago Up 3 days 28011-28090/tcp, 0.0.0.0:29011-29090->29011-29090/tcp, 0.0.0.0:2121->21/tcp dreamy_pare
dc82bd3f873c kiwix/library "/bin/sh -c start.sh" 8 days ago Up 8 days 80/tcp library
676ada85547a kiwix/mirrorbrain "/bin/sh -c start.sh" 9 days ago Up 9 days 80/tcp mirrorbrain-web
a246c60319ce kiwix/mirrorbrain "/bin/sh -c start.sh" 9 days ago Up 9 days 80/tcp mirrorbrain-update
eb70d10aa0c1 matomo "/entrypoint.sh apac…" 9 days ago Up 9 days 80/tcp matomo_web
2fe7c0af952f mysql:5 "docker-entrypoint.s…" 9 days ago Up 9 days 3306/tcp, 33060/tcp matomo_mysql
044387a66f9d kiwix/reverse-proxy "/app/docker-entrypo…" 9 days ago Up 9 days 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp reverse-proxy
159215c227e8 jrcs/letsencrypt-nginx-proxy-companion "/bin/bash /app/entr…" 9 days ago Up 9 days letsencrypt
0647ef09736f postgres "docker-entrypoint.s…" 12 days ago Up 9 days 5432/tcp mirrorbrain-db
39a8ec87f636 kiwix/rsyncd "/bin/sh -c 'rsync -…" 12 days ago Restarting (11) 52 seconds ago rsyncd
a960683009be kiwix/matomo-log-analytics "/bin/sh -c '{ ech…" 12 days ago Up 9 days matomo-log-analytics_download
4ef4cc0a0eed gimoh/pureftpd "/usr/local/sbin/dkr…" 12 days ago Up 9 days 0.0.0.0:21->21/tcp, 0.0.0.0:30000-30050->30000-30050/tcp ftpd
root@download:/home/kelson# docker logs --tail 50 --follow --timestamps rsyncd
2018-11-16T18:05:08.781090614Z 2018/11/16 18:05:08 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:05:08.781315289Z 2018/11/16 18:05:08 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:06:09.591067663Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:06:09.591219618Z 2018/11/16 18:06:09 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:06:09.591235984Z 2018/11/16 18:06:09 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:07:10.417616402Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:07:10.417823161Z 2018/11/16 18:07:10 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:07:10.417845678Z 2018/11/16 18:07:10 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:08:11.306718261Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:08:11.306905071Z 2018/11/16 18:08:11 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:08:11.306922148Z 2018/11/16 18:08:11 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:09:12.054487615Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:09:12.054690613Z 2018/11/16 18:09:12 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:09:12.054730164Z 2018/11/16 18:09:12 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:10:12.934073380Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:10:12.934241176Z 2018/11/16 18:10:12 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:10:12.934267774Z 2018/11/16 18:10:12 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:11:13.907780159Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:11:13.907859784Z 2018/11/16 18:11:13 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:11:13.907879040Z 2018/11/16 18:11:13 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:12:14.692802711Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:12:14.693040041Z 2018/11/16 18:12:14 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:12:14.693086388Z 2018/11/16 18:12:14 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:13:15.612575162Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:13:15.612863823Z 2018/11/16 18:13:15 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:13:15.612905333Z 2018/11/16 18:13:15 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:14:16.637888146Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:14:16.638108427Z 2018/11/16 18:14:16 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:14:16.638801840Z 2018/11/16 18:14:16 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:15:17.473054555Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:15:17.473137537Z 2018/11/16 18:15:17 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:15:17.473154103Z 2018/11/16 18:15:17 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:16:19.249662879Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:16:19.249886583Z 2018/11/16 18:16:19 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:16:19.249934972Z 2018/11/16 18:16:19 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:17:20.142091063Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:17:20.142241131Z 2018/11/16 18:17:20 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:17:20.142521620Z 2018/11/16 18:17:20 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:18:20.901606884Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:18:20.901858269Z 2018/11/16 18:18:20 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:18:20.901897996Z 2018/11/16 18:18:20 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:19:21.963085761Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:19:21.963132608Z 2018/11/16 18:19:21 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:19:21.976045214Z 2018/11/16 18:19:21 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:20:22.889703400Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:20:22.889763190Z 2018/11/16 18:20:22 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:20:22.889807059Z 2018/11/16 18:20:22 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
2018-11-16T18:21:23.758098124Z failed to create pid file /var/run/rsyncd.pid: File exists
2018-11-16T18:21:23.758122689Z 2018/11/16 18:21:23 [8] rsync: failed to create pid file /var/run/rsyncd.pid: File exists (17)
2018-11-16T18:21:23.758145103Z 2018/11/16 18:21:23 [8] rsync error: error in file IO (code 11) at clientserver.c(1113) [Receiver=3.1.2]
Massive SPAM problem on wiki.kiwix.org
Since the 28 January 2019.
See:
https://wiki.kiwix.org/w/index.php?title=Special:RecentChanges&limit=250
All visitors have a same IP on Sloppy
When I receive mail confirmation, the IP adresse is always the same : 192.168.18.27. Then the limite of 99 account creation is reacheable in 24 hours, and nobody can create a new account. Maybe a bot creates accounts.
Motomo (db) does not work properly
I strongly suspect that the DB is not configured properly for this amount of data.
kiwix-serve should be started in background in library container
I do not get "the hand" anymore if I start the library
container, I should.
Wikifundi on Sloppy is suspended
The container seem to have been removed, but none has been created new!
zim directory in zim directory
$ pwd
/data/download/zim
$ ls -lad zim
lrwxrwxrwx 1 root root 32 Dec 2 16:37 zim -> /var/www/download.kiwix.org/zim/
@florentk I suspect you have added it, but of course this is wrong and I have removed it. Any reason why we have this link here?
MWOffliner for Ted.com
I would love to create a new Ted.com zim file, that latest one is from 2016-06. I read the Wiki (http://wiki.kiwix.org/wiki/TED) but I couldn't find any source code.
Is anyone aware of some code that works (or used to work). I may update/fix it so we have an uptodate archive.
Can stats.kiwix.org be behind the reverse proxy?
Even if we use an other docker-compose file?
mirrors.html seems not to be updated one time a day like it should
http://mirror.download.kiwix.org/mirrors.html date is 7 november.
Nginx logs should be splited by virtualhosts
mirror.download.kiwix.org cert seems not to be created properly
docker-compose restart letsencrypt
Does not allow me to connect with a working SSL layout to https://mirror.download.kiwix.org/.
It seems the certificates are not generated properly in data/certs/mirror.download.kiwix.org/
# ls mirror.download.kiwix.org/
account_key.json
download.kiwix.org alwys computing hash from files with "mb"
As we are almost not adding files, this is wrong. I suspect that (part of) the problem comes from the the fact that the hash are not store in the DB but in /usr/share/mirrorbrain
which is not part of any volume. This should be at least fixed.
Update mirrors.hml
It seems there is no code migrated to docker to update https://mirror.download.kiwix.org/mirrors.html
.
This code is in the cron
on the old server and looks like following:
mb mirrorlist -f xhtml | grep -v "href=\"\"" > /var/www/download.kiwix.org/mirrors.html
It should be put IMO in the mirrorbrain-update
container.
Support zsync in Mirrorbrain
To go ahead in zsync support, see kiwix/overview#12
Library is regularly down
The monitoring of the https service tells me:
The monitor Library (https://library.kiwix.org) is currently DOWN (HTTP 502 - Bad Gateway).
I had a brief look to the error log and it seems the limit is not configured properly.
Support Kiwix-android user-agent in Matomo
Necessary to better analyse where we looks users.
I have started to have a look to Matomo to find out how to deal with this, it looks like there is no clean solution:
1 - matomo-org/matomo#11641 (so we need to write a plugin)
2 - Modify directly in the source code the list of browser to support this new user agent.
Have a look also to kiwix/kiwix-android#73
https://download.openzim.org fails
I have migrated the service but seems impossible to get the SSL cert configured properly.
Tunisia mirror seems not to be taken in account by mirrorbrain
stats are down since 29/11
From @Popolechien on December 7, 2018 9:56
But only for download.kiwix.org
Any idea why?
Copied from original issue: kiwix/web#47
Library kiwix-serve is not restarted one time a day
I see in the process that it runs for more than 24h. One time a day, the library.kiwix.org.xml
should be newly generated and kiwix-serve
restarted to take in account the new library.kiwix.org.xml
file.
Apache links to /usr/local/apache2/icons/ should be in Apache config
Not in the data directory itself.
New Mirrorbrain seems to fail
After having:
- put
195.154.156.115 download.kiwix.org
to my/etc/hosts
- accepted the invalid certificate
I get:
A "500 Internal Server Error" on https://download.kiwix.org/
[URGENT] Cronjobs run should be protected to avoid overlapping
To avoid cronjobs scripts to run in parallel, they should be "protected" by the usage of flock
. All of this is properly configured on the current download server but has been removed in the Docker images/containers.
The overall server stability is put currently in danger because of this bug.
Can we give the mwoffliner scripts more descriptive names?
While debugging, I sometimes try to find the command for a particular build. For example, to determine whether a particular Kiwix file is built in mobile mode or not.
The existing script names give me no clue which file will have a particular build. I just have to search them all.
In particular, the endings _dump, _shm, and _zim don't appear to be different as far as I can tell. The files just seem to have different wikis and languages. Do those suffixes mean anything?
And of course 1, 2, 3, 4, 5 don't give any insight.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.