[ For new users, please see the wiki page at <a href="https://github.com/gnosygnu/xowa

XOWA on Android,about gnosygnu/xowa

Comments (26)

markusmaresch commented on September 15, 2024

My 2 cents on "reducing image quality to 50%":
I believe this is a good option in order to reduce the size of the image database - however to be used very carefully !!
I would reduce anything larger than a reasonable screen resolution (beyond HD) to HD - downsampling. The ImageMagick 'convert' is great for things like that. For any image below, it might be ok to decrease the quality (say to 70% from 95%) - for JPG that might be easy - for PNG not so much probably. Converting all PNG's to JPG's might be an option - but then you have to deal with all the references pointing to that one image with a different name .. not sure.

from xowa.

gnosygnu commented on September 15, 2024

I believe this is a good option in order to reduce the size of the image database - however to be used very carefully !!

Yup, I haven't much experience with this, and as there are over 4 million images in the XOWA English Wikipedia image databases, I've been a bit hesitant to go this route. Hence the delay.

I would reduce anything larger than a reasonable screen resolution (beyond HD) to HD - downsampling

I was planning on downsampling all images, not just those which are high resolution (HD?). I know there will be loss of quality for lower-resolution images, but they're not much noticeable to the eye. For example, this link shows a pretty progression between different quality settings (for Lightroom, but still relevant) http://regex.info/blog/lightroom-goodies/jpeg-quality

The ImageMagick 'convert' is great for things like that. For any image below, it might be ok to decrease the quality (say to 70% from 95%) - for JPG that might be easy - for PNG not so much probably. Converting all PNG's to JPG's might be an option - but then you have to deal with all the references pointing to that one image with a different name .. not sure.

Yup. Agreed here as well. My naive approach is to use ImageMagick to convert...

leave all GIFs and SVGs as is
convert all JPGs to 50%
convert all PNGs to JPGs and then downscale to 50%. I'd still keep the image name as PNG so that XOWA could still reference the same file (as you pointed out). It doesn't matter to the web-browser whether I'm loading a JPG-format file with a PNG name (Wikipedia actually does this with a few files). Also, the last time I looked at this PNGs are only something like 10% - 20% of images. The vast majority are JPGs

I'd adjust depending on results, but this would be where I'd start.

Let me know if I'm missing anything. Otherwise, thanks for the feedback!

from xowa.

markusmaresch commented on September 15, 2024

I agree mostly with your approach. However I would not downsample by the fixed 50% - but rather GRADUALLY - see attached worksheet.

Realize - for an offline Wiki it is much more important to have AN image (with possibly lower quality), than having NO image at all. If the high resolution full quality image is required - one can go to the original online version anyway in most cases.
I like the 'convert to JPG', but leave the PNG extension trick :-) .. most viewers probably look at the magic header anyway, than relying on the extension.

xowa_image_scale_down.xlsx.zip

from xowa.

gnosygnu commented on September 15, 2024

I agree mostly with your approach. However I would not downsample by the fixed 50% - but rather GRADUALLY - see attached worksheet.

Hey, that spreadsheet is great! How'd you come up with those formulas? I wouldn't have thought that compression rations can be generalized from aspect ratios. I assume you've done this sort of thing before?

Realize - for an offline Wiki it is much more important to have AN image (with possibly lower quality), than having NO image at all.

I do agree. Keep in mind though that it was easier to just download the "original" thumb from Wikipedia than to do additional conversion. In addition, I've found ImageMagick to be flaky (about 1% on converted images). Besides, I sort of liked being able to say that the image is exactly the same thing one would have gotten online from Wikipedia.

That said, Android is really forcing my hand, as storage space is at a premium, and it's hard to justify taking up a full 128 GB card.

I like the 'convert to JPG', but leave the PNG extension trick :-) .. most viewers probably look at the magic header anyway, than relying on the extension.

Yup. That's what I believe they do. I know Wikimedia commons has occasional pngs that have a ".jpg" extension as well as vice versa.

from xowa.

gnosygnu commented on September 15, 2024

I uploaded the alpha tonight. This comment will detail requirements, installation, wiki import. It also lists next release items as well as feedback requests.

Requirements:

Android device with KitKat or higher.
2.5 GB free space

Installation

(This assumes a Windows computer with a microSD card, but other setups are possible. If you have problems, please comment below and I'll help)

Download Simple Wiki: https://archive.org/download/Xowa_simplewiki_latest/Xowa_simplewiki_2015-12-06.7z
Unzip it. You'll have four files
- simple.wikipedia.org-text.xowa
- simple.wikipedia.org-html.xowa
- simple.wikipedia.org-file.xowa
- simple.wikipedia.org-file-user.xowa
Copy them to your microSD card. For example, if you're on Windows and your microSD card is attached as the X drive, then copy them to "X:\xowa\wiki\simple.wikipedia.org".
Download the apk: https://github.com/gnosygnu/xowa_android/releases/download/v0.1.2.0/xowa_android_app.apk
Copy the apk to your microSD card. For example, copy it to X:\xowa\xowa_android_app.apk
Insert the microSD card in to your Android phone
Open up File Manager and navigate to the xowa_android_app.apk directory. For example, on an LG G4, you would navigate to "All files" -> "SD card" -> "xowa"
Open up "xowa_android_app.apk" and install it

Wiki import

(Please note that the initial setup is rather rough and unintuitive. I will fix this for the next release, but you'll have to endure it this one time.)

XOWA opens to a page that says "Main Page" with only the text "Page not found".
Press the menu button in the upper-left hand corner (the three horizontal bars)
After the menu slides open, press the "XOWA" item
The XOWA file browser opens. Navigate to your simple.wikipedia.org directory by pressing the directories. For example, on an LG G4 phone, you would navigate to "storage", "external_SD", "xowa", "wiki", "simple.wikipedia.org"
In the simple.wikipedia.org directory, check the file "simple.wikipedia.org-text.xowa"
Press the "import" link (located above the table)
Another screen opens. Press the "simple.wikipedia.org Main Page" link
The Main Page opens. Press any link to navigate within Simple Wikipedia.
Press "Search XOWA". Enter in "Earth" and select the "Earth" entry
The "Earth" page opens

Next release

Upload English Wikipedia: I'm generating the 7z files and will be uploading them to archive.org over the next few days
Polish the Wiki import UI: The UI is very rough. I'm planning to fix items like the initial "Page not found" to a cleaner "import" process
Load images immediately if available: This is probably easier seen than explained. XOWA always dynamically updates the HTML with images, even if the page has already been viewed. Instead, XOWA should dynamically update the HTML the first time a page is viewed, but when the page is viewed again, generate the HTML correctly from the start.
Improve Search performance: The search function on English Wikipedia can be slow. Some terms like "Earth" may take up to a minute to retrieve. I'm going to try a quick-fix via UI responsiveness, but if this doesn't work, then I'll need to add more indices to the search database. Unfortunately, that may take several weeks.
Remove Wikipedia-specific items: There are still many places which say "Wikipedia" instead of "XOWA". I removed the most prominent ones, but I'm still debating how to change the rest. See Feedback requests below.

Feedback requests

Include entire Wikipedia mobile app? XOWA is built into the Wikipedia Mobile app, so it automatically inherits most of its functions. However, many features don't apply to the offline world, such as "Log in to Wikipedia", "Nearby", "Wikipedia Zero", "Saved Pages", "Today". I'm thinking about removing them, unless there are users that are thinking of installing only XOWA, and using it for both offline and online. Do you expect to use the XOWA app as a hybrid app? For example, read offline, but if you need to edit a page, press a switch, and then edit directly within XOWA. Or would you rather keep two separate Android apps (the XOWA offline app and the Wikipedia Mobile online app) and jump between the two?
Expand tables by default? The Wikipedia Mobile app automatically collapses tables. For example, the infobox is always collapsed, and must be manually expanded. I decided to change the default to automatically expand tables. This may make the page more "noisy" and increase scrolling, but I like seeing as much content as possible without having to do additional presses.
How does it work on your device? Couldn't get it to run? Too slow? Errors? Please comment below!
Other priorities? What else do you think is needed before the Play Store release?

from xowa.

gnosygnu commented on September 15, 2024

I uploaded the English Wikipedia html dumps today.

Background

The XOWA Android app uses XOWA HTML dumps. The XOWA Wikitext dumps currently are not readable on Android (they will be in the future). However, due to the slow processors on mobile devices and the heavy requirements of MediaWiki parsing (luaj and jtidy particularly), English Wikipedia will be released primarily as pre-compiled XOWA HTML dumps. These XOWA HTML dumps will be released on archive.org
The current HTML dumps are for 2015-12-06 and are about 32 GB. Note that this is an alpha copy, though it's mostly functional (there are still some bugs I'm working on). I will re-upload a new English Wikipedia copy for 2016-01 for the Google Play Store release. If downloading 32 GB is a problem, you may want to wait. That said, I've been using this dump for the past several days, and it works very well.
I will release another update of the alpha this Sunday. This will have features like "Random" as well as some other minor UI tweaks.
If you want images, you can move your XOWA image databases to the same directory. However, you will need a full 128 GB microSD card for HTML + images. My next priority is to reduce the size of both to somewhere around 64 GB, though this won't happen for at least a few months.

Installation

Download the HTML dumps
- main index: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_core.7z
- 1st 2 million articles in main namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_ns-000_part-1.7z
- 2nd 2 million articles in main namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_ns-000_part-2.7z
- remaining articles in main namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_ns-000_part-3.7z
- Portal namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_ns-100.7z
- Project namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_ns-004.7z
- Category namespace (not really usable unless you find a direct link from an article): https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2015-12-06_html_ns-014.7z
Unzip them and copy all files to one folder on your microSD card. For example, if you're on Windows and your microSD card is attached as the X drive, then copy them to "X:\xowa\wiki\en.wikipedia.org".
Copy as many image databases as you can to your microSD card.
See Wiki Import above, but replace en.wikipedia.org for simple.wikipedia.org
Also, pick "en.wikipedia.org-core.xowa" as the file to be imported.

from xowa.

sjgard2001 commented on September 15, 2024

Hi,
Here's quick (and slightly simplistic) error report for you.

Handset:
Samsung Galaxy S6

Android version:
v6.0.1 (Android Security Patch Level: 1 January 2016
Samsung Official UK Marshmallow public pre-release Build MMB29K.G920FXXU3PZA1

Application version:
Xowa v0.1.2.0 [based on the filename I downloaded and installed](according to Settings -> Applications -> version 2.1.136-xowa-2016-01-10)

Upon launching the Xowa for Android for the first time it appeared to render the main application splash screen with the Xowa logo in the middle. That was then replaced by the Search XOWA screen which appeared momentarily. The application then halted and end on a "We're sorry, the XOWA app has experienced an error and was terminated. Would you like to start over or quit?" error screen with Start Over and Quit buttons. Starting over just results in the same behaviour repeated.

Logcat appeared to show that this was the result of a directory not existing.

01-15 23:40:33.369 27978 27978 E org.wikipedia.crash.hockeyapp.HockeyAppExceptionHandler: gplx.Err: /mnt/sdcard/external_sd/temp/current/session.txt: open failed: ENOENT (No such file or directory)
01-15 23:40:33.369 27978 27978 E org.wikipedia.crash.hockeyapp.HockeyAppExceptionHandler: at gplx.Err_.cast_or_make(Err_.java:57)

When I created the directory manually (creating temp and then temp/current under /mnt/sdcard/external_sd) the application would crash in the same way with a permissions problem

01-15 23:43:32.109 29513 29513 E org.wikipedia.crash.hockeyapp.HockeyAppExceptionHandler:
01-15 23:43:32.109 29513 29513 E org.wikipedia.crash.hockeyapp.HockeyAppExceptionHandler: gplx.Err: /mnt/sdcard/external_sd/temp/current/session.txt: open failed: EACCES (Permission denied)

That's somewhat understandable given the combination of per application file ownerships + (potentially) SElinux intervention. Next I deleted the directories I created and checked the applications permissions. The Storage permission was listed but this wasn't enabled (I was never asked during installion to grant it). After I granted the storage permission the application continues to crash in the same manner but /mnt/sdcard/external_sd/temp/current/session.txt is now being created by the application and we've now moved on to something other than file creation / permissions it seems

01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: 01-15 23:55:48.078 9899 9948 D HockeyApp: android.database.sqlite.SQLiteException: no such table: user_site (code 1): , while compiling: SELECT site_id, site_priority, site_domain, site_name, site_path, site_xtn FROM user_site ################################################################# Error Code : 1 (SQLITE_ERROR) Caused By : SQL(query) error or missing database. (no such table: user_site (code 1): , while compiling: SELECT site_id, site_priority, site_domain, site_name, site_path, site_xtn FROM user_site) ################################################################# at android.database.sqlite.SQLiteConnection.nativePrepareStatement(Native Method) at android.database.sqlite.SQLiteConnection.acquirePreparedStatement(SQLiteConnection.java:1058) at android.database.sqlite.SQLiteConnection.prepare(SQLiteConnection.java:623) at android.database.sqlite.SQLiteSession.prepare(SQLiteSession.java:588) at android.database.sqlite.SQLiteProgram.<init>(SQLiteProgram.java:59) at android.database.sqlite.SQLiteQuery.<init>(SQLiteQuery.java:37) at android.database.sqlite.SQLiteDirectCursorDriver.query(SQLiteDirectCursorDriver.java:44) at android.database.sqlite.SQLiteDatabase.rawQueryWithFactory(SQLiteDatabase.java:1454) at android.database.sqlite.SQLiteDatabase.queryWithFactory(SQLiteDatabase.java:1301) at android.database.sqlite.SQLiteDatabase.query(SQLiteDatabase.java:1172) at android.database.sqlite.SQLiteDatabase.query(SQLiteDatabase.java:1340) at gplx.dbs.Drd_db_stmt.Exec_select(Drd_db_mgr.java:197) at gplx.dbs.Drd_db_stmt.Exec_select__rls_auto(Drd_db_mgr.java:187) at gplx.xowa.users.data.Xoud_site_tbl.Select_all(Xoud_site_tbl.java:37) at gplx.xowa.users.data.Xoud_site_mgr.Get_all(Xoud_site_mgr.java:11) at gplx.xowa.apps.Xoav_wiki_mgr.Load_by_user_data(Xoav_wiki_mgr.java:17) at gplx.xowa.drds.Xod_app_mgr.Init_app(Xod_app_mgr.java:125) at gplx.xowa.drds.Xod_app_mgr.Get_page_or_load(Xod_app_mgr.java:88) at gplx.xowa.drds.OfflinePageLoadStrategy.loadLeadSection(OfflinePageLoadStrategy.java:756) at gplx.xowa.drds.OfflinePageLoadStrategy.performActionForState(OfflinePageLoadStrategy.java:270) at gplx.xowa.drds.OfflinePageLoadStrategy.loadPageFromNetwork(OfflinePageLoadStrategy.java:456) at gplx.xowa.drds.OfflinePageLoadStrategy.loadPageOnWebViewReady(OfflinePageLoadStrategy.java:375) at gplx.xowa.drds.OfflinePageLoadStrategy.access$300(OfflinePageLoadStrategy.java:76) at gplx.xowa.drds.OfflinePageLoadStrategy$1.onMessage(OfflinePageLoadStrategy.java:177) at gplx.xowa.drds.OfflinePageLoadStrategy$SynchronousBridgeListener.onMessage(OfflinePageLoadStrategy.java:895) at org.wikipedia.bridge.CommunicationBridge$2.handleMessage(CommunicationBridge.java:123) at android.os.Handler.dispatchMessage(Handler.java:98) at android.os.Looper.loop(Looper.java:158) at android.app.ActivityThread.main(ActivityThread.java:7197) at java.lang.reflect.Method.invoke(Native Method) at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:1232) at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:1122)

Keep in mind I haven't yet managed to get the application to run for more than about a second so I haven't had a chance to select the Simple Wiki files.

session.txt contains just two lines per execution
a20160116_000209.473 db.schema.load.bgn: conn=data source=/data/user/0/org.xowa/files/user-anonymous.sqlite3;version=3;a
a20160116_000209.477 db.schema.load.enda

I wasn't sure this was exactly the kind of bugreport you were looking for or actually how much use it is but there you go.

Kind Regards
Stephen

from xowa.

gnosygnu commented on September 15, 2024

Hey, thanks for the detailed error report! It was really helpful.

I think there are 2 issues. You clearly identified the first: the log file was causing a permissions exception (/mnt/sdcard/external_sd/temp/current/session.txt). I'm a little puzzled why I didn't run across it on my device (or Anselm on his devices), but it may be a Marshmallow thing. I'll remove it for the next build as it's not really necessary.

The 2nd error (user_site) is also unexpected, but I think it's b/c the 1st error placed XOWA in an inconsistent state. Can you try uninstalling XOWA and then installing again, but this time with the permissions in place for session.txt? If that doesn't work, I'll take another look at the code.

Let me know if I missed anything else. Thanks again!

from xowa.

sjgard2001 commented on September 15, 2024

Thank you for the quick response.

I uninstalled the app, re-installed it and granted it the Storage permission prior to launching it for the first time and am happy to report that Xowa worked as expected. It is certainly notable that when the application was installed it didn't prompt for any permissions but clearly the manifest does include requests for Location and Storage access. Whether this lack of permissions is a security measure on Samsung's particular flavour of Android when installed from unknown source or perhaps even a bug in this pre-release version of the OS.. who can tell.

During today I downloaded the HTML dumps for English wikipedia and tried that out and that instigated another crash which left the application inconsistent and needing re-install. Unlike the permissions issue this one wasn't fixed by a re-installed ie. it recurred when I tried the same sequence of events (Storage permission was of course added prior to first launch). I think this is a lack of understanding on my part because although I added what I believed to be the correct set of files to the storage on the phone it's complaining that a whole bunch of required files are missing.

I downloaded an unzip'd

Xowa_enwiki_2015-12-06_html_core.7z
Xowa_enwiki_2015-12-06_html_ns-000_part-1.7z
Xowa_enwiki_2015-12-06_html_ns-000_part-2.7z
Xowa_enwiki_2015-12-06_html_ns-000_part-3.7z
Xowa_enwiki_2015-12-06_html_ns-004.7z
Xowa_enwiki_2015-12-06_html_ns-014.7z
Xowa_enwiki_2015-12-06_html_ns-100.7z

to the phone into

/sdcard/xowa/wiki/en.wikipedia.org

as listed in your installation instructed and then decided not to add the images databases beleving them to be unnecessary. But after selecting en.wikipedia.org-core.xowa I ended up in the crashing cycle previously mentioned. I now see that I'm missing (according to logcat)

wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.004.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.118.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.012.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.100.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.006.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.010.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.008.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.002.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.014.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.003.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.004.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.828.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.005.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.108.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.006.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.004-db.002.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.007.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.710.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.008.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.2600.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.2300.xowa
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000-db.009.xowa
wiki.db:missing db; tid=xtn.category.core url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.core.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.001.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.002.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.003.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.004.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.005.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.006.xowa
wiki.db:missing db; tid=xtn.category.link url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-xtn.category.link-db.007.xowa
wiki.db:missing db; tid=html url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-html-ns.014.xowa

As you stated that the XOWA wikitext dumps aren't currently readable I'm confused about the requirement for what seems to be a large number of "text" related files. Also can I ask where / which 7z(s) these missing files come from? After downloading a fair number of the Xowa_enwiki_2015-06-02_file* files it seems clear they don't contain any "text" files the app wants. Any help / clarification would be appreciated.

For reference my device currently has the following in its /sdcard/xowa/wiki/en.wikipedia.org directory

en.wikipedia.org-core.xowa
en.wikipedia.org-file-ns.000-db.013.xowa
en.wikipedia.org-file-ns.000-db.014.xowa
en.wikipedia.org-file-ns.000-db.015.xowa
en.wikipedia.org-html-ns.000-db.002.xowa
en.wikipedia.org-html-ns.000-db.003.xowa
en.wikipedia.org-html-ns.000-db.004.xowa
en.wikipedia.org-html-ns.000-db.005.xowa
en.wikipedia.org-html-ns.000-db.006.xowa
en.wikipedia.org-html-ns.000-db.007.xowa
en.wikipedia.org-html-ns.000-db.008.xowa
en.wikipedia.org-html-ns.000-db.009.xowa
en.wikipedia.org-html-ns.000-db.010.xowa
en.wikipedia.org-html-ns.000-db.011.xowa
en.wikipedia.org-html-ns.000-db.012.xowa
en.wikipedia.org-html-ns.000-db.013.xowa
en.wikipedia.org-html-ns.000-db.014.xowa
en.wikipedia.org-html-ns.000-db.015.xowa
en.wikipedia.org-html-ns.000-db.016.xowa
en.wikipedia.org-html-ns.000.xowa
en.wikipedia.org-html-ns.004-db.002.xowa
en.wikipedia.org-html-ns.004-db.003.xowa
en.wikipedia.org-html-ns.004.xowa
en.wikipedia.org-html-ns.100.xowa
en.wikipedia.org-xtn.search.core.xowa

Thanks,
Stephen

from xowa.

gnosygnu commented on September 15, 2024

I uninstalled the app, re-installed it and granted it the Storage permission prior to launching it for the first time and am happy to report that Xowa worked as expected.

Great! Glad to hear that worked. I'll fix the broken "user_site" table for a future release.

It is certainly notable that when the application was installed it didn't prompt for any permissions but clearly the manifest does include requests for Location and Storage access.

Yeah, I'm puzzled by that. However, this shouldn't be an issue anymore, as I deactivated logging to the external SD card for tomorrow night's release. Thanks for reporting it!

But after selecting en.wikipedia.org-core.xowa I ended up in the crashing cycle previously mentioned. I now see that I'm missing (according to logcat)
wiki.db:missing db; tid=text url=/sdcard/xowa/wiki/en.wikipedia.org/en.wikipedia.org-text-ns.000.xowa

That's fine. Those are warning messages that are for log purposes. The Android App works fine without the wikitext dumps; it just complains to the log file.

Also can I ask where / which 7z(s) these missing files come from?

Sure! They're from the first row - the one with "wiki.core", "wiki.main articles-part-1", etc.. Here they are:

You can also build these from within XOWA by doing Main Menu -> Tools -> Import Online (or Import Offline)

For reference my device currently has the following in its /sdcard/xowa/wiki/en.wikipedia.org directory

This is a bug on my side related to Android and the read-only file systems. For now, you'll need two additional files: en.wikipedia.org-file-core.xowa and en.wikipedia.org-file-user.xowa . You can get them by doing either of the following:

Download them from my Google Drive account
- https://drive.google.com/file/d/0B9cb52zjL2rIVHFjYWRqdXBrU1E/view?usp=sharing
- https://drive.google.com/file/d/0B9cb52zjL2rId2lsQi1zVEhxSjA/view?usp=sharing
Or generate them automatically within desktop XOWA
- Setup English Wikipedia (by downloading from archive.org or by building it manually)
- Run XOWA and navigate to en.wikipedia.org
- Copy the 2 files from the en.wikipedia.org directory. If your XOWA is at C:\xowa, then the files would be at C:\xowa\wiki\en.wikipedia.org

More explanation follows.

XOWA can run without file databases, but it still expects stubs for file databases. Desktop XOWA creates these on the fly if they don't exist. Android XOWA also tries to create them on the fly, but since the file-system is read-only, it fails and the program crashes.

I'll try to fix this for a future version, but it may take me a while

from xowa.

sjgard2001 commented on September 15, 2024

Thank you so much for the explanations, assistance and work on this project. As someone's who has pursued getting English wikipedia (particularly an up to date version) onto a mobile device for many years Xowa for Android really is somewhat of a Holy Grail for me. Having bashed my head against various increasingly complex options including reverse engineering Wikidroyd (which used a neat somewhat hacky trick to index LZMA compressed blobs and re-use a minimal LZMA compression header with a native LZMA library), to running wp-mirror with mysql instances running on 256GB RAM drives to hold the database to try and get enough performance to dump out a Mediawiki instance, to my own crazied scripts to process the try and process the XML dumps .. it was quite something to add the two extra files you provided and see Xowa spring into life. My investment in a 128GB Galaxy S6 (and previous over-spec'd phones) to hold this monster has finally paid off.

I'll add one query here in and then a couple of separate comments regarding images compression strategies later once I've re-read the comments made above.

Is it possible to use Xowa to import the XML dump and then produce the HTML dumps suitable for Xowa for Android? From what I've seen Xowa produces text dumps. If HTML dumps can be generated from Xowa can they be generated with a script via the command-line? You can probably see where this is going.. I've got plenty of compute power and space at my disposal so if I can take a XML bz2 dump, throw it on to a Linux box at work and let it process overnight and generate the HTML dump I can produced updated copies with minimal remote downloading on-demand. I do understand that given that Wikipedia itself is a moving target this process wouldn't always produce viable output but I'd at least like to understand if this sort fo thing is possible in principle.

Regards
Stephen

from xowa.

sjgard2001 commented on September 15, 2024

As someone who tends to use the dark themes in applications whenever possible to minimize power usage on my AMOLED display I thought I'd mention a trival issue with the presentation. I'm not sure how you produce the dark theme, presumably through an alternative CSS.

The most notable aspect common to many pages that remains light (grey / white) when the dark theme is selected is

table.wikitable > tr > th
table.wikitable > * > tr > th {
background-color: #F2F2F2;
}

This seems to be about the only issue I've seen that can probably be adjusted. There are many individual elements which retain their light / bright luminence but as have per-element color values short they're not really addressable globally short of some kind of pre-render "color" substitution pass through the HTML.

from xowa.

gnosygnu commented on September 15, 2024

@sjgard2001

Thank you so much for the explanations, assistance and work on this project.

No problem! Thank you for sticking with it. I know the current version is pretty rough around the corners, but I'm glad that someone was able to figure it out.

As someone's who has pursued getting English wikipedia (particularly an up to date version) onto a mobile device for many years Xowa for Android really is somewhat of a Holy Grail for me. Having bashed my head against various increasingly complex options including reverse engineering Wikidroyd (which used a neat somewhat hacky trick to index LZMA compressed blobs and re-use a minimal LZMA compression header with a native LZMA library)

Wow, that's quite some dedication on your side. I haven't used Wikidroyd before, but I'm surprised they had anything that was a complete Wikipedia dump.

to running wp-mirror with mysql instances running on 256GB RAM drives to hold the database to try and get enough performance to dump out a Mediawiki instance

WP-mirror is a good project, but I think the main bottleneck is the PHP code. I think it takes something like 30+ days to generate an English Wikipedia dump. Java may be slow, but I did pay some attention to performance when writing the code. Currently, XOWA can generate a dump using one machine in about 3+ days: an approximate throughput of 20 articles per second over 5 million articles

to my own crazied scripts to process the try and process the XML dumps ..

Yeah, I feel your pain. I started off that way before working on XOWA. :)

Is it possible to use Xowa to import the XML dump and then produce the HTML dumps suitable for Xowa for Android?

Yes! If you're brave enough to try. :) I'd love to have as many people as possible generating HTML dumps on their side.

There is already a page in XOWA for this functionality: home/wiki/Help:Import/Command-line/Thumbs . I will update it in tonight's release for the second-generation HTML dumps. You will need a dedicated high-performance machine and about 70 hours of processing time. If you run into any problems, open up another issue, and I'll help.

Let me know if there's any other questions. Thanks again!

from xowa.

gnosygnu commented on September 15, 2024

As someone who tends to use the dark themes in applications whenever possible to minimize power usage on my AMOLED display

Just so you know, I tend to use the light theme by default. The dark theme is nice, but the light theme is "closer" to the actual look and feel of the Wikipedia website. It helps me spot issues quicker.

I'm not sure how you produce the dark theme, presumably through an alternative CSS.

XOWA uses the Wikipedia Mobile App for almost all its UI. The Wikipedia Mobile app has two css files: styles.css and night.css. The latter is responsible for all the CSS changes.

You can see the actual CSS by taking the XOWA apk and unzipping it using 7-zip. The files are in /assets/

The most notable aspect common to many pages that remains light (grey / white) when the dark theme is selected

Yeah, this color is hard-coded into the wikitext. For example, see the infobox for surface temp in en.w:Earth and then look at en.w:Template:Infobox_planet . You'll see the F2F2F2 hard-coded directly there.

I'll add this simple css rule for tonight's release:

  .infobox    *
, .wikitable  *
{
  background-color: black !important;
}

This seems to take care of most of the issues, though there may still be individual elements as you mentioned.

There are many individual elements which retain their light / bright luminence but as have per-element color values short they're not really addressable globally short of some kind of pre-render "color" substitution pass through the HTML.

If you spot any others and want to suggest other CSS, you should be able to set-up a test bed on your side. Let me know if you're interested, and I'll put up a wiki page for doing limited HTML reviews using the Wikipedia Mobile / XOWA stylesheets.

Ultimately though, it would be difficult to do a complete night conversion. The main issue is that many images have a white background hard-coded. In addition, there are probably many other wikitext-encoded colors that can't be clobbered to night-mode using a CSS rule.

That said, if you have any ideas, I'm willing to hear them out. :)

Finally, I like to credit users for reporting bugs. There's currently a line in the XOWA change log with text like "Change background-color to black for infobox / wikitable in night-mode {detected by anonymous}". Let me know if you prefer an alias like sjgard2001, or would rather remain anonymous

Thanks!

from xowa.

gnosygnu commented on September 15, 2024

For new users, I'm summarizing the instructions from above in one comment. As always, let me know if you have questions, and I'll answer below as well as amend here.

Thanks.

Requirements:

Android device with KitKat or higher.
2.5 GB free space

Installation

(This assumes a Windows computer with a microSD card, but other setups are possible. If you have problems, please comment below and I'll help)

Download Simple Wiki: https://archive.org/download/Xowa_simplewiki_latest/Xowa_simplewiki_2015-12-06.7z
Unzip it. You'll have four files
- simple.wikipedia.org-text.xowa
- simple.wikipedia.org-html.xowa
- simple.wikipedia.org-file.xowa
- simple.wikipedia.org-file-user.xowa
Copy them to your microSD card. For example, if you're on Windows and your microSD card is attached as the X drive, then copy them to "X:\xowa\wiki\simple.wikipedia.org".
Download the apk: https://github.com/gnosygnu/xowa/releases/download/v3.3.2.1/xowa_app_android_v3.3.2.1.apk
Copy the apk to your microSD card. For example, copy it to X:\xowa\xowa_android_app.apk
Insert the microSD card in to your Android device
Open up File Manager and navigate to the xowa_android_app.apk directory. For example, on an LG G4, you would navigate to "All files" -> "SD card" -> "xowa"
Open up "xowa_android_app.apk" and install it

Wiki import

(Please note that the initial setup is rather rough and unintuitive. I will fix this for a future release, but you'll have to endure it once only when you setup a wiki.)

XOWA opens to a page that says "Main Page" with only the text "Page not found".
Press the menu button in the upper-left hand corner (the three horizontal bars)
After the menu slides open, press the "XOWA" item
The XOWA file browser opens. Navigate to your simple.wikipedia.org directory by pressing the directories. For example, on an LG G4 phone, you would navigate to "storage", "external_SD", "xowa", "wiki", "simple.wikipedia.org"
In the simple.wikipedia.org directory, check the file "simple.wikipedia.org-text.xowa"
Press the "import" link (located above the table)
Another screen opens. Press the "simple.wikipedia.org Main Page" link
The Main Page opens. Press any link to navigate within Simple Wikipedia.
Press "Search XOWA". Enter in "Earth" and select the "Earth" entry
The "Earth" page opens

English Wikipedia

The process is similar to Simple Wikipedia, except for the following:
Download the HTML dumps
- main index: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_core_v3.7z
- 1st 2 million articles in main namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_ns-000_part-1.7z
- 2nd 2 million articles in main namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_ns-000_part-2.7z
- remaining articles in main namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_ns-000_part-3.7z
- Portal namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_ns-100.7z
- Project namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_ns-004.7z
- Category namespace: https://archive.org/download/Xowa_enwiki_latest/Xowa_enwiki_2016-02-04_html_ns-014.7z
Unzip them and copy all files to one folder on your microSD card. For example, if you're on Windows and your microSD card is attached as the X drive, then copy them to "X:\xowa\wiki\en.wikipedia.org".
Copy as many image databases as you can to your microSD card.
If you don't have en.wikipedia.org-file-core.xowa and en.wikipedia.org-file-user.xowa, do one of the following:
- Generate them automatically within desktop XOWA
  - Run XOWA and navigate to any page in en.wikipedia.org
  - Copy the 2 files from the en.wikipedia.org directory. If your XOWA is at C:\xowa, then the files would be at C:\xowa\wiki\en.wikipedia.org
- Or download them from my Google Drive account
  - https://drive.google.com/file/d/0B9cb52zjL2rIVHFjYWRqdXBrU1E/view?usp=sharing
  - https://drive.google.com/file/d/0B9cb52zjL2rId2lsQi1zVEhxSjA/view?usp=sharing
Follow the process in Wiki Import above, but navigate to the en.wikipedia.org directory
Choose "en.wikipedia.org-core.xowa" as the file to be imported.

from xowa.

sjgard2001 commented on September 15, 2024

[ Apologies for the brevity. I only have a short catch-up window today ]

Thank you for the change to the table.wikitable.tr.hr color, it's made a substantial improvement to the dark theme.

I will read the HTML dump documentation you've included. Hopefully I'll get a chance to experiment with that over the weekend.

One additional issue that I think is worth looking at.

If you open any wiki page (including the Main Page) and scroll to the very bottom and then slowly begin zooming in using pinch-to-zoom it appears as though the page footer begins moving up the screen and eventually disappears off the top as the zoom level increases. As the page context pop-up button (the circular green button, with three lines + dots) seems to the tied in some way to the bottom of the footer if that button is visible (before its fade out) it will follow the page footer up and off the top of the screen.

If that's unclear I can grab a screenshot later.

from xowa.

gnosygnu commented on September 15, 2024

Thank you for the change to the table.wikitable.tr.hr color, it's made a substantial improvement to the dark theme.

No problem! Thanks for the confirm, as well as the original suggestion. Please feel free to suggest others. :)

If you open any wiki page (including the Main Page) and scroll to the very bottom and then slowly begin zooming in using pinch-to-zoom

Yeah, this is an issue with the Wikipedia Mobile App. They actually implement these panels (View in Browser; Date modified; CC license) as Android widgets and try to dock it to the bottom of the screen. This layout goes haywire in pinch-zoom mode.

I think they realized this, as the Wikipedia Mobile App doesn't allow pinch-zoom. I added pinch-zoom to XOWA manually because I thought it was useful.

I'll replace these bottom panels with HTML instead. I may not be able to do the nice page-shadow effect, but at least it'll fix the issue.

from xowa.

PhongTong commented on September 15, 2024

How can I use Xowa Android for other wikis instead?

from xowa.

gnosygnu commented on September 15, 2024

Hi! Right now the only way is to build an html-dump yourself. See home/wiki/Help:Import/Command-line/Dumps . If you're comfortable with a command-line, you should be able to get it going.

If there's a specific wiki that you want, let me know, and I'll try to build it earlier. I'm working on the other English wikis now (English Wiktionary, Wikiquote, etc) Eventually, I'm going to release Android versions for all wikis to archive.org, but it will take me a few months.

Also, I'm planning to get XOWA Android to read the wikitext-dump. However, compiling wikitext on the Android will probably be slow for most devices, so this would probably only be useful for small wikis.

Anyway, let me know if there is any other information. Thanks!

from xowa.

PhongTong commented on September 15, 2024

I'm planning use Xowa Android for some wikis from wikia sites. Could it be some way different than creating Html dump from wikipedia?
Thank you. :3

from xowa.

gnosygnu commented on September 15, 2024

Yeah, it is different than the current way. :(

Right now, here are your options:

Let me know which wikia site you're interested in and I'll build the html-dump for you
Try to build the html-dumps on your own using the command-line (home/wiki/Help:Import/Command-line/Dumps)
Wait a few months and I'll get the Android version to read directly from the XOWA wikitext dumps.

Let me know which direction you're leaning. Thanks.

from xowa.

PhongTong commented on September 15, 2024

I think i want to do it by myself. So I don't depend on anyone and can be more active :3
Thanks :)

from xowa.

gnosygnu commented on September 15, 2024

No problem. Let me know if there are any issues. Thanks!

from xowa.

anarky321 commented on September 15, 2024

a 200GB microsd card is only $100, a 128gb one far cheaper.... i think that the database image quality should be left as is

from xowa.

gnosygnu commented on September 15, 2024

Yup, I do have one of those $100 200 GB cards. :)

With that said, I think I'm still going to get XOWA to use less space. At the very least, one can fit more XOWA wikis with the extra GB.

Also, I'll always offer the "original" db quality. There will basically be two sets: the "original quality" one and the "reduced quality" one.

from xowa.

gnosygnu commented on September 15, 2024

The XOWA Android App was released to the Play Store tonight. See the following links for more info:

Play Store Link: https://play.google.com/store/apps/details?id=org.xowa
Youtube video: https://www.youtube.com/watch?v=jsMTBxGweUw
Android Setup: http://xowa.org/home/wiki/Android/Setup.html

This ticket is now closed. Thanks to all who've tested as well as those who've patiently waited.

from xowa.

XOWA on Android about xowa HOT 26 CLOSED

Comments (26)

Requirements:

Installation

Wiki import

Next release

Feedback requests

Background

Installation

Requirements:

Installation

Wiki import

English Wikipedia

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent