Coder Social home page Coder Social logo

kiwix / libkiwix Goto Github PK

View Code? Open in Web Editor NEW
108.0 10.0 52.0 8.13 MB

Common code base for all Kiwix ports

Home Page: https://download.kiwix.org/release/libkiwix/

License: GNU General Public License v3.0

C++ 76.24% C 0.15% Python 1.63% Meson 0.86% Shell 0.13% CSS 1.87% JavaScript 17.25% HTML 1.77% Roff 0.09%
kiwix

libkiwix's Introduction

Libkiwix

The Libkiwix provides the Kiwix software suite core. It contains the code shared by all Kiwix ports (Windows, GNU/Linux, macOS, Android, iOS, ...).

Release Repositories Build Status Doc CodeFactor Codecov License: GPL v3

Disclaimer

This document assumes you have a little knowledge about software compilation. If you experience difficulties with the dependencies or with the Libkiwix compilation itself, we recommend to have a look to kiwix-build.

Preamble

Although the Libkiwix can be (cross-)compiled on/for many systems, the following documentation explains how to do it on POSIX ones. It is primarily thought for GNU/Linux systems and has been tested on recent releases of Ubuntu and Fedora.

Dependencies

The Libkiwix relies on many third party software libraries. They are prerequisites to the Libkiwix compilation. Following libraries need to be available:

  • ICU (package libicu-dev on Ubuntu)
  • ZIM (package libzim-dev on Ubuntu)
  • Pugixml (package libpugixml-dev on Ubuntu)
  • Mustache (Just copy the header mustache.hpp somewhere it can be found by the compiler and/or set CPPFLAGS with correct -I option). Use Mustache version 4.1 or above.
  • Libcurl (libcurl4-gnutls-dev, libcurl4-nss-dev or libcurl4-openssl-dev on Ubuntu)
  • Microhttpd (package libmicrohttpd-dev on Ubuntu)
  • Zlib (package zlib1g-dev on Ubuntu)

To test the code:

The following dependency needs to be available at runtime:

  • Aria2 (package aria2 on Ubuntu)

These dependencies may or may not be packaged by your operating system. They may also be packaged but only in an older version. The compilation script will tell you if one of them is missing or too old. In the worst case, you will have to download and compile bleeding edge version by hand.

If you want to install these dependencies locally, then use the libkiwix directory as install prefix.

Environment

The Libkiwix builds using Meson version 0.45 or higher. Meson relies itself on Ninja, pkg-config and few other compilation tools.

Install first the few common compilation tools:

These tools should be packaged if you use a cutting edge operating system. If not, have a look to the Troubleshooting section.

Compilation

Once all dependencies are installed, you can compile the Libkiwix with:

meson . build
ninja -C build

By default, it will compile dynamic linked libraries. All binary files will be created in the build directory created automatically by Meson. If you want statically linked libraries, you can add --default-library=static option to the Meson command.

Depending of you system, ninja may be called ninja-build.

The android wrapper uses deprecated methods of libkiwix so it cannot be compiled with werror=true (the default). So you must pass -Dwerror=false to meson:

meson . build -Dwrapper=android -Dwerror=false
ninja -C build

Static files compilation

Libkiwix has a few static files 'compiled' within the binary code. This is mostly Javascript/HTML/pictures necessary for the HTTP daemon.

These static files are available in the static directory and are compiled by custom Python code available in this repository scripts directory. This happens automatically at compilation time without any additional command to run.

To avoid HTTP caching issues, the URLs (to the static content) are appended with a cacheid parameter (this is called "cache busting"). This cacheid value derived from the sha1sum of each targeted static file. As a consequence, each time you change a static file, the corresponding cacheid value will change.

To properly test this feature, this cacheid needs to be added manually to the automated tests and has to be commited. After modifying the needed static file, run the automated tests. They will fail, but the inspection of the testing log will give you the new cacheid value(s). Finally update test/server.cpp with the appropriate cacheid value(s) which have changed.

Testing

To run the automated tests:

cd build
meson test

Installation

If you want to install the Libkiwix and the headers you just have compiled on your system, here we go:

ninja -C build install

You might need to run the command as root (or using sudo), depending where you want to install the libraries. After the installation succeeded, you may need to run ldconfig (as root).

Uninstallation

If you want to uninstall the Kiwix library:

ninja -C build uninstall

Like for the installation, you might need to run the command as root (or using sudo).

Custom Index Page

to use custom welcome page mention customIndexPage argument in kiwix::internalServer() or use kiwix::server->setCustomIndexTemplate(). (note - while using custom html file please mention all external links as absolute path.)

to create a HTML template with custom JS you need to have a look at various OPDS based endpoints as mentioned here to load books.

To use JS provided by kiwix-serve you can use the following template to start with ->

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width,initial-scale=1" />
    <title><-- Custom Tittle --></title>
    <script src="{{root}}/skin/isotope.pkgd.min.js" defer></script>
    <script src="{{root}}/skin/iso6391To3.js"></script>
    <script type="text/javascript" src="{{root}}/skin/index.js" defer></script>
  </head>
  <body>
  </body>
</html>
  • To get books listed using index.js add - <div class="book__list"></div> under body tag.
  • To get number of books listed add - <h3 class="kiwixHomeBody__results"></h3> under body tag.
  • To add language select box add - <select id="languageFilter"></select> under body tag.
  • To add category select box add - <select id="categoryFilter"></select> under body tag.
  • To add search box for books use following form -
        <form id='kiwixSearchForm'>
        <input type="text" name="q" placeholder="Search" id="searchFilter" class='kiwixSearch filter'>
        <input type="submit" class="kiwixButton" value="Search"/>
        </form>
    

If you compile manually Libmicrohttpd, you might need to compile it without GNU TLS, a bug here will impeach further compilation otherwise.

If the compilation still fails, you might need to get a more recent version of a dependency than the one packaged by your Linux distribution. Try then with a source tarball distributed by the problematic upstream project or even directly from the source code repository.

Troubleshooting

If you need to install Meson "manually":

virtualenv -p python3 ./ # Create virtualenv
source bin/activate      # Activate the virtualenv
pip3 install meson       # Install Meson
hash -r                  # Refresh bash paths

If you need to install Ninja "manually":

git clone git://github.com/ninja-build/ninja.git
cd ninja
git checkout release
./configure.py --bootstrap
mkdir ../bin
cp ninja ../bin
cd ..

License

GPLv3 or later, see COPYING for more details.

libkiwix's People

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

libkiwix's Issues

Meson: Unknown variable "ctpp2_include_path"

I tried upgrading kiwix-lib from a3d01b6 to 4485cc8 and I'm having a problem with Meson. I'm on 0.39.1, the latest available from pypi. I'm building on Debian Jessie. For reference, I had similar issues in #10.

The Meson build system
Version: 0.39.1
Source dir: /opt/app/kiwix-lib
Build dir: /opt/app/kiwix-lib/build
Build type: native build
Project name: kiwixlib
Native cpp compiler: c++ (gcc 4.9.2)
Build machine cpu family: x86_64
Build machine cpu: x86_64
Dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.28)
Native dependency icu-i18n found: YES 52.1
Native dependency libzim found: YES 1.4
Native dependency pugixml found: YES 1.8
Has header "ctpp2/CTPP2Logger.hpp": YES
Library ctpp2 found: YES
Program compile_resources.py found: YES (/opt/app/kiwix-lib/scripts/compile_resources.py)
Configuring kiwix_config.h using configuration

Meson encountered an error in file meson.build, line 87, column 5:
Unknown variable "ctpp2_include_path".

Please include a COPYING/LICENSE file

The files I skimmed have GPL file headers, please also include a full copy of the GPL (or whatever the appropriate license is) in a COPYING or LICENSE file. Thanks!

Unable to build kiwix-lib on Debian (packaging)

Hi,

I'm trying to build Debian package for kiwix-lib and have added libzim-dev libpugixml-dev and all other dependencies. But meson fails to find libzim

mkdir build && meson . build
WARNING: You are using 'ANSI_X3.4-1968' which is not a a Unicode-compatible locale.
WARNING: You might see errors if you use UTF-8 strings as filenames, as strings, or as file contents.
WARNING: Please switch to a UTF-8 locale for your platform.
The Meson build system
Version: 0.40.0
Source dir: /build/kiwix-lib-0.1
Build dir: /build/kiwix-lib-0.1/build
Build type: native build
Project name: kiwixlib
Native cpp compiler: c++ (gcc 6.3.0)
Build machine cpu family: x86
Build machine cpu: i686
Dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29)
Native dependency icu-i18n found: YES 57.1

Meson encountered an error in file meson.build, line 10, column 0:
Native dependency 'libzim' not found

If I comment ouf dependency(libzim) line in meson.build it then fails to find pugixml. More closer look on meson's build log indicates that its trying to find libzim using pkg-config but libzim package does not ship pkg-config file, same with pugixml. Which means probably it needs custom meson snippets to detect these library similar to ctpp2 snippet in current meson.build file.

Can you please help us to get it built properly?.

Suggestion seems to be buggy

Reported here https://sourceforge.net/p/kiwix/discussion/604121/thread/cbada626/?limit=25#a80b

In English Wikipedia, with full-text search off. Many examples:

1) If you type "List or Russian dishes" in the search, it doesn't show. But go to Russian cuisine#See also, and you see the article exists and is available by clicking. Also: "List of Russian " returns very few options, clearly buggy.

2) If you type "Château de ", there are many hits, but only from "Château de Bagatelle" to "Château de Clérans". But add a "B" or a "C" and the search now has no return. Almost no article will be findable by search if you input "Château de [A-Z]". A few will show up for weird reasons (Why does "Château de F" shows "Château de fieuzal", and nothing else?), and it's clear that the result has less hits than the real. Try it.

3) If you type "Soy tu d", only result is "Soy tu duena", which redirects to "Soy tu dueña". Why "Soy tu dueña" doesn't appear in the list? Then try also typing "Soy tu due" and adding the "ñ". The hit changes from "Soy tu duena" to "Soy tu dueña". I think "ñ" in search is matching "ñ" in results, but "n" in search is matching only "n", while it could also match "ñ".

4) ISO 7001 also cannot be searched by autocomplete in Wikipedia English, but article exists. Try it.

5) "Kelly Mc" returns many results in autocomplete, including "Kelly McBroom". But add a "B", and you see "Kelly McB" returns no results.

Might be a bug related to matching query results that have mixed case.

Please include a AUTHORS file

Hi,
It would be good to list down all the contributors to project in AUTHORS file and provide a software publication period in it. This will be useful for me to fill into the Copyright field of debian/copyright file as described here

Rewrite resourceTools.

The resourceTool allow to embed resource file in the binary itself.

When everything was in the same git repository, the tool itself and the resources (in static directory) where in the same place and so handled together.

But now, with the split of the repositories, the resourceTool is kiwix-lib and the resources should be with the program code (using the tool), ie kiwix-tools.

We have to found a way to store the resources in kiwix-tools and make the resources compiled automatically by meson when compiling the tools.

Fix stemming

Like for stopwords, this is already implemented but this should really be checked to be sure everything works fine.

Rethink how we integrate resources in binaries.

We have currently a small solution to embedded resource file in binaries.
The resource file is just store in a C array and the code just read it and interpret it as a file content.

This work well but it could be improved :

  • Some files/resources work together and can be seen as a pack. (stop words list for xapian, all html/css/js for the topbar, ...)
  • It is somehow complicated to allow customization (kiwix/kiwix-tools#17) or update of the resources. For now the two solutions are either to recompile the binary or use env variables to overwrite the content.

This is somehow a pity as we are a project who embedded different files in a read efficient archive, handling file hierarchy and so. We can probably use our own code base to store and read those resource files :)

Compilation errors on Ubuntu 14.04

I've tried to build kiwix-lib on Ubuntu 14.04 yet I get the following:

$ ~/bin/ninja 
[1/4] c++  -Isrc/kiwix@sha -Isrc -Istatic -I../src -Iinclude -I../include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/local/include/pugixml-1.8 -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -fPIC -pthread -MMD -MQ 'src/kiwix@sha/reader.cpp.o' -MF 'src/kiwix@sha/reader.cpp.o.d' -o 'src/kiwix@sha/reader.cpp.o' -c ../src/reader.cpp
FAILED: src/kiwix@sha/reader.cpp.o 
c++  -Isrc/kiwix@sha -Isrc -Istatic -I../src -Iinclude -I../include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/local/include/pugixml-1.8 -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -fPIC -pthread -MMD -MQ 'src/kiwix@sha/reader.cpp.o' -MF 'src/kiwix@sha/reader.cpp.o.d' -o 'src/kiwix@sha/reader.cpp.o' -c ../src/reader.cpp
../src/reader.cpp: In member function ‘bool kiwix::Reader::hasFulltextIndex() const’:
../src/reader.cpp:593:29: error: ‘class zim::File’ has no member named ‘is_multiPart’
         && !zimFileHandler->is_multiPart() );
                             ^
../src/reader.cpp: In member function ‘bool kiwix::Reader::searchSuggestionsSmart(const string&, unsigned int)’:
../src/reader.cpp:696:36: error: ‘class zim::File’ has no member named ‘suggestions’
       = this->getZimFileHandler()->suggestions(prefix, 0, suggestionsCount);
                                    ^
../src/reader.cpp: In member function ‘bool kiwix::Reader::hasFulltextIndex() const’:
../src/reader.cpp:594:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
^[[B[2/4] c++  -Isrc/kiwix@sha -Isrc -Istatic -I../src -Iinclude -I../include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/local/include/pugixml-1.8 -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -fPIC -pthread -MMD -MQ 'src/kiwix@sha/searcher.cpp.o' -MF 'src/kiwix@sha/searcher.cpp.o.d' -o 'src/kiwix@sha/searcher.cpp.o' -c ../src/searcher.cpp
FAILED: src/kiwix@sha/searcher.cpp.o 
c++  -Isrc/kiwix@sha -Isrc -Istatic -I../src -Iinclude -I../include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/local/include/pugixml-1.8 -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -fPIC -pthread -MMD -MQ 'src/kiwix@sha/searcher.cpp.o' -MF 'src/kiwix@sha/searcher.cpp.o.d' -o 'src/kiwix@sha/searcher.cpp.o' -c ../src/searcher.cpp
../src/searcher.cpp: In member function ‘void kiwix::Searcher::suggestions(std::string&, bool)’:
../src/searcher.cpp:232:13: error: ‘class zim::Search’ has no member named ‘set_suggestion_mode’
     search->set_suggestion_mode(true);
             ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_readerIndex()’:
../src/searcher.cpp:295:19: error: ‘zim::Search::iterator’ has no member named ‘get_fileIndex’
   return iterator.get_fileIndex();
                   ^
../src/searcher.cpp:296:1: warning: control reaches end of non-void function [-Wreturn-type]
 }
 ^
[3/4] c++  -Isrc/kiwix@sha -Isrc -Istatic -I../src -Iinclude -I../include -I/usr/include/x86_64-linux-gnu -I/usr/local/include -I/usr/local/include/pugixml-1.8 -pipe -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -Wnon-virtual-dtor -std=c++11 -O0 -g -fPIC -pthread -MMD -MQ 'src/kiwix@sha/manager.cpp.o' -MF 'src/kiwix@sha/manager.cpp.o.d' -o 'src/kiwix@sha/manager.cpp.o' -c ../src/manager.cpp
ninja: build stopped: subcommand failed.

I got libzim from http://www.openzim.org/download/libzim-2.0.0.tar.gz and Xapian by installing the libxapian-dev package.

Search title has html tags

Calling kiwix::Result::get_title sometimes results in a string with html tags.

For example, in wikipedia_en_medicine_novid_2017-08.zim, if search 't', you can get results with title like <I>Thermococcus</i>

Wrong redirection handling

Currently the kiwix-lib, if a redirect is found, search the target and return the target page body. It works so far the targeted page is at the same path level like the redirect. If not all relative URLs will probably fail to load properly.

As a workaround, in kiwix-serve a node as been introduced, but this is not the proper answer to that problem and this fixes only kiwix-serve.

To fix that problem, probably the kiwix-lib should returns a proper code or HTML redirection code.

Licensing kiwix-lib so it can be used by kiwix-apache

Kiwix-apache uses the Apache 2.0 license as it's primary audience is for users and contributors to the Apache Web Server project. It currently uses kiwix-lib. Collectively we would like to make sure that we've got compatible licenses for kiwix-lib, and if necessary it's dependencies so that both the Apache community and GPL communities accept that the licenses are compatible and acceptable to the various contributors of the respective projects.

Note: There are 15 contributors to kiwix-lib according to https://github.com/kiwix/kiwix-lib/graphs/contributors currently.

libxapian-dev on Debian Jessie doesn't seem to satisfy (optional) dependency

I notice that when I build, even having installed libxapian-dev via Debian Jesse, it says that the xapian-core dependency was not satisfied. It built anyway since it's an optional dependency, so perhaps the person writing the README did not notice. Or, maybe Debian Jesse has too old a version of libxapian-dev, such that it doesn't have the pkg-config thing.

kiwix-serve should propose widgets

For now there is no solution to export (part of) the ZIM library served by a kiwix-serve instance in a human readable fashion. For computers, we have the OPDS feed, but for humans there is nothing. If this is need, for the moment, there is only two solutions:

  • build an OPDS parser
  • maintain manually via a third party tool links to the content (but most of ZIM files are renewed monthly)
  • Just provide a simple link to the homepage of kiwix-serve (so no integrated experience)

This is not realistic to expect this to satsify all use cases like:

  • Our own library https://wiki.kiwix.org/wiki/Content_in_all_languages (currently updated via a dedicated script)
  • Our own Web Site which should advert the most proeminent ZIM files
  • Tools like Rachel, IIAB, OLIPS which might be interested to have a simpler solution to integrate the content
  • Any Web site which for example has its content scraped and would be interested to advert it
  • Basically any Web site which want to put an emphasis on a ZIM file or a set of ZIM files.

This is why, all these use-cases, would benefit of kiwix-serve providing a new-endpoint exporting a widget on the kiwix-serve home page.

Here a few ideas/properties of this new feature:

  • Widget should be laded in an iframe on client side
  • It should be possible to make a filter to select which ZIM files we want to display (reuse same system like already working on kiwix-serve)
  • It should be possible to decide if the widget content has the search filters or no (maybe just through CSS tweaking?)
  • Tiles should allow (but should be possible to disable) - if available at all - to view the content via a simple click on the tile. Then probably the content should open in a new tab (to be discussed).
  • Tiles should allow (but should be possible to disable) - if available at all - to download a ZIM file
  • Solution to integrate the widget should be easy, ie. the end-point should provide the HTML code via a simple click.
  • Documentation about all the options of this new feature should be written in the kiwix-tools documentation
  • It should be able to change the stylesheet via a custom CSS within the iframe
  • It should be possible to run a custom javascript to custom the DOM within the ifram

Copied from original issue: kiwix/kiwix-xulrunner#245

Does not seem to compile correctly in static

$ git pull
Already up-to-date.
(kiwix-tools) kelson@kelson-VirtualBox:~/kiwix-tools$ rm -rf build/ ; mkdir build ; meson --default-library=static . build
The Meson build system
Version: 0.38.0
Source dir: /home/kelson/kiwix-tools
Build dir: /home/kelson/kiwix-tools/build
Build type: native build
Project name: kiwix-tools
Native cpp compiler: c++ (gcc 5.4.0)
Build machine cpu family: x86_64
Build machine cpu: x86_64
Dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Native dependency kiwix found: YES 1.0
Native dependency libmicrohttpd found: YES 0.9.44
Native dependency zlib found: YES 1.2.8
Has header "ctpp2/CTPP2Logger.hpp": YES
Library ctpp2 found: YES
Program compile_resources.py found: YES (/usr/local/bin/compile_resources.py)
Build targets in project: 7
(kiwix-tools) kelson@kelson-VirtualBox:~/kiwix-tools$ cd build/
(kiwix-tools) kelson@kelson-VirtualBox:~/kiwix-tools/build$ ninja
[1/14] Compiling cpp object 'src/reader/kiwix-read@exe/kiwix-read.cpp.o'
../src/reader/kiwix-read.cpp: In function ‘int main(int, char**)’:
../src/reader/kiwix-read.cpp:91:18: warning: unused variable ‘contentLength’ [-Wunused-variable]
     unsigned int contentLength = 0;
                  ^
[3/14] Compiling cpp object 'src/searcher/kiwix-search@exe/kiwix-search.cpp.o'
../src/searcher/kiwix-search.cpp: In function ‘int main(int, char**)’:
../src/searcher/kiwix-search.cpp:109:12: warning: deleting object of abstract class type ‘kiwix::Searcher’ which has non-virtual destructor will cause undefined behaviour [-Wdelete-non-virtual-dtor]
     delete searcher;
            ^
../src/searcher/kiwix-search.cpp:38:8: warning: variable ‘verboseFlag’ set but not used [-Wunused-but-set-variable]
   bool verboseFlag = false;
        ^
../src/searcher/kiwix-search.cpp:41:20: warning: variable ‘backend’ set but not used [-Wunused-but-set-variable]
   supportedBackend backend = XAPIAN;
                    ^
[9/14] 'Generating resources with a custom command.'
jquery-ui/jquery-ui.structure.min.css ['jquery_ui', 'jquery_ui_structure_min_css']
jquery-ui/jquery-ui.min.js ['jquery_ui', 'jquery_ui_min_js']
jquery-ui/external/jquery/jquery.js ['jquery_ui', 'external', 'jquery', 'jquery_js']
jquery-ui/images/ui-bg_flat_0_aaaaaa_40x100.png ['jquery_ui', 'images', 'ui_bg_flat_0_aaaaaa_40x100_png']
jquery-ui/images/ui-bg_flat_75_ffffff_40x100.png ['jquery_ui', 'images', 'ui_bg_flat_75_ffffff_40x100_png']
jquery-ui/images/ui-icons_222222_256x240.png ['jquery_ui', 'images', 'ui_icons_222222_256x240_png']
jquery-ui/images/ui-bg_glass_55_fbf9ee_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_55_fbf9ee_1x400_png']
jquery-ui/images/ui-bg_highlight-soft_75_cccccc_1x100.png ['jquery_ui', 'images', 'ui_bg_highlight_soft_75_cccccc_1x100_png']
jquery-ui/images/ui-bg_glass_65_ffffff_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_65_ffffff_1x400_png']
jquery-ui/images/ui-icons_2e83ff_256x240.png ['jquery_ui', 'images', 'ui_icons_2e83ff_256x240_png']
jquery-ui/images/ui-icons_cd0a0a_256x240.png ['jquery_ui', 'images', 'ui_icons_cd0a0a_256x240_png']
jquery-ui/images/ui-icons_888888_256x240.png ['jquery_ui', 'images', 'ui_icons_888888_256x240_png']
jquery-ui/images/ui-bg_glass_75_e6e6e6_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_75_e6e6e6_1x400_png']
jquery-ui/images/animated-overlay.gif ['jquery_ui', 'images', 'animated_overlay_gif']
jquery-ui/images/ui-bg_glass_75_dadada_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_75_dadada_1x400_png']
jquery-ui/images/ui-icons_454545_256x240.png ['jquery_ui', 'images', 'ui_icons_454545_256x240_png']
jquery-ui/images/ui-bg_glass_95_fef1ec_1x400.png ['jquery_ui', 'images', 'ui_bg_glass_95_fef1ec_1x400_png']
jquery-ui/jquery-ui.theme.min.css ['jquery_ui', 'jquery_ui_theme_min_css']
jquery-ui/jquery-ui.min.css ['jquery_ui', 'jquery_ui_min_css']
home.html.tmpl ['home_html_tmpl']
include.html.part ['include_html_part']
taskbar.css ['taskbar_css']
taskbar.html.part ['taskbar_html_part']
[12/14] Compiling cpp object 'src/server/kiwix-serve@exe/kiwix-serve.cpp.o'
../src/server/kiwix-serve.cpp: In function ‘MHD_Response* build_response(const void*, unsigned int, const string&, const string&, bool, bool)’:
../src/server/kiwix-serve.cpp:200:36: warning: ‘MHD_Response* MHD_create_response_from_data(size_t, void*, int, int)’ is deprecated [-Wdeprecated-declarations]
   struct MHD_Response * response = MHD_create_response_from_data(length,
                                    ^
In file included from ../src/server/kiwix-serve.cpp:74:0:
/usr/include/microhttpd.h:2022:1: note: declared here
 MHD_create_response_from_data (size_t size,
 ^
../src/server/kiwix-serve.cpp:200:36: warning: ‘MHD_Response* MHD_create_response_from_data(size_t, void*, int, int)’ is deprecated [-Wdeprecated-declarations]
   struct MHD_Response * response = MHD_create_response_from_data(length,
                                    ^
In file included from ../src/server/kiwix-serve.cpp:74:0:
/usr/include/microhttpd.h:2022:1: note: declared here
 MHD_create_response_from_data (size_t size,
 ^
../src/server/kiwix-serve.cpp:203:73: warning: ‘MHD_Response* MHD_create_response_from_data(size_t, void*, int, int)’ is deprecated [-Wdeprecated-declarations]
                                                                  MHD_YES);
                                                                         ^
In file included from ../src/server/kiwix-serve.cpp:74:0:
/usr/include/microhttpd.h:2022:1: note: declared here
 MHD_create_response_from_data (size_t size,
 ^
../src/server/kiwix-serve.cpp: In function ‘MHD_Response* handle_random(MHD_Connection*, int&, kiwix::Reader*, kiwix::Searcher*, const string&, const string&, bool)’:
../src/server/kiwix-serve.cpp:429:8: warning: unused variable ‘cacheEnabled’ [-Wunused-variable]
   bool cacheEnabled = false;
        ^
../src/server/kiwix-serve.cpp: In function ‘MHD_Response* handle_content(MHD_Connection*, int&, kiwix::Reader*, kiwix::Searcher*, const string&, const string&, bool)’:
../src/server/kiwix-serve.cpp:452:16: warning: unused variable ‘contentLength’ [-Wunused-variable]
   unsigned int contentLength;
                ^
../src/server/kiwix-serve.cpp: In function ‘int accessHandlerCallback(void*, MHD_Connection*, const char*, const char*, const char*, const char*, size_t*, void**)’:
../src/server/kiwix-serve.cpp:596:14: warning: unused variable ‘acceptRange’ [-Wunused-variable]
   const bool acceptRange = acceptRangeHeaderValue != NULL;
              ^
../src/server/kiwix-serve.cpp: In function ‘int main(int, char**)’:
../src/server/kiwix-serve.cpp:959:10: warning: unused variable ‘host’ [-Wunused-variable]
     char host[NI_MAXHOST];
          ^
[14/14] Linking target src/server/kiwix-serve
(kiwix-tools) kelson@kelson-VirtualBox:~/kiwix-tools/build$ ./src/server/kiwix-serve
./src/server/kiwix-serve: error while loading shared libraries: libkiwix.so: cannot open shared object file: No such file or directory

I conclude that it is not build statically if it needs libkiwix.so

Compiler errors, seemingly related to zim

I jumped ahead and tried the code based on this fix: #51

It appears to sit on top of master, so I don't think I'm missing any more recent fixes. I also updated openzim, I think to the latest release (though perhaps I didn't point to the tip of master). Here's what I get:

Compiler Errors (click to expand)
../src/searcher.cpp:40:48: error: ‘zim::Search::iterator’ has not been declared
       _Result(Searcher* searcher, zim::Search::iterator& iterator);
                                                ^
../src/searcher.cpp:52:20: error: ‘iterator’ in ‘class zim::Search’ does not name a type
       zim::Search::iterator iterator;
                    ^
../src/searcher.cpp:57:18: error: ‘iterator’ in ‘class zim::Search’ does not name a type
     zim::Search::iterator current_iterator;
                  ^
../src/searcher.cpp: In member function ‘void kiwix::Searcher::search(std::string&, unsigned int, unsigned int, bool)’:
../src/searcher.cpp:119:62: error: ‘class zim::File’ has no member named ‘search’
       internal->_search = this->reader->getZimFileHandler()->search(unaccentedSearch, resultStart, resultEnd);
                                                              ^
../src/searcher.cpp:120:17: error: ‘struct kiwix::SearcherInternal’ has no member named ‘current_iterator’
       internal->current_iterator = internal->_search->begin();
                 ^
../src/searcher.cpp:120:55: error: ‘const class zim::Search’ has no member named ‘begin’
       internal->current_iterator = internal->_search->begin();
                                                       ^
../src/searcher.cpp:121:55: error: ‘const class zim::Search’ has no member named ‘get_matches_estimated’
       this->estimatedResultCount = internal->_search->get_matches_estimated();
                                                       ^
../src/searcher.cpp: In member function ‘void kiwix::Searcher::restart_search()’:
../src/searcher.cpp:128:15: error: ‘struct kiwix::SearcherInternal’ has no member named ‘current_iterator’
     internal->current_iterator = internal->_search->begin();
               ^
../src/searcher.cpp:128:53: error: ‘const class zim::Search’ has no member named ‘begin’
     internal->current_iterator = internal->_search->begin();
                                                     ^
../src/searcher.cpp: In member function ‘kiwix::Result* kiwix::Searcher::getNextResult()’:
../src/searcher.cpp:132:19: error: ‘struct kiwix::SearcherInternal’ has no member named ‘current_iterator’
     if (internal->current_iterator != internal->_search->end()) {
                   ^
../src/searcher.cpp:132:58: error: ‘const class zim::Search’ has no member named ‘end’
     if (internal->current_iterator != internal->_search->end()) {
                                                          ^
../src/searcher.cpp:133:52: error: ‘struct kiwix::SearcherInternal’ has no member named ‘current_iterator’
       Result* result = new _Result(this, internal->current_iterator);
                                                    ^
../src/searcher.cpp:134:17: error: ‘struct kiwix::SearcherInternal’ has no member named ‘current_iterator’
       internal->current_iterator++;
                 ^                                                                                                                                                                   [99/1698]
../src/searcher.cpp: At global scope:
../src/searcher.cpp:167:53: error: ‘zim::Search::iterator’ has not been declared
   _Result::_Result(Searcher* searcher, zim::Search::iterator& iterator):
                                                     ^
../src/searcher.cpp: In constructor ‘kiwix::_Result::_Result(kiwix::Searcher*, int&)’:
../src/searcher.cpp:169:5: error: class ‘kiwix::_Result’ does not have any field named ‘iterator’
     iterator(iterator)
     ^
../src/searcher.cpp: In member function ‘virtual std::string kiwix::_Result::get_url()’:
../src/searcher.cpp:174:20: error: missing template arguments before ‘.’ token
     return iterator.get_url();
                    ^
../src/searcher.cpp: In member function ‘virtual std::string kiwix::_Result::get_title()’:
../src/searcher.cpp:178:20: error: missing template arguments before ‘.’ token
     return iterator.get_title();
                    ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_score()’:
../src/searcher.cpp:182:20: error: missing template arguments before ‘.’ token
     return iterator.get_score();
                    ^
../src/searcher.cpp: In member function ‘virtual std::string kiwix::_Result::get_snippet()’:
../src/searcher.cpp:186:20: error: missing template arguments before ‘.’ token
     return iterator.get_snippet();
                    ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_size()’:
../src/searcher.cpp:190:22: error: missing template arguments before ‘.’ token
       return iterator.get_size();
                      ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_wordCount()’:
../src/searcher.cpp:194:22: error: missing template arguments before ‘.’ token
       return iterator.get_wordCount();
                      ^
../src/searcher.cpp: In member function ‘virtual std::string kiwix::_Result::get_url()’:
../src/searcher.cpp:175:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
../src/searcher.cpp: In member function ‘virtual std::string kiwix::_Result::get_title()’:
../src/searcher.cpp:179:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_score()’:
../src/searcher.cpp:183:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
../src/searcher.cpp: In member function ‘virtual std::string kiwix::_Result::get_snippet()’:
../src/searcher.cpp:187:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_size()’:
../src/searcher.cpp:191:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
../src/searcher.cpp: In member function ‘virtual int kiwix::_Result::get_wordCount()’:
../src/searcher.cpp:195:3: warning: control reaches end of non-void function [-Wreturn-type]
   }
   ^
[7/14] Compiling cpp object 'src/kiwix@sha/reader.cpp.o'
FAILED: src/kiwix@sha/reader.cpp.o 
c++  '-Isrc/kiwix@sha' '-fdiagnostics-color=always' '-Isrc' '-I../src' '-Iinclude' '-I../include' '-pipe' '-Wall' '-Winvalid-pch' '-Wnon-virtual-dtor' '-std=c++11' '-O0' '-g' '-fPIC' '-I/usr
/include/x86_64-linux-gnu' '-I/usr/local/include' '-I/usr/local/include/pugixml-1.8' '-pthread' '-Istatic' '-MMD' '-MQ' 'src/kiwix@sha/reader.cpp.o' '-MF' 'src/kiwix@sha/reader.cpp.o.d' -o '
src/kiwix@sha/reader.cpp.o' -c ../src/reader.cpp
../src/reader.cpp: In member function ‘std::map<const std::basic_string<char>, unsigned int> kiwix::Reader::parseCounterMetadata() const’:
../src/reader.cpp:106:45: error: no matching function for call to ‘std::basic_stringstream<char>::basic_stringstream(zim::Blob)’
     stringstream ssContent(article.getData());
                                             ^

Easy download of kiwix-lib dependences should be added to kiwix-lib

kiwix-lib should provide a way to download necessary dependences "quickly" (with an adhoc solution) from http://download.kiwix.org/dev/. This is the case with the "old" Kiwix download script and this simplify a lot the static compilation of kiwix(-lib). As a remember static0compilation is the "standard" compilation, non-static compilation is probably almost only needed for GNU/Linux distribution.

It seems that most of this code has been done in the kiwix-tools repo (for a reason which I ignore). This seems wrong to me, this should be done at kiwix-lib level, non kiwix-lib code should only bother of the kiwix-lib, not of kiwix-lib dependences.

See a related ticket in kiwix-tools kiwix/kiwix-tools#8

Check search pertinence

A small investigation on the overall search engine pertinence to see if a few things can not be quickly be improved.

Merge suggestions and ft search

To find an article with have two methods:

  • Suggestions, based on a search in a sorted list of title.
  • Fulltext

This ticket is about to provide a solution which (if the ft index is there) to give a list of best match. The whole system should be better than the current one (so return everything in one shot as an array). It should also allow to make case insensitive search.

Add document to the Searcher API.

Searcher is a bit complex and methods do not speak by themselves.
I don't know who write this (probably me :) ), but it was a bad thing.

Fix stopwords

Stopwords are words which should not be indexed (during the FT index process) and also be ignore during the FT search. This stopwords are language specific. Lists are provided by Xapian and are at least used in zimwriterfs. This task is about to check that everything works fine on the indexing part but also on the reader side.

Rethink a bit the searcher API about readable names.

Readable names are used only by the getHtml method.
But this method is used only by kiwix-serve. It is totally useless for android or apple systems and so readable names are nonsense in these cases.

We should improve the API on this.
Maybe remove the getHtml method and move elsewhere. kiwix-serve seems to be a good place but we want to move things from kiwix-serve to kiwix-lib, so maybe not. A "formater" using a searcher as backend is maybe the solution.

Index files too large for Fat32

Even though the zim files are divided so that I can copy them to Fat32 drives/SD cards, the index files aren't. Therefore, I cannot perform a full-text-search on Kiwix for Android. I wonder if I can divide those index files myself. Will Kiwix recognize them?

Abbreviations cannot be searched anymore

I'm using Kiwix 2.2
Searching for abbreviations (like FFT) doesn't work anymore.
Some versions ago (2.1 or 1.97) it worked properly.
Maybe refs #66
(Maybe Disambiguation sites not working anymore is the underlying reason)

Problem compiling kiwix-lib (with ctpp2)

On Ubuntu Xenial

$ meson ..
The Meson build system
Version: 0.29.0
Source dir: /home/kelson/kiwix-lib
Build dir: /home/kelson/kiwix-lib/build
Build type: native build
Build machine cpu family: x86_64
Build machine cpu: x86_64
Project name: kiwixlib
Native cpp compiler: c++ (gcc 5.4.0-6ubuntu1)
Dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29.1)
Native dependency icu-i18n found: YES 55.1
Native dependency libzim found: YES 1.4
Native dependency pugixml found: YES 1.8
Program aria2c found: YES (/usr/bin/aria2c)
Has header "ctpp2/CTPP2Logger.hpp": YES

Meson encountered an error in file meson.build, line 24, column 2:
Unknown method "find_library" in object.

Improve kiwix-lib README

README should clearly explain the compilation procedure. This means step-by-step procedures which can be follow blindly with short explanation about what is going on:

  • What are the dependencies (with minimal version)
  • How to compile under Linux static/dynamic
  • Where to go to compile under Windows & if user wants to download locally the deps

Multizim search support

so the search feature should allow to make search an all or a subset of all open ZIM files and give the results. This probably implies to add multi-zim support to the lib

Unable to build using meson due to pugixml dependency

I'm pretty new to meson, so I'm not sure if I'm doing something totally wrong...but I can't get this to build with meson. Here's what I did:

# apt install build-essential liblzma-dev libicu-dev libpugixml-dev
# Install libzim with https://gerrit.wikimedia.org/r/#/c/329759/ so pkg-config knows about it
# Edit meson.build to use "libzim" instead of "zimlib" (will submit a PR momentarily)
# mkdir build
# meson . build
The Meson build system
Version: 0.37.1
Source dir: /home/km/projects/kiwix-lib
Build dir: /home/km/projects/kiwix-lib/build
Build type: native build
Project name: kiwixlib
Native cpp compiler: c++ (gcc 6.2.1)
Build machine cpu family: x86_64
Build machine cpu: x86_64
Dependency threads found: YES
Found pkg-config: /usr/bin/pkg-config (0.29)
Native dependency icu-i18n found: YES 57.1
Native dependency libzim found: YES 1.4

Meson encountered an error in file meson.build, line 10, column 0:
Native dependency 'pugixml' not found

pugixml is definitely installed, and I know that meson uses pkg-config to find dependencies, so I filed zeux/pugixml#128, but in the meantime, how do I build this?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.