natlibfi / kivepa-booksearch-workshop

An application which was used at the workshop of automated indexing of fiction books at Kirjastoverkkopäivät 2023.

THIS REPOSITORY HAS BEEN ARCHIVED

Booksearch application for workshop at Kirjastoverkkopäivät 2023

An application which was used at the workshop of automated indexing of fiction books at Kirjastoverkkopäivät 2023.

Developed with help by ChatGPT.

See the article about the workshop: Lehtinen M., Inkinen J. & Suominen O. (2023). Kaunokirjallisuuden automaattinen kuvailu – Tarinaluotsi vs. Juonenjyvä. Tietolinja, 2023(2). Pysyvä osoite: https://urn.fi/URN:NBN:fi-fe20231218155441

Set up local environment

python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Get subjects for books from Annif

time python get-annif-subjects.py

Local development

Set up Elasticsearch and indices

docker run --rm --net elastic --name es-node01 -p 9200:9200 -p 9300:9300 -e "xpack.security.enabled=false" -e "discovery.type=single-node" -v es-data:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:8.9.1

# curl -X DELETE "localhost:9200/books"  # Delete existing index
time python import-books-to-es.py &> import-books.out  # Takes 5 min for 1000 books, 15 min for 4000 books, etc.

# curl -X DELETE "localhost:9200/autocomplete"  # Delete existing index
python create-autocomplete-index.py

Alternatively use Elasticsearch running in OpenShift

export ELASTICSEARCH_URL=https://kvp-2023-workshop-elasticsearch.apps.kk-test.k8s.it.helsinki.fi:443

Start application

flask run --debug
# gunicorn app:app  # Alternatively use gunicorn

Deploy in OpenShift

The application will be publicly available at the address given in route template.

If this repository is private a deploykey needs to be in place as a secret in Openshift.

Install/update application

helm upgrade --install kvp-2023-workshop helm-charts/
# oc start-build kvp-2023-workshop-app  # Rebuild Docker image

Create indices

# curl -X DELETE "https://kvp-2023-workshop-elasticsearch.apps.kk-test.k8s.it.helsinki.fi:443/books"  # Delete old
# curl -X DELETE "https://kvp-2023-workshop-elasticsearch.apps.kk-test.k8s.it.helsinki.fi:443/autocomplete"  # Delete old

time python import-books-to-es.py https://kvp-2023-workshop-elasticsearch.apps.kk-test.k8s.it.helsinki.fi:443 &> import-books.out
python create-autocomplete-index.py https://kvp-2023-workshop-elasticsearch.apps.kk-test.k8s.it.helsinki.fi:443

Database resetting and copying to local machine for analysis

# oc rsh deployment/kvp-2023-workshop-app rm sqlite3-data/database.db  # Delete
oc rsync <pod-name>:sqlite3-data/ ./os-sqlite3-data

Elasticsearch queries

curl http://localhost:9200/_aliases
curl http://localhost:9200/books?pretty
curl http://localhost:9200/books/_count
# Note: Search returns only 10 results by default
curl localhost:9200/books/_search?pretty -H "Content-Type: application/json" -d '{
  "query": {
    "match": {
      "title": "Kissa"
    }
  }
}'