brown-university-library / iip-production

Overview

This is the code for the production iip project.

---

Installation

• make a dir iip_webapp_stuff
• cd into that dir
• $ python3 -m venv ./env3_iip_web
• $ git clone https://github.com/Brown-University-Library/iip_production.git ./iip_web_project
• $ mkdir ./settings
• $ mkdir ./logs
• $ mkdir ./db
• $ mkdir ./pull_from_github
• $ cd ./iip_web_project
• $ source ../env3_iip_web/bin/activate
• $ pip install --upgrade pip
• $ pip install -r ./config/requirements.txt
• put submitted settings file in the settings dir
  • watch for line-endings
• put submitted db file in the db dir
• point the env/bin/activate file to the envar_settings.sh file
  • give lines for bottom
  • give deactivate line
• go through the settings file, updating all paths
• $ python ./manage.py check
• $ python ./manage.py runserver

iip-production's Issues

Current Location display and solr field

Add a label "Current Location" underneath "Place Found" label. This will be followed by the contents of the solr "provenance" field. Assume only one provenance now.

Elli creates this in solr.

bold target word in the kwic view

the word whose lemma is being viewed should be bold in the KWIC display.

images not showing up as expected (3days?)

Images that have been added to new inscriptions aren't showing up. We know that we cannot display multiple images at the moment, however, there is a strange behavior that may have to do with how images are identified for an inscription. Until recently, we only had single images for a few inscriptions. The image names were the same as the file name/inscription ID. Although we index image captions, we don't index image names in our Solr indexing script. This makes me think that the code assumes not only that there is only 1 image per file, but that its name is the same as the file name, but with a .jpg extension.

We now have inscriptions with multiple images, so there are images named sepp0018a.jpg and not just sepp0018.jpg This means that the name of the image doesn't always match the name of the file or the inscription ID. For inscriptions for whom there exists at least one image in our image folder that has the same name as the inscription file/ID, the image displays. This occurs even when the vanilla image file name is the second item. (https://library.brown.edu/iip/viewinscr/sepp0020/)

In cases where both image file names differ from the inscription ID/file name, the image fails to show up and the noimg.png file is shown instead. https://library.brown.edu/iip/viewinscr/jeru0552/

Examples:

1. inscription that has one image that is the same as the inscription file name/ID, where that is the image that shows up. https://library.brown.edu/iip/viewinscr/arch0002/ (This is expected behavior)
2. We don't have any inscriptions with only one image that has a name different from the IIP ID.
3. inscription with two images both of whose names differ from the inscription file name/ID (no images are displayed in this case) https://library.brown.edu/iip/viewinscr/anna0001/
4. inscription with two images one whose names differ from the inscription file name/ID. The image that has the same name as the inscription ID displays.(https://library.brown.edu/iip/viewinscr/sepp0020/)

Need to:

1. add indexing for file names, and also figure out syntax for multiple image names and image captions
2. change code to use actual file names
3. change code to handle multiple images (there is another issue on this)

Item counts on search page are incorrect

The search page on the new site is doing various incorrect things.

• Many Location category items display a 0 count of inscriptions. They seem to return correct result counts, however, when selected and searched.
• Type of Inscription and Physical Type display non-0 counts on the search page but return a different count.. For example, "boundary marker" says it has 22 results, but returns 30.
• Language and Religion return a different count than what the search page shows.
• Material returns everything in the database, because the term doesn't seem to be transmitted. This is the status line for results of a search on an item in the Material category:
  Your search for display status:approved AND * yielded 3929 results
• The full text search, and metadata search seem to work

redirects for new site still need to be done

http://cds.library.brown.edu/projects/Inscriptions/ needs to be redirected

Idumea doesn't appear in the list of locations on the main search page

This is probably because it has no geographical information - no Pleiades ID or coordinates. Should check to see what solr facet generates this information. or if it's because of how the main page is generated.

incorporate word-list work into production

done -- just adding this with awaiting-review label to track it.

Zotero

There is a Zotero account and a Zotero group for IIP. The old site seems to be using the Zotero account API and the new one is using the Zotero group API call (to the group).

They should all be using the Zotero group. I think you need to change the old site.

This is used in the page that displays the inscription. (fiull page and possibly results list)

make language & part-of-speech selections bookmarkable

Add the rest of the grammatical information when displaying oblique form

When a particular form is shown from the word list, it should display part of speech and the rest of parsed information, for example, number, gender, case, etc.

List of words on main page: formatting

Words should be bold, but all lower case.

css and CETEIcean should live in git and update on push (3days?)

there may be more complications with this one, but like the solr xsl, the CSS and perhaps the js library for CETEIcean should also be part of the iip-texts git repo and update on push as the inscription text do already.

facets not working properly

I did a search for "funerary" or "funerary.epitaph" or "funerary.malediction" and have 1670 results. The URL string is:
https://library.brown.edu/iip/mapsearch/?q=(type:funerary%20OR%20type:funerary.epitaph%20OR%20type:funerary.malediction)

On the results page, I tried to select the facet "jewish" and then had NO results. Because the search string was rewritten as
display status:approved AND type:funerary AND type:funerary.epitaph AND type:funerary.malediction AND religion:jewish

The search is now looking for the intersecton of all the items, and not the union of the first three and the intersection of the last one. When I change the first two "AND" values in the URL string to "OR" it works as expected.

The rewrite should append the new value to the search string with AND, but it should preserve what's already there.

Add Doubletree view back into the wordlist functionalities

perhaps have some buttons next to each lemma, 1 for the dictionary, 1 for the doubletree view.

Finish Location/Provenance Display handling

This is related to #51 - this created a new solr field called provenance and displayed it separately from the original location information. This issue is a request to clean up the display.

Currently, the full inscription display, for example, https://library.brown.edu/iip/viewinscr/caes0305/ , shows location metadata like this:
Place Found: Caesarea, Coastal Plain.
Current Location: Caesarea Museum

Place Found: Caesarea (Coastal Plain)

Current Location: Caesarea Museum

where the first line comes from the Solr field called "place_found" which used to have extra provenance information appended to it because there was no provenance field. That extra text "Current location: Caesaria Museum" has now been removed in the Solr document. That line should remain; its content is being managed in Solr.

The 2nd line is extraneous, and shouldn't be here at all. It comes from the Solr field "placeMenu" and seems to have been introduced by mistake.

The 3rd line is the new Solr field called "provenance" and appears as it should.

So: the change requested in this issue is to remove the 2nd line which derives from the "placeMenu" field.

word lists are missing words

The word list, in production is missing words from most inscriptions. For example:
jeru0491:
online view https://library.brown.edu/iip/viewinscr/jeru0491/
the word list has 4 words from this inscription:
https://github.com/Brown-University-Library/iip-word-lists/blob/master/new-version/Step%204%20New%20Output.csv (search for jeru0491 in the file)

The table is correct in the line number and word count that it has for each word. It's just that other words and lines are missing. I looked in the step 1,2,3 output files and the words are all there. Something goes wrong when step 4 output is created.

Bibliography display small chunk needs removal (1day?)

https://dlibwwwcit.services.brown.edu/iip/viewinscr/abur0001/

XXXI needs to be removed also

wordlist url to inscription only works on local-development

• problem: the 'runserver' environment assumes a root url segment, but the linux servers do not. This means that urls have to be crafted in such a way as to work in both environments.
• to replicate issue (assuming wordlist branch is active):
  • go to https://dlibwwwcit.services.brown.edu/iip/wordlist/
  • select 'Latin'
  • click the plus symbol next to a word, eg AC (1)
  • attempt to click on the link for caes0009
  • the link fails because the root url segment iip is missing.
• solution: use 'url' template-tag.
• example of a template-tag where a parameter is passed to it.
• django url templatetag documentation

---

New parsed Latin Spreadsheet

This is the file with the corrected POS information
https://docs.google.com/spreadsheets/d/1k6pADGfsEJkPff-bXftkVeORFJ_NbcTJFzDBPYil0f8/edit?usp=sharing

landing-page map doesn't display

GanttStart: 2020-11-19
GanttDue: 2021-02-01

##GanttDue: 2020-11-21

• The search landing-page isn't displaying maps on load.

  • Note: clicking the Satellite View overlay works (as do the other overlays).

• Briefly looking into this, it appears the js calls to mapbox.com, to get the tiles, are failing.

• This example call yields the response:

  {"message":"Classic styles are no longer supported; see https://blog.mapbox.com/deprecating-studio-classic-styles-d8892ac38cb4 for more information"}

---

results page error

• example failing url

partial word searches

right now, searches seem to only work for whole words. We'd like to be able to search for parts of words, for ex. "bask" should also get "basket"

the two transcription displays are different

• The transcription on the initial results page is different from the transcription on the inscription page
  • (look at "Imp eratori ")
• Goal: document how each transcription is being built
• Next goal: make the results transcription look like and work like the actual inscription-page transcription

define good query for the to_approve items.

stories page error

• example url

ability to re-index all inscriptions

can be link or URL.

Missing bib info

• Bibliography is missing (displays nothing) for some inscriptions even though the data is present in xml file (i.e. masa0772).

• A cause: Bibliography for inscriptions with wrong format in xml tag “bibl” : [“bibl=IIP-xxx|nType=|n=834”] do not show up. Instead, it seems that it needs proper format “bibl” : [“bibl=IIP-xxx.xml|nType=|n=834”].


• Examples

  • masa0772 example of Inscription with error
  • abil0001 example of Inscription with no error

• Note: error does not appear in production because the new code extracts some bib info in a different way, using regex which does not account for the data inconsistency.

doubletree url not working on deploy-servers

The doubletree url works locally in the runserver environment, but not on the dev/production servers.

Example:

• on the dev-server clicking the tree icon for agrippa yields the url: https://dlibwwwcit.services.brown.edu/wordlist/latinword/doubletree/agrippa/N
• that's missing the iip url-segment; adding that segment makes the url work properly.

This is probably because the runserver environment assumes the root iip part of the segment; the deployment-servers do not. The way around that is to use, in code, the syntax reverse( urls.py desired-url-root) -- or, in a template, the syntax <a href="{% url 'desired-url-root' %}">

---

small map tiles also not working

On pages with a list of inscriptions (search results) and on individual inscription pages, there is a small map image showing the location of the inscription. This is also not displaying the map tile, just a dot on a white background. Could change this to the satellite tile as with the map on the main search page.

Search results:
https://library.brown.edu/iip/mapsearch/?q=(physical_type:fragment)

Single inscription:
https://library.brown.edu/iip/viewinscr/caes0064/

upload the new p.o.speech CSV

re-enable delete-orphans for solr-index.

images are being referenced from dev server

On this page (and perhaps others) https://library.brown.edu/iip_development/about/why_inscription/
the image at the top of the page is being served from
https://dlibwwwcit.services.brown.edu/django_media/iip_student_dev_media/about/img/about1.jpg
and is not visible when outside Brown. It should be in the prod, not dev server. The image is available here:
https://library.brown.edu/django_media/iip_student_dev_media/about/img/about1.jpg
The URL is pointing to the wrong server however.

Is it possible to grep for other such image links in the new IIP?

proofreading workflow - change status using Solr dashboard

Birkin investigate if this is feasible.

change URLs used for search/facet API from old site to current site

title says it all.

transcription and diplomatic divs with empty elements as contents don't display content

For ex: in .https://dlibwwwcit.services.brown.edu/iip/viewinscr/masa0295/
The contents of the transcription are:

<div type="edition" subtype="transcription" ana="b1" xml:lang="heb">
           <p>
               <g type="phoen-YOD"/><g type="phoen-NUN"/>
            </p>
</div>

the display shows [no transcription]

This is generated from one of the following files (found by grepping recursively for "no transcription"

./static/mapsearch/js/result_display.js:                $(domTarget).find(".transcription").append("<tei-div>[no transcription]</tei-div>");

./static/mapsearch/js/result_display.js:                $(domTarget).find(".transcription").html("[no transcription]");

./static/iip_search_app/js/result_display.js:                $(domTarget).find(".transcription").append("<tei-div>[no transcription]</tei-div>");

./static/iip_search_app/js/result_display.js:                $(domTarget).find(".transcription").html("[no transcription]");
./templates/iip_search_templates/results.html:                                    [no transcription]

Apparently the code is testing for a <div> element with no text() or spaces in it. However, the <g> elements generate content - they are standing in for non-displayable elements. In other cases, where there is a non-space text node, the <g> elements show up, because all content is pulled from the source file. See
https://dlibwwwcit.services.brown.edu/iip/viewinscr/masa0301/
This is probably true for diplomatic divs as well.

default noimage.jpg being loaded from old repository (2hrs?)

On pages like this one: https://library.brown.edu/iip/viewinscr/caes0064/
There is no image of the inscription, and we display a default line drawing "Image not yet available." The jpg file is being loaded from: https://raw.githubusercontent.com/Brown-University-Library/iip-production/zak/iip_smr_web_app/static/resources/img/noimg.png

I think this is a different branch of the code tree. The correct image also exists in the main branch, in the corresponding directory.

Need to change the code to point to the image in the main branch. One place it is used is:
/iip_smr_web_app/templates/iip_search_templates/viewinscr.html line 54

The image is also used on a search results page: https://library.brown.edu/iip/mapsearch/?q=(physical_type:fragment)
On this page, the image is being called from a different, also out of date directory: https://library.brown.edu/django_media/iip_student_dev_media/resources/img/noimg.png

I think that both should be displaying the image that is in: iip_smr_web_app/static/resources/img/noimg.png

To find other place (if any) that this image is used, grep codebase for string "noimg.png."

Analytics in new site

add google analytics code to some header or footer for the new site. EM check to make sure that we can differentiate the sites in the same analytics account.

solr indexing xsl file update on push

Need to make sure that the xsl file used for solr indexing is part of the git repo - it should be one of the files that Elli and others can edit and push to update on the server. (maybe it already does this - pls check)

test-issue

GanttStart: 2020-11-01
GanttDue: 2020-11-20

Image URLs to change in order to make the old site display properly

change all files so URL reads ...projects/iip-images/... instead of ...projects/Inscriptions/images/...

Page: https://library.brown.edu/cds/projects/iip/info/about/
http://cds.library.brown.edu/projects/Inscriptions/images/sun_floor.jpg
http://cds.library.brown.edu/projects/Inscriptions/images/arches.jpg
http://cds.library.brown.edu/projects/Inscriptions/images/mosaic1.jpg
http://cds.library.brown.edu/projects/Inscriptions/images/mosaic2.jpg

Page https://library.brown.edu/cds/projects/iip/info/research-and-teaching/
http://cds.library.brown.edu/projects/Inscriptions/images/carv2.jpg

Page https://library.brown.edu/cds/projects/iip/info/related-sites/
http://cds.library.brown.edu/projects/Inscriptions/images/floor_circle.jpg
http://cds.library.brown.edu/projects/Inscriptions/images/mosaic3.jpg

Page https://library.brown.edu/cds/projects/iip/info/bibliography/
http://cds.library.brown.edu/projects/Inscriptions/images/israel_quad.png

investigate iip-processing hang

explore solution for dev only data flow

so for ex. only update dev when changes are made on a branch. Update Production when changes are made on master branch.

jaff0035 image not coming in as thumbnail

covers whole screen

new proofreading views

The old site where proofreading works will be more and more out of date as we improve the display in the new site. Need to figure out how to proofread.

inscription-page css thought

• consider moving the main css file for an inscription page (tei_custom_elements.css') from being a part of this repo to being a part of a data-repo, so that a wider group of people can trigger automatic updates.

New html for photo captions

Add underneath photo, and use solr field "photo_caption"

Elli makes xsl to generate that field.

check iso language codes and make them all consistent in Data and Code

title says it all.

link each lemma to a dictionary online, or pull definition in via API

The lemma should also have a link to a dictionary. @emylonas needs to look into best site to link to or to use API.

optimize map loading on main search page

The page is non-functional until all the facet data and the map have loaded. This is especially burdensome in places with slow net connections. It would be great to figure out a way to optimize the loading.

inscriptions from idum... are not showing up at all in location lists.

test with idum0044
no Idumea in location list on search page, and if you look for all inscriptions, and try the facets for settlement (city) then it doesn't show up. It appears in the old site. You can see the inscription if you just type the URL https://library.brown.edu/iip_development/viewinscr/idum0044/