ghini / server

ghini reloaded - merging desktop & web

ghini.server

Ghini is a suite of software programs, helping the management and presentation of botanical data. It aims at botanical gardens, plant collectors, nurseries, and their visitors.

This is the central ghini.server, serving data to all other Ghini family members. ghini.server handles data from one garden, or for several.

• in combination with the included ghini.web client, it lets logged in garden conservators plan their work, produce logs, and review their garden history.
• in combination with the ghini.web web client, it provides non logged in visitors a read-only access to portions of the database.
• in combination with ghini.pocket, it lets gardeners streamline their daily activities, and make it visible to the garden conservator.
• in combination with ghini.tour, it lets visitors find their way in the garden.
• ghini.server can cooperate with other instances of itself, contributing selected data to a ghini.server instance running as a garden aggregator. A ghini.web client accessing a garden aggregator server will now show data from several gardens at the same time.

server's Issues

after context-menu edit, refresh row

a context menu action can result in changes in the database, either limited to the active line, or propagating to other objects. let's do start optimistically, and just refresh the edited row.

choose better icons for depending objects

the way it works now: you get a results row, it has a search button;

you click on it, and the client asks the server what objects are linked to the one in question, and you get as many buttons as the collections.
it may be: Taxon, Accession, Plant, Location, Contact.
once you get the 'depending' collections, you can click on the icon and the collection unfolds.
what we miss:

• choose a better icon for each category
• make sure that the icon of the unfolded collection looks different than when folded.

implement filtered query

the client (master) lets the user insert a search query.
the same master lets the user indicate on which collections the search query should be applied.

now the server should understand such a query. (and answer in a way that the client can handle regardless the type of objects being served. see #8).

reconsider BETWEEN as 'range'

https://docs.djangoproject.com/en/2.2/ref/models/querysets/#range
it's now implemented as intersection of two queries.

expose aggregate functions in query

Django has aggregate functions, I'm afraid many more than I would like to offer our users.
https://docs.djangoproject.com/en/2.2/ref/models/querysets/#aggregation-functions
let's just start with count and sum.

add URL for results line representation

same as the infobox URL, I am now missing the equivalent to ghini.desktop's search_view_markup_pair. this would allow me update the results line after updating the object.

add vernacular names

in ghini.desktop we have a 1 to many, but shouldn't it be a many-to-many? think of "ciruelo" in Latin America, or "mostaza" in Chinese shops, or "espinaca" … don't know, just a thought.

add stored queries

in ghini.desktop stored queries ended up in a generic table with uncategorised data, and we split the name to make space to different fields. django has this nice migrate thing, it will be ways easier.

add overview page as ghini home

I miss the initial home page, with an overview of the content, and links to queries.
the query language isn't in place yet, so we could only select "all" objects of a specific type, but thinking in perspective, we can already add the page with partially non-functional links.

find an acceptable source for genera

in issue #1 I stopped with an incomplete solution. we let the user import whatever they used to have in ghini.desktop, while we the new TSGhini database does not have the same family/genus/species limitation.
we have the taxonomic derivation of all orders, we have included most families, we have some of the taxa at intermediate ranks between family and genus, for a few families, but below 'ordo' the tree is not complete.
can we use Wikispecies, maybe?
for i in $(cat /tmp/1.0/genus.txt | sed -n -e 's/^\([A-Z][a-z]*\).*/\1/p' | sort); do echo $i; wget -qO- https://species.wikimedia.org/wiki/$i | sed -n -e 's/.*\(Familia\|Subfamilia\|Tribus\|Subtribus\|Genus\):.*>\([A-Z][a-z]*\).*/\1: \2/p'; echo; done > /tmp/1.0/taxonomy.txt
this collects all intermediate epithet information, no synonyms, no authorships.

implement uniform answer to web client search

when the master client sends a request for matching objects, it expects objects of different types, which will behave more or less uniformly, or at least respecting the same idioms.

so an object in the results list should know how to present itself in the list, should know how to update the infobox, and should know of whatever objects depend from it.

I think we should implement these in the browse app, not in the distinct data apps (taxonomy, collection, garden).

the result could be

• a piece of html, or data to build it, similar to ghini's two lines representation;
• the url to ask for infobox data/view;
• a dictionary that associates a name to an url, to retrieve dependent data.

keep history of changes

like bauble/ghini.desktop, and like Eric's taxasoft BG

generalize to multiple gardens

speaking of object ULRs, if we generalize the database to model more than one garden, we will need to associate accessions to gardens (and their accession code will stop being unique across the database), we will probably identify gardens with a stub, and will prepend accession urls with a garden stub code. as of now, we only deal with a single garden.

copy the 'two lines summary' into reloaded

• change inline from single element to three parts
• typeset it as bold - thin - second line
• make sure only bold is copied to infobox

one of the nicer features according to me: the results overview showing a minimal but relevant summary of the object.

align buttons in 'depending' sections

I'm not so sure how to do this, I have to dive a bit more into css, bootstrap, and such stuff. or someone could help. I'm sure it's simple, once you know how to do that.

when object is deleted, remove from results list

add context menu to results items

accession: edit; add plant; delete
plant: edit; split; delete
others: edit.

on activation of the context menu,

• steadily highlight the associated row,
• show the associated infobox,
• disable the menu when the user moves the mouse out of it. - won't do this.

implement search by alternative fields

it now only matches the search string on a single field.
at least for locations, it should match on either code or name.

let ghini.server handle the r/w client

the r/w client, that would become ghini.boss, or ghini.chief, or ghini.master, I don't yet know how to call it. maybe master. anyhow, this would become the replacement for ghini.desktop.

add data insertion modal boxes

logic

some objects (Contact, Accession, Location, Taxon) can be added without relation to previously inserted data. they don't need a parent object. Taxon is a border case, since a Taxon has a 'parent' taxon property, but that's not characterizing the object, it may even change. Plant and Verification only make sense in combination with an Accession and that's an immutable property. in fact, the URL to a Plant as well as the URL to a Verification contain their related Accession code.

objects are added, or edited. the javascript client requests editing forms from the django server. the django server serves pre-populated forms for specific objects, and empty forms for new objects. forms are not cached. forms served by django only contain table rows, while the client holds the rest of the form. the javascript client adds the POST verb to empty forms, because that's for new objects, and the PUT verb to pre-populated forms, because that's for editing objects. we use PUT and not PATCH because every time we're rewriting the whole edited object.

URL logic

let's collect the URL involved in this story.

all URLs are named, after the class, plus an optional suffix.

• list API url, it lets you GET the list, and POST a new object, suffix -list;
• detail API url, meant for individual GET, and PUT, suffix -detail;
• infobox URL, for populating the infobox, suffix -infobox;
• empty form URL, suffix -post-form;
• populated form URL, suffix -put-form;

concrete urls regarding the same object or the same collection can be computed one from the other, like they all share the same prefix, so that the javascript code can juggle with replacing suffixes.

things to do

• add context menu - drop down
• add main menu
• define Django view for both empty and populated forms content
• systematically use url names
• put more structure in url values
• add target property to forms
• allow for POST/PUT in target and let javascript change it
• populate / clean form before displaying it
• add new objects
• edit existing objects
• add dependent objects (Plant, Verification)
• freeze properties that should not be edited (split into #38)
• do not let user post incomplete request (split into #23)

data input/edit forms should have immutable fields

when adding a dependent object, like a plant to an accession, there's no point in letting the user again select the plant. better freeze the fields corresponding to the parent object.

implement infobox links

one of the strongest points in ghini.desktop, inherited from bauble.classic, is the pervasive presence of hyperlinks in the infobox section. they come in two different flavours (some replace the search, some add to the results list), and it's sometimes surprising which one is which. I suggest we only implement the replace search style.

enhance linking to elements of lists

fields as 'intended locations', or 'taxa' may contain lists of objects.
each element has its own link.
we should work with divs or something, and make sure that each link can be followed individually.

add keyboard shortcuts

like for moving the selected active row,
for entering the search text
for invoking the context menu
…

let notes be added to objects

like tags, one of the nicer things in bauble/ghini, all objects could accept notes, signed, timestamped, categorized notes.

document the REST interface

the rest interface to our database should offer reading, creating, patching, linking, deleting.
it should follow a logic, and the logic should be documented, and the documentation should match the code, and the code should be under continuous integration testing.

• add documentation initial structure
• register it on readthedocs
• add the development document

then describe what I did, so it makes sense if we need to expand it.

• the main GET, POST, PUT, DELETE structure,
• the other GET entry points, all derived from the above.
• what about unit testing? see issue #33
• what about deployment? see note in https://github.com/mfrasca/ghini/projects/1

allow for data import from ghini.desktop

the best thing would be to be able to import a csv backup.
considering how limited the current ts-ghini database, I would start with a small script, and have it grow as we expand the target.
like we iterate and import per table, saving a dictionary associating the original integer key with the new or found database object. the sequence could be: location, family, genus, species, accession, plant.

allow adding verification from taxon

you can add a verification only from the accession: the entry in the taxon context menu isn't working. the point is that verifications are identified starting from the accession. so when the accession isn't yet known, there's no way to even get the form for entering the data.

add user login page

users need to be logged in if they want to perform "unsafe" operations. (POST, PUT, DELETE). we now use the django admin user interface, or is it accounts …
this url: http://localhost:8080/accounts/login/?next=/browse/

new accession: one-step

when adding a new accession, we may want to add its verification, and the location of its physical plants, all from the same input page.

add python-like string formatter

Python3 has a nice way to format string, powerful and flexible, and a guy has implemented it just as powerfully and flexibly in javascript, in very few lines of code.
https://github.com/davidchambers/string-format

display pictures

with bootstrap, we should really be able to display pictures just as nicely as one can wish. only our power to imagine is the limiting factor.
consider this library: https://github.com/richardbarran/django-photologue

let non-authenticated users browse the data

we are making a read/write interface, requiring everybody to be logged in.
in reality, we should hide parts of the user interface for not logged in users, add a login/logout button, and secure the views.

• secure the non-safe api URLs,
• add a login-logout button
• identify the user interface to be protected
• use css to hide/show those parts

review infobox hyperlinks

I have created infobox hyperlinks while I still had no search language. links to epithets like autonyms or species epithets do work, but also include stuff that is either wrong or simply unrelated. like asking for genus Tillandsia and getting both genus and the autonym subgenus, or asking for indica and getting all sorts of species with that epithet.
we can do better now.

compute height of results frame

as of now, the results frame has hard-coded height.
this works for me on my computer today, but we obviously have to do it better and generic.

process all drop down menus via select2

this is not for performance, rather for a more uniform look&feel.

don't repeat same GET infobox

if you click on a row that is already the active row, the software will issue the same GET command that was used to populate the infobox.
we should know which infobox we are showing, and we should not refresh the infobox if it's the same URL.

add is (not) null to query language

https://docs.djangoproject.com/en/2.2/ref/models/querysets/#isnull

decide and implement the minimal collection needs

when ghini.server passes the javascript program containing ghini.master, we need the master to give orders to the server. that is, we need to decide what kind of command have to be given by one and understood by the other. minimally, initially.

add expanders for dependent objects

todo

• add an expander button on each row.
• clicking on it will retrieve the 'depending' objects,
• do not not show the rows, only the buttons corresponding to the depending collections.
• clicking on the new created buttons will expand/collapse the corresponding block.
• clicking on a button expands ONLY that section, collapsing any other.
  ⛔ collapsing a section acts recursively, collapsing everything below it. -

original text

when you have a row in the results list, like a taxon, or an accession, or a location, there's a whole list of objects associated to it, and actually more than one list:
taxa have dependent taxa, and identified accessions.
accessions have plants (and identifications?).
plants have propagations.
propagations aren't there yet.
locations have plants.

bootstrap has "collapsers", https://www.w3schools.com/bootstrap4/bootstrap_ref_js_collapse.asp, they have very flexible behaviour, we can work with them, recursively.

we need to enhance the answer to filter_json, split it depending on the query type, add the option for dependent objects,

add pictures to model

we should allow people keep pictures of their plants, and associate pictures to plant species (actually, taxon at whatever rank).

add menu bar

check the content of the current ghini.desktop menu bar

implement back button (previous search)

there's no need to go back one page, you only get out of the program.
it does make sense going back one search term.
we need a small queue of search terms, and two buttons.

make sure form is only posted via ajax

data input forms should not change the url. this goes fine if you activate the submit button with a click, but not if you press on enter while in a form input element.

implement search per domain

when performing a search, if the search query starts with a search domain, the searched string should only be matched against the desired collections, not against everything. this is useful in general, but necessary in the hyperlinks.

let ghini.server handle the geographic client

the geographic client, we already have it as ghini.web, it can serve as an initial step towards ghini.guest. once it's served, we have to tie it to ghini.server.

todo

• add a map page next to results and pictures.
• make it show the map from osm (Leaflet).
• set initial view (initially hard coded)
• let institution data define initial view (#83),
• add coordinates to Plant model (#79),
• propagate plant coordinates to Accession and Taxon,
• store search in hash,
• restore search from hash,
• show plants on the map, matching the Results,
• before any row is selected, centre map according to all Results,
• make sure we can associate a single Results row to multiple plant markers,
• show active plant(s) in different colour,
• when row in Results is clicked, Marker is highlighted too,
• if highlighted Marker falls outside map, map is re-centred,
• when plant in Map is clicked, row in Results is highlighted,
• if highlighted Results row falls outside view, view is re-centred,
• report same two-lines in popup,
• disable popup when moving out of map,

move to own issues:

• when selects area on map, search is updated,
• show icon in centre of marker (initially hard coded),
• add .icon and .layer_zoom fields to Plant model,

showcase site at ghini.me

• decide a name - it's become cuaderno
• configure the CNAME
• create separate configuration
• copy the collection
• make sure pictures shows correctly
• configure guest user
• configure admin user

taxon infobox should show number of accessions/plants

ghini.desktop reports the amount of accessions connected to a species, plus amount of living plants, plus lots of extra information, which we now miss in ghini reloaded.
let's start at least with the infobox.

add unit testing and register on travis-ci

as it is now, it's quite unpractical. I have written a few data migrations, that add a whole mass of taxonomic information to the database. this takes some 10 minutes to run. now running tests needs to first create the initial database, which includes all migrations, also this 10 minutes one.

explain the colour code in the picture

the family picture has to be explained. colours have a meaning, both for streams and for installed or not installed software. there's a star which also has a meaning. it's not obvious and without a legenda it is very likely that I made mistakes, like inconsistent colouring. we need an explanation, and a second check against it.