julielab / costosys Goto Github PK

I rare cases, we need to insert/update the XMI data of a XMI document column. Add a CLI function for this.

Neither the README nor the docbook mention how to set the database configuration file (it's the -dbc parameter) or that it defaults to dbcConfiguration.xml.

This should be part of the 'Quickstart' section of the doc.

Please add the following entry to the documentation:

In case you removed the mirror subset of a table via a database statement (DROP TABLE name), it is also necessary to remove the corresponding entry to it in the _mirrorsubsets table of _data. This is not done automatically. Should you try to create another mirror subset with the same name as the deleted one, CoStoSys will throw an error message like this:

Error executing SQL command: INSERT INTO _data._mirrorSubsets VALUES ('_data.documents','user.medline',true)
org.postgresql.util.PSQLException: ERROR: duplicate key value violates unique constraint "_data_mirrorsubsets_pkey"
  Detail: Key (subsettablename)=(user.medline) already exists.

The strings 'user' and 'medline' in the message are to be exchanged with the name of your schema and table, of course.
This problem is revolved by issuing an DELETE query against the _mirrorsubsets table. In the example above, this statement would be:
DELETE FROM _data._mirrorsubsets WHERE subsettablename ='user.medline';

Right now, the deleter that removes MEDLINE documents marked for deletion in the MEDLINE update files works strictly on the default data table.
Let instead a list of tables be given in the configuration.

There is a corrupt PMC update file. Read as many archive items as possible and skip the rest.

Currently, the code to update pubmed and memorize the updates already applied is external code. Create some kind of addon, plugin or whatsoever to enable the update functionality of CoStoSys to do this.

Currently, all CoStoSys connections can be shared via obtainOrReserveConnection. However, this can cause issues when two code snippets use the same connection but one needs auto commit and the other doesn't.
Thus, it should be possible to get a connection that is not shared with other code.

Allow the CoStoSys CLI to drop tables. This is especially important for mirror tables that should then automatically be removed from the _mirrorsubsets table. See #2 .

Add a -iap, --ignore-already-processed switch to the -im (import Medline) mode. Specifying this should cause update files to be applied to database import and by deleters no matter if they were imported already.

Until now, XML namespace awareness was switched on by default.
The new nsAware attribute for the tableSchema element allows to en- or disable namespace awareness for the underlying VTD XML library.

The binary format is more efficient in storage but cannot be queries as easily as text or XML storage.

For decoding the data we need to

• indicate in the table schema that it stores binary data
• provide a facility to specify UIMA type descriptors which are required by the decoding algorithm

Historically, we made a difference between "import" and "update": Import does not check for duplicates but just issues a series of INSERT statements. At some point however, document IDs occurred multiple times not only in update files but even in the baseline files. At this point we began to use "updates" all the time.

The next step will now be to not make any difference between baseline and updates at all. All that matters is order: First the baseline files, then the update files in ascending order relative to the file name.

Thus, we will just define an ordered list of directories in the Medline/Pubmed update XML configuration file that will all be taken as update files. Hence, they will be written into the database as update files and marked as being imported.

When then a new Medline baseline is issued, we only need to point to the new files and the update will begin from scratch, overwriting old documents with the same PMID.

Currently there is no specification, using Postgres 9.6.