islandora / documentation Goto Github PK

Explain what is going on. That would probably be helpful.

The compound object solution pack adds an 'isConstituentOf' relationship to child objects' RELS-EXT. This should be listed on the Islandora namespace RELS-EXT predicates page. I can open a PR - let me know.

We should be using semantic versioning for the overall version number of the software in addition to each 'sub-project' (e.g. sync, services, etc...). Drupal modules will remain in the Drupal versioning scheme.

Feel free to completely ignore this if it doesn't align with your preferences and direction right now but I think it may already, even at this early stage, be advantageous to split the different projects from each other and have a git repository for each. Along the lines of:

• islandora-sync
• islandora-modules (a repository that at least in the short term will hold all islandora modules, so "islandora-modules/islandora" would be the 1st)
• islandora-vagrant
• islandora-techdocs (or something like that)

As the future will inevitably move in this direction anyway it would address the only point in the process that looks unrealistic / artificial from a prototype deployment / devops perspective. I'm thinking of:

https://github.com/Islandora-Labs/islandora/blob/7.x-2.x/install/drupal.sh#L52
-- where it would be more natural to clone the islandora modules repository into drupal, and ...

https://github.com/Islandora-Labs/islandora/blob/7.x-2.x/install/sync.sh#L5
-- where the sync script knows about the download location of the "umbrella" repository and works with it directly, unlike how the other, similar scripts (fcrepo, fuseki, solr) are (for the most part) self contained and can be understood independently.

See this crazy conversation here: https://github.com/Islandora/Islandora-Fedora4-Interest-Group/issues/11

Long story short, check out jquery.xmleditor and doctored.js and see if they're any good.

Let us know if one of them is a clear winner.

Directly related to #15

New users who have never installed VirtualBox, will go for the prominently marketed 5.0 version on virtualbox.org. Existing VirtualBox users, may naturally assume the need to upgrade to the latest version (my story).

Requirements on the install/README page should be updated to reflect this pitfall, maybe with a specific recommendation for VirtualBox 4.3.

Error when "vagrant up" is ran against VirtualBox 5.0:

This ticket applies to https://github.com/daniel-dgi/migration-utils/tree/camel-service

Use the transactional functionality of the fcrepo-camel component when performing upgrations.

See the README at https://github.com/acoburn/fcrepo-camel for more. Should be a matter of spring config and and adding a .transacted() statement at the beginning of things.

The README is off to a good start but it would be awesome if you could add a bit more documentation to bring greater clarity to the build and deployment process. For example, karaf is mentioned in the deployment section but isn't listed in the requirements, and having a sequence of steps that could be followed (dare I say copy-and-pasted) to get setup would be helpful.

We need to get a handle on how we're gong to nid -> path map nodes in drupal to containers in Fedora. Does it live in the drupal db? Externally? Make sure it can handle multisites, as there may be node collisions. Figuring this out will let us work on implementing services defined in the services docs.

Islandora Services needs a test suite.

This ticket applies to https://github.com/daniel-dgi/migration-utils/tree/camel-service

Create a custom processor to pretty print FOXML on the way into the service, as the way Fedora 3 exports it is not parsible by Saxon! Feel free to use a third party library to help you pretty print. No need to re-code the wheel here.

The processing should occur at the point of entry, just before the call to ingestContainer here: https://github.com/daniel-dgi/migration-utils/blob/camel-service/migration-service/src/main/java/org/fcrepo/migration/service/routes/FoxmlMigratorRoute.java#L15

add fcrepo camel component to install

The url is embarassingly hardcoded at the moment. Use the spring config to setup the fcrepo-camel component using property-placeholders so you can load in the url for the rest endpoint, as well as auth creds, from an external config file.

This applies to both this repo and the camel-service branch of https://github.com/daniel-dgi/migration-utils

Create a drupal module for basic image to hold the drupal content type for it. It should be a node. You should have certain basic fields, along with file fields for MEDIUM_SIZE and TN datastreams, as well as xml fields for MODS/FITS (etc) datastreams. The particulars haven't been set in stone, and this will be subject to change over time. So let's just get the process started and try to model content in Drupal!

We should have this exported in code so that it is available to end users. Do not make this a drupal feature, as it should be considered a starting point for customization and not holy law.

Don't forget to add any extra modules you need (xml fields, file, etc...) to the vagrant install!

1. Identify triplestores
2. Identify benchmark tests

Pull down the xpath field module and give it a whirl. Let's see how well it works on our xml datastreams in drupal.

Source is here: https://github.com/alxp/xpath_field

If it works well enough, add it to the vagrant install!

Should Fuseki have an Upstart script? In development it isn't much of a concern, but it would be good to have one for production later on.

As asked here: https://groups.google.com/forum/m/#!topic/islandora-dev/QE8uUpuqNzA

How will data flow from Drupal to Fedora?

...and add install scripts to it.

add apache camel to install

Islandora Sync needs a test suite.

Some basic end-user documentation that needs to be written, with screenshots. The 7.x-1.x documentation has an example: https://wiki.duraspace.org/display/ISLANDORA/How+to+Purge+an+Object

While poking around, I changed the types of the dc.Creator and dc.description fields (in structure > Content Types > Basic Image > manage fields to allow filtered HTML.

When I then enter some allowed html tags (e.g. <em />) and click save, at that instant the Drupal content will show up with the tags rendered. But if I reload the page, the <em> tags are sticking out again.

This probably needs a conversation: Should HTML even be allowed in DC fields?

"Does Pathauto also filter and replace mentions of repository URLs in the body of responses (i.e. returned RDF)?"

For some reason I brought up a completely new copy of this vagrant box and it set.

DocumentRoot /var/www/html/drupal/drupal

in /etc/apache2/site-enabled/000-default.conf, possibly the Document Root already has drupal in it from somewhere else and this line sticks a second one in it?

Pull together a set of Islandora upgration test fixtures.

• 1 Collection object
• 5 Basic image objects
• 5 Large image objects
• 5 Audio objects
• 5 Video objects
• 5 Web ARChive objects
• 5 PDF objects
• 5 Compound objects
• 1 Book object
• 1 Newspaper object

This is a followup to a discussion started during the March 27 meeting of the Fedora 4 Interest Group.

During the call, we discussed the impact of not using content model objects (as Islandora currently uses) but instead expressing collections as RDF properties on member objects. Some of the implications we identified are:

• There would be no more islandora:root collection
• Some people are using these to create inherited properties in new content-models
• Content-models act as useful examples of the content type
• How do we express collection policies (e.g., allowed Mimetypes. or sub-content models such as the current Paged Content within Books)?

Topic for discussion: is it preferable to model collections content models as full Fedora 4 objects or to express collection membership content models as RDF simple string properties. Examples of implementations to illustrate your arguments would be useful.

This ticket applies to https://github.com/daniel-dgi/migration-utils/tree/camel-service

Add the applicable object properties from /foxml:digitalObject/foxml:objectProperties to the parent container when migrating from foxml. Do this before ingesting datastreams. See the TODO here: https://github.com/daniel-dgi/migration-utils/blob/camel-service/migration-service/src/main/java/org/fcrepo/migration/service/routes/ContainerIngesterRoute.java#L32

Created a collection via the Drupal interface.

(It did have the title "Archives & Special Collections")

When I went to delete it, it is removed from Drupal. But this is what I saw in the catalina.out logs.

[2015-09-28 18:06:09] LogUpdateProcessor INFO  [collection1] webapp=/solr path=/update params={wt=json} {} 0 1
[2015-09-28 18:06:09] SolrCore   ERROR org.apache.solr.common.SolrException: undefined field sm_parent_document_id
    at org.apache.solr.schema.IndexSchema.getDynamicFieldType(IndexSchema.java:1269)
    at org.apache.solr.schema.IndexSchema$SolrQueryAnalyzer.getWrappedAnalyzer(IndexSchema.java:434)
    at org.apache.lucene.analysis.DelegatingAnalyzerWrapper$DelegatingReuseStrategy.getReusableComponents(DelegatingAnalyzerWrapper.java:74)
    at org.apache.lucene.analysis.Analyzer.tokenStream(Analyzer.java:175)
    at org.apache.lucene.util.QueryBuilder.createFieldQuery(QueryBuilder.java:207)
    at org.apache.solr.parser.SolrQueryParserBase.newFieldQuery(SolrQueryParserBase.java:374)
    at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:742)
    at org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:387)
    at org.apache.solr.parser.SolrQueryParserBase.handleQuotedTerm(SolrQueryParserBase.java:555)
    at org.apache.solr.parser.QueryParser.Term(QueryParser.java:403)
    at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:185)
    at org.apache.solr.parser.QueryParser.Query(QueryParser.java:139)
    at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:96)
    at org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:151)
    at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:50)
    at org.apache.solr.search.QParser.getQuery(QParser.java:141)
    at org.apache.solr.update.DirectUpdateHandler2.getQuery(DirectUpdateHandler2.java:354)
    at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:384)
    at org.apache.solr.update.processor.RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:80)
    at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:55)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:931)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:1433)
    at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:1226)
    at org.apache.solr.update.processor.LogUpdateProcessor.processDelete(LogUpdateProcessorFactory.java:121)
    at org.apache.solr.handler.loader.XMLLoader.processDelete(XMLLoader.java:349)
    at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:278)
    at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:174)
    at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:99)
    at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
    at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:418)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:207)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1041)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:313)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

It is not removed from Fedora.

Some basic end-user documentation that needs to be written, with screenshots. The 7.x-1.x documentation has an example: https://wiki.duraspace.org/display/ISLANDORA/How+to+Add+an+Item+to+a+Digital+Collection

This includes version bump, and updating the download url, since the url pattern changed slightly with the 4.3.0 release.

https://github.com/fcrepo4-exts/fcrepo-camel-toolbox/releases/tag/fcrepo-camel-toolbox-4.3.0.

Also, verify messages are being send. I did a preliminary investigates, and saw a lot of 61616 message being unable to be send in catalina.out.

This enhancement is for work that has already been produced. If the design is not desirable by the solution pack owners, we will redesign and reimplement as they see fit. The goal is to contribute code that solves the issue below.

Currently the audio solution pack requires that LAME is installed. The main issue faced by someone trying to use the solution pack without LAME is that the drupal hook_requirements method of the solution pack uses REQUIREMENT_WARNING which will not allow normal operation of drupal's update.php to update the database - requiring an override. The minor issue is that messages are sent to the watchdog log on every audio ingest.

In this case, someone is not able to install LAME on the Drupal server, but is able to generate the PROXY_MP3 as a derivative to store in Fedora. The desired result is that the Drupal administrator should be able to install the module, use update.php, and the user would be able to use the player that the solution pack uses.

This work has already been done in the following way, but my customer (Smithsonian Institution) is open to redesign and rework by the company I work for:
islandora_audio.module:

• included a string constant to indicate that the LAME encoder should not be run
• if executable path is set to this string value, no attempt to run LAME or message to the watchdog log on audio ingest
  includes/admin.form.inc:
• include text on form for what should be used in the field that points to the executable to indicate LAME is not installed
  islandora_audio.install:
• if the executable path is set to the string constant, issue REQUIREMENT_INFO as severity instead of REQUIREMENT_WARNING

See: https://github.com/daniel-dgi/migration-utils/blob/camel-service/migration-service/src/main/java/org/fcrepo/migration/service/routes/DatastreamIngesterRoute.java#L16

No attempt is made to handle different versions of datastreams. I haven't even checked to see what the result of the current work is w/r/t versions.

You should split on foxml:datastreamVersion and process each individually. You can check out an example of splitting on xpath here: https://github.com/daniel-dgi/migration-utils/blob/camel-service/migration-service/src/main/java/org/fcrepo/migration/service/routes/ContainerIngesterRoute.java#L35

Utilizing the vagrant aws and digitalocean plugins, do what the title says.

See https://github.com/daniel-dgi/migration-utils/blob/camel-service/migration-service/src/main/java/org/fcrepo/migration/service/routes/DatastreamIngesterRoute.java#L31

You should make another route to handle this to try and keep things clean.

The parent container's path is accessible as a property on the exchange, although it has already been set as a header that will be utilized by the fcrepo camel component. See: https://github.com/daniel-dgi/migration-utils/blob/camel-service/migration-service/src/main/java/org/fcrepo/migration/service/routes/DatastreamIngesterRoute.java#L23

It would be great to have a visual diagram of how of how the components fit together. There is an example here but I don't know how much use it will be as a model for how we should approach this one.

Some basic end-user documentation that needs to be written, with screenshots. The 7.x-1.x documentation has an example: https://wiki.duraspace.org/display/ISLANDORA/How+to+Edit+an+Object%27s+Metadata

Looking at the existing documentation, and pulling on discussions from the last couple Fedora 4 IG calls, I'd like to clarify what exactly folks are looking for documentation-wise. Currently we have documentation for:

• About
• Technical documenation
  • Architecture
  • Middleware Services
  • Islandora Commands
  • How to build documentation
• Contributing
  • How to contribute
  • Hacking on Islandora
• Migration
  • Property mappings
  • fcrepo3 namespace predicates
  • islandora namespace predicates
• Vagrant

@sprklinginfo @rosiel @dmoses would you mind articulating what else we could focus on documentation-wise? I can then pull that in a task list for this ticket, and we can collectively work on it during the sprint.

See the following "sub-issues" for this ticket:

• #66 - Documentation - How To Create A Collection
• #67 - Documentation - How To Ingest An Object
• #68 - Documentation - How To Edit An Object's Medadata
• #70 - Documentation - How To Purge (Delete?) An Object
• #71 - Documentation - Islandora 2.x for Islandora 1.x Users
• #72 - Documentation - How To Add An Object To A Collection
• #74 - Documentation - How To Create A Collection View
• #75 - Documentation - How To Edit An Ingest Form

See #50 @daniel-dgi comment #50 (comment) & #50 (comment)

Some basic end-user documentation that needs to be written, with screenshots. The 7.x-1.x documentation has an example: https://wiki.duraspace.org/display/ISLANDORA/How+to+Create+a+New+Islandora+Collection