Automatic File ordering on Ingest/Update about strawberryfield

[giancarlobi]

@DiegoPino my thoughts:

• ordering on file names on first upload is better/simpler/(only?) method
  People are used to file names ordering so with strange file names I think people have to take care to rename correctly them, as probably a lot of people are already doing.
• after first upload, automatic ordering must be disable as IMHO never it will be useful/coherent with my needs (also with AI ... 😄 )
• do we need a procedure/UI/SimpleEditSBFJSONSequenceNumber to manually reordering, right?
  So, we can use it also when we upload a new file to add to an existing list.
  So, upload a new file will put it at the high sequence number free (= to the end).

Sorry for this very basic point of view, I'm following Archipelago philosophy ... simple, simple, simple ...
Have a nice day, amigo

[alliomeria]

Little late to the party on this, but do have some thoughts to share, agree on.

• ordering on file names on first upload is better/simpler/(only?)

YES. We do that. We should tell people how file naming goes (documentation)

Also concur that the existing method of ordering on filenames on first upload is good approach, a familiar procedure to most GLAM professionals.

In the documentation, it could be helpful to include recommendations about having a single page per image (not scans of two page spread) for neat, logical display of book/manuscript objects in Book Reader (other potential viewers, such as Universal Viewer).

@DiegoPino I think the simple/best way is move key without sequence number at the end then people needs to use manual procedure to reorder it.

Make sense to me too.

I think so because if user removes sequence key I suppose he knows what he do and if he doesn't know then that was for testing so he needs to test also manual reordering procedure

Testing leads to learning! :) But it might also be useful (time-saving, life-saving) for users to have an option to reapply/reset the automatic ordering after any manual edits too, just in case.

[giancarlobi]

@giancarlobi i totally agree.
So concretely.

• ordering on file names on first upload is better/simpler/(only?)

YES. We do that. We should tell people how file naming goes (documentation)

Sure, of course.

-after first upload, automatic ordering must be disable as IMHO never it will be useful/coherent with my needs (also with AI ... smile )

I agree. AI is not even a real word! hahaha

But i have question is. If we go for never ever touch ordering, are there any exclusions?conditions we want there?. Like what happens if people are playing around remove ordering for certain keys. Should we assume a single missing sequence implies we need to reoder? or do we push the ones without order to the end? LIke in that example. To get the order right i ended renaming the files directly in the JSON.

@DiegoPino I think the simple/best way is move key without sequence number at the end then people needs to use manual procedure to reorder it.
I think so because if user removes sequence key I suppose he knows what he do and if he doesn't know then that was for testing so he needs to test also manual reordering procedure 😄

• do we need a procedure/UI/SimpleEditSBFJSONSequenceNumber to manually reordering, right?

Yes. We need the ordering file tab. Should also be a replace file tab, remove file tab, in general a File/Attachment tab in every Object

Could be useful/clear separated tabs for CRUD and REORDER ?

[DiegoPino]

@giancarlobi

Could be useful/clear separated tabs for CRUD and REORDER ?
Yes. We should have subtabs there. So one mega tab for Attachments, subtabs (tasks like what we get in the manage display tab) for each type of action, CRUD, regenerate stuff, order and sequence.

Also: Now that i remember @giancarlobi and @alliomeria . There is a second option we have not discussed. Table of content ordering. That is actually a good secondary option to automatic sequencing. We allow an extra structure to live in the JSON (we just need to figure out on that is simple enough to create) and then users can order pages there, many times, in many ways. and even keep substructures, like chapters, etc. Does that sound like a good idea (extra one) for another ticket inside this time, https://github.com/esmero/webform_strawberryfield (so we have a widget that allows that) and https://github.com/esmero/format_strawberryfield (so we have a standard function that allows an extra setting in each formatter/viewer to be able to read from it?)

How cool are we all, right? 🏷️

[marlo-longley]

@DiegoPino this is a nice approach coming directly from use cases.

It seems like you addressed point 2 in your comments for #61 - If the current order, is already totally not in Natural Order, but sequence keys with values are present, when assume its manually ordered. So to force an automatic ordering we just would need a function that removes all sequence keys.- so we can basically know the sequence_type without a key.

The question of new files and where they fall in the order. Yes, if automatic, the filename will dictate position in the sort. If manual, add the end of the sequence and allow user to place. But what is the UI. File naming scheme only, a Textfield for editing file name?. If there are a large number of files to order, a drag/drop will not be a good UI. Scrolling up a page with something you are dragging is bad.

[DiegoPino]

@marlo-longley i guess the questions is where we set the toggle for 'force' reordering in the UI. One way would be to override the managed file upload widget, but that would only apply to things ingested via that way. About the actual ordering UI, i think we already discussed somewhere else how to approach that: esmero/webform_strawberryfield#12 The idea is not to use table drags (bad bad UI) but allow some automatic ordering options and also a UI where you can give a given File (via file name, eg) and an autocomplete, a order and it would push others down.
Still, we have to do it!

[DiegoPino]

@giancarlobi @alliomeria i'm trying to figure out a few things related to automatic ordering and would love to hear your input (nothing fancy, just some ideas)

Fact: right now i we order, always, files based on their filenames. This ordering is happening by adding a sequence key to each structured file data. The order used is natural order which means if the file ends in _1 it will come before of one that ends in _10

As seen here
https://play.archipelago.nyc/do/cf48b5f8-4e4d-4d3d-b3f7-2ec12f0b298d#mode/2up

Problem:

I don't like the re ordering i do every time. because if i need to manually tune the order (let's say i want to move a file that is named Cover.jpeg to -1 because all was already ordered correctly and i uploaded the Cover after the initial creation and i manually (RAW) edit add a -1 to the sequence of that file, Archipelago will simply ignore me and reorder.

Possible solutions:
Be smarter. But how? RAW ideas here

• Order newly added files first. now get the offset (max sequence number of existing files != file count because someone could have modified them manually?). If the 'name' of the first file in the sequence of newly added object is higher in sequence than the filename of the file with highest sequence in the existing ones list, then simply add that number (offset) to new ones. Opposite, push then before. This approach has many many issues. Its assuming either before or after the current block. Not relative internal orders.
• Simpler: ask the user. Do you want to reorder your files?
• Ignore order and keep manual?
• push new files at the end?

The thing is this: if i have the need of changing the order, means automatic will never work out. Means the system/will/not/be able to deduce a better order than myself right? So. If i detect that the automatic order was modified by the user. Do i simply assume no automatic ordering, unless asked for explicitly?

And then all this brings me back to 'ask the user'
How and when?

Any of you see at the bottom of the page some automatic ordering mechanism that could make sense in an cultural heritage institution scenario? https://www.geeksforgeeks.org/sort-array-according-order-defined-another-array/

This whole thing comes also because i uploaded that book (example i pasted) and tiff files were named using _recto and verso at the end, which totally broke my sorting. And then, i found some that ended just in _r and _v.

[DiegoPino]

@giancarlobi i totally agree.
So concretely.

• ordering on file names on first upload is better/simpler/(only?)

YES. We do that. We should tell people how file naming goes (documentation)

-after first upload, automatic ordering must be disable as IMHO never it will be useful/coherent with my needs (also with AI ... 😄 )

I agree. AI is not even a real word! hahaha

But i have question is. If we go for never ever touch ordering, are there any exclusions?conditions we want there?. Like what happens if people are playing around remove ordering for certain keys. Should we assume a single missing sequence implies we need to reoder? or do we push the ones without order to the end? LIke in that example. To get the order right i ended renaming the files directly in the JSON.

• do we need a procedure/UI/SimpleEditSBFJSONSequenceNumber to manually reordering, right?

Yes. We need the ordering file tab. Should also be a replace file tab, remove file tab, in general a File/Attachment tab in every Object

[DiegoPino]

@alliomeria i agree we need to document a lot more here. E.g People can have verso and recto in a single file. But the viewer needs to be setup for that. So they either need to conform to that recommendation (use single pages) or make sure they have a specific view mode setup to deal with that fact, or, a twig template that is able to get info from the upload (so toggle, like a tiny checkbox in the webform?) that tells us, handle this as two pages. I feel this needs to go into https://github.com/esmero/format_strawberryfield as a new issue.

Testing leads to learning! :)

Yes. But then also testing and failing on 100s of images can lead to frustration. So, yes. I need to work on that extra tab for ordering files, deleting, replacing ones, etc. Lots to do! Will keep asking for feedback. Thanks everyone here!!