joefutrelle / oii Goto Github PK

right now expiration policy is completely controlled by the client per-request for all products, which is not fine-grained enough.

instead it should be possible to control the policy per-product, as there is no intermediate granularity.

the simplest approach is to set an expiration time, which is probably easiest represented as a delta in seconds. the expire call can then check the current time of each product against its update time + the delta.

allowing an absolute time to be specified is probably unnecessary and would complicate the update process.

They should be removed.

That results in a 500 server error.

Instead the exception should be caught and ignored.

It tries to add the rois to the roi view container div, which is not present on the bin landing page.

Possible issue introduced in 2f57294

I'm getting a blank graph when I try to log plot fluorescenceLow vs. scatteringLow.

The workflow web service could provide some simple Jinja2 templated views of useful things such as events ordered by time, errors, expiration times, etc.

It's showing 15 files, should only be showing three.

Currently there are no GUI controls allowing the user to specify a time range for metrics plots. This ought to be handled using a Javascript library that provides interactive navigation through the time series.

currently dashboard mosaics are just of one file; the requested feature is viewing a mosaic for a time range that spans more than one file.

right now the vagrant build doesn't install the apps in Apache

It works, just doesn't show the correct state.

The monolithic accession script is calling code to compute bin metrics but they're coming back zero.

For example there's /delete/{pid} which deletes a product, but /get_product/{pid} which retrieves it.
Make these consistent.

It's easy for users to accidentally create unusable time series labels or instrument names in the admin interface. These should be validated and constrained to a simple regex allowing only alpha characters and safe characters like underbar.

I think even if a bin isn't in the database it should still be viewable in the dashboard, at least via its direct links. Same thing with skipped bins.

I believe this is how V2 worked and even earlier versions of V3, but that something now is raising a 404 if the bin can't be found in the database.

@ecrockford:

when scrolling my mouse over the red timeline at top of dashboard, I would prefer to have timestamp pop up as opposed to GB of data for each bin.

right now there is no worker provisioning

So it can be used in the bin view and elsewhere like on the new day management view.

A workflow that produces products will do so and then attempt to submit them to the dashboard, even if the dashboard is not configured with any data directory to hold the products. In that case CPU time is wasted if all works properly then the product which cannot be deposited will expire, then be re-generated, resubmitted, and fail repeatedly.

What's needed is an API call for the dashboard that will simply indicate whether or not it is accepting a specific kind of product.

It may be necessary to handle manual and automatic class labels very differently from one another.

checksums, while useful, should not delay accession. they can be computed later, asynchronously, as time permits.

The dashboard now provides access to raw feature files, but no means of plotting them viewing them the way it is possible to plot and view ADC data.

related: there is no permalink to a specific bin view (always defaults to page 1 of mosaic at default size)

in oii.ifcb2.acquisition, get_copy_from sorts filesets, but do_copy puts them back into a dict, then iterates over them without sorting them.

currently the dashboard GUI only allows mosaics of four preset sizes and four preset scalings.

Batch accession is not working. After one job, wakeup fails, giving this message on the client side:

[2015-07-27 18:33:27,413: ERROR/MainProcess] Task oii.workflow.async.wakeup[ba007d77-0b8e-4cf3-a3cc-36f7fb87429f] raised unexpected: RuntimeError(u'mutex ifcb:acc:test in unknown state',)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 218, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/celery/app/trace.py", line 398, in __protected_call__
    return self.run(*args, **kwargs)
  File "/vagrant/oii/workflow/async.py", line 64, in func_wrapper
    func(*a,**kw)
  File "/vagrant/oii/ifcb2/workflow/batch_acc_worker.py", line 80, in acc_wakeup
    client.wakeup()
  File "/vagrant/oii/workflow/client.py", line 184, in __exit__
    raise RuntimeError("mutex %s in unknown state" % self.mutex_pid)
RuntimeError: mutex ifcb:acc:test in unknown state

This is a concurrency exception in the database. First a commit is attempted on a closed transaction, then key constraint violations start accumulating.

[Mon Jul 27 18:33:27.774500 2015] [:error] [pid 1572:tid 140193296103168] [remote 127.0.0.1:19538] InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (raised as a result of Query-invoked autoflush; consider using a session.no_autoflush block if this flush is occurring prematurely) (psycopg2.IntegrityError) duplicate key value violates unique constraint "products_pkey"
[Mon Jul 27 18:33:27.774507 2015] [:error] [pid 1572:tid 140193296103168] [remote 127.0.0.1:19538] DETAIL:  Key (id)=(15) already exists.
[Mon Jul 27 18:33:27.774511 2015] [:error] [pid 1572:tid 140193296103168] [remote 127.0.0.1:19538]  [SQL: 'INSERT INTO products (id, pid, state, event, message, ts, ttl, expires, priority) VALUES (%(id)s, %(pid)s, %(state)s, %(event)s, %(message)s, %(ts)s, %(ttl)s, %(expires)s, %(priority)s)'] [parameters: {'expires': None, 'pid': u'http://demi.whoi.edu/test/IFCB2_2007_182_021542_raw', 'ts': datetime.datetime(2015, 7, 27, 18, 33, 25, tzinfo=<UTC>), 'event': 'created', 'priority': 100, 'state': 'waiting', 'ttl': None, 'message': None, 'id': 15}]```

Tested with Firefox, but does not work on Chrome, at least for Mac OS.

for some reason, the "go to day" AJAX call on the bin view seems to be delayed until after the mosaic loads. I don't remember there being any reason for that in the code--if there's some way to accelerate that, it would make the interface smoother.

Haven't dug into the angular stuff, but the current behavior is that pressing the button doesn't do anything where I expect the GUI for editing a time series to appear.

I'm getting collision on supposedly atomic transactions. As far as I'm aware this is because I'm using an SQLite backend, which does not support row locking.

I will confirm that my row locking implementation works with PostgreSQL. To continue to support atomic transactions in the workflow web service API when it is backed with SQLite, I will need to provide an extra level of locking within the webapi layer.

This is another angular / restless caching issue where changes don't propagate to the database. This only seems to affect newly-created time series--exactly the ones where you would need to change the path. If you reload the admin page or return to it later, path changes "take".

products can be deleted and can expire, but their dependencies cannot. the only facility for deleting dependencies is low-level, and it's not available through the web API.

feature extraction needs to be wrapped in workflow stuff

Introduced in 2f57294

There's a FIXME-style comment in the code pointing this out:

Hard coded simply because I don't know / want to find out how to pull this from Python where the csv is read. This is easier

could cause issues when plotting datasets with different sets of features.

There's some SQLite specific code in our services right now, and my initial attempts to reconfigure those services to run on PostgreSQL have failed.

What happens at startup of e.g., the workflow service is that schema creation succeeds, then the web service correctly responds to the first request (I think) then fails on the second request. What's happening is that the PostgreSQL connection drops but the associated SQLAlchemy ORM session stays in the pool, so the next time it's used it dies with SSL errors (EOF, other conditions indicating a dropped connection that SQLAlchemy hasn't caught).

Reading examples on the web is more confusing than helpful because there is all kinds of request scope magic happening in Flask that is muddying the waters for me. This is one of those things where the solution is probably simple, but it's just not obvious on any level.

In the admin interface a filesystem check is run before allowing accession but that has the effect of preventing accession for cases where product directories are configured but empty,

in 4ad9f81 I introduced a custom patch method for user admin because changes to users weren't "taking". That could have been related to the Restangular bug that was affecting time series in #68, and so the workaround for that bug may be a much simpler approach than what I ended up doing earlier.

Time series access fails with "Internal server error" if the description field of a time series is blank (NULL in the database).

In the case that instrument data has already been acquired and simply needs to go through accession, the dashboard needs to provide a worker that does that and can be initiated periodically.

When I attempt to revoke a keychain item, the database fails with:
InvalidRequestError: Object '<APIKey at 0x10c6b8350>' is already attached to session '2' (this is '1')

should be configurable

I have successfully run almost exactly the same code before, so I'm not sure why it's failing on the wf_tools branch, but I'm getting this exception in the blob worker trying to call parse_pid on a known good pid. The good news is that stack traces are now propagating to the database so I can track worker errors. The bad news is I've got an error to track.

Traceback (most recent call last):
  File "/home/ubuntu/dev/oii/workflow/client.py", line 117, in do_all_work
    callback(pid,job)
  File "/home/ubuntu/dev/oii/ifcb2/workflow/blob_worker.py", line 40, in extract_blobs
    parsed_pid = parse_pid(pid)
  File "/home/ubuntu/dev/oii/ifcb2/identifiers.py", line 23, in parse_pid
    return next(get_resolver().ifcb.pid(pid))
  File "/home/ubuntu/dev/oii/utils.py", line 80, in inner
    new_value = fn(*args,**kw)
  File "/home/ubuntu/dev/oii/ifcb2/__init__.py", line 38, in get_resolver
    return ldr.get_resolver(locate_resolver('ifcb.xml'))
  File "/home/ubuntu/dev/oii/ldr.py", line 734, in get_resolver
    return Resolver(locate_resolver(relative_path))
  File "/home/ubuntu/dev/oii/utils.py", line 80, in inner
    new_value = fn(*args,**kw)
  File "/home/ubuntu/dev/oii/ldr.py", line 731, in locate_resolver
    return search_path(relative_path)
  File "/home/ubuntu/dev/oii/utils.py", line 34, in search_path
    cands = [os.path.join(d,pathname_suffix) for d in sys.path]
  File "/usr/lib/python2.7/posixpath.py", line 75, in join
    if b.startswith('/'):
AttributeError: 'NoneType' object has no attribute 'startswith'

some of these are more like metadata fields (lat/lon/depth), others are more like annotations (tags)

The relative time reported for a bin in the bin view is now broken--it shows up blank.