pcdshub / happi

Heuristic Access to Positions of Photon Instruments

HAPPI

Heuristic Access to Positioning of Photon Instrumentation

Motivation • Features • Installation • Basic Usage • Documentation

Motivation

LCLS endstations deal with dynamic sets of instrumentation. Information like ports, triggers and aliases are all important for operation, but hard to manage when spread across a multitude of applications. Happi solves this problem by creating a single access point for all the metadata required to interface with LCLS instrumentation. Using a flexible container based system Happi allows the enforcement of specific conventions while still permitting flexible data entry where required.

Features

• Manage information for specific device types using containers
• Input arbitrary metadata associated with a specific device
• Flexible backend support for multiple types of databases; MongoDB, JSON e.t.c
• Easily search database entries for device/s that match a set of keys

Installation

Install the most recent tagged build: conda install happi -c pcds-tag -c conda-forge

Install the most recent development build: conda install happi -c pcds-dev -c conda-forge

Basic Usage

The happi.Client is your main interface to the underlying device database. You have the choice of either creating your database backend by hand or using the environment variable $HAPPI_BACKEND to create a persistent reference to your database type. By default, the Client assumes a JSON file database:

   import happi

   client = happi.Client(path='path/to/my_db.json')

If your database has entries, you should either be able to search by key variables for individual or multiple devices.

   client.find_device(name="My Device")

   client.search(stand='DG2')

Once you have the device you want, you can edit the information just as you would any other Python object. View the device information in a convenient table using .show_info:

   dev = client.find_device(name="My Device")

   dev.z = 432.1

   dev.show_info()

Output

+--------------+----------------------+
| EntryInfo    | Value                |
+--------------+----------------------+
| active       | True                 |
| beamline     | LCLS                 |
| name         | My Device            |
| parent       | None                 |
| prefix       | MY:DEV:01            |
| stand        | None                 |
| system       | None                 |
| z            | 432.10000            |
+--------------+----------------------+

After you are satisfied with your changes, push the information back to the database using the .save method. If this is a new device, you will have to call Client.add_device. Before the entry is modified in the database, the happi.Client confirms that the new changes meet all the requirements specified by the container.

   dev.save()

happi's Issues

Idea: Netconfig Backend

Add a backend that connects to the netconfig database and makes host information available through the happi API. This unites the information under one API, but doesn't duplicate the information in multiple databases

Thoughts? @ZLLentz

Motor Container

Expected Behavior

We need a container for basic motors. This will probably start referring to PCDSMotorBase and require a more specific setting from the operator.

Track Client on Device

Expected Behavior

Would be nice when we load_device or add_device to keep track of the original client. You could then also have a property like on happi.Device

@property
def in_database(self):
       if self.client:
            return True
       else:
            return False

Get metadata by attribute

Implement custom getattribute to grab information from the extraneous dictionary

Port Threaded Loading from hutch_python

There are quite a few applications that will want to load a large number of devices from happi. It would be nice to batch and thread loading where we can.

Lightpath Keyword

Expected Behavior

In the near future we will like to add devices that our "active" but do not get included in the lightpath. We can either add a "lightpath=True" default to Device or we lightpath just needs to be aware of the device classes it needs.

Docstrings for EntryInfo

Docstrings for EntryInfo are not available via IPython terminal?

Update documentation for new beamline naming scheme

I updated the naming conventions for the beamlines, this should be reflected in the documentation on the Conventions page.

Support multiple configurations

Allow a user to have multiple backends loaded. This allows for shared configuration files for common areas. The database should be smart enough to know which configuration file to add the device.

The only really tricky part is allowing the user to use different backends and still add new devices

Cache Loading

The from_container function is quiet when it uses the cache and when it doesn't. This should be clearer for users. We can also be more aggressive when we don't use the cache. Currently we only check that the prefix is the same, but we could also check that md itself is the same!

Automated Container Creation

Expected Behavior

We should not maintain a list of container with hard-coded paths to devices. Instead we should supply useful helpful functions to turn objects/classes into happi devices. Much of the manual labor can be avoided avoided by inspecting the class itself.

Example Syntax

container = container_from_class(pcdsdevices.device_types.GateValve)
happi_client.add_device(container(prefix='Tst:VGC:01', name='Easy Valve'))
container.name == 'Easy Valve'
container.device_class == 'pcdsdevices.device_types.GateValve'

Discussion about Device and Mandatory Fields

Expected Behavior

Since other labs are looking into Happi, should we make Device be completely flexible with additional metadata and create the PCDSDevice or some other name that will have mandatory fields such as functional_group and location_group ?

Current Behavior

Device requires functional_group and location_group along with some other information that is a bit LCLS specific.

Possible Solution

Create a new type of container that is the PCDS basic Device and make Device very relaxed.

Raise Exception if HAPPI_CFG points to non-existing file

If the HAPPI_CFG variable is incorrect, loading the Client.from_cfg produces a non-sensical error message.

Documentation Keyword

Expected Behavior

Add a new default keyword to happi to store relevant documentation

Expand Device Support from LCLS Questionnaire

Currently we only load motors. This should include:

• Motors
• Triggers
• Cameras
• Analog I/O

Copy method for Device

Device Container class should have a copy special method

JSONBackend should be alphabetized

Expected Behavior

The devices with the produced json file should be alphabetized for better diffs.

Add method should re-write save

Description

When you add a device to happi then want to edit it, you are force to reload the container. This has to do with how the save method is attached to the object

Expected Behavior

d = happi.Device(...)
client.add_device(d)
d.field = 'blah'
d.save()   # Currently fails with NotImplementedError

Current Behavior

d = happi.Device(...)
client.add_device(d)
d = client.find_device(name=d.name)  # Forced to reload to have save method works
d.field = 'blah'
d.save()

Questionnaire Plugin loads devices without Python names

Expected Behavior

The Questionnaire should ignore devices that haven't been given valid Python names

Current Behavior

The plugin just checks that the name key exists not that it is valid. We then load devices with name ''.

Possible Solution

Have more stringent requirements for devices that are loaded from the Questionnaire

Support fuzzy search

LUCID will be using fuzzywuzzy (pcdshub/lucid#10) so we may as well use the dependency here, too.

Add via Command Line

Expected Behavior

We should have a command line prompted version to add a new device. Imagining this is very similar to the netconfig system.

graphviz map

Expected Behavior

Draw a nice graphviz diagram of all the beamlines with all their devices and all their parents. This could be helpful when we start getting nested "parent-age"

Original Idea -> @rajanplumley

Timeout on Questionnaire

Questionnaire backend hangs for far too long when can't connect to the server.

Device Caching

Expected Behavior

The device loading should keep track of which devices it has loaded so far. This way we do not attempt to load the same device twice.

Current Behavior

No caching done.

Possible Solution

Have a happi.cache that has a mapping of loaded devices to containers. This way it is possible to go find the container information for a prior device. Also if a device is requested twice we just return a reference to the first one.

Fix questionnaire tests

Questionnaire tests have data that does not comply with happi's field value enforcement.
For example, names that start with [A-Z] in test_backends

Avoid requirement to register new device classes

happi does not easily support Containers created outside of the module

happi/happi/client.py

Line 48 in 70bb1bc

self.device_types.update(dict([(name, cls) for (name, cls) in

Separate base Container and Device

There are a few LCLS specific things in Device. In reality we should have a base Container with the bare minimum globally useful keys, then have Device contain the more LCLS specific information i.e stand, z, beamline.

Loading Routine

Standardized loading of multiple devices. Exceptions will be caught and reported.

Add an Ophyd object

Expected Behavior

Add a provided Ophyd object to the database. This allows users to skip the intermediary step of creating a happi Container. This should be done using the device _repr_info

Questionnaire Backend

Expected Behavior

Have a happi Client that connect directly to the LCLS Questionnaire and provide a read-only inteface

Current Behavior

No support for Questionnaire loading. This would currently have to be dumped to JSON.

Possible Solution

Hopefully this can be an in-memory JSONBackend with no save method.

Use jinja2 templating

For devices that need to repeat information in multiple places. Allow users to supply jinja2 templates that use other device attributes

Happi Database restructuring / unique key and removal of `_id`

Problem

Had a revelation about how to restructure the happi database. Right now when the database is written we structure the JSON dump as a single large dictionary, the key is determined by Client._id. This is the only key that happi checks to be unique

Restructuring

Add a keyword to a piece of EntryInfo called unique. This means that this piece of information has to be unique among all entries (i.e PV prefix, name, e.t.c). The database is then stored as a flat list of dictionaries, but when when we load it in happi.Client we can create a look-up table for all of our unique keys. This will give us fast look-up on any unique key and we can remove the _id nonsense.

Standardize new device loading attributes

Instead of counting on our device containers to have the same name as a pcdsdevice, each device should have device class , args, and kwargs. This will be used to quickly load the device regardless of package

Use register_instance instead of custom happi cache

The cache that is created when loading devices should be replaced by the new register_instance mechanics in ophyd.

Update screen storage information

Expected Behavior

Change the fields that specify screens to be:

• display_screen
• embedded_screen
• engineering_screen

I also want to delete macros. For L2S-I just the happi object itself can be passed in as JSON as the macros 😄

Current Behavior

Tailored for EDM

Loading Documentation

None of the loading functions made their way into the higher level Sphinx docs.

Tree view filter: group matches should show all members

From #107 :

There is this little detail demonstrated at the last piece of the GIF above in which if we filter and no child match the filter but the parent does, only the parent is displayed.
[...] in order to fix that we would need to subclass the QSortFilterProxyModel and tweak the filterAcceptsRow method. I'm not sure how much effort we should put into something like this. Up for discussion.

In-Vacuum field is not found from Questionnaire

The happi QSBackend reports all information in the Questionnaire table except for the in-vacuum column. This is probably do to an error in which we find keys in the Questionnaire JSON dump.

Travis by Conda

Modify Travis to build and test using the Conda recipe then upload to pcds-tag, pcds-dev

DB Creation

Current Behavior

• If db file does not exist Happi does not create a file
• If db file is empty, exception is raised

Avoid Repeated Information During Device Creation

Expected Behavior

It would be nice if there was a method that made an intelligent stab at populating information fields based on EPICS prefix
i.e 'TST:STAND:01`

name="test_stand_01",
beamline='TST'
stand='STAND'

This avoids some of the required grunt work.

Add Qt model for happi devices

• To be used as the model in lucid for a replacement suite view
• Also requested from LBL/xi-cam

Documentation

Update docs for 1.0.0 features. Include a much nicer README

Search on Regex

Be able to search the database on Regex patterns

Standard Storage of Metadata

Desired Behavior

After pcdsdevicesno longer handles loading from happi, there is no standard way to attach the metadata from a happi Container to the instantiated device. This should be done in .load_devices so that all devices loaded through happi have a reliable way to retrieve the metadata.

setattr(obj, 'metadata', container)

Current Behavior

Device metadata that isn't included in the class creation is dropped on the floor.

PEP8

happi should conform entirely with PEP8

Equivalance method returns tuple

Creation and Last Edits

Expected Behavior

This information should be accessible via keyword, but not writable by a user

Current Behavior

Currently it is just placed in the extraneous metadata dictionary

Update containers to point to pcdsdevices

Expected Behavior

We are no longer expecting to explicitly match the name of a container to a pcdsdevice. We should have the defaults of these classes point to the correct class using the device_class. Each should also have the correct args and kwargs formulated to pipe EntryInfo names into the device constructor.

Current Behavior

Specialized containers have None set for device_class

Global Happi Configuration File

Expected Behavior

If you launch happi.Client with no arguments we should look for a configuration file that specifies where and what type of happi database we are expecting. The thought is as applications that depend on pulling configuration from happi become more common, we want to avoid passing the information explicitly into each one. That would would mean if we move the db.json file for example we would have to go update -> hutch-python, lightpath, typhon ...

Current Behavior

Database configuration information is specified at __init__ every time

Possible Solution

Use configparser to search for .happi.cnf. This should specify database types and the keywords necessary to instantiate it