nsls2-conda-envs / conda-pack-template Goto Github PK

As mentioned in #17 (review), we need to update the code so that it can upload to the existing deposition when a new version of an environment is generated.

Checklist:

• a link to the deposition that was automatically created via CI that contains a new version of n2sn v0.3.7.

Nikea/xray-vision#98

With a 2021-1.2 environment

[tcaswell@xf28id2-srv1 ~]$ conda activate /opt/bluesky/qserver/conda_envs/qserver-collection-2021-1.2/(/opt/bluesky/qserver/conda_envs/qserver-collection-2021-1.2) [tcaswell@xf28id2-srv1 ~]$ conda list | grep -i epic
epics-base                7.0.4                h68659b9_0    nsls2forge
pyepics                   3.4.3                    py37_0    nsls2forge

Trying to import pyepics will eventually fail with:

(/opt/bluesky/qserver/conda_envs/qserver-collection-2021-1.2) [tcaswell@xf28id2-srv1 ~]$ ipython
impPython 3.7.7 (default, Mar 26 2020, 15:48:22) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.21.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import epics.ca

In [2]: epics.ca.find_libca()
---------------------------------------------------------------------------
ChannelAccessException                    Traceback (most recent call last)
<ipython-input-2-476db334b910> in <module>
----> 1 epics.ca.find_libca()

/opt/bluesky/qserver/conda_envs/qserver-collection-2021-1.2/lib/python3.7/site-packages/epics/ca.py in find_libca()
    331 
    332 def find_libca():
--> 333     return _find_lib('ca')
    334 
    335 def find_libCom():

/opt/bluesky/qserver/conda_envs/qserver-collection-2021-1.2/lib/python3.7/site-packages/epics/ca.py in _find_lib(inp_lib_name)
    327         return dllpath
    328 
--> 329     raise ChannelAccessException('cannot find Epics CA DLL')
    330 
    331 

ChannelAccessException: cannot find Epics CA DLL

Debugging this, the search path for pyepics ad-hoc library search path does not include where the epics libraries actually are (the underlying bug here is that pyepics re-implements library discovery but unfortunately that is not actually fixable upstream). A work-around is to do

export PYEPICS_LIBCA=/opt/bluesky/qserver/conda_envs/qserver-collection-2021-1.2/epics/lib/linux-x86_64/libca.so

before starting IPython, but this is obviously not a generalizable solution.

It is possible that this will need to be addressed in the CF builds of either pyepics or epics-base.

That should include the following fixes:

• pyolog v4.5.0
• silx v0.15.2
• xmidas v0.1.1
• xpdan v0.8.2
• xray-vision v0.1.1
• nslsii v0.3.2
• mxtools v0.0.2
• nyxtools v0.0.2
• intake v0.6.4 (see NSLS-II/sirepo-bluesky#67)

It would be great to implement the code to cover the following:

• check if the deposition with the same name exists and create a version of it rather than creating a new deposition
• upload all 3 files, not only one .tar.gz, e.g.:
  • n2sn-sha256sum.txt
  • n2sn.tar.gz
  • n2sn.yml
• add an option to not publish, but still, deposit files
• check if ZENODO_ACCESS_TOKEN is empty/undefined, and exit immediately (I observed the case in the CI when the token wasn't defined, and the code actually hit the Zenodo server and got 401 error code; we better check the token in the very beginning before attempting to communicate with the server).

Please use this manual deposition as an example of what we’d like to expect: https://zenodo.org/record/4421838. That one is used in the profile_collection CI.

@dmgav reported that xraylarch is not working with silx v1.0.0 released recently in December. We need to pin it to 0.15.2

Currently, the version conda picked for us is 0.5.2 (observed during the deployment visit).

The following code can be reworked to use the tabulate library to have a nice table with the results:
https://github.com/NSLS-II/conda-pack-template/blob/31045f9954cf5d77a4002b95cfeb6bcc3a24d434/zenodo_uploader.py#L57-L78

Right now it looks like:

Checklist:

• table is formatted with tabulate
• an example snapshot is posted here and/or in the associated PR in the comment
• optionally, add the date of the found depositions

lsdc-server and lsdc-gui

xmidas (developed at HXN) package is ready to be included in the environment

Implement automated uploading of the resulting Docker images to:

• DockerHub: https://hub.docker.com
• GitHub Container Registry: https://github.com/orgs/NSLS-II/packages/docker

xref: the second task from #9.

NSLS-II/pyOlog#19