cdluc3 / ezid

CDLUC3 ezid

EZID dev setup

Keeping setup instructions separate from the original README for now.

Packaged dependencies

sudo bash -c '
    apt update
    apt dist-upgrade -fy
    apt install -y \
        \
        build-essential \
        llvm \
        make \
        \
        python2-dev \
        python-openssl \
        python-setuptools \
        \
        bash \
        curl \
        wget \
        git \
        xz-utils \
        \
        libbz2-dev \
        libc6-dev \
        libdb-dev \
        libedit-dev \
        libffi-dev \
        libgdbm-dev \
        liblzma-dev \
        libmariadbclient-dev
        libncurses5-dev \
        libncursesw5-dev \
        libreadline-dev \
        libsqlite3-dev \
        libssl-dev \
        libz-dev \
        tk-dev \
        zlib1g \
        zlib1g-dev \
        \
        sqlite3 \
'

Python 2.7

Prepare pyenv

curl https://pyenv.run | bash

• Follow the instructions printed at the end of the pyenv setup.

Install Python 2.7

bash -c '
    pyver=2.7.18
    venv=ezid_${pyver}
    export CONFIGURE_OPTS=--enable-shared
    export CFLAGS=-O2
    pyenv install ${pyver}
    pyenv virtualenv ${pyver} ${venv}
    pyenv global ${venv}
    pip install --upgrade pip
'

EZID

bash -c '
    export ez=$HOME/dev
    mkdir -p $ez
    cd $ez
    
    git clone https://github.com/CDLUC3/ezid.git
    git clone https://github.com/CDLUC3/ezid-info-pages.git ezid/templates/info
    
    cd $ez/ezid && git checkout initial-setup
    cd $ez/ezid/templates/info && git checkout initial-setup 

    cd $ez/ezid
    pip install -r requirements.txt

    ./dev_prep_env.sh
'

DB Init or reinit

bash -c '
    rm -f ./db/*.sqlite3
    ./manage.py migrate
    ./manage.py migrate --database=search
    ./manage.py loaddata store-init
    ./manage.py loaddata search-init --database=search
'

Optionally, load more complete test database

bash -c '
    yes 'yes' | ./manage.py flush
    ./manage.py loaddata store-test -e auth -e contenttypes -e sessions
'

# TODO: Outline setup via setup.py
# Dev env with: ./setup.py develop
# pyenv rehash
# Tools now available as ez-* (use tab to get a list) 

Start

$ export ez=$HOME/dev
$ cd $ez/ezid
$ ./manage.py runserver

Git

Set Git to ignore bulk edits in git blame:

$ cd <ezid root>
$ git config blame.ignoreRevsFile .git-blame-ignore-revs

ezid's Issues

Support identifier registrations that do not use the EZID minter

Need to support scenarios in which users are registering identifiers without using the EZID minter.

1. Super-shoulder
User provides their own string on a standalone prefix or NAAN with no shoulder extension
Example:
https://doi.org/10.15310/2334-3591.1058

2. No-minter
User creates an identifier using their own minter. They may or may not be using an EZID shoulder.
Examples:
https://n2t.net/ark:/21547/DaU2
https://n2t.net/ark:/81986/caida.data.100001
https://n2t.net/ark:/65665/hc8

Setup an N2T test instance for identifier minting

Having an instance of N2T running will help verify that identifiers are being created properly with the python code

Review python implementation of noid

See: https://pypi.org/project/noid-mint/

And: https://ucop.zoom.us/rec/play/uZArdrr9-m03G9Oc5ASDU_5xW9S4K6is2ykb-fsJmkjkW3AEMVKgMOZBNrc90qZlF3mWVleCTp-TenaZ?continueMode=true&_x_zm_rtaid=lf1xWU4QR-e_Y8t32eFvRg.1586370084908.624f347ad5e5f5083ba24b5da7e11216&_x_zm_rhtaid=719

See Also:

• https://github.com/no-reply/pynoid

• Looks like the main thing missing in the library is automatic extension of the template. EZID minters extend the template by 3 characters each time the previous one is exhausted.

Add passwords for n2t stage from dev

version request broken on stage2

Requesting /version on stage 2 returns an error document. Expected response is e.g.:

curl -q "https://ezid.cdlib.org//version"                                                                               [12:02:22]
success: version information follows
startup.time: Thu Sep 24 05:30:34 2020
startup.ezid_version: unknown
startup.info_version: unknown
last_reload.time: Thu Sep 24 05:30:34 2020
last_reload.ezid_version: unknown
last_reload.info_version: unknown

Align configurations across stage and dev

Need to align the configuration on dev with the practices that are currently in use on production and stage.

The goal is to use yum to manage system level dependencies, though override the configuration with the ezid application requriements.

Verify requirements for how N2T will read shoulder information from EZID

Maria: Check with John Kunze about this.

Rely on list-shoulders command, perhaps?

Add management command to activate / deactivate a shoulder

Create a test copy of and EZID database for use with the EZID dev instance

Should be pre-populated with some content so as to emulate an operating system

RDS instance on dev and dev instance will talk to dev database and we'll do a dump from stage to dev

Identify the performance requirements for correct identifier minting

Linux2 migration of N2T NOG

Create instance of EC2 for hosting EZID dev instance

• Enable SSH access for Roger and Dave
• Ensure instance is allowed to connect with the test RDS database

Test shoulder creation / identifier minting on Stage 2

Test associating shoulders and groups

Verify `reload` management operation functionality

Replace EZID call to N2T identifier minting with python method

Test coverage of refactored functionality by Travis CI

Resolve high-severity security alerts

Github is reporting a number of security alerts for UI components of EZID:

https://github.com/CDLUC3/ezid/network/alerts

These all need to be reviewed and addressed as needed.

Make sure that search works on dev

Item 8 in the test doc

https://docs.google.com/document/d/130tmwSRibh5IUWdDDGFqXmT2aNNdnhPtbQcLwoyZSRs/edit

Enable appropriate permissions for GitHub repo

Fix path to minters, make it one level above the git clone level

See also #34

Complete shoulder generation management command

check on colon issue #24

Adjust logging for verifying new code actions

(Update this issue with specific actions for improving logging)

Need to improve logging messages and the coverage of logging.

• Consistent time stamps for log messages
• Separate logs for detail vs operational
• Increase coverage of logging statements to help diagnose issues as they emerge
• Ensure that messages resulting in an email are easily found
• Support logging from multiple threads
• Ensure that management operations are logged

Fix bug with admin button on dev

Configure Django templates for working with latest Python2 version of Django

Investigate requirements for automated CI using Travis

Initial focus is on covering the functionality being replaced by removing N2T from EZID.

The CI should pull from GitHub and run tests as specified.

Possibly consider GiHub actions for continuous integration.

Test deployed EC2 instance to ensure complete functionality

Need to develop a playbook outlining the procedure for testing functionality of the EZID instance.

Functionality:

• Create shoulder
• Mint identifier with existing shoulder
• Mint identifier with new shoulder
• Edit metadata for ARK identifier
• Search metadata for identifiers

Playbook in google docs: https://docs.google.com/document/d/130tmwSRibh5IUWdDDGFqXmT2aNNdnhPtbQcLwoyZSRs/edit

N2T prod instance access

Need to be able to reload EZID after installing a shoulder

May be fixed by configuration updates in #34.

Additional capability is to ensure automatic reload after edits

Connect dev instance to RDS database (mysql)

Ensure support for DOI minting

Super shoulder support using the management command. Adjust implementation to remove support for multiple prefixes

Make the status of async operations more transparent for monitoring

The issue is that several background processes can operate (e.g. updating metadata on N2T) though it is not obvious if such operations fail or do not complete.

Would be nice to have a management / admin page or dashboard that provided some insight to the status of various background tasks.

A lighter-weight approach is to add in additional logging information about these background tasks.

See also #89

Verify and document `pause` management operation functionality

• Clarify what it means to pause
• Update documentation that pause functionality is no longer necessary since EZID is no longer relying on N2T

Need to review operation with respect to background sub-processes

Need to review this with respect to re-implementation with SystemD

Support editing / updating shoulder name and datacenter metadata using a management command.

Check end-to-end datacenter change: mint identifier on datacenter A, change datacenter, check in DataCite to make sure change is reflected.

Deploy EZID dev instance on EC2

This will result in a test instance of EZID that operates entirely independently of N2T for shoulder creation and identifier minting

Load balancer setup for Stage 2

Placeholder. Ashley tracking separately in Pivotal.

Hook up EZID Stage to DataCite and Crossref test instances

Hook up https://uc3-ezidx2-dev.cdlib.org/ to DataCite test instance (https://doi.test.datacite.org) so we can do end-to-end testing of DOI registrations. Hook up to Crossref test as well.

Investigate performance issues

Look into this much later.

Implement python method that mints identifiers

The method should take as input the state information that is necessary to generate and identifier and output an identifier that is functionally equivalent to one generated by N2T

Ensure documentation is up to date with management commands

API coverage with Travis testing

Create a pytest fixture that enables testing of the API. This will likely require a mock of the django installation.

Enable a mechanism to exercise the API in Travis

Replace EZID call to N2T for shoulder generation with Python

Document the workflow for deploying the EZID application to production.

@ashleygould will add documentation to EZID Operations page in Confluence

Make NOTES file (in Github) more prominent in Confluence page and EZID README

Confluence page: specific deployment details internal to CDL

Github repo: public documentation about application

Fix colon issue in generated identifiers

Practice run for production upgrade using the EZID staging server

See also notes at: https://github.com/CDLUC3/ezid/blob/master/ec2_upgrade.adoc

See #62 for preparation (moved from here)

See #63 for production upgrade scheduled for 2020-11-12.

Practice Upgrade during the maintenance window (starting 2020-10-29 05:00 PT):

Outcome will be:

EC2 instance: uc3-ezidui01x2-prd
RDS instance: rds-ias-ezid-search4-prd

• ezid: On the day before, ensure that the EZID link-checker has been paused
• ezid: configure apache service on uc3-ezidx2-prd to display maintenance page
• ezid: Ensure apache on uc3-ezidui01x2-prd is shutdown:

~/etc/init.d/httpd stop

• ias: clone rds instance: rds-ias-ezid-search-prd -> rds-ias-ezid-search4-prd
• ezid: reconfigure apache service on uc3-ezidx2-prd to resume normal operation
• ezid: copy most recent version of master_shoulders.txt and minters to uc3-ezidui01x2-prd
• ezid: update database host name in ~/ezid/settings/ezid.conf.shadow setting:

[databases]
store_host: rds-ias-ezid-search4-prd.cmcguhglinoa.us-west-2.rds.amazonaws.com
search_host: rds-ias-ezid-search4-prd.cmcguhglinoa.us-west-2.rds.amazonaws.com

• ezid: upgrade db schema on rds-ias-ezid-search4-prd

./manage.py migrate

• ezid: check minters

./manage.py  shoulder-check-minters

• ezid: Merge shoulders

./manage.py shoulder-merge-master

• ezid: Startup apache service on uc3-ezidui01x2-prd:

~/etc/init.d/httpd stop

• ezid: test application functionality. url: https://uc3-ezid-ui-prd.cdlib.org
• ezid: verify login by admin and regular user through the EZID UI
• ezid: perform a restart of uc3-ezidui01x2-prd
• ezid: verify EZID is back on line
• ezid: continue testing EZID application API and UI
• ezid: don't forget to restore the EZID link-checker

New Pending issues

1. How do we roll back the upgrade if the new DB has been running for several hours already?
2. What criteria do we used to determine a roll back is necessary?

Formerly Pending issues

There are several items related to the upgrade process that need to be resolved before then:

1. Is the new production VM ready for installation?
   Yes, the new ec2 instance hostname is uc3-ezidui01x2-prd.cdlib.org
2. Do Roger and Dave have access to the new production VM (access to be removed after upgrade)?
   Yes
3. Who will clone the production database?
   IAS will clone the existing RDS instance during downtime window.
   The RDS instance hostname is rds-ias-ezid-search2-prd.cmcguhglinoa.us-west-2.rds.amazonaws.com. The user eziddba user is able to create and drop any database starting with "ezid".
4. How long does it take to clone the database?
   26 minutes.
5. Are resources available for two additional clones of the production database?
   Yes, the new RDS instance has 300GB total disk space of which 200GB is unused. This is after the initial cloning from the old RDS instance.
6. Is the load balancer ready for use with the production environment? Any DNS preparations to be made?
   Yes. Currently we can access the the ALB using the domain uc3-ezid-ui-prd.cdlib.org. This is a DNS cname which points to the ALB DNSname uc3-ezidui-prd-alb-1936286154.us-west-2.elb.amazonaws.com. When we are ready to cut over, we request IAS to update DNS so that ezid.cdlib.org points to the ALB DNSname.

Create unit test for testing minting

N2T has some test cases, these should be duplicated in python where approrpriate

HTTP/2 broken on stage 2.

for example:

curl -v "https://ezid-stg.cdlib.org/status"                                                                             [11:53:44]
*   Trying 44.231.82.25...
* TCP_NODELAY set
* Connected to ezid-stg.cdlib.org (44.231.82.25) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=ezid-stg.cdlib.org
*  start date: Aug 26 00:00:00 2020 GMT
*  expire date: Sep 25 12:00:00 2021 GMT
*  subjectAltName: host "ezid-stg.cdlib.org" matched cert's "ezid-stg.cdlib.org"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f971580c400)
> GET /status HTTP/2
> Host: ezid-stg.cdlib.org
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
* http2 error: Invalid HTTP header field was received: frame type: 1, stream: 1, name: [upgrade], value: [h2,h2c]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* stopped the pause stream!
* Connection #0 to host ezid-stg.cdlib.org left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
* Closing connection 0

versus dev:

curl -v "https://uc3-ezidx2-dev.cdlib.org/status"                                                                       [11:56:07]
*   Trying 35.164.191.193...
* TCP_NODELAY set
* Connected to uc3-ezidx2-dev.cdlib.org (35.164.191.193) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; postalCode=94607; ST=California; L=Oakland; street=1111 Franklin Street; O=Regents of the University of California; OU=CDL; CN=uc3-ezidx2-dev.cdlib.org
*  start date: Jun  4 00:00:00 2020 GMT
*  expire date: Jun  4 23:59:59 2022 GMT
*  subjectAltName: host "uc3-ezidx2-dev.cdlib.org" matched cert's "uc3-ezidx2-dev.cdlib.org"
*  issuer: C=US; ST=MI; L=Ann Arbor; O=Internet2; OU=InCommon; CN=InCommon RSA Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fc98280f600)
> GET /status HTTP/2
> Host: uc3-ezidx2-dev.cdlib.org
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 200
< date: Tue, 29 Sep 2020 15:56:40 GMT
< server: Apache/2.4.46 () OpenSSL/1.0.2k-fips mod_wsgi/4.7.1 Python/2.7
< content-length: 19
< vary: Accept-Language,Cookie
< content-language: en
< content-type: text/plain; charset=UTF-8
<
* Connection #0 to host uc3-ezidx2-dev.cdlib.org left intact
success: EZID is up* Closing connection 0

Address shadow ark validation issue with super-shoulder creation

Creating a super-shoulder results in a shadow ark validation error

Update django templates for latest version of django in python2 (security fixes)