ska-sciencedataprocessor / integration-prototype

SDP Integration Prototype

SDP Integration Prototype

---

---

PLEASE NOTE THIS REPO IS NO LONGER UNDER DEVELOPMENT AND SHOULD BE CONSIDERED ARCHIVED

---

---

Introduction

The SDP Integration prototype is a project to develop a lightweight prototype of the major components of the SDP system. The focus of this work is to:

• provide verification, testing and analysis of the SDP architecture,
• test external and internally facing SDP software interfaces,
• provide limited tests of horizontal scaling on a representative SDP hardware prototype, the SDP performance prototype platform (P3) also known as ALaSKA.

The SIP Code

The code in this repository is designed to provide a set of loosely coupled services, example workflows and pipelines, and supporting libraries. These are intended to be independently testable but also capable of being combined as part of a larger deployment for testing various combination of the SDP software components.

This repository also consists of a number of emulators which are used to provide synthetic data to mock or test various interfaces.

The main SIP code can be found in the sip directory and is organised according to a structure inspired by the SDP high-level architecture module view. Emulator code can be found in the emulators directory.

As, by design, code folders in this repository are loosely coupled most will contain a README.md describing their function and how they can be run and tested. In fact, while folders are grouped into this single repository for convenience, the majority of sub-directories in this repository can be equally thought of as standalone sub-repositories.

In order to test collections of SIP code which providing various test deployments scripts facilitating this can be found in the deploy directory and at higher levels in the code tree.

SIP Docker Containers

https://hub.docker.com/r/skasip/

integration-prototype's Issues

master controller dies horribly when the sip stack is removed even if the state is "off"

Describe the bug
When the sip stack is removed, the master controller throws an exception. This happens even if the current state is 'off'

To Reproduce

1. Follow the 01_states_demo instructions
2. Set the target_state = "OFF" with iTango
3. remove the sip stack with "docker stack rm sip'
4. Observe a long, ugly stack dump in the master controller log

Expected behavior
A graceful exit. Since there is no way out of the off state the master controller should stop monitoring the database.

Screenshots &/or terminal output

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 184, in _read_from_socket
    raise socket.error(SERVER_CLOSED_CONNECTION_ERROR)
OSError: Connection closed by server.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2952, in _execute
    return command(*args)
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 636, in read_response
    raise e
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 633, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 291, in read_response
    response = self._buffer.readline()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 223, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 198, in _read_from_socket
    (e.args,))
redis.exceptions.ConnectionError: Error while reading from socket: ('Connection closed by server.',)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 493, in connect
    sock = self._connect()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 550, in _connect
    raise err
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 538, in _connect
    sock.connect(socket_address)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/sip/app/__main__.py", line 331, in <module>
    main()
  File "/home/sip/app/__main__.py", line 322, in main
    _process_state_change_events()
  File "/home/sip/app/__main__.py", line 293, in _process_state_change_events
    _state_event = state_events.get()
  File "/usr/local/lib/python3.6/site-packages/sip_config_db/_events/event_queue.py", line 58, in get
    message = self._queue.get_message()
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 3057, in get_message
    response = self.parse_response(block=False, timeout=timeout)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2974, in parse_response
    return self._execute(connection, connection.read_response)
  File "/usr/local/lib/python3.6/site-packages/redis/client.py", line 2959, in _execute
    connection.connect()
  File "/usr/local/lib/python3.6/site-packages/redis/connection.py", line 498, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 111 connecting to ec_config_database:6379. Connection refused.

EventQueue.get() behaviour

The EventQueue.get() method returns a single event everytime a new event is published. The problem is that if there is a list of historical events in the queue, it only pops the oldest one on getting new event so unprocessed events remain in the queue.

Docker stack Tango database funny

Describe the bug
Using the itango3 client

d.get_server_list()
DB_SQLError: Failed to query TANGO database (error=Lost connection to MySQL server during query)
The query was: SELECT DISTINCT server FROM device WHERE server LIKE "%" ORDER BY server
(For more detailed information type: tango_error)

In [9]: d.get_server_list()
Out[9]: DbDatum(name = 'server', value_string = ['DataBaseds/2', 'processing_block_ds/1', 'sdp_master_ds/1', 'subarray_ds/1', 'TangoAccessControl/1', 'TangoTest/test'])

To Reproduce
Leave the tango-db service idle for some ill-defined time!

1. '...'
2. '...'
3. See error '...'

Expected behavior
No error

Screenshots &/or terminal output
If applicable, add screenshots or terminal output to help explain the problem.

Context
Please add as much context as possible to help narrow down the problem.

Master Controller CLI rejects shutdown command if there are no service tasks defined

If there are no services defined as "online": true in sip/etc/slave_map.json), the Master Controller will not accept the shutdown command and therefore has to be killed manually.

This should be tested and resolved when refactoring the Master Controller in TSK-1687

Trying to go from standby to disable causes critical error

If the master controller is in standby and the target state is set to disable the master controller dies with a CRITICAL error

To Reproduce

1. Follow the 01_states_demo until you have an iTango proxy device
2. md.target_state = 'standby'
3. md.target_state = 'disable'

The master controller logs a CRITICAL error and then exits.

Expected behavior
Either the master controller changes state or an error is reported.

Screenshots &/or terminal output

2019-03-18 11:51:34,401 - sip.ec.master_controller - INFO - SDP target state changed to 'disable'
2019-03-18 11:51:34,483 - sip.ec.master_controller - INFO - Processing target state change request ...
2019-03-18 11:51:34,587 - sip.ec.master_controller - INFO - Done processing target state change request!
2019-03-18 11:51:34,588 - sip.ec.master_controller - CRITICAL - Value error: Invalid current state update: 'disable'. 'standby' can be transitioned to states: ['off', 'on', 'alarm', 'fault', 'standby']
2019-03-18 11:51:34,589 - sip.ec.master_controller - INFO - Exiting!

Context

SBI/PB registration with invalid workflow.

Describe the bug
Registering a processing block whose workflow has not been defined still results in a SBI in the configuration database with no PBs attached. This seems to be because the PB is rejected when the workflow is undefined but at that point the SBI has already been added to the DB.

To Reproduce
List steps to reproduce the behavior:
1a. Configure a SBI using the Tango Master Controller containing a PB with a workflow which is not currently registered.
1b. (not tested) use the CLI method skasip_config_db_add_sbi with a SBI with a PB where the workflow has not been registered
3. An error is reported but the SBI is entered into the DB.

Expected behavior
A clear error message should be reported and nothing should be entered into the DB so that the SBI can resubmitted.

Context
This was found running the SIP november 2018 demos.

test

comment

replace pep8 check with pycodestyle

The pep8 package has now been renamed to pycodestyle.

The SIP Jenkins file should be updated accordingly.

See:

• PyCQA/pycodestyle#572
• PyCQA/pycodestyle#466