irothschild / stefuna Goto Github PK

Hello,
I was trying to test this module, facing below error when I try to run a simple worker/server.

Traceback (most recent call last):
File "/Users/kiran/Documents/installs/python-virtenvs/aws-python27/bin/stefuna", line 7, in
from stefuna.stefuna import main
File "/Users/kiran/Documents/installs/python-virtenvs/aws-python27/lib/python2.7/site-packages/stefuna/stefuna.py", line 5, in
from stefuna import Server, configure_logger # noqa
ImportError: cannot import name Server

The worker_config is not available in the worker instance as self.config. Not the case with server_config as I can see the foo/bar json in the log output.

I'm trying to run a task that does some audio and video processing, and I'm using the library librosa (https://librosa.github.io/librosa/) to do some audio analysis.

When I try to load the audio file with librosa, the worker ends silently and logs the message:
[INFO][ForkPoolWorker-5/20516] Initializing worker

The code is:

# Load the audio as a waveform `y`
# Store the sampling rate as `sr`
y, sr = librosa.load(audio_file)

Also, even thought it says ' Initializing worker', from that point onwards the server as stopped listening for new activities (until I stop and start the server).

The code and this library are working fine in all other environment (EC2/Docker etc), although the code is not threaded anywhere else.

Hi,
Stefuna looks like a great project! I tried a very basic configuration but got the following error. My Stefuna configuration is identical the default, except with a different ARN of course. I have not modified the hello_worker.py yet (so it is also very basic). I will try this on a different OS to check results.

Andrew

OS is Ubuntu 18.04, Python 3.6.6:

stefuna --config=hello_config
2018-10-23 23:20:14,999 [INFO][MainProcess/20890] Running examples.hello_worker.HelloWorker for activity HelloExample arn:aws:states:us-east-1:754185987656:activity:Process_Imagery with 8 workers
Traceback (most recent call last):
File "/home/ubuntu/.local/bin/stefuna", line 11, in
sys.exit(main())
File "/home/ubuntu/.local/lib/python3.6/site-packages/stefuna/stefuna.py", line 97, in main
healthcheck=config['healthcheck'])
TypeError: 'NoneType' object is not callable

I am trying to understand this package in a better way. It works great for step function with a single activity.

How it works if a step function is having more than one state and each state is having an activity?

"Comment": "A Hello World example",
"StartAt": "One",
"States": {
    "One": {
      "Type": "Task",
      "Resource": "arn:aws:states:****:one",
      "Next": "Two"
    },
    "Two": {
      "Type": "Task",
      "Resource": "arn:aws:states:****:two",
      "End": true
    }
  }
}

#hello_worker.py
class One(Worker):
    def run_task(self, task_token, input_data):
        return {"message": "One"}

class Two(Worker):
    def run_task(self, task_token, input_data):
        return {"message": "Two"}

1. How can we execute stefuna now? Like this??

stefuna --worker=hello_worker.One --activity-arn=arn:aws:states:****:one
stefuna --worker=hello_worker.Two --activity-arn=arn:aws:states:****:two

If I execute like above am getting this error on the second command since the health check for both command listens to the same port.

Traceback (most recent call last):
  File "/bin/stefuna", line 8, in <module>
    sys.exit(main())
  File "/lib/python3.6/site-packages/stefuna/stefuna.py", line 113, in main
    loglevel=loglevel_numeric)
  File "/lib/python3.6/site-packages/stefuna/server.py", line 86, in __init__
    self._create_healthcheck(healthcheck)
  File "/lib/python3.6/site-packages/stefuna/server.py", line 169, in _create_healthcheck
    self.healthcheck_http_server = HTTPServer(('', port), HealthcheckHTTPRequestHandler)
  File "lib/python3.6/socketserver.py", line 453, in __init__
    self.server_bind()
  File "lib/python3.6/http/server.py", line 136, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "lib/python3.6/socketserver.py", line 467, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use

2. Do I need to create a separate python environment for each activity related code and change health check port?

3. Can you add this information to Readme?

If I run:
pipenv run stefuna --config '{"a":1}'

Then:
Error loading config {"a":1}

Because:

    if args.config:
        local_config = locate(args.config)
        if local_config is None:
            sys.stderr.write('Error loading config {0}\n'.format(args.config))
            sys.exit(-1)

locate() returns empty.

pydoc out of python 3.7

Hi,
I am running the worker code in EC2 - Private subnet without internet connection. I am facing below error. It works fine if I run from EC2 public subnet(with internet gateway). Is there any workaround for this?

botocore.vendored.requests.exceptions.ConnectTimeout: HTTPSConnectionPool(host='states.us-east-1.amazonaws.com', port=443): Max retries exceeded with url: / (Caused by ConnectTimeoutError(<botocore.awsrequest.AWSHTTPSConnection object at 0x7fc11e517ef0>, 'Connection to states.us-east-1.amazonaws.com timed out. (connect timeout=60)'))

Being able to specify the AWS Step Function Activity ARN on the command line would be greatly beneficial. The existing override logic that exists for the worker definition would be a perfect pattern to follow here as well.

    parser.add_argument('--activity-arn', dest='activity_arn', action='store', required=False,
                        help='Step Function Activity ARN, Overrides config setting.')

I'm happy to do the work and provide a PR.

~Drew