bfansports / cloudprocessingengine-decider Goto Github PK

The decider never terminate the workflow.
If no more activities, then the workflow should end.

Default logging is very verbose and not very useful. A lot of Boto requests debug output.
Could we have more meaningful logs?

I am lacking an element as input for one of the step. It is required by my plan.
The decider detect the issue but crash

The Decider needs to be super reliable and never crash! It should throw errors and start over his loop. It needs try {} catch {} desperately.

NFO:pydecider.state:Updating step 'TranscodeAssetStep' status: pending -> ready
Traceback (most recent call last):
File "./bin/decider.py", line 64, in
main()
File "./bin/decider.py", line 57, in main
while d.run():
File "/usr/local/lib/python2.7/dist-packages/pydecider/swf_decider.py", line 44, in run
decisions = self._run(events)
File "/usr/local/lib/python2.7/dist-packages/pydecider/swf_decider.py", line 52, in _run
results = self.statemachine.eval(events)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state_machine.py", line 50, in eval
results = self._run_event(event)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state_machine.py", line 66, in _run_event
handler_fun(event)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state_machine.py", line 149, in __ev_completed
self.state.step_update(step_name, 'succeeded', output)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state.py", line 120, in step_update
new_output=new_data)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state.py", line 329, in update
child.check_requirements(context)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state.py", line 312, in check_requirements
self.update('ready', context)
File "/usr/local/lib/python2.7/dist-packages/pydecider/state.py", line 323, in update
self.input = self._prepare()
File "/usr/local/lib/python2.7/dist-packages/pydecider/state.py", line 261, in _prepare
render = self.step.prepare(context)
File "/usr/local/lib/python2.7/dist-packages/pydecider/step.py", line 172, in prepare
self.activity.check_input(activity_input)
File "/usr/local/lib/python2.7/dist-packages/pydecider/activity.py", line 155, in check_input
return self._input_validator.validate(activity_input)
File "/usr/local/lib/python2.7/dist-packages/pydecider/schema.py", line 39, in validate
self._input_validator.validate(some_input)
File "/usr/local/lib/python2.7/dist-packages/jsonschema/validators.py", line 117, in validate
raise error
jsonschema.exceptions.ValidationError: 'client' is a required property

Failed validating 'required' in schema:
{'properties': {'client': {'type': 'object'},
'input_asset': {'type': 'object'},
'input_asset_metadata': {'type': 'object'},
'output_asset': {'type': 'object'},
'workflow': {'type': 'object'}},
'required': ['client',
'workflow',
'input_asset',
'input_asset_metadata',
'output_asset'],
'type': 'object'}

On instance:
{u'input_asset': {u'bucket': u'cloudtranscode-dev',
u'file': u'/input/video1.mp4',
u'type': u'VIDEO'},
u'input_asset_metadata': {u'format': {u'bit_rate': u'760321',
u'duration': u'213.785000',
u'filename': u'/tmp/CloudTranscode/55ac6ad6b3fc43.58332425//input/video1.mp4',
u'format_long_name': u'QuickTime / MOV',
u'format_name': u'mov,mp4,m4a,3gp,3g2,mj2',
u'nb_programs': 0,
u'nb_streams': 2,
u'probe_score': 100,
u'size': u'20318159',
u'start_time': u'0.000000',
u'tags': {u'compatible_brands': u'isommp42',
u'creation_time': u'2013-11-17 00:58:44',
u'major_brand': u'mp42',
u'minor_version': u'0'}},
u'streams': [{u'avg_frame_rate': u'30000/1001',
u'bit_rate': u'661697',
u'bits_per_raw_sample': u'8',
u'chroma_location': u'left',
u'codec_long_name': u'H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10',
u'codec_name': u'h264',
u'codec_tag': u'0x31637661',
u'codec_tag_string': u'avc1',
u'codec_time_base': u'1001/60000',
u'codec_type': u'video',
u'coded_height': 368,
u'coded_width': 640,
u'display_aspect_ratio': u'0:1',
u'disposition': {u'attached_pic': 0,
u'clean_effects': 0,
u'comment': 0,
u'default': 1,
u'dub': 0,
u'forced': 0,
u'hearing_impaired': 0,
u'karaoke': 0,
u'lyrics': 0,
u'original': 0,
u'visual_impaired': 0},
u'duration': u'213.780233',
u'duration_ts': 12826814,
u'has_b_frames': 0,
u'height': 360,
u'index': 0,
u'is_avc': u'1',
u'level': 30,
u'nal_length_size': u'4',
u'nb_frames': u'6407',
u'pix_fmt': u'yuv420p',
u'profile': u'Constrained Baseline',
u'r_frame_rate': u'30000/1001',
u'refs': 1,
u'sample_aspect_ratio': u'0:1',
u'start_pts': 0,
u'start_time': u'0.000000',
u'tags': {u'handler_name': u'VideoHandler',
u'language': u'und'},
u'time_base': u'1/60000',
u'width': 640},
{u'avg_frame_rate': u'0/0',
u'bit_rate': u'96022',
u'bits_per_sample': 0,
u'channel_layout': u'stereo',
u'channels': 2,
u'codec_long_name': u'AAC (Advanced Audio Coding)',
u'codec_name': u'aac',
u'codec_tag': u'0x6134706d',
u'codec_tag_string': u'mp4a',
u'codec_time_base': u'1/44100',
u'codec_type': u'audio',
u'disposition': {u'attached_pic': 0,
u'clean_effects': 0,
u'comment': 0,
u'default': 1,
u'dub': 0,
u'forced': 0,
u'hearing_impaired': 0,
u'karaoke': 0,
u'lyrics': 0,
u'original': 0,
u'visual_impaired': 0},
u'duration': u'213.786122',
u'duration_ts': 9427968,
u'index': 1,
u'max_bit_rate': u'102864',
u'nb_frames': u'9207',
u'profile': u'LC',
u'r_frame_rate': u'0/0',
u'sample_fmt': u'fltp',
u'sample_rate': u'44100',
u'start_pts': 0,
u'start_time': u'0.000000',
u'tags': {u'creation_time': u'2013-11-17 00:58:46',
u'handler_name': u'IsoMedia File Produced by Google, 5-11-2011',
u'language': u'und'},
u'time_base': u'1/44100'}]},
u'output_asset': {u'bucket': u'cloudtranscode-dev',
u'file': u'/output/video1.mp4',
u'keep_ratio': False,
u'no_enlarge': False,
u'preset': u'360p-4.3-generic',
u's3_encrypt': True,
u's3_rrs': True,
u'type': u'VIDEO',
u'watermark': {u'bucket': u'cloudtranscode-dev',
u'file': u'/no-text-96px.png',
u'opacity': 0.2,
u'size': u'96:96',
u'x': -20,
u'y': -20}},
u'workflow': {u'domain': u'SADomain',
u'name': u'Transcode',
u'taskList': u'basic_transcode',
u'version': u'1.0'}}

We need a way to simple way to do two things:

• Pass the WF input (JSON payload) to an Activity (Any activity).
  Keep the existing logic of variables but we need a keyword to inject the WF payload in a variable or as the entire input for the activity.
• Simply pass the output of one Activity to another.

The use of keywords here would be great. Similar to init

thanks

Would be great to be able to:

• Create condition on output values. Similar to "success" / "failed".
• Allow people to create conditional workflows

It is needed for SA. Could be powerful feature for general use too.

nico

We need to be able instantiate Boto with the region located in config environment variable if any provided.

It uses us-east-1 otherwise ad today it cannot be changed.

When I add timeout and heartbeat info in the plan I expect it to override the default value.
It doesn't.

Documentation say that the following should be used in my plan:

---
plan:

In your plan example it is not used. Also it makes my decider crash if I add it

The Decider must allow users to provide the name of the Activity TaskList they want to use for their steps. Generating it automatically is very restrictive and done on the activity level not on the step level.

Example:
I want to use the same activity (same version) for two different steps. One step generate thumbnails and I want a dedicated worker for it. The other step generates videos and I provision many workers for it.

I want to make sure the thumbnail worker stays dedicated to the thumbnail generation thus it should listen to a specific TaskList for Thumb jobs only. My videos workers should listen to another TaskList for video jobs only.

The problem today is that the TaskList is generated on the "activity" level. It should be generated on the "step" level. Each step should accept a specific TaskList if provided. This way each step can be processed by a separate activity TaskList.

The only to make this work today, is to have several version of Activities or having several deciders all using different domains.

Within the same domain, you can't segregate activities of same type and version.

The Decider just crash if the plan format is little off.

Is there a way to at least catch that a exit gracefully?
If there is a way to show what the error it would be great.

We need to document the plan format.
You can use Hackpad for now:
https://sportarchive.hackpad.com/Cloud-Trandscode-New-Decider-TIAP4yPeoyD

Plan needs to handle the options provided by SWF.
Especially the timeout options are they are crucial for scaling and preventing workflow to timeout silently.

The only way to prevent building a retry logic or a queue logic outside of CPE is to allow workflows to never timeout. If they timeout we are dead in the water.

If they never timeout, we can track the size of the pool and scale at will. For this we must be able to configure SWF workflows when the Decider register them.
Configuring the activities options would also be a plus but not as critical. However for large transcodes, we will need to control the activities timeout as well.

All those options can be submitted at creation time by the client. This is something I'm looking into, but for simplicity and clarity it makes more sense to have this in the plan. We already have the name and versions.

thanks

input_specs: ~

is throwing an error.
Both the workflow input_spec and Activities input_spec need something.

Work around: use:
type: "object"

Would be nice to exit gracefully

We need to be able to specify the activity TaskList name instead of having it automatically generated by the Decider.

Following the documentation, I don't need to specify:

input_spec:
  type: object

At the workflow level.

If I don't I get the following error:

Traceback (most recent call last):
  File "./bin/decider.py", line 57, in <module>
    main()
  File "./bin/decider.py", line 50, in main
    while d.run(): pass
  File "/usr/local/lib/python2.7/dist-packages/pydecider/swf_decider.py", line 40, in run
    decisions = self._run(events)
  File "/usr/local/lib/python2.7/dist-packages/pydecider/swf_decider.py", line 48, in _run
    results = self.statemachine.eval(events)
  File "/usr/local/lib/python2.7/dist-packages/pydecider/state_machine.py", line 46, in eval
    results = self._run_event(event)
  File "/usr/local/lib/python2.7/dist-packages/pydecider/state_machine.py", line 62, in _run_event
    handler_fun(event)
  File "/usr/local/lib/python2.7/dist-packages/pydecider/state_machine.py", line 111, in __ev_start
    self.plan.check_input(input_data)
  File "/usr/local/lib/python2.7/dist-packages/pydecider/schema.py", line 22, in check_input
    if self._input_validator is not None:
AttributeError: _input_validator

IT should be mentioned in the documentation

The decider must send updates to the client through SQS:

• Workflow started
• Workflow timeout
• Errors
  etc.

Without this, there is no way for the client to know about workflow status and possible errors.

You need to integrate the SQS library. The client SQS queue information can be submitted with the JSON payload or in the Plan to determine

Need a catchall for unwanted Decision tasks.
This task make the Decider go in loop