You Folks Rock! (I'm a noob and could use some install help.) about cloudtranscode

[Parent5446]

Yep, basically this project has a number of modules:

• The "pollers", which perform the actual encoding task. That's this repository. It calls out to AWS SWF and asks for things to do.
• The "decider", which uses a plan file to decide what encoding tasks should be performed. You can find the decider repository here: https://github.com/sportarchive/CloudProcessingEngine-Decider

As far as running it, I'd say the hardest part is just making sure all your AWS backing is correct. You need an input queue and output queue, and the machine running CloudTranscode needs the appropriate IAM permissions in order to post to those queues and to talk with SWF.

Let us know if you run into any other particular issues!

[koxon]

Correct, I will update the documentation as it is not a submodule anymore.
If you have other questions, let us know!

[adcprod]

Thanks so much guys! By the way, I was just at the StreamingMedia East
Conference in NYC. I think I saw that at least one of you is from the NYC
area. Were either of you at the conference? (I'm based in Washington, DC
and work in IT at Voice of America as my day job.)

Anyway, there was a great session on how Viacom is doing just about all of
there web and OTT encoding via FFMPEG and other custom special sauce so
their files play out correctly when using VBR. I've made a few contacts
there and would eventually like to write an article for the magazine (if
possible) about this project if all works out well if you guys are cool
with that.

Thanks again for the prompt reply!

-Dave

On Wed, May 18, 2016 at 4:20 PM, Nicolas Menciere [email protected]
wrote:

Closed #54 #54.

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#54 (comment)

[koxon]

I receive the StreamingMedia newsletter but we didn't attend this year.
No problem for the article. We always welcome some exposure and thank you for your interest!

If you need any help with the project setup let me know, we can do a quick call to get it working if you run into troubles.

As @Parent5446 said, AWS understanding and setup are key. The project itself it pretty straight forward once the SWF concepts are understood correctly.

Best regards!

[adcprod]

Thanks so much! I'll try not to bug you guys too much and absorb what I
can on my own. I've been doing some stuff on AWS recently, but not a ton.
I think I did notice that this system utilizes EC2 storage, which seems
like it would impose a practical limit of 5Gb on individual files sent to
AWS for transcoding. (I think I've read about some hacks to get beyond
5GB, but they don't seem well supported.) Of course you could send those
to an on-premises server instead, which is what I'd do.

-Dave

On Wed, May 18, 2016 at 5:15 PM, Nicolas Menciere [email protected]
wrote:

I receive the StreamingMedia newsletter but we didn't attend this year.
No problem for the article. We always welcome some exposure and thank you
for your interest!

If you need any help with the project setup let me know, we can do a quick
call to get it working if you run into troubles.

As @Parent5446 https://github.com/Parent5446 said, AWS understanding
and setup are key. The project itself it pretty straight forward once the
SWF concepts are understood correctly.

Best regards!

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#54 (comment)

[koxon]

Hi,

We store all files in S3, and S3 file size limit is 5TB.
In order to process those files without making first a TMP copy locally, you can reference the file HTTP url in your transcoding job, instead of the S3 bucket/key.
FFMpeg will stream the file from HTTP and transcode it on the fly. If the file is on S3 but private, then you can just sign the URL.

The problem may be the resulting file as this one is being stored locally by FFMPeg, on the EBS partition of the instance in /tmp (or local disk if not running in Ec2) before being shipped back to S3.

I am trying to find Linux EBS partition specs and limitation but no luck so far.
But if it respects Ext3 partitions standard then the max file size is 2TB. This can go up to 64TB file size if they use 8 Kbytes block size.

So you may just need to associate the correct EBS storage size to your instance to make sure you have enough space for your resulting files. Once they are uploaded to S3, they are being deleted locally to restore original capacity.

Hope this help

[adcprod]

Sorry, another dumb install question.

I was under the impression that Docker would handle all/most of the requirements. However, I ran into an issue with my current Ubuntu 14.04 LTS image on AWS and it seemed to be due to Python requirements. Ubuntu 14.04 I think is using Python 2.74 and I would assuming you want us to be on Python 3.X as the setup.py file failed miserably when I ran it.

Rather than try to bolt on everything to Ubuntu 14.04, I'd rather take a recommended path from start to finish. I assume that for this system you folks would recommend users install the Amazon Linux distro rather than Ubuntu or something else. If so, I didn't see that stated in the setup docs. I'm happy to trash what I have on Ubuntu and start over with Amazon Linux (or another distro you suggest). Since AWS gives you several options for distros, it would be good to see that in writing as others might run into the same kinds of issues and waste your time. (Hey, I like the Debian package manager system. Can you blame me? I've been burned too many times by RPM in the past.)

Thanks so much!

[sportarc]

Hi,

We are using Python 2 and Ubuntu.
For the requirements, so there is no conflict nor issues with the current
env, I suggest you use this:
http://docs.python-guide.org/en/latest/dev/virtualenvs/

This will create a Python environment just for the project. I had similar
issues in the past and using this solved them.

You should be able to run this on any distro that supports Python. It
shouldn't be a problem.

Give it a shot a let me know

On Mon, May 23, 2016 at 8:28 AM, adcprod [email protected] wrote:

Sorry, another dumb install question.

I was under the impression that Docker would handle all/most of the
requirements. However, I ran into an issue with my current Ubuntu 14.04 LTS
image on AWS and it seemed to be due to Python requirements. Ubuntu 14.04 I
think is using Python 2.74 and I would assuming you want us to be on Python
3.X as the setup.py file failed miserably when I ran it.

Rather than try to bolt on everything to Ubuntu 14.04, I'd rather take a
recommended path from start to finish. I assume that for this system you
folks would recommend users install the Amazon Linux distro rather than
Ubuntu or something else. If so, I didn't see that stated in the setup
docs. I'm happy to trash what I have on Ubuntu and start over with Amazon
Linux (or another distro you suggest). Since AWS gives you several options
for distros, it would be good to see that in writing as others might run
into the same kinds of issues and waste your time. (Hey, I like the Debian
package manager system. Can you blame me? I've been burned too many times
by RPM in the past.)

Thanks so much!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#54 (comment)

Nicolas Menciere
Co-founder and CTO
http://bfansports.com

[adcprod]

Thanks! Will do.
On May 23, 2016 9:35 AM, "sportarc" [email protected] wrote:

Hi,

We are using Python 2 and Ubuntu.
For the requirements, so there is no conflict nor issues with the current
env, I suggest you use this:
http://docs.python-guide.org/en/latest/dev/virtualenvs/

This will create a Python environment just for the project. I had similar
issues in the past and using this solved them.

You should be able to run this on any distro that supports Python. It
shouldn't be a problem.

Give it a shot a let me know

On Mon, May 23, 2016 at 8:28 AM, adcprod [email protected] wrote:

Sorry, another dumb install question.

I was under the impression that Docker would handle all/most of the
requirements. However, I ran into an issue with my current Ubuntu 14.04
LTS
image on AWS and it seemed to be due to Python requirements. Ubuntu
14.04 I
think is using Python 2.74 and I would assuming you want us to be on
Python
3.X as the setup.py file failed miserably when I ran it.

Rather than try to bolt on everything to Ubuntu 14.04, I'd rather take a
recommended path from start to finish. I assume that for this system you
folks would recommend users install the Amazon Linux distro rather than
Ubuntu or something else. If so, I didn't see that stated in the setup
docs. I'm happy to trash what I have on Ubuntu and start over with Amazon
Linux (or another distro you suggest). Since AWS gives you several
options
for distros, it would be good to see that in writing as others might run
into the same kinds of issues and waste your time. (Hey, I like the
Debian
package manager system. Can you blame me? I've been burned too many times
by RPM in the past.)

Thanks so much!

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
<
#54 (comment)

Nicolas Menciere
Co-founder and CTO
http://bfansports.com

—
You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub
#54 (comment)