distribute binary modules for common platforms/architectures about node-webrtc

[DamonOehlman]

Alan - I'm happy to take this on if you like. I think we may need to take a different approach to the fibers module as the fibers binary looks to be about ~35K whereas the webrtc.node looks to be around 8.5Mb.

I think we can use a similar approach though and just host binaries somewhere online, and download based on a platform detection script...

[piranna]

I agree on that, there are free CDNs for open source projects. Also, this
way we can left the grunt toolchain only as devDependencies.

Since standard Node.js compiled modules are static libraries without
dependencies, I propose to start having versions for the usual suspects
(Mac, Windows & Linux) both in x86 and x86_64, and later add more as they
are suggested (ARM, FreeBSD...), with grunt it should be easy to build more
veesions in just one shoot...

Send from my Samsung Galaxy Note II
El 11/02/2014 08:13, "Damon Oehlman" [email protected] escribió:

Alan - I'm happy to take this on if you like. I think we may need to take
a different approach to the fibers module as the fibers binary looks to be
about ~35K whereas the webrtc.node looks to be around 8.5Mb.

I think we can use a similar approach though and just host binaries
somewhere online, and download based on a platform detection script...

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-34731146
.

[DamonOehlman]

I was just looking into the hosting options, and while I'm not sure that any of the CDNs will agree to host a 8Mb binary for free, it looks like sourceforge and github have some kind of arrangement:

http://sourceforge.net/publish/?source=github

(from the github info here: https://help.github.com/articles/distributing-large-binaries)

This is probably a pretty good option, and wonderfully retro too :)

[piranna]

Another alternative is http://cdnjs.com/, and also GitHub pages can be used
to serve content as a CDN...

2014-02-11 10:38 GMT+01:00 Damon Oehlman [email protected]:

I was just looking into the hosting options, and while I'm not sure that
any of the CDNs will agree to host a 8Mb binary for free, it looks like
sourceforge and github have some kind of arrangement:

http://sourceforge.net/publish/?source=github

(from the github info here:
https://help.github.com/articles/distributing-large-binaries)

This is probably a pretty good option, and wonderfully retro too :)

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-34738743
.

"Si quieres viajar alrededor del mundo y ser invitado a hablar en un monton
de sitios diferentes, simplemente escribe un sistema operativo Unix."

• Linus Tordvals, creador del sistema operativo Linux

[modeswitch]

@DamonOehlman Thanks, I've assigned the issue to you.

We can use ghpages for this. If that doesn't work, I have a number of other options so I'm not worried about this.

[DamonOehlman]

@modeswitch Great. I think we might be asking a bit much of github pages in the long term but it will probably do for now.

Also do you know if we need to take into account the v8 version as fibers does?

https://github.com/laverdet/node-fibers/blob/master/build.js#L29

Secondly, I'm of the opinion that we should have builds available that match released package versions (and adopt a process for archiving older versions), so for example if we had binaries available now they would be stored somewhere like:

http://js-platform.github.io/node-webrtc/binaries/0.0.9/linux-x64-v8-3.14/webrtc.node

or (if the v8 version is not required):

http://js-platform.github.io/node-webrtc/binaries/0.0.9/linux-x64/webrtc.node

When version 0.0.10 was released it would sit at:

http://js-platform.github.io/node-webrtc/binaries/0.0.10

Thoughts?

[piranna]

I like the files scheme, +1.

Send from my Samsung Galaxy Note II
El 11/02/2014 23:39, "Damon Oehlman" [email protected] escribió:

@modeswitch https://github.com/modeswitch Great. I think we might be
asking a bit much of github pages in the long term but it will probably do
for now.

Also do you know if we need to take into account the v8 version as fibers
does?

https://github.com/laverdet/node-fibers/blob/master/build.js#L29

Secondly, I'm of the opinion that we should have builds available that
match released package versions (and adopt a process for archiving older
versions), so for example if we had binaries available now they would be
stored somewhere like:

http://js-platform.github.io/node-webrtc/binaries/0.0.9/linux-x64-v8-3.14/webrtc.node

or (if the v8 version is not required):

http://js-platform.github.io/node-webrtc/binaries/0.0.9/linux-x64/webrtc.node

When version 0.0.10 was released it would sit at:

http://js-platform.github.io/node-webrtc/binaries/0.0.10

Thoughts?

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-34817341
.

[modeswitch]

Also do you know if we need to take into account the v8 version as fibers does?

I believe we do.

Secondly, I'm of the opinion that we should have builds available that match released package versions (and adopt a process for archiving older versions), so for example if we had binaries available now they would be stored somewhere like:

Yep. We can look for a prebuilt version in the install script and download that.

I wonder if it's worth keeping binaries around for each release, or to have a release branch with a set of published binaries that track it. Is there a use case for grabbing old prebuilt binaries?

[DamonOehlman]

I think we should probably keep binaries around for the latest stable version and say the 2 versions prior to that. After that it should be a you have to build it yourself situation.

If we use a pathing scheme as I suggested, then attempts to get an older binary will 404 and the install script can invoke the build script....

[piranna]

We can have stored the prebuild files just only on the gh-pages branch, so
this way the main source code one doesn't get poluted nor people need to
download all of them wasting disk space.

Send from my Samsung Galaxy Note II
El 12/02/2014 07:41, "Alan K" [email protected] escribió:

Also do you know if we need to take into account the v8 version as fibers
does?

I believe we do.

Secondly, I'm of the opinion that we should have builds available that
match released package versions (and adopt a process for archiving older
versions), so for example if we had binaries available now they would be
stored somewhere like:

Yep. We can look for a prebuilt version in the install script and download
that.

I wonder if it's worth keeping binaries around for each release, or to
have a release branch with a set of published binaries that track it. Is
there a use case for grabbing old prebuilt binaries?

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-34843381
.

[jbuck]

Setup an S3 bucket, and use https://github.com/springmeyer/node-pre-gyp ?

[piranna]

Seems really nice :-) Only point, I don't understand why it requires to be
included as a bundle dependency... :-/

Send from my Samsung Galaxy Note II
El 14/02/2014 18:39, "Jon Buckley" [email protected] escribió:

Setup an S3 bucket, and use https://github.com/springmeyer/node-pre-gyp ?

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-35106232
.

[piranna]

I have been lately thinking about having a precompiled library as a .deb package and let node-webrtc bindings to link against it, and this huge list of dependencies just only confirm this. Since it's being used by other several projects (Android, C++...), why don't move libWebrtc/libJingle to an external library and just link against it instead of requiring to the users of node-webrtc to download and compile everything?

I know we are discussing about having pre-compiled binaries, but that's a half-way solution and in fact since the last time I was able to compile it I only needed to update the Javascript files since also the C++ bindings were untouched...

[DamonOehlman]

@piranna I think that a .deb package (and appropriate other binary packages for various platforms) is something that would be of benefit to both this project and other projects working against the core "libwebrtc" codebase. It's likely to be a pretty massive job though.

After having a bit of a look at google's depot_tools I can see they have put a lot of work into making cross platform builds (for chrome and sub-projects) easier, but I'm not sure there is an equivalent for packaging binaries. I recall a tool somewhere that helped simplify the process of creating the appropriate package types for the various linux package managers, but I can't remember what it is (I just remember someone built this tool because they were frustrated by the current process - which is hard).

In short, I think it might be easier for us to generate binaries for just the node-webrtc project that go down the packages path. That said I don't think we should decide before doing a bit more investigation into both options.

[piranna]

I agree that's a masive effort and an independent project, but once the
library is on .so or .a format, this could be packaged and installed "as
is", so the only pending step is just packaging.

Send from my Samsung Galaxy Note II
El 26/02/2014 10:59, "Damon Oehlman" [email protected] escribió:

@piranna https://github.com/piranna I think that a .deb package (and
appropriate other binary packages for various platforms) is something that
would be of benefit to both this project and other projects working against
the core "libwebrtc" codebase. It's likely to be a pretty massive job
though.

After having a bit of a look at google's depot_tools I can see they have
put a lot of work into making cross platform builds (for chrome and
sub-projects) easier, but I'm not sure there is an equivalent for packaging
binaries. I recall a tool somewhere that helped simplify the process of
creating the appropriate package types for the various linux package
managers, but I can't remember what it is (I just remember someone built
this tool because they were frustrated by the current process - which is
hard).

In short, I think it might be easier for us to generate binaries for just
the node-webrtc project that go down the packages path. That said I don't
think we should decide before doing a bit more investigation into both
options.

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-36108922
.

[modeswitch]

This might be helpful: https://github.com/mapbox/node-pre-gyp

There's some related discussion here: npm/npm#1891

[DamonOehlman]

That does look useful. I'll be honest and say I'm struggling with the new build process at the moment, it seems more fragile (at least on linux) than what we had before. Having a look at it now to see if can work out what is going on.

[wouldgo]

I agree with the precompiled binary for platforms and the node-pre-gyp project is very interesting.

P.S.
Just my 2 cents about the previous building script:

• it didn't provide a way to understand why the build process was faulty (plus, even if this failed, the npm install stated that the installation was ok) forcing to intervene by hand running a `depot_tools`` command;
• it didn't declare an isolated execution environment where append the downloaded depot_tools to $PATH, forcing to have these in the $PATH (another thing to do by hand);
• the options provided to the commands used are not so correct and sometimes drove the process in a faulty state.
• it was structured as a big pyramid of doom callbacks that didn't help to understand what it did.

@piranna @modeswitch @DamonOehlman can you provide some reproducible facts that is more fragile than the old?

[DamonOehlman]

@wouldgo I did say it "seems" more fragile, and I don't think either @piranna or @modeswitch have experienced the problems I have, so I think it really is just me saying this.

You have my thanks for your improvements as per your points 1 and 3. I'm not sure 2 goes far enough (as I mention in #91) and I'm not going to comment on 4 as that is something that goes down to personal coding style (as we have seen in so many promises vs callbacks debates before). No need to have another one here.

[piranna]

When we'll go fully down the Grunt way, code styles will be fixed and
probably the stabilities issues too.

Send from my Samsung Galaxy Note II
El 02/03/2014 13:45, "Damon Oehlman" [email protected] escribió:

@wouldgo https://github.com/wouldgo I did say it "seems" more fragile,
and I don't think either @piranna https://github.com/piranna or
@modeswitch https://github.com/modeswitch have experienced the problems
I have, so I think it really is just me saying this.

You have my thanks for your improvements as per your points 1 and 3. I'm
not sure 2 goes far enough (as I mention in #91#91)
and I'm not going to comment on 4 as that is something that goes down to
personal coding style (as we have seen in so many promises vs callbacks
debates before). No need to have another one here.

Reply to this email directly or view it on GitHubhttps://github.com//issues/29#issuecomment-36453236
.

[willscott]

Also worth taking a look at the packaging of https://github.com/Obvious/phantomjs

They distribute a similarly sized binary, which is hosted on bitbucket and downloaded upon install.

[DamonOehlman]

OK, Sorry I've been slack on this, going to take a look at this today / tonight...

[modeswitch]

@DamonOehlman No worries. This is a slightly big task so it might be worth breaking it up. Specifically, there's the issue of building binaries for target platforms (I think the build script will need to accept some extra flags, I started using argparse to do that). After that, the install script needs to check for available binaries and download them. Let's plan on using the gh-pages for this project initially and open another issue for moving the binaries to a more permanent home.

[DamonOehlman]

Agreed on using the gh-pages branch as previously discussed. I'm not sure if you've seen it but another command-line arg parser I can highly recommend is nopt which is used in npm itself. I've personally used it quite a few times and it's been awesome on every occasion.

[modeswitch]

That works too :)

[modeswitch]

I have initial support for this done. Any one linux x64 and node v11 should be able to download and use a prebuilt module.

[DamonOehlman]

Just tested and it works a treat on my machine:

[develop][doehlman@nicta-djo node-webrtc]$ uname -rvm
3.11.0-18-generic #32-Ubuntu SMP Tue Feb 18 21:11:14 UTC 2014 x86_64

[modeswitch]

Still needs some work:

• module_path isn't transmitted to binding.gyp
• install script needs to check for dynamically loaded dependencies before proceeding (like libnss and libssl)
• need to build more binary modules for other platforms

[wouldgo]

Tested on my workstation now and It works.

Linux: 3.11.0-19-generic #33-Ubuntu SMP Tue Mar 11 18:48:34 UTC 2014 x86_64
node: v0.10.24

👍

[modeswitch]

I refactored the build process and the native backend so this should be much easier to build. Deferring precompiled modules for now.