Complexity concern about node-gyp

[TooTallNate]

On Tue, Feb 7, 2012 at 3:52 PM, paddybyers <
[email protected]

wrote:

I don't like to stir things up, but I'm concerned that this is building
layers of complexity on top of something that should be really simple and,
more than anything, needs to be completely transparent to the developer.

All I really want to be able to do is to do

gyp (some options I understand) mymodule.gyp

Perhaps a script can take away the need to add the options. This node-gyp
seems to be going more into the direction of being an autotools - which has
the superficial allure of being super-intelligent and automatic, but in
fact makes things harder because everything becomes that bit more obscure
for the developer, and the only build options I get are the ones that the
tool chose to give me, instead of the ones that are supported by the
underlying utilities.

While not currently documented, you can pass additional gyp arguments to
the 'configure' step if you really need that. In most cases you can
specify the options in your gyp file directly.

Additionally, this module is really just a wrapper around the existing
gyp_addon script in the node source tree. I don't think it tries to be
intelligent, except for knowing whether to invoke 'msbuild' or 'make'
depending on the platform. This was definitely a problem in the gyp native
module space, which this module now takes care of (a consistent
cross-platform API to build your module).

Whatever qualms you have with the "complexity" of this module could just a
easily be said about the waf build system and node-waf.

Already looking at this I don't know, wlthout looking in detail, whether
or not I can use the xcode generator; or what options it can and cannot
pass to gyp, for example. Or what option to give it to see exactly what gyp
command is being generated, or to see what the complete compiler/linker
commands are.

Currently, it's either the MSVC target on Windows or the "make" target
elsewhere. What purpose do you need the XCode generator for? Just curious
as I have not needed it; make has worked fine. Additionally, I don't think
the current node source works with the xcode target (I saw your patch), so
if the core team doesn't care about xcode then what else can we do?

Added to all of that, we still have the issue that gyp itself is not
mature, and you often end up having to debug gyp when you first try to do
something new. That becomes a lot harder when you're trying to tell it what
to do from the other side of this stack.

Can you elaborate on that? There's 1 existing bug that prevents 64-bit
builds on OSX (I might try to work around that in node-gyp), but other
than that gyp has seemed a lot more stable to me than waf.

---

Reply to this email directly or view it on GitHub:
#11

Additionally, I'm always open to suggestions! Thanks for your concerns
though!

[TooTallNate]

Also note that if you pass -v for verbose, then the "build" step prints out the verbose output from make/msbuild, so you do see the full commands/flags being used and their entire output.

[paddybyers]

Hi Nathan,

Thanks for the response. Please understand I'm not being critical of what you've written here because I think it can make a nice utility. I'm just saying that the convenience comes at a cost if it makes it harder to understand what's going on.

Additionally, this module is really just a wrapper around the existing
gyp_addon script in the node source tree. I don't think it tries to be
intelligent, except for knowing whether to invoke 'msbuild' or 'make'
depending on the platform. This was definitely a problem in the gyp native
module space, which this module now takes care of (a consistent
cross-platform API to build your module).

Yes .. but gyp already does that:

https://github.com/joyent/node/blob/master/tools/gyp/pylib/gyp/__init__.py#L339

At some point (I can't find where right now) something was added (to gyp-node I think) to force -f make and then the other platforms broke, but I don't think it needs a wrapper to do that. We should first be figuring out why that's not working, if it isn't, rather than writing the same logic again somewhere else.

Whatever qualms you have with the "complexity" of this module could just a
easily be said about the waf build system and node-waf.

Yes, I fully agree with that. node-waf was way too opaque, and I want to make sure that we learn the lessons from that.

What purpose do you need the XCode generator for? Just curious
as I have not needed it; make has worked fine. Additionally, I don't think
the current node source works with the xcode target (I saw your patch), so
if the core team doesn't care about xcode then what else can we do?

The main point of gyp as opposed to the hundred other cross-platform build systems we could have used (Cmake, waf, Scons, ant, etc) is that it generates project file for IDEs instead of just being a build framework. For those who do want to be able to develop in the IDE, which means Xcode on Mac, then it's a big deal.

For a while it's been possible to build node using xcodebuild (I haven't tried very recently, but if it's broken then it's easily fixed) and it's a very reasonable aspiration IMO to want to be able to develop addons this way.

Furthermore, I want to be able to take advantage transparently when gyp gets extended to support additional platforms; Android for example. Did you know that with gyp you can do -f make-android (ie it supports 'flavors' of each individual generator) ?

Really, though, this is my point - whether you are right or wrong on that question (do you want/need to support xcode or not), that has ended placing an arbitrary restriction on what you can do, when that restriction wasn't there in the underlying tools. It's not a question of whether or not your view, or my view, is the right or wrong one; because other developers will legitimately have a different view and have reasonable expectation that the tools don't get in their way if they know what they want to do. The convenience is great, and having the "correct" default options for the mainstream use-cases is great, but it's bad if that comes at the cost of reducing clarity and/or functionality, so the non-mainstream use-cases are harder.

Can you elaborate on that? There's 1 existing bug that prevents 64-bit
builds on OSX (I might try to work around that in node-gyp), but other
than that gyp has seemed a lot more stable to me than waf.

Well, there was that issue we discussed fairly recently:

https://gist.github.com/1590684#gistcomment-74849

The gyp documentation on that point was incorrect, so we ended up having to debug what was going on in gyp. Sooner or later you need to start to play with gyp command lines directly, without it being wrapped in gyp-xxx or anything else.

It's not a biggie, just another example of where layers of "convenience" end up making some things harder.

And yes, please don't think I'm saying we should go back to waf. The sooner we can move across to gyp (and have it work compatibly for 0.6 as well as 0.7+) the better.

So please take this as constructive input - well done for committing the effort to this, and keep up the good work.

[TooTallNate]

Ok, I appreciate the input, it is definitely constructive. So it seems to me your main concern is limiting gyp's existing features in this abstraction, since node-gyp tries to guess the "correct" default options for most use-cases (well said).

node-gyp configure is the step that actually invokes gyp, so how about we add the ability to overwrite node-gyp's guessed target with one specified with the -f/--format parameter? We could even go on further as to have node-gyp build run xcodebuild, but I think that would just be sugar, as it seems more likely that you are compiling in the IDE (correct me if I'm wrong there).

Another option is to add a switch that makes configure not specify a format to use, and have it fall back on gyp's guessing logic (which is what I think you meant by "take advantage transparently when gyp gets extended"), which I am open to. But I don't think node-gyp forcing make on OSX will change by default, as long as node itself forces make.

Do let me know if there was another concern I'm not addressing. It's late, I'm tired; need sleep :)

[TooTallNate]

P.S. how do you do the nice quote formatting on GitHub? Responding on Gmail made it look like crap here, haha.

[paddybyers]

Hi,

node-gyp configure is the step that actually invokes gyp, so how about we add the ability to overwrite
node-gyp's guessed target with one specified with the -f/--format parameter?

Yes, I think at most you should mirror what gyp itself does, and only add your default if that option wasn't explicitly specified by the user. Then -f xcode, or any other options (including options defined in the future that we don't yet know about) should be passed through transparently.

I agree that if node itself is using make for Mac, then this has to default to the same thing.

We could even go on further as to have node-gyp build run xcodebuild, but I think that would
just be sugar, as it seems more likely that you are compiling in the IDE (correct me if I'm wrong there).

Yes, that isn't going to add anything. Running xcodebuild explicitly from the commandline will still happen (ie it's not just to build in the IDE) but you don't need help to do that.

Another option is to add a switch that makes configure not specify a format to use, and have it
fall back on gyp's guessing logic (which is what I think you meant by "take advantage transparently
when gyp gets extended"), which I am open to.

No, I'm not asking for that; only that I can pass options myself to gyp, and you add the defaults only for those options that aren't explicitly specified. Then if gyp does support some new generator, I don't have to wait for node-gyp to catch up, that's all.

P.S. how do you do the nice quote formatting on GitHub? Responding on Gmail made it look like crap here, haha.

:) It happened by accident. Just literally quote with > at the start of the line, ie

> this is quoted
> text

[TooTallNate]

@paddybyers So the configure step already passes through additional arguments that are passed through to the gyp_addon script, only currently you have to pass -- before the arguments that get passed (I'll ask @isaacs if there's a best practice way to fix that with nopt):

$ node-gyp configure -- -f xcode

I just added a check that doesn't force 'make' if another format was specified, so that should allow you to use the xcode target safely. Please let me know what else you need to make using node-gyp an enjoyable and convenient experience!

[TooTallNate]

@paddybyers I'm gonna close. Reopen or create a new ticket if you need anything else.

[paddybyers]

ok, thanks for the feedback

[isaacs]

There's no way currently to make nopt ignore all options it doesn't know about, though you can use -- to stop parsing as @TooTallNate noted. That'd be a nice feature, though.

You can get at the original argv list by looking at the parsed.argv object. It has "remain" for the remaining positional args, "cooked" for what it saw when it replaced shorthands, and "original" for the actual original argv list.