danath / gyp Goto Github PK

we should put BuildIndependentTargetsInParallel under some control in the xcode 
generator so it 
can be turned off.

Currently we end up having incremental linking off in chromium because 32-bit 
systems run out of address space. If we make this selective based on 32/64, 
we can have it on for folks with 64-bit.

What steps will reproduce the problem?
1. Run gyp -f make -DOS=freebsd build/all.gyp
2. (Don't) see the failing gyp file

Here's the output before the patch:

: flz@mayday:/home/flz/home/chrome-svn/tarball/chromium/src; ./tools/gyp/gyp -f 
make -
DOS=freebsd build/all.gyp
Traceback (most recent call last):
  File "./tools/gyp/gyp", line 14, in <module>
    sys.exit(gyp.main(sys.argv[1:]))
  File "./tools/gyp/pylib/gyp/__init__.py", line 228, in main
    includes, options.depth)
  File "./tools/gyp/pylib/gyp/__init__.py", line 61, in Load
    depth, generator_input_info)
  File "./tools/gyp/pylib/gyp/input.py", line 1573, in Load
    LoadTargetBuildFile(build_file, data, aux_data, variables, includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 281, in LoadTargetBuildFile
    includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 281, in LoadTargetBuildFile
    includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 281, in LoadTargetBuildFile
    includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 244, in LoadTargetBuildFile
    build_file_path)
  File "./tools/gyp/pylib/gyp/input.py", line 620, in ProcessVariablesAndConditionsInDict
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 636, in ProcessVariablesAndConditionsInList
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 620, in ProcessVariablesAndConditionsInDict
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 636, in ProcessVariablesAndConditionsInList
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 620, in ProcessVariablesAndConditionsInDict
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 640, in ProcessVariablesAndConditionsInList
    expanded = ExpandVariables(item, is_late, variables, build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 356, in ExpandVariables
    ' in ' + input
KeyError: 'Undefined variable source_files in <@(source_files)'


Here's the output after the patch:

: flz@mayday:/home/flz/home/chrome-svn/tarball/chromium/src; ./tools/gyp/gyp -f 
make -
DOS=freebsd build/all.gyp
Traceback (most recent call last):
  File "./tools/gyp/gyp", line 14, in <module>
    sys.exit(gyp.main(sys.argv[1:]))
  File "./tools/gyp/pylib/gyp/__init__.py", line 228, in main
    includes, options.depth)
  File "./tools/gyp/pylib/gyp/__init__.py", line 61, in Load
    depth, generator_input_info)
  File "./tools/gyp/pylib/gyp/input.py", line 1573, in Load
    LoadTargetBuildFile(build_file, data, aux_data, variables, includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 281, in LoadTargetBuildFile
    includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 281, in LoadTargetBuildFile
    includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 281, in LoadTargetBuildFile
    includes, depth)
  File "./tools/gyp/pylib/gyp/input.py", line 244, in LoadTargetBuildFile
    build_file_path)
  File "./tools/gyp/pylib/gyp/input.py", line 620, in ProcessVariablesAndConditionsInDict
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 636, in ProcessVariablesAndConditionsInList
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 620, in ProcessVariablesAndConditionsInDict
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 636, in ProcessVariablesAndConditionsInList
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 620, in ProcessVariablesAndConditionsInDict
    build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 640, in ProcessVariablesAndConditionsInList
    expanded = ExpandVariables(item, is_late, variables, build_file)
  File "./tools/gyp/pylib/gyp/input.py", line 356, in ExpandVariables
    ' in ' + build_file
KeyError: 'Undefined variable source_files in /usr/home/flz/home/chrome-
svn/tarball/chromium/src/third_party/ffmpeg/ffmpeg.gyp'

There should be a standard library suffix type variable.

Currently gyp on windows assumes that the semi-hermetic version of cygwin 
used in chromium has been configured in the registry with the setup_mount.bat 
present in the source tree. Ideally this would either not be required, or 
alternatively their should be a way to properly express this dependency.

Both cygwin and python are currently needed by msvs gyp. Ideally only python 
should be needed. The cygwin dependency makes some potential users 
uncomfortable.

Several people are hitting the issue that variables within the same dict cannot 
refer to each other. 
This limitation is problematic.
We should work on lifting it.

Apparently 'copies' is not working correctly in all cases in which the 
source file is generated in the same project.

the make generator is the only one that supports -G output_dir=name, it 
would be nice to support it on the other generators that dictate a path (xcode 
currently doesn't so it might not need to honor this)

In the case where msvs_precompiled_source is used simultaneously with 
msvs_settings, the two can clash because msvs_precompiled_source assumes that 
only a single source file will be listed. In general there should only be a 
single one, but failing with a mysterious string collision error is not a 
good failure mode. We should either support multiple, or complain more 
coherently.

What steps will reproduce the problem?
1. Try gyp -f make -DOS=freebsd
2. See that gyp file will be parsed with OS=linux
3. Cry to sleep

I'm using gyp on FreeBSD 7.2-RELEASE.

The attached patch uses the generator default variables first then the
command line variables.

What steps will reproduce the problem?
1. Set 'process_outputs_as_sources': 1 in an action as done in 
http://src.chromium.org/viewvc/chrome/trunk/src/webkit/webkit.gyp
2. See MSVC chocking on dependency processing for incremental build.

What is the expected output? What do you see instead?
MSVC200x doesn't support updating the dependency tree while processing a single 
project. The dependency tree is only updated after processing a whole project. 
So generated sources must be in a separate project from the steps that generate 
the said sources to have the dependency scanning done properly. This makes 
process_outputs_as_sources effectively broken and useless and should be removed 
accordingly. webkit.gyp and installer.gyp need to be updated to not use this 
flag anymore and support for this flag must be removed until we drop support 
for 
vcproj exports.

Currently the proper AdditionalDependencies are not being generated for 
'rule' in the windows generators. actions and copies seem to be alright.

It really is a pain that gyp litters things all over the place.
It would be nice if things got put into a parallel tree.

The version of python_24 checked into the chromium tree isn't fully hermetic 
on vista64. We should check in a fixed up version.

What steps will reproduce the problem?
1. Check out gyp into a path with spaces
2. Try to run gyp.bat
3. It doesn't work

In Windows, if a path has spaces you are expected to enclose it in quotes.  
gyp.bat fails to do this so python errors out because it can't find the 
path specified.

Using revision 536 on Windows XP SP3.

I would commit the fix myself but cygwin svn.exe gives me some error 
message that makes no sense to me.  I guess I have to set something up 
first.

Fixed batch file is attached.

xcode and scons generators currently support 'message' as an optional 
extended description that is displayed when a rule or action executes. Visual 
Studio supports this, so we should add it.

rules and actions were intended to run scripts which can assume that the full 
directory path to ouputs has been created. The windows generator currently 
does not add mkdir commands automatically to do this.

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?


Please provide any additional information below.

Currently if any target in a gyp file is used, the entire gyp file is 
processed, including targets that 
are not used. This means that included gypi's get included, even if they are 
not relevant. Ideally, this 
would not happen, as this prevents gating in and out sections of a project.

What steps will reproduce the problem?
1. Have a target of type none
2. Attempt to place a link_settings section inside of it

Take a look at ffmpeg.gyp:
http://src.chromium.org/viewvc/chrome/trunk/src/third_party/ffmpeg/ffmpeg.gyp
?revision=19579&view=markup

The settings do not get propagated to executables and shared libraries.  The 
current workaround is to place it inside of a direct_dependent_settings 
section, which will make the media lib pick up the link_settings, where it 
will then propagate to executables and shared libraries.

In the self-contained example below, "gyp -f gypd" generates an error:

KeyError: 'Undefined variable BRANDING in <(BRANDING)'

As described in the comments below, the variable *is* defined if the target 
is itself outside of a condition.  It also doesn't matter if the two 
condition blocks are swapped in the 'conditions' list (i.e., you try to put 
the condition that defines the BRANDING variable "before" the target 
condition), you get the undefined variable error regardless of the order.


{
  'variables': {
    'OS': 'win',
    'branding': 'Chrome',
    # 'BRANDING' is set in the 'conditions' section at the bottom.
  },
  # BRANDING *is* defined for this target outside of the condition:
  #'targets': [
  #  {
  #    'target_name': 'mini_installer',
  #    'type': 'none',
  #    'sources': [
  #      '<(BRANDING)',
  #    ],
  #  },
  #],
  'conditions': [
    ['OS=="win"', {
      # BRANDING is undefined for this conditional target
      'targets': [
        {
          'target_name': 'setup',
          'type': 'none',
          'sources': [
            '<(BRANDING)',
          ],
        },
      ],
    }],
    [ 'branding == "Chrome"', {
      'variables': {
         'BRANDING': '../../chrome/app/theme/google_chrome/BRANDING',
      },
    }, { # else branding!="Chrome"
      'variables': {
         'BRANDING': '../../chrome/app/theme/chromium/BRANDING',
      },
    }],
  ],
}

Due to the migration of settings out of vsprops in the debug configuration in 
gyp, FASTBUILD does not work correctly. We should figure out how to duplicate 
this correctly using gyp generation.

When does the problem occur?
The problem occurs when a user attempts to compile GYP with Python 3.0.x.

What is the expected output? What do you see instead?
Rather than a properly compiled GYP, Python outputs a plethora of errors, 
mostly syntactical.

What version of the product are you using? On what operating system?
GYP: Revision 520
OS: Windows 7 Ultimate RC1 x64
Python: Python 3.0.1 AMD64

Please provide any additional information below.
I am almost done updating the source for Python 3.

Apparently visual studio 2008 can be supported with a simple version change 
to the emitted projects/solutions. We should support this, possibly auto-
detecting which version to emit.

Since gyp only handles ascii input at this point. It should complain loudly if 
given something else.

Visual Studio normally only keeps track of changes to project properties 
(including those that affect the command-lines run) for the duration of one 
session. Nothing is kept on disk.

So for instance if you add a command line flag to a particular project. 
Visual studio will correctly understand that this project needs to be 
rebuild if you do it right away. If you exit and restart visual studio 
after having changed and saved the project, it will not detect that a 
rebuild is needed.

We can use gyp to compensate for this behavior using the following trick: 
Store a file containing the complete set of information known at gyp time 
related to each configuration of a target. Then add a dependency on that 
file to its related target. Only touch the file if the settings have 
changed.

vs2008 shows the projects in the sln out of order.
Presumably we need to sort them by name in the sln.

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

What steps will reproduce the problem?
 'Release': { }
whereas not:: 'Release': { 'foo': 'bar' }

What is the expected output? What do you see instead?
Not a syntax error!

On windows if an include_dir uses a variable whose name has underscores, the 
contents of the 
variable do not seem to be escaped correctly with regard to backslashes. The 
path is interpreted as 
single \path which causes stray control characters to appear in the generated 
vcproj files.

beng was trying to have app.gyp:app and chrome.gyp:app, one was a library, 
one an executable that used the library. This caused inconsistent behavior 
on some machines and some builds. Also it is not clear how you would build 
only one module at the command line.

We should either fix this, or detect it and complain on all the generators.

It would be nice to be able to build libraries like base without having to 
drag in icu. Unfortunately, when you depend on another library, there's no 
easy way to do so without dragging in all the link dependencies associated 
with it. A global variable could gate things out project wide, but what if 
you want to build one way in some places but not others in the same build?

If you run gyp from somewhere other than the top of the source tree on windows, 
it generates 
junked up slns that show only one of the vcproj files.
Path handling likely got messed up at some point.

The msvs generator uses the built-in xml module. This module accepts an eol 
parameter which can 
be used to specify an alternate line ending. This is used for the msvs 
generator since visual studio 
and incredibuild expect crlf. Unfortunately some versions of python2.4 
(including unfortunately the 
one chromium has checked in) ignore this parameter when emitting the xml 
header. Fortunately, 
visual studio and incredibuild will accept either variant for the header. But 
it would be nice to be 
consistent. We should find an alternative that works on all python versions.

It would be good if the startup project was predictable...

The 'copies' command be used to copy individual files into a destination 
directory.  It would be nice to have a command to copy a directory tree 
instead of just a single file.

Currently changes to slns/vcproj made in the IDE move things around a lot. We 
should canonicalize 
them by uppercasing and sorting them.

Currently dependencies have no ordering guarantees on windows. It would be 
useful to have the option to do this (possibly for release builds).

The attached patch defines a default generator for FreeBSD 7.x and 8.x.

It also changes the shebang line in gyp from /usr/bin/python to
/usr/bin/env python, as python doesn't live in /usr/bin on FreeBSD. The
same change could be done to all python sources but gyp is the only one
being called directly.

Currently since actions in msvs need to be attached to a particular file,
we pick the second file in the list, unless there is only one.
This works ok, but is a hack.
We should fix it.

A syntax like the following should work, but currently you need to 
explicitly gate precompiled headers from windows out on other platforms, 
but not on windows.
{
   'sources': [
         .....
       'bob_precompiled.cc',
       'bob_precompiled.h',
    }
    'sources!': [
       'bob_precompiled.cc',
       'bob_precompiled.h',
    ],
    'configurations': {
        'Debug': {
             'msvs_precompiled_header': 'bob_precompiled.h',
             'msvs_precompiled_source': 'bob_precompiled.cc',
         },
     },  
}

Specifying raw msvs_settings for incremental linking and subsystem is error 
prone enough that we should add explicit gyp level msvs_ options for this.

Currently actions without inputs emit a useless out of range exception.
It should let you know what's wrong.

Currently, sln files are generated which list all targets in the current 
directory at the top level, and all others under a folder called 
'dependencies'. This is okay for smaller modules, but webkit and chromium 
would benefit from more specific grouping.

There are two options:
1. Support an extended syntax which would allow category groupings like 
'tools', 'tests', 'submodules' similar to the pre-gyp sln files.
2. Do something intelligent with actual target dependencies. For instance 
list only direct dependencies in the main 'dependencies' folder and have 
subdependencies list down a layer deeper.

There is no direct equivalent of this problem for xcode, since targets 
outside the current gyp file are external project references. This 
unfortunately doesn't work in visual studio, which needs fully populated 
sln files.

Even if 'process_outputs_as_sources' is 0, an output lib seems to get treated 
as an input to the lib 
line, this is bad. Caused a problem with tcmalloc's libcmt magic.

GYP_GENERATOR_FLAGS doesn't exist yet but we
should add it and make it work with --generator-flags the same way
that GYP_DEFINES works with -D

If you have a .c file as an input to an action, the msvs generator will also 
link it into the target.

gyp seems to provide no way to modify the output file of a target. I'd like 
to be able to change the file name of the output file (e.g., to add a 
"_debug" suffix to the name in Debug configurations).

IDL files which are inputs to actions end up getting treated as MIDL files by 
the IDE.
This doesn't happen if a rule define something else for .idl
This is just for actions.