Xcode.munki.recipe fails at xip extract stage about recipes-for-autopkg

[nmcspadden]

I can't reproduce this. Can you please provide the full output of autopkg run -vvvv Xcode.munki, and sanitize out your Apple ID?

[nmcspadden]

I'm unable to reproduce this on either 1.4 or 2.0b3:

XcodeXIPUnpacker
{'Input': {'PKG': u'/Users/Shared/AutoPkg/Cache/com.facebook.autopkg.xcode.downloader/downloads/Xcode_11.3_beta.xip'}}
XcodeXIPUnpacker: Extracting xip archive, please be patient, this could take a long time...
XcodeXIPUnpacker: Finished xip unpack.

[mikethehill]

Output below:

Processing Xcode.munki...
Use of undefined key in variable substitution: u'RECIPE_CACHE_DIR'
{u'APPLE_ID': u'[email protected]',
 'AUTOPKG_VERSION': u'1.4',
 u'BETA': u'',
 u'DESTINATION_APP_NAME': u'Xcode.app',
 u'ICON_NAME': u'Xcode.png',
 u'MUNKI_REPO': u'/Volumes/Data',
 u'MUNKI_REPO_SUBDIR': u'apps/apple/xcode/',
 u'NAME': u'Xcode',
 u'NOSKIP': u'',
 'PARENT_RECIPES': [u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode/Xcode.munki.recipe',
                    u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode/Xcode.extract.recipe',
                    u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode/Xcode.download.recipe'],
 u'PASSWORD': u'',
 u'PASSWORD_FILE': u'secret.txt',
 u'PATTERN': u'((?!.*beta).*\\/Xcode_.*\\/Xcode.*.xip)',
 'RECIPE_CACHE_DIR': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode',
 'RECIPE_DIR': '/Users/autopkg/Library/AutoPkg/RecipeOverrides',
 'RECIPE_OVERRIDE_DIRS': ['~/Library/AutoPkg/RecipeOverrides'],
 'RECIPE_PATH': '/Users/autopkg/Library/AutoPkg/RecipeOverrides/Xcode.munki.recipe',
 u'RECIPE_REPOS': {u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.HobbitHardcase-recipes': {u'URL': u'https://github.com/autopkg/HobbitHardcase-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.apettinen-recipes': {u'URL': u'https://github.com/autopkg/apettinen-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.apizz-recipes': {u'URL': u'https://github.com/autopkg/apizz-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.arubdesu-recipes': {u'URL': u'https://github.com/autopkg/arubdesu-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.chilcote-recipes': {u'URL': u'https://github.com/autopkg/chilcote-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.dataJAR-recipes': {u'URL': u'https://github.com/autopkg/dataJAR-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.foigus-recipes': {u'URL': u'https://github.com/autopkg/foigus-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.hansen-m-recipes': {u'URL': u'https://github.com/autopkg/hansen-m-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes': {u'URL': u'https://github.com/autopkg/hjuutilainen-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.homebysix-recipes': {u'URL': u'https://github.com/autopkg/homebysix-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.its-unibas-recipes': {u'URL': u'https://github.com/autopkg/its-unibas-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.jessepeterson-recipes': {u'URL': u'https://github.com/autopkg/jessepeterson-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.jlehikoinen-recipes': {u'URL': u'https://github.com/autopkg/jlehikoinen-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.keeleysam-recipes': {u'URL': u'https://github.com/autopkg/keeleysam-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.killahquam-recipes': {u'URL': u'https://github.com/autopkg/killahquam-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.munkireport-recipes': {u'URL': u'https://github.com/autopkg/munkireport-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes': {u'URL': u'https://github.com/autopkg/recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.rtrouton-recipes': {u'URL': u'https://github.com/autopkg/rtrouton-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.scriptingosx-recipes': {u'URL': u'https://github.com/autopkg/scriptingosx-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.valdore86-recipes': {u'URL': u'https://github.com/autopkg/valdore86-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.wardsparadox-recipes': {u'URL': u'https://github.com/autopkg/wardsparadox-recipes'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg': {u'URL': u'https://github.com/facebook/Recipes-for-AutoPkg.git'},
                   u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.gitlab.techahoy.autopkg': {u'URL': u'https://gitlab.com/techahoy/autopkg'}},
 u'RECIPE_SEARCH_DIRS': [u'.',
                         u'~/Library/AutoPkg/Recipes',
                         u'/Library/AutoPkg/Recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.gitlab.techahoy.autopkg',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.hjuutilainen-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.jessepeterson-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.jlehikoinen-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.arubdesu-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.killahquam-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.munkireport-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.apizz-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.wardsparadox-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.foigus-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.its-unibas-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.dataJAR-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.homebysix-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.hansen-m-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.keeleysam-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.scriptingosx-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.chilcote-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.HobbitHardcase-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.rtrouton-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.valdore86-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.autopkg.apettinen-recipes',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg',
                         '/Users/autopkg/Library/AutoPkg/RecipeOverrides',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode',
                         u'/Users/autopkg/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Xcode'],
 u'VERSION_EMIT_PATH': u'%RECIPE_CACHE_DIR%/xcode_tag',
 u'pkginfo': {
    catalogs =     (
        testing
    );
    description = Xcode;
    "display_name" = Xcode;
    name = Xcode;
    "unattended_install" = 1;
},
 'verbose': 4}
AppleDataGatherer
{'Input': {'appID_key': u'891bd3417a7776362562d2197f89480a8547b108fd934911bcbea0110d07f757',
           'apple_id': u'[email protected]',
           'password_file': u'secret.txt'}}
AppleDataGatherer: Writing data to file
{'Output': {'data_pathname': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/login_data'}}
AppleCookieDownloader
{'Input': {'login_data': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/login_data'}}
AppleCookieDownloader: No value supplied for CURL_PATH, setting default value of: /usr/bin/curl
AppleCookieDownloader: Getting login cookie
AppleCookieDownloader: Getting download cookie
AppleCookieDownloader: Unzipping download list
{'Output': {'download_cookies': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/download_cookies'}}
AppleURLSearcher
{'Input': {'re_pattern': u'((?!.*beta).*\\/Xcode_.*\\/Xcode.*.xip)'}}
AppleURLSearcher: No value supplied for result_output_var_name, setting default value of: match
AppleURLSearcher: Beta flag not set, searching More downloads list...
AppleURLSearcher: Sorted list of possible filenames: [u'Xcode_8', u'Xcode_8.1', u'Xcode_8.2', u'Xcode_8.2.1', u'Xcode_8.3', u'Xcode_8.3.1', u'Xcode_8.3.2', u'Xcode_8.3.3', u'Xcode_9', u'Xcode_9.0.1', u'Xcode_9.1', u'Xcode_9.2', u'Xcode_9.3', u'Xcode_9.3.1', u'Xcode_9.4', u'Xcode_9.4.1', u'Xcode_10', u'Xcode_10.1', u'Xcode_10.2', u'Xcode_10.2.1', u'Xcode_10.3', u'Xcode_11', u'Xcode_11.1', u'Xcode_11.2', u'Xcode_11.2.1']
AppleURLSearcher: Found matching item: Xcode_11.2.1
AppleURLSearcher: Full URL: https://download.developer.apple.com/Developer_Tools/Xcode_11.2.1/Xcode_11.2.1.xip
{'Output': {'match': u'https://download.developer.apple.com/Developer_Tools/Xcode_11.2.1/Xcode_11.2.1.xip'}}
AppleCookieDownloader
{'Input': {'CURL_PATH': '/usr/bin/curl',
           'login_data': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/login_data'}}
AppleCookieDownloader: Getting login cookie
AppleCookieDownloader: Getting download cookie
AppleCookieDownloader: Unzipping download list
{'Output': {'download_cookies': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/download_cookies'}}
XcodeVersionEmitter
{'Input': {'dont_skip': u'',
           'output_filepath': u'%RECIPE_CACHE_DIR%/xcode_tag',
           'url': u'https://download.developer.apple.com/Developer_Tools/Xcode_11.2.1/Xcode_11.2.1.xip'}}
XcodeVersionEmitter: dont_skip is false, so skipping this Processor.
{'Output': {}}
EndOfCheckPhase
{'Input': {}}
{'Output': {}}
URLDownloader
{'Input': {'CURL_PATH': '/usr/bin/curl',
           'curl_opts': (
    "--request",
    GET,
    "--cookie",
    "/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/download_cookies"
),
           'url': u'https://download.developer.apple.com/Developer_Tools/Xcode_11.2.1/Xcode_11.2.1.xip'}}
URLDownloader: No value supplied for CHECK_FILESIZE_ONLY, setting default value of: False
URLDownloader: No value supplied for prefetch_filename, setting default value of: False
URLDownloader: Item at URL is unchanged.
URLDownloader: Using existing /Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/Xcode_11.2.1.xip
{'Output': {'pathname': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/Xcode_11.2.1.xip'}}
PathDeleter
{'Input': {'path_list': (
    "/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/login_data"
)}}
PathDeleter: Deleted /Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/login_data
{'Output': {}}
XcodeXIPUnpacker
{'Input': {'PKG': u'/Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/Xcode_11.2.1.xip'}}
XcodeXIPUnpacker: Extracting xip archive, please be patient, this could take a long time...
Traceback (most recent call last):
  File "/usr/local/bin/autopkg", line 2810, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/bin/autopkg", line 2804, in main
    exit(subcommands[verb]["function"](argv))
  File "/usr/local/bin/autopkg", line 2324, in run_recipes
    autopackager.process(recipe)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 684, in process
    log_err(unicode(err))  # noqa TODO
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 24: ordinal not in range(128)```

[nmcspadden]

What happens if you manually run xip --expand /Users/autopkg/Library/AutoPkg/Cache/local.munki.xcode/downloads/Xcode_11.2.1.xip? Sounds like there's an issue extracting the package, but it doesn't seem to be related to the recipe itself. The recipe failed because it tried to log a unicode string incorrectly (which I can fix).

[nmcspadden]

I can't reproduce this on any machine, so I'm closing this.