facebook / Recipes-for-AutoPkg

Collection of AutoPkg recipes authored at Facebook.

Facebook's AutoPkg Recipes

This repository is a collection of recipes for use with AutoPkg. This collection represents AutoPkg recipes that the Client Platform Engineering (CPE) team has authored.

You can see other projects from the CPE team at the IT-CPE GitHub repo.

Usage

Clone the repo using autopkg repo-add or git clone: autopkg repo-add https://github.com/facebook/Recipes-for-AutoPkg.git
or
git clone https://github.com/facebook/Recipes-for-AutoPkg.git ~/Library/AutoPkg/RecipeRepos/com.github.autopkg.facebook-recipes

Then run any of the recipes with autopkg run.

Recipe List

Android NDK

The Android Native Development Kit recipes a custom processor: "AndroidNDKVersioner".

There are two versions of note for the NDK: the release number, which is arbitrarily determined by Google, and the version number inside the download. Release number is determined naively by parsing the download file name itself. The version is parsed by looking in the "source.properties" file for the value of "Pkg.Version".

Unlike prior versions of this recipe, this is now idempotent.

Android SDK

The Android Software Development Kit recipe uses a number of custom processors to determine versioning and URLs.

This recipe parses two XML files that Google uses to inform the URLs, versions, and names of all packages related to the SDK. The Android SDK installer, which is a Java app, can download all of these manually (and uses those two XML URLs to figure out what to download).

This is a minimal set of packages necessary to start developing with the Android SDK. It is not the complete SDK.

The Android SDK requires Java 7 in order to function.

BlueJeans

BlueJeans is a video conferencing tool. Previously, BlueJeans was only available as browser plugins, but they now offer desktop applications for Windows and OS X.

This recipe will download the BlueJeans installer package and create a straightforward app-dmg that copies the BlueJeans.app into /Applications.

DbVisualizer

DbVisualizer is a database visualizer tool. The software has a Free version, and can be upgraded into Pro version with proper licensing.

Duo

Duo is a 2 factor authentication system. This recipe downloads the latest version of Duo Unix and compiles it to be placed into /usr/local/bin/ rather than /usr/bin/, for compatibility with 10.11 El Capitan.

Since this is a standard configure / make / make install Linux installer, the custom processors provided here will create a tiny sandbox to do the custom install into the AutoPkg cache, so it won't affect your current system.

Intellij IDEA

IntelliJ IDEA is a lightweight developer IDE for Java, Groovy, and Scala.

This recipe contains a custom URL provider, "IntellijURLProvider" that searches their versioning API for the latest download.

Lobby Video

This recipe is more of a proof of concept than a usable tool for others. This recipe serves as an example for a way to take an unversionable binary file, such as a large movie file, and create a versionable item that Munki can handle.

This particular example uses a custom processor "DateVersioner" that simply uses the current date and time to name the item, and set a version for Munki importing.

Since the version is not deterministic and not repeatable unless you change the system clock to go back in time, this recipe is not idempotent. It should be run manually whenever the source media file is updated.

Mosh

The Mobile Shell, known as "Mosh", is a remote terminal application ideal for certain network conditions and environments.

Mosh can be obtained via GitHub and built from source, installed via Homebrew, or downloaded as a prebuilt package. This recipe set provides downloads for GitHub releases as well as the prebuilt package hosted on the website.

Two custom processors are included, "FBGitHubReleasesInfoProvider" and "Mosh Versioner".

Versioning is accomplished with the custom processor by naively looking at the package name, which is fragile and dependent on the developers maintaining consistency with filenames.

The "FBGitHubReleasesInfoProvider" is nearly identical to the AutoPkg GitHubReleasesInfoProvider processor, but contains some additional logic to parse the correct version, since the developers have been listing the release version as "mosh-1.2.x".

SQLDeveloper

SQLDeveloper is an integrated development environment from Oracle that makes it easier to work with Oracle Databases and database applications.

Unfortunately, the software is a native .app wrapped around a Java app, and the native .app version is static and does not change with updates. A custom versioning processor looks for a file inside the Java core that thankfully is updated with new builds when new versions are released.

VMware Fusion (Mass Deploy)

VMWare Fusion needs no introduction. This recipe follows the instructions on creating a mass-deployment package that is pre-serialized.

The parent recipe for this is in Justin Rummel's VMware Fusion recipe set. You must add Justin Rummel's recipes to your search directories before you can use this recipe:
autopkg repo-add justinrummel-recipes

To use this recipe, you must, at a bare minimum, create an override and replace the license key with your own. The deploy.ini file contents are an input variable, so you can place in your own settings as you see fit.

Though this recipe does include both .pkg and .munki variants, the .pkg recipe only creates a bundle package, not a flat package, and therefore will need to be put into a DMG / archive of some sort to be used with other deployment mechanisms (JSS, Absolute, etc.).

Shared Processors

Facebook also provides a number of Shared Processors that can be used in multiple processors.

Facebook's common convention for custom processors is to bundle together recipe-specific tools with the recipe itself - such as unique Versioners, URL providers, etc. Some processors serve useful purposes for many recipes (such as rsync), and therefore make more sense to be shared in a common location.

These shared processors can be referred to by using the stub recipe identifier com.facebook.autopkg.shared:

    <dict>
      <key>Processor</key>
      <string>com.facebook.autopkg.shared/SubDirectoryList</string>
      <key>Comment</key>
      <string>Get list of folder contents</string>
      <key>Arguments</key>
      <dict>
        <key>root_path</key>
        <string>%RECIPE_CACHE_DIR%/unpack</string>
      </dict>
    </dict>

See the README inside the Shared_Processors directory for more details about the provided processors.

Recipes-for-AutoPkg's Issues

IntelliJ update address has changed

The JetBrains URL used by the IntellijURLProvider AutoPkg processer appears to be no longer updated:

http://www.jetbrains.com/js2/version.js

Reference: https://github.com/facebook/Recipes-for-AutoPkg/blob/master/Intellij/IntellijURLProvider.py#L19

The new update address appears to be the URL below:

http://www.jetbrains.com/updates/updates.xml

Reference: https://github.com/JetBrains/intellij-community/blob/master/community-resources/src/idea/IdeaApplicationInfo.xml#L16

Could the IntellijURLProvider processer please be updated to use the new update URL?

Xcode.munki.recipe fails at xip extract stage

Running AutoPkg v1.4

XcodeXIPUnpacker: Extracting xip archive, please be patient, this could take a long time... Traceback (most recent call last): File "/usr/local/bin/autopkg", line 2810, in <module> sys.exit(main(sys.argv)) File "/usr/local/bin/autopkg", line 2804, in main exit(subcommands[verb]["function"](argv)) File "/usr/local/bin/autopkg", line 2324, in run_recipes autopackager.process(recipe) File "/Library/AutoPkg/autopkglib/__init__.py", line 698, in process log_err(unicode(err)) # noqa TODO UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 24: ordinal not in range(128)

VMwareFusionDeploy should have an installs item, minimum_os_version

In the past when I've deployed VMwareFusion with the deployment pkg, I've also added an installs item of VMware Fusion.app, because VMware's included package is simply 1.0 and so Munki doesn't have enough to go on to keep it up to date. Also, it would be good to use the LSMinimumSystemVersion in the app for the pkginfo's minimum_os_version.

Would you consider a pull request that adds these?

Thanks for this recipe!

Xcode names on disk as variable for overrides

It would be most excellent if the names on disk could be a variable that one could employ in an override for the autopkg.

The .munki recipes for Xcode include version and beta numbers in the names on disk.

• /Applications/%NAME%_%version%.app
• /Applications/%NAME%_%version%_beta_%BETA_NUMBER%.app

Xcode, as if from app store, comes as:

• /Applications/%NAME%.app
• /Applications/%NAME%-beta.app

Perhaps the name could use new variable like %APP_NAME%.

• /Applications/%APP_NAME% where %APP_NAME% == %NAME%.app
• /Applications/%APP_NAME% where %APP_NAME% == %NAME%-beta.app

rsync processor cannot be used with a mix of short and long arguements e.g. `-Phav --delete`

Mixing short and long arguments results in an unrecognised argument error as the whole argument string is sent as one argument. This is mostly fine except if you need to use delete as there is no short argument for delete.

Xcode code signature

Could a code signature verification step be added to the Xcode.extract.recipe? This can make it easier to check Xcode download is trusted.

Mosh URL has changed

       <key>Processor</key>
        <string>URLTextSearcher</string>
        <key>Arguments</key>
        <dict>
            <key>url</key>
            <string>https://mosh.org/#getting</string>
            <key>re_pattern</key>
            <string>https://mosh.org/mosh-[0-9,.]\..\..\.pkg</string>

URL is https://mosh.mit.edu/#getting on https://github.com/facebook/Recipes-for-AutoPkg/blob/master/Mosh/mosh.download.recipe
Snippet above has working URL. https://mosh.org/#getting

Xcode Extract Recipe with 9.2

In its current form the XcodeXip extract processor doesn’t work with 9.2. @pudquick if you could sync the internal changes that would be super swell. 🤗

It looks like Xcode 9 was patched with 76e4f98

but 9.2 keeps giving me the following ditto error
ditto: cpio read error: bad file format

IntellijURLProvider generating invalid download links

When running Intellij.download.recipe, IntellijURLProvider fails with the following output:

<key>RecipeError</key>
		<string>Error in com.facebook.autopkg.download.intellij: Processor: URLDownloader: Error: Curl failure: The requested URL returned error: 403 Forbidden (exit code 22)</string>

The URL currently generated by the processor, https://download.jetbrains.com/idea/ideaIC-2.1.3.dmg, does not appear to point to a valid download.

Missing processor

I'm getting this error:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<array>
	<dict>
		<key>RecipeError</key>
		<string>Unknown processor 'XcodeBuildNumberEmitter'.</string>
	</dict>
</array>
</plist>

I deleted and made again the override. I'm always getting this error and I can't find this processor.

Xcode recipe crashes Autopkg's Python

I am seeing Autopkg's Python crash when running the Xcode recipe.

Run output:

Processing Xcode.munki.recipe...
Use of undefined key in variable substitution: 'RECIPE_CACHE_DIR'
Use of undefined key in variable substitution: 'RECIPE_CACHE_DIR'
AppleDataGatherer
AppleDataGatherer: Writing data to file
AppleCookieDownloader
AppleCookieDownloader: Getting login cookie
AppleCookieDownloader: Getting download cookie
AppleCookieDownloader: Unzipping download list
AppleURLSearcher
AppleURLSearcher: Beta flag not set, searching More downloads list...
AppleURLSearcher: Found matching item: Xcode_13
AppleURLSearcher: Full URL: https://download.developer.apple.com/Developer_Tools/Xcode_13/Xcode_13.xip
AppleCookieDownloader
AppleCookieDownloader: Getting login cookie
AppleCookieDownloader: Getting download cookie
AppleCookieDownloader: Unzipping download list
XcodeVersionEmitter
XcodeVersionEmitter: dont_skip is false, so skipping this Processor.
EndOfCheckPhase
URLDownloader
URLDownloader: Item at URL is unchanged.
URLDownloader: Using existing /Users/graham_gilbert/Library/AutoPkg/Cache/local.munki.Xcode/downloads/Xcode_13.xip
PathDeleter
PathDeleter: Deleted /Users/graham_gilbert/Library/AutoPkg/Cache/local.munki.Xcode/downloads/login_data
XcodeXIPUnpacker
XcodeXIPUnpacker: Extracting xip archive, please be patient, this could take a long time...
XcodeXIPUnpacker: Finished xip unpack.
FileFinder
FileFinder: Found file match: '/Users/graham_gilbert/Library/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/Xcode.app' from globbed '/Users/graham_gilbert/Library/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/*.app'
CodeSignatureVerifier
CodeSignatureVerifier: Code signature verification disabled for this recipe run.
PlistReader
PlistReader: Reading: /Users/graham_gilbert/Library/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/Xcode.app/Contents/version.plist
PlistReader: Assigning value of '19234' to output variable 'bundle_version'
PlistReader: Assigning value of '13.0' to output variable 'version'
PlistReader: Assigning value of '13A233' to output variable 'build_version'
XcodeVersioner
XcodeVersioner: Major version: 13
XcodeVersioner: Minor version: 0
XcodeVersioner: Normalizing patch to 0
/opt/airbnb/bin/run_autopkg: line 26: 63533 Segmentation fault: 11  AUTOPKG run "${@}" --prefs /tmp/autopkg_prefs.json --ignore-parent-trust-verification-errors -v

macOS Crash Dump

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Python [63533]
Path:                  /Library/AutoPkg/*/Python.framework/Versions/3.7/Resources/Python.app/Contents/MacOS/Python
Identifier:            org.python.python
Version:               3.7.5 (3.7.5)
Code Type:             X86-64 (Native)
Parent Process:        Exited process [63531]
Responsible:           iTerm2 [24933]
User ID:               501

Date/Time:             2021-09-23 09:58:57.0351 -0700
OS Version:            macOS 12.0 (21A5522h)
Report Version:        12
Bridge OS Version:     6.0 (19P50522d)
Anonymous UUID:        1330ED8A-9CBF-9F19-594E-ADD257085B8B

Sleep/Wake UUID:       F50E61FA-7EAA-437D-96AF-49BDC54869DF

Time Awake Since Boot: 50000 seconds
Time Since Wake:       4428 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000000000000000
Exception Codes:       0x0000000000000001, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [63533]

VM Region Info: 0 is not in any region.  Bytes before following region: 4400484352
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      UNUSED SPACE AT START
--->  
      __TEXT                      1064a1000-1064a2000    [    4K] r-x/rwx SM=COW  .../MacOS/Python

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   ???                           	               0x0 ???
1   DVTFoundation                 	       0x10a41d4f1 type metadata completion function for DVTCoreDevice + 97
2   libswiftCore.dylib            	    0x7ff8293f49bb swift_getSingletonMetadata + 923
3   DVTFoundation                 	       0x10a41d47d type metadata accessor for DVTCoreDevice + 29
4   libobjc.A.dylib               	    0x7ff81b61f087 realizeAllClasses() + 111
5   libobjc.A.dylib               	    0x7ff81b6238f9 objc_getClassList + 59
6   _objc.cpython-37m-darwin.so   	       0x106c32fea PyObjC_GetClassList + 26
7   _objc.cpython-37m-darwin.so   	       0x106c27014 loadBundle + 372
8   Python                        	       0x1068e908b _PyMethodDef_RawFastCallKeywords + 379
9   Python                        	       0x1068e85ba _PyCFunction_FastCallKeywords + 42
10  Python                        	       0x1069a7804 call_function + 724
11  Python                        	       0x1069a4924 _PyEval_EvalFrameDefault + 25508
12  Python                        	       0x1068e89f0 function_code_fastcall + 128
13  Python                        	       0x1069a7812 call_function + 738
14  Python                        	       0x1069a47c3 _PyEval_EvalFrameDefault + 25155
15  Python                        	       0x1068e89f0 function_code_fastcall + 128
16  Python                        	       0x1069a7812 call_function + 738
17  Python                        	       0x1069a47c3 _PyEval_EvalFrameDefault + 25155
18  Python                        	       0x1068e89f0 function_code_fastcall + 128
19  Python                        	       0x1069a7812 call_function + 738
20  Python                        	       0x1069a47c3 _PyEval_EvalFrameDefault + 25155
21  Python                        	       0x1068e89f0 function_code_fastcall + 128
22  Python                        	       0x1069a7812 call_function + 738
23  Python                        	       0x1069a47c3 _PyEval_EvalFrameDefault + 25155
24  Python                        	       0x1068e89f0 function_code_fastcall + 128
25  Python                        	       0x1069a7812 call_function + 738
26  Python                        	       0x1069a47c3 _PyEval_EvalFrameDefault + 25155
27  Python                        	       0x1069a8363 _PyEval_EvalCodeWithName + 2467
28  Python                        	       0x1068e8581 _PyFunction_FastCallKeywords + 257
29  Python                        	       0x1069a7812 call_function + 738
30  Python                        	       0x1069a487e _PyEval_EvalFrameDefault + 25342
31  Python                        	       0x1068e89f0 function_code_fastcall + 128
32  Python                        	       0x1069a7812 call_function + 738
33  Python                        	       0x1069a487e _PyEval_EvalFrameDefault + 25342
34  Python                        	       0x1069a8363 _PyEval_EvalCodeWithName + 2467
35  Python                        	       0x10699e4a4 PyEval_EvalCode + 100
36  Python                        	       0x1069dbb91 PyRun_FileExFlags + 209
37  Python                        	       0x1069db40a PyRun_SimpleFileExFlags + 890
38  Python                        	       0x1069fa893 pymain_main + 6915
39  Python                        	       0x1069fadfa _Py_UnixMain + 58
40  dyld                          	       0x1112164d5 start + 421

Thread 1:
0   libsystem_pthread.dylib       	    0x7ff81b77000c start_wqthread + 0


Thread 0 crashed with X86 Thread State (64-bit):
  rax: 0x00007ff85e687a08  rbx: 0x00007ff85e687b68  rcx: 0xffffffffffdefbb4  rdx: 0x0000000000000000
  rdi: 0x000000000000013f  rsi: 0x00007ff7b9a5bed0  rbp: 0x00007ff7b9a5bf50  rsp: 0x00007ff7b9a5bdc8
   r8: 0x0000000000000001   r9: 0x000000000000003f  r10: 0xffffffffffffffff  r11: 0x0000000000000012
  r12: 0x000000010a62db90  r13: 0x0000000000000000  r14: 0x000000010a7ec8d0  r15: 0x000000010a62dba8
  rip: 0x0000000000000000  rfl: 0x0000000000010202  cr2: 0x0000000000000000
  
Logical CPU:     7
Error Code:      0x00000014 (no mapping for user instruction read)
Trap Number:     14


Binary Images:
               0x0 - 0xffffffffffffffff ??? (*) <00000000-0000-0000-0000-000000000000> ???
       0x10a0d6000 -        0x10a735fff com.apple.dt.DVTFoundation (13.0) <d563257a-b418-387a-8d9b-08faf2632851> /Users/USER/Library/AutoPkg/*/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/Versions/A/DVTFoundation
    0x7ff8290f4000 -     0x7ff8294adfff libswiftCore.dylib (*) <8dc93af1-1409-3829-a971-513405d1c92a> /usr/lib/swift/libswiftCore.dylib
    0x7ff81b608000 -     0x7ff81b641fff libobjc.A.dylib (*) <00864e0c-0325-380e-bf76-b3fc47f3d5e8> /usr/lib/libobjc.A.dylib
       0x106c04000 -        0x106c56fff _objc.cpython-37m-darwin.so (*) <8295b1a1-1136-3e32-aee9-c82b3feb4b21> /Library/AutoPkg/*/Python.framework/Versions/3.7/lib/python3.7/site-packages/objc/_objc.cpython-37m-darwin.so
       0x1068c7000 -        0x106aa0fff org.python.python (3.7.5, (c) 2001-2019 Python Software Foundation.) <62582eaa-87f5-3553-8feb-c0a33b8660c8> /Library/AutoPkg/*/Python.framework/Versions/3.7/Python
       0x111211000 -        0x111278fff dyld (*) <9fc670fc-4d35-3582-a044-8f06bf2528af> /usr/lib/dyld
    0x7ff81b76e000 -     0x7ff81b779fff libsystem_pthread.dylib (*) <ad356414-9c4c-3df7-a0e6-57ccac56de46> /usr/lib/system/libsystem_pthread.dylib

External Modification Summary:
  Calls made by other processes targeting this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by this process:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0
  Calls made by all processes on this machine:
    task_for_pid: 0
    thread_create: 0
    thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=942.8M resident=0K(0%) swapped_out_or_unallocated=942.8M(100%)
Writable regions: Total=1.6G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.6G(100%)

                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Activity Tracing                   256K        1 
Kernel Alloc Once                    8K        1 
MALLOC                           264.1M       33 
MALLOC guard page                   24K        5 
MALLOC_MEDIUM (reserved)         960.0M        8         reserved VM address space (unallocated)
MALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)
STACK GUARD                          8K        2 
Stack                             16.5M        3 
VM_ALLOCATE                        252K        4 
__CTF                               759        1 
__DATA                            23.5M      454 
__DATA_CONST                      17.9M      253 
__DATA_DIRTY                      1061K      138 
__FONT_DATA                          4K        1 
__LINKEDIT                       642.8M       56 
__OBJC_RO                         81.4M        1 
__OBJC_RW                         3104K        2 
__TEXT                           300.1M      471 
__UNICODE                          588K        1 
dyld private memory               1024K        1 
mapped file                       28.3M        2 
shared memory                       60K        6 
===========                     =======  ======= 
TOTAL                              2.7G     1445 
TOTAL, minus reserved VM space     1.3G     1445 



-----------
Full Report
-----------

{"app_name":"Python","timestamp":"2021-09-23 09:58:58.00 -0700","app_version":"3.7.5","slice_uuid":"4b030ec4-815e-34b7-90e7-d0720c31e072","build_version":"3.7.5","platform":1,"bundleID":"org.python.python","share_with_app_devs":1,"is_first_party":0,"bug_type":"309","os_version":"macOS 12.0 (21A5522h)","incident_id":"40B1CF16-1EA3-450B-A075-971F81D33126","name":"Python"}
{
  "uptime" : 50000,
  "procLaunch" : "2021-09-23 09:28:47.6975 -0700",
  "procRole" : "Unspecified",
  "version" : 2,
  "userID" : 501,
  "deployVersion" : 210,
  "modelCode" : "MacBookPro16,2",
  "procStartAbsTime" : 48433801072053,
  "coalitionID" : 806,
  "osVersion" : {
    "train" : "macOS 12.0",
    "build" : "21A5522h",
    "releaseType" : "User"
  },
  "captureTime" : "2021-09-23 09:58:57.0351 -0700",
  "incident" : "40B1CF16-1EA3-450B-A075-971F81D33126",
  "bug_type" : "309",
  "pid" : 63533,
  "procExitAbsTime" : 50243163209979,
  "cpuType" : "X86-64",
  "procName" : "Python",
  "procPath" : "\/Library\/AutoPkg\/*\/Python.framework\/Versions\/3.7\/Resources\/Python.app\/Contents\/MacOS\/Python",
  "bundleInfo" : {"CFBundleShortVersionString":"3.7.5","CFBundleVersion":"3.7.5","CFBundleIdentifier":"org.python.python"},
  "storeInfo" : {"deviceIdentifierForVendor":"8A1598B5-7CFE-5835-85CB-799EFB237C05","thirdParty":true},
  "parentProc" : "Exited process",
  "parentPid" : 63531,
  "coalitionName" : "com.googlecode.iterm2",
  "crashReporterKey" : "1330ED8A-9CBF-9F19-594E-ADD257085B8B",
  "responsiblePid" : 24933,
  "responsibleProc" : "iTerm2",
  "wakeTime" : 4428,
  "bridgeVersion" : {"build":"19P50522d","train":"6.0"},
  "sleepWakeUUID" : "F50E61FA-7EAA-437D-96AF-49BDC54869DF",
  "sip" : "enabled",
  "vmRegionInfo" : "0 is not in any region.  Bytes before following region: 4400484352\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  \n      __TEXT                      1064a1000-1064a2000    [    4K] r-x\/rwx SM=COW  ...\/MacOS\/Python",
  "isCorpse" : 1,
  "exception" : {"codes":"0x0000000000000001, 0x0000000000000000","rawCodes":[1,0],"type":"EXC_BAD_ACCESS","signal":"SIGSEGV","subtype":"KERN_INVALID_ADDRESS at 0x0000000000000000"},
  "termination" : {"flags":0,"code":11,"namespace":"SIGNAL","indicator":"Segmentation fault: 11","byProc":"exc handler","byPid":63533},
  "vmregioninfo" : "0 is not in any region.  Bytes before following region: 4400484352\n      REGION TYPE                    START - END         [ VSIZE] PRT\/MAX SHRMOD  REGION DETAIL\n      UNUSED SPACE AT START\n--->  \n      __TEXT                      1064a1000-1064a2000    [    4K] r-x\/rwx SM=COW  ...\/MacOS\/Python",
  "extMods" : {"caller":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"system":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"targeted":{"thread_create":0,"thread_set_state":0,"task_for_pid":0},"warnings":0},
  "faultingThread" : 0,
  "threads" : [{"triggered":true,"id":696315,"threadState":{"r13":{"value":0},"rax":{"value":140704712522248,"symbolLocation":64,"symbol":"value witness table for Builtin.NativeObject"},"rflags":{"value":66050},"cpu":{"value":7},"r14":{"value":4471048400,"symbolLocation":0,"symbol":"type metadata for DVTCoreDevice"},"rsi":{"value":140701948296912},"r8":{"value":1},"cr2":{"value":0},"rdx":{"value":0},"r10":{"value":18446744073709551615},"r9":{"value":63},"r15":{"value":4469218216,"symbolLocation":32,"symbol":"method descriptor for DVTCoreDevice.DDILocation()"},"rbx":{"value":140704712522600,"symbolLocation":64,"symbol":"value witness table for Builtin.UnknownObject"},"trap":{"value":14,"description":"(no mapping for user instruction read)"},"err":{"value":20},"r11":{"value":18},"rip":{"value":0},"rbp":{"value":140701948297040},"rsp":{"value":140701948296648},"r12":{"value":4469218192,"symbolLocation":8,"symbol":"method descriptor for DVTCoreDevice.DDILocation()"},"rcx":{"value":18446744073707387828},"flavor":"x86_THREAD_STATE","rdi":{"value":319}},"queue":"com.apple.main-thread","frames":[{"imageOffset":0,"imageIndex":0},{"imageOffset":3437809,"symbol":"type metadata completion function for DVTCoreDevice","symbolLocation":97,"imageIndex":1},{"imageOffset":3148219,"symbol":"swift_getSingletonMetadata","symbolLocation":923,"imageIndex":2},{"imageOffset":3437693,"symbol":"type metadata accessor for DVTCoreDevice","symbolLocation":29,"imageIndex":1},{"imageOffset":94343,"symbol":"realizeAllClasses()","symbolLocation":111,"imageIndex":3},{"imageOffset":112889,"symbol":"objc_getClassList","symbolLocation":59,"imageIndex":3},{"imageOffset":192490,"symbol":"PyObjC_GetClassList","symbolLocation":26,"imageIndex":4},{"imageOffset":143380,"symbol":"loadBundle","symbolLocation":372,"imageIndex":4},{"imageOffset":139403,"symbol":"_PyMethodDef_RawFastCallKeywords","symbolLocation":379,"imageIndex":5},{"imageOffset":136634,"symbol":"_PyCFunction_FastCallKeywords","symbolLocation":42,"imageIndex":5},{"imageOffset":919556,"symbol":"call_function","symbolLocation":724,"imageIndex":5},{"imageOffset":907556,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25508,"imageIndex":5},{"imageOffset":137712,"symbol":"function_code_fastcall","symbolLocation":128,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907203,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25155,"imageIndex":5},{"imageOffset":137712,"symbol":"function_code_fastcall","symbolLocation":128,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907203,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25155,"imageIndex":5},{"imageOffset":137712,"symbol":"function_code_fastcall","symbolLocation":128,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907203,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25155,"imageIndex":5},{"imageOffset":137712,"symbol":"function_code_fastcall","symbolLocation":128,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907203,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25155,"imageIndex":5},{"imageOffset":137712,"symbol":"function_code_fastcall","symbolLocation":128,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907203,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25155,"imageIndex":5},{"imageOffset":922467,"symbol":"_PyEval_EvalCodeWithName","symbolLocation":2467,"imageIndex":5},{"imageOffset":136577,"symbol":"_PyFunction_FastCallKeywords","symbolLocation":257,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907390,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25342,"imageIndex":5},{"imageOffset":137712,"symbol":"function_code_fastcall","symbolLocation":128,"imageIndex":5},{"imageOffset":919570,"symbol":"call_function","symbolLocation":738,"imageIndex":5},{"imageOffset":907390,"symbol":"_PyEval_EvalFrameDefault","symbolLocation":25342,"imageIndex":5},{"imageOffset":922467,"symbol":"_PyEval_EvalCodeWithName","symbolLocation":2467,"imageIndex":5},{"imageOffset":881828,"symbol":"PyEval_EvalCode","symbolLocation":100,"imageIndex":5},{"imageOffset":1133457,"symbol":"PyRun_FileExFlags","symbolLocation":209,"imageIndex":5},{"imageOffset":1131530,"symbol":"PyRun_SimpleFileExFlags","symbolLocation":890,"imageIndex":5},{"imageOffset":1259667,"symbol":"pymain_main","symbolLocation":6915,"imageIndex":5},{"imageOffset":1261050,"symbol":"_Py_UnixMain","symbolLocation":58,"imageIndex":5},{"imageOffset":21717,"symbol":"start","symbolLocation":421,"imageIndex":6}]},{"id":696320,"frames":[{"imageOffset":8204,"symbol":"start_wqthread","symbolLocation":0,"imageIndex":7}]}],
  "usedImages" : [
  {
    "size" : 0,
    "source" : "A",
    "base" : 0,
    "uuid" : "00000000-0000-0000-0000-000000000000"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4463616000,
    "CFBundleShortVersionString" : "13.0",
    "CFBundleIdentifier" : "com.apple.dt.DVTFoundation",
    "size" : 6684672,
    "uuid" : "d563257a-b418-387a-8d9b-08faf2632851",
    "path" : "\/Users\/USER\/Library\/AutoPkg\/*\/Xcode.app\/Contents\/SharedFrameworks\/DVTFoundation.framework\/Versions\/A\/DVTFoundation",
    "name" : "DVTFoundation",
    "CFBundleVersion" : "19207"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703817482240,
    "size" : 3907584,
    "uuid" : "8dc93af1-1409-3829-a971-513405d1c92a",
    "path" : "\/usr\/lib\/swift\/libswiftCore.dylib",
    "name" : "libswiftCore.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64h",
    "base" : 140703587926016,
    "size" : 237568,
    "uuid" : "00864e0c-0325-380e-bf76-b3fc47f3d5e8",
    "path" : "\/usr\/lib\/libobjc.A.dylib",
    "name" : "libobjc.A.dylib"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4408229888,
    "size" : 339968,
    "uuid" : "8295b1a1-1136-3e32-aee9-c82b3feb4b21",
    "path" : "\/Library\/AutoPkg\/*\/Python.framework\/Versions\/3.7\/lib\/python3.7\/site-packages\/objc\/_objc.cpython-37m-darwin.so",
    "name" : "_objc.cpython-37m-darwin.so"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4404834304,
    "CFBundleShortVersionString" : "3.7.5, (c) 2001-2019 Python Software Foundation.",
    "CFBundleIdentifier" : "org.python.python",
    "size" : 1941504,
    "uuid" : "62582eaa-87f5-3553-8feb-c0a33b8660c8",
    "path" : "\/Library\/AutoPkg\/*\/Python.framework\/Versions\/3.7\/Python",
    "name" : "Python",
    "CFBundleVersion" : "3.7.5"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 4582346752,
    "size" : 425984,
    "uuid" : "9fc670fc-4d35-3582-a044-8f06bf2528af",
    "path" : "\/usr\/lib\/dyld",
    "name" : "dyld"
  },
  {
    "source" : "P",
    "arch" : "x86_64",
    "base" : 140703589392384,
    "size" : 49152,
    "uuid" : "ad356414-9c4c-3df7-a0e6-57ccac56de46",
    "path" : "\/usr\/lib\/system\/libsystem_pthread.dylib",
    "name" : "libsystem_pthread.dylib"
  }
],
  "sharedCache" : {
  "base" : 140703586193408,
  "size" : 15209971712,
  "uuid" : "1fe7a76a-83ec-36ba-b7e7-fe48df279b93"
},
  "vmSummary" : "ReadOnly portion of Libraries: Total=942.8M resident=0K(0%) swapped_out_or_unallocated=942.8M(100%)\nWritable regions: Total=1.6G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.6G(100%)\n\n                                VIRTUAL   REGION \nREGION TYPE                        SIZE    COUNT (non-coalesced) \n===========                     =======  ======= \nActivity Tracing                   256K        1 \nKernel Alloc Once                    8K        1 \nMALLOC                           264.1M       33 \nMALLOC guard page                   24K        5 \nMALLOC_MEDIUM (reserved)         960.0M        8         reserved VM address space (unallocated)\nMALLOC_NANO (reserved)           384.0M        1         reserved VM address space (unallocated)\nSTACK GUARD                          8K        2 \nStack                             16.5M        3 \nVM_ALLOCATE                        252K        4 \n__CTF                               759        1 \n__DATA                            23.5M      454 \n__DATA_CONST                      17.9M      253 \n__DATA_DIRTY                      1061K      138 \n__FONT_DATA                          4K        1 \n__LINKEDIT                       642.8M       56 \n__OBJC_RO                         81.4M        1 \n__OBJC_RW                         3104K        2 \n__TEXT                           300.1M      471 \n__UNICODE                          588K        1 \ndyld private memory               1024K        1 \nmapped file                       28.3M        2 \nshared memory                       60K        6 \n===========                     =======  ======= \nTOTAL                              2.7G     1445 \nTOTAL, minus reserved VM space     1.3G     1445 \n",
  "legacyInfo" : {
  "threadTriggered" : {
    "queue" : "com.apple.main-thread"
  }
},
  "trialInfo" : {
  "rollouts" : [
    {
      "rolloutId" : "607844aa04477260f58a8077",
      "factorPackIds" : {
        "SIRI_MORPHUN_ASSETS" : "60ef663fa2f864203caf575c"
      },
      "deploymentId" : 250000055
    },
    {
      "rolloutId" : "60356660bbe37970735c5624",
      "factorPackIds" : {

      },
      "deploymentId" : 250000016
    },
    {
      "rolloutId" : "60da5e84ab0ca017dace9abf",
      "factorPackIds" : {

      },
      "deploymentId" : 250000007
    },
    {
      "rolloutId" : "5ffde50ce2aacd000d47a95f",
      "factorPackIds" : {

      },
      "deploymentId" : 250000036
    },
    {
      "rolloutId" : "601d9415f79519000ccd4b69",
      "factorPackIds" : {
        "SIRI_TEXT_TO_SPEECH" : "614108fd74f55d4a733ca879"
      },
      "deploymentId" : 250000207
    },
    {
      "rolloutId" : "5fc94383418129005b4e9ae0",
      "factorPackIds" : {

      },
      "deploymentId" : 250000124
    },
    {
      "rolloutId" : "602ad4dac86151000cf27e46",
      "factorPackIds" : {

      },
      "deploymentId" : 250000189
    }
  ],
  "experiments" : [

  ]
}
}

Model: MacBookPro16,2, BootROM 1715.40.12.0.0 (iBridge: 19.16.10522.5.4,0), 4 processors, Quad-Core Intel Core i7, 2.3 GHz, 32 GB, SMC 
Graphics: Intel Iris Plus Graphics, Intel Iris Plus Graphics, Built-In
Display: Color LCD, 2560 x 1600 Retina, Main, MirrorOff, Online
Display: DELL U3219Q, 6016 x 3384, MirrorOff, Online
Memory Module: BANK 0/ChannelA-DIMM0, 16 GB, LPDDR4X, 3733 MHz, SK Hynix, H9HCNNNFAMALTR-NME
Memory Module: BANK 2/ChannelB-DIMM0, 16 GB, LPDDR4X, 3733 MHz, SK Hynix, H9HCNNNFAMALTR-NME
AirPort: spairport_wireless_card_type_wifi (0x14E4, 0x7BF), wl0: Jul 12 2021 18:02:56 version 9.30.464.0.32.5.76 FWID 01-c081cfed
Bluetooth: Version (null), 0 services, 0 devices, 0 incoming serial ports
Network Service: Wi-Fi, AirPort, en0
USB Device: USB31Bus
USB Device: YubiKey FIDO+CCID
USB Device: hub_device
USB Device: HD Pro Webcam C920
USB Device: 4-Port USB 2.0 Hub
USB Device: Blue Microphones
USB Device: composite_device
USB Device: USB31Bus
USB Device: T2Bus
USB Device: Touch Bar Backlight
USB Device: Touch Bar Display
USB Device: Apple Internal Keyboard / Trackpad
USB Device: Headset
USB Device: Ambient Light Sensor
USB Device: FaceTime HD Camera (Built-in)
USB Device: Apple T2 Controller
Thunderbolt Bus: MacBook Pro, Apple Inc., 85.0

Thunderbolt Bus: MacBook Pro, Apple Inc., 85.0

TypeError on Mosh recipe

Fair chance I'm missing something here. Running autopkg 2.0.2

$ autopkg run -v mosh.munki Processing mosh.munki... URLTextSearcher URLTextSearcher: Found matching text (match): https://mosh.org/mosh-1.3.2.pkg URLDownloader URLDownloader: Item at URL is unchanged. URLDownloader: Using existing /Users/john/Library/AutoPkg/Cache/local.munki.mosh/downloads/mosh-1.3.2.pkg MoshVersioner Traceback (most recent call last): File "/Library/AutoPkg/autopkglib/__init__.py", line 673, in process self.env = processor.process() File "/Library/AutoPkg/autopkglib/__init__.py", line 480, in process self.main() File "/Users/john/Library/AutoPkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Mosh/MoshVersioner.py", line 56, in main self.env["version"] = unfixed.lstrip("mosh-") TypeError: a bytes-like object is required, not 'str' File "/Library/AutoPkg/autopkglib/__init__.py", line 673, in process self.env = processor.process() a bytes-like object is required, not 'str'

If I print out unfixed I get b'mosh-1.3.2' which makes sense after line 54 in MoshVersioner. Since lstrip is looking for a string if I decode unfixed with unfixed = unfixed.decode("utf-8") before self.env["version"] = unfixed.lstrip("mosh-") everything works fine.

XcodeVersioner fails to load DVTFoundation

XcodeVersioner
{'Input': {'app_path': u'/Volumes/Data/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/Xcode.app',
           'version': u'11.3.1'}}
XcodeVersioner: Major version: 11
XcodeVersioner: Minor version: 3
2020-02-17 12:04:13.972 Python[90079:1556035] Error loading /Volumes/Data/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/DVTFoundation:  dlopen(/Volumes/Data/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/DVTFoundation, 265): Library not loaded: /System/Library/Frameworks/UserNotifications.framework/Versions/A/UserNotifications
  Referenced from: /Volumes/Data/AutoPkg/Cache/local.munki.Xcode/Xcode_unpack/Xcode.app/Contents/SharedFrameworks/DVTFoundation.framework/DVTFoundation
  Reason: image not found
Bundle could not be loaded
Failed.

From my quick research, this appears to be because the Xcode 11.3 version of DVTFoundation is attempting to load /System/Library/Frameworks/UserNotifications.framework which is not present on macOS 10.13 High Sierra.

sw_vers 
ProductName:	Mac OS X
ProductVersion:	10.13.6
BuildVersion:	17G11023

I'm mostly documenting this issue for awareness, not necessarily looking for a fix. I need to just upgrade my AutoPkg box.

More suggestions for improving VMwareDeploy

Some additional possible improvements to the VMwareDeploy Munki recipes, as per discussed in Slack.

• Set LICENSE_KEY default override to one that's not commented, or...
• ..alternatively, allow the override to specify a path to a Deploy.ini file, or give its contents as the override value.
• Improve uninstaller logic - the pkg cannot actually be removed using only remove_packages. It should be either not uninstallable or have its own uninstall_script.
• Break up packaging and Munki-related logic into two recipes for use by systems other than Munki.

Using FileAppender to append variables in .yaml file

I am trying to populate a .yaml file after Autopkg runs, including some variables from earlier processors in a .download recipe.
Text seems to be fine but variables are only passed along in cleartext.
Am I missing something or is it not possible to pass e.g. %NAME% / %version% variables directly to a .yaml file with this custom processor?
Tried both .recipe and .yaml recipes to pass along variables from with the same result.

XcodeVersionedName.munki FileMover: [Errno 66] Directory not empty

After once successful autopkg run XcodeVersionedName.munki, subsequent runs produce a [Errno 66] Directory not empty error. Maybe the PathDeleter process in Xcode.download.recipe should have moar paths?

IntelliJ Recipe and `urllib2` dep

Seems like the current release of y'all's IntelliJ recipe is wonky.

Processing IntelliJ.munki...
WARNING: /Users/Shared/autopkg/RecipeRepos/com.github.facebook.Recipes-for-AutoPkg/Intellij/IntellijURLProvider.py: No module named 'urllib2'
Failed.

Error with PackageInfoVersioner

To be clear, up until very recently I was still on autopkg 1.4.1, so all of my recipe testing was done pre-2.0. Prior to upgrading to the latest (2.1), I created a recipe for importing Pro Tools into munki. This processes not only the Pro Tools PKG installer but also 2 other PKGs in the DMG which are version specific to the included version of Pro Tools. As such the recipe grabs the PKG versions of these two other installers so they can be included with the Pro Tools pkginfo file. After upgrading to 2.1, I now get an error.

System info:
autopkg: 2.1
macOS: 10.14.6

Applicable lines in the recipe: https://github.com/autopkg/apizz-recipes/blob/master/ProTools/ProToolsImport.munki.recipe#L64-L139

I was reviewing my recipe list run and noted that the run failed to continue after running PackageInfoVersioner the second time it's called in the recipe. Given the error, my assumption is it has to do with the processor rather than autopkg, but I could be wrong given my lack of Python know-how. If it turns out to be an autopkg thing, I can move this issue there.

Interestingly, PackageInfoVersioner does not fail on the first Codecs pkg, but fails on the second HD driver package. Not sure what info would be helpful for isolating what's going on or what's different with this second package. I don't expect you to have the software, so if want a copy of the pkg let me know.

autopkg run ProToolsImport.or.munki -vv
Processing ProToolsImport.or.munki...
FileFinder
{'Input': {'pattern': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/*.dmg'}}
FileFinder: No value supplied for find_method, setting default value of: glob
FileFinder: Found file match: '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg' from globbed '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/*.dmg'
{'Output': {'found_filename': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg'}}
CodeSignatureVerifier
{'Input': {'expected_authority_names': ['Developer ID Installer: Avid '
                                        'Technology Inc (4UYUA773XD)',
                                        'Developer ID Certification Authority',
                                        'Apple Root CA'],
           'input_path': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg/Install*.pkg'}}
CodeSignatureVerifier: Mounted disk image /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg
CodeSignatureVerifier: Using path '/private/tmp/dmg.qgNL5D/Install Pro Tools 2019.12.0.pkg' matched from globbed '/private/tmp/dmg.qgNL5D/Install*.pkg'.
CodeSignatureVerifier: Verifying installer package signature...
CodeSignatureVerifier: Package "Install Pro Tools 2019.12.0.pkg":
CodeSignatureVerifier:    Status: signed by a certificate trusted by Mac OS X
CodeSignatureVerifier:    Certificate Chain:
CodeSignatureVerifier:     1. Developer ID Installer: Avid Technology Inc (4UYUA773XD)
CodeSignatureVerifier:        SHA1 fingerprint: 52 C7 74 17 B8 F7 11 F5 92 B1 B3 F3 9A C4 A5 54 E8 B1 86 36
CodeSignatureVerifier:        -----------------------------------------------------------------------------
CodeSignatureVerifier:     2. Developer ID Certification Authority
CodeSignatureVerifier:        SHA1 fingerprint: 3B 16 6C 3B 7D C4 B7 51 C9 FE 2A FA B9 13 56 41 E3 88 E1 86
CodeSignatureVerifier:        -----------------------------------------------------------------------------
CodeSignatureVerifier:     3. Apple Root CA
CodeSignatureVerifier:        SHA1 fingerprint: 61 1E 5B 66 2C 59 3A 08 FF 58 D1 4A E2 24 52 D1 98 DF 6C 60
CodeSignatureVerifier: 
CodeSignatureVerifier: Signature is valid
CodeSignatureVerifier: Authority name chain is valid
{'Output': {}}
com.github.homebysix.VersionSplitter/VersionSplitter
{'Input': {'index': -2,
           'split_on': '_',
           'version': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg'}}
VersionSplitter: Split version: 2019.12.0
{'Output': {'version': '2019.12.0'}}
Copier
{'Input': {'destination_path': '/Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/ProTools2019-2019.12.0.pkg',
           'overwrite': True,
           'source_path': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg/Install*.pkg'}}
Copier: Parsed dmg results: dmg_path: /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg, dmg: .dmg/, dmg_source_path: Install*.pkg
Copier: Mounted disk image /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg
Copier: Using path '/private/tmp/dmg.an3QU2/Install Pro Tools 2019.12.0.pkg' matched from globbed '/private/tmp/dmg.an3QU2/Install*.pkg'.
Copier: Copied /private/tmp/dmg.an3QU2/Install Pro Tools 2019.12.0.pkg to /Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/ProTools2019-2019.12.0.pkg
{'Output': {}}
FileFinder
{'Input': {'find_method': 'glob',
           'pattern': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg/Codec '
                      'Installers/*.pkg'}}
FileFinder: Mounted disk image /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg
FileFinder: Found file match: '/private/tmp/dmg.vbKpCd/Codec Installers/Install Avid Codecs LE.pkg' from globbed '/private/tmp/dmg.vbKpCd/Codec Installers/*.pkg'
FileFinder: DMG-relative file match: 'Codec Installers/Install Avid Codecs LE.pkg'
{'Output': {'dmg_found_filename': 'Codec Installers/Install Avid Codecs LE.pkg',
            'found_filename': '/private/tmp/dmg.vbKpCd/Codec '
                              'Installers/Install Avid Codecs LE.pkg'}}
FlatPkgUnpacker
{'Input': {'destination_path': '/Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/unpack',
           'flat_pkg_path': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg/Codec '
                            'Installers/Install Avid Codecs LE.pkg'}}
FlatPkgUnpacker: Mounted disk image /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg
FlatPkgUnpacker: Unpacked /private/tmp/dmg.KuH5PD/Codec Installers/Install Avid Codecs LE.pkg to /Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/unpack
{'Output': {}}
com.facebook.autopkg.shared/PackageInfoVersioner
{'Input': {'package_info_path': '/Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/unpack/AvidCodecsLE_component.pkg/PackageInfo'}}
PackageInfoVersioner: Found version b'2.7.3.39170'
{'Output': {'version': b'2.7.3.39170'}}
FileFinder
{'Input': {'find_method': 'glob',
           'pattern': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg/Driver '
                      'Installers/*.pkg'}}
FileFinder: Mounted disk image /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg
FileFinder: Found file match: '/private/tmp/dmg.4iSMDD/Driver Installers/Install Avid HD Driver.pkg' from globbed '/private/tmp/dmg.4iSMDD/Driver Installers/*.pkg'
FileFinder: DMG-relative file match: 'Driver Installers/Install Avid HD Driver.pkg'
{'Output': {'dmg_found_filename': 'Driver Installers/Install Avid HD '
                                  'Driver.pkg',
            'found_filename': '/private/tmp/dmg.4iSMDD/Driver '
                              'Installers/Install Avid HD Driver.pkg'}}
FlatPkgUnpacker
{'Input': {'destination_path': '/Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/unpack',
           'flat_pkg_path': '/Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg/Driver '
                            'Installers/Install Avid HD Driver.pkg'}}
FlatPkgUnpacker: Mounted disk image /Volumes/EXT_DATA/Autopkgr/Manual_Imports/ProTools/Pro_Tools_2019.12.0_Mac.dmg
FlatPkgUnpacker: Unpacked /private/tmp/dmg.hj8yld/Driver Installers/Install Avid HD Driver.pkg to /Volumes/EXT_DATA/Autopkgr/Cache/local.munki.or.ProToolsImport/unpack
{'Output': {}}
com.facebook.autopkg.shared/PackageInfoVersioner
Traceback (most recent call last):
  File "/usr/local/bin/autopkg", line 2887, in <module>
    sys.exit(main(sys.argv))
  File "/usr/local/bin/autopkg", line 2881, in main
    exit(subcommands[verb]["function"](argv))
  File "/usr/local/bin/autopkg", line 2398, in run_recipes
    autopackager.process(recipe)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 664, in process
    processor.inject(step.get("Arguments", {}))
  File "/Library/AutoPkg/autopkglib/__init__.py", line 465, in inject
    update_data(self.env, key, value)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 375, in update_data
    a_dict[key] = do_variable_substitution(value)
  File "/Library/AutoPkg/autopkglib/__init__.py", line 357, in do_variable_substitution
    item = RE_KEYREF.sub(getdata, item)
TypeError: sequence item 0: expected str instance, bytes found

Can't find PASSWORD_FILE

I like the new password_file thingy, but including a password in the override no longer works:

❯ autopkg run XcodeVersionedName.munki -vv
Processing XcodeVersionedName.munki...
Use of undefined key in variable substitution: u'RECIPE_CACHE_DIR'
AppleDataGatherer
{'Input': {'appID_key': u'891bd3417a7776362562d2197f89480a8547b108fd934911bcbea0110d07f757',
           'apple_id': ### u'REDACTED',
           'password_file': u'secret.txt'}}
[Errno 2] No such file or directory: u'secret.txt'
Failed.
Receipt written to /Users/macj/Library/AutoPkg/Cache/local.munki.XcodeVersionedName/receipts/XcodeVersionedName-receipt-20191002-130802.plist

The following recipes failed:
    XcodeVersionedName.munki
        Error in local.munki.XcodeVersionedName: Processor: AppleDataGatherer: Error: [Errno 2] No such file or directory: u'secret.txt'

Nothing downloaded, packaged or imported.

I think it'd be fine to deprecate the override password but FYI.

Another ditto error Xcode10.1

Error while extracting Xcode10.1 or Xcode10 with XcodeXIPUnpacker.

Error: ditto: cipo read error: bad file format.

Manual extract does work. Tried on multiple machines. Always the same.

Unused variables

Recipes-for-AutoPkg/Xcode/XcodeVersioner.py

Line 66 in ef5dee3

framework_bundle = objc.loadBundle( # NOQA

Variables declared but not used, or reassigned before they are used, can be the result of programmer negligence, which often means that there is a bug in the program.