contrib: macdeploy: Correctly generate macOS SDK

Previously, we did not include the macOS SDK libc++ headers in our SDK
creation process and instead used whichever libc++ headers shipped with
the clang package we downloaded in depends.

This change adds a script (which works on both GNU/Linux and macOS) to
correctly generate the macOS SDK including the libc++ headers. This can
be thought of as a simplified rewrite of tpoechtrager's script:

d3392f4eae/tools/gen_sdk_package.sh

The location within the SDK where we place the libc++ headers is chosen
such that clang's search path detection logic for sysroots would pick up
the headers properly.

We also document this change.

contrib/macdeploy/README.md

@@ -14,13 +14,17 @@ When complete, it will have produced `Bitcoin-Qt.dmg`.
 
 ## SDK Extraction
 
-Our current macOS SDK (`macOSX10.14.sdk`) can be extracted from
+### Step 1: Obtaining `Xcode.app`
+
+Our current macOS SDK
+(`Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz`) can be
+extracted from
 [Xcode_10.2.1.xip](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
 An Apple ID is needed to download this.
 
-`Xcode.app` is packaged in a `.xip` archive.
+After Xcode version 7.x, Apple started shipping the `Xcode.app` in a `.xip`
-This makes the SDK less-trivial to extract on non-macOS machines.
+archive. This makes the SDK less-trivial to extract on non-macOS machines. One
-One approach (tested on Debian Buster) is outlined below:
+approach (tested on Debian Buster) is outlined below:
 
 ```bash
 
@@ -41,17 +45,28 @@ popd
 xar -xf Xcode_10.2.1.xip -C .
 
 ./pbzx/pbzx -n Content | cpio -i
-
-find Xcode.app -type d -name MacOSX.sdk -exec sh -c 'tar --transform="s/MacOSX.sdk/MacOSX10.14.sdk/" -c -C$(dirname {}) MacOSX.sdk/ | gzip -9n > MacOSX10.14.sdk.tar.gz' \;
 ```
 
-on macOS the process is more straightforward:
+On macOS the process is more straightforward:
 
 ```bash
 xip -x Xcode_10.2.1.xip
-tar -s "/MacOSX.sdk/MacOSX10.14.sdk/" -C Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.14.sdk.tar.gz MacOSX.sdk
 ```
 
+### Step 2: Generating `Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz` from `Xcode.app`
+
+To generate `Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz`, run
+the script [`gen-sdk`](./gen-sdk) with the path to `Xcode.app` (extracted in the
+previous stage) as the first argument.
+
+```bash
+# Generate a Xcode-10.2.1-10E1001-extracted-SDK-with-libcxx-headers.tar.gz from
+# the supplied Xcode.app
+./contrib/macdeploy/gen-sdk '/path/to/Xcode.app'
+```
+
+### Historial macOS SDK Extraction Notes
+
 Our previously used macOS SDK (`MacOSX10.11.sdk`) can be extracted from
 [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
 The script [`extract-osx-sdk.sh`](./extract-osx-sdk.sh) automates this. First
@@ -91,13 +106,13 @@ and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done
 
 To complicate things further, all builds must target an Apple SDK. These SDKs are free to
 download, but not redistributable. To obtain it, register for an Apple Developer Account,
-then download [Xcode 10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
+then download [Xcode_10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
 
 This file is many gigabytes in size, but most (but not all) of what we need is
 contained only in a single directory:
 
 ```bash
-Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
+Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk
 ```
 
 See the SDK Extraction notes above for how to obtain it.

contrib/macdeploy/gen-sdk

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+import argparse
+import plistlib
+import pathlib
+import sys
+import tarfile
+import gzip
+import os
+import contextlib
+
+@contextlib.contextmanager
+def cd(path):
+    """Context manager that restores PWD even if an exception was raised."""
+    old_pwd = os.getcwd()
+    os.chdir(str(path))
+    try:
+        yield
+    finally:
+        os.chdir(old_pwd)
+
+def run():
+    parser = argparse.ArgumentParser(
+        description=__doc__, formatter_class=argparse.RawTextHelpFormatter)
+
+    parser.add_argument('xcode_app', metavar='XCODEAPP', nargs=1)
+    parser.add_argument("-o", metavar='OUTSDKTGZ', nargs=1, dest='out_sdktgz', required=False)
+
+    args = parser.parse_args()
+
+    xcode_app = pathlib.Path(args.xcode_app[0]).resolve()
+    assert xcode_app.is_dir(), "The supplied Xcode.app path '{}' either does not exist or is not a directory".format(xcode_app)
+
+    xcode_app_plist = xcode_app.joinpath("Contents/version.plist")
+    with xcode_app_plist.open('rb') as fp:
+        pl = plistlib.load(fp)
+        xcode_version = pl['CFBundleShortVersionString']
+        xcode_build_id = pl['ProductBuildVersion']
+    print("Found Xcode (version: {xcode_version}, build id: {xcode_build_id})".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id))
+
+    sdk_dir = xcode_app.joinpath("Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk")
+    sdk_plist = sdk_dir.joinpath("System/Library/CoreServices/SystemVersion.plist")
+    with sdk_plist.open('rb') as fp:
+        pl = plistlib.load(fp)
+        sdk_version = pl['ProductVersion']
+        sdk_build_id = pl['ProductBuildVersion']
+    print("Found MacOSX SDK (version: {sdk_version}, build id: {sdk_build_id})".format(sdk_version=sdk_version, sdk_build_id=sdk_build_id))
+
+    out_name = "Xcode-{xcode_version}-{xcode_build_id}-extracted-SDK-with-libcxx-headers".format(xcode_version=xcode_version, xcode_build_id=xcode_build_id)
+
+    xcode_libcxx_dir = xcode_app.joinpath("Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include/c++/v1")
+    assert xcode_libcxx_dir.is_dir()
+
+    if args.out_sdktgz:
+        out_sdktgz_path = pathlib.Path(args.out_sdktgz_path)
+    else:
+        # Construct our own out_sdktgz if not specified on the command line
+        out_sdktgz_path = pathlib.Path("./{}.tar.gz".format(out_name))
+
+    def tarfp_add_with_base_change(tarfp, dir_to_add, alt_base_dir):
+        """Add all files in dir_to_add to tarfp, but prepent MEMBERPREFIX to the files'
+        names
+
+        e.g. if the only file under /root/bazdir is /root/bazdir/qux, invoking:
+
+            tarfp_add_with_base_change(tarfp, "foo/bar", "/root/bazdir")
+
+        would result in the following members being added to tarfp:
+
+            foo/bar/             -> corresponding to /root/bazdir
+            foo/bar/qux          -> corresponding to /root/bazdir/qux
+
+        """
+        def change_tarinfo_base(tarinfo):
+            if tarinfo.name and tarinfo.name.startswith("./"):
+                tarinfo.name = str(pathlib.Path(alt_base_dir, tarinfo.name))
+            if tarinfo.linkname and tarinfo.linkname.startswith("./"):
+                tarinfo.linkname = str(pathlib.Path(alt_base_dir, tarinfo.linkname))
+            return tarinfo
+        with cd(dir_to_add):
+            tarfp.add(".", recursive=True, filter=change_tarinfo_base)
+
+    print("Creating output .tar.gz file...")
+    with out_sdktgz_path.open("wb") as fp:
+        with gzip.GzipFile(fileobj=fp, compresslevel=9, mtime=0) as gzf:
+            with tarfile.open(mode="w", fileobj=gzf) as tarfp:
+                print("Adding MacOSX SDK {} files...".format(sdk_version))
+                tarfp_add_with_base_change(tarfp, sdk_dir, out_name)
+                print("Adding libc++ headers...")
+                tarfp_add_with_base_change(tarfp, xcode_libcxx_dir, "{}/usr/include/c++/v1".format(out_name))
+    print("Done! Find the resulting gzipped tarball at:")
+    print(out_sdktgz_path.resolve())
+
+if __name__ == '__main__':
+    run()