mirror of
https://github.com/bitcoin/bitcoin.git
synced 2025-02-13 11:25:02 -05:00
build: use macOS 10.14 SDK
Co-Authored-By: Carl Dong <accounts@carldong.me>
This commit is contained in:
fanquake
parent
ca5055a5aa
commit
7e2104433c
6 changed files with 133 additions and 106 deletions
|
|
@ -8,7 +8,7 @@ export LC_ALL=C.UTF-8
|
||||||
|
|
||||||
export HOST=x86_64-apple-darwin16
|
export HOST=x86_64-apple-darwin16
|
||||||
export PACKAGES="cmake imagemagick libcap-dev librsvg2-bin libz-dev libbz2-dev libtiff-tools python3-dev python3-setuptools"
|
export PACKAGES="cmake imagemagick libcap-dev librsvg2-bin libz-dev libbz2-dev libtiff-tools python3-dev python3-setuptools"
|
||||||
export OSX_SDK=10.11
|
export OSX_SDK=10.14
|
||||||
export RUN_UNIT_TESTS=false
|
export RUN_UNIT_TESTS=false
|
||||||
export RUN_FUNCTIONAL_TESTS=false
|
export RUN_FUNCTIONAL_TESTS=false
|
||||||
export GOAL="deploy"
|
export GOAL="deploy"
|
||||||
|
|
|
||||||
|
|
@ -209,7 +209,7 @@ def main():
|
||||||
args.macos = 'm' in args.os
|
args.macos = 'm' in args.os
|
||||||
|
|
||||||
# Disable for MacOS if no SDK found
|
# Disable for MacOS if no SDK found
|
||||||
if args.macos and not os.path.isfile('gitian-builder/inputs/MacOSX10.11.sdk.tar.gz'):
|
if args.macos and not os.path.isfile('gitian-builder/inputs/MacOSX10.14.sdk.tar.gz'):
|
||||||
print('Cannot build for MacOS, SDK does not exist. Will build for other OSes')
|
print('Cannot build for MacOS, SDK does not exist. Will build for other OSes')
|
||||||
args.macos = False
|
args.macos = False
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ remotes:
|
||||||
- "url": "https://github.com/bitcoin/bitcoin.git"
|
- "url": "https://github.com/bitcoin/bitcoin.git"
|
||||||
"dir": "bitcoin"
|
"dir": "bitcoin"
|
||||||
files:
|
files:
|
||||||
- "MacOSX10.11.sdk.tar.gz"
|
- "MacOSX10.14.sdk.tar.gz"
|
||||||
script: |
|
script: |
|
||||||
set -e -o pipefail
|
set -e -o pipefail
|
||||||
|
|
||||||
|
|
@ -90,7 +90,7 @@ script: |
|
||||||
BASEPREFIX="${PWD}/depends"
|
BASEPREFIX="${PWD}/depends"
|
||||||
|
|
||||||
mkdir -p ${BASEPREFIX}/SDKs
|
mkdir -p ${BASEPREFIX}/SDKs
|
||||||
tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.11.sdk.tar.gz
|
tar -C ${BASEPREFIX}/SDKs -xf ${BUILD_DIR}/MacOSX10.14.sdk.tar.gz
|
||||||
|
|
||||||
# Build dependencies for each host
|
# Build dependencies for each host
|
||||||
for i in $HOSTS; do
|
for i in $HOSTS; do
|
||||||
|
|
|
||||||
|
|
@ -1,15 +1,135 @@
|
||||||
### MacDeploy ###
|
# MacOS Deployment
|
||||||
|
|
||||||
For Snow Leopard (which uses [Python 2.6](http://www.python.org/download/releases/2.6/)), you will need the param_parser package:
|
The `macdeployqtplus` script should not be run manually. Instead, after building as usual:
|
||||||
|
|
||||||
sudo easy_install argparse
|
```bash
|
||||||
|
make deploy
|
||||||
|
```
|
||||||
|
|
||||||
This script should not be run manually, instead, after building as usual:
|
During the deployment process, the disk image window will pop up briefly
|
||||||
|
when the fancy settings are applied. This is normal, please do not interfere,
|
||||||
|
the process will unmount the DMG and cleanup before finishing.
|
||||||
|
|
||||||
make deploy
|
When complete, it will have produced `Bitcoin-Qt.dmg`.
|
||||||
|
|
||||||
During the process, the disk image window will pop up briefly where the fancy
|
## SDK Extraction
|
||||||
settings are applied. This is normal, please do not interfere.
|
|
||||||
|
|
||||||
When finished, it will produce `Bitcoin-Qt.dmg`.
|
`Xcode.app` is packaged in a `.xip` archive.
|
||||||
|
This makes the SDK less-trivial to extract on non-macOS machines.
|
||||||
|
One approach (tested on Debian Buster) is outlined below:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
|
||||||
|
apt install clang cpio git liblzma-dev libxml2-dev libssl-dev make
|
||||||
|
|
||||||
|
git clone https://github.com/tpoechtrager/xar
|
||||||
|
pushd xar/xar
|
||||||
|
./configure
|
||||||
|
make
|
||||||
|
make install
|
||||||
|
popd
|
||||||
|
|
||||||
|
git clone https://github.com/NiklasRosenstein/pbzx
|
||||||
|
pushd pbzx
|
||||||
|
clang -llzma -lxar pbzx.c -o pbzx -Wl,-rpath=/usr/local/lib
|
||||||
|
popd
|
||||||
|
|
||||||
|
xar -xf Xcode_10.2.1.xip -C .
|
||||||
|
|
||||||
|
./pbzx/pbzx -n Content | cpio -i
|
||||||
|
|
||||||
|
find Xcode.app -type d -name MacOSX.sdk -execdir sh -c 'tar -c MacOSX.sdk/ | gzip -9n > /MacOSX10.14.sdk.tar.gz' \;
|
||||||
|
```
|
||||||
|
|
||||||
|
on macOS the process is more straightforward:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
xip -x Xcode_10.2.1.xip
|
||||||
|
tar -C Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.14.sdk.tar.gz MacOSX.sdk
|
||||||
|
```
|
||||||
|
|
||||||
|
Our previously used macOS SDK (`MacOSX10.11.sdk`) can be extracted from
|
||||||
|
[Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
|
||||||
|
The script [`extract-osx-sdk.sh`](./extract-osx-sdk.sh) automates this. First
|
||||||
|
ensure the DMG file is in the current directory, and then run the script. You
|
||||||
|
may wish to delete the `intermediate 5.hfs` file and `MacOSX10.11.sdk` (the
|
||||||
|
directory) when you've confirmed the extraction succeeded.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
apt-get install p7zip-full sleuthkit
|
||||||
|
contrib/macdeploy/extract-osx-sdk.sh
|
||||||
|
rm -rf 5.hfs MacOSX10.11.sdk
|
||||||
|
```
|
||||||
|
|
||||||
|
## Deterministic macOS DMG Notes
|
||||||
|
Working macOS DMGs are created in Linux by combining a recent `clang`, the Apple
|
||||||
|
`binutils` (`ld`, `ar`, etc) and DMG authoring tools.
|
||||||
|
|
||||||
|
Apple uses `clang` extensively for development and has upstreamed the necessary
|
||||||
|
functionality so that a vanilla clang can take advantage. It supports the use of `-F`,
|
||||||
|
`-target`, `-mmacosx-version-min`, and `--sysroot`, which are all necessary when
|
||||||
|
building for macOS.
|
||||||
|
|
||||||
|
Apple's version of `binutils` (called `cctools`) contains lots of functionality missing in the
|
||||||
|
FSF's `binutils`. In addition to extra linker options for frameworks and sysroots, several
|
||||||
|
other tools are needed as well such as `install_name_tool`, `lipo`, and `nmedit`. These
|
||||||
|
do not build under Linux, so they have been patched to do so. The work here was used as
|
||||||
|
a starting point: [mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
|
||||||
|
|
||||||
|
In order to build a working toolchain, the following source packages are needed from
|
||||||
|
Apple: `cctools`, `dyld`, and `ld64`.
|
||||||
|
|
||||||
|
These tools inject timestamps by default, which produce non-deterministic binaries. The
|
||||||
|
`ZERO_AR_DATE` environment variable is used to disable that.
|
||||||
|
|
||||||
|
This version of `cctools` has been patched to use the current version of `clang`'s headers
|
||||||
|
and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done in `toolchain4`.
|
||||||
|
|
||||||
|
To complicate things further, all builds must target an Apple SDK. These SDKs are free to
|
||||||
|
download, but not redistributable. To obtain it, register for an Apple Developer Account,
|
||||||
|
then download [Xcode 10.2.1](https://download.developer.apple.com/Developer_Tools/Xcode_10.2.1/Xcode_10.2.1.xip).
|
||||||
|
|
||||||
|
This file is many gigabytes in size, but most (but not all) of what we need is
|
||||||
|
contained only in a single directory:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk
|
||||||
|
```
|
||||||
|
|
||||||
|
See the SDK Extraction notes above for how to obtain it.
|
||||||
|
|
||||||
|
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries which are
|
||||||
|
created using these tools. The build process has been designed to avoid including the
|
||||||
|
SDK's files in Gitian's outputs. All interim tarballs are fully deterministic and may be freely
|
||||||
|
redistributed.
|
||||||
|
|
||||||
|
`genisoimage` is used to create the initial DMG. It is not deterministic as-is, so it has been
|
||||||
|
patched. A system `genisoimage` will work fine, but it will not be deterministic because
|
||||||
|
the file-order will change between invocations. The patch can be seen here: [cdrkit-deterministic.patch](https://github.com/bitcoin/bitcoin/blob/master/depends/patches/native_cdrkit/cdrkit-deterministic.patch).
|
||||||
|
No effort was made to fix this cleanly, so it likely leaks memory badly, however it's only used for
|
||||||
|
a single invocation, so that's no real concern.
|
||||||
|
|
||||||
|
`genisoimage` cannot compress DMGs, so afterwards, the DMG tool from the
|
||||||
|
`libdmg-hfsplus` project is used to compress it. There are several bugs in this tool and its
|
||||||
|
maintainer has seemingly abandoned the project.
|
||||||
|
|
||||||
|
The DMG tool has the ability to create DMGs from scratch as well, but this functionality is
|
||||||
|
broken. Only the compression feature is currently used. Ideally, the creation could be fixed
|
||||||
|
and `genisoimage` would no longer be necessary.
|
||||||
|
|
||||||
|
Background images and other features can be added to DMG files by inserting a
|
||||||
|
`.DS_Store` before creation. This is generated by the script `contrib/macdeploy/custom_dsstore.py`.
|
||||||
|
|
||||||
|
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in
|
||||||
|
order to satisfy the new Gatekeeper requirements. Because this private key cannot be
|
||||||
|
shared, we'll have to be a bit creative in order for the build process to remain somewhat
|
||||||
|
deterministic. Here's how it works:
|
||||||
|
|
||||||
|
- Builders use Gitian to create an unsigned release. This outputs an unsigned DMG which
|
||||||
|
users may choose to bless and run. It also outputs an unsigned app structure in the form
|
||||||
|
of a tarball, which also contains all of the tools that have been previously (deterministically)
|
||||||
|
built in order to create a final DMG.
|
||||||
|
- The Apple keyholder uses this unsigned app to create a detached signature, using the
|
||||||
|
script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs).
|
||||||
|
- Builders feed the unsigned app + detached signature back into Gitian. It uses the
|
||||||
|
pre-built tools to recombine the pieces into a deterministic DMG.
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
OSX_MIN_VERSION=10.12
|
OSX_MIN_VERSION=10.12
|
||||||
OSX_SDK_VERSION=10.11
|
OSX_SDK_VERSION=10.14
|
||||||
OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION).sdk
|
OSX_SDK=$(SDK_PATH)/MacOSX$(OSX_SDK_VERSION).sdk
|
||||||
darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK)
|
darwin_CC=clang -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK)
|
||||||
darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++
|
darwin_CXX=clang++ -target $(host) -mmacosx-version-min=$(OSX_MIN_VERSION) --sysroot $(OSX_SDK) -stdlib=libc++
|
||||||
|
|
|
||||||
|
|
@ -113,96 +113,3 @@ tail -f $HOME/Library/Application\ Support/Bitcoin/debug.log
|
||||||
* Tested on OS X 10.12 Sierra through macOS 10.15 Catalina on 64-bit Intel
|
* Tested on OS X 10.12 Sierra through macOS 10.15 Catalina on 64-bit Intel
|
||||||
processors only.
|
processors only.
|
||||||
* Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714).
|
* Building with downloaded Qt binaries is not officially supported. See the notes in [#7714](https://github.com/bitcoin/bitcoin/issues/7714).
|
||||||
|
|
||||||
## Deterministic macOS DMG Notes
|
|
||||||
Working macOS DMGs are created in Linux by combining a recent `clang`, the Apple
|
|
||||||
`binutils` (`ld`, `ar`, etc) and DMG authoring tools.
|
|
||||||
|
|
||||||
Apple uses `clang` extensively for development and has upstreamed the necessary
|
|
||||||
functionality so that a vanilla clang can take advantage. It supports the use of `-F`,
|
|
||||||
`-target`, `-mmacosx-version-min`, and `--sysroot`, which are all necessary when
|
|
||||||
building for macOS.
|
|
||||||
|
|
||||||
Apple's version of `binutils` (called `cctools`) contains lots of functionality missing in the
|
|
||||||
FSF's `binutils`. In addition to extra linker options for frameworks and sysroots, several
|
|
||||||
other tools are needed as well such as `install_name_tool`, `lipo`, and `nmedit`. These
|
|
||||||
do not build under Linux, so they have been patched to do so. The work here was used as
|
|
||||||
a starting point: [mingwandroid/toolchain4](https://github.com/mingwandroid/toolchain4).
|
|
||||||
|
|
||||||
In order to build a working toolchain, the following source packages are needed from
|
|
||||||
Apple: `cctools`, `dyld`, and `ld64`.
|
|
||||||
|
|
||||||
These tools inject timestamps by default, which produce non-deterministic binaries. The
|
|
||||||
`ZERO_AR_DATE` environment variable is used to disable that.
|
|
||||||
|
|
||||||
This version of `cctools` has been patched to use the current version of `clang`'s headers
|
|
||||||
and its `libLTO.so` rather than those from `llvmgcc`, as it was originally done in `toolchain4`.
|
|
||||||
|
|
||||||
To complicate things further, all builds must target an Apple SDK. These SDKs are free to
|
|
||||||
download, but not redistributable. To obtain it, register for an Apple Developer Account,
|
|
||||||
then download the [Xcode 7.3.1 dmg](https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/Xcode_7.3.1/Xcode_7.3.1.dmg).
|
|
||||||
|
|
||||||
This file is several gigabytes in size, but only a single directory inside is needed:
|
|
||||||
```
|
|
||||||
Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
Unfortunately, the usual Linux tools (7zip, hpmount, loopback mount) are incapable of
|
|
||||||
opening this file. To create a tarball suitable for Gitian input, there are two options:
|
|
||||||
|
|
||||||
Using macOS, you can mount the DMG, and then create it with:
|
|
||||||
```shell
|
|
||||||
hdiutil attach Xcode_7.3.1.dmg
|
|
||||||
tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.11.sdk.tar.gz MacOSX10.11.sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
Alternatively, you can use 7zip and SleuthKit to extract the files one by one. The script
|
|
||||||
[`extract-osx-sdk.sh`](./../contrib/macdeploy/extract-osx-sdk.sh) automates this. First
|
|
||||||
ensure the DMG file is in the current directory, and then run the script. You may wish to
|
|
||||||
delete the `intermediate 5.hfs` file and `MacOSX10.11.sdk` (the directory) when you've
|
|
||||||
confirmed the extraction succeeded.
|
|
||||||
|
|
||||||
```shell
|
|
||||||
apt-get install p7zip-full sleuthkit
|
|
||||||
contrib/macdeploy/extract-osx-sdk.sh
|
|
||||||
rm -rf 5.hfs MacOSX10.11.sdk
|
|
||||||
```
|
|
||||||
|
|
||||||
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries which are
|
|
||||||
created using these tools. The build process has been designed to avoid including the
|
|
||||||
SDK's files in Gitian's outputs. All interim tarballs are fully deterministic and may be freely
|
|
||||||
redistributed.
|
|
||||||
|
|
||||||
`genisoimage` is used to create the initial DMG. It is not deterministic as-is, so it has been
|
|
||||||
patched. A system `genisoimage` will work fine, but it will not be deterministic because
|
|
||||||
the file-order will change between invocations. The patch can be seen here: [theuni/osx-cross-depends](https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff).
|
|
||||||
No effort was made to fix this cleanly, so it likely leaks memory badly. But it's only used for
|
|
||||||
a single invocation, so that's no real concern.
|
|
||||||
|
|
||||||
`genisoimage` cannot compress DMGs, so afterwards, the DMG tool from the
|
|
||||||
`libdmg-hfsplus` project is used to compress it. There are several bugs in this tool and its
|
|
||||||
maintainer has seemingly abandoned the project. It has been forked and is available
|
|
||||||
(with fixes) here: [theuni/libdmg-hfsplus](https://github.com/theuni/libdmg-hfsplus).
|
|
||||||
|
|
||||||
The DMG tool has the ability to create DMGs from scratch as well, but this functionality is
|
|
||||||
broken. Only the compression feature is currently used. Ideally, the creation could be fixed
|
|
||||||
and `genisoimage` would no longer be necessary.
|
|
||||||
|
|
||||||
Background images and other features can be added to DMG files by inserting a
|
|
||||||
`.DS_Store` before creation. This is generated by the script
|
|
||||||
`contrib/macdeploy/custom_dsstore.py`.
|
|
||||||
|
|
||||||
As of OS X 10.9 Mavericks, using an Apple-blessed key to sign binaries is a requirement in
|
|
||||||
order to satisfy the new Gatekeeper requirements. Because this private key cannot be
|
|
||||||
shared, we'll have to be a bit creative in order for the build process to remain somewhat
|
|
||||||
deterministic. Here's how it works:
|
|
||||||
|
|
||||||
- Builders use Gitian to create an unsigned release. This outputs an unsigned DMG which
|
|
||||||
users may choose to bless and run. It also outputs an unsigned app structure in the form
|
|
||||||
of a tarball, which also contains all of the tools that have been previously (deterministically)
|
|
||||||
built in order to create a final DMG.
|
|
||||||
- The Apple keyholder uses this unsigned app to create a detached signature, using the
|
|
||||||
script that is also included there. Detached signatures are available from this [repository](https://github.com/bitcoin-core/bitcoin-detached-sigs).
|
|
||||||
- Builders feed the unsigned app + detached signature back into Gitian. It uses the
|
|
||||||
pre-built tools to recombine the pieces into a deterministic DMG.
|
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Reference in a new issue