And, I always prefer to actually pick my glibc independently from the host os and ship it with the binary (generally in an OCI image). This way you can patch/update the host without breaking the service.
But what about any tools compiled from source and used during the build? Those can also suffer from these issues.
You can have every developer cross-compile from a different os/cpu platform
Mysterious?
This is by the way why many binary Python packages use https://github.com/pypa/manylinux for builds: if you build on an old glibc, your library will still (generally) work with newer versions.
zig cc -target x86_64-linux-gnu.2.17 file.c
A heterogenous build cluster with non-hermetic builds and shared caching. The fact that this is only a glibc symbol versioning problem and not something far more severe is, well, a blessing.
At the bare fucking minimum, I would expect the CI builds to have a homogenous configuration and their own cache, not writable from outside CI. If you’re lazy, just wipe the cache every time you upgrade the CI cluster. Maybe I’ve just been living too long in environments where we care about artifact provenance and I’m blind to the whims of people who don’t care about provenance.
I want to feel sympathetic, because I know Bazel is a pain in the ass to learn, but it sounds like the author was juggling knives in a windstorm and caught a few pointy ends.
If you can’t create your own sysroot image, you can simply download Chromium’s prebuilt one and configure your C++ compile rules correctly. Problem solved.
We also have an dockerfile for clang/LLVM in that repo so the whole thing is hermetic. It’s a bit of shame Bazel doesn’t come with stronger options/defaults here, because I feel like I want to reproduce this same toolchain on every C++ project with Bazel
The first project I was able to change their workflow to build inside a 20.04 container. The other project uses tauri and it requires some very recent libraries so I don't know if an older container will work.
Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations?
This approach does _not_ work because you end up with the `node` that runs GitHub Actions not being able to run, certainly this will happen if you end using a sufficiently old container.
> Do you have any documentation or generic recommendations for solving these issues caused by blindly using GitHub Actions for all compilations?
Install these pkgs in an `ubuntu-latest` image:
- debootstrap debian-archive-keyring
- software-properties-common
- schroot fakeroot fakechroot
- name: 'Cache sysroot'
# This comes after checking out the sources because
# actions/checkout@v4 cleans $PWD!
id: cache-sysroot
uses: actions/cache@v3
with:
path: ${{ github.workspace }}/sysroot-DEBIAN_RELEASE
key: sysroot-DEBIAN_RELEASE-${{ runner.os }}-${{ runner.arch }}-v1
- name: 'Setup cross-compilation sysroot'
if: steps.cache-sysroot.oututs.cache-hit != 'true'
run: |
set -vx
SYSROOT_PATH="${{ github.workspace }}/sysroot-DEBIAN_RELEASE"
echo "SYSROOT_PATH=$SYSROOT_PATH" >> $GITHUB_ENV
if [ ! -d sysroot-DEBIAN_RELEASE ]; then
sudo debootstrap --arch=$(dpkg --print-architecture) DEBIAN_RELEASE sysroot-DEBIAN_RELEASE http://archive.ubuntu.com/ubuntu
fi
sudo chroot sysroot-DEBIAN_RELEASE apt-get update
sudo chroot sysroot-DEBIAN_RELEASE apt-get install -y build-essential git wget curl sudo unzip zip autoconf libfreetype6-dev libcups2-dev libx11-dev libxext-dev libxrender-dev libxrandr-dev libxtst-dev libxt-dev libasound2-dev libffi-dev file binutils libfontconfig-dev
sudo chroot sysroot-DEBIAN_RELEASE apt-get install -y software-properties-common
sudo chroot sysroot-DEBIAN_RELEASE sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo chown -R $USER:$USER sysroot-DEBIAN_RELEASE
- configure your project's build to use that sysroot.
If your project does not support sysroots, make it do so. In general compilers will support sysroots, so it's just a matter of making your build configuration facility support sysroots.
Why would you cache developer builds on CI?
shykes•4mo ago
jeffbee•4mo ago
paulddraper•4mo ago
But that’s only a minority of what it does.
klodolph•4mo ago
Containers tend to be coarse-grained. For example, maybe you are writing a program, so you put the entire build environment in a container. If you have lots of programs with different dependencies, do you need lots of containers? Do you need to rebuild lots of containers when dependencies change? Bazel is much more fine-grained, and in Bazel, each individual build action gets its own build environment.
By default, that build environment includes the host (build system) glibc, compiler, system headers, system libraries, etc. If you want repeatable builds, you turn that off, and give Bazel the exact glibc, compiler, libraries, etc. to use for building your program.
You get the isolation of containers, but much faster builds and better caching.
josephg•4mo ago
Ambient, implicit dependencies are the devil’s playthings.
shykes•4mo ago
Adopting containers would solve this, but it seems to be a major blind spot for the Bazel community. My theory is that for them to adopt container technology, they will have to reinvent it themselves, hence my tongue-in-cheek comment.