To solve for this I made this extension which disables the Github merge button when checks are failing and/or a user defined label is on the PR.
There's a content script that uses a MutationObserver to know when to disable/enable. That content script also listens to chrome.storage.sync.onChanged to know the values set in the configuration pop-over.