Ability to "break glass" and bypass Github branch protection for emergency fixes?

Question

For our project we're using Github's branch protection to enforce at least 1 reviewer for all changes, no exceptions for administrators.

The day will come however, when we need to rush an emergency fix. This will require expedited merging of a branch, which can only happen if someone else is around to review. But what happens if no one is around?

We would need to bypass the code review requirement.

We could disable the code review requirement temporarily while merging, but that is not desirable because there's no insight into when that was done, and it's a solution that only works for admins.

I'd like to have an auditable way to do this. A common term for this is "break glass" ie. you're breaking the glass and doing an emergency merge and deploy, because it's an emergency and no one is around to review your code.

Has anyone worked out a way to achieve this on a Github protected branch?

user_212774 · Answer

Where I work we manage our branch protection using a program called Github Organization Manager, which basically uses a file inside of a single git repository to manage the settings for our repositories. This allows settings to be exposed without giving admin access (and the organizer tool doesn't let users do things like make repositories public or delete them). There are other similar tools.

Outside of something like that, or simply giving users admin access to the repository, there isn't a way to do it built into Github. If it's something you're really worried about it wouldn't be too difficult to use the Github API to build an "break emergency glass" dashboard or slackbot that could remove the branch protection or even merge the pull request directly.

user_1110798 · Answer

Whether you do it by hand or with a tool/API, I'd be careful with temporarily modifying branch protection rules because it's easy to unblock ALL open pull requests during that window (i.e. you have one branch protection rule for main/master, which applies to most or all PRs). Depends on your situation obviously, but I've written tools to do that in the past it could really cause some headaches when you consider things like the auto-merge features and the fact that people might click merge as soon as it's possible!

Best would be to handle it on a per-PR basis. Like if you set up a bot to respond to a PR comment and leave an approval review? That would take some doing but you could probably use GitHub Actions (might try something like this for a head start on reacting to comments).

Some people also use PullApprove for this, which gives a lot more flexibility around code review rules. One way is to use labels (here's an example in practice on github.com/angular/angular). Looks a little clunky in this case but you can say here's when PullApprove is not enabled and it should actually have a passing status:

# .pullapprove.yml
version: 3

pullapprove_conditions:
- condition: "'emergency' not in labels"
  unmet_status: success
  explanation: "Review disabled with the emergency label"

groups:
  ...

Label changes show up directly in the PR timeline (including who changed them) so the audit trail would be pretty clear:

VonC_6309 · Answer

We could disable the code review requirement temporarily while merging, but that is not desirable because there's no insight into when that was done, and it's a solution that only works for admins.

Actually not just for admins.

Since Aug. 2022, you now have:

Bypass branch protections with a new permission

You can now create a custom role to bypass branch protections without having to grant the Admin role.

Previously, to bypass branch protections you had to be an Admin which provides additional permissions that may not be needed.

For tighter control of Admin permissions, you can now craft a custom role that has the Bypass branch protections permission, allowing just the right amount of access.

To enforce branch protections for all Admins and roles with the "Bypass branch protections" permission, enable Do not allow bypassing the above settings in your branch protection rules.

This permission differs from the Push commits to protected branches permission, which allows pushing to a protected branch, but branch protection rules will still apply and could result in a push being denied.

For more information, visit Managing custom repository roles for an organization in the GitHub documentation.

Ability to "break glass" and bypass Github branch protection for emergency fixes?

3 Answers

Bypass branch protections with a new permission