Apple doesn’t publish much for users about the security systems in macOS, and documentation for developers is also limited to what it feels we need to know. One of the features that we’ve all come to know about is Gatekeeper – the system which checks all our apps when they first run, if they’ve been downloaded from the Internet outside the App Store.

Only when you look inside macOS, you realise that Gatekeeper as such doesn’t really exist. It’s a convenient name for several different systems which work together at times, but most often operate independently. This article looks at what really goes on inside the mirage of Gatekeeper.

The best-known set of checks which are performed on macOS apps is when they’re downloaded using an app like Safari which places a quarantine flag on them. If they happen to arrive in an archive such as a Zip file, that archive has a quarantine flag attached to it (as an extended attribute), and most apps used to extract its contents attach the quarantine flag to those files which are extracted, including any apps.

When you then double-click that quarantined app, LaunchServices handles that action; for an app with a quarantine flag being run for the first time, it’s translocated to a special folder for now. With a quarantine flag set, according to security policy, this in turn results in the Apple Mobile File Integrity service amfid calling for the app’s signature to be checked fully – the first of many such checks over the next fraction of a second.

The signature check is performed by the Trust service trustd; if it completes without error, then amfid is satisfied, and the app is next checked by XProtect to determine if its bears any signs of known malware. When amfid checks an app like this, if errors are found in the signature, they are taken seriously and escalated, resulting in macOS crashing the app with that security error.

If the app passes XProtect’s scrutiny, its quarantine flag is cleared, ensuring that it won’t need to undergo those same checks again. Interestingly, those flags set on code bundles and code within the app are cleared, but not those on non-executable documents within the bundle, even the signature file itself and Info.plist.

Launch of the app is only the start of a long series of checks. For instance, in order for the app to be granted entitlements and to gain access to protected data, the Transparency Consent and Control system TCC asks the security system to perform further checks, which include signature checks run by trustd. There’s an important difference now, though, in that errors returned during TCC checks might not result in action taken to terminate the app. For this, security policy is checked, and in some circumstances, such as undeclared attempts to access camera or microphone, TCC may cause the app to crash.

Some of this sequence will also be run in other circumstances, such as when LaunchServices is asked to run an app from a path which it hasn’t previously known that particular app. In this case, amfid should call trustd to run a full signature check, and will still crash the app if that fails. However, without the quarantine flag being set, XProtect’s checks won’t be run.

These are summarised diagrammatically below.

So why don’t some app installations follow the same routes? When an app updates itself in place using the Sparkle system, or another update installer is run, the updated app doesn’t appear to pass through some or all of those checks, even though it may have been downloaded from the Internet.

Such updates can either be total, in which case the new version of the app is installed and the old moved to the Trash, or incremental, in which the updater assembles the updated app using a combination of old and new files within the app bundle. Where the update has been downloaded using a mechanism which leads to attachment of quarantine flags, those are removed, so that the updated app doesn’t pass through the normal sequence of checks. Instead, once the new app is ready to run, the updater runs its own check on its signature, in place of that normally run by amfid.

Provided that the updated app doesn’t have a quarantine flag set, and the app has the same ID and is running from the same path as before, this won’t trigger either the full check by amfid and XProtect, or amfid‘s check of the signature alone.

Sparkle has some additional features designed to ensure good security: it will only connect to and obtain updates from a designated HTTPS server, using a current version of TLS, and it has its own signature system in addition to that in macOS. Its omission of standard XProtect checks may at one time have been seen as a compromise, when Apple kept that protection up to date with the latest malware, but many recent malware threats have not been suitable for XProtect’s signature-based approach, and XProtect isn’t as useful as it once was.

Other installers, either total or incremental, may use similar tricks to work their way around standard checks, but should always check the integrity of the end result using its code signature.

The sequence of checks known popularly as Gatekeeper isn’t hard for malware to bypass. Remove quarantine flags, sign with a rogue developer ID, hide your malicious code so that XProtect can’t spot it, or use of a vulnerability in the process, and malware can and does get past. Although macOS 10.15 is most unlikely to abandon this system, Apple is now placing greater emphasis on its checking and approval of apps, either through the App Store or with notarization.

We’ll soon see how that works out.