How do I detect that an iOS app is running on a jailbroken phone?

It depends what you mean by jailbreak. In the simple case, you should be able to see if Cydia is installed and go by that - something like

NSString *filePath = @"/Applications/Cydia.app";
if ([[NSFileManager defaultManager] fileExistsAtPath:filePath])
{
   // do something useful
}

For hacked kernels, it's a little (lot) more involved.

BOOL isJailbroken()
{
#if TARGET_IPHONE_SIMULATOR
    return NO;
#else
    FILE *f = fopen("/bin/bash", "r");

    if (errno == ENOENT)
    {
        // device is NOT jailbroken
        fclose(f);
        return NO;
    }
    else {
        // device IS jailbroken
        fclose(f);
        return YES;
    }
#endif
}

Please use following code for Swift 4 and above: Add the following code in the appdelegate:

private func getJailbrokenStatus() -> Bool {
    if TARGET_IPHONE_SIMULATOR != 1 {
        // Check 1 : existence of files that are common for jailbroken devices
        if FileManager.default.fileExists(atPath: "/Applications/Cydia.app")
            || FileManager.default.fileExists(atPath: "/Library/MobileSubstrate/MobileSubstrate.dylib")
            || FileManager.default.fileExists(atPath: "/bin/bash")
            || FileManager.default.fileExists(atPath: "/usr/sbin/sshd")
            || FileManager.default.fileExists(atPath: "/etc/apt")
            || FileManager.default.fileExists(atPath: "/private/var/lib/apt/")
            || UIApplication.shared.canOpenURL(URL(string:"cydia://package/com.example.package")!) {
            return true
        }
        // Check 2 : Reading and writing in system directories (sandbox violation)
        let stringToWrite = "Jailbreak Test"
        do {
            try stringToWrite.write(toFile:"/private/JailbreakTest.txt", atomically:true, encoding:String.Encoding.utf8)
            //Device is jailbroken
            return true
        } catch {
            return false
        }
    }
    else {
        return false
    }
}

Inside Appdelegate methods, write code as below

func applicationDidBecomeActive (_ application: UIApplication) {
    
    if getJailbrokenStatus() {
        let alert = UIAlertController(title: LocalizedKeys.Errors.jailbreakError, message: LocalizedKeys.Errors.jailbreakErrorMessage, preferredStyle: UIAlertController.Style.alert)
        let jailBrokenView = UIViewController()
        
        jailBrokenView.view.frame = UIScreen.main.bounds
        jailBrokenView.view.backgroundColor = .white
        self.window?.rootViewController = jailBrokenView
        jailBrokenView.present(alert, animated: true, completion: nil)
    }
    
    if #available(iOS 11.0, *) {
        if !UIScreen.main.isCaptured {
            DispatchQueue.main.async {
                self.blockImageView.removeFromSuperview()
            }
        }
    }
}

I am not aware of any "APIs" that exist for this. If there were, then a jailbreak-masking product would quickly cover them up.

As lots of people point out, it is a cat-and-mouse game. And after both players become expert, it all comes down to who gets the first move. (Person holding the device.)

I found many good suggestions for detecting jailbreak in Zdziarski's new book "Hacking and Securing iOS Apps". (Personally, I paid more for the O'Reilly eBook because they permit copy-and-paste.)

No, I am not affiliated with the publishers. But I did find it a good book. I don't like to just publish hackers' mistakes so they can fix them, so I thought I'd point to the book.

I'd suggest looking for files that aren't present on a "vanilla" iPhone. All jailbreak kits I've seen install ssh. That might be a good indicator of a jailbroken phone.

Here's my solutions: Step 1

extension UIDevice {
    func isJailBroken() -> Bool {
        let cydiaPath = "/Applications/Cydia.app"
        let aptPath = "/private/var/lib/apt/"
        if FileManager.default.fileExists(atPath: cydiaPath) || FileManager.default.fileExists(atPath: aptPath) {
            return true
        }
        return false
    }
}

Step 2: Call it inside viewDidLoad() inside your launch screen view controller(or whatever VC you are calling for the first time):

       // show a blank screen or some other view controller
       let viewController = UIDevice.current.isJailBroken() ? JailBrokenViewController() : NextViewController()
       self.navigationController?.present(viewController, animated: true, completion:nil)

In iOS 14 there is a service App Attest. Check this article.

Also i used this repo https://github.com/fiber-inc/SecurityDetector, but some users tell, that they didn't have jailbreak, when detector triggered.

So i decided to test this repo https://github.com/wearebeatcode/SwiftJailbreakDetection/blob/master/Sources/SwiftJailbreakDetection/JailbreakDetection.swift. Still the algorithm isn't good and gives a result that the jailbreak active in not jailbreaked phones. Searching further..

Now i'm trying this: https://github.com/securing/IOSSecuritySuite