First, I read this. But I would like to expand. To summarize:
When designing safety-critical systems a designer has to evaluate some metrics to get the confidence that the system will work as expected. It is, kind of, a mathematical proof with low enough complexity to be accessible to a human being. It has to do with accountability, reliability, auditability, etc...
On the other hand, at this point, AI is a black box that seems to work very well, but most of the times we do not have a proof of its correctness (mainly because the thing going on in the box is too complex to be analyzed), it is more like a statistical certainty: We trained the system and the system performed well for all the tests.
So, some questions?
Q1. Do these two vague thoughts make sense nowadays?
Q2. Is possible to use AI for safety-critical system and be sure of its performance? Can we have certainty about the deterministic behavior of AI? Any reference?
Q3. I guess there are already some companies selling safety-critical systems based on AI in the automotive realm for example. How do they manage to certify their products for such a restrictive market?
EDIT
About Q1: thanks to Peter, I realized that, for example, for the automotive example, there are not requirements about total certainty. ASIL D level, the most restrictive level for automotive systems, requires only an upper bound for the probability of failure. So do other ISO26262 standards and levels. I would refine the question:
Q1. Is there any safety standard in system design, at any level/subcomponent, in any field/domain, that requires total certainty?
About Q2: Even though total certainty were not required, the question still holds.
About Q3: Now I understand how they would be able to achieve certification. Anyhow, any reference would be very welcome.