As I understand, your goal is to impose restrictions by providing a single point of extension in your API and at the same time expose your top-level abstractions to end-users in order to allow them to utilize these abstractions in the code.
You can achieve both accessibility and control over the class hierarchy with sealed classes and interfaces introduced with Java 17.
Sealed
The feature was finalized by JEP 409.
A sealed class or interface can be extended or implemented only by
those classes and interfaces permitted to do so.
I.e. if sealed modifier is being applied to a class/interface it implies that only classes or interfaces listed in the permits clause of its declaration are allowed to extend/implement this class/interface.
The permits clause needs to be always present in the declaration of a sealed class/interface. And for all subclasses/subinterfaces mentioned in the permits clause of their parent it's mandatory to extend/implement the sealed the parent directly.
In turn, every subclass/subinterface has to be declared either:
- sealed (with it own permits clause etc.);
- non-sealed - a modifier that "breaks the seal" and turns this child into a normal class/interface which can be extended without any restrictions;
- final (only for classes).
Note that both sealed and non-sealed modifiers play well with abstract modifier. Which is particularly useful for this task.
A class which is sealed or non-sealed may be abstract, and have
abstract members. A sealed class may permit subclasses which are
abstract.
Implementation
The idea is to make both interfaces (existing and the one that needs to be implemented by the end-user) and existing abstract class sealed to impose restrictions on their extension.
Existing final class will remain to be final.
The extension point will be represented by a non-sealed abstract class, which extends the existing abstract class and implements the interface that defines a contract for user-specific behavior.

With this approach, you can prevent the critical behaviour defined by the Base interface and Root class from being implemented in an undesired way, by marking these methods inside the OpenChild class with final modifier. And at the same time provide a contract for user-specific behavior which all custom classes are expected to conform to.
That how it might look in the code:
public sealed interface Base permits Root {
// critical behaviour
void m1();
void m2();
}
public sealed abstract class Root implements Base
permits ClosedChild, OpenChild {
// critical behaviour
public void m1() {} // default implementation
}
public final class ClosedChild extends Root { // can't be extended further
// critical behaviour
public void m2() { /* custom implementation */ }
}
public sealed interface Outer {
// end-user-specific behaviour
void m3();
void m4();
}
public non-sealed abstract class OpenChild extends Root implements Outer {
// critical behaviour
final public void m1() { super.m1(); } // locked
final public void m2() { /* custom implementation */ } // locked
// has to be implemented by the end user
public abstract void m5(); // + methods declared by Outer
}
Alternatives
If you are using an earlier version of Java and therefore can't utilize sealed feature, then unfortunately you need to make a choice between the ability to control the extension of your top-level abstractions and accessibility of these abstractions to the end-users.
As a partial solution, you can hide from the end-user everything apart from the existing final class ClosedChild and extension point OpenChild.
In order to do that, all the classes and interfaces mentioned above have to reside in a separate package. Classes OpenChild and ClosedChild must be marked as public. Interfaces Base, Outer and abstract class Root have to be package private, i.e. they will be visible to OpenChild and ClosedChild, but their behaviour will accessible from everywhere.
Note: methods declared inside the Base abstract class in order to make them assemble to the end-user must be public, all methods in interfaces are public by default even the interface itself package private.
An example of this approach in the JDK is a package private abstract class AbstractStringBuilder, which is not publicly accessible. And its public final subclasses StringBuilder and StringBuffer share its behaviour.