How does java compiler deal with code like `public interface A extends B<A>`

To answer your explicit question, the compiler validates generics by checking constraints against each other, validating that the specific types in question are legal/matching the generic references. After that (for the output code) generic types are erased. This is called 'type erasure'. The explicit steps are explained here: https://docs.oracle.com/javase/tutorial/java/generics/erasure.html

But more interesting are the semantics of what you apparently encountered.

Generic types don't mean 'use to create' but rather 'regarding' (in whatever way). Take List for instance. List<A> means 'List containing instances of class A'. For the code calling an instance of that, it means that all methods referencing the generic type will directly or indirectly regard object(s) of type A.

Any class implementing public interface B<E extends B<E>> will be required to regard themselves in that respect. The point of this might be that at a later point methods or attributes that refer to E will then yield the type implementing them. An example could be:

public interface Chainable<E extends Chainable<E>> {
  public void append(E followup);

  public E getNext();
}

...

public class MyLink implements Chainable<MyLink> {
  public void append(MyLink l) {
    ...
  }

  public MyLink getNext() {
    ...
  }
}

This way, it is ensured, that any class implementing Chainable will have a method taking, and one returning an object of that class, rather than just any object.

While the above alone is useful in terms of constraints I cannot see the better use of your example's public interface A extends B<A>. The reason for this is, that any class now implementing A is just guaranteed to regard A but not the implementing class itself (which would be much more useful). Example:

public class C implements A {
  public A methodDeclaredInB(A someParam) {
    ...
  }
}

The method above only knows A at compile time, but not C. If instead someone wrote:

public class C implements B<C> {
  public C methodDeclaredInB(C someParam) {
    ...
  }
}

then class C could be used instead of 'only' A.

There may be cases where it is sufficient to know type A, but usually you want type C, so making a (itself non-generic) interface A that extends B in such a way seems a fragile shortcut to the more verbose but more useful example above.

Yes it is astonishing what the compiler is capable of. As said by the other answer(s), generic parameters are used by reference. But there are compilations (.class) and their dependent relations. Notice: some runtime errors are caused by unsynchronized compilation, and you can store a .java file in a jar - I believe).

That juggling with self-references is often used to pass the class of a child class to a super class for restricting things (methods) to the child class.

But the same trick of cyclic behavior can be done as pure classes:

public interface Foo {
    Foo ZERO = new Bar();
}

public class Bar implements Foo {
    ... ZERO ...
}

So the java compiler solves this chicken/egg problem.