when exactly are we supposed to use "public static final String"?

You do not have to use final, but the final is making clear to everyone else - including the compiler - that this is a constant, and that's the good practice in it.

Why people doe that even if the constant will be used only in one place and only in the same class: Because in many cases it still makes sense. If you for example know it will be final during program run, but you intend to change the value later and recompile (easier to find), and also might use it more often later-on. It is also informing other programmers about the core values in the program flow at a prominent and combined place.

An aspect the other answers are missing out unfortunately, is that using the combination of public final needs to be done very carefully, especially if other classes or packages will use your class (which can be assumed because it is public).

Here's why:

1. Because it is declared as final, the compiler will inline this field during compile time into any compilation unit reading this field. So far, so good.
2. What people tend to forget is, because the field is also declared public, the compiler will also inline this value into any other compile unit. That means other classes using this field.

What are the consequences?

Imagine you have this:

class Foo {
  public static final String VERSION = "1.0";
}

class Bar {
  public static void main(String[] args) {
    System.out.println("I am using version " + Foo.VERSION);
  }
}

After compiling and running Bar, you'll get:

I am using version 1.0

Now, you improve Foo and change the version to "1.1". After recompiling Foo, you run Bar and get this wrong output:

I am using version 1.0

This happens, because VERSION is declared final, so the actual value of it was already in-lined in Bar during the first compile run. As a consequence, to let the example of a public static final ... field propagate properly after actually changing what was declared final (you lied!;), you'd need to recompile every class using it.

I've seen this a couple of times and it is really hard to debug.

If by final you mean a constant that might change in later versions of your program, a better solution would be this:

class Foo {
  private static String version = "1.0";
  public static final String getVersion() {
    return version;
  }
}

The performance penalty of this is negligible, since JIT code generator will inline it at run-time.

It is kind of standard/best practice. There are already answers listing scenarios, but for your second question:

Why do they have to do that? Why do they have to initialize the value as final prior to using it?

Public constants and fields initialized at declaration should be "static final" rather than merely "final"

These are some of the reasons why it should be like this:

1. Making a public constant just final as opposed to static final leads to duplicating its value for every instance of the class, uselessly increasing the amount of memory required to execute the application.

2. Further, when a non-public, final field isn't also static, it implies that different instances can have different values. However, initializing a non-static final field in its declaration forces every instance to have the same value owing to the behavior of the final field.

This is related to the semantics of the code. By naming the value assigning it to a variable that has a meaningful name (even if it is used only at one place) you give it a meaning. When somebody is reading the code that person will know what that value means.

In general is not a good practice to use constant values across the code. Imagine a code full of string, integer, etc. values. After a time nobody will know what those constants are. Also a typo in a value can be a problem when the value is used on more than one place.

I think these are all clear explanations. But, Let me clarify it by giving a java inbuild example.

In java, most would have used System.out.println()

The system is a class and out is a PrintStream class.

So what java says is I will take care of the initialization of the out object(PrintStream) and keep the initialization private to myself in the System class.

public final class System {

    public final static PrintStream out = null;

    //Some initialization done by system class which cannot be changed as it is final.
}

You just access the println method statically without worrying about its initialization.