Can a singleton initialize thread-safely just use public static field rather a Lazy<T> or other Singleton solution?

First off you´re not really using a singleton here. A singleton is just the one and only instance of a specific class. A static field in itself has no instance at all. There might be dozens of ConcurrentDictionary within your code and also a dozens instances of your class.

static just means: all instances - regardless on how many of those exist - share the same member, in your case the dictionary.

Apart from this your two appraoches aren´t equivalent, even though in your specific example they seem similar. The most important difference between a static readonly field and a static Lazy field, is that the former is initialized the very first time your class is used, whereas the latter is initialized when that member is used the very first time. Chances are the lazy member isn´t called in your entire code, so it wont be initialized at all.

Two make this more clear see this example:

public class CacheData
{
    private static readonly int myint = 3;
    private static Lazy<ConcurrentDictionary<string, string>> _concurrentDic = new Lazy<ConcurrentDictionary<string, string>>(() =>
    {
        return new ConcurrentDictionary<string, string>();
    });

    public static ConcurrentDictionary<string, string> ConcurrentDic
    {
        get
        {
            return _concurrentDic.Value;
        }
    }
}

Now you have two completely independ members here. While myint is initialized when your class is used, the dictionary stays un-initialized. Only when you try to access that dictionary it will be initialized:

class MyClass
{
    void DoSomething()
    {
        // this will initialize myint if not already done
        CacheData.DoSomething(); 
        // however the dictionary is not initialized, only if you try to access it
        console.WriteLine(CacheData.ConcurrentDic["Hello"]);
        // now the dictionary is also initialized
    }
}

To keep long stories short: Lazy is ment to defer (heavy) initialization as far as possible until needed, whereas a static readonly field will be initialized on class-initialization.

Fundamentally, you are seeing behaviour that isn't guaranteed. Because it isn't guaranteed, you can't rely on it. Take the below code as an example. What output will it generate? The answer is it depends.

17.4.5.1: "If a static constructor (§17.11) exists in the class, execution of the static field initializers occurs immediately prior to executing that static constructor. Otherwise, the static field initializers are executed at an implementation-dependent time prior to the first use of a static field of that class."

implementation-dependent means multiple orders are valid, as long as the rules are kept.

It might generate:

The Dictionary has been initialized now
Start Running...
Start Running...

Or it might generate:

Start Running...
Start Running...
The Dictionary has been initialized now

You can validate it yourself at https://dotnetfiddle.net/d5Tuev (switch runtimes on the left side).

Now, in your testing you have experienced one of those behaviours. But that behaviour isn't guaranteed, and thus you shouldn't rely on it.

using System.Collections.Concurrent;
using System.Threading;

namespace Samples.Console
{
    public class Program
    {
        public static void Main(string[] args)
        {
            Thread thread1 = new Thread(() => StaticField.StaticFieldTest());
            Thread thread2 = new Thread(() => StaticField.StaticFieldTest());

            thread1.Start();
            thread2.Start();

            Thread.Sleep(1000);
        }

        public class StaticField
        {
            public static readonly ConcurrentDictionary<string, string> _dic = InitDictionary();

            public static void StaticFieldTest()
            {
                System.Console.WriteLine("Start Running...");

                _dic.TryAdd(string.Empty, string.Empty);
                _dic.TryAdd(string.Empty, string.Empty);
                _dic.TryAdd(string.Empty, string.Empty);
            }

            public static ConcurrentDictionary<string, string> InitDictionary()
            {
                System.Console.WriteLine("The Dictionary has been initialized now");
                Thread.Sleep(500);
                return new ConcurrentDictionary<string, string>();
            }
        }
    }
}

In addition - Lazy or LazyWithNoExceptionCaching are quite useful in some contexts - e.g. if you have an object which takes 20 seconds to create, but you want to ensure you create only one of them.