For your code, line-by-line, the following is happening:
\Vertex[L=$x_{1}$]{x};
This creates a vertex (at the origin) with the name x and a Label $x_1$.
\Vertex[L=$h_{1}$]{h};
This also creates a vertex (at the origin) with the name h with a Label $h_1$.
\EA(x){h}
Here you add a vertex h to the EAst of x. The label name is h as well as the label.
\Edges(x)(h)
Now you try to add an edge between x and h.
Perhaps this is something that you might be interested in:
\Vertex[x = 1cm, L = $x_1$]{x};
Place a vertex called x with a Label $x_1$ (1cm east of the origin, just because).
\Vertex[x = 4cm, L = $h_1$]{h};
Place a vertex h with Label $h_1$ (4cm east of the origin, just so it doesn't sit on/close to x).
\Edges(x,h)
Draw an edge between nodes x and h.
\SetVertexMath
Node names and labels will include math content.
\EA(h){h_2}
Create a new vertex h_2 to the EAst of h.
\Edges(h,h_2)
Draw another edge between h and h_2.
Here is a complete minimal example:

\documentclass{article}
\usepackage{tkz-berge}
\usetikzlibrary{shapes}
\begin{document}
\begin{figure}
\tikzstyle{VertexStyle} = [
shape = ellipse,
minimum width = 6ex,
draw]
\tikzstyle{EdgeStyle} = [
->,
> = stealth']
\begin{tikzpicture}[scale=0.5]
\SetGraphUnit{4}
\Vertex[x = 1cm, L = $x_1$]{x};
\Vertex[x = 4cm, L = $h_1$]{h};
\Edges(x,h)
\SetVertexMath
\EA(h){h_2}
\Edges(h,h_2)
\end{tikzpicture}
\end{figure}
\end{document}