Here is one way to binarize the image with Markov chain:
Assuming (by Markov property) that a pixel value depends only on its neighbors (let's assume a 4-nbd), let's estimate the probability that a pixel is white given that n of its nbd pixels are white, i.e., for 4-nbd, let's first compute the conditional probabilities that P(x(i,j)=1 | n of its nbrs are also 1), where n=0,1,2,3,4 for 4-nbd (also, let's use a global thresholding to compute the probabilities as shown in the following code, this can be thought of as the training phase):
from skimage.io import imread
im = imread('https://i.stack.imgur.com/r9XCE.png')
def count_nbrs(im, i, j, th): # counts number of nbd pixels are 1 given a pixel, with threshold th
count = 0
count += np.mean(im[i-1,j]) > th
count += np.mean(im[i+1,j]) > th
count += np.mean(im[i,j-1]) > th
count += np.mean(im[i,j+1]) > th
return count
th = 140 #np.mean(im) # a global threshold
nnbrs = 5 # use 4-nbd
freq = np.zeros(nnbrs)
tot = np.zeros(nnbrs)
h, w, _ = im.shape
for i in range(1, h-1):
for j in range(1, w-1):
count = count_nbrs(im, i, j, th)
if np.mean(im[i,j]) > th:
freq[count] += 1
tot[count] += 1
prob = freq/tot
print(prob)
# Prob(x(i,j)=1|n of its nbrs are 1) in the image, for n=0,1,2,3,4
# [0.00775595 0.09712838 0.48986784 0.91385768 0.99566323]
Now let's use these estimated probabilities to change each of the pixels in the colored image to black and white, depending on its nbd (this can be thought of as testing phase):
h, w, _ = im.shape
im1 = np.zeros((h, w))
for i in range(1, h-1):
for j in range(1, w-1):
c = count_nbrs(im, i, j, th) # count number of neighbors with white pixel
im1[i,j] = 255*(prob[c] > 0.5) # use Prob vector to determine value of the pixel
plt.imshow(im1, 'gray')
plt.show()
The binary image obtained has far better quality than global thresholding (check it out).

You could randomly choose pixels and compute the probability that a pixel value will be 1 given the number of 1 nbrs (thresholded), accordingly setting the pixel value, using the following code with large number of iterations N, it will result in similar binary image.
N = 100000
h, w, _ = im.shape
im1 = np.zeros((h, w))
for k in range(N):
i = np.random.randint(1,h-1,1)[0]
j = np.random.randint(1,w-1,1)[0]
c = count_nbrs(im, i, j, th)
im1[i,j] = 255*(prob[c] > 0.5)
plt.imshow(im1, 'gray')
plt.show()

The next animation shows how the binary image is generated using the above code.
