How to convert Pandas Timestamp to Matplotlib's float64 format?

The code in below plots a Pandas Time Series with milliseconds unit. However, Matplotlib stores this milliseconds timestamp as float64 internally, so that I can reformat them to hh:mm:ss as shown in below. Does anyone know how to manually convert Pandas timestamp to Matplotlib's float64 format? Thanks

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates

x = np.arange(5000)
df = pd.DataFrame(x, columns=['y'])
df['timestamp'] = pd.to_datetime(df.index, unit='ms')
df.set_index('timestamp', inplace=True)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(df.index, df.y)
ax.xaxis.set_major_formatter(mdates.DateFormatter('%H:%M:%S'))

Matplotlib stores date in float64 format

print(ax.get_xlim())
>> (-2.892939814814815e-06, 6.075173611111111e-05)

print(type(ax.get_xlim()[0]))
>> numpy.float64

print(df.index[0])    # how to convert to -2.892939814814815e-06
>> 1970-01-01 00:00:00

print(df.index[-1])   # how to convert to 6.075173611111111e-05
>> 1970-01-01 00:00:04.999000

The mdates.date2num() doesn't work either

print(mdates.date2num(df.index[0]))
>> 0.0

print(mdates.date2num(df.index[-1]))
>> 5.785879629629629e-05

Final Update:

Sorry, I was wrong. The mdates.date2num() actually works! Initially I thought ax.get_xlim() was returning (x[0], x[-1]), but it actually returns something before x[0], and something after x[-1] so that the figure has some padding around the line. That's why it returns (-2.892939814814815e-06, 6.075173611111111e-05) instead of (0.0, 5.785879629629629e-05). Now I finally get it. Thanks guys!!