Matplotlib: Formatting time on x-axis of stacked horizontal bar graph

I am trying to graph the amounts of time a user spent on different tasks in a stacked horizontal bar graph using matplotlib. The x-axis is time. Each portion of the bar represents the amount of time spent on the task. However, I am getting an unexpected graph with incorrect formatting. The formatting of the graph requires me to zoom very far on the graph in order to see all of the bars.

The graph does resemble what I need, but at first it looks like this: original graph

And I have to zoom in very far in order to get this: zoomed-in graph

The zoomed-in graph seems to have the correct proportions of the bars except for the first and last data points.

My data is a list of timedelta objects so they may be added together. I am converting them to datetime objects to graph the data.

The code that gives me undesired results is:

import matplotlib.pyplot as plt
from datetime import datetime
from datetime import timedelta
import matplotlib.dates as dt

def sum_times(data, N):
    '''sums the first N datetime elements in a list
       changes from timedelta object to datetime object
    '''
    # initialize
    summ = datetime(1900, 1, 1, 0, 0, 0)
    for i in range(len(data[0:N])):
        summ += data[i]
    return summ

start_date = datetime(1900, 1, 1, 0, 0, 0)
times = [timedelta(0, 737), 
         timedelta(0, 110), 
         timedelta(0, 356), 
         timedelta(0, 171), 
         timedelta(0, 306)]

fig, ax1 = plt.subplots(1,1, sharex=True, sharey=True)

ax1.set_title("Bug being fixed")
ax1.set_xlabel('time')
ax1.xaxis_date()

# the graph uses datetime objects
ax1.barh(1, times[0] + start_date)
ax1.barh(1, times[1] + start_date, left=sum_times(times, 1))
ax1.barh(1, times[2] + start_date, left=sum_times(times, 2))
ax1.barh(1, times[3] + start_date, left=sum_times(times, 3))
ax1.barh(1, times[4] + start_date, left=sum_times(times, 4))

ax1.set_xticks(range(1,10))   # arbitrary 

plt.show()

What can I do to change the formatting?

I have tried using ax1.set_xticklabels() with either datetime or timedelta objects, but I get the exception: ValueError: ordinal must be >= 1.

When I use

ax1.xaxis.set_major_locator(dt.MinuteLocator(interval=60))
ax1.xaxis.set_major_formatter(dt.DateFormatter('%M:%S'))

the formatting is still incorrect.

I am using python 3.6, matplotlib, PyDev IDE, and Windows 8.

If there are any libraries that are better suited to display this sort of data or can supplement matplotlib/pyplot, I would appreciate a solution using one.

Also, I have tried using a loop to automate the calls to ax1.barh(...) but the graph window doesn't display and becomes unresponsive. Since I would like to graph larger data sets, I could use a more elegant solution.