GPU Ops not shown on tensorboard graph

I have made a quick network using tensorflow, and it seems to be training properly. It uses the GPU for all network operations, which I have verified by creating the session with tf.Session(config=tf.ConfigProto(log_device_placement=True))

I set up the network like this:

train, image_guess = network(images, labels)

tf.summary.image('Guess', image_guess, max_outputs=3)
tf.summary.image('Input', images, max_outputs=3)
tf.summary.image('Target', tf.image.hsv_to_rgb(labels), max_outputs=3)

sess.run([
    tf.local_variables_initializer(),
    tf.global_variables_initializer(),
])

Where the network function sets up the graph and returns the training op and a resulting image tensor for logging.

def network(images, labels):
    with tf.variable_scope("NN") as scope:
        ...
        return train_step, image_guess

The trouble comes with visualizing the graph in tensorboard. For some reason it only shows a graph for my input handling operations like so:

How can I get it to properly display my entire network?