How can I add click-to-play to my HTML5 videos without interfering with native controls?

I'm using the following code to add click-to-play functionality to HTML5 video:

$('video').click(function() {
  if ($(this).get(0).paused) {
    $(this).get(0).play();
  }
  else {
    $(this).get(0).pause();
  }
});

It works okay except that it interferes with the browser's native controls: that is, it captures when a user clicks on the pause/play button, immediately reversing their selection and rendering the pause/play button ineffective.

Is there a way to select just the video part in the DOM, or failing that, a way to capture clicks to the controls part of the video container, so that I can ignore/reverse the click-to-play functionality when a user presses the pause/play button?