Google Maps v3 performance issues

I'm experiencing massive performance issues with the google maps v3 api and could not find anyone with similar issues when searching for this problem. This problem can be seen on any browser, but I'm focussing on Chrome here.

The Symptoms:

When I zoom into the map or out of the map, the FPS rate drops drastically leading to an increadibly bad experience.

My Code:

I implemented the google map using their documentation with the most simple example.

<!DOCTYPE html>
<html>
  <head>
    <title>Simple Map</title>
    <meta name="viewport" content="initial-scale=1.0">
    <meta charset="utf-8">
    <style>
      html, body {
        height: 100%;
        margin: 0;
        padding: 0;
      }
      #map {
        height: 100%;
      }
    </style>
  </head>
  <body>
    <div id="map"></div>
    <script>
      var map;
      function initMap() {
        map = new google.maps.Map(document.getElementById('map'), {
          center: {lat: -34.397, lng: 150.644},
          zoom: 8
        });
      }
    </script>
    <script src="https://maps.googleapis.com/maps/api/js?callback=initMap"
    async defer></script>
  </body>
</html>

What I tested so far:

First of all, going to https://maps.google.com and trying out the zoom feature (with the mouse wheel) yields really good results. Everything is smooth and the FPS rate is at 60 fps. During my tests, I opened both the official google maps website and my small example page, and did the following:

In the Chrome development console I enabled "Show FPS meter". Here are two screenshots taken during a 20-second zoom-in-and-zoom-out session for both the original website and my website.

Note that the original google maps website does not seem to be using the GPU Memory at all, while my website makes extreme use of it. Actually in the moment of taking the screenshot it was at 146 MB, however, this was fluctuating between 0-200MB with very high speed.

The next thing I did was to enable the Paint Flashing feature of the development console in Chrome, which shows the rectangles that are redrawn. I again zoomed in and out of the map and made screenshots:

Here it becomes very clear that my website is trying to repaint only what is changing whilst the original google maps website always repaints the whole picture. No matter how long I keep zooming, the original website always repaints the whole thing and does not make any use of the GPU memory at all.

Now my Question:

Has anyone every experienced similar behavior and is there any way (maybe some settings) that let me also render the map just the way google is doing it on their website? My specific use case really makes it impossible to use google maps with the performance I am experiencing.