Problem

I only need the color when the user is finished with his selection or respectively closes the color picker.

I have the problem that the change event fires to often if an user tries to pick a color from the following view in the color picker.

Research

The first thing I did was test this in Firefox, Chrome, and Safari to see how they differed. I posted the three videos of the tests here. Only in Firefox did it behave as expected. (Side note: It worked as expected for Android Chrome.)

Furthermore, I found that it behaved as expected in Windows using Chrome & Firefox. So there appears to be discrepancies in both the OS and Browser type.

Now, according to the official HTML standard, change is...

Fired at controls when the user commits a value change (see also the input event)

I haven't found to much more in my research, but I propose the following reason for why this is happening. On macOS, there is nowhere to confirm the color you selected like there is on Windows, such as a "Okay" or "Select" button. When the user closes the color picker on macOS, the only way to do so is by clicking the "X". Now, how is the browser/website supposed to know if the color the user selected before closing that window is what the user wanted, or if they closed it because that wasn't the color they wanted? Therefore, browsers like Chrome and Safari are always searching for what color the user selects, constantly updating it. However browsers like Firefox take a different approach and simple search for when the user closes the color selector, and then uses that value.

In summary, the browsers may be interpreting the standard different ways. Whenever the user moves his mouse over a new color, this could be though of as "committing a value change" per the standard for the change event.

Possible Solutions

There is a thread with a question very similar to yours here. As you can see, everyone seems to have varying results with recreating the problem based on what browser/OS they are using.

The best solution I could some up with is based on a combination of discussion in this thread, and looking at the docs. By using onblur attribute in the input tag, you can call a function when the user no longer selects the input box. Perhaps you could put a "submit" button to guide them into clicking off the input box. When this function is called, take the value in the input form and you can do with you want with that, such as printing it to the console.

Here is an example of me testing a small jsfiddle I made: https://imgur.com/a/4k0zJC4

Notice how after I select a color, I close the selector, and then I click in the blank space somewhere on the screen to deselect the input field. Then, the value I choose was printed to the console.

Another solution is to use a custom made color picker like jscolor or spectrum. (There was an issue similar to what you described with spectrum here, but it appears they fixed it.)

Cross-browser event differences are a common problem. My suggestion is below:

Try the oninput attribute to trigger a function when an input is finally confirmed.

e.g.

<input type="color" value="#ff0000" id="bgcolor" name="bgcolor" oninput="changeBackground(bgcolor.value)">

The triggered function looks like:

function changeBackground(colorValue){
    document.body.style.backgroundColor = colorValue;
}

Change the style to the style you wish to change if it is not background color.