Unaligned x scales for two Chartjs-chart types with the same configuration - only difference is "chart type" (bar/line)

I built a line chart & a bar chart in Chartjs with exactly the same configuration, but for some reason the x-scales of both charts are different as you can see in the picture below.

The configuration for the both charts is the same, the only difference is the chart type:

function getOptions() {
    return {
        animation: false,
        responsive: true,
        maintainAspectRatio: false,
        plugins: {
            tooltip: false,
            legend: {
                display: false
            },
            title: {
                display: false
            }
        },
        elements: {
            line: {
                fill: true
            },
            point: {
                radius: 0
            }
        },
        scales: {
            x: {
                type: "time",
                time: {
                    displayFormats: {
                        second: 'HH:mm:ss',
                        minute: 'HH:mm',
                        hour: 'HH'
                    },
                    unit: 'minute',
                    stepSize: 1,
                    min: '00:00:00',
                    max: '23:59:59',
                },
                grid: {
                    display: true,
                    drawBorder: true,
                    drawOnChartArea: true,
                    drawTicks: true,
                    color: function(context) {
                        return '#fff';
                    },
                }
            },
            y: {
                beginAtZero: true,
                grid: {
                    display: true,
                    drawBorder: false,
                    color: function(context) {
                        return '#fff';
                    },
                },
            }
        }
    };
}

I noticed that the x axes do align perfectly, if I set the "stepSize" to 'seconds'. This probably has to do with the fact that the data inserted into the chart is second-data.