matSuffix not working when wrapped inside of matFormField

I am currently trying to write a global datepicker. For that I created a component observing the given example from angular (https://material.angular.io/guide/creating-a-custom-form-field-control). I only want to outsource a part of the whole mat-form-field, so this what I was doing:

app.component.html:

  <mat-form-field color="accent" floatLabel="always">
    <mat-label>Date of Birth</mat-label>
    <my-date-picker formControlName="birthday"></my-date-picker>
  </mat-form-field>

my-date-picker.component.html:

<input matInput [matDatepicker]="picker" placeholder="TT.MM.JJJJ" [ngModel]="value"
  (ngModelChange)="modelChanged($event)">
<mat-datepicker-toggle matSuffix [for]="picker"></mat-datepicker-toggle>
<mat-datepicker #picker></mat-datepicker>

(I think the .ts shouldn't be important for my problem)

But now it looks like this:

This also happens if I would wrap it like this in the app.component.html:

  <mat-form-field color="accent" floatLabel="always">
    <mat-label>Date of Birth</mat-label>
    <div>
      <input matInput [matDatepicker]="picker" placeholder="TT.MM.JJJJ" [ngModel]="value"
      (ngModelChange)="modelChanged($event)">
      <mat-datepicker-toggle matSuffix [for]="picker"></mat-datepicker-toggle>
      <mat-datepicker #picker></mat-datepicker>
    </div>
  </mat-form-field>

Because then it renders the matSuffix as part of the mat-infix. So it seems like the matSuffix is not recognized correctly.

Whilst it would render it like this without the extra wrapper (divs):

Putting:

<mat-datepicker-toggle matSuffix [for]="picker"></mat-datepicker-toggle>
<mat-datepicker #picker></mat-datepicker>

in the app.component.html is not an option. I tried this and this just leads to other errors, which are worse than the css error.