I looked into multiple options of setting the textColor through various tags using a custom theme but none of them seem to work. I have attached screenshots for reference. Here's the custom theme I am using for when the theme of the app is Dark.
<style name="CalendarWeekDateTextDark" parent="TextAppearance.AppCompat.Button">
<item name="android:textColor">@color/colorWhite</item>
</style>
and I have used it in the XML file as
<CalendarView
android:id="@+id/calendarView"
android:layout_width="0dp"
android:layout_height="0dp"
android:dateTextAppearance="?attr/CalendarWeekDatTextDark"
android:weekDayTextAppearance="?attr/CalendarWeekDatTextDark"
app:layout_constraintBottom_toTopOf="@+id/bottomGuideline"
app:layout_constraintEnd_toEndOf="@+id/endGuideline"
app:layout_constraintStart_toStartOf="@+id/startGuideline"
app:layout_constraintTop_toBottomOf="@+id/topGuideline" />
Screenshots:

