To change the background color of the label you can use a custom style:
<com.google.android.material.slider.RangeSlider
style="@style/Myslider"
...>
with:
<style name="Myslider" parent="@style/Widget.MaterialComponents.Slider">
<item name="labelStyle">@style/My_Tooltip</item>
<item name="materialThemeOverlay">@style/ThemeOverlay.Slider</item>
</style>
<style name="My_Tooltip" parent="Widget.MaterialComponents.Tooltip">
<!-- background color of the Tooltip -->
<item name="backgroundTint">@color/...</item>
</style>
<style name="ThemeOverlay.Slider" parent="">
<!-- color used by the text in the Tooltip -->
<item name="colorOnPrimary">@color/...</item>
</style>
To customize the value in the label you can a LabelFormatter.
Something like:
RangeSlider slider = findViewById(R.id.slider);
slider.setLabelFormatter(new LabelFormatter() {
@NonNull
@Override
public String getFormattedValue(float value) {
//It is just an example
if (value == 3.0f)
return "TEST";
return String.format(Locale.US, "%.0f", value);
}
});

About the behavior of the label there are only these values (with 1.2.0-beta01 and 1.3.0-alpha01):
LABEL_FLOATING: The label will only be visible on interaction. It will float above the slider and may cover views above this one. This is the default and recommended behavior.
LABEL_WITHIN_BOUNDS: The label will only be visible on interaction. The label will always be drawn within the bounds of this view. This means extra space will be visible above the slider when the label is not visible.
LABEL_GONE: The label will never be drawn.
Currently it seems impossible to show always the label.
You can set the label behavior using the setLabelBehavior method or the labelBehavior in the layout or in a style.
<com.google.android.material.slider.RangeSlider
app:labelBehavior="withinBounds"/>
If you want to change the shape of the label you can check this answer.