WooCommerce Filter woocommerce_adjust_non_base_location_prices in cart

I try to show prices based on the customers country (price incl. country based tax). This works great. The customer pays always € 119.00, no matter where he lives. Only the tax is adjusting.

This works only with the following code:

add_filter( 'woocommerce_adjust_non_base_location_prices', '__return_false' );

Now I have the following challenge: There are countries with 0% tax, because they are paying the tax in their own country (like Norway). If the tax rate is 0% I want to charge the price without tax (standard rate of our shop is 19%) - in the case of Norway it is just € 100.00.

So I tried the following code:

function disable_woo_vat_adjustment(){

    $total_tax = WC()->cart->get_subtotal_tax();

    if( isset( $total_tax ) && $total_tax > 0 ){
        add_filter( 'woocommerce_adjust_non_base_location_prices', '__return_false' );
    }

}

do_action( 'after_setup_theme', 'disable_woo_vat_adjustment' );

This code still works for Norway, the price is € 100.00 and this is correct. But now the price for all other countries is wrong, because the price is not always the same (€ 119.00), instead it is € 100.00 + tax (for Austria 20.00% => € 120.00 instead of € 119.00).

I tried to add a second check in the if clause:

function disable_woo_vat_adjustment(){

    $total_tax = WC()->cart->get_subtotal_tax();

    if( isset( $total_tax ) && !empty( $total_tax ) && $total_tax > 0 ){
        add_filter( 'woocommerce_adjust_non_base_location_prices', '__return_false' );
    }else{
        remove_filter( 'woocommerce_adjust_non_base_location_prices', '__return_false' );
    }

}

do_action( 'after_setup_theme', 'disable_woo_vat_adjustment' );

I've also tried not to remove the function, but to set the filter to __return_true in the else condition, but didn't work as well for all other countries (like Austria).

What am I missing?

Thank you.