Flutter Provider Mixing Local State with Global Change Notifier

I have a bit of a philosophical question here regarding providers.

I have a user provider as such:

@JsonSerializable(explicitToJson: true)
class ZUser extends ChangeNotifier {
  final String uid;
  String? displayName;
  String? email;
  String? phoneNumber;
  String? photoURL;
  String? did;
  List<String>? interests = [];

  @JsonKey(ignore: true)
  Database _db = Database();

  ZUser({required this.uid}) {
    Database().getUser(uid).listen((user) async {
      displayName = user?.displayName;
      email = user?.email;
      phoneNumber = user?.phoneNumber;
      photoURL = user?.photoURL;
      did = user?.did;
      interests = user?.interests;
      notifyListeners();
    });
  }

  Future addInterest(String interest) async {
    interests ??= [];
    if (!interests!.contains(interest)) {
      interests!.add(interest);
      return _db.updateUser(uid, {"interests": interests});
    }
  }

  Future removeInterest(String interest) async {
    interests ??= [];
    if (interests!.contains(interest)) {
      interests!.remove(interest);
      return _db.updateUser(uid, {"interests": interests});
    }
  }

  factory ZUser.fromJson(Map<String, dynamic> json) => _$ZUserFromJson(json);

  Map<String, dynamic> toJson() => _$ZUserToJson(this);
}

Notice that I listen on DB changes with a stream, and then notify listeners. Now I have a local class that I want to listen to interests for the user. In this class, I want to show a loading indicator when an item is selected/deselected, and then remove said indicator when the item syncs with the DB. I tried this, but I run into race condition issues, and many times the indicator persists far longer than it took to sync with the DB;

class _HomeState extends State<Home> {
  bool generalLoading = false;


  static const String generalStr = "GENERAL";

  @override
  Widget build(BuildContext context) {
    var zuser = Provider.of<ZUser?>(this);
    return zuser == null
        ? const Loading()
        : Scaffold(
            backgroundColor: context.backgroundColor,
            appBar: const PreferredSize(
              preferredSize: Size.fromHeight(Constants.BarHeight),
              child: ZLandingMenuBar(),
            ),
            body: Column(
              mainAxisAlignment: context.isMobile
                  ? MainAxisAlignment.start
                  : MainAxisAlignment.center,
              children: [
                BlockContainer(
                  child: Column(
                    crossAxisAlignment: CrossAxisAlignment.center,
                    mainAxisAlignment: MainAxisAlignment.center,
                    children: [
                      ZCheckBoxTile(
                        title: "General Interests",
                        loading: generalLoading,
                        value: zuser.interests?.contains(generalStr),
                        onPressed: () {
                          if (generalLoading) return;
                          setState(() {
                            generalLoading = true;
                          });

                          zuser.interests != null &&
                                  zuser.interests!.contains(generalStr)
                              ? zuser.removeInterest(generalStr).whenComplete(
                                    () => setState(() {
                                      generalLoading = false;
                                    }),
                                  )
                              : zuser.addInterest(generalStr).whenComplete(
                                    () => setState(() {
                                      generalLoading = false;
                                    }),
                                  );
                        },
                      ),
                      context.sd,
                    ],
                  ),
                ),
              ],
            ),
          );
  }
}

I want to remove the loading if and only if I get an update from the provider (and not for any other build of the widget). Any ideas for a cleaner way to do this?