Google drive API : DetailedApiRequestError(error : Invalid credentials)

I am using Google drive api for downloading and uploading the files using my Flutter app. My app requires google signIn by users by which I store their auth headers in shared_preferences(local disk of phone's storage) and use it to initialize the driveApi. But after some time it throws error of Invalid credentials.
I know it is getting expired and read some documentation about refresh tokens but couldn't able to understand and integrate in my app. Some of the post are recommending to use curl requests and your_client_id, your_client_secret, refresh_token but I couldn't able to find these terms. Could anyone suggest some improvements in my code to get the refresh tokens ?

Google sign in code :

final googleSignIn = GoogleSignIn(scopes: [drive.DriveApi.driveFileScope],);
  GoogleSignInAccount _user;
  GoogleSignInAccount get user => _user;

  Future<void> signInWithGoogle() async {

    final GoogleSignInAccount googleuser = await googleSignIn.signIn();
    if (googleuser == null) return;
    _user = googleuser;

    final GoogleSignInAuthentication googleAuth = await googleuser
        .authentication;

    final
    GoogleAuthCredential credential = GoogleAuthProvider.credential(
        idToken: googleAuth.idToken,
        accessToken: googleAuth.accessToken
    );


    await FirebaseAuth.instance.signInWithCredential(credential);
}

Storing the authHeaders in users phone storage :

Provider.of<GoogleSignInProvider>(context, listen: false);
await provider.signInWithGoogle();
SharedPreferences prefs = await SharedPreferences.getInstance();
final account = provider.user;
final Map<String,String> authHead = await account.authHeaders;
authHead.forEach((key, value){        //Storing the authHeaders in prefs(shared-preferences)
     prefs.setString(key,value);
});

Now using this prefs I'm initializing my driveApi, Initializing driveApi :

final Map<String, String> authHeaders = {};
SharedPreferences prefs = await SharedPreferences.getInstance();
authHeaders['Authorization'] = prefs.getString('Authorization')!;  //retrieving the authHeaders back
authHeaders['X-Goog-AuthUser'] = prefs.getString('X-Goog-AuthUser')!;
final authenticateClient = GoogleAuthClient(authHeaders);    //authenticating the client
driveApi = drive.DriveApi(authenticateClient); //Initialized