How to add new members in Group using Firebase

UPDATE

After reading your comment I would propose another approach.

When the Group creator user adds users to a group, if a user does not already exists you could, from the front-end, call a Callable Cloud Function (CF) that creates a temporary Firestore document in a specific collection. The ID of this document will be the user Id.

Then, still in this same Cloud Function, you send an email to the email address (you need to generate yourself the email, for example with the email extension) with a link containing the userId as query string value.

Example of code for this first CF:

exports.provisionNewAccount = functions
    .https.onCall(async (data, context) => {

        try {

            // New user email
            const userEmail = data.email;

            // Generate the new user docID
            let fakeDocRef = admin.firestore().collection('_').doc();
            let requestId = fakeDocRef.id;
            fakeDocRef = admin.firestore().collection('_').doc()
            requestId += fakeDocRef.id;

            // Create the doc in a specific collection
            await admin.firestore().collection('usersCreationRequests').doc(requestId).set({ email: userEmail, treated: false });

            // Generate the link to include in the email
            const linkURL = 'https://your.app/register.html?requestId=' + requestId
            
            // Send the email by creating a doc in the Extension collection
            await db
                .collection("emails")
                .add({
                    to: userEmail,
                    message: {
                        subject: "....",
                        html: `<a href="${linkURL}">Click to create your account</a>`  // adapt the html to add some text
                    },
                });
                
                return {result: 'OK'}

        } catch (error) {
            console.log(JSON.stringify(error));
            throw new functions.https.HttpsError('internal', JSON.stringify(error));
        }

    });

You call it as explained here, by passing the future user's email.

When the email recipient clicks on the link, you open a specific page of your web app that shows a set of fields for the future user to enter his password, display name etc. Then on clicking on a sign-in button in this page you call a Callable Cloud Function passing it the Firestore document ID plus the field values (you get the document ID from the query string).

As shown below, this Cloud Function creates the user in the Authentication service (using the Admin SDK) and flag the Firestore document as treated. Upon getting back the Cloud Function result in the web app you authenticate the user (you have his email an password, since he/she entered it in the form).

exports.createNewAccount = functions
    .https.onCall(async (data, context) => {

        try {

            const userEmail = data.email;
            const userId = data.userId;
            const userPassword = data.password;
            const userDisplayName = data.displayName;

            // Fetch the user doc created in the first CF
            const snapshot = await admin.firestore().collection('usersCreationRequests').doc(userId).get();

            const treated = snapshot.get('treated');
            const email = snapshot.get('email');

            if (!treated && userEmail === email) {

                const createUserPayload = {
                    email,
                    emailVerified: false,
                    password: userPassword,
                    displayName: userDisplayName
                };

                const userRecord = await admin.auth().createUser(createUserPayload);

                return { result: 'OK' }

            } else {

                return { result: 'User already created' }

            }


        } catch (error) {
            console.log(JSON.stringify(error));
            throw new functions.https.HttpsError('internal', JSON.stringify(error));
        }

    });

I’m actually using this exact approach for a B2B collaborative web app in which users can invite new users by email.

INITIAL ANSWER

(Totally different from the update)

So they don't have user id property in the app… How do I fix this problem? When someone creates new group, should I develop sign up functions new user using Firebase auth?

You can use the Anonymous Authentication mode, it exactly corresponds to your needs:

You can use Firebase Authentication to create and use temporary anonymous accounts to authenticate with Firebase. These temporary anonymous accounts can be used to allow users who haven't yet signed up to your app to work with data protected by security rules. If an anonymous user decides to sign up to your app, you can link their sign-in credentials to the anonymous account so that they can continue to work with their protected data in future sessions.

When signing-in with Anonymous Authentication a userId (uid) will be created and later you will be able to convert an anonymous account to a permanent account

I always use userId to achive this kind of feature, you can use anonymous authentication to get userId after user click invite link, Then if needed unlock more feature with furter authentication(add more provider).

If you only using mail address without authentication, It's hard to write rules for prevent user access unwanted data, Like anyone knew your email are able to access your account.