There is a novel (Feb 2022) technique showcased in 2 minute papers.

Autors call it the "Mixbox", and the technique is open source (algorithm can be found on github, mind the license though). Currently Rebelle 5 is implementing this color mixing technique, more programs may follow in the nearest future.

Imo the strongest point of this is its simplicity in implementation and how easily you can use it out of the box. Their paper is also (supposedly -- I didn't read it) well written.

Wondering if calculation of inversion of the RGB value work. Since it's about subtraction of lights, technically the subtraction part can be calculated by simple math.

For example cyan + yellow

cyan = 0x00ffff yellow = 0xffff00

Their inversions are 0xff0000 and 0x0000ff, meaning they absorbed red and blue lights completely. Their 1:1 mixture should absorb half of red and blue lights (since the other half of the mixture can still reflects some red and blue light), which is consistent with (0xff0000 + 0x00ffff) / 2 = 0x7f007f. Now we subtract the value from 0xffffff we have 0x80ff80 which is green!

I would like to add that I have creted a library in python that handles this problem in a way that spares you from having to implement complex interpolation and color system convertion functions. It is called colorir.

Example of blue + yellow:

PolarGrad(["0000ff", "ffff00"]).perc(0.5)

Gives you: "#00c5a1"

Behind the curtains, colorir interpolates the colors in HCLuv, a perceptually uniform color system. This does not mimic a "real" mixture of colors in the sense that it wont account for the subtractive properties of real pigments (i.e. the final mixed color will not look darker than the originals). It will, however, mix the colors as in finding what the almost exact average of the visual properties of the two colors look like. The following steps are all handled automatically by the PolarGrad class:

1. Interpret the value of the input colors (in this case the hex rgb values "0000ff" - blue and "ffff00" - yellow)
2. Convert these colors to a perceptually uniform color coordinate system (by default HCLuv, but other color spaces can be used)
3. Sample a color from the middle of the gradient (0.5)
4. Convert the sampled color back to RGB or other format that you may want

Mixing Colors like Pigments

1. Remove white from all colors, keeping the white parts and color parts
2. Average the RGB values of the white parts removed from the colors
3. Average the RGB values of the color parts
4. Take out the white from the averaged color parts
5. Half the white value removed and add that value to the Green of the averaged color parts
6. Add the averaged white parts back in and make whole number

Javascript Implementation

// rgbColor is an array of colors,
// where each color is an array with the three color values (RGB, 0 to 255)
function mixRGB(rgbColor) {
        function sumColors(summedColors, nextColor) {
            return summedColors.map(
                (summedColor, i) => nextColor[i] + summedColor,
            );
        }

        function averageColors(colors) {
            return colors
                .reduce(sumColors, [0, 0, 0])
                .map(c => c/colors.length);
        }

        // Remove white from all colors
        const whiteParts = [];
        const colorParts = [];
        rgbColor.forEach(color => {
            const whiteVal = Math.min(...color);
            whiteParts.push([whiteVal, whiteVal, whiteVal]);
            colorParts.push(color.map(val => val - whiteVal));
        });
        
        // Average the whites from each selection
        const averagedWhite = averageColors(whiteParts);
        // Average all non-white colors
        let averagedColor = averageColors(colorParts);

        // Take out the white from the averaged colors
        const whitePart = Math.min(...averagedColor);
        averagedColor = averagedColor.map(color => color - whitePart);

        // Half the white value removed and add that value to the Green
        averagedColor[1] += (whitePart / 2);
        
        // Add the averaged white back in and make whole number
        averagedColor = averagedColor.map((color, i) => Math.floor(color + averagedWhite[i]));
    
        return averagedColor;
  }

I built a color game based off this you can try