I have an application that I have implemented via Azure Function App. The application values individual financial products (up to 1 million) many times (up to 1000) for different scenarios. Each function invocation is independent so Azure functions seems the most appropriate as it can scale out as required. Each calculation (function invocation) produces as matrix of results (up to 100 x 100 x 100). The results from each function invocation need to be added together to produce the results for the entire application run.
I am using Redis Cache as the mechanism to store the total of the results. After each function has calculated its results it takes a lock on the Redis Cache, loads the total results from Redis (matrix of up to 100 X 100 X 100), deserializes the results into a C# matrix of doubles, adds the functions results to the total, serializes the C# matrix of doubles, and then stores this back to Redis Cache, then releases the Redis lock.
This all works, but performance it terrible. The bottleneck is in serializing and deserializing the matrix from a redis string.
Is there a better way of doing this, maybe an in memory cache of some sort that can store native C# values?