How do we replay projection in a production environment?
For example, we have about 100k events, to replay, it takes about 15 minutes, if we do this live, new events may come in and the projection will not be up to date after the replay.
So aside from schedule a system down time, how do we replay the projection gracefully?
Projection is always (potentially) not up to date. Projections are Data on the Outside -- unlocked, non-authoritative copies of the real data.
The fact that projection updates lag behind the changes to the authoritative copies of the data is an inevitable consequence of distributing copies of the data.
So aside from schedule a system down time, how do we replay the projection gracefully?
You accept into your design that the projections are data "as at" some time in the past; and you let the system run with the previously cached projection while the new projection is assembled.
We typically name our projections. If you projected all your order events into a projected-orders-v1 you can create a projected-orders-v2 in parallel and let it build up in the background.
When it's ready you do the code change required to access the new projections.
After that you can delete your old projection if you want.
This requires that your projection mechanism can read your event log from the beginning independently.
Update: Designing your system according to CQRS, separating READS from WRITES, solves this as there will be separate non-conflicting processes. One process is responsible for writing events to the end of the event stream, and (at least) one is responsible for reading from the beginning of the event stream. The process reading the events don't have to care if the event is new or not, it will only have to keep track of it's position (last known event) and keep reading forever.
