How can I continuously draw in SwiftUI Canvas / TimelineView without clearing the previous frame

I want to draw into the context using SwiftUI's Canvas and Timeline views, but I don't want the context to be cleared each frame. In the below example I want to draw random points to the screen forever, eventually building up to an entire image of random pixels. Instead I just get a single pixel in a random colour and location.

This is just a simple demo to demonstrate an issue I am trying to solve, so I am not looking for another way to do this such as drawing a Rect view using normal SwiftUI views. I really would like to solve this using Canvas and Timeline as it fits more with what I want to do for a series of projects.

Basically I want to do what you can do with p5.js with the draw call when you don't clear the background.

struct RandomPoints: View {
    
    let colors : [Color] = [ .red, .orange, .yellow, .green, .mint, .teal, .cyan, .blue, .indigo, .purple, .pink, .brown ]
    
    var body: some View {
        
            ZStack {
                Color.black
                
                TimelineView(.animation) { timeline in
                    Canvas { context, size in
                                                
                        // Accessing the timeline inside Canvas is what performs the animation
                        let _ = timeline.date.timeIntervalSinceReferenceDate
                        
                        context.fill(Path(CGRect(x: Double.random(in: 0..<640.0),
                                                 y: Double.random(in: 0..<480.0),
                                                 width: 2, height: 2)),
                                     with: .color(colors.randomElement()!))
                    }
                    .frame(width: 640.0, height: 480.0)
                }
            }
    
    }
}