How to reduce the memory footprint of a large list of images in SwiftUI

I'm playing with SwiftUI and I'm currently struggling with images. Basically, I want to display an list of images with an infinite scroll but I want to keep the memory usage reasonable.

I have the following (truncated) code:

struct HomeView: View {
    @State private var wallpapers: Loadable<[Wallpaper]> = .notRequested
    @State private var currentPage = 1

    @Environment(\.container) private var container

    var body: some View {
        content
            .onAppear { loadWallpapers() }
    }

    private var content: some View {
        VStack {
            wallpapersList(data: wallpapers.value ?? [])

            // ...
        }
    }

    private func wallpapersList(data: [Wallpaper]) -> some View {
        ScrollView {
            LazyVStack(spacing: 5) {
                ForEach(data) { w in
                    networkImage(url: w.thumbs.original)
                        .onAppear { loadNextPage(current: w.id) }
                }
            }
        }
    }

    private func networkImage(url: String) -> some View {
        // I use https://github.com/onevcat/Kingfisher to handle image loading
        KFImage(URL(string: url))
            // ...
    }

    private func loadWallpapers() {
        container.interactors.wallpapers.load(data: $wallpapers, page: currentPage)
    }

    private func loadNextPage(current: String) {
        // ...
    }
}

struct WallpapersInteractor: PWallpapersInteractor {
    let state: Store<AppState>
    let agent: PWallpapersAgent

    func load(data: LoadableSubject<[Wallpaper]>, page: Int) {
        let store = CancelBag()
        data.wrappedValue.setLoading(store: store)

        Just.withErrorType((), Error.self)
            .flatMap { _ in
                agent.loadWallpapers(page: page) // network call here
            }
            .map { response in
                response.data
            }
            .sink { subCompletion in
                if case let .failure(error) = subCompletion {
                    data.wrappedValue.setFailed(error: error)
                }
            } receiveValue: {
                if var currentWallpapers = data.wrappedValue.value {
                    currentWallpapers.append(contentsOf: $0) // /!\
                    data.wrappedValue.setLoaded(value: currentWallpapers)
                } else {
                    data.wrappedValue.setLoaded(value: $0)
                }
            }
            .store(in: store)
    }
}

Because I append the new data to my Binding every time I request a new batch of images, the memory consumption quickly becomes stupidly high.

I tried to remove data from the array using .removeFirst(pageSize) once I get to the third page so that my array contains at most 2 * pageSize elements (pageSize being 64 in this case). But doing so makes my list all jumpy because the content goes up, which creates more problems than it solves.

I tried searching for a solution but I surprisingly didn't find anything on this particular topic, am I missing something obvious ?