Let's assume large objects (multi-megabyte attachments in my case, several thousands per day) are received via a REST endpoint and stored in a case class like this:
case class Box(largeBase64Object: String, results: List[String])
Now, instances of this case class are processed in multiple, consecutive steps (chain). Each chain step may change the instance by calling box.copy(results = "foo" :: box.results) (actually, this example is simplified, it is actually a shapeless HList which stores the result of each step).
Single steps might be, e.g. virus scanning, which would add the scanner's result (infected/not infected as a Boolean) to the results list.
This approach, however, would always create a new copy of the potentially large attachments. Yes, garbage collection would collect the outdated copy sooner or later, but I'm still afraid of the potential overhead as we are talking of about several gigabytes per day of attachment data.
The other obvious approach would be to store the attachment in a global mutable Map and to just store a reference in the Box. This would avoid copying the attachments once per step but the nice properties of completely immutable data structures would be gone.
What is the canonical way to handle these situation? Has anyone pointers to benchmarks reflecting this scenario (copy vs global storage)?