Let's do the math. For the sake of this discussion, we'll ignore the size of static mesh geometry and other data that isn't updated every frame (things like inverse bind matrices, if you're doing matrix palette skinning). How big is each one of those skinning instance data buffers?
Instance count * joint count * elements per skinning matrix * bytes per matrix element
50,000 * 50 * 12 * 4 = 120 MB
(I'm discounting other things that might vary per frame, so if it helps, replace these numbers with the orders of magnitude of your real data. The rest of the analysis still applies.)
So yes, an appreciable amount of space, but not necessarily infeasible. We might choose to drop back to double buffering instead of triple-buffering everything, cutting the total space required from 360 MB to 240 MB.
We've also assumed that we're doing matrix palette skinning using 3x4 matrices. We could cut the required space in half again by using dual quaternion skinning.
One concern here is that transferring 60 MB of skinning data per frame to the GPU will consume a large amount of bandwidth if we're running on a discrete GPU. We might therefore consider moving our joint calculations to the GPU. On iOS, with a shared memory architecture, this is less of a concern. Either way, we'll want to be savvy about how we compute our transform hierarchy, since we're going to be doing a lot of multiplications if every joint moves every frame.
As for varying instance counts, the one thing we definitely don't want to do is reallocate new buffers every frame. Instead, we can reallocate the buffer, increasing its size geometrically, when we notice that the number of instances exceeds the current maximum capacity of our buffers.
Alternatively, we could take a "paged" approach and split the instances across multiple buffers, each of which can hold some large-ish number of instances, and which get recycled by adding them to a reusable buffer pool once the GPU is done rendering from them. This does require splitting the rendering into multiple draw calls (unless perhaps we're using argument buffers), but the overhead of issuing, say, 10 instanced draw calls, each with 1/10th of the instances, will be negligible compared to the time required to actually render.
We could also consider using MTLHeap to create transient buffers more cheaply than allocating buffers directly from our device, but this still requires selecting a sensible initial upper bound and growing/reallocating as the maximum number of instances increases.
At the end of the day, theory is trumped by benchmarks. Naturally, you should start by analyzing your data, as we're trying to do here, but building a test app and putting these ideas into practice, noticing where problems actually arise, is the only real way to be sure how much optimization you need to do to run smoothly on your target platform.
