We plan to use Apache Kafka in our product as it seems to handle well horizontal scaling and shock absorbing - and our cloud environment requires these features.
But we also plan to deploy our product to envrironments, where RAM consumption is a priority. These environments will however not require big throughput. Let's say 1MB/s, with bigger deployments 5MB/s. Messages will be rather small, 1kB each.
We do not require HA in for these deployments, nor any perfect performance. We just need it to be stable, 24h retention (we'll have enough space on reasonable disks).
Would it be possible to run Kafka under these conditions with 1GB RAM? (or even less). If not, what will be the minimal setup?