I make some request to an API and I download a large protobuf file containing several messages (around 2.1 GB) and after deserialization (see the code below, using the DecodeVarint32() function available from: https://github.com/fkgruber/protobuf_reader) I obtain a very small R object in comparison (<30 MB).
All messages are well deserialized and i get all the data inside.
The data size of the httr request content before deserialization = 2.1 GB

The data size of the R object after deserialization = 1.5 MB

How can we explain this large difference?
EDIT:
Here's a reproductible example with a subset of the data. The rg object is 1.1 MB and its deserialized outputlist (containing two messages) is only 1904 bytes.
Link to download the protobuf response object obtained from the request
rg = readRDS("rg.RDS")
library(dplyr)
library(RProtoBuf)
readProtoFiles("message2.proto", package = 'RProtoBuf')
alldata = rg$content
n = 1
i = 1
outputlist = list()
while (n < length(alldata)) {
pos = DecodeVarint32(alldata,as.integer(n))
clen = pos[[1]]
n = pos[[2]]
nend = n + clen - 1
tmp = alldata[n:nend]
outputlist[[i]] = read(Spectrum, tmp)
n = nend + 1
i = i+1
}
object.size(outputlist)
Code of the DecodeVarint32() function below:
library(bitops)
DecodeVarint32 = function(buffer, pos){
mask = (2 ^ 32 - 1)
result = 0 %>% as.raw
shift = 0
while(TRUE){
b = buffer[pos]
result = bitOr(as.numeric(result),
bitShiftL(
bitAnd(as.numeric(b), as.numeric(0x7f)),
shift))
pos = pos + 1
if(!bitAnd(b,0x80)){
result = bitAnd(result,mask)
result = as.integer(result)
return(list(result, pos))
}
shift = shift + 7
if(shift >= 64){
stop("Too many bytes when decoding varint")
}
}
}