Chronicle queue size determination

user_5216060 0
Viewed 601

If we need to limit total size of chronicle queue to let say 1GB or 10GB. What would be the best way to do it ?

We store buffers as bytes so I was trying to calculate total size just by summing buffer sizes but looks like there is no easy way to correlate this with actual size of queue. One way is to calculate total directory size using file utils every 5 minutes or so but that would be error prone if there is huge data in the interval and it may overflow

chronicle-queue chronicle
2 Answers

you would have to add up the size of each cq4 files

the writePosition gives you the length of each cq4 file in bytes

public class Example {

    public static void main(String[] args) throws FileNotFoundException {
        SingleChronicleQueue q = SingleChronicleQueueBuilder.builder().path("tmp").build();

        ExcerptAppender appender = q.createAppender();

        try (DocumentContext dc = appender.writingDocument()) {
            long l = dc.wire().bytes().writePosition();
            dc.wire().write().text("lastx");
        }

       DumpQueueMain.dump(q.fileAbsolutePath());

    }
}

outputs the following

--- !!meta-data #binary
header: !SCQStore {
  writePosition: [
    131328,
    564049465049088
  ],
  indexing: !SCQSIndexing {
    indexCount: !short 8192,
    indexSpacing: 64,
    index2Index: 184,
    lastIndex: 64
  }
}

# position: 184, header: -1
--- !!meta-data #binary
index2index: [
  # length: 8192, used: 1
  65760 # truncated trailing zeros
]
# position: 65760, header: -1
--- !!meta-data #binary
index: [
  # length: 8192, used: 1
  131328 # truncated trailing zeros
]
# position: 131328, header: 0
--- !!data #binary
"": lastx

...
# 83754737 bytes remaining

--- !!meta-data #binary
header: !STStore {
  wireType: !WireType BINARY_LIGHT,
  recovery: !TimedStoreRecovery {
    timeStamp: 0
  },
  metadata: !SCQMeta {
    roll: !SCQSRoll { length: !int 86400000, format: yyyyMMdd, epoch: 0 },
    deltaCheckpointInterval: 64,
    sourceId: 0
  }
}

# position: 225, header: 0
--- !!data #binary
listing.highestCycle: 17780

# position: 264, header: 1
--- !!data #binary
listing.lowestCycle: 17780

# position: 304, header: 2
--- !!data #binary
listing.modCount: 1

# position: 336, header: 3
--- !!data #binary
chronicle.write.lock: -9223372036854775808

# position: 376, header: 4
--- !!data #binary
chronicle.lastIndexReplicated: -1

# position: 432, header: 5
--- !!data #binary
chronicle.lastAcknowledgedIndexReplicated: -1

...
# 65044 bytes remaining

when the length of the .cq4 file is defined by

writePosition: [
    131328,
     ....
  ],

in other words 131328 bytes

user_41066150

The chronicle queue files roll based on time not file size. Why do you wish to limit the size of the .cq4 file ?

If you wish you could roll the queue file every Minute by changing the roll cycle.

user_66876360
Related