MQ - Data Compression on queue

Question

Problem: MQ7 has a hard limit of a maximum 100MB jms message. For large payloads ( xml ) that are close that, can this be compressed on the queue to shorten the data length?

I tried compressing a 7MB jms string message using the MQ ZLIB compression on the svr.def.conn channel and it didn't make any difference to the data length of the jms message. I only set the one channel and expected the channel that is used would compress the data going into the queue.

MQ Server: 7.5 Client: JAVA Message Type: String

user_974186 · Answer

I would recommend to commpress the payload and use a ByteMessage. Message properties can be used to qualify the payload type, similar to HTTP e.g. "Content-Encoding", "Content-Type"

String payload = ...; // the xml

Session session = ...;
BytesMessage bytesMessage = session.createBytesMessage();

bytesMessage.writeBytes(compressGZIP(payload, StandardCharsets.UTF_8));
bytesMessage.setStringProperty("Content-Encoding", "gzip");
bytesMessage.setStringProperty("Content-Type", "text/xml; charset=utf-8");

Here is the compressGZIP method:

private byte[] compressGZIP(String string, Charset charset) throws IOException {
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();
    try (GZIPOutputStream out = new GZIPOutputStream(byteArrayOutputStream)) {
        StringReader stringReader = new StringReader(string);
        // IOUtils from apache commons-io
        IOUtils.copy(stringReader, out, charset);
    }
    return byteArrayOutputStream.toByteArray();
}

The consumer can then ask for the message properties, decompress and re-create the xml based on the "Content-Encoding" and "Content-Type" message properties.

Something like this

public void onMessage(Message message) {
    BytesMessage bytesMessage = (BytesMessage) message;

    long bodyLength = bytesMessage.getBodyLength();
    byte[] rawPayload = new byte[(int) bodyLength];
    InputStream payloadInputStream = new ByteArrayInputStream(rawPayload);

    String contentEncoding = bytesMessage.getStringProperty("Content-Encoding");
    if("gzip".equals(contentEncoding)) {
        payloadInputStream = new GZIPInputStream(payloadInputStream);
    }

    String contentType = bytesMessage.getStringProperty("Content-Type");
    MimeType mimeType = new MimeType(contentType); // from javax.activation

    if("text".equals(mimeType.getPrimaryType())) {
        if("xml".equals(mimeType.getSubType())) {
            Charset charset;

            String charsetString = mimeType.getParameter("charset");
            if(charsetString != null) {
                charset = Charset.forName(charsetString);
            } else {
                charset = StandardCharsets.UTF_8; // default
            }

            Reader reader = new InputStreamReader(payloadInputStream, charset);

            String xml = IOUtils.toString(reader);
            IOUtils.closeQuietly(reader);
        }
    }
}

The advantage of this solution is that you stay on the standard JMS api instead of using a provider specific configuration.

The disadvantage is that the sender and receiver must implement content-type handling.

Thus you have to make a decision between portability and implementation effort.

MQ - Data Compression on queue

2 Answers