Kafka and Golang with SSL

Question

I am trying to connect to a topic that requires SSL. Can anyone suggest an approach to achieve this?

I have a ca.crt, ca.p12 and password provided by the kafka cluster administrator. I can connect just fine using a test utility that leverages the keytool/keystore, but these don't work for Go.

The code to connect to the cluster is as follows:

func (p *KafkaPublisher) Initialise() {
    configFile := "./config/kafka.properties"
    fmt.Printf("Reading config file from: %s
", configFile)
    conf := ReadConfig(configFile)

    var err error
    kafkaProducer, err = kafka.NewProducer(&conf)

    // rest of code commented out.
}

And this works just fine for SASL using the following config file:

bootstrap.servers=markets-cb--vog--pu-taejvaba.bf2.kafka.rhcloud.com:443
security.protocol=SASL_SSL
sasl.mechanisms=PLAIN
# Service account user name
sasl.username=700e9c83-b4be-4f23-8697-b6cfa5921354
sasl.password=ab955a0d-e78d-4d96-bd8e-35de3b6b83e5

# Best practice for Kafka producer to prevent data loss
acks=all

I reviewed the following documentation to try and find the right properties: https://docs.confluent.io/platform/current/clients/librdkafka/html/md_CONFIGURATION.html#autotoc_md91 and have been unsuccessful

bootstrap.servers=my-cluster-kafka-listener1-bootstrap-climate.violet-cluster-new-2761a99850dd8c23002367ac6ce7f9ad-0000.au-syd.containers.appdomain.cloud:443

security.protocol=SSL
ssl.key.password=ebuSuzkDbfFK
ssl.certificate.location="ca.pem"

# Best practice for Kafka producer to prevent data loss
acks=all

I have also tried putting the pem file inline as a single quote-delimited line using the property ssl.certificate.pem. This produced the the most encouraging result with an error:

Failed to create producer: ssl.certificate.pem failed: not in PEM format?: error:0909006C:PEM routines:get_name:no start line: Expecting: CERTIF

But it is a PEM file format...

Here is the certificate (Don't worry, there is no security risk with me pasting this in here as the cert is now removed from the server and I changed the url and password above.)

Kafka and Golang with SSL

0 Answers