AWS MSK trigger for AWS Lamba - multiple topics in the same execution context

It is indeed supported, you have to set up each topic in a separate MSK/Kafka event source. Lambda will even consume from multiple topics using the same consumer group if you give each event source the same consumer group name.

The event looks like what you mocked up in the original question. The dictionary items at the top level of records have a key topic_partition, and the topic is also present in each Kafka record in the value of each dictionary item.

From my understanding, the event is a serialized version of the Java object below, which is from the repository here. You can derive a data structure to deserialize the event to whatever platform you are targeting in your Lambda.

Another post mentions using Glue (Spark Streaming) for this use case. That is a fine idea for high volume, big data, or analytics problems. But it is probably overkilled unless you need a distributed computing solution. But consuming low-volume topics or topics that don't need a lot of consuming logic or distributed computing is a fine fit for Lambda.

https://github.com/aws/aws-lambda-java-libs/blob/master/aws-lambda-java-events/src/main/java/com/amazonaws/services/lambda/runtime/events/KafkaEvent.java

/*
 * Copyright 2015-2020 Amazon.com, Inc. or its affiliates. All Rights Reserved.
 *
 * Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file except in compliance with
 * the License. A copy of the License is located at
 *
 * http://aws.amazon.com/apache2.0
 *
 * or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
 * CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions
 * and limitations under the License.
 */
package com.amazonaws.services.lambda.runtime.events;

import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
import java.util.List;
import java.util.Map;

@Data
@NoArgsConstructor
@AllArgsConstructor
@Builder(setterPrefix = "with")
/** Represents a Kafka Event. **/
public class KafkaEvent {
    private Map<String, List<KafkaEventRecord>> records;
    private String eventSource;
    private String eventSourceArn;
    private String bootstrapServers;
    
    @Data
    @NoArgsConstructor
    @AllArgsConstructor
    @Builder(setterPrefix = "with")
    public static class KafkaEventRecord {
        private String topic;
        private int partition;
        private long offset;
        private long timestamp;
        private String timestampType;
        private String key;
        private String value;
        private List<Map<String, byte[]>> headers;
    }

    @Data
    @NoArgsConstructor
    @AllArgsConstructor
    @Builder(setterPrefix = "with")
    public static class TopicPartition {
        private  String topic;
        private  int partition;

        @Override
        public String toString() {
            //Kafka also uses '-' for toString()
            return topic + "-" + partition;
        }
    }
}

better to use glue/spark to connect multiple topic kafka