Data streaming in Apache-Ignite through Apache-Kafka consumes high CPU

I am using Ignite Source Connector (Single Kafka connector node) to export the Ignite Events from my ignite cluster(2 Nodes) to Kafka broker nodes (2 Nodes) with single topic and 29 partitions.

I have processed 0.1 Million events (PUT, DELETE) per minute with the 1k size messages(in Avg).

My connector node consumes 90% of the CPU (you can see it below).

My Connector node machine config.

RAM --> 30 GB
ROM --> 1TB
Configured Heap --> 15GB
No of CPU Core --> 40 (20 x 2)



#connector
name=my-ignite-source-connector
connector.class=class of my source connector
tasks.max=1
topicNames=ignite-data

#cache
cacheName=DOCIDS
cacheAllowOverwrite=true
cacheEvts=put , removed
evtBatchSize=100
numberOfPartitions=29
igniteCfg=/myconfig/ignite-config.xml

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:util="http://www.springframework.org/schema/util"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans
        http://www.springframework.org/schema/beans/spring-beans.xsd
        http://www.springframework.org/schema/util
        http://www.springframework.org/schema/util/spring-util.xsd">
    <bean abstract="true" id="ignite.cfg" class="org.apache.ignite.configuration.IgniteConfiguration">
        <!-- Set to true to enable distributed class loading for examples, default is false. -->
        <property name="communicationSpi"><bean class="org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi"><property name="socketWriteTimeout" value="60000"/></bean></property><property name="peerClassLoadingEnabled" value="true"/><property name="clientFailureDetectionTimeout" value="10000"/><property name="dataStorageConfiguration"><bean class="org.apache.ignite.configuration.DataStorageConfiguration"><property name="walPath" value="wal_data/"/><property name="walArchivePath" value="wal_data/"/><property name="defaultDataRegionConfiguration"><bean class="org.apache.ignite.configuration.DataRegionConfiguration"><property name="persistenceEnabled" value="true"/><property name="maxSize" value="#{15L * 1024 * 1024 * 1024}"/></bean></property></bean></property>

        <!-- Enable task execution events for examples. -->
        <property name="includeEventTypes">
            <list>
                <!--Task execution events-->
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_STARTED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FINISHED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_TASK_FAILED"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_PUT"/>
                <util:constant static-field="org.apache.ignite.events.EventType.EVT_CACHE_OBJECT_REMOVED"/>
                <!--Cache events-->



            </list>
        </property>
 <!-- Explicitly configure TCP discovery SPI to provide list of initial nodes. -->
        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi">
                <property name="ipFinder">
                    <!--
                        Ignite provides several options for automatic discovery that can be used
                        instead os static IP based discovery. For information on all options refer
                        to our documentation: http://apacheignite.readme.io/docs/cluster-config
                    -->
                    <!-- Uncomment static IP finder to enable static-based discovery of initial nodes. -->
                    <!--<bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">-->
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder">
                        <property name="addresses">
                            <list>
                                <!-- In distributed environment, replace with actual host IP address. -->
                                <value>ignite-server-node1:47500..47509</value><value>ignite-server-node2:47500..47509</value><value>kafka-conect-node:47500..47509</value>
                            </list>
                        </property>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>
</beans>

The data transfer process is working as intended only for a few hours because of this high CPU consumption, after that the Kafka connector node will be OFFLINE in ignite server topology.

Any pointers will help.

Thanks in advance.

Data streaming in Apache-Ignite through Apache-Kafka consumes high CPU

0 Answers