AccessDeniedException while running Apache Kafka 3 on Windows

user_1497944 0
Viewed 7042

Apache Kafka 3.0 was released yesterday and I decided to try it on Windows 10. Although, ZooKeeper starts perfectly, Kafka shuts down after logging few exceptions.

I understand that AccessDeniedException is quite self explanatory but the permission isn't the issue because Kafka is able to create the very files for which AccessDeniedException is logged.

[2021-09-22 23:58:18,295] ERROR Failed to write meta.properties due to (kafka.server.BrokerMetadataCheckpoint)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:214)
        at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:204)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2(KafkaServer.scala:772)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2$adapted(KafkaServer.scala:770)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
        at kafka.server.KafkaServer.checkpointBrokerMetadata(KafkaServer.scala:770)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:322)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-09-22 23:58:18,301] ERROR [KafkaServer id=0] Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.BrokerMetadataCheckpoint.liftedTree1$1(BrokerMetadataCheckpoint.scala:214)
        at kafka.server.BrokerMetadataCheckpoint.write(BrokerMetadataCheckpoint.scala:204)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2(KafkaServer.scala:772)
        at kafka.server.KafkaServer.$anonfun$checkpointBrokerMetadata$2$adapted(KafkaServer.scala:770)
        at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
        at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
        at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
        at scala.collection.IterableOps$WithFilter.foreach(Iterable.scala:889)
        at kafka.server.KafkaServer.checkpointBrokerMetadata(KafkaServer.scala:770)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:322)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)

and

[2021-09-22 23:58:19,115] INFO Shutting down. (kafka.log.LogManager)
[2021-09-22 23:58:19,200] ERROR Error while writing to checkpoint file D:\tmp\kafka-logs\recovery-point-offset-checkpoint (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:114)
        at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:92)
        at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:67)
        at kafka.log.LogManager.$anonfun$checkpointRecoveryOffsetsInDir$1(LogManager.scala:675)
        at kafka.log.LogManager.$anonfun$checkpointRecoveryOffsetsInDir$1$adapted(LogManager.scala:671)
        at scala.Option.foreach(Option.scala:437)
        at kafka.log.LogManager.checkpointRecoveryOffsetsInDir(LogManager.scala:671)
        at kafka.log.LogManager.$anonfun$shutdown$9(LogManager.scala:542)
        at kafka.log.LogManager.$anonfun$shutdown$9$adapted(LogManager.scala:535)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
        at kafka.log.LogManager.shutdown(LogManager.scala:535)
        at kafka.server.KafkaServer.$anonfun$shutdown$18(KafkaServer.scala:701)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
        at kafka.server.KafkaServer.shutdown(KafkaServer.scala:701)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:435)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-09-22 23:58:19,202] ERROR Disk error while writing recovery offsets checkpoint in directory D:\tmp\kafka-logs: Error while writing to checkpoint file D:\tmp\kafka-logs\recovery-point-offset-checkpoint (kafka.log.LogManager)
[2021-09-22 23:58:19,233] ERROR Error while writing to checkpoint file D:\tmp\kafka-logs\log-start-offset-checkpoint (kafka.server.LogDirFailureChannel)
java.nio.file.AccessDeniedException: D:\tmp\kafka-logs
        at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:89)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103)
        at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108)
        at java.base/sun.nio.fs.WindowsFileSystemProvider.newFileChannel(WindowsFileSystemProvider.java:121)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:298)
        at java.base/java.nio.channels.FileChannel.open(FileChannel.java:357)
        at org.apache.kafka.common.utils.Utils.flushDir(Utils.java:953)
        at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:941)
        at kafka.server.checkpoints.CheckpointFile.liftedTree1$1(CheckpointFile.scala:114)
        at kafka.server.checkpoints.CheckpointFile.write(CheckpointFile.scala:92)
        at kafka.server.checkpoints.OffsetCheckpointFile.write(OffsetCheckpointFile.scala:67)
        at kafka.log.LogManager.$anonfun$checkpointLogStartOffsetsInDir$1(LogManager.scala:698)
        at kafka.log.LogManager.$anonfun$checkpointLogStartOffsetsInDir$1$adapted(LogManager.scala:694)
        at scala.Option.foreach(Option.scala:437)
        at kafka.log.LogManager.checkpointLogStartOffsetsInDir(LogManager.scala:694)
        at kafka.log.LogManager.$anonfun$shutdown$9(LogManager.scala:545)
        at kafka.log.LogManager.$anonfun$shutdown$9$adapted(LogManager.scala:535)
        at kafka.utils.Implicits$MapExtensionMethods$.$anonfun$forKeyValue$1(Implicits.scala:62)
        at scala.collection.mutable.HashMap$Node.foreachEntry(HashMap.scala:633)
        at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:499)
        at kafka.log.LogManager.shutdown(LogManager.scala:535)
        at kafka.server.KafkaServer.$anonfun$shutdown$18(KafkaServer.scala:701)
        at kafka.utils.CoreUtils$.swallow(CoreUtils.scala:68)
        at kafka.server.KafkaServer.shutdown(KafkaServer.scala:701)
        at kafka.server.KafkaServer.startup(KafkaServer.scala:435)
        at kafka.Kafka$.main(Kafka.scala:109)
        at kafka.Kafka.main(Kafka.scala)
[2021-09-22 23:58:19,234] ERROR Disk error while writing log start offsets checkpoint in directory D:\tmp\kafka-logs: Error while writing to checkpoint file D:\tmp\kafka-logs\log-start-offset-checkpoint (kafka.log.LogManager)
[2021-09-22 23:58:19,236] INFO Shutdown complete. (kafka.log.LogManager)

ZooKeeper scripts -

bin\windows\kafka-server-start.bat .\config\server.properties

Kafka starting scripts -

bin\windows\kafka-server-start.bat .\config\server.properties

I am using default configuration files for ZooKeeper and Kafka in following environment -

Environment

JDK 11.0.2
Kafka 3.0
Windows 10
apache-kafka windows java
1 Answers

Nowdays in Windows 10 it's possible to use wsl2 to run kafka. Just please follow this guide:

https://www.confluent.io/blog/set-up-and-run-kafka-on-windows-linux-wsl-2/#enable-windows-subsystem

then you can write code and launch it in wsl

Sometimes there is an issue with windows and wsl communication so you have to issue in windows command prompt wsl --shutdown and then restart it (issue wsl again), after that you can call your wsl launched application and call its exposed services from windows

user_61454060
Related