I have a small pyspark code which writes into a csv file in my local machine. Each time i am running the code,it is using different ports as the previous port is couldn't bind. here is the error codes. how can i use the same port over and over again while running same code multiple times
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4046. Attempting port 4047.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4047. Attempting port 4048.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4048. Attempting port 4049.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4049. Attempting port 4050.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4050. Attempting port 4051.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4051. Attempting port 4052.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4052. Attempting port 4053.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4053. Attempting port 4054.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4054. Attempting port 4055.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4055. Attempting port 4056.
21/08/31 21:05:45 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Failed to bind to /0.0.0.0:4056: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at org.sparkproject.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.sparkproject.jetty.server.ServerConnector.open(ServerConnector.java:308)
at org.sparkproject.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.sparkproject.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.spark.ui.JettyUtils$.newConnector$1(JettyUtils.scala:301)
at org.apache.spark.ui.JettyUtils$.httpConnect$1(JettyUtils.scala:332)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$5(JettyUtils.scala:335)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$5$adapted(JettyUtils.scala:335)
at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2256)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2248)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:336)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:146)
at org.apache.spark.SparkContext.$anonfun$new$10(SparkContext.scala:470)
at org.apache.spark.SparkContext.$anonfun$new$10$adapted(SparkContext.scala:470)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:470)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)