I am running this code with spark-submit with the following configuration
spark-submit --conf spark.kubernetes.pyspark.pythonVersion=3 --queue=OID --num-executors=10 Proceso_Match_20220905.py
But it returns me the following error:
Traceback (most recent call last):
File "/home/aic_proceso_vfs/rjaimea/vfs_504/bin/Proceso_Match_20220905.py", line 20, in <module>
print rddCentralMap.PipelinedRDD()
AttributeError: 'PipelinedRDD' object has no attribute 'PipelinedRDD'
Proceso_Match_20220905.py
import os
import sys
from pyspark.sql import HiveContext, Row
from pyspark import SparkContext, SparkConf
from pyspark.sql.functions import *
from pyspark.sql import functions as F
from pyspark.sql.types import *
if __name__ =='__main__':
conf=SparkConf().setAppName("Spark RDD").set("spark.speculation","true")
sc=SparkContext(conf=conf)
sc.setLogLevel("WARN")
sqlContext = HiveContext(sc)
secuencia = 1
exit
rddCentral = sc.textFile("/user/aic_proceso_vfs/rjaimea/vfs_504/archivos_csv/rja_gdr_mss_20201023_axis_tar.csv")
rddCentralMap = rddCentral.map(lambda line : line.split(","))
print rddCentralMap.PipelinedRDD()
sc.stop()
What is missing, why does this error occur?
Execution log
22/09/14 19:57:38 INFO spark.SparkContext: Running Spark version 2.4.7.7.1.7.1000-141
22/09/14 19:57:38 INFO logging.DriverLogger: Added a local log appender at: /tmp/spark-910a8a00-9047-4d82-8231-ca83e76db2b6/__driver_logs__/driver.log
22/09/14 19:57:38 INFO spark.SparkContext: Submitted application: Spark RDD
22/09/14 19:57:38 INFO spark.SecurityManager: Changing view acls to: aic_proceso_vfs
22/09/14 19:57:38 INFO spark.SecurityManager: Changing modify acls to: aic_proceso_vfs
22/09/14 19:57:38 INFO spark.SecurityManager: Changing view acls groups to:
22/09/14 19:57:38 INFO spark.SecurityManager: Changing modify acls groups to:
22/09/14 19:57:38 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users with view permissions: Set(aic_proceso_vfs); groups with view permissions: Set(); users with modify permissions: Set(aic_proceso_vfs); groups with modify permissions: Set()
22/09/14 19:57:39 INFO util.Utils: Successfully started service 'sparkDriver' on port 34173.
22/09/14 19:57:39 INFO spark.SparkEnv: Registering MapOutputTracker
22/09/14 19:57:39 INFO spark.SparkEnv: Registering BlockManagerMaster
22/09/14 19:57:39 INFO storage.BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
22/09/14 19:57:39 INFO storage.BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
22/09/14 19:57:39 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-e7c499b2-eb05-466a-9f19-788317d276ce
22/09/14 19:57:39 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB
22/09/14 19:57:39 INFO spark.SparkEnv: Registering OutputCommitCoordinator
22/09/14 19:57:39 INFO util.log: Logging initialized @4971ms to org.spark_project.jetty.util.log.Slf4jLog
22/09/14 19:57:40 INFO server.Server: jetty-9.4.40.v20210413; built: 2021-04-13T20:42:42.668Z; git: b881a572662e1943a14ae12e7e1207989f218b74; jvm 1.8.0_232-b09
22/09/14 19:57:40 INFO server.Server: Started @5215ms
22/09/14 19:57:40 INFO server.AbstractConnector: Started ServerConnector@1f0f597f{HTTP/1.1, (http/1.1)}{0.0.0.0:4040}
22/09/14 19:57:40 INFO util.Utils: Successfully started service 'SparkUI' on port 4040.
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@312644f8{/jobs,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@14cf8c93{/jobs/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4ef688b8{/jobs/job,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@68926063{/jobs/job/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7335855{/stages,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1c481ccb{/stages/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@4f85f5a5{/stages/stage,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1ea258c7{/stages/stage/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@55f05592{/stages/pool,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@724a1f1f{/stages/pool/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@77e6ff3f{/storage,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3b195202{/storage/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@30f377f3{/storage/rdd,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@10f5c50b{/storage/rdd/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2823d73e{/environment,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@1f57dab2{/environment/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@9bd21cd{/executors,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@2ddf3301{/executors/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@7e77bc67{/executors/threadDump,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@41a03889{/executors/threadDump/json,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@3ec6dfa6{/static,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@5449b4b6{/,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@673197e4{/api,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@a4b7463{/jobs/job/kill,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@60144ede{/stages/stage/kill,null,AVAILABLE,@Spark}
22/09/14 19:57:40 INFO ui.SparkUI: Bound SparkUI to 0.0.0.0, and started at http://cl-hdp-cdp-nn1.cse-cph.int:4040
22/09/14 19:57:40 INFO security.YARNHadoopDelegationTokenManager: Attempting to load user's ticket cache.
22/09/14 19:57:41 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1967764497_19, ugi=aic_proceso_vfs@LABDOM.LOCAL (auth:KERBEROS)]] with renewer yarn/cl-hdp-cdp-nn1.cse-cph.int@LABDOM.LOCAL
22/09/14 19:57:42 INFO hdfs.DFSClient: Created token for aic_proceso_vfs: HDFS_DELEGATION_TOKEN owner=aic_proceso_vfs@LABDOM.LOCAL, renewer=yarn, realUser=, issueDate=1663203462163, maxDate=1663808262163, sequenceNumber=8093, masterKeyId=112 on ha-hdfs:pgdacl-hdp-nm-ha
22/09/14 19:57:42 INFO security.HadoopFSDelegationTokenProvider: getting token for: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1967764497_19, ugi=aic_proceso_vfs@LABDOM.LOCAL (auth:KERBEROS)]] with renewer aic_proceso_vfs@LABDOM.LOCAL
22/09/14 19:57:42 INFO hdfs.DFSClient: Created token for aic_proceso_vfs: HDFS_DELEGATION_TOKEN owner=aic_proceso_vfs@LABDOM.LOCAL, renewer=aic_proceso_vfs, realUser=, issueDate=1663203462203, maxDate=1663808262203, sequenceNumber=8094, masterKeyId=112 on ha-hdfs:pgdacl-hdp-nm-ha
22/09/14 19:57:42 INFO security.HadoopFSDelegationTokenProvider: Renewal interval is 86400151 for token HDFS_DELEGATION_TOKEN
22/09/14 19:57:42 INFO conf.HiveConf: Found configuration file file:/etc/hive/conf.cloudera.hive_on_tez/hive-site.xml
22/09/14 19:57:42 WARN conf.HiveConf: HiveConf of name hive.start.db.check does not exist
22/09/14 19:57:42 WARN conf.HiveConf: HiveConf of name hive.stats.fetch.partition.stats does not exist
22/09/14 19:57:42 WARN conf.HiveConf: HiveConf of name hive.masking.algo does not exist
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.5.5-141-f8239b25373d3268aad95a601d0fd27fb86b28fd, built on 03/24/2022 16:18 GMT
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:host.name=cl-hdp-cdp-nn1.cse-cph.int
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_232
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.vendor=AdoptOpenJDK
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_232-cloudera/jre
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/logs/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/spark/conf/
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.library.path=:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/hadoop/lib/native:/opt/cloudera/parcels/CDH-7.1.7-1.cdh7.1.7.p1000.24102687/lib/hadoop/lib/native:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:os.version=3.10.0-1160.el7.x86_64
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:user.name=aic_proceso_vfs
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/aic_proceso_vfs
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/aic_proceso_vfs/rjaimea/vfs_504/bin
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:os.memory.free=606MB
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:os.memory.max=910MB
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Client environment:os.memory.total=732MB
22/09/14 19:57:45 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=cl-hdp-cdp-dn1.cse-cph.int:2181,cl-hdp-cdp-nn2.cse-cph.int:2181,cl-hdp-cdp-nn1.cse-cph.int:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$59/585695520@13986759
22/09/14 19:57:45 INFO common.X509Util: Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
22/09/14 19:57:45 INFO zookeeper.ClientCnxnSocket: jute.maxbuffer value is 4194304 Bytes
22/09/14 19:57:45 INFO zookeeper.ClientCnxn: zookeeper.request.timeout value is 0. feature enabled=
22/09/14 19:57:45 INFO zookeeper.ClientCnxn: Opening socket connection to server cl-hdp-cdp-nn2.cse-cph.int/172.20.0.127:2181. Will not attempt to authenticate using SASL (unknown error)
22/09/14 19:57:45 INFO zookeeper.ClientCnxn: Socket connection established, initiating session, client: /172.20.0.126:42532, server: cl-hdp-cdp-nn2.cse-cph.int/172.20.0.127:2181
22/09/14 19:57:45 INFO zookeeper.ClientCnxn: Session establishment complete on server cl-hdp-cdp-nn2.cse-cph.int/172.20.0.127:2181, sessionid = 0x20000025106142f, negotiated timeout = 90000
22/09/14 19:57:47 INFO security.HBaseDelegationTokenProvider: Get token from HBase: Kind: HBASE_AUTH_TOKEN, Service: 1311d655-37c8-4df8-8d7c-9d4d78745fd8, Ident: ((username=aic_proceso_vfs@LABDOM.LOCAL, keyId=71, issueDate=1663203467518, expirationDate=1663808267518, sequenceNumber=3))
22/09/14 19:57:47 INFO deploy.SparkHadoopUtil: Updating delegation tokens for current user.
22/09/14 19:57:47 INFO zookeeper.ZooKeeper: Session: 0x20000025106142f closed
22/09/14 19:57:47 INFO zookeeper.ClientCnxn: EventThread shut down for session: 0x20000025106142f
22/09/14 19:57:47 INFO util.Utils: Using initial executors = 10, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
22/09/14 19:57:48 INFO yarn.Client: Requesting a new application from cluster with 5 NodeManagers
22/09/14 19:57:48 INFO conf.Configuration: resource-types.xml not found
22/09/14 19:57:48 INFO resource.ResourceUtils: Unable to find 'resource-types.xml'.
22/09/14 19:57:48 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (92416 MB per container)
22/09/14 19:57:48 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead
22/09/14 19:57:48 INFO yarn.Client: Setting up container launch context for our AM
22/09/14 19:57:48 INFO yarn.Client: Setting up the launch environment for our AM container
22/09/14 19:57:48 INFO yarn.Client: Preparing resources for our AM container
22/09/14 19:57:48 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
22/09/14 19:57:48 INFO yarn.Client: Uploading resource file:/tmp/spark-910a8a00-9047-4d82-8231-ca83e76db2b6/__spark_conf__4067743511513927631.zip -> hdfs://pgdacl-hdp-nm-ha/user/aic_proceso_vfs/.sparkStaging/application_1663087208786_0217/__spark_conf__.zip
22/09/14 19:57:48 WARN ipc.Client: Exception encountered while connecting to the server : org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.ipc.StandbyException): Operation category READ is not supported in state standby. Visit https://s.apache.org/sbnn-error
22/09/14 19:57:48 INFO spark.SecurityManager: Changing view acls to: aic_proceso_vfs
22/09/14 19:57:48 INFO spark.SecurityManager: Changing modify acls to: aic_proceso_vfs
22/09/14 19:57:48 INFO spark.SecurityManager: Changing view acls groups to:
22/09/14 19:57:48 INFO spark.SecurityManager: Changing modify acls groups to:
22/09/14 19:57:48 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls enabled; users with view permissions: Set(aic_proceso_vfs); groups with view permissions: Set(); users with modify permissions: Set(aic_proceso_vfs); groups with modify permissions: Set()
22/09/14 19:57:49 INFO yarn.Client: Submitting application application_1663087208786_0217 to ResourceManager
22/09/14 19:57:49 INFO impl.YarnClientImpl: Submitted application application_1663087208786_0217
22/09/14 19:57:50 INFO yarn.Client: Application report for application_1663087208786_0217 (state: ACCEPTED)
22/09/14 19:57:50 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: AM container is launched, waiting for AM container to Register with RM
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: OID
start time: 1663203469058
final status: UNDEFINED
tracking URL: http://cl-hdp-cdp-nn1.cse-cph.int:8088/proxy/application_1663087208786_0217/
user: aic_proceso_vfs
22/09/14 19:57:51 INFO yarn.Client: Application report for application_1663087208786_0217 (state: ACCEPTED)
22/09/14 19:57:52 INFO yarn.Client: Application report for application_1663087208786_0217 (state: ACCEPTED)
22/09/14 19:57:53 INFO yarn.Client: Application report for application_1663087208786_0217 (state: ACCEPTED)
22/09/14 19:57:54 INFO yarn.Client: Application report for application_1663087208786_0217 (state: ACCEPTED)
22/09/14 19:57:55 INFO yarn.Client: Application report for application_1663087208786_0217 (state: RUNNING)
22/09/14 19:57:55 INFO yarn.Client:
client token: Token { kind: YARN_CLIENT_TOKEN, service: }
diagnostics: N/A
ApplicationMaster host: 172.20.0.135
ApplicationMaster RPC port: -1
queue: OID
start time: 1663203469058
final status: UNDEFINED
tracking URL: http://cl-hdp-cdp-nn1.cse-cph.int:8088/proxy/application_1663087208786_0217/
user: aic_proceso_vfs
22/09/14 19:57:55 INFO cluster.YarnClientSchedulerBackend: Application application_1663087208786_0217 has started running.
22/09/14 19:57:55 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1663087208786_0217 and attemptId None
22/09/14 19:57:55 INFO cluster.YarnScheduler: Starting speculative execution thread
22/09/14 19:57:55 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37015.
22/09/14 19:57:55 INFO netty.NettyBlockTransferService: Server created on cl-hdp-cdp-nn1.cse-cph.int:37015
22/09/14 19:57:55 INFO storage.BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
22/09/14 19:57:55 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> cl-hdp-cdp-nn1.cse-cph.int,cl-hdp-cdp-nn2.cse-cph.int, PROXY_URI_BASES -> http://cl-hdp-cdp-nn1.cse-cph.int:8088/proxy/application_1663087208786_0217,http://cl-hdp-cdp-nn2.cse-cph.int:8088/proxy/application_1663087208786_0217, RM_HA_URLS -> cl-hdp-cdp-nn1.cse-cph.int:8088,cl-hdp-cdp-nn2.cse-cph.int:8088), /proxy/application_1663087208786_0217
22/09/14 19:57:55 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, cl-hdp-cdp-nn1.cse-cph.int, 37015, None)
22/09/14 19:57:55 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-nn1.cse-cph.int:37015 with 366.3 MB RAM, BlockManagerId(driver, cl-hdp-cdp-nn1.cse-cph.int, 37015, None)
22/09/14 19:57:55 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, cl-hdp-cdp-nn1.cse-cph.int, 37015, None)
22/09/14 19:57:55 INFO storage.BlockManager: external shuffle service port = 7337
22/09/14 19:57:55 INFO storage.BlockManager: Initialized BlockManager: BlockManagerId(driver, cl-hdp-cdp-nn1.cse-cph.int, 37015, None)
22/09/14 19:57:55 INFO ui.ServerInfo: Adding filter to /metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
22/09/14 19:57:55 INFO handler.ContextHandler: Started o.s.j.s.ServletContextHandler@f0fc770{/metrics/json,null,AVAILABLE,@Spark}
22/09/14 19:57:56 INFO scheduler.EventLoggingListener: Logging events to hdfs://pgdacl-hdp-nm-ha/spark2-history/application_1663087208786_0217
22/09/14 19:57:56 INFO util.Utils: Using initial executors = 10, max of spark.dynamicAllocation.initialExecutors, spark.dynamicAllocation.minExecutors and spark.executor.instances
22/09/14 19:57:56 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
22/09/14 19:57:56 INFO logging.DriverLogger$DfsAsyncWriter: Started driver log file sync to: /user/spark/driverLogs/application_1663087208786_0217_driver.log
22/09/14 19:57:57 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
22/09/14 19:58:01 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.135:34428) with ID 2
22/09/14 19:58:01 INFO dynalloc.ExecutorMonitor: New executor 2 has registered (new total is 1)
22/09/14 19:58:01 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn7.cse-cph.int:41318 with 366.3 MB RAM, BlockManagerId(2, cl-hdp-cdp-dn7.cse-cph.int, 41318, None)
22/09/14 19:58:01 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.135:34436) with ID 6
22/09/14 19:58:01 INFO dynalloc.ExecutorMonitor: New executor 6 has registered (new total is 2)
22/09/14 19:58:01 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn7.cse-cph.int:39286 with 366.3 MB RAM, BlockManagerId(6, cl-hdp-cdp-dn7.cse-cph.int, 39286, None)
22/09/14 19:58:04 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.129:54986) with ID 9
22/09/14 19:58:04 INFO dynalloc.ExecutorMonitor: New executor 9 has registered (new total is 3)
22/09/14 19:58:04 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.128:46112) with ID 5
22/09/14 19:58:04 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn2.cse-cph.int:34997 with 366.3 MB RAM, BlockManagerId(9, cl-hdp-cdp-dn2.cse-cph.int, 34997, None)
22/09/14 19:58:04 INFO dynalloc.ExecutorMonitor: New executor 5 has registered (new total is 4)
22/09/14 19:58:04 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.130:60364) with ID 7
22/09/14 19:58:04 INFO dynalloc.ExecutorMonitor: New executor 7 has registered (new total is 5)
22/09/14 19:58:04 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.129:54988) with ID 4
22/09/14 19:58:04 INFO dynalloc.ExecutorMonitor: New executor 4 has registered (new total is 6)
22/09/14 19:58:04 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.130:60366) with ID 1
22/09/14 19:58:04 INFO dynalloc.ExecutorMonitor: New executor 1 has registered (new total is 7)
22/09/14 19:58:05 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn1.cse-cph.int:35435 with 366.3 MB RAM, BlockManagerId(5, cl-hdp-cdp-dn1.cse-cph.int, 35435, None)
22/09/14 19:58:05 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn3.cse-cph.int:45960 with 366.3 MB RAM, BlockManagerId(7, cl-hdp-cdp-dn3.cse-cph.int, 45960, None)
22/09/14 19:58:05 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn2.cse-cph.int:36926 with 366.3 MB RAM, BlockManagerId(4, cl-hdp-cdp-dn2.cse-cph.int, 36926, None)
22/09/14 19:58:05 INFO storage.BlockManagerMasterEndpoint: Registering block manager cl-hdp-cdp-dn3.cse-cph.int:45118 with 366.3 MB RAM, BlockManagerId(1, cl-hdp-cdp-dn3.cse-cph.int, 45118, None)
22/09/14 19:58:05 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (172.20.0.136:55720) with ID 3
22/09/14 19:58:05 INFO dynalloc.ExecutorMonitor: New executor 3 has registered (new total is 8)
22/09/14 19:58:05 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
Traceback (most recent call last):
File "/home/aic_proceso_vfs/rjaimea/vfs_504/bin/Proceso_Match_20220905.py", line 20, in <module>
print rddCentralMap.PipelinedRDD()
AttributeError: 'PipelinedRDD' object has no attribute 'PipelinedRDD'