hive shell shows many logs/warning in my hive 3.1.2

Viewed 1231

My hive shell(3.1.2) on hadoop 3.1 shows many log/warning on both start up and during execution of any queries.

Startup

2020-05-10 12:10:26,988 INFO [main] session.SessionState: Created HDFS directory: /tmp/hive-hduser/hduser/022ddbe2-5580-4b63-be1b-2cd82dfe9f0f/_tmp_space.db 2020-05-10 12:10:27,037 INFO [main] conf.HiveConf: Using the default value passed in for log id: 022ddbe2-5580-4b63-be1b-2cd82dfe9f0f 2020-05-10 12:10:27,038 INFO [main] session.SessionState: Updating thread name to 022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main 2020-05-10 12:10:30,631 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore 2020-05-10 12:10:30,779 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.ObjectStore: datanucleus.autoStartMechanismMode is set to unsupported value null . Setting it to value: ignored 2020-05-10 12:10:30,791 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.ObjectStore: ObjectStore, initialize called 2020-05-10 12:10:30,794 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] conf.MetastoreConf: Found configuration file file:/home/hduser/apache-hive-3.1.2-bin/conf/hive-site.xml 2020-05-10 12:10:30,796 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] conf.MetastoreConf: Unable to find config file hivemetastore-site.xml 2020-05-10 12:10:30,797 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] conf.MetastoreConf: Found configuration file null 2020-05-10 12:10:30,799 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] conf.MetastoreConf: Unable to find config file metastore-site.xml 2020-05-10 12:10:30,799 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] conf.MetastoreConf: Found configuration file null 2020-05-10 12:10:31,951 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.Persistence: Property datanucleus.cache.level2 unknown - will be ignored 2020-05-10 12:10:35,553 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order" 2020-05-10 12:10:36,071 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.MetaStoreDirectSql: Using direct SQL, underlying DB is MYSQL 2020-05-10 12:10:36,077 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.ObjectStore: Initialized ObjectStore 2020-05-10 12:10:36,865 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:36,867 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:36,869 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:36,870 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:36,871 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:36,871 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:41,551 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:41,561 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:41,563 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:41,563 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:41,564 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:41,565 WARN [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] DataNucleus.MetaData: Metadata has jdbc-type of null yet this is not valid. Ignored 2020-05-10 12:10:46,889 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.HiveMetaStore: Added admin role in metastore 2020-05-10 12:10:46,903 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.HiveMetaStore: Added public role in metastore 2020-05-10 12:10:46,992 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.HiveMetaStore: No user is added in admin role, since config is empty 2020-05-10 12:10:47,483 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.RetryingMetaStoreClient: RetryingMetaStoreClient proxy=class org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient ugi=hduser (auth:SIMPLE) retries=1 delay=1 lifetime=0 2020-05-10 12:10:47,613 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.HiveMetaStore: 0: get_all_functions 2020-05-10 12:10:47,617 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_all_functions Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases. 2020-05-10 12:10:47,734 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] CliDriver: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.

During execution

hive> show databases

; 2020-05-10 12:12:11,062 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] conf.HiveConf: Using the default value passed in for log id: 022ddbe2-5580-4b63-be1b-2cd82dfe9f0f 2020-05-10 12:12:11,356 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Compiling command(queryId=hduser_20200510121211_0213cb21-be5b-462d-9040-0f87736b9889): show databases

2020-05-10 12:12:13,078 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Concurrency mode is disabled, not creating a lock manager 2020-05-10 12:12:13,143 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Semantic Analysis Completed (retrial = false) 2020-05-10 12:12:13,244 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Returning Hive schema: Schema(fieldSchemas:[FieldSchema(name:database_name, type:string, comment:from deserializer)], properties:null) 2020-05-10 12:12:14,270 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] exec.ListSinkOperator: Initializing operator LIST_SINK[0] 2020-05-10 12:12:14,295 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Completed compiling command(queryId=hduser_20200510121211_0213cb21-be5b-462d-9040-0f87736b9889); Time taken: 3.002 seconds 2020-05-10 12:12:14,297 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] reexec.ReExecDriver: Execution #1 of query 2020-05-10 12:12:14,298 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Concurrency mode is disabled, not creating a lock manager 2020-05-10 12:12:14,298 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Executing command(queryId=hduser_20200510121211_0213cb21-be5b-462d-9040-0f87736b9889): show databases

2020-05-10 12:12:14,328 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Starting task [Stage-0:DDL] in serial mode 2020-05-10 12:12:14,339 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] metastore.HiveMetaStore: 0: get_databases: @hive# 2020-05-10 12:12:14,339 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] HiveMetaStore.audit: ugi=hduser ip=unknown-ip-addr cmd=get_databases: @hive# 2020-05-10 12:12:14,355 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] exec.DDLTask: results : 3 2020-05-10 12:12:14,407 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Completed executing command(queryId=hduser_20200510121211_0213cb21-be5b-462d-9040-0f87736b9889); Time taken: 0.109 seconds OK 2020-05-10 12:12:14,408 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: OK 2020-05-10 12:12:14,408 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] ql.Driver: Concurrency mode is disabled, not creating a lock manager 2020-05-10 12:12:14,427 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] Configuration.deprecation: mapred.input.dir is deprecated. Instead, use mapreduce.input.fileinputformat.inputdir 2020-05-10 12:12:14,604 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] mapred.FileInputFormat: Total input files to process : 1 2020-05-10 12:12:14,675 INFO [022ddbe2-5580-4b63-be1b-2cd82dfe9f0f main] exec.ListSinkOperator: RECORDS_OUT_INTERMEDIATE:0, RECORDS_OUT_OPERATOR_LIST_SINK_0:3, d1 default demo Time taken: 3.123 seconds, Fetched: 3 row(s)

Please advise on how to get rid of these messages.

2 Answers

I am answering my own question here as the answer seems to be Hadoop 3.1.2 did not support the above mentioned version of hive. Reinstalled Hadoop 3.3.0...

add necessary log4j-sl4j jar files. I have also faced the same issue. Once you added jar files log and warn messages will get minimised.

Related