I am migrating some ETL sql from mysql to Spark sql . Encounter very strange exception . I hope some one can help me out. the test table :
create table spark_hdw.test_bug(
a decimal(20,2),
b decimal(20,2),
c decimal(21,2)
)
Spark sql:
%spark.sql
SELECT
*
FROM
(
SELECT a AS kpi_04
FROM
spark_hdw.test_bug
UNION ALL
SELECT
a+b AS kpi_04
FROM
spark_hdw.test_bug
) t
Exception:
Resolved attribute(s) kpi_04#17864 missing from kpi_04#17872 in operator !Project [kpi_04#17864]. Attribute(s) with the same name appear in the operation: kpi_04. Please check if the right attribute(s) are used.;;
!Project [kpi_04#17864]
+- SubqueryAlias t
+- Union
:- Project [cast(kpi_04#17864 as decimal(21,2)) AS kpi_04#17872]
: +- Project [a#17866 AS kpi_04#17864]
: +- SubqueryAlias test_bug
: +- HiveTableRelation `spark_hdw`.`test_bug`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [a#17866, b#17867, c#17868]
+- Project [kpi_04#17865]
+- Project [CheckOverflow((promote_precision(cast(a#17869 as decimal(21,2))) + promote_precision(cast(b#17870 as decimal(21,2)))), DecimalType(21,2)) AS kpi_04#17865]
+- SubqueryAlias test_bug
+- HiveTableRelation `spark_hdw`.`test_bug`, org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, [a#17869, b#17870, c#17871]