The join behavior for a dynamic frame is different than what I expected. We provide two lists of columns to the join method, and it seems that if the columns from the first match any permutation of the columns from the second, then that is considered equality. Here is an example to demonstrate - the pyspark dataframe join gives what I would expect, whereas the dynamic frame join seems to consider permutations of column values to be a match.
from awsglue import DynamicFrame
from pyspark import SparkContext
from awsglue.context import GlueContext
sc = SparkContext.getOrCreate()
gc = GlueContext(sc)
col_names = ['a', 'b']
df1 = spark.createDataFrame([(1, 2), (2, 1)], col_names)
df2 = spark.createDataFrame([(1, 2)], col_names)
joined_df = df1.join(df2, col_names)
joined_df.show()
+---+---+
| a| b|
+---+---+
| 1| 2|
+---+---+
dyf1 = DynamicFrame.fromDF(df1, gc, 'dyf1')
dyf2 = DynamicFrame.fromDF(df2, gc, 'dyf2')
joined_dyf = dyf1.join(col_names, col_names, dyf2)
joined_dyf.show()
{"b": 2, ".b": 2, ".a": 1, "a": 1}
{"b": 2, ".b": 1, ".a": 2, "a": 1}
joined_dyf.toDF().show() # convert to dataframe before the show for better visualization
+---+---+---+---+
| b| .b| a| .a|
+---+---+---+---+
| 2| 2| 1| 1|
| 2| 1| 1| 2|
+---+---+---+---+
I did not expect the second row in the last table above to be included. Is this behavior for dynamic frame joins intentional?