PyArrow : how to concatenate two columns into a new one ? (multi index handling)

So, I've been using pyarrow recently, and I need to use it for something I've already done in dask / pandas : I have this multi index dataframe, and I need to drop the duplicates from this index, and select rows based on their index to replace them.

To handle the multi index, I want to concatenate my two index columns into another column, that I can use just like an index. That's what I did with dask, and it worked...

How to concatenate two columns into a new one with pyarrow ?

My dataframe looks like this :

table1 = pa.table({'num1': [1, 2, 3, 5, 8, 5],
                   'date': [2017-4, 2018-6, 2017-4, 2018-9, 2016-7, 2018-6],
                   'data': ["some-data", "other-stuff", Null, Null, "data", "data", Null]})

table2 = pa.table({'num1': [2, 3, 5],
                   'date': [ 2018-6, 2017-4, 2018-9],
                   'data': ["new-data", "data that was missing", "data"]})

I need to concatenate num1 and date. I have already tried the concat_arrays function, but it adds the second array's elements in the end, so, not what I want.

I also tried that :

a = lct.select(["num1"])
b = lct.select(["date"])

lst = []
for i in range(len(a)):
    for j in range(len(a[i])):
        lst.append(str(a[i][j]) + str(b[i][j]))

I know that pyarrow types are immutables, so I wanted to create a list that I would transform to an array later, and iterate over the arrays I've got to concatenate the data together. But there are a few problems :

• first, it takes time.
• Then, retrieving something with the index here is a bit more complicated, it throws an IndexError (index out of bounds) for the a[i].

I have also thought about concatenating the two dataframes with concat_tables and sorting what I needed and what I didn't need, but I still have the same problem : how do I select data using two columns ?

I also tried joining the two dataframes together, I wanted to join the dataframes on the key columns and then drop the useless data.

table_valid = table1.join(table2, keys=['num1', 'date'], join_type="left anti")

I intended to use several join types to see what worked, but I can't do any join because keys can't have null type. And I have some keys that have somme null values in them, I guess. So, knowing that, I wanted to drop the null values from the keys :

iter1 = pc.drop_null(table1["num1"])
iter2 = pc.drop_null(iter1["date"])

table_valid = iter2.join(table2, keys=['num1', 'date'], join_type="left anti")

This could have been a good idea, but trying to join iter2 to table2, I got the following error : 'pyarrow.lib.ChunkedArray' object has no attribute 'join'

I know I could just be doing all of this with pandas, but I really want to do it with pyarrow and really have no clue how...

Thank you for your time.