How to invalidate metadata for Impala using spark?

Viewed 448

I'm using PySpark to inset data into an empty table at first, but then I will have to automate the process. Using PySpark, how do I invalidate the metadata or refresh the data so that it can be read properly in Impala?

Here is a sample of my code:

spark.sql("""
select
 gps_data_adj.trip_duration
 , gps_data_adj.geometry
 , trip_summary.TRIP_HAVERSINE_DISTANCE
 , trip_summary.TRIP_GPS_DURATION
 , gps_data_adj.HAVERSINE_DISTANCE
 , gps_data_adj.GPS_INTERVAL
 , gps_data_adj.HAVERSINE_DISTANCE/trip_summary.TRIP_HAVERSINE_DISTANCE AS HAVERSINE_DISTANCE_FRACTION
 , gps_data_adj.GPS_INTERVAL/trip_summary.TRIP_GPS_DURATION AS GPS_INTERVAL_FRACTION
 , (gps_data_adj.HAVERSINE_DISTANCE/trip_summary.TRIP_HAVERSINE_DISTANCE)*gps_data_adj.trip_distance_travelled AS HAVERSINE_DISTANCE_ADJ
 , (gps_data_adj.GPS_INTERVAL/trip_summary.TRIP_GPS_DURATION)*gps_data_adj.trip_duration AS GPS_INTERVAL_ADJ
    FROM
        gps_data_adj
    INNER JOIN
        (
            SELECT
                trip_id 
                , sum(COSINES_DISTANCE) as TRIP_COSINES_DISTANCE
                , sum(HAVERSINE_DISTANCE) as TRIP_HAVERSINE_DISTANCE
                , sum(GPS_INTERVAL) AS TRIP_GPS_DURATION
            FROM
                gps_data_adj
            GROUP BY
                trip_id
        ) trip_summary
on gps_data_adj.trip_id = trip_summary.trip_id
""").write.format('parquet').mode('append').insertInto('driving_data_TEST')
0 Answers
Related