I have a directory, such as /user/name/folder.
Inside this directory, I have more sub-directories named dt=2020-06-01, dt=2020-06-02, dt=2020-06-03, etc.
These directories contain parquet files. They all have the same schema.
Is it possible to create an Impala table using /user/name/folder?
Each time I do, I get a Table with 0 records. Is there a way to tell Impala to pull the parquet files from all of the sub-directories?
One way to do that is loading data with static partitioning in which you manually define the different partitions. With static partitioning, you create a partition manually, using an ALTER TABLE … ADD PARTITION statement, and then load the data into the partition.
CREATE TABLE customers_by_date
(cust_id STRING, name STRING)
PARTITIONED BY (dt STRING)
STORED AS PARQUET;
ALTER TABLE customers_by_country
ADD PARTITION (dt='2020-06-01')
SET LOCATION '/user/name/folder/dt=2020-06-01';
If the location is not specified then the location is created
ALTER TABLE customers_by_date
ADD PARTITION (dt='2020-06-01');
and you could load data with HDFS commands too
$ hdfs dfs -cp /user/name/folder/dt=2020-06-01 /user/directory_impala/table/partition
You could follow these links to the Cloudera documentation for further details: