Can hive single orcfile'stripe be splited to smaller size?

user_13549539 0
Viewed 29

Example, an orcfile's block and stripesize are same, are 256M, but the split size is 100M. How does the orcfile be splitted?

mapper orc hive
1 Answers

ORC is splitted on stripe level, in your case 256M, smaller split is not possible.

Stripe size is configurable in TBLPROPERTIES("orc.stripe.size"="268435456"), split size is configurable in these settings (example for Tez):

set hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set tez.grouping.max-size=104857600; 
set tez.grouping.min-size=32000; --smaller files will be possibly be combined

In this case each mapper will receive whole stripes (256M) or less if there are smaller stripes, stripes will not be splitted on smaller chunks.

user_27003440
Related