Convert an array into a Map

Viewed 3136

I have a table with a column like

[{"key":"e","value":["253","203","204"]},{"key":"st","value":["mi"]},{"key":"k2","value":["1","2"]}]

Which is of the format array<struct<key:string,value:array<string>>>

I want to convert the column into below format :

{"e":["253","203","204"],"st":["mi"],"k2":["1","2"]}

which is of the type map<string,array<string>>

I have tried exploding the array but that does not work. Any ideas how I can do this in hive.

3 Answers

Without use of external libraries it's impossible. Please refer to brickhouse or create your own UDAF.


Note: further code provides snippets to reproduce the problem and solving the problem that Hive's built-in functions can solve. i.e map<string,string> not map<string, array<string>>.

-- reproducing the problem
CREATE TABLE test_table(id INT, input ARRAY<STRUCT<key:STRING,value:ARRAY<STRING>>>);
INSERT INTO TABLE test_table 
SELECT 
    1 AS id,
    ARRAY(
        named_struct("key","e",  "value", ARRAY("253","203","204")),
        named_struct("key","st", "value", ARRAY("mi")),
        named_struct("key","k2", "value", ARRAY("1", "2"))
    ) AS input;

SELECT id, input FROM test_table;
+-----+-------------------------------------------------------------------------------------------------------+--+
| id  |                                                 input                                                 |
+-----+-------------------------------------------------------------------------------------------------------+--+
| 1   | [{"key":"e","value":["253","203","204"]},{"key":"st","value":["mi"]},{"key":"k2","value":["1","2"]}]  |
+-----+-------------------------------------------------------------------------------------------------------+--+

With exploding and using STRUCT features, we can split the keys and values.

SELECT id, exploded_input.key, exploded_input.value
FROM (
    SELECT id, exploded_input
    FROM test_table LATERAL VIEW explode(input) d AS exploded_input
) x;

+-----+------+----------------------+--+
| id  | key  |        value         |
+-----+------+----------------------+--+
| 1   | e    | ["253","203","204"]  |
| 1   | st   | ["mi"]               |
| 1   | k2   | ["1","2"]            |
+-----+------+----------------------+--+

The idea is to use your UDAF to "collect" a map while aggregating on id.


What Hive can solve with built in function is generating map<string,string> by converting rows to strings with a special delimiter, aggregate rows via another special delimiter and use str_to_map built-in function on the delimiters to generate map<string, string>.

SELECT
    id,
    str_to_map(

        -- outputs:  e:253,203,204#st:mi#k2:1,2 with delimiters between aggregated rows
        concat_ws('#', collect_list(list_to_string)), 
        '#', -- first delimiter
        ':'  -- second delimiter
    )  mapped_output
FROM ( 
    SELECT 
        id, 
        -- outputs 3 rows: (e:253,203,203), (st:mi), (k2:1,2)
        CONCAT(exploded_input.key,':' , CONCAT_WS(',', exploded_input.value)) as list_to_string
    FROM (
        SELECT id, exploded_input
        FROM test_table LATERAL VIEW explode(input) d AS exploded_input
    ) x
) y
GROUP BY id;

Which outputs a string to string map like:

+-----+-------------------------------------------+--+
| id  |               mapped_output               |
+-----+-------------------------------------------+--+
| 1   | {"e":"253,203,204","st":"mi","k2":"1,2"}  |
+-----+-------------------------------------------+--+
with input_set as (
select array(named_struct('key','e','value',array('253','203','204')),named_struct('key','st','value',array('mi')),named_struct('key','k2','value',array('1','2'))) as input_array
), break_input_set as (
select y.col_num as y_col_num,y.col_value as y_col_value from input_set lateral view posexplode(input_set.input_array) y as col_num, col_value
), create_map as (
select map(y_col_value.key,y_col_value.value) as final_map from break_input_set
)
select * from create_map;
var Array = [{"key":"e","value":["253","203","204"]},{"key":"st","value":["mi"]},{"key":"k2","value":["1","2"]}];

var obj = {}
for(var i=0;i<Array.length;i++){
  obj[Array[i].key] = Array[i].value
}

obj will be in the required format

Related