AWS Kinesis Firehose: how to put multiple files containing JSONs using aws cli & bash

user_11699868 0
Viewed 1062

I have >100 files where each line is a json. It looks something like this (no commas & no []):

{"one":"one","two":{"tree":...}}
{"one":"one","two":{"tree":...}}
...
{"one":"one","two":{"tree":...}}

To be able to use aws firehose put-record-batch, file needs to be in the format:

[
  {
    "Data": blob
  },
  {
    "Data": blob
  },
  ...
]

I want to put all of these files to aws Firehose from terminal.

I'm looking to write a shell script that looks something like this:

for file in files
do
  aws firehose put-record-batch --delivery-stream-name <name> --records file://$file
done

So there're 2 questions:

  1. How to transform the files into the applicable format
  2. And, how to iterate through all the files
amazon-kinesis-firehose aws-cli amazon-web-services shell
1 Answers
for file in *.json;
do
    jq -s . "${file}" >${file}.tmp && mv ${file}.tmp $file    
done

This will read all the json file in the current directory and change it into the desired form and save to the file .

OR if you do not have jq here is alternate way using python's json module. 

for file in *.json;do
  while read line ; do 
      echo $line | python -m json.tool 
  done < ${file} |awk 'BEGIN{print "["}{print}END{print "]"}'
done
user_63096010
Related