Unable to create a source for reading table error when trying to query a glue table

user_8066997 0
Viewed 689

I'm new to kinesis analytics studio using apache flink, basically I have a data stream with hundreds of records going through it, and I want to make some real-time basic analysis, so I went through this tutorial and created a table in glue using below code:

%flink.ssql(type=update)
CREATE TABLE active_users(
user_id varchar(120),
platform varchar(60),
event_time timestamp(3),
WATERMARK FOR event_time AS event_time - INTERVAL '5' SECOND
 )
 PARTITIONED BY (user_id)
 WITH (
'connector' = 'kinesis',
'stream' = 'stream-id',
'aws.region' = 'us-east-1',
'scan.stream.initpos' = 'LATEST',
'format' = 'json',
'json.timestamp-format.standard' = 'ISO-8601');

and the table was created successfully and I can see it in glue data catalogue, but when I try to query the table with a basic select statement

%flink.ssql(type=update)
SELECT * FROM active_users limit 10;

I get below error

Unable to create a source for reading table 'hive.stream-id.active_users'.

Table options are:

'aws.region'='us-east-1'
'connector'='kinesis'
'format'='json'
'json.timestamp-format.standard'='ISO-8601'
'scan.stream.initpos'='LATEST'
'stream'='stream-id'

I looked into the IAM policy and everything looks good, any thoughts on how to tackle this?.

amazon-kinesis-analytics amazon-kinesis apache-flink amazon-web-services sql
1 Answers

There are two things that I tried to solve that issue when I encountered.

  1. The versión of flink. I changed it to 1.11 and

  2. The IAM policies, have to add them manually:

     {
     "Sid": "VisualEditor00",
     "Effect": "Allow",
     "Action": [
         "kinesis:DescribeStream",
         "kinesis:PutRecord",
         "kinesis:PutRecords",
         "kinesis:GetShardIterator",
         "kinesis:GetRecords",
         "kinesis:ListShards",
         "kinesis:DescribeStreamSummary",
         "kinesis:RegisterStreamConsumer"
     ],
     "Resource": [
         "arn:aws:kinesis:eu-west-1:376596651109:stream/edwarinputstream"
     ]
 },

Once I did the above it ran properly.

user_80356690
Related