Apache Flink with AWS Kinesis Analytics

user_10346701 0
Viewed 723

The actual situation

I want to use Apache Flink with Kinesis Analytics. Like I don't have experience with Java and Maven and going to try to summary the steps I followed and the results.

  1. Use IntelliJ IDEA as IDE
  2. Test this example without any problems
  3. Change to AWS, following their instruction. The problems start in the step: Create and Compile the Apache Flink Streaming Java Code

The problems

If you want to use the Kinesis connector, you can not do it like other connector due to the ASL license and they do not deploy the artifact to the Maven central repository on Flink releases, so you will need to build the Kinesis connector by yourself download the source code for the connector from here Apache Flink, and install it to your local Maven repository.

After download it, you have build the module following this steps:

mvn clean install -Pinclude-kinesis -DskipTests

So I move to the unziped folder, and run the mvn command, with this error:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 50.670 s
[INFO] Finished at: 2018-12-27T14:35:13+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (default) on project flink-parent: Too many files with unapproved license: 2 See RAT report in: C:\Users\...\flink-master\target\rat.txt -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :flink-parent

Trying to find a solution, I tried what this user said here

mvn clean install -Pinclude-kinesis -DskipTests  -Drat.ignoreErrors=true package

But again, I had an error:

[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 01:15 min
[INFO] Finished at: 2018-12-27T14:40:47+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal on project flink-shaded-hadoop2: Could not resolve dependencies for project org.apache.flink:flink-shaded-hadoop2:jar:1.8-SNAPSHOT: Could not find artifact jdk.tools:jdk.tools:jar:1.6 at specified path C:\Program Files\Java\jdk-11/../lib/tools.jar -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :flink-shaded-hadoop2

The questions

  1. How should I build the connector module?
  2. How should I install it to my local Maven repository?
amazon-kinesis amazon-web-services maven intellij-idea java
2 Answers

After receiving similar errors, even after using the other answer's solution for checking out a release branch, I found a solution that works for me. According to the build guide in the Flink docs, you can also add the -Dfast option to speed up build time. In my case, this resolved the licensing errors:

mvn clean install -Pinclude-kinesis -DskipTests -Dfast
user_9705890

I think that the problem is that you're trying to build the "snapshot" build of the Flink connector, not the release build.

Since you're not familiar with Java and Maven, here's what that means: Maven differentiates between "snapshot" builds, which are currently under active development, and "release" builds. Normally a project only pushes its release builds to Maven Central. However, for long-running development, the "master" branch is often a snapshot.

The POM in your linked instructions references version 1.6.2, so that's what you should be building. Check out the tag release-1.6.2, and also make sure that you're reading the Flink documentation for that version (your link above is for the latest development version).

user_108394820
Related