Hadoop on Mac in intelliJ IDEA setup

user_961143 0
Viewed 5682

Installed hadoop using brew, now want to run hadoop jobs in intelliJ IDEA. How to setup the environment and resolve dependencies?

hadoop mapreduce intellij-idea macos
1 Answers

After spending lot's of time, was finally able to run hadoop jobs from intelliJ IDEA. Hope this might save time for others.

brew upgrade hadoop

installed/updated hadoop in /usr/local/Cellar/hadoop/2.x.x Now, we will try to import dependency from this location in our intelliJ

  1. Launch IntelliJ and click on ‘Create New Project’
  2. Create a java project
  3. Go to File -> Project Structure -> Select Project Module in Modules -> Select Dependencies Tab -> Select ‘+’ in bottom and click Jars or Directories -> Import dependencies from Hadoop installation

enter image description here

  1. For importing dependencies, first select all folders inside ../hadoop- 2.x.x/share/hadoop/

enter image description here

Repeat Step 4 and import ../Hadoop-2.x.x/share/hadoop/common/lib folder

enter image description here

Hadoop environment is integrated with intelliJ Java Project. You can create new java classes to implement your drivers, mappers, reducers etc.

user_9611430
Related