How to change the temporary used directory by sqoop-import?

user_5556798 0
Viewed 813

I've been using a sqoop-import command like this:

sqoop import --connect jdbc:oracle:thin:@${machine}:${port}/${schema} --username ${user} --password ${pw} --table "${table}" --columns "${cols}" --where "${machine}" --m 1 --fields-terminated-by ';' --target-dir ${dir} --hive-table "${hive_table}"

It has worked properly until now, when I receive this message:

Picked up _JAVA_OPTIONS: -Djava.io.tmpdir=/data/tmp
Java HotSpot(TM) 64-Bit Server VM warning: Insufficient space for shared memory file:
   64215
Try using the -Djava.io.tmpdir= option to select an alternate temp location.

Indeed my /tmp is full:

/dev/mapper/system-lv_tmp 5136000 4875796 0 100% /tmp

I cannot delete files that are not mine in /tmp so i can't free up enough space there.

I've tried changing the _JAVA_OPTIONS to another directory but still get the same error.

Is there any way to make the command use a different directory for temporal files?

tmp sqoop hdfs hive java
2 Answers

This is caused by the Java property java.io.tmpdir and has nothing to do with Sqoop.

From the Java Docs:

The default temporary-file directory is specified by the system property java.io.tmpdir.

Sqoop is not setting the value for tmpdir. But you can override this value when running your sqoop import command. 

sqoop import -Djava.io.tmpdir=/path/to/new/tmp ...
user_73034470

Maybe your problem is the code generated for Sqoop, there are several code generation control arguments like:

enter image description here

For example, you could try this:

sqoop import \
--connect jdbc:oracle:thin:@${machine}:${port}/${schema} \
--username ${user} \
--password ${pw} \
--table "${table}" \
--columns "${cols}" \
--where "${machine}" \
--m 1 \
--fields-terminated-by ';' \
--target-dir ${dir} \
--hive-table "${hive_table}" \
--outdir /home/user/outdir_directory \
--bindir /home/user/bindir_directory

and the java files generated for Sqoop are going to go to {outdir_directory and bindir_directory} so you can delete these whenever you want.

user_85714980
Related