Winutils is mapped as hadoop home when I write spark applications which does stuff on hadoop. But just wanted to understand how winutils makes this possible?
1 Answers
Spark does not require a Hadoop cluster installation, it can be installed locally, but it is required to use an element of the Hadoop code base known as ‘WinUtils’. This allows management of the POSIX file system permissions that the HDFS file system requires of the local file system. More details here.
Spark requires that you have set POSIX compatible permissions for a temporary directory used by the Hive metastore, which defaults to C:\tmp\hive
WinUtils provides POSIX chmod along with other POSIX commands
It seems it is not a big deal though, possible to remove dependency on winutils. Read this article: Apache Spark on Windows without winutils.exe - this can not be used as a permanent solution because of hardcoded fix, for example:
private void loadPermissionInfo() {
IOException e = null;
try {
/* String output = FileUtil.execCommand(new File(getPath().toUri()),
Shell.getGetPermissionCommand());*/
// Let put it hard instead of calling winutils.exe
String output = "drwx-wx-wx|1|xxxxx|yyyyyyy|0|Jun|21|2018|"+getPath();
but that code will definitely give you better idea about what winutils is used for.
Related