How to get rid of the warning “util.NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable”

If you installed Apache Hadoop on a 64bit operating system from a precompiled package , you will see this warning message whenever you run a Hadoop command (Hadoop from Apache, I did not see the warning in Cloudera):

util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

It happens when you stop or start Hadoop, use hdfs, hive or any other Hadoop related tool. It is not a critical situation and everything will still work using java libraries, but with some performance degradation.

The problem is with the shared library that is located in $HADOOP_HOME/lib/native

It comes precompiled for 32bit and hence cannot be used with 64bit OS.

To fix this we first have to install some prerequisite packages:

yum install build-essential autoconf automake libtool cmake zlib-devel openssl-devel gcc-c++ glibc-headers svn git

Then install maven:

wget -O /etc/yum.repos.d/epel-apache-maven.repo

yum install apache-maven

And finally install google protocol buffers.
Download protocol buffers version 2.5.0 source package from here and Unzip or untar it on your hosts (you specifically need version 2.5 although newer versions exist).

You should run the following commands as root:

cd protobuf-2.5.0



Make install


After all prerequisites are met we can download the Hadoop source from here. Make sure you download the same Hadoop version that you already have installed (2.7 in my case).

Unzip the source file, enter the unzipped directory and run:

mvn package -Pdist,native -DskipTests
This can take a while to complete, depending on your computer hardware and network. If everything goes well you will get this message:

[INFO] ------------------------------------------------------------------------
[INFO] Total time: 27:20 min
[INFO] Finished at: 2016-05-17T10:51:54+03:00
[INFO] Final Memory: 213M/837M
[INFO] ------------------------------------------------------------------------


Under the source directory, go to ./hadoop-dist/target/hadoop-2.7.1/lib/native and find (the exact patch may change according to your Hadoop version).
Copy the file to $HADOOP_HOME/lib/native (overwrite existing file).

Now start Hadoop, the warning disappeared !

You can now cleanup the Hadoop and protobuf sources.



This entry was posted in Hadoop and tagged , . Bookmark the permalink.

Leave a Reply