Saturday, May 11, 2013

Usual Errors encountered when using HBase

Error 1: Error on the hbase shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/jruby/Main
Caused by: java.lang.ClassNotFoundException: org.jruby.Main
      at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: org.jruby.Main. Program will exit.

Solution: This is most likely a version incompatibility issue, which I had encountered when trying to run hbase-0.20.4 with hadoop 0.20.1
The same setup of Hadoop was working fine with hbase-0.20.1, hbase-0.20.2 and hbase-0.20.3. Hence try with some other version of either Hadoop or HBase.

Error 2: Error on the hbase shell
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/zookeeper/ZKServerTool
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.zookeeper.ZKServerTool
      at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: org.apache.hadoop.hbase.zookeeper.ZKServerTool. Program will exit.
 
Solution: Same as that for Error 1 above

Error 3: Error in the master logs
ERROR org.apache.hadoop.hbase.master.HMaster: Cannot start master
Caused by: java.net.ConnectException: Call to hadoop-4252-nameserver/192.168.41.123:9000 failed on connection exception: java.net.ConnectException: Connection refused

Solution: When getting the above error, my /etc/hosts file on the HBase Master node was something like below:

127.0.0.1   hbase-master      localhost.localdomain   localhost
::1   hbase-master      localhost6.localdomain6 localhost6
192.168.41.123  hbase-master
192.168.41.123  localhost
192.168.41.123  localhost.localdomain
And to resolve the error, I edited it to be

192.168.41.123  hbase-master
192.168.41.123  localhost
192.168.41.123  localhost.localdomain

After which the above error went away.

Error 4: Error in the master logs
ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
      at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1105)
      at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:142)
      at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:102)
      at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
      at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
      at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1119)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
      at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
      at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:37)
      at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:34)
      at org.apache.hadoop.security.UgiInstrumentation.create(UgiInstrumentation.java:51)
      at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:216)
      at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:184)
      at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:236)
      at org.apache.hadoop.security.KerberosName.<clinit>(KerberosName.java:79)
      at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:209)
      at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:184)
      at org.apache.hadoop.security.UserGroupInformation.isSecurityEnabled(UserGroupInformation.java:236)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
      at org.apache.hadoop.hbase.security.User.call(User.java:457)
      at org.apache.hadoop.hbase.security.User.callStatic(User.java:447)
      at org.apache.hadoop.hbase.security.User.access$200(User.java:49)
      at org.apache.hadoop.hbase.security.User$SecureHadoopUser.isSecurityEnabled(User.java:435)
      at org.apache.hadoop.hbase.security.User$SecureHadoopUser.login(User.java:406)
      at org.apache.hadoop.hbase.security.User.login(User.java:146)
      at org.apache.hadoop.hbase.master.HMaster.<init>(HMaster.java:204)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
      at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
      at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
      at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1100)
      ... 5 more

Solution: Remove the commons-configuration-*.jar(if present) from your hbase setup lib folder and replace with the commons-configuration-*.jar of the hadoop being used.
 
Error 5: Error in the master logs
java.io.IOException: Call to /10.0.1.37:50070 failed on local exception: java.io.EOFException at org.apache.hadoop.ipc.Client.wrapException(Client.java:1139) at org.apache.hadoop.ipc.Client.call(Client.java:1107) at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:226) ....

Solution: Remove the hadoop-core-*.jar from your hbase setup lib folder and replace with the hadoop-core jar of the hadoop being used.
 
Error 6: Error in the master logs
FATAL org.apache.hadoop.hbase.master.HMaster: Unhandled exception. Starting shutdown.
java.lang.IllegalArgumentException: � 13955@hbase-masterVMENABLED
      at org.apache.hadoop.hbase.zookeeper.ZKTable.getTableState(ZKTable.java:119)
      at org.apache.hadoop.hbase.zookeeper.ZKTable.populateTableStates(ZKTable.java:94)
      at org.apache.hadoop.hbase.zookeeper.ZKTable.<init>(ZKTable.java:79)
      at org.apache.hadoop.hbase.master.AssignmentManager.<init>(AssignmentManager.java:171)
      at org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:353)
      at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:281)
2012-11-15 19:03:12,275 INFO org.apache.hadoop.hbase.master.HMaster: Aborting
2012-11-15 19:03:12,277 DEBUG org.apache.hadoop.hbase.master.HMaster: Stopping service threads
2012-11-15 19:03:12,279 INFO org.apache.hadoop.ipc.HBaseServer: Stopping server on 60000
2012-11-15 19:03:12,292 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 0 on 60000: exiting
2012-11-15 19:03:12,296 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 1 on 60000: exiting
2012-11-15 19:03:12,301 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 2 on 60000: exiting
2012-11-15 19:03:12,303 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 3 on 60000: exiting
2012-11-15 19:03:12,306 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 4 on 60000: exiting
2012-11-15 19:03:12,308 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 5 on 60000: exiting
2012-11-15 19:03:12,315 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 6 on 60000: exiting
2012-11-15 19:03:12,318 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 7 on 60000: exiting
2012-11-15 19:03:12,321 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 8 on 60000: exiting
2012-11-15 19:03:12,322 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server handler 9 on 60000: exiting
2012-11-15 19:03:12,329 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server listener on 60000
2012-11-15 19:03:12,361 INFO org.apache.hadoop.ipc.HBaseServer: Stopping IPC Server Responder
2012-11-15 19:03:12,471 DEBUG org.apache.hadoop.hbase.catalog.CatalogTracker: Stopping catalog tracker org.apache.hadoop.hbase.catalog.CatalogTracker@73305c
2012-11-15 19:03:12,472 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to hconnection-0x13b0445c6c90001 has been closed.
2012-11-15 19:03:12,473 INFO org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: Closed zookeeper sessionid=0x13b0445c6c90001
2012-11-15 19:03:12,498 INFO org.apache.zookeeper.ZooKeeper: Session: 0x13b0445c6c90001 closed
2012-11-15 19:03:12,499 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation: The connection to null has been closed.
2012-11-15 19:03:12,499 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down
2012-11-15 19:03:12,532 INFO org.apache.zookeeper.ZooKeeper: Session: 0x13b0445c6c90000 closed
2012-11-15 19:03:12,532 INFO org.apache.hadoop.hbase.master.HMaster: HMaster main thread exiting
2012-11-15 19:03:12,533 INFO org.apache.zookeeper.ClientCnxn: EventThread shut down

Solution: Remove the commons-lang-*.jar from your hbase setup lib folder and replace with the commons-lang-*.jar of the hadoop being used.
 
Posted by at
Labels:

12 comments:

  1. UnknownDecember 2, 2013 at 10:37 AM

    Greetings,
    I have installed Hadoop + HBase 6 node cluster with 1 master and 5 slaves/regionservers. However, mine setup is not working properly and I can not understnd why. I have installed Hadoop and I have all its's processes running. After I setup HBase and run /start-hbase.sh it says that all nodes have started but when I run "jps" I only have Hadoop processes running and nothing from HBase. After, when I try to use shell I get : "ERROR: org.apache.hadoop.hbase.MasterNotRunningException: Retried 7 times"
    Also, when I try to /stop-hbase.sh I get "localhost: no zookeeper to stop because kill -0 of pid 22876 failed with status 1".
    On logs I have: ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
    java.lang.RuntimeException: HMaster Aborted
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:153)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:103)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:1668)

    Any ideas?
    Thanks :)

    ReplyDelete
    Replies
    1. JayatiDecember 2, 2013 at 11:11 PM

      Hello,

      Clearly the HBase Master is not starting and hence all the errors.

      Could you check if the port on which HBase Master runs is not already occupied?

      Because with the given information, its not clear what's the issue which is aborting the master.

      Jayati

      Delete
  2. UnknownMarch 4, 2014 at 7:14 PM

    hi,

    when i load 1T data in Hbase(standalone), hbase aborted and show below error:

    ERROR org.apache.hadoop.hbase.master.HMasterCommandLine: Failed to start master
    java.lang.RuntimeException: HMaster Aborted
    at org.apache.hadoop.hbase.master.HMasterCommandLine.waitOnMasterThreads(HMasterCommandLine.java:201)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:150)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:104)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:76)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2120)
    2014-03-04 08:29:10,994 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=Thread[Thread-67,5,main]
    2014-03-04 08:29:10,994 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: STOPPED: Shutdown hook


    2014-03-04 08:29:10,994 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Starting fs shutdown hook thread.
    2014-03-04 08:29:10,995 INFO org.apache.hadoop.hbase.regionserver.ShutdownHook: Shutdown hook finished.



    Any ideas?
    Thanks

    ReplyDelete
  3. UnknownApril 8, 2014 at 10:44 PM

    hi,
    I want to use yarn + hbase + hive together but i m facing version incompatibility problem.
    Can u suggest me which version of Hbase and hive i should use with yarn-2.2.0.

    Thanks

    ReplyDelete
  4. varunSeptember 3, 2014 at 11:35 PM

    hi,
    i am new in hbase , i am using hadoop-2.5.0 with hbase-0.98.5 version both are stable version when i started hbase start-hbase.sh then i got error like

    Error: Could not find or load main class org.apache.hadoop.hbase.util.HBaseConfTool
    Error: Could not find or load main class org.apache.hadoop.hbase.zookeeper.ZKServerTool
    starting master, logging to /usr/lib/hbase/hbase-0.98.5/logs/hbase-root-master-Varun.out
    Error: Could not find or load main class org.apache.hadoop.hbase.master.HMaster
    hadoop-slave-1: starting regionserver, logging to /usr/lib/hbase/hbase-0.98.5/logs/hbase-root-regionserver-Root.out
    hadoop-master: starting regionserver, logging to /usr/lib/hbase/hbase-0.98.5/logs/hbase-root-regionserver-Varun.out
    hadoop-slave-1: Error: Could not find or load main class org.apache.hadoop.hbase.regionserver.HRegionServer
    hadoop-master: Error: Could not find or load main class org.apache.hadoop.hbase.regionserver.HRegionServer

    my hbase-env.sh is:

    # The java implementation to use. Java 1.6 required.
    export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.9.x86_64

    export HADOOP_CONF_DIR=/opt/hadoop/hadoop/etc/hadoop

    export HBASE_HEAPSIZE=4096

    export HBASE_MANAGES_ZK=true

    my region server is :

    hadoop-master
    hadoop-slave-1

    and hbase-site.xml is :


    hbase.rootdir
    hdfs://hadoop-master:9000/hbase


    　　hbase.cluster.distributed
    　　true

    　
    　　hbase.zookeeper.quorum
    　　10.0.0.133,10.0.0.126


    hbase.zookeeper.property.dataDir
    /home/hduser/hbase/zookeeper


    　　hbase.master
    　　hdfs://hadoop-master:60000


    please help me its urgent please ,please help me

    ReplyDelete
    Replies
    1. AnonymousJanuary 14, 2015 at 12:06 AM

      Facing the same issues, Varun, were you able to solve it?

      Delete
  5. ShashJune 14, 2015 at 5:13 AM

    Installed Hadoop-2.6.0 and Hbase-1.0.1.1 on multi node cluster. Region servers are not starting. Getting the following error:

    java.lang.RuntimeException: Failed construction of Regionserver: class org.apache.hadoop.hbase.regionserver.HRegionServer
    at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2496)
    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:64)
    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2511)
    Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.constructRegionServer(HRegionServer.java:2494)
    ... 5 more
    Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/HTraceConfiguration
    at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:526)
    ... 10 more
    Caused by: java.lang.ClassNotFoundException: org.apache.htrace.HTraceConfiguration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:358)

    Any help is appreciated. Thanks.

    ReplyDelete
  6. UnknownAugust 14, 2015 at 12:08 AM

    Hi Jayati,
    Kudos for your blog, i keep following all your contents, its really helpful...i have solved many issues after finding a solution from your blog.

    I am facing the Hbase issue, Failed construction of Master: class, i have tried the solution which you have provided above, but still i get this.

    Hbase is not starting at the first run.. i am trying to install hbase from cloudera manager.


    below is the stacktrace of the issue,

    ERROR org.apache.hadoop.hbase.master.HMasterCommandLine
    Master exiting
    java.lang.RuntimeException: Failed construction of Master: class org.apache.hadoop.hbase.master.HMaster
    at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2049)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:198)
    at org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:139)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
    at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:2063)
    Caused by: java.net.BindException: Address already in use
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.apache.hadoop.hbase.ipc.RpcServer.bind(RpcServer.java:2373)
    at org.apache.hadoop.hbase.ipc.RpcServer$Listener.(RpcServer.java:528)
    at org.apache.hadoop.hbase.ipc.RpcServer.(RpcServer.java:1903)
    at org.apache.hadoop.hbase.regionserver.RSRpcServices.(RSRpcServices.java:821)
    at org.apache.hadoop.hbase.master.MasterRpcServices.(MasterRpcServices.java:204)
    at org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:494)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:498)
    at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:347)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2044)
    ... 5 more

    ReplyDelete
  7. SaurabhDecember 11, 2015 at 4:57 AM

    Harshawardhan Kulkarni did you find any solution for your problem?

    ReplyDelete
  8. Venkatarami ReddyMay 24, 2016 at 4:29 AM

    Hi Jayathi,

    Please give me solution on this error

    ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
    java.lang.RuntimeException: HRegionServer Aborted
    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:68)
    at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:126)
    at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2651)

    ReplyDelete
  9. Venkatarami ReddyMay 26, 2016 at 5:27 AM

    Hi Jayathi,
    Pls help on that error.

    ReplyDelete
  10. Intern4January 10, 2023 at 7:22 AM

    Hello Jayati, thanks for this bog post. A nice one. Business Analytics courses in Pune

    ReplyDelete

Subscribe to: Post Comments (Atom)