hadoop cluster Unable to start standby Namenode

0 votes
we have HDP Hadoop cluster with two name-node services ( one active name-node and the secondary is the standby name-node )

due of unexpected electricity failure , the standby name-node failed to start with the flowing exception , while the active name-node starting successfully

    2024-02-02 08:47:11,497 INFO  common.Storage (Storage.java:tryLock(776)) - Lock on /hadoop/hdfs/namenode/in_use.lock acquired by nodename 36146@master1.delax.com
    2024-02-02 08:47:11,891 INFO  namenode.FSImage (FSImage.java:loadFSImageFile(745)) - Planning to load image: FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000052670667141, cpktTxId=0000000052670667141)
    2024-02-02 08:47:11,897 ERROR namenode.FSImage (FSImage.java:loadFSImage(693)) - Failed to load image from FSImageFile(file=/hadoop/hdfs/namenode/current/fsimage_0000000052670667141, cpktTxId=0000000052670667141)
    java.io.IOException: Premature EOF from inputStream
            at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:204)
            at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:221)
            at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:898)
            at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:882)
            at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:755)
            at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:686)
            at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:303)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:1077)
            at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:724)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:697)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:761)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:1001)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:985)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1710)
            at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1778)
    2024-02-02 08:47:12,238 WARN  namenode.FSNamesystem (FSNamesystem.java:loadFromDisk(726)) - Encountered exception loading fsimage
    java.io.IOException: Failed to load FSImage file, see error(s) above for more info.

we can see from above exception - `Failed to load image from FSImageFile` , and seems it is as results of when machine failed because unexpected shutdown

as I understand one of the options to recover the standby name-node could be with the following procedure:

1. Put Active NN in safemode

    sudo -u hdfs hdfs dfsadmin -safemode enter

2. Do a savenamespace operation on Active NN

    sudo -u hdfs hdfs dfsadmin -saveNamespace

3. Leave Safemode

    sudo -u hdfs hdfs dfsadmin -safemode leave

4. Login to Standby NN

5. Run below command on Standby namenode to get latest fsimage that we saved in above steps.

    sudo -u hdfs hdfs namenode -bootstrapStandby -force

we glad to receive any suggestions , or if my above suggestion is good enough for our problem
Feb 4, 2024 in Big Data Hadoop by anonymous

edited Mar 5 4 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP