Category: Linux

Recovering from Linux boot failure with fsck

A virtual Linux machine used for development was behaving unreliably.

Programs would fail to start and running program would freeze.

I decided to restart the system, however it failed to start up again.

 

It simply displayed this message:

BusyBox v1.22.1 (Ubuntu 1:1.22.0-15ubuntu1) built-in shell (ash)
Enter 'help' for a list of built-in commands.

(initramfs)

 

I tried to exit the shell, which lead to this message:

/dev/sda1 contains a file system with errors, check forced.
/dev/sda1:
Inodes that were part of a corrupted orphan linked list found.

/dev/sda1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
fsck exited with status code 4
The root filesystem on /dev/sda1 requires a manual fsck

 

File system corruption seemed to explain the reliability problems and the subsequent boot failure.

Followed the suggestion of running fsck with:

fsck /dev/sda1

 

Accepted all file system repairs suggested by fsck.

(Should have used the -y parameter)

 

Then rebooted the system with:

reboot

 

After the file system repairs the system booted and seemed fully functional.

Conclusion

If a Linux system fails to boot and only displays a BusyBox / initramfs prompt, try exiting the shell.

This may provide information about the actual problem.

Linux hang at boot due to authentication problems with Samba mount in /etc/fstab

A Raspberry Pi 3 had a problem after I had performed a number of system changes.

When booting it would hang for around 2 minutes showing:

systemd-hostnamed.service

 

I examined system messages from the boot process with:

dmesg

Part of the result was:

[   98.748722] CIFS VFS: Send error in SessSetup = -13
[   98.749092] CIFS VFS: cifs_mount failed w/return code = -13
[  125.427742] usb 1-1.5.2.3: reset high-speed USB device number 9 using dwc_otg
[  188.946712] Status code returned 0xc000006d NT_STATUS_LOGON_FAILURE

 

I remembered that I had experimented with various ways to mount a Samba share in: /etc/fstab

This didn’t work because of authentication problems and (as a side effect) it delayed startup significantly.

 

I decided to solve the problem by:

1. Removing the non-functional line from /etc/fstab

2. Creating a script to be run on demand:

mount_RPServer_shared.sh

With content similar to this:

#/bin/bash
sudo mount -t cifs -o user=pi //server/shared /mnt/server_shared/

 

The script relies on the mount.cifs command.

For Debian based Linux distributions this is part of the cifs-utils package.

Linux problem: package is in a very bad inconsistent state

While updating a newly installed Raspberry Pi 2 with Raspbian Linux, it seemed to hang while upgrading with:

sudo apt-get upgrade

 

After 10 minutes of inactivity I decided to reset the system.

As expected this caused problems.

 

After running:

sudo apt-get update

I noticed this error:

E: dpkg was interrupted, you must manually run 'sudo dpkg --configure -a' to correct the problem.

 

I tried running the suggested command:

sudo dpkg --configure -a

It fixed some problems, but it left me with:

dpkg: error processing package ntp (--configure):
package is in a very bad inconsistent state; you should
reinstall it before attempting configuration
Errors were encountered while processing:
ntp

 

Various ways to reinstall and remove the ntp package were unsuccessful.

Eventually I found a solution here:

http://askubuntu.com/questions/148715/how-to-fix-package-is-in-a-very-bad-inconsistent-state-error

sudo dpkg --remove --force-remove-reinstreq ntp

sudo apt-get install ntp

 

Conclusion

If a package is broken on a Debian based Linux system it can be a solution to remove it forcefully with the –force-remove-reinstreq option for dpkg.