MAServer will no longer boot with Kernal Panic

[generatorlabs]

Here is my situation very briefly.
MAServer was running but has a noisy WD hard drive. Figured I would be proactive and clone the drive before it got worse. Server is a Dell Optiplex workstation.
I clone WD harddrive with Clonezilla, copying the entire drive and boot sectors. I removed the cloned drive and figured I would just hang on to it in case the WD failed.
Returned everything back to the original way it was set up and proceeded to boot.
Then I get the following while booting:

md: created md126
md: bind<sda1>
md: running: <sda1>
raid1: raid set md126 active with 1 out of 2 mirrors
md: autorun DONE.
Scanning logical volumes
   Reading all physical volumes.  This may take a while...
Activating logical volumes
   Volume group "main" not found
Creating root device.
Mounting root filesystem.
mount: could not find filesystem '/dev/root'
Setting up other filesystems.
Setting up new root fs
setuproot: moving /dev failed: No such file or directory
no fstab.sys, mounting internal defaults
setuproot: error mounting /proc: No such file or directory
setuproot: error mounting /sys: No such file or directory
Switching to new root and running init.
unmounting old /dev
unmounting old /proc
unmounting old /sys
switchroot: mount failed: No such file or directory
Kernal panic - not syncing: Attempted to kill init!

What in the world?!?! I swear I did nothing to the original HD image. Even the cloned drive boots the same way. Can someone guide me to what is going on here?
My gut feeling tells me the Kernal Panic is not the problem but rather a symptom of something happening much earlier.

Thanks

[sarond]

Did you take a backup from the server-manager? Probably should have done that before a clone of the drive.

This sound as if it could be an LVM problem but I'm not sure.

[generatorlabs]

Did you take a backup from the server-manager? Probably should have done that before a clone of the drive.

I did not do a backup from server manager.
I was thinking "what backup is better than a complete clone".
At least that is the way I see it when backing up virtual machines.
I don't know what happened on this one though.
I am gonna try and mount the hard drive in another Linux box and see what I can see on the drive.

[acejavelin]

I'm confused... The first post says

I removed the cloned drive and figured I would just hang on to it in case the WD failed.
Returned everything back to the original way it was set up and proceeded to boot.

Is the original drive in the MSL server or the cloned one? Did you try the cloned one?

I imagine the cloned one would fail because it's UUID of the clone is different than the original, and the fstab file would not be able to find any of the partitions to mount since they are all based on the UUID of the drive. If you are really good with Linux and have a bootable live distrobution you MIGHT be able to fix this on the cloned copy.

You likely need to reinstall and restore from a backup... when was your last backup done?

[johnp]

I think that a back up, fresh install, clear hardware id, restore is the route i would have done.

[generatorlabs]

Is the original drive in the MSL server or the cloned one? Did you try the cloned one?

The original drive was in the server when I first saw the Kernal Panic.
So then I tried the cloned drive, thinking the original drive might have failed.
They both produce the same failure.

I decided to poke around with a G-Parted Live CD I had lying around. With the original drive installed I see:
/dev/sda of 232.83GiB
/dev/sda1 of 101.94 MiB, Marked active as "boot"
/dev/sda2 of 232.73 GiB, Marked active.

If I boot into an Ubuntu Live CD I see the same thing but I can also see that sda1 is EXT3 and sda2 is another type of file system which I am not familiar. It is something like LVMPS (it slips my mind, I am not near the machine right now)

That gives me a feeling that the drive is responding properly but if I try to mount the partition in the Ubuntu Live environment it says it is an unknown filesystem type.
I am not an expert in Linux but I know enough to hurt myself.

Do you think I should be able to mount the EXT3 partition and edit the UUID that Grub may be referencing?
Can I use any Linux Live distribution to edit Grub?
The (C)ommand Line option offered inside Grub does not seem user friendly at all and I don't think it can do much to help me.
At this point I will try anything. I do not have a regular backup to fall back on so my only other option is to rebuild the whole thing. Hindsight is 20/20.

[sarond]

Are you an end user or a dealer?

How much config is there? It may be easier to just rebuild depending on what the MAS was used for.

What version was it?

Although the partition type is standard ext3 not LVM does it still have the data in there?

As a long shot you could try a new install to get the partitioning correct and then copy the files over the top from the source. (I think it's normally /etc /home /root /var that is backed up) Sounds like you have nothing to lose. Maybe just back up the files to a separate drive before stuffing around too much.

Try and keep the original drive untouched as someone here might be able to talk you through recovering.

[generatorlabs]

Okay I got it back up and running.
I took the server over to my buddies place. He is an old time Linux head who knows more than most have forgotten.
Anyway booted the server up with a USB Ubuntu Live thumb drive.
We were able to see some partition information but not really access or mount them.
He installed some MSADM tools and voila he was able to mount the partitions and we were able to see the data.
We then copied it over to another drive as a fail safe measure.
Now we had the opportunity to get messy.
He realized there was something screwy with the RamDisk files.
I reinstalled the base MAS software to another empty hard drive.
We now had the option or dropping the backed up files onto the newly installed image or taking the working RamDisk files and dropping them back on the drive that was not booting. We went with the second option and poof it started up and all systems are GO!
Now I can do a backup thru the console which I intend to do quickly.

Anyway, thanks for any input. Acejavelin led me to reach out to a Linux-head as most of these advanced commands are something I have not seen before.

[generatorlabs]

copy the files over the top from the source. (I think it's normally /etc /home /root /var that is backed up) Sounds like you have nothing to lose.

Now that I have the server working I quickly performed a backup. This does raise one additional question. I performed a backup via "Backup to Desktop".
I downloaded the file 'smeserver.tgz' and noticed it was not very large, just over 17.1 MB.
Peeking into the compressed files I can see the /etc, /home, & /root folders, with /home being the largest.

I was expecting this to be much larger.
What exactly is being backed up? Is the auto-attendant message tree and all voice mails included in that backup? I have never had to do a restore from a file like this so I am not too familiar with what happens behind the scenes. I was imagining the backup would be so much larger. We don't have a lot of users but I would think most of our users have a bunch of undeleted v-mails sitting in their box.

For a comparison I tried to do the same backup at the the server via a USB hard drive. It said it could not mount the hard drive. I had the hard drive formatted to EXT4 and it mounts fine on another Linux box. Is the console expecting to see something smaller like a thumb drive? Do I have to format it in any particular way?

Thanks again!

[johnp]

I took a quick look at a recent customer backup. It appears that the applications get backed up in the /home and /etc is passwords, /root is what ever is in /root. I myself create a tree in route for any modifications that might have been done so that I can review it. Make like and original directory and a modified and copy the changes or original flies to each as named.