one of the SuSE 12 LinuxServers has reported disk failure. Fortunately the Database Server has Software Raid hence the system is still up and running. But as recommended, we would like to replace the failed disk with a new one and rebuild the software raid on it.
System Information is :
Total 4 Internal Disks. sda, sdb , sdc and sdd
The fdisk partitions are :
fdisk -l
Disk /dev/sdb: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x0007d757
Device Boot Start End Sectors Size Id Type
/dev/sdb1 * 2048 2105343 2103296 1G fd Linux raid autodetect
/dev/sdb2 2105344 39858175 37752832 18G fd Linux raid autodetect
/dev/sdb3 39858176 1953523711 1913665536 912.5G fd Linux raid autodetect
Disk /dev/sdc: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x000a0e8a
Device Boot Start End Sectors Size Id Type
/dev/sdc1 2048 1953523711 1953521664 931.5G fd Linux raid autodetect
Disk /dev/sdd: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: dos
Disk identifier: 0x000caaad
Device Boot Start End Sectors Size Id Type
/dev/sdd1 2048 1953523711 1953521664 931.5G fd Linux raid autodetect
Software RAID --> sda + sdb (sda is failed disk) Software RAID --> sdb + sdc
DBServer# cat /proc/mdstat
Personalities : [raid1]
md3 : active raid1 sdc1[0] sdd1[1] 976760640 blocks super 1.0 [2/2] [UU] bitmap: 2/8 pages [8KB], 65536KB chunk
md0 : active raid1 sdb1[1] sda10 1051584 blocks super 1.0 [2/1] [_U] bitmap: 1/1 pages [4KB], 65536KB chunk
md2 : active raid1 sdb3[1] sda30 956832576 blocks super 1.0 [2/1] [_U] bitmap: 2/8 pages [8KB], 65536KB chunk
md1 : active raid1 sdb2[1] sda20 18876288 blocks super 1.0 [2/1] [_U] bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices:
So md0,md1 and md2 have failed devices namely sda1,sda2 and sda3
Please note that it also has 2 VGs defined as shown below,
1 VG - system (/dev/md2) 2 VG - ora_db (/dev/md3)
pvdisplay
--- Physical volume ---
PV Name /dev/md3
VG Name ora_db
PV Size 931.51 GiB / not usable 3.81 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 238466
Free PE 84866
Allocated PE 153600
PV UUID vgPdWQ-x6CW-vvdF-moxh-FKyb-wpSU-NdJqSm
--- Physical volume ---
PV Name /dev/md2
VG Name system
PV Size 912.51 GiB / not usable 2.81 MiB
Allocatable yes
PE Size 4.00 MiB
Total PE 233601
Free PE 182401
Allocated PE 51200
PV UUID rdff2n-ztxd-lcBY-nAqk-8O9u-fnFG-BVI91v
The grub.conf shows : (Relevant part)
if [ x$feature_default_font_path = xy ] ; then
font=unicode
else
insmod part_msdos msdos
insmod diskfilter mdraid1x lvm
insmod ext2
set root='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c
else
search --no-floppy --fs-uuid --set=root 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c
fi
font="/share/grub2/unicode.pf2"
fi
if loadfont $font ; then
set gfxmode=auto
load_video
insmod gfxterm
set locale_dir=$prefix/locale
set lang=POSIX
insmod gettext
fi
terminal_output gfxterm
insmod part_msdos msdos
insmod diskfilter mdraid1x
insmod ext2
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526
else
search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
fi
Quote: The procedure to follow should go like this,
First we mark /dev/sda1 as failed:
mdadm --manage /dev/md0 --fail /dev/sda1
Then we remove /dev/sda1 from /dev/md0:
mdadm --manage /dev/md0 --remove /dev/sda1
Now we do the same steps again for /dev/sda2 and sda3 (which is part of /dev/md1 and /dev/md2)
Then power down the system:
shutdown -h now
and replace the old /dev/sdb hard drive with a new one
After inserting new SATA disk /dev/sda, boot the system.
Then we create the exact same partitioning as on /dev/sda. We can do this with one simple command:
sfdisk -d /dev/sdb | sfdisk /dev/sdaCheck if both the disks have same partitions (fdisk -l) 8.Next we add /dev/sda1 to /dev/md0 and /dev/sda2 to /dev/md1 and /dev/sda3 to /dev/md3:
mdadm --manage /dev/md0 --add /dev/sda1 mdadm --manage /dev/md1 --add /dev/sda2 mdadm --manage /dev/md2 --add /dev/sda3
Confirm the synchronisation in progress
cat /proc/mdstat
Please let me know if I have missed something. 2 important points I guess would be, how should I take care of lvm and grub in this case.
Do I have to do something extra to take care of it or the command sfdisk -d /dev/sdb | sfdisk /dev/sda , should take care of LVM as well.
How should I take care of grub in this case? As grun.conf shows entries pertaining to LVM as well as MDADM. Do I have to change anything here before I shutdown the system?
I understand the system has 2 pointers to take care of mdadm+lvm. Which have complicated things. Else would it be easier to setup completely new system?
Not yet tried anything. Only collecting information.
grub.cfg
### BEGIN /etc/grub.d/00_header ###
if [ -f ${config_directory}/grubenv ]; then
load_env -f ${config_directory}/grubenv
elif [ -s $prefix/grubenv ]; then
load_env
fi
if [ "${env_block}" ] ; then
load_env -f "${env_block}"
fi
if [ "${next_entry}" ] ; then
set default="${next_entry}"
set next_entry=
save_env next_entry
if [ "${env_block}" ] ; then
save_env -f "${env_block}" next_entry
fi
set boot_once=true
else
set default="${saved_entry}"
fi
if [ x"${feature_menuentry_id}" = xy ]; then
menuentry_id_option="--id"
else
menuentry_id_option=""
fi
export menuentry_id_option
if [ "${prev_saved_entry}" ]; then
set saved_entry="${prev_saved_entry}"
save_env saved_entry
set prev_saved_entry=
save_env prev_saved_entry
set boot_once=true
fi
function savedefault {
if [ -z "${boot_once}" ]; then
saved_entry="${chosen}"
save_env saved_entry
fi
}
function load_video {
if [ x$feature_all_video_module = xy ]; then
insmod all_video
else
insmod efi_gop
insmod efi_uga
insmod ieee1275_fb
insmod vbe
insmod vga
insmod video_bochs
insmod video_cirrus
fi
}
if [ x$feature_default_font_path = xy ] ; then
font=unicode
else
insmod part_msdos msdos
insmod diskfilter mdraid1x lvm
insmod ext2
set root='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='lvmid/m7AEp0-79EG-D2Vi-ELzE-BTzh-C8mN-CLxrpz/S0eZEl-PlBX-E1ZL-oCwL-SmUx-4Qe4-Mz9NHX' 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c
else
search --no-floppy --fs-uuid --set=root 7c2e3a9c-5f5b-47e3-8a0a-d1e66f12747c
fi
font="/share/grub2/unicode.pf2"
fi
if loadfont $font ; then
set gfxmode=auto
load_video
insmod gfxterm
set locale_dir=$prefix/locale
set lang=POSIX
insmod gettext
fi
terminal_output gfxterm
insmod part_msdos msdos
insmod diskfilter mdraid1x
insmod ext2
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526
else
search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
fi
insmod gfxmenu
loadfont ($root)/grub2/themes/SLE/DejaVuSans-Bold14.pf2
loadfont ($root)/grub2/themes/SLE/DejaVuSans10.pf2
loadfont ($root)/grub2/themes/SLE/DejaVuSans12.pf2
loadfont ($root)/grub2/themes/SLE/ascii.pf2
insmod png
set theme=($root)/grub2/themes/SLE/theme.txt
export theme
if [ x${boot_once} = xtrue ]; then
set timeout=0
elif [ x$feature_timeout_style = xy ] ; then
set timeout_style=menu
set timeout=8
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
set timeout=8
fi
### END /etc/grub.d/00_header ###
### BEGIN /etc/grub.d/10_linux ###
menuentry 'SLES12' --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-690785da-f0f0-4250-b693-5a008acbba10' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_msdos msdos
insmod diskfilter mdraid1x
insmod ext2
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526
else
search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
fi
echo 'Loading Linux 3.12.28-4-default ...'
linux /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10 resume=/dev/md1 splash=silent quiet crashkernel=232M-:116M showopts
echo 'Loading initial ramdisk ...'
initrd /initrd-3.12.28-4-default
}
submenu 'Advanced options for SLES12' --hotkey=1 $menuentry_id_option 'gnulinux-advanced-690785da-f0f0-4250-b693-5a008acbba10' {
menuentry 'SLES12, with Linux 3.12.28-4-default' --hotkey=2 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-advanced-690785da-f0f0-4250-b693-
5a008acbba10' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_msdos msdos
insmod diskfilter mdraid1x
insmod ext2
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526
else
search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
fi
echo 'Loading Linux 3.12.28-4-default ...'
linux /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10 resume=/dev/md1 splash=silent quiet crashkernel=232M-:116M showopts
echo 'Loading initial ramdisk ...'
initrd /initrd-3.12.28-4-default
}
menuentry 'SLES12, with Linux 3.12.28-4-default (recovery mode)' --hotkey=3 --class sles12 --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-3.12.28-4-default-recovery-
690785da-f0f0-4250-b693-5a008acbba10' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_msdos msdos
insmod diskfilter mdraid1x
insmod ext2
set root='mduuid/531cd341e2c7d5a71c542ad04d9ea589'
if [ x$feature_platform_search_hint = xy ]; then
search --no-floppy --fs-uuid --set=root --hint='mduuid/531cd341e2c7d5a71c542ad04d9ea589' 96c11697-c3b7-4f11-90fc-3aef207db526
else
search --no-floppy --fs-uuid --set=root 96c11697-c3b7-4f11-90fc-3aef207db526
fi
echo 'Loading Linux 3.12.28-4-default ...'
linux /vmlinuz-3.12.28-4-default root=UUID=690785da-f0f0-4250-b693-5a008acbba10 showopts apm=off noresume edd=off powersaved=off nohz=off highres=off processor.max_cstate=1 nomodeset
x11failsafe crashkernel=232M-:116M
echo 'Loading initial ramdisk ...'
initrd /initrd-3.12.28-4-default
}
}
### END /etc/grub.d/10_linux ###
### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###
### BEGIN /etc/grub.d/20_memtest86+ ###
### END /etc/grub.d/20_memtest86+ ###
### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###
### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries. Simply type the
# menu entries you want to add after this comment. Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###
### BEGIN /etc/grub.d/41_custom ###
if [ -f ${config_directory}/custom.cfg ]; then
source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then
source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###
### BEGIN /etc/grub.d/80_suse_btrfs_snapshot ###
### END /etc/grub.d/80_suse_btrfs_snapshot ###
### BEGIN /etc/grub.d/90_persistent ###
### END /etc/grub.d/90_persistent ###
I did install grub on /dev/sdb. Please find the output.
# grub2-install /dev/sdb
Installing for i386-pc platform.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
error: cannot read `/dev/sda': Input/output error.
grub2-install: warning: Couldn't find physical volume `(null)'. Some modules may be missing from core image..
Installation finished. No error reported.
Not sure why it gave sda errors though.
And then cross checked with following command again.
# dd bs=512 count=1 if=/dev/sdb 2>/dev/null| strings
ZRr=
`|f
\|f1
GRUB
Geom
Hard Disk
Read
Error
