Replacing a disk in a software RAID1 array in Linux

We have a server with two disks: /dev/sda and /dev/sdb. These disks are configured in a software RAID1 using mdadm. One of the disks has failed, in our case, it's /dev/sdb. 1. Removing the Disk from the Array Before replacing the disk, it's advisable to remove the failed disk from the array. First, let's check how the disk is partitioned in the array:

# cat /proc/mdstat
Personalities : [raid1] [raid0] [raid6] [raid5] [raid4]
md2: active raid1 sda4[0] sdb4[1]
      1456504640 blocks super 1.2 [2/2] [UU]

md1: active raid1 sda3[0] sdb3[1]
      7996352 blocks super 1.2 [2/2] [UU]

md0: active raid1 sda2[0] sdb2[1]
      499392 blocks super 1.2 [2/2] [UU]

unused devices: <none>

In this case, the array is set up with md0 consisting of sda2 and sdb2, md1 with sda3 and sdb3, and md2 with sda4 and sdb4. On this server, md0 is /boot, md1 is swap, and md2 is the root filesystem. We need to remove sdb from all these devices.

# mdadm /dev/md0 --remove /dev/sdb2
# mdadm /dev/md1 --remove /dev/sdb3
# mdadm /dev/md2 --remove /dev/sdb4

If the partitions are not removable from the array, as in our case, mdadm does not consider the disk as failed and continues to use it. When you try to remove it, you may encounter an error that the device is still in use. In this situation, you should mark the disk as faulty before removing it:

# mdadm /dev/md0 -f /dev/sdb2
# mdadm /dev/md1 -f /dev/sdb3
# mdadm /dev/md2 -f /dev/sdb4

Then, you can proceed with the removal of partitions from the array. You have successfully removed the faulty disk from the array, and you can now request a replacement disk from your data center. 2. Adding a New Disk to the Array after Replacement 2.1. Determining the Partition Table (GPT or MBR) and Transferring it to the New Disk After replacing the failed disk, you need to add the new disk to the array. To do this, you should first determine whether your partition table is GPT or MBR. You can use the gdisk tool for this. Install gdisk with the following command:

# apt-get install gdisk -y

Then, run the following command to check the partition table:

# gdisk -l /dev/sda

Replace /dev/sda with the working disk in your RAID. The output will indicate whether you have MBR or GPT: For MBR, the output will be similar to:

Partition table scan:
MBR: MBR only
BSD: not present
APM: not present
GPT: not present

For GPT, the output will be similar to:

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

After determining the partition table, you need to create partitions on the new disk that match the partitions on sda. The process for creating partitions depends on the disk layout. Copying the partition layout for GPT:

# sgdisk -R /dev/sdb /dev/sda

Be careful here; the first disk is the destination, and the second one is the source. Mixing them up can result in the loss of the partition layout on the originally working disk. Generate a new random UUID for the disk:

# sgdisk -G /dev/sdb

Copying the partition layout for MBR:

# sfdisk -d /dev/sda | sfdisk /dev/sdb

In this case, the first disk is the source, and the second is the destination. If the partitions are not visible in the system, you can rescan the partition table using:

# sfdisk -R /dev/sdb

2.2. Adding the Disk to the Array After creating the partitions on /dev/sdb, you can add the disk to the array:

# mdadm /dev/md0 -a /dev/sdb2
# mdadm /dev/md1 -a /dev/sdb3
# mdadm /dev/md2 -a /dev/sdb4

3. Installing the Boot Loader After adding the disk to the array, you need to install the boot loader on it. If the server is booting normally, you can use the following command:

# grub-install /dev/sdb

If the server is booting in recovery or rescue mode (e.g., from a live CD), the bootloader installation is done differently. Mount the root filesystem to /mnt:

# mount /dev/md2 /mnt

Mount /boot:

# mount /dev/md0 /mnt/boot

Mount /dev, /proc, and /sys:

# mount --bind /dev /mnt/dev
# mount --bind /proc /mnt/proc
# mount --bind /sys  /mnt/sys

Then, chroot into the mounted system:

# chroot /mnt

And install Grub on sdb:

# grub-install /dev/sdb

Now, you can try booting the server in normal mode.