🌐 🇵🇱 Polski · 🇬🇧 EN
Sometimes a situation arises where one of the disks in a RAID partition fails. In such a case, it is necessary to remove the faulty disk and replace it with a new, functional one. We can manually simulate a disk failure to see what happens.
Step 1 - To perform a controlled disk failure in a RAID partition, use the following command:
# mdadm /dev/md0 -f /dev/hdd1
mdadm: set /dev/hdd1 faulty in /dev/md0
Step 2 - Checking the disks in the RAID after one of them has failed
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Jan 27 14:09:23 2011
Raid Level : raid5
Array Size : 16769024 (15.99 GiB 17.17 GB)
Used Dev Size : 8384512 (8.00 GiB 8.59 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu Jan 27 14:28:55 2011
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 1
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : RHEL-01:0 (local to host RHEL-01)
UUID : a02fb98a:63a7cbbf:762c7a7f:e681a8ee
Events : 19
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/hdb1
1 8 33 1 active sync /dev/hdc1
2 0 0 2 removed
3 8 49 - faulty spare
/dev/hdd1
It is visible which disk has failed and needs to be replaced with a new one.
Step 3 - Removing the disk from the pool of disks included in the RAID
# mdadm /dev/md0 -r /dev/hdd1
mdadm: hot removed /dev/hdd1 from /dev/md0
Step 4 - Re-checking detailed information about the RAID
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Jan 27 14:09:23 2011
Raid Level : raid5
Array Size : 16769024 (15.99 GiB 17.17 GB)
Used Dev Size : 8384512 (8.00 GiB 8.59 GB)
Raid Devices : 3
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Jan 27 14:31:23 2011
State : clean, degraded
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : RHEL-01:0 (local to host RHEL-01)
UUID : a02fb98a:63a7cbbf:762c7a7f:e681a8ee
Events : 22
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/hdb1
1 8 33 1 active sync /dev/hdc1
2 0 0 2 removed
In the last line of this command's output, you can see that one of the disks has been removed. In a real-world environment, in such a situation, we remove the faulty disk and replace it with a new one. Then, we activate the new disk. We set up the same partition on the new disk as on the faulty one (hdd) and configure the RAID.
The steps provided discuss a command that simulates a failure of one of the disks and a subsequent command that removes the faulty partition. This can be done in one go, and it may be one of the topics on RedHat exams.
# mdadm -v /dev/md0 -f /dev/hdd1 -r /dev/hdd1
Step 5 - If the new primary partition has been created, you can add it to the RAID partition.
mdadm: re-added /dev/hdd1
Step 6 - Checking if the new disk has been added correctly.
# mdadm -D /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Thu Jan 27 14:09:23 2011
Raid Level : raid5
Array Size : 16769024 (15.99 GiB 17.17 GB)
Used Dev Size : 8384512 (8.00 GiB 8.59 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu Jan 27 14:34:21 2011
State : clean, degraded, recovering
Active Devices : 2
Working Devices : 3
Failed Devices : 0
Spare Devices : 1
Layout : left-symmetric
Chunk Size : 512K
Rebuild Status : 3% complete
Name : RHEL-01:0 (local to host RHEL-01)
UUID : a02fb98a:63a7cbbf:762c7a7f:e681a8ee
Events : 26
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/hdb1
1 8 33 1 active sync /dev/hdc1
3 8 49 2 spare rebuilding
/dev/hdd1
The visible information "rebuilding" indicates that the new disk is being added to the RAID.
Step 7 - Querying the system kernel
# cat /proc/mdstat
Personalities : [raid0] [raid6] [raid5] [raid4]
md0 : active raid5 hdd1[2] hdc1[1] hdb1[0]
16769024 blocks super 1.2 level 5, 512k chunk, algorithm 2
[3/2] [UU_]
[===>.................] recovery = 17.4% (697276/4000640)
finish=3.9min speed=13843K/sec
unused devices: <none>
Here you can see more details regarding the data rebuilding on the new partition.
Step 8 - If something goes wrong with the RAID as a whole, you can stop the operation for the entire partition.
# mdadm -vS /dev/md0
mdadm: stopped /dev/md0
When the entire RAID partition is stopped, you can perform repair work, after which you can restart the RAID.
Comments