#acl PaulHowarth:read,write,admin,revert,delete All:read === Tuesday 27th May 2008 === ==== Shrinking Disk? ==== After installing Fedora 9 onto my firewall/Internet server box using the existing partitioning/RAID/LVM layout, I found the kernel complaining that one of my disk partitions extended past the end of the drive, something that was never an issue on Fedora 8. {{{ May 26 15:20:27 goalkeeper kernel: scsi 0:0:1:0: Direct-Access ATA WDC WD800BB-75CA 16.0 PQ: 0 ANSI: 5 May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] 156250000 512-byte hardware sectors (80000 MB) May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write Protect is off May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] 156250000 512-byte hardware sectors (80000 MB) May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write Protect is off May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA May 26 15:20:27 goalkeeper kernel: sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 > May 26 15:20:27 goalkeeper kernel: sdb: p6 exceeds device capacity May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Attached SCSI disk May 26 15:20:27 goalkeeper kernel: attempt to access beyond end of device May 26 15:20:27 goalkeeper kernel: sdb: rw=0, want=156296145, limit=156250000 May 26 15:20:27 goalkeeper kernel: Buffer I/O error on device sdb6, logical block 4608608 May 26 15:20:27 goalkeeper kernel: attempt to access beyond end of device May 26 15:20:27 goalkeeper kernel: sdb: rw=0, want=156296145, limit=156250000 }}} I wonder whether it was always past the end of the disk and Fedora 8 just didn't care (and it wasn't an issue because I wasn't using all of the space in the affected volume group), or whether the kernel's view of the number of cylinders on the disk changed in Fedora 9. Anyway, after booting Fedora 9, this is what I had: {{{ # fdisk -l /dev/sdb Disk /dev/sdb: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x9dc96e9e Device Boot Start End Blocks Id System /dev/sdb1 * 1 19 152586 fd Linux raid autodetect /dev/sdb2 20 38 152617+ fd Linux raid autodetect /dev/sdb3 39 5392 43006005 fd Linux raid autodetect /dev/sdb4 5393 9729 34836952+ 5 Extended /dev/sdb5 5393 7434 16402333+ fd Linux raid autodetect /dev/sdb6 7435 9729 18434556 fd Linux raid autodetect }}} So the extended partition `/dev/sdb4` and the logical partition `/dev/sdb6` both extended past the end of the drive by 3 cylinders. Now fortunately I'm in the habit of building everything on top of RAID1 these days, so the machine was still running happily with a degraded array, having removed `/dev/sdb6` as faulty. Moreover, since I run LVM on top of my RAID1, and the volume group was not fully utilized, I was able to "fix" this issue with minimal disruption and no data loss... The first step was to shrink the offending partitions down by 3 cylinders so that they fit on the disk. This would destroy the content of `/dev/sdb6` but that was OK as it had already been automatically failed out of the RAID array and wasn't in use. I removed the oversized extended partition `/dev/sdb4` and that took out the logical partitions `/dev/sdb5` and `/dev/sdb6` too: {{{ # fdisk /dev/sdb The number of cylinders for this disk is set to 9726. There is nothing wrong with that, but this is larger than 1024, and could in certain setups cause problems with: 1) software that runs at boot time (e.g., old versions of LILO) 2) booting and partitioning software from other OSs (e.g., DOS FDISK, OS/2 FDISK) Command (m for help): p Disk /dev/sdb: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x9dc96e9e Device Boot Start End Blocks Id System /dev/sdb1 * 1 19 152586 fd Linux raid autodetect /dev/sdb2 20 38 152617+ fd Linux raid autodetect /dev/sdb3 39 5392 43006005 fd Linux raid autodetect /dev/sdb4 5393 9729 34836952+ 5 Extended /dev/sdb5 5393 7434 16402333+ fd Linux raid autodetect /dev/sdb6 7435 9729 18434556 fd Linux raid autodetect Command (m for help): d Partition number (1-6): 4 Command (m for help): p Disk /dev/sdb: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x9dc96e9e Device Boot Start End Blocks Id System /dev/sdb1 * 1 19 152586 fd Linux raid autodetect /dev/sdb2 20 38 152617+ fd Linux raid autodetect /dev/sdb3 39 5392 43006005 fd Linux raid autodetect }}} I then added a new extended partition `/dev/sdb4` of the correct size, restored `/dev/sdb5` in exactly the same position as the old one, and a slightly smaller `/dev/sdb6` that now fit the remaining space: {{{ Command (m for help): n Command action e extended p primary partition (1-4) e Selected partition 4 First cylinder (5393-9726, default 5393): Using default value 5393 Last cylinder or +size or +sizeM or +sizeK (5393-9726, default 9726): Using default value 9726 Command (m for help): n First cylinder (5393-9726, default 5393): Using default value 5393 Last cylinder or +size or +sizeM or +sizeK (5393-9726, default 9726): 7434 Command (m for help): n First cylinder (7435-9726, default 7435): Using default value 7435 Last cylinder or +size or +sizeM or +sizeK (7435-9726, default 9726): Using default value 9726 Command (m for help): p Disk /dev/sdb: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x9dc96e9e Device Boot Start End Blocks Id System /dev/sdb1 * 1 19 152586 fd Linux raid autodetect /dev/sdb2 20 38 152617+ fd Linux raid autodetect /dev/sdb3 39 5392 43006005 fd Linux raid autodetect /dev/sdb4 5393 9726 34812855 5 Extended /dev/sdb5 5393 7434 16402333+ 83 Linux /dev/sdb6 7435 9726 18410458+ 83 Linux Command (m for help): t Partition number (1-6): 5 Hex code (type L to list codes): fd Changed system type of partition 5 to fd (Linux raid autodetect) Command (m for help): t Partition number (1-6): 6 Hex code (type L to list codes): fd Changed system type of partition 6 to fd (Linux raid autodetect) Command (m for help): p Disk /dev/sdb: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x9dc96e9e Device Boot Start End Blocks Id System /dev/sdb1 * 1 19 152586 fd Linux raid autodetect /dev/sdb2 20 38 152617+ fd Linux raid autodetect /dev/sdb3 39 5392 43006005 fd Linux raid autodetect /dev/sdb4 5393 9726 34812855 5 Extended /dev/sdb5 5393 7434 16402333+ fd Linux raid autodetect /dev/sdb6 7435 9726 18410458+ fd Linux raid autodetect Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot. Syncing disks. }}} Before I could do any work with the smaller new `/dev/sdb6` partition, I needed to reboot so that the kernel would see the updated partition table. {{{ # reboot Broadcast message from root@goalkeeper.city-fan.org (/dev/pts/0) at 12:09 ... The system is going down for reboot NOW! }}} The next step was to create a new RAID1 array to replace `/dev/md2` that `/dev/sdb6` had been part of. The new one would be three cylinders smaller of course. I would eventually add `/dev/sda6` (the other half of `/dev/md2`) to the new array, but `/dev/md2` (and hence `/dev/sda6`) was still in use as part of LVM volume group `vgOSa`, so I created the the new array `/dev/md6` with `/dev/sdb6` and a missing mirror to be added later: {{{ # mdadm --create --level=1 --raid-devices=2 --auto=md /dev/md6 /dev/sdb6 missing mdadm: array /dev/md6 started. [root@goalkeeper ~]# mdadm --detail /dev/md6 /dev/md6: Version : 00.90.03 Creation Time : Tue May 27 12:14:58 2008 Raid Level : raid1 Array Size : 18410368 (17.56 GiB 18.85 GB) Used Dev Size : 18410368 (17.56 GiB 18.85 GB) Raid Devices : 2 Total Devices : 1 Preferred Minor : 6 Persistence : Superblock is persistent Update Time : Tue May 27 12:14:58 2008 State : clean, degraded Active Devices : 1 Working Devices : 1 Failed Devices : 0 Spare Devices : 0 UUID : bf013ddf:1da1e663:2bb59604:4d695037 Events : 0.1 Number Major Minor RaidDevice State 0 8 22 0 active sync /dev/sdb6 1 0 0 1 removed }}} I could now create an LVM physical volume on the new array. However, `pvcreate` was concerned that I was overwriting an existing physical volume, which of course I was: {{{ # pvcreate /dev/md6 Found duplicate PV biKO5Kz1nDpu1nBuT3Pc3vZsXvKWgRTO: using /dev/md2 not /dev/md6 Can't initialize physical volume "/dev/md6" of volume group "vgOSa" without -ff }}} So I blew away the metadata by filling the first megabyte of the array with zeroes and tried again: {{{ # dd if=/dev/zero of=/dev/md6 bs=1024 count=1024 1024+0 records in 1024+0 records out 1048576 bytes (1.0 MB) copied, 0.176955 s, 5.9 MB/s # pvcreate /dev/md6 Physical volume "/dev/md6" successfully created }}} I could now add the new array to the volume group that the existing `/dev/md2` belonged to, `vgOSa`: {{{ # vgextend vgOSa /dev/md6 Volume group "vgOSa" successfully extended # vgdisplay -v vgOSa --- Volume group --- VG Name vgOSa System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 8 VG Access read/write VG Status resizable MAX LV 0 Cur LV 6 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 35.09 GB PE Size 32.00 MB Total PE 1123 Alloc PE / Size 484 / 15.12 GB Free PE / Size 639 / 19.97 GB VG UUID h029AZ-TRm5-gS2F-iq1l-MSxC-5eko-Vu6J6G ... --- Physical volumes --- PV Name /dev/md2 PV UUID biKO5K-z1nD-pu1n-BuT3-Pc3v-ZsXv-KWgRTO PV Status allocatable Total PE / Free PE 562 / 78 PV Name /dev/md6 PV UUID yyQPHZ-xM8o-qbgM-kv5U-6G1M-0oMZ-Le9L8P PV Status allocatable Total PE / Free PE 561 / 561 }}} The next step was to move all of the data off the old array `/dev/md2` and onto the new one `/dev/md6`. This is easily done with the system fully live using `pvmove`: {{{ # pvmove /dev/md2 /dev/md6 # vgdisplay -v vgOSa Using volume group(s) on command line Finding volume group "vgOSa" --- Volume group --- VG Name vgOSa System ID Format lvm2 Metadata Areas 2 Metadata Sequence No 16 VG Access read/write VG Status resizable MAX LV 0 Cur LV 6 Open LV 0 Max PV 0 Cur PV 2 Act PV 2 VG Size 35.09 GB PE Size 32.00 MB Total PE 1123 Alloc PE / Size 484 / 15.12 GB Free PE / Size 639 / 19.97 GB VG UUID h029AZ-TRm5-gS2F-iq1l-MSxC-5eko-Vu6J6G ... --- Physical volumes --- PV Name /dev/md2 PV UUID biKO5K-z1nD-pu1n-BuT3-Pc3v-ZsXv-KWgRTO PV Status allocatable Total PE / Free PE 562 / 562 PV Name /dev/md6 PV UUID yyQPHZ-xM8o-qbgM-kv5U-6G1M-0oMZ-Le9L8P PV Status allocatable Total PE / Free PE 561 / 77 }}} With no data left on `/dev/md2`, I could now remove it from the volume group, the volume group now being fully migrated onto the new array: {{{ # vgreduce vgOSa /dev/md2 Removed "/dev/md2" from volume group "vgOSa" # vgdisplay -v vgOSa Using volume group(s) on command line Finding volume group "vgOSa" --- Volume group --- VG Name vgOSa System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 17 VG Access read/write VG Status resizable MAX LV 0 Cur LV 6 Open LV 0 Max PV 0 Cur PV 1 Act PV 1 VG Size 17.53 GB PE Size 32.00 MB Total PE 561 Alloc PE / Size 484 / 15.12 GB Free PE / Size 77 / 2.41 GB VG UUID h029AZ-TRm5-gS2F-iq1l-MSxC-5eko-Vu6J6G ... --- Physical volumes --- PV Name /dev/md6 PV UUID yyQPHZ-xM8o-qbgM-kv5U-6G1M-0oMZ-Le9L8P PV Status allocatable Total PE / Free PE 561 / 77 }}} The old array `/dev/md2` was no longer in use and could be stopped: {{{ # mdadm --stop /dev/md2 mdadm: stopped /dev/md2 # mdadm --detail /dev/md2 mdadm: md device /dev/md2 does not appear to be active. }}} I could now add the `/dev/sda6` partition into the new array `/dev/md6` since it was no longer in use by `/dev/md2`. It didn't matter that the partition size was slightly larger than the array size; the extra space just doesn't get used. {{{ # fdisk -l /dev/sda Disk /dev/sda: 82.3 GB, 82348277760 bytes 255 heads, 63 sectors/track, 10011 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x0007b59d Device Boot Start End Blocks Id System /dev/sda1 * 1 19 152586 fd Linux raid autodetect /dev/sda2 20 38 152617+ fd Linux raid autodetect /dev/sda3 39 5392 43006005 fd Linux raid autodetect /dev/sda4 5393 10011 37102117+ 5 Extended /dev/sda5 5393 7687 18434556 fd Linux raid autodetect /dev/sda6 7688 9982 18434556 fd Linux raid autodetect # fdisk -l /dev/sdb Disk /dev/sdb: 80.0 GB, 80000000000 bytes 255 heads, 63 sectors/track, 9726 cylinders Units = cylinders of 16065 * 512 = 8225280 bytes Disk identifier: 0x9dc96e9e Device Boot Start End Blocks Id System /dev/sdb1 * 1 19 152586 fd Linux raid autodetect /dev/sdb2 20 38 152617+ fd Linux raid autodetect /dev/sdb3 39 5392 43006005 fd Linux raid autodetect /dev/sdb4 5393 9726 34812855 5 Extended /dev/sdb5 5393 7434 16402333+ fd Linux raid autodetect /dev/sdb6 7435 9726 18410458+ fd Linux raid autodetect # mdadm /dev/md6 -a /dev/sda6 mdadm: added /dev/sda6 # mdadm --detail /dev/md6 /dev/md6: Version : 00.90.03 Creation Time : Tue May 27 12:14:58 2008 Raid Level : raid1 Array Size : 18410368 (17.56 GiB 18.85 GB) Used Dev Size : 18410368 (17.56 GiB 18.85 GB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 6 Persistence : Superblock is persistent Update Time : Tue May 27 12:49:15 2008 State : clean, degraded, recovering Active Devices : 1 Working Devices : 2 Failed Devices : 0 Spare Devices : 1 Rebuild Status : 1% complete UUID : bf013ddf:1da1e663:2bb59604:4d695037 Events : 0.34 Number Major Minor RaidDevice State 0 8 22 0 active sync /dev/sdb6 2 8 6 1 spare rebuilding /dev/sda6 }}} A few minutes later the array was fully recovered and the oversize partition problem had been fixed, all online with no data loss and no need to backup and restore the data. Another related niggle I had was that another RAID1 array, `/dev/md0`, made up from `/dev/sda1` and `/dev/sdb1`, which was used as the `/boot` partition for the previous Fedora 8 installation, wasn't showing up in Fedora 9 (I partition my system up so that I can do a fresh install of new Fedora versions without writing over the the current version in case I have big problems and need to go back to the old version for a while). I suspect that the installer had changed something in the metadata: {{{ # mdadm --query /dev/sda1 /dev/sda1: is not an md array /dev/sda1: device 0 in 2 device undetected raid1 /dev/.tmp.md0. Use mdadm --examine for more detail. # mdadm --query /dev/sdb1 /dev/sdb1: is not an md array /dev/sdb1: device 1 in 2 device undetected raid1 /dev/md0. Use mdadm --examine for more detail. # mdadm --examine /dev/sdb1 /dev/sdb1: Magic : a92b4efc Version : 00.90.00 UUID : c0be5227:82b8bda1:055163eb:7970b0f9 Creation Time : Sun Dec 3 15:19:27 2006 Raid Level : raid1 Used Dev Size : 152512 (148.96 MiB 156.17 MB) Array Size : 152512 (148.96 MiB 156.17 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 0 Update Time : Mon May 26 15:08:35 2008 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Checksum : f9147855 - correct Events : 0.58 Number Major Minor RaidDevice State this 1 8 49 1 active sync /dev/sdd1 0 0 8 33 0 active sync /dev/sdc1 1 1 8 49 1 active sync /dev/sdd1 }}} I'm not sure why `/dev/sda1` had been tweaked to be in `/dev/.tmp.md0` rather than `/dev/md0` but that was easily fixed: {{{ # mdadm --create --level=1 --raid-devices=2 --auto=md /dev/md0 /dev/sd[ab]1 mdadm: /dev/sda1 appears to contain an ext2fs file system size=152512K mtime=Mon May 26 15:05:33 2008 mdadm: /dev/sda1 appears to be part of a raid array: level=raid1 devices=2 ctime=Sun Dec 3 15:19:27 2006 mdadm: /dev/sdb1 appears to contain an ext2fs file system size=152512K mtime=Mon May 26 15:05:33 2008 mdadm: /dev/sdb1 appears to be part of a raid array: level=raid1 devices=2 ctime=Sun Dec 3 15:19:27 2006 Continue creating array? y mdadm: array /dev/md0 started. # mdadm --detail /dev/md0 /dev/md0: Version : 00.90.03 Creation Time : Tue May 27 12:53:46 2008 Raid Level : raid1 Array Size : 152512 (148.96 MiB 156.17 MB) Used Dev Size : 152512 (148.96 MiB 156.17 MB) Raid Devices : 2 Total Devices : 2 Preferred Minor : 0 Persistence : Superblock is persistent Update Time : Tue May 27 12:53:46 2008 State : clean Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 UUID : 8fa04abe:c3894ab6:29fb567d:92fd1cec Events : 0.1 Number Major Minor RaidDevice State 0 8 1 0 active sync /dev/sda1 1 8 17 1 active sync /dev/sdb1 }}} I was then able to mount `/dev/md0` and the existing data was still there as expected (i.e. the contents of the Fedora 8 `/boot` partition). ----