PaulHowarth/Blog/2008-05-27

Tuesday 27th May 2008

Shrinking Disk?

After installing Fedora 9 onto my firewall/Internet server box using the existing partitioning/RAID/LVM layout, I found the kernel complaining that one of my disk partitions extended past the end of the drive, something that was never an issue on Fedora 8.

May 26 15:20:27 goalkeeper kernel: scsi 0:0:1:0: Direct-Access     ATA      WDC WD800BB-75CA 16.0 PQ: 0 ANSI: 5
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] 156250000 512-byte hardware sectors (80000 MB)
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write Protect is off
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] 156250000 512-byte hardware sectors (80000 MB)
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write Protect is off
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
May 26 15:20:27 goalkeeper kernel:  sdb: sdb1 sdb2 sdb3 sdb4 < sdb5 sdb6 >
May 26 15:20:27 goalkeeper kernel:  sdb: p6 exceeds device capacity
May 26 15:20:27 goalkeeper kernel: sd 0:0:1:0: [sdb] Attached SCSI disk
May 26 15:20:27 goalkeeper kernel: attempt to access beyond end of device
May 26 15:20:27 goalkeeper kernel: sdb: rw=0, want=156296145, limit=156250000
May 26 15:20:27 goalkeeper kernel: Buffer I/O error on device sdb6, logical block 4608608
May 26 15:20:27 goalkeeper kernel: attempt to access beyond end of device
May 26 15:20:27 goalkeeper kernel: sdb: rw=0, want=156296145, limit=156250000

I wonder whether it was always past the end of the disk and Fedora 8 just didn't care (and it wasn't an issue because I wasn't using all of the space in the affected volume group), or whether the kernel's view of the number of cylinders on the disk changed in Fedora 9.

Anyway, after booting Fedora 9, this is what I had:

# fdisk -l /dev/sdb

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          19      152586   fd  Linux raid autodetect
/dev/sdb2              20          38      152617+  fd  Linux raid autodetect
/dev/sdb3              39        5392    43006005   fd  Linux raid autodetect
/dev/sdb4            5393        9729    34836952+   5  Extended
/dev/sdb5            5393        7434    16402333+  fd  Linux raid autodetect
/dev/sdb6            7435        9729    18434556   fd  Linux raid autodetect

So the extended partition /dev/sdb4 and the logical partition /dev/sdb6 both extended past the end of the drive by 3 cylinders. Now fortunately I'm in the habit of building everything on top of RAID1 these days, so the machine was still running happily with a degraded array, having removed /dev/sdb6 as faulty.

Moreover, since I run LVM on top of my RAID1, and the volume group was not fully utilized, I was able to "fix" this issue with minimal disruption and no data loss...

The first step was to shrink the offending partitions down by 3 cylinders so that they fit on the disk. This would destroy the content of /dev/sdb6 but that was OK as it had already been automatically failed out of the RAID array and wasn't in use. I removed the oversized extended partition /dev/sdb4 and that took out the logical partitions /dev/sdb5 and /dev/sdb6 too:

# fdisk /dev/sdb

The number of cylinders for this disk is set to 9726.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
   (e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): p

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          19      152586   fd  Linux raid autodetect
/dev/sdb2              20          38      152617+  fd  Linux raid autodetect
/dev/sdb3              39        5392    43006005   fd  Linux raid autodetect
/dev/sdb4            5393        9729    34836952+   5  Extended
/dev/sdb5            5393        7434    16402333+  fd  Linux raid autodetect
/dev/sdb6            7435        9729    18434556   fd  Linux raid autodetect

Command (m for help): d
Partition number (1-6): 4

Command (m for help): p

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          19      152586   fd  Linux raid autodetect
/dev/sdb2              20          38      152617+  fd  Linux raid autodetect
/dev/sdb3              39        5392    43006005   fd  Linux raid autodetect

I then added a new extended partition /dev/sdb4 of the correct size, restored /dev/sdb5 in exactly the same position as the old one, and a slightly smaller /dev/sdb6 that now fit the remaining space:

Command (m for help): n
Command action
   e   extended
   p   primary partition (1-4)
e
Selected partition 4
First cylinder (5393-9726, default 5393): 
Using default value 5393
Last cylinder or +size or +sizeM or +sizeK (5393-9726, default 9726): 
Using default value 9726

Command (m for help): n
First cylinder (5393-9726, default 5393): 
Using default value 5393
Last cylinder or +size or +sizeM or +sizeK (5393-9726, default 9726): 7434

Command (m for help): n
First cylinder (7435-9726, default 7435): 
Using default value 7435
Last cylinder or +size or +sizeM or +sizeK (7435-9726, default 9726): 
Using default value 9726

Command (m for help): p

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          19      152586   fd  Linux raid autodetect
/dev/sdb2              20          38      152617+  fd  Linux raid autodetect
/dev/sdb3              39        5392    43006005   fd  Linux raid autodetect
/dev/sdb4            5393        9726    34812855    5  Extended
/dev/sdb5            5393        7434    16402333+  83  Linux
/dev/sdb6            7435        9726    18410458+  83  Linux

Command (m for help): t
Partition number (1-6): 5
Hex code (type L to list codes): fd
Changed system type of partition 5 to fd (Linux raid autodetect)

Command (m for help): t
Partition number (1-6): 6
Hex code (type L to list codes): fd
Changed system type of partition 6 to fd (Linux raid autodetect)

Command (m for help): p

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          19      152586   fd  Linux raid autodetect
/dev/sdb2              20          38      152617+  fd  Linux raid autodetect
/dev/sdb3              39        5392    43006005   fd  Linux raid autodetect
/dev/sdb4            5393        9726    34812855    5  Extended
/dev/sdb5            5393        7434    16402333+  fd  Linux raid autodetect
/dev/sdb6            7435        9726    18410458+  fd  Linux raid autodetect

Command (m for help): w
The partition table has been altered!

Calling ioctl() to re-read partition table.

WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table.
The new table will be used at the next reboot.
Syncing disks.

Before I could do any work with the smaller new /dev/sdb6 partition, I needed to reboot so that the kernel would see the updated partition table.

# reboot

Broadcast message from root@goalkeeper.city-fan.org
        (/dev/pts/0) at 12:09 ...

The system is going down for reboot NOW!

The next step was to create a new RAID1 array to replace /dev/md2 that /dev/sdb6 had been part of. The new one would be three cylinders smaller of course. I would eventually add /dev/sda6 (the other half of /dev/md2) to the new array, but /dev/md2 (and hence /dev/sda6) was still in use as part of LVM volume group vgOSa, so I created the the new array /dev/md6 with /dev/sdb6 and a missing mirror to be added later:

# mdadm --create --level=1 --raid-devices=2 --auto=md /dev/md6 /dev/sdb6 missing
mdadm: array /dev/md6 started.
[root@goalkeeper ~]# mdadm --detail /dev/md6
/dev/md6:
        Version : 00.90.03
  Creation Time : Tue May 27 12:14:58 2008
     Raid Level : raid1
     Array Size : 18410368 (17.56 GiB 18.85 GB)
  Used Dev Size : 18410368 (17.56 GiB 18.85 GB)
   Raid Devices : 2
  Total Devices : 1
Preferred Minor : 6
    Persistence : Superblock is persistent

    Update Time : Tue May 27 12:14:58 2008
          State : clean, degraded
 Active Devices : 1
Working Devices : 1
 Failed Devices : 0
  Spare Devices : 0

           UUID : bf013ddf:1da1e663:2bb59604:4d695037
         Events : 0.1

    Number   Major   Minor   RaidDevice State
       0       8       22        0      active sync   /dev/sdb6
       1       0        0        1      removed

I could now create an LVM physical volume on the new array. However, pvcreate was concerned that I was overwriting an existing physical volume, which of course I was:

# pvcreate /dev/md6
  Found duplicate PV biKO5Kz1nDpu1nBuT3Pc3vZsXvKWgRTO: using /dev/md2 not /dev/md6
  Can't initialize physical volume "/dev/md6" of volume group "vgOSa" without -ff

So I blew away the metadata by filling the first megabyte of the array with zeroes and tried again:

# dd if=/dev/zero of=/dev/md6 bs=1024 count=1024
1024+0 records in
1024+0 records out
1048576 bytes (1.0 MB) copied, 0.176955 s, 5.9 MB/s
# pvcreate /dev/md6
  Physical volume "/dev/md6" successfully created

I could now add the new array to the volume group that the existing /dev/md2 belonged to, vgOSa:

# vgextend vgOSa /dev/md6
  Volume group "vgOSa" successfully extended
# vgdisplay -v vgOSa
  --- Volume group ---
  VG Name               vgOSa
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  8
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               35.09 GB
  PE Size               32.00 MB
  Total PE              1123
  Alloc PE / Size       484 / 15.12 GB
  Free  PE / Size       639 / 19.97 GB
  VG UUID               h029AZ-TRm5-gS2F-iq1l-MSxC-5eko-Vu6J6G
   
...
   
  --- Physical volumes ---
  PV Name               /dev/md2     
  PV UUID               biKO5K-z1nD-pu1n-BuT3-Pc3v-ZsXv-KWgRTO
  PV Status             allocatable
  Total PE / Free PE    562 / 78
   
  PV Name               /dev/md6     
  PV UUID               yyQPHZ-xM8o-qbgM-kv5U-6G1M-0oMZ-Le9L8P
  PV Status             allocatable
  Total PE / Free PE    561 / 561

The next step was to move all of the data off the old array /dev/md2 and onto the new one /dev/md6. This is easily done with the system fully live using pvmove:

# pvmove /dev/md2 /dev/md6
# vgdisplay -v vgOSa
    Using volume group(s) on command line
    Finding volume group "vgOSa"
  --- Volume group ---
  VG Name               vgOSa
  System ID             
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  16
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               0
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               35.09 GB
  PE Size               32.00 MB
  Total PE              1123
  Alloc PE / Size       484 / 15.12 GB
  Free  PE / Size       639 / 19.97 GB
  VG UUID               h029AZ-TRm5-gS2F-iq1l-MSxC-5eko-Vu6J6G
   
...
   
  --- Physical volumes ---
  PV Name               /dev/md2     
  PV UUID               biKO5K-z1nD-pu1n-BuT3-Pc3v-ZsXv-KWgRTO
  PV Status             allocatable
  Total PE / Free PE    562 / 562
   
  PV Name               /dev/md6     
  PV UUID               yyQPHZ-xM8o-qbgM-kv5U-6G1M-0oMZ-Le9L8P
  PV Status             allocatable
  Total PE / Free PE    561 / 77

With no data left on /dev/md2, I could now remove it from the volume group, the volume group now being fully migrated onto the new array:

# vgreduce vgOSa /dev/md2
  Removed "/dev/md2" from volume group "vgOSa"
# vgdisplay -v vgOSa
    Using volume group(s) on command line
    Finding volume group "vgOSa"
  --- Volume group ---
  VG Name               vgOSa
  System ID             
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  17
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                6
  Open LV               0
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               17.53 GB
  PE Size               32.00 MB
  Total PE              561
  Alloc PE / Size       484 / 15.12 GB
  Free  PE / Size       77 / 2.41 GB
  VG UUID               h029AZ-TRm5-gS2F-iq1l-MSxC-5eko-Vu6J6G
   
...
   
  --- Physical volumes ---
  PV Name               /dev/md6     
  PV UUID               yyQPHZ-xM8o-qbgM-kv5U-6G1M-0oMZ-Le9L8P
  PV Status             allocatable
  Total PE / Free PE    561 / 77

The old array /dev/md2 was no longer in use and could be stopped:

# mdadm --stop /dev/md2
mdadm: stopped /dev/md2
# mdadm --detail /dev/md2
mdadm: md device /dev/md2 does not appear to be active.

I could now add the /dev/sda6 partition into the new array /dev/md6 since it was no longer in use by /dev/md2. It didn't matter that the partition size was slightly larger than the array size; the extra space just doesn't get used.

# fdisk -l /dev/sda

Disk /dev/sda: 82.3 GB, 82348277760 bytes
255 heads, 63 sectors/track, 10011 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x0007b59d

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          19      152586   fd  Linux raid autodetect
/dev/sda2              20          38      152617+  fd  Linux raid autodetect
/dev/sda3              39        5392    43006005   fd  Linux raid autodetect
/dev/sda4            5393       10011    37102117+   5  Extended
/dev/sda5            5393        7687    18434556   fd  Linux raid autodetect
/dev/sda6            7688        9982    18434556   fd  Linux raid autodetect
# fdisk -l /dev/sdb

Disk /dev/sdb: 80.0 GB, 80000000000 bytes
255 heads, 63 sectors/track, 9726 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x9dc96e9e

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          19      152586   fd  Linux raid autodetect
/dev/sdb2              20          38      152617+  fd  Linux raid autodetect
/dev/sdb3              39        5392    43006005   fd  Linux raid autodetect
/dev/sdb4            5393        9726    34812855    5  Extended
/dev/sdb5            5393        7434    16402333+  fd  Linux raid autodetect
/dev/sdb6            7435        9726    18410458+  fd  Linux raid autodetect
# mdadm /dev/md6 -a /dev/sda6
mdadm: added /dev/sda6
# mdadm --detail /dev/md6
/dev/md6:
        Version : 00.90.03
  Creation Time : Tue May 27 12:14:58 2008
     Raid Level : raid1
     Array Size : 18410368 (17.56 GiB 18.85 GB)
  Used Dev Size : 18410368 (17.56 GiB 18.85 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 6
    Persistence : Superblock is persistent

    Update Time : Tue May 27 12:49:15 2008
          State : clean, degraded, recovering
 Active Devices : 1
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 1

 Rebuild Status : 1% complete

           UUID : bf013ddf:1da1e663:2bb59604:4d695037
         Events : 0.34

    Number   Major   Minor   RaidDevice State
       0       8       22        0      active sync   /dev/sdb6
       2       8        6        1      spare rebuilding   /dev/sda6

A few minutes later the array was fully recovered and the oversize partition problem had been fixed, all online with no data loss and no need to backup and restore the data.

Another related niggle I had was that another RAID1 array, /dev/md0, made up from /dev/sda1 and /dev/sdb1, which was used as the /boot partition for the previous Fedora 8 installation, wasn't showing up in Fedora 9 (I partition my system up so that I can do a fresh install of new Fedora versions without writing over the the current version in case I have big problems and need to go back to the old version for a while). I suspect that the installer had changed something in the metadata:

# mdadm --query /dev/sda1
/dev/sda1: is not an md array
/dev/sda1: device 0 in 2 device undetected raid1 /dev/.tmp.md0.  Use mdadm --examine for more detail.
# mdadm --query /dev/sdb1
/dev/sdb1: is not an md array
/dev/sdb1: device 1 in 2 device undetected raid1 /dev/md0.  Use mdadm --examine for more detail.
# mdadm --examine /dev/sdb1
/dev/sdb1:
          Magic : a92b4efc
        Version : 00.90.00
           UUID : c0be5227:82b8bda1:055163eb:7970b0f9
  Creation Time : Sun Dec  3 15:19:27 2006
     Raid Level : raid1
  Used Dev Size : 152512 (148.96 MiB 156.17 MB)
     Array Size : 152512 (148.96 MiB 156.17 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0

    Update Time : Mon May 26 15:08:35 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : f9147855 - correct
         Events : 0.58


      Number   Major   Minor   RaidDevice State
this     1       8       49        1      active sync   /dev/sdd1

   0     0       8       33        0      active sync   /dev/sdc1
   1     1       8       49        1      active sync   /dev/sdd1

I'm not sure why /dev/sda1 had been tweaked to be in /dev/.tmp.md0 rather than /dev/md0 but that was easily fixed:

# mdadm --create --level=1 --raid-devices=2 --auto=md /dev/md0 /dev/sd[ab]1
mdadm: /dev/sda1 appears to contain an ext2fs file system
    size=152512K  mtime=Mon May 26 15:05:33 2008
mdadm: /dev/sda1 appears to be part of a raid array:
    level=raid1 devices=2 ctime=Sun Dec  3 15:19:27 2006
mdadm: /dev/sdb1 appears to contain an ext2fs file system
    size=152512K  mtime=Mon May 26 15:05:33 2008
mdadm: /dev/sdb1 appears to be part of a raid array:
    level=raid1 devices=2 ctime=Sun Dec  3 15:19:27 2006
Continue creating array? y
mdadm: array /dev/md0 started.
# mdadm --detail /dev/md0
/dev/md0:
        Version : 00.90.03
  Creation Time : Tue May 27 12:53:46 2008
     Raid Level : raid1
     Array Size : 152512 (148.96 MiB 156.17 MB)
  Used Dev Size : 152512 (148.96 MiB 156.17 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Tue May 27 12:53:46 2008
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0

           UUID : 8fa04abe:c3894ab6:29fb567d:92fd1cec
         Events : 0.1

    Number   Major   Minor   RaidDevice State
       0       8        1        0      active sync   /dev/sda1
       1       8       17        1      active sync   /dev/sdb1

I was then able to mount /dev/md0 and the existing data was still there as expected (i.e. the contents of the Fedora 8 /boot partition).


Recent