Skip to content. | Skip to navigation

IT Virtualization Blog

Personal tools

This is SunRain Plone Theme
You are here: Home / Users / lmarzke / howto / FreeNAS ZFS mirror expansion issue

FreeNAS ZFS mirror expansion issue

by lmarzke last modified May 01, 2018 08:10 PM
FreeNAS 11.1U4 on FreeBSD. Expansion of both mirror disks did not result in expansion of the vDEV as expected. After some troubleshooting a GPT partition issue was the cause, and the fix explained.

Issue

I have a FreeNAS system as described at here, composed of 6 x 2TB disks in 3 mirrors.   After failure of one of the 2TB disks I replaced it with a 4GB part,  and after re-silvering replaced the 2nd mirror device,  however the mirror still kept reporting 2TB usable.

2TB drives are SUN / Seagate Constellation ES SATA,  and  4TB drives are Western Digital RED NAS

Troubleshooting

The clue leading to resolution was the output of the gpart command.

Note the [CORRUPT] label on da1

root@freenas:~ # gpart show
=> 34 3907029101 da0 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 4294967228 da1 GPT (3.6T) [CORRUPT]
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 4290772824 2 freebsd-zfs (2.0T)
4294967256 6 - free - (3.0K)

=> 34 3907029101 da2 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 3907029101 da3 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 3907029101 da4 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 7814037101 da5 GPT (3.6T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 7809842696 2 freebsd-zfs (3.6T)
7814037128 7 - free - (3.5K)

 

The gpart man page references issues causing corruption,  so looking though the logs we see the following:

root@freenas:~ # dmesg | grep -i gpt
GEOM: da1: the secondary GPT header is not in the last LBA.

The man page references this error and recommend running recover option. 

Note: no corruption here - but a prior disk resizing caused the 2nd GPT header to not be placed correctly at end of disk, so there is no backup to the single GPT table.

Running 'gpart recover' on the disk fixes this easily.

root@freenas:~ # gpart recover /dev/da1
da1 recovered

root@freenas:~ # gpart show
=> 34 3907029101 da0 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 40 7814037088 da1 GPT (3.6T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 4290772824 2 freebsd-zfs (2.0T)
4294967256 3519069872 - free - (1.6T)
=> 34 3907029101 da2 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 3907029101 da3 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 3907029101 da4 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 34 7814037101 da5 GPT (3.6T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 7809842696 2 freebsd-zfs (3.6T)
7814037128 7 - free - (3.5K)

Note however,  that now  da1 partition 2 (freebsd-zfs) is still 2.0 T,  with 1.6T of free-space immediately following.

In the case of contiguous free space after the partition ,  gpart can resize the partition easily.

root@freenas:~ # gpart resize -i 2 da1
da1p2 resized

root@freenas:~ # gpart show
=> 34 3907029101 da0 GPT (1.8T)
34 94 - free - (47K)
128 4194304 1 freebsd-swap (2.0G)
4194432 3902834696 2 freebsd-zfs (1.8T)
3907029128 7 - free - (3.5K)

=> 40 7814037088 da1 GPT (3.6T)
40 88 - free - (44K)
128 4194304 1 freebsd-swap (2.0G)
4194432 7809842696 2 freebsd-zfs (3.6T)

Now the 2nd partition (freebsd-zfs) shows 3.6T as well as the entire disk.

After this zpool list -v  still did NOT show the expanded size.

root@freenas:~ # zpool list -v

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bigpool 5.61T 3.81T 1.80T - 46% 67% 1.00x ONLINE /mnt
mirror 1.81T 1.22T 608G - 46% 67%
gptid/06967635-9d22-11e4-9294-00221992d64d - - - - - -
gptid/7cf831d6-d697-11e6-9391-00221992d64d - - - - - -
mirror 1.98T 1.37T 633G - 46% 68%
gptid/c72c07c4-2085-11e7-b172-00221992d64d - - - - - -
gptid/6e147570-04e0-11e7-b8fe-00221992d64d - - - - - -
mirror 1.81T 1.22T 607G - 46% 67%
gptid/08a5cba0-9d22-11e4-9294-00221992d64d - - - - - -
gptid/09170759-9d22-11e4-9294-00221992d64d - - - - - -
log - - - - - -
gptid/ddc212bd-5462-11e7-b10e-00221992d64d 91G 27.7M 91.0G - 79% 0%
cache - - - - - -
gptid/a76755b4-4ebf-11e7-ab31-00221992d64d 74.5G 60.8G 13.7G - 0% 81%
freenas-boot 7.25G 1.64G 5.61G - - 22% 1.00x ONLINE -
mirror 7.25G 1.64G 5.61G - - 22%
da7p2 - - - - - -
da8p2

Running the zpool online expansion command now worked.

root@freenas:~ # zpool online -e bigpool gptid/c72c07c4-2085-11e7-b172-00221992d64d 
root@freenas:~ # zpool online -e bigpool gptid/6e147570-04e0-11e7-b8fe-00221992d64d

root@freenas:~ # zpool list -v
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
bigpool 7.25T 3.81T 3.44T - 35% 52% 1.00x ONLINE /mnt
mirror 1.81T 1.22T 608G - 46% 67%
gptid/06967635-9d22-11e4-9294-00221992d64d - - - - - -
gptid/7cf831d6-d697-11e6-9391-00221992d64d - - - - - -
mirror 3.62T 1.37T 2.26T - 25% 37%
gptid/c72c07c4-2085-11e7-b172-00221992d64d - - - - - -
gptid/6e147570-04e0-11e7-b8fe-00221992d64d - - - - - -
mirror 1.81T 1.22T 607G - 46% 67%
gptid/08a5cba0-9d22-11e4-9294-00221992d64d - - - - - -
gptid/09170759-9d22-11e4-9294-00221992d64d - - - - - -
log - - - - - -
gptid/ddc212bd-5462-11e7-b10e-00221992d64d 91G 2.16M 91.0G - 79% 0%
cache - - - - - -
gptid/a76755b4-4ebf-11e7-ab31-00221992d64d 74.5G 60.8G 13.7G - 0% 81%
freenas-boot 7.25G 1.64G 5.61G - - 22% 1.00x ONLINE -
mirror 7.25G 1.64G 5.61G - - 22%
da7p2 - - - - - -
da8p2

At last,  the mirror vDev, and pool now show increased size.

 

Lee Marzke      <lmarzke@4aero.com>

Infrastructure / VMware Consultant, 4aero

 

Document Actions