Expanding a ZFS pool

As some users of ZFS come asks - is it possible to expand a pool?

The standard answer per zfs manual is to add more devices or vdevs (raidz, mirrors). As this gives you more space, it does comes with a backside and that is the need for more harddisk that again consumes more space. Space in commidity hardware is not something we have plentyfull of so the best approach would be to replace the existing harddrives.

The zfs documentation vaguely mentiones that you can do a one by one replacement of the drives within a vdev, thus indirectly expanding.

Earlier I have never been in need for expanding any vdev since upgrading space have come hand in hand with upgrade the entire server. But now the data growth have superseded the lifetime of the server, I had to solve this issue.

Server I use for storage today have two pools, each with one single vdev in raidz. The first goal was to upgrade the biggest pool. This consisted of 6 x 750GB harddrives, new drives was 1.5TB.

The clue is that the zpool will scale the vdev by the smallest device within its entity. And what one need to do is to replace the disks one by one.

To be absolutely sure on the method and the result I did first a quick test on a small install. I create a instance in vmware with 6 virtual disks.

da1: 1024MB (2097152 512 byte sectors: 64H 32S/T 1024C)

da2: 1024MB (2097152 512 byte sectors: 64H 32S/T 1024C)

da3: 1024MB (2097152 512 byte sectors: 64H 32S/T 1024C)

da4: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)

da5: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)

da6: 8192MB (16777216 512 byte sectors: 255H 63S/T 1044C)

# zpool create test raidz da{1,2,3}

# zpool status

pool: test

state: ONLINE

scrub: none requested

config:

NAME STATE READ WRITE CKSUM

test ONLINE 0 0 0

raidz1 ONLINE 0 0 0

da1 ONLINE 0 0 0

da2 ONLINE 0 0 0

da3 ONLINE 0 0 0

errors: No known data errors

# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT

test 2.98G 141K 2.98G 0% ONLINE -

As you can see the pool test is created as usual with 3 x 1GB disks and the total unconsumed space is approx 3 GB.

Now we continue to replace all the disks with 8 GB's.

# zpool replace test da1 da4

# zpool replace test da2 da5

# zpool replace test da3 da6

Note: you must wait for the resilver process to complete for each drive until you continue with the next one.

While replacing zpool status will report this:

# zpool status

pool: test

state: ONLINE

scrub: resilver completed after 0h0m with 0 errors on Sat Aug 29 19:52:20 2009

config:

NAME STATE READ WRITE CKSUM

test ONLINE 0 0 0

raidz1 ONLINE 0 0 0

da4 ONLINE 0 0 0 30.5K resilvered

da5 ONLINE 0 0 0 50K resilvered

da3 ONLINE 0 0 0 29.5K resilvered

errors: No known data errors

Now, the resilver process is very quick, since there where no data stored in the pool. This is a major benefit with ZFS, it only cares for the blocks that actually have data. In a hardware raid implementation, a replacement always results in a full XOR recalculation.

There is still one drive left to replace, you can see that the total space is still unaffected by the two first drives been replaced.

# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT

test 2.98G 165K 2.98G 0% ONLINE -

After the last drive been replaced the capacity have increased:

# zpool list

NAME SIZE USED AVAIL CAP HEALTH ALTROOT

test 24.0G 168K 24.0G 0% ONLINE -

In comparison with hardware and dedicated solutions (SAN) the approach is exactly the same. There are actually not many implementations you can expand a raid in this manner. Most solutions are based on moving a volume on a raid to a bigger raid or spanning the volume over several raid sets.

Many have complained ZFS lack of been able to resize raids, but they need to remember that very few implementions can do this. Resizing requires heavy recalculations and a major logistical works since the data is allready stored in a specific pattern with checksums.

Again, zfs shows us the future of filesystems. The flexibillity, ease of use and end to end intergrity puts most other fs's in the dust.

Note for FreeBSD: The current tested version, zfs v13. You most reboot the server when all replacements have been done in order for the zfs to detect the increased capacity. If there are other better methods, please comment.

Hva vil man?

I den siste tiden har jeg selv stått ovenfor valg som vil påvirke fremtiden i større grad. Veiskille, fritid, miljø etc er stikkord for de tanker og konsekvenser av valg jeg har måtte foreta. En annen like interessant sak er at jeg kjenner flere som har havnet i samme situasjon, å høre andres tanker og prioriteringer i slike settinger setter ofte ting i et godt perspektiv.

Det å stå på sidelinjen, men samtidig egentlig være med i samme båt og se de fleste prioritere og velge utifra hva andre forventer av dem er tankevekkende. Det virker som de fleste velger med horisont og perspektiv som er styrt av samfundets roller og normer - hva andre forventer.

Istedenfor burde man reflektere på hva man egentlig vil, og da med trykk på egentlig. De fleste utfordrer seg ikke nok på dette og tar rett og slett ikke hensyn til seg selv. Den andre side kan være det at mange setter seg i situasjoner hvori de i ikke simpelthet lenger har mulighet til å velge utifra det man vil.

For mange fengsler seg inn i den situasjon man allerede befinner seg i. Et typisk eksempel på dette er økonomiske rammer hvor stadig høyere inntekt senker terskelen for å binde seg til forpliktelser hvor den opprinnelige grunntanken var å ta ibruk muligheter man fikk.

(artikkel ikke ferdig)

Disconnecting

Before christmas, I received a new so called smartphone. Not that I am new to the smartphone concept, but this phone i rather unique in that it really connects you to internet - all the time.

From before I have used Palm Treo 650, Iphone and now this item. From going to very basic phones, the first one i had couldnt even send sms (only receive) to actually requiring some effort in order to not connecting it to internet.

A couple days ago, I realized that I was becomming more and more a slave of this item. Not the item itself, but being connected to internet. Checking n mail accounts, checking who's online, chatting, tweeting, facebook, friendfeed, linkedin etc etc.

The simple fix for me was to simply disconnecting myself from all this (noise). I shelfed all the smartphones and bought one of the simplest phones I could find. A Nokia 1200. It can't check email, you cant do online chatting, no rss feed and certanly not facebook.

Ironically the simple phone was a timesaver (not only due to less frequent need for charging :) ), and let me spend more time on what I really like. In this time of year it is mostly anything related to step slopes and snow and reading email is not compatible with that. When interneting, I maybe spend 20 minutes and finish all the tasks in that timeperiod, instead of randomly checking via a smartphone n times a day.

Tempus edax rerum.