2

I have a monster server I'm provisioning at the minute.

There are 15 x 8TB HDD's connected to a SATA interface card that I'm using to create a ZFS volume. The drives are all detected and I've got the ZFS libraries installed and they're ready etc.

I created my ZFS volume using zpool. I opted for RAIDZ2, as I want double parity for the extra fault tolerance.

zpool create -f diskpool1 raidz2 sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm sdn sdo sdp sdq

and if I check the status, that's all good:

[root@BACKUPNAS-I ~]# zpool status
pool: diskpool1
state: ONLINE
scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    diskpool1   ONLINE       0     0     0
      raidz2-0  ONLINE       0     0     0
        sdc     ONLINE       0     0     0
        sdd     ONLINE       0     0     0
        sde     ONLINE       0     0     0
        sdf     ONLINE       0     0     0
        sdg     ONLINE       0     0     0
        sdh     ONLINE       0     0     0
        sdi     ONLINE       0     0     0
        sdj     ONLINE       0     0     0
        sdk     ONLINE       0     0     0
        sdl     ONLINE       0     0     0
        sdm     ONLINE       0     0     0
        sdn     ONLINE       0     0     0
        sdo     ONLINE       0     0     0
        sdp     ONLINE       0     0     0
        sdq     ONLINE       0     0     0

errors: No known data errors

But if I check the disk space, I'm only showing 87Tb :(

[root@BACKUPNAS-I ~]# df -h
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   50G  3.0G   47G   6% /
devtmpfs                 7.7G     0  7.7G   0% /dev
/dev/mapper/centos-home  154G   54M  154G   1% /home
/dev/md126p1             497M  188M  310M  38% /boot
diskpool1                 87T  256K   87T   1% /diskpool1

I know the raid levels vary from the RAID standards when using ZFS, due to its origins, however I anticipated that I'd have around 104TB usable with a RAID6-like configuration, giving me a fault tolerance of 2 disks in the pool.

Am I not doing something correctly, or is it simply the case that using what is essentially 'software RAID' with ZFS uses takes up a lot of space (2 or so of my 8TB HDD's worth!)

Improve this question
6
  • I'd suggest you to update your title. The current one is not at all expressive, as it doesn't tell what your problem is. Commented Mar 23, 2016 at 13:17
  • Are you using ZFS on the disks or are the disks partitioned and you run ZFS on the partitions? Unrelated side note: 15*8TB in a RAIDZ2 will take ages to resilver. I would recommend splitting that up into two vdevs (maybe add one or three additional disks). Commented Mar 23, 2016 at 14:43
  • Are these 8 TiB drives (8*2^40 or 8*1024^4 or 8,796,093,022,208 bytes) or 8 TB (8*10^12 or 8*1000^4 or 8,000,000,000,000 bytes) drives? most likely the latter because marketers at hard disk companies long ago decided to specify drives in powers of 10 or 1000 rather than powers of 2 or 1024 because 8TB looks bigger than 7.27TiB. Try df -H rather than df -h. read man df to discover that -h is powers-of-1024 while -H is powers-of-1000. see en.wikipedia.org/wiki/Tebibyte Commented Mar 24, 2016 at 2:06
  • BTW, this is worth a read for anyone thinking about using ZFS jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz Commented Mar 24, 2016 at 2:27
  • @cas, if it were just a metric-vs-imperial-terabyte issue, df would be showing 94.5 TB, not 87TB. Commented Apr 13, 2016 at 23:42

1 Answer 1

Reset to default
2

Looks like there's two problems.

First hard drives are in TB (1000^4) while df reports in TiB (1024^4). Calculating your expected 104TB to TiB gets you to 94.5TiB. Still off.

But you said you had two drives, so (15*8)-(2*8)=104TB. What if you had a third drive in for redundancy or not in use? Then you get (15*8)-(3*8)=96TB. Convert to TiB and you get 87.3TiB which is what df is reporting to you.

Improve this answer
2
  • yes, this vdev looks like it's raidz-3 rather than raidz-2. try zpool history | grep 'zpool create' for the exact command line used to create it. Commented Mar 24, 2016 at 2:36
  • So indeed the difference between TB and TiB makes up for a decent chunk of my missing space. The rest, I've found, is due to 'padding' with ZFS and there isn't a whole lot that can be done about it. Commented Mar 30, 2016 at 9:10

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.