Make ganging redundancy respect redundant_metadata property

[pcd1193182]

Motivation and Context

The redundant_metadata setting in ZFS allows users to trade resilience for performance and space savings. This applies to all data and metadata blocks in zfs, with one exception: gang blocks. Gang blocks currently just take the copies property of the IO being ganged and, if it's 1, sets it to 2. This means that we always make at least two copies of a gang header, which is good for resilience. However, if the users care more about performance than resilience, their gang blocks will be even more of a penalty than usual.

Description

We add logic to calculate the number of gang headers copies directly, and store it as a separate IO property. This is stored in the IO properties and not calculated when we decide to gang because by that point we may not have easy access to the relevant information about what kind of block is being stored. We also check the redundant_metadata property when doing so, and use that to decide whether to store an extra copy of the gang headers, compared to the underlying blocks.

There is also a new test to verify that gang headers are stored redundantly in roughly the way that we want, plus some changes to the other gang block tests to make the new test (and other tests) easier to work with.

How Has This Been Tested?

Mostly the new zfs-test suite test, plus similar manual testing.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair, libuutil and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist:

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[amotin]

I have no objections to this part, while base one I haven't looked yet.

PS: GitHub reports some merge conflict.

[pcd1193182]

Here is the change separated out from the dynamic gang header stuff. I'll work on separating out the dynamic allocation changes next, though that one is a little more entwined.

[amotin]

Thank you. Just a small nit:

[amotin]

@pcd1193182 "error: commit message body contains line over 72 characters"

[tonyhutter]

At a high level this PR makes sense as the gang block headers could be considered "metadata" and thus should follow redundant_metdata rules. There's some details to consider though,

redundant_metadata requires an extra copy of the metadata per the actual data:

redundant_metadata=all|most|some|none

Controls what types of metadata are stored redundantly. ZFS stores an extra copy of metadata, so that if a single block is corrupted, the amount of user data lost is limited. This extra copy is in addition to any redundancy provided at the pool level (e.g. by mirroring or RAID-Z), and is in addition to an extra copy specified by the copies property (up to a total of 3 copies). For example if the pool is mirrored, copies=2, and redundant_metadata=most, then ZFS stores 6 copies of most metadata, and 4 copies of data and some metadata.

https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#redundant_metadata

So if we consider gang block headers as redundant_metadata, then with copies=3, we should actually be storing 4 copies of the gang block headers. That would be difficult to do.

Also, ZFS prefers storing redundant_metadata on a special device. Are we currently trying to allocate gang block headers on special class devices? It seems like we should if we're considering them as redundant_metadata.

[pcd1193182]

redundant_metadata requires an extra copy of the metadata per the actual data:
...
So if we consider gang block headers as redundant_metadata, then with copies=3, we should actually be storing 4 copies of the gang block headers. That would be difficult to do.

Sort of, but...

(up to a total of 3 copies).

We don't store more than 3 copies, regardless of the redundant_metadata setting. The only sense in which we do is if mirroring is in play (which, as the man page says, is "redundancy provided at the pool level"). The logic here is the same as for normal metadata blocks, which also respect the copies property (see dmu_write_policy; copies always starts at os->os_copies, whether the block is metadata or not).

Also, ZFS prefers storing redundant_metadata on a special device. Are we currently trying to allocate gang block headers on special class devices? It seems like we should if we're considering them as redundant_metadata.

That's an interesting question. My gut says that yes, we should ideally be sticking gang headers on special devices, if they're available. But that could easily be a separate change. It's not particularly complicated (just an extra call to metaslab_alloc in zio_write_gang_block, but the intent is separate and I'd like to get a little more feedback from other people on whether they agree with the idea without holding this change up.

[tonyhutter]

So if we consider gang block headers as redundant_metadata, then with copies=3, we should actually be storing 4 copies of the gang block headers. That would be difficult to do.

Sort of, but...

  (up to a total of 3 copies).

Ah you're right - I totally misread that. I thought they were talking about the limits on the copies property, not the limits of the copies of the metadata. So it's like:

copies=1: 2 metadata copies
copies=2: 3 metadata copies
copies=3: 3 metadata copies

My gut says that yes, we should ideally be sticking gang headers on special devices, if they're available. But that could easily be a separate change.

That's true, let's consider it for a separate PR.