Return to Answer

This is an older question, but still relevant in 2019.

Yes, parity files are a viable solution to bit-rot

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:

Portability.

The fanciest filesystem-based error-checking in the world could theoretically keep your data bitrot-free forever, but protection is locked to that filesystem. As soon as files leave they are unprotected.

"too heavy"by brapps is licensed under CC BY-NC 2.0 (He should really switch to ZFS and a roof before putting the data in)

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write incorrect metadata, for example)

PAR2 is a viable choice of parity file

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

PAR files are not a complete backup solution on thier own, however

They still need to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (an off-site backup solution with built-in change history would be the winner here).

Next steps

The answer to the original question of “Is there any script or program that would [automatically create and maintain parity files]?” is still No in 2019, but making a script yourself or having one made is fairly trivial. If you go this route the best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.

And, if anyone reading this does go this route, consider open-sourcing it for the benefit of everyone.

This is an older question, but still relevant in 2019.

Yes, parity files are a viable solution to bit-rot

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:

Portability.

The fanciest filesystem-based error-checking in the world could theoretically keep your data bitrot-free forever, but protection is locked to that filesystem. As soon as files leave they are unprotected.

"too heavy"by brapps is licensed under CC BY-NC 2.0 (He should really switch to ZFS and a roof before putting the data in)

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write incorrect metadata, for example)

PAR2 is a viable choice of parity file

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

PAR files are not a complete backup solution on thier own, however

They still need to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (an off-site backup solution with built-in change history would be the winner here).

Next steps

The answer to the original question of “Is there any script or program that would [automatically create and maintain parity files]?” is still No in 2019, but making a script yourself or having one made is fairly trivial. If you go this route the best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.

And, if anyone reading this does go this route, consider open-sourcing it for the benefit of everyone.

Edited addendum: The technology used for PAR files is https://en.wikipedia.org/wiki/Reed–Solomon_error_correction , and while PAR development has stalled, there are dozens if not hundreds of projects using Reed-Solomon for file-level data integrity/repair.

This is an older question, but still relevant in 2019.

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:

Portability.

The fanciest filesystem-based system in the world could theoretically keep your data bitrot-free forever, but it's locked to that filesystem. As soon as it leaves it's unprotected.

"too heavy"by brapps is licensed under CC BY-NC 2.0 (He should really switch to ZFS and a roof before putting the data in)

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write, for example)

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

It still needs to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (off-site backup with built-in change history would be the winner here).

The original question of "is there something that would do this" is answered still with No, but making a script yourself or having one made is fairly trivial. The best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.

This is an older question, but still relevant in 2019.

Yes, parity files are a viable solution to bit-rot

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:

Portability.

The fanciest filesystem-based error-checking in the world could theoretically keep your data bitrot-free forever, but protection is locked to that filesystem. As soon as files leave they are unprotected.

"too heavy"by brapps is licensed under CC BY-NC 2.0 (He should really switch to ZFS and a roof before putting the data in)

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write incorrect metadata, for example)

PAR2 is a viable choice of parity file

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

PAR files are not a complete backup solution on thier own, however

They still need to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (an off-site backup solution with built-in change history would be the winner here).

Next steps

The answer to the original question of “Is there any script or program that would [automatically create and maintain parity files]?” is still No in 2019, but making a script yourself or having one made is fairly trivial. If you go this route the best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.

And, if anyone reading this does go this route, consider open-sourcing it for the benefit of everyone.

This is an older question, but still relevant in 2019.

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:

Portability.

The fanciest filesystem-based system in the world could theoretically keep your data bitrot-free forever, but it's locked to that filesystem. As soon as it leaves it's unprotected.

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write, for example)

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

It still needs to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (off-site backup with built-in change history would be the winner here).

The original question of "is there something that would do this" is answered still with No, but making a script yourself or having one made is fairly trivial. The best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.

This is an older question, but still relevant in 2019.

While there are detriments to non-filesystem-level parity checking, there is also one huge benefit:

Portability.

The fanciest filesystem-based system in the world could theoretically keep your data bitrot-free forever, but it's locked to that filesystem. As soon as it leaves it's unprotected.

"too heavy"by brapps is licensed under CC BY-NC 2.0 (He should really switch to ZFS and a roof before putting the data in)

Images with parity files next to them can be moved to USB drives, transferred over packlossy connections, compressed, backed up individually to other media, and a hundred other things that I can think up off the top of my head.

They can also prevent changes based on bad programs (metadata analyzers that accidentally write, for example)

PAR2 files particularly can be used cross-platform, and their biggest detriments are not actually a problem in the use-case of protecting large photos (those being that PAR2 doesn't support subfolders, and doesn't handle <1KB files efficiently).

It still needs to be part of a 3-copy backup strategy to be "bulletproof", especially as with PAR files there's no change history: If a change is detected as valid, and the parity files recreated, there's no going back without referencing a separate backup (off-site backup with built-in change history would be the winner here).

The original question of "is there something that would do this" is answered still with No, but making a script yourself or having one made is fairly trivial. The best advice I can give is: create a test environment with every edge case you can think of, and run any script through all tests before trusting it.