37
votes
Accepted
snap /dev/loop at 100% utilization -- no free space
No, having Snap images which consume 100% of their filesystem is perfectly acceptable. In fact, it's supposed to work that way.
A snap is a squashfs file carrying content and a bit of metadata that ...
- 7,496
27
votes
Accepted
Best way to delete large amount of files by date
If the files are not modified after initial creation, you could delete if they have not been modified in over 90 days:
find /path/to/folder ! -type d -mtime +90 -delete
or
find /path/to/folder ! -...
- 9,415
19
votes
Check data integrity after copying thousands of files
Using MD5 sums is a good way, but the canonical way to use it is:
cd to the directory of the source files and issue:
md5sum * >/path/to/the/checksumfile.md5
If you have directories with many ...
- 9,328
19
votes
Accepted
Is my drive failing and is this the same reason my laptop is running slower?
Yes, your drive is failing:
5 Reallocated_Sector_Ct 0x0033 001 001 005 Pre-fail Always FAILING_NOW 9800
There’s nothing you can do to make the drive “better”. What you need to do next ...
- 481k
16
votes
dd backup fails with "File too large" error despite ample free space on FAT32 partition
dd fails because your FAT filesystem does not support large files.
The obvious solution is to use a different filesystem.
The alternative is to write multiple files.
For this, you can use split:
sudo ...
- 52.1k
15
votes
Persistent disk name /dev/sd'x', changing with almost every reboot
/dev/sdX has not been a stable identifier for a drive for a very long time (and indeed probably never was). Those are allocated in the order they're discovered, and different controllers are probed in ...
- 113k
14
votes
Accepted
Check data integrity after copying thousands of files
Unmount, eject, and remount the device. Then use
diff -r source destination
In case you used rsync to do the copy, rsync -n -c might be very convenient, and it is nearly as good as diff. It doesn't ...
- 53.5k
12
votes
Accepted
Inaccessable partition with 477GB
nvme0n1 is not a partition, it is an entire disk: "NVMe controller #0, namespace #1". Unless you have some serious enterprise hardware, your NVMe disks have only a single namespace on them.
...
- 114k
11
votes
Accepted
tmpfs usage and resizing
Now, from what I read the tmpfs doesn't take physical storage, but uses the virtual memory of the machine. Is it correct?
Correct. tmpfs appears as a mounted file system, but it's stored in volatile ...
- 2,899
11
votes
Accepted
persistent device naming for NVMe storage devices
Short: No. You can not rely on the name of the descriptor. And you most likely never will.
The NVMe naming standard describes:
nvme0: first registered device's device controller
nvme0n1: first ...
- 156
11
votes
Accepted
What is the "~/.stack" directory?
From https://docs.haskellstack.org/en/stable/GUIDE/
The stack root directory holds packages belonging to snapshots and any stack-installed versions of GHC.
You can safely remove any or all of it, ...
- 246
10
votes
dd backup fails with "File too large" error despite ample free space on FAT32 partition
FAT32¹ only uses 32-bits for file sizes in directory entries, which means that any one file can be at most 2^32-1 bytes long (one byte short of 4 GiB). There is no way to get around this limitation in ...
- 13.5k
9
votes
Accepted
why is the internal storage mmc and not sd?
/dev/sdX indeed once meant SCSI disk, but nowadays it covers anything with a SCSI-like interface, including SATA, USB storage and even old PATA when libata drivers are used. Some actually have ...
- 114k
8
votes
Clear unused space with zeros (btrfs)
The correct and "smart" way to clear the free space is using:
fstrim /mountpoint
Use this on a mounted filesystem.
It's important to note that there is no security advantage to this. If you wish to ...
- 222
8
votes
Check data integrity after copying thousands of files
rsync -rc original-dir/ copied-dir/
-c causes rsync to compare files by MD5 checksum (without it, it normally uses only the timestamp and size for quicker comparisons).
This will also cause rsync to ...
- 5,019
8
votes
Replicating an SD card
If you're just looking to copy an SD card exactly from one to another then you can do so with dd on the command line.
You should NOT do this from your raspberry pi from it's own OS. This is because ...
- 21k
8
votes
Accepted
Is it true that rsync does not handle "File renames" gracefully?
There is no mechanism for rsync to track renames as it does not maintain state other than while it is running. If you rename /test/10GBfile to /test/10GBfile_newname on the source computer, then by ...
- 128k
8
votes
Is it possible to replace the disks in LVM raid with larger disks?
Yes, this is possible, but you need to follow the instructions in the chapter on Replacing a failed RAID device in a logical volume:
Back up your data.
Power down the system, remove one disk, add a ...
- 481k
6
votes
Accepted
df command shows 100% full, even after deleting files it shows same usage (100%)
A reboot and re-mount would solve the problem I believe.
Reason: The reason behind this is that df utilize statfs(2) system call to get the File system stat. What it means that it checks open kernel ...
- 1,599
6
votes
How to make journald increase logs storage capacity?
It's necessary to setup SystemMaxUse=100G e.g.
- 255
6
votes
Accepted
Report how much space is used and available in storage in ZFS on FreeBSD
zfs list
Use the list option on the zfs command built into FreeBSD.
zfs list
Example:
$ zfs list
NAME USED AVAIL REFER MOUNTPOINT
zroot 4.41G 17.4G 88K none
...
- 1,691
5
votes
Clear unused space with zeros (ext3,ext4)
if you don't want to resort to additional tools this method should be faster than cat /dev/zero > /mnt/fs/zeros:
dd if=/dev/zero of="/mnt/fs/filler" bs=1048576
dd if=/dev/zero of="/...
- 1,733
5
votes
Accepted
Mirror Linux disk writes to secondary disk
You can use a MD RAID1 with the option --write-mostly.
subsequent devices listed in a --build, --create, or --add command will be flagged as 'write-mostly'. This is valid for RAID1 only and means ...
- 9,239
5
votes
Check data integrity after copying thousands of files
Along with the other fine answers above, I would like to also recommend considering hashdeep, from http://md5deep.sourceforge.net/. It has a good sized userbase in the scientific community, where they ...
- 150
5
votes
Is it true that rsync does not handle "File renames" gracefully?
--fuzzy has already been answered but there is another interesting hack involving hard links.
After the first transfer
$ rsync -avHP --delete-after ~/family/Photos remotebox:backups
You create a hard ...
- 6,353
5
votes
Accepted
dd faster than it should be
What you're observing is called "Dirty Pages" and is a feature of most modern Operating Systems. The kernel doesn't physically write data to the disk as soon as a file is written to - this ...
- 32.9k
5
votes
what is "dd if=/dev/zero of=/storage2/test4.img bs=1M count=256" usage?
It creates an 256 MB file filled with zeros in the directory /storage2. This can be used as a disk image and formatted and mounted as any disk using a loop device.
See:
man dd
man mount
Loop device
...
- 2,511
4
votes
Protecting data against bit rot
This is an older question, but still relevant in 2019.
Yes, parity files are a viable solution to bit-rot
While there are detriments to non-filesystem-level parity checking, there is also one huge ...
- 141
4
votes
Rebuilding mdadm RAID 5 array with multiple failed drives
You can't reassemble without data loss a RAID-5 array with more than one drive dead. In theory, you could read data from the rest of the drives, but you'd have chunks missing in between.
The ...
- 148k
4
votes
Check data integrity after copying thousands of files
It is possible to generate hashsums for individual files and output them into one text file, of which the MD5 hash can be generated. For that text file, you can use any hash function you like because ...
- 1,720
Only top scored, non community-wiki answers of a minimum length are eligible