10

I am tring to create random 1G test file via dd command.

dd status=progress if=/dev/zero of=/tmp/testfile.zer bs=100M count=10
dd status=progress if=/dev/urandom of=/tmp/testfile1.ran bs=100M count=10
dd status=progress if=/dev/urandom of=/tmp/testfile2.ran bs=100M count=20

The output is:

-rw-rw-r-- 1 dorinand dorinand  320M dub 21 12:37 testfile1.ran
-rw-rw-r-- 1 dorinand dorinand  640M dub 21 12:37 testfile2.ran
-rw-rw-r-- 1 dorinand dorinand 1000M dub 21 12:37 testfile.zer

Why is the output testfile generate from /dev/urandom three times smaller? I would expect that the size of testfile1.ran will be 1000M and size of testfile2.ran will be 2000M. Could anybody why this happening? How should I generate random testfile?

Improve this question
1

1 Answer 1

Reset to default
19

With larger blocksize, there is a risk of getting incomplete reads. This also happens a lot when reading from a pipe, rather than a block device.

If you expect to receive a certain size (count*bs) you also have to supply iflag=fullblock.

It might not be necessary for bs=1M or smaller, but it's still recommended either way.

dd will also try to show you how many incomplete reads it got. It copies n+m blocks, n complete and m incomplete ones. When copying files that are not multiple of blocksize, it's normal for the last block to be incomplete.

Example:

$ dd status=progress if=/dev/urandom of=/dev/null bs=100M count=20 
dd: warning: partial read (33554431 bytes); suggest iflag=fullblock
536870896 bytes (537 MB, 512 MiB) copied, 2 s, 254 MB/s
0+20 records in
0+20 records out
671088620 bytes (671 MB, 640 MiB) copied, 2.64391 s, 254 MB/s

In this case it got only incomplete reads and not a single full 100M block. Obviously /dev/urandom is unwilling to serve that much data in a single read. My version of dd even tells you to use iflag=fullbock directly.

With fullblock everything is OK:

$ dd status=progress if=/dev/urandom of=/dev/null bs=100M count=20 iflag=fullblock
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 8 s, 255 MB/s 
20+0 records in
20+0 records out
2097152000 bytes (2.1 GB, 2.0 GiB) copied, 8.22914 s, 255 MB/s

It takes longer because it actually copies more than twice the amount of data.

Improve this answer
8
  • Did you tweak any kernel settings to get that kind of speed from /dev/urandom? Commented Apr 21, 2018 at 18:12
  • 1
    @kasperd I get the same ~250 MB/s on basic Ubuntu kernel on a few years old laptop. Commented Apr 21, 2018 at 19:13
  • @jpa I tried a bunch of different Ubuntu systems, and the fastest I got was 24.3 MB/s Commented Apr 21, 2018 at 19:18
  • @kasperd No, but the urandom algorithm changes from time to time. It was replaced with a significantly faster algorithm not too long ago. If you are using a recent-ish 4.1x.x kernel you should see improvements. Commented Apr 21, 2018 at 22:17
  • 1
    @jpa Yeah, I should probably ask a separate question about upgrading HWE, as it seems to not be included in apt-get dist-upgrade. I guess such a question would be better suited to Ask Ubuntu. Commented Apr 22, 2018 at 11:17

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.