Return to Revisions

From the spec:

• If the bs=expr operand is specified and no conversions other than sync, noerror, or notrunc are requested, the data returned from each input block shall be written as a separate output block; if the read() returns less than a full block and the sync conversion is not specified, the resulting output block shall be the same size as the input block.

So this is probably what causes your confusion. Yes, because dd is designed for blocking, by default partial read()s will be mapped 1:1 to partial write()s, or else syncd out on tail padding NUL or space chars to bs= size when conv=sync is specified.

However, this is only true when bs= is specified or obs= is not specified, as the very next sentence in the spec states:

• If the bs=expr operand is not specified, or a conversion other than sync, noerror, or notrunc is requested, the input shall be processed and collected into full-sized output blocks until the end of the input is reached.

Without ibs= and/or obs= arguments this can't matter - because ibs and obs are both the same size by default. However, you can get explicit about input buffering by specifying different sizes for either and not specifying bs= (because it takes precedence).

For example, if you do:

IN| dd ibs=1| OUT

...then a POSIX dd will write() in chunks of 512 bytes by collecting every singly read() byte into a single output block.

Otherwise, if you do...

IN| dd obs=kxk| OUT

...a POSIX dd will read() at maximum 512 bytes at a time, but write() every megabyte-sized output block (excepting possibly the last - because that's EOF) in full by collecting input into full-sized output blocks.

Also from the spec, though:

• count=n
  • Copy only n input blocks.

count= maps to [i]?bs= blocks, so let's change it up a little bit and try your yes thing again with an actual blocking factor. Like:

yes | dd ibs=k obs=kxk |     #input bs = 1k, ouput bs= 1k * 1k
dd bs=kxk count=10 of=out    #input and output bs = 1k * 1k

If you have a POSIX dd on site, your results will be:

10+0 records in
10+0 records out
10485760 bytes (10 MB) copied, 0.127955 s, 81.9 MB/s
$ ls -l out
-rw-r--r-- 1 mikeserv mikeserv 10485760 Mar 23 17:37 out

You have to synchronize i/o w/ dd to handle non-seekable inputs. In other words, make pipe-buffers explicit and they cease to be a problem. That's what dd is for. I suggest you read through the POSIX spec on dd. Particularly important is that you learn the different implications of any declarations for [ioc]?bs=.

In the above case the unknown quantity is the number of bytes yes write()s to the pipe each time. dd is spec'd for blocking out inputs, and so its [i]?bs= argument - when obs= or certain cbs= conversions are not specified - will result in input blocks being copied whole and singly from input to output. This means that count=10 results in only 10 read()s from the pipe and likewise 10 write()s to out.

However, when explicitly specified, dd will buffer input blocks to aggregate into a single obs= sized output block. And so you just need a second dd and a blocking factor. Tell the first dd to aggregate as many input blocks as are necessary to write your 1M block-size requirement, and from that point on both dds will synchronize their i/o - one writing 1M blocks to the pipe, the second reading them in the same size and counting up to 10 1M blocks written into out.

Of course, if you knew certainly the size of the writes yes was doing you wouldn't even need two dds because you could just synchronize dd w/ yes instead. But when you can't know, dd is there. That's why we have it. Look:

yes | dd ibs=333333 obs=333333x44 | 
dd bs=333333x44 count=10 of=/dev/null

10+0 records in
10+0 records out
146666520 bytes (147 MB) copied, 1.69232 s, 86.7 MB/s

See? It works with i/o blocks of any shape or size, as long as the system's definition for (at least) PIPE_BUF will allow it. The above examples were run on a linux system, but may not work the same on others, specifically because the system kernel may not guarantee atomic read()s and write()s at such high capacities, but, if it is a POSIX system, it will guarantee such atomic things are atomic within the limits of PIPE_BUF as defined in limits.h. POSIX guarantees that PIPE_BUF be at least 512 bytes (which also happens to be the default dd i/o blocksize).

This is where the idea of a block factor comes in: it can be applied in several ways, and is always a multiple of some common denominator. For example, the exact same output as before can be achieved more portably like:

yes | dd obs=44 | dd bs=44 count=3333330 of=/dev/null

3333330+0 records in
3333330+0 records out
146666520 bytes (147 MB) copied, 2.5654 s, 57.2 MB/s

...where we're still working with the product of 333333, 44 and 10. The above case is probably overly conservative - if you don't want to check limits.h yourself, it is probably safe in most conditions to assume at least 4k can be written/read atomically across either end of the pipe, and therefore synchronized between our blocking dd processes.

Here's a snippet from the POSIX spec:

• ibs=expr
  • Specify the input block size, in bytes, by expr (default is 512).
• obs=expr
  • Specify the output block size, in bytes, by expr (default is 512).
• bs=expr
  • Set both input and output block sizes to expr bytes, superseding ibs= and obs=. If no conversion other than sync, noerror, and notrunc is specified, each input block shall be copied to the output as a single block without aggregating short blocks.

You'll also find some of this explained better here.