dd be like: yeah, I'm not showing you progress
pv be like: I'm instantly finished, but please let me continue for 15 minutes or else.....

why linux be like that

@kaia Linux is made of autism, with its good and its bad things

@toast I just waited 15 minutes for it to complete

@kaia sync; sync; sync

@kaia > I'm instantly finished, but please let me continue for 15 minutes
That's an really interesting observation
No super expert, but afaik that's due to I/O buffering.
A process ships data from the file system to the kernel which in turn sends the data to the I/O controller that finally writes it to hardware.
Now, pv has no idea whatever happens after it has sent the data to the kernel, since there is no feedback other than "process has terminated successfully" after the I/O buffer has been emptied.
If pv has sent all data to the kernel, but the process has not finished yet, it looks like the data transfer is finished but all that has happened is that all the data has been put into the I/O buffer waiting for the controller to work it down.
When the connection to the I/O peripheral is dropped while the buffer is non-empty you will have a partial transfer with probably some file corruption for the data that had been written mid-connectiondrop.

@lain this is the way.

status=LEVEL

         The  LEVEL  of information to print to stderr; 'none' suppresses everything but error messages,  'noxfer'  suppresses  the  final transfer  statistics, 'progress' shows periodic transfer statistics

@kaia status=progress for dd, idk what pv does, access like that should be uncached

@kaia btw, if you haven't read it yet, this book is hilarious and educational, too: https://web.mit.edu/~simsong/www/ugh.pdf

@lain @kaia I see, somebody used an ancient UNIX system here.

@snacks @kaia It's because pipes and I/O are buffered by default. It throws a gigabyte into the I/O buffer and then stalls for some time. With dd, you should be able to bypass that with oflag=direct.

@kaia
>dd be like: yeah, I'm not showing you progress
There's the "status=progress" option.
Example:

dd if=.iso of=/dev/sd* bs=4M status=progress

@kaia >Linux
`dd --version`

Here, have a GNU bash function that tells you how much data Linux has claimed to have written, but hasn't in fact done so yet;
watchSync() {
watch -n1 'grep -E "(Dirty|Write)" /proc/meminfo; echo; ls /sys/block/ | while read device; do awk "{ print \"$device: \" \$9 }" "/sys/block/$device/stat"; done'
}

@lain @kaia There is no point running sync more than once.

@lain @kaia The book is wrong and is about seething over skill issues.

The foreword is by some genius that doesn't even realize he swapped to a Unix by surrendering his freedom to macos.


Also, GNU's Not Unix.

@Suiseiseki @kaia That is absolutely dreadful shell and you should be ashamed of yourself

@ullard @kaia I didn't even write it, but it is indeed quite well written.

@kaia for peak linuxy you should be piping dd to pv

@kaia and for peak windowsy you can look at the writes in a process monitor instead

@Suiseiseki @kaia @lain

I've been told that sync is in fact asynchronous, meaning it can return before the data it began flushing is done being flushed. But if you call it again, it must wait for the previous operation to finish before triggering another. So calling it twice ensures that the first run of sync has fully completed.

@taylan @kaia @lain >meaning it can return before the data it began flushing is done being flushed
Cursed - of course Linux would go full retard.

In my experience GNU sync has always blocked until data has been flushed.

So it needs to be `sync ; sync` to be safe huh?

@Suiseiseki @kaia @ullard

It would probably be better to use a for loop with a glob instead of piping ls into a while loop.

@Suiseiseki @kaia @lain

Hmm, according to this man page, it's actually synchronous on Linux:

https://man7.org/linux/man-pages/man2/sync.2.html

Apparently, the problem I've described is based on the guarantees (not) made by the POSIX standard.

So I guess it's important to call it twice in cross platform scripts, but on GNU+Linux it shouldn't matter.

In fact, based on the following, it seems that even the hack of calling it multiple times may not help, because this doesn't describe the behaviour that I mentioned earlier, of subsequent calls having to wait for the first to finish:

https://pubs.opengroup.org/onlinepubs/9799919799/functions/sync.html

@taylan @kaia @lain GNU's not POSIX.

Scripts working fine on GNU/Linux-libre, but not on proprietary Unix-likes is a feature - so single `sync` all the way.


Such bug existed in Linux 1.3.20, but it doesn't exist anymore.

@taylan @kaia @Suiseiseki @ullard Glob glob glob. 🫧 🐟

@kaia

dd if=/dev/urandom of=/dev/null status=progress