Crypto on the Command Line, Part 1: Hash Functions

A hash function turns a file of any size into a short digest, and the commands that compute one are already on every Linux machine. This post runs the hashing scenes of the Security Primitives tool by hand:

• hashing a real document, a few thousand words long;
• changing one bit of it, and counting how much of the digest changes;
• checking a file against a digest published beside it;
• guessing the input back from a digest, which works only when the input is short.

The posts follow the tool's parties: Alice and Bob at the two ends, Eve reading the wire and Mallory changing it. The names are cryptography's standard cast, each standing for its role.

The tools on Ubuntu 26.04

The commands are shown as a terminal session:

• $ starts a command: type what follows it;

• > marks a line that continues the command above it, for a command too long for one line;

• a line with neither is output. A command with no lines under it prints nothing.

• sha256sum, sha512sum, b2sum, md5sum come with coreutils, installed on every Ubuntu.

  • Since 25.10 Ubuntu ships the Rust rewrite of coreutils (uutils); sha256sum --version says uutils coreutils. The output is the same as GNU's.

• openssl is installed on the desktop and server images. A container or minimal image may lack it:

  1$ sudo apt update
  2$ sudo apt install openssl xxd
  

• xxd prints bytes as hex. It is its own package (xxd), installed with vim, so it may be missing.

Every command below runs in a scratch directory:

1$ mkdir -p ~/crypto-lab && cd ~/crypto-lab

A document to hash

Every Ubuntu carries the text of the GNU GPL, version 3: 674 lines, about twelve printed pages.

1$ cp /usr/share/common-licenses/GPL-3 doc.txt
2$ wc -l -c doc.txt
3  674 35149 doc.txt

Hashing a file

sha256sum reads the file and prints its SHA-256 digest in hex, then the file's name:

1$ sha256sum doc.txt
23972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986  doc.txt

• Deterministic: the same file gives this digest on every machine, so yours matches the one above.
• Keyless: no key or password went in, so anyone can compute it.
• openssl dgst -sha256 doc.txt computes the same digest and prints it as SHA2-256(doc.txt)= 3972dc97….

The digest's fixed size

SHA-256 gives 32 bytes, whatever goes in. The 64 hex characters above are those 32 bytes, two characters each:

1$ openssl dgst -sha256 -binary doc.txt | xxd
200000000: 3972 dc97 44f6 499f 0f9b 2dbf 7669 6f2a  9r..D.I...-.vio*
300000010: e7ad 8af9 b23d de66 d6af 86c9 dfb3 6986  .....=.f......i.

An empty input gives 32 bytes too. /dev/null is a device file that reads as empty: zero bytes, then the end of the file.

1$ sha256sum /dev/null
2e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855  /dev/null

So does a large one. /dev/zero is a device file that reads as an endless run of zero bytes:

1$ head -c 100M /dev/zero > big.bin
2$ time sha256sum big.bin
320492a4d0d84f8beb1767f6616229f85d44c2827b64bdbfb260ee12fa1109e0e  big.bin
4
5real	0m0.496s

• head -c 100M copies the first 100 MiB (104,857,600 bytes) of it into big.bin.
• time runs the command after it and prints how long it took: real is the time on the clock.
• 100 MB hashed in half a second: SHA-256 is fast, and modern CPUs have instructions for it.
• rm big.bin deletes the file afterwards.

So the digest keeps almost nothing of what went in:

• 35,149 bytes of licence went in and 32 came out, so almost all of the information is gone.
• So a digest is not a compressed copy, as a zip file is: nothing can rebuild the document from it.

One changed bit

A digest still stands for its document, because any change to the input gives a very different digest. The licence's second line holds its date; change 29 June to 28 June:

1$ sed '2s/29 June/28 June/' doc.txt > doc2.txt
2$ cmp -l doc.txt doc2.txt
3   83  71  70

• cmp -l lists each differing byte: byte 83, octal 71 in the first file (9, hex 39) and 70 in the second (8, hex 38).
• 0x39 is 0011 1001 and 0x38 is 0011 1000: one bit of the 281,192 in the file changed.

1$ sha256sum doc.txt doc2.txt
23972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986  doc.txt
342ce29fbda3461960d22dc866de007a54e264e7c1873ab62aace0f97f9816eb4  doc2.txt

A short loop counts the differing bits of the two digests, one hex digit (four bits) at a time:

1$ a=$(sha256sum < doc.txt | cut -c1-64)
2$ b=$(sha256sum < doc2.txt | cut -c1-64)
3$ n=0
4$ for i in $(seq 0 63); do
5>   x=$(( 0x${a:i:1} ^ 0x${b:i:1} ))
6>   while (( x )); do n=$(( n + (x & 1) )); x=$(( x >> 1 )); done
7> done
8$ echo "$n of 256 bits differ"
9119 of 256 bits differ

• About half the bits flip, which is what a digest of unrelated input would give: nothing in the second digest hints that the documents are nearly the same.
• So two equal digests mean two equal files, and comparing 32 bytes replaces comparing the whole document.

Checking a file against a published digest

sha256sum writes its output in a form it can read back with -c:

1$ sha256sum doc.txt > doc.sha256
2$ sha256sum -c doc.sha256
3doc.txt: OK

With the changed file in its place, the check fails and the exit code is 1, so a script can stop on it:

1$ cp doc2.txt doc.txt
2$ sha256sum -c doc.sha256; echo "exit $?"
3doc.txt: FAILED
4sha256sum: WARNING: 1 computed checksum did NOT match
5exit 1
6$ cp /usr/share/common-licenses/GPL-3 doc.txt

• This is how a download is checked. Ubuntu publishes a SHA256SUMS file beside its ISO images, and sha256sum -c SHA256SUMS --ignore-missing checks the one image downloaded.
• A digest says nothing about who made it. Whoever can replace the image can replace SHA256SUMS too, as Mallory does in the tool's second scene. So Ubuntu signs it as well (SHA256SUMS.gpg), and Part 5 shows what a signature adds.

The trailing newline

echo adds a newline, and the newline is hashed with the rest:

1$ echo hello | sha256sum
25891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03  -
3$ printf hello | sha256sum
42cf24dba5fb0a30e26e83b2ac5b9e29e1b161e5c1fa7425e73043362938b9824  -

• echo hello | xxd shows the sixth byte, 0a.
• The - in place of a file name means standard input.
• Use printf (or echo -n) to hash a string as a program would.

Other hash functions

MD5 and SHA-1 still catch a corrupted download, but nothing else: an attacker can make a second file with the same digest. openssl list -digest-algorithms lists every digest openssl knows.

Guessing the input back

A digest is one-way: there is no calculation from it back to the input, only guessing inputs and hashing each. That works when the input is short. Here is the digest of a 4-digit PIN, and a loop that tries all 10,000:

1$ target=$(printf 4827 | sha256sum | cut -c1-64)
2$ time for p in $(seq -w 0 9999); do
3>   [ "$(printf $p | sha256sum | cut -c1-64)" = "$target" ] && echo "found $p" && break
4> done
5found 4827
6
7real	0m28.093s

• The loop starts a new process for every guess; a program hashing in memory tries millions a second.
• So a digest hides a 60-page document, and hides nothing about a PIN or a short password. How passwords are stored is a subject of its own, for a later post.