Crypto on the Command Line, Part 4: Public-Key Encryption

calendar Oct 12, 2026 · 5 min read · security cryptography linux  ·
Share on: linkedin copy

Anyone holding Bob's public key can encrypt to him, and only his private key decrypts. No secret has to travel first. This post runs the public-key scene of the Security Primitives tool with openssl pkeyutl:

  • encrypting a message to bob.pub and decrypting it with bob.key;
  • trying the public key on the ciphertext, as Eve would;
  • the size limit, which a document is far over;
  • key wrapping, step by step: a fresh AES key for the document, and RSA for the key.

The tools on Ubuntu 26.04

The commands are shown as a terminal session:

  • $ starts a command: type what follows it;
  • > marks a line that continues the command above it, for a command too long for one line;
  • a line with neither is output. A command with no lines under it prints nothing.

openssl (3.5 on 26.04) and xxd, and Bob's key pair from Part 3 :

1$ sudo apt update
2$ sudo apt install openssl xxd
3$ mkdir -p ~/crypto-lab && cd ~/crypto-lab
4$ openssl genpkey -quiet -algorithm RSA -out bob.key     # skip if Part 3 made it
5$ openssl pkey -in bob.key -pubout -out bob.pub

Encrypting to the public key

pkeyutl runs one public-key operation. -pubin says the key file is a public key, and the two -pkeyopt options choose OAEP padding with SHA-256:

1$ printf 'pay carol 10' > msg.txt
2$ openssl pkeyutl -encrypt -pubin -inkey bob.pub \
3>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 \
4>   -in msg.txt -out msg.rsa
5$ wc -c msg.rsa
6256 msg.rsa
  • The ciphertext is as long as the key: 2048 bits, 256 bytes. RSA turns one number smaller than n into another.
  • OAEP pads the message with random bytes before the arithmetic. Run the command twice and the two ciphertexts differ.
  • That matters: without it, Eve could encrypt pay carol 10 to bob.pub herself and compare.
  • pkeyutl's default padding is the older PKCS#1 v1.5, open to known attacks on decryption; name OAEP.

Trying the public key

Eve holds bob.pub too. It does not decrypt:

1$ openssl pkeyutl -decrypt -pubin -inkey bob.pub \
2>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -in msg.rsa
3A private key is needed for this operation
4pkeyutl: Error loading key

openssl refuses before trying, but the arithmetic would fail anyway: undoing the public exponent e takes the private exponent d, and d is not in bob.pub.

Decrypting with the private key

1$ openssl pkeyutl -decrypt -inkey bob.key \
2>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 -in msg.rsa
3pay carol 10

The size limit

RSA encrypts one number, so the message has to fit in one. With a 2048-bit key and OAEP over SHA-256, that is 256 − 2 × 32 − 2 = 190 bytes:

1$ cp /usr/share/common-licenses/GPL-3 doc.txt
2$ head -c 190 doc.txt | openssl pkeyutl -encrypt -pubin -inkey bob.pub \
3>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | wc -c
4256
5$ head -c 191 doc.txt | openssl pkeyutl -encrypt -pubin -inkey bob.pub \
6>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 | wc -c
7Public Key operation error
8...:error:0200006E:rsa routines:ossl_rsa_padding_add_PKCS1_OAEP_mgf1_ex:data too large for key size:...
90
  • The 35,149-byte licence would take 185 separate RSA operations.
  • RSA is also hundreds of times slower than AES per byte: openssl speed -seconds 1 rsa2048 aes-256-cbc measures both.

Key wrapping

So RSA carries one small thing, a fresh AES key, and AES carries the document. Each step is one command.

Alice makes a key and an IV for this one document (teal.key, 32 raw bytes, not hex this time):

1$ openssl rand 32 > teal.key
2$ openssl rand 16 > doc.iv

She encrypts the document with AES ( Part 2 ); xxd -p turns the raw bytes into the hex that -K and -iv take:

1$ openssl enc -aes-256-cbc -K $(xxd -p -c 32 teal.key) -iv $(xxd -p doc.iv) \
2>   -in doc.txt -out doc.enc

She wraps the key: encrypts its 32 bytes to bob.pub:

1$ openssl pkeyutl -encrypt -pubin -inkey bob.pub \
2>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 \
3>   -in teal.key -out teal.wrapped
4$ wc -c doc.enc doc.iv teal.wrapped
535152 doc.enc
6   16 doc.iv
7  256 teal.wrapped
835424 total

She sends those three files, and deletes teal.key. Eve copies all three, and none helps her: the IV is not secret, and the key inside teal.wrapped opens only with bob.key.

Bob, in a directory of his own, unwraps the key with his private key, then decrypts the document with it:

1$ mkdir -p bob && cp doc.enc doc.iv teal.wrapped bob/ && cd bob
2$ openssl pkeyutl -decrypt -inkey ../bob.key \
3>   -pkeyopt rsa_padding_mode:oaep -pkeyopt rsa_oaep_md:sha256 \
4>   -in teal.wrapped -out teal.key
5$ openssl enc -d -aes-256-cbc -K $(xxd -p -c 32 teal.key) -iv $(xxd -p doc.iv) \
6>   -in doc.enc -out doc.txt
7$ sha256sum doc.txt
83972dc9744f6499f0f9b2dbf76696f2ae7ad8af9b23dde66d6af86c9dfb36986  doc.txt
9$ cd ..

The digest is the licence's from Part 1 : Bob has the document, to the bit.

  • This is how encrypted mail and files work. gpg --encrypt and openssl cms -encrypt run these steps in one command, and pack the wrapped key, the IV and the ciphertext into one file.
  • A message to three readers is encrypted once, with its key wrapped three times, once per public key.
  • TLS once agreed its keys this way (RSA key exchange). TLS 1.3 uses key agreement instead, Part 7 .