lib: Add post-quantum cryptography support

This adds initial support for runtime-configurable encryption and
post-quantum Key Encapsulation Mechanisms (KEMs) and authentication
(ML-DSA).

Supported key exchange algorithms:

  ECDH: prime256v1, secp384r1, secp521r1, X25519, X448
  Finite Field DH: ffdhe2048, ffdhe3072, ffdhe4096
  ML-KEM (FIPS 203): ML-KEM-512, ML-KEM-768, ML-KEM-1024
  Hybrid KEMs: X25519MLKEM768, X448MLKEM1024

Supported ciphers:
  AEAD: aes-128-gcm, aes-192-gcm, aes-256-gcm, chacha20-poly1305
  CTR: aes-128-ctr, aes-192-ctr, aes-256-ctr

Supported HKDFs:
  sha256, sha384, sha512, sha3-256, sha3-384, sha3-512,
  blake2b512, blake2s256

Supported Digests for DSA:
  sha256, sha384, sha512, sha3-256, sha3-384, sha3-512,
  blake2b512, blake2s256

PQC support requires OpenSSL 3.4.0+ and is detected automatically via
CMake. A DISABLE_PQC option allows building without PQC even when
available.

KEMs differ from traditional DH in that they require asymmetric roles:
one party encapsulates to the other's public key. This creates a
coordination problem during simultaneous reconnection attempts. The
kem_mode configuration parameter resolves this by pre-assigning roles:

  kem_mode=server  # Server encapsulates (1-RTT, full forward secrecy)
  kem_mode=client  # Client encapsulates (0-RTT, cached server key)

The enc.conf file format supports:

  kex=<algorithm>      # Key exchange algorithm
  cipher=<algorithm>   # Symmetric cipher
  kdf=<KDF>            # Key derivation function
  digest=<digest>      # Digest for DSA
  kem_mode=<mode>      # Server (default) or client
  none                 # Disable encryption

The OAP protocol is extended to negotiate algorithms and exchange KEX
data. All KEX messages are signed using existing authentication
infrastructure for integrity and replay protection.

Tests are split into base and _pqc variants to handle conditional PQC
compilation (kex_test.c/kex_test_pqc.c, oap_test.c/oap_test_pqc.c).

Bumped minimum required OpenSSL version for encryption to 3.0
(required for HKDF API). 1.1.1 is long time EOL.

Signed-off-by: Dimitri Staessens <dimitri@ouroboros.rocks>
Signed-off-by: Sander Vrijders <sander@ouroboros.rocks>

1 files changed, 18 insertions, 3 deletions

diff --git a/cmake/dependencies/openssl.cmake b/cmake/dependencies/openssl.cmake
index 604d5d99..19edfa2b 100644
--- a/cmake/dependencies/openssl.cmake
+++ b/cmake/dependencies/openssl.cmake
@@ -1,21 +1,36 @@
 find_package(OpenSSL QUIET)
 if (OPENSSL_FOUND)
   set(HAVE_OPENSSL_RNG TRUE)
-  if (OPENSSL_VERSION VERSION_LESS "1.1.0")
-    message(STATUS "Install version >= \"1.1.0\" to enable OpenSSL support "
+  if (OPENSSL_VERSION VERSION_LESS "3.0.0")
+    message(STATUS "Install version >= 3.0.0 to enable OpenSSL support "
                    "(found version \"${OPENSSL_VERSION}\")")
   else ()
     set(DISABLE_OPENSSL FALSE CACHE BOOL "Disable OpenSSL support")
     if (NOT DISABLE_OPENSSL)
       message(STATUS "OpenSSL support enabled")
       set(HAVE_OPENSSL TRUE CACHE INTERNAL "")
+      set(IRMD_SECMEM_MAX 1048576 CACHE STRING "IRMd secure heap size")
+      set(PROC_SECMEM_MAX 1048576 CACHE STRING "Process secure heap size")
+      set(SECMEM_GUARD 32 CACHE STRING "Secure heap min size")
+      set(DISABLE_PQC FALSE CACHE BOOL "Disable post-quantum cryptography support")
+      if (OPENSSL_VERSION VERSION_GREATER_EQUAL "3.4.0")
+        if (NOT DISABLE_PQC)
+          set(HAVE_OPENSSL_PQC TRUE CACHE INTERNAL "")
+          message(STATUS "OpenSSL PQC support enabled")
+        else()
+          message(STATUS "OpenSSL PQC support disabled by user")
+          unset(HAVE_OPENSSL_PQC CACHE)
+        endif()
+      else()
+        message(STATUS "Install OpenSSL >= 3.4.0 for PQC support")
+      endif()
     else()
       message(STATUS "OpenSSL support disabled")
       unset(HAVE_OPENSSL CACHE)
     endif()
   endif ()
 else()
-  message(STATUS "Install openSSL version >= \"1.1.0\" to enable OpenSSL support")
+  message(STATUS "Install OpenSSL version >= 3.0.0 to enable OpenSSL support")
   unset(HAVE_OPENSSL_RNG)
   unset(HAVE_OPENSSL CACHE)
   set(OPENSSL_INCLUDE_DIR "")