feature: added ngx.ssl.compress_certs()

[u5surf]

The Lua side of openresty/lua-nginx-module#2530, which fixes openresty/lua-nginx-module#2516. That PR carries the root-cause analysis: a certificate installed with clear_certs() + set_cert() is always sent uncompressed, even under ssl_certificate_compression on;, because OpenSSL sends a TLS 1.3 CompressedCertificate only for a chain that was pre-compressed, and clear_certs() throws away the pre-compressed copies nginx made for the configured certificate.

API

ssl_certificate_by_lua_block {
    local ssl = require "ngx.ssl"

    assert(ssl.clear_certs())
    assert(ssl.set_cert(cert))
    assert(ssl.set_priv_key(key))

    local ok, err = ssl.compress_certs("zlib")   -- or "brotli", "zstd", or an
    if not ok then                               -- RFC 8879 number; omit it for
        ngx.log(ngx.WARN, err)                   -- every algorithm OpenSSL has
    end
}

Returns true, or nil plus an error string. It needs OpenSSL 3.2.0+, an OpenSSL built with the algorithm in question, and ssl_certificate_compression on; (nginx 1.29.1+) for nginx to put the compressed certificate on the wire; lib/ngx/ssl.md documents all three and the error each missing one produces.

The C binding is resolved through pcall, as lib/ngx/pipe.lua does, because indexing C.<symbol> resolves it eagerly: without the guard, ngx.ssl as a whole would stop loading against an ngx_lua that does not carry ngx_http_lua_ffi_ssl_compress_certs() yet. Unknown algorithm names are rejected in Lua, before any C call.

Tests

t/ssl-cert-compression.t covers the no-argument and named forms, an unknown name, and compressing before any certificate is set. The two blocks whose outcome depends on which algorithms OpenSSL was built with accept either result unless TEST_NGINX_CERT_COMP_ALGS=1 is set; the other two are deterministic.

The API was exercised end to end against nginx 1.31.6 + OpenSSL 3.5.4 built with zlib: ssl.compress_certs("zlib") returned true and openssl s_client -trace showed CompressedCertificate, Length=1396 where the same server without the call sent Certificate, Length=1566. The test file itself has not been run as a file: t::TestCore reads /proc/net/tcp, so the suite does not start on macOS — the same holds for the existing t/ssl.t — and CI is its first run.

I hereby granted the copyright of the changes in this pull request
to the authors of this lua-resty-core project.