[BUG] Inexact encryption with cms protocol that prevents later decrytion with OpenSSL

[tadam98s]

Source code and results are in the attached zip.

My OpenSSL encrypts files converted to Base64 so that we only deal with text files. The resul is also a PEM text file (I prefer text files).
openssl cms -encrypt -in myfile.txt.b64 -aes256 -rand {noiseFile} -out myfile.txt.b64.enc -recip mickey_cert.pem -keyopt ecdh_kdf_md:sha256
I am using the latest 2.6.2 release downloaded from nuget: BouncyCastle.Cryptography 2.6.2 nuget

Here is my call to Bounty Castle which created a binary DER file:
msEcdhInteropTest.EncryptFile(b64OutBC, $"{b64OutBC}.BCenc", receiverCertBC)

A clear and concise description of what the bug is.

Attached in a zip my code, certificate and the command to run it.

Expected Behavior

A clear and concise description of what you expected to happen.

Screenshots and Logs

myfile.txt.b64.zip

If applicable, add screenshots and logs to help explain your problem.

Product Deployment

Please complete the following information:

• Deployment format: [e.g. software, container]
• Version [e.g. 8.0.0]

Desktop

Please complete the following information:

• OS: [e.g. iOS] Widnows 11
• Browser [e.g. chrome, safari]
• Version [e.g. 22]

Additional Context

Analysis of the attached results:

OpenSSL: (11 bytes total — just the OID, no parameters field)
parameter: SEQUENCE:
0:d=0 hl=2 l= 11 cons: SEQUENCE
2:d=1 hl=2 l= 9 prim: OBJECT :id-aes256-wrap

Bouny Castle:
parameter: SEQUENCE: (13 bytes — OID plus an explicit NULL parameters field)
0:d=0 hl=2 l= 13 cons: SEQUENCE
2:d=1 hl=2 l= 9 prim: OBJECT :id-aes256-wrap
13:d=1 hl=2 l= 0 prim: NULL

Claude.ai analysis:
This is the exact discrepancy that breaks interop. That inner wrap-algorithm AlgorithmIdentifier gets DER-encoded into the ECC-CMS-SharedInfo structure (RFC 5753) that's fed into the X9.63 KDF alongside the raw ECDH secret Z. Two extra bytes there means different KDF input, means a different derived key-encryption-key — everything else lines up (curve, KDF OID, wrap OID, content cipher), but the actual key bytes silently diverge. That produces exactly the deterministic "checksum failed" you're seeing, with no other symptom, because the algorithms genuinely do all match — only the encoding of one of them differs.

This is a known category of interop bug: some ASN.1 libraries include an explicit NULL for "no parameters" (strict X.690 tag-length-value completeness), others correctly follow RFC 5753's convention of omitting the parameters field entirely for this specific algorithm. OpenSSL follows the RFC. BC 2.6.2, apparently, does not — matching what you were "hinted" about their ASN.1 not being bug-free.
This means BC's high-level AddKeyAgreementRecipient API can't produce openssl-compatible output as-is, and likely can't be worked around by changing OIDs or wrap choices — the NULL insertion is happening inside BC's internal SharedInfo/AlgorithmIdentifier construction, not something the public API surface lets you override.

Please check.
Thanks,
Mickey

[tadam98s]

The probkem is corrected by the issue is real.
We just "bypassed".
Kindly study.
Best,
Mickey

Encrypt with OpenSSL:
OpenSSL cms -encrypt -in \"{b64Out}\" -outform PEM -aes256 -rand \"{noiseFile}\" -out \"{encOut}\" -recip \"{receiverCert}\" -keyopt ecdh_kdf_md:sha256
Encrypt wth BC:
CmsEcdhInteropTest.EncryptFile(b64Out, encOut, receiverCert);

Decrypt with OpenSSL:
OpenSSL cms -decrypt -in \"{verifiedFile}\" -inform PEM -recip \"{receiverCert}\" -inkey \"{receiverKey}\" -out \"{b64DecOut}\"
Decrypt with BC:
CmsEcdhInteropTest.DecryptFileManual(verifiedFile, b64DecOut, receiverKey);

All combinations now work (I did not yet add noise on the BC encrypt)

• Root cause identified and documented: BC 2.6.2's high-level CMS decrypt reconstructs SharedInfo with its own conventions instead of reusing transmitted bytes — a real, confirmed bug, not a misconfiguration.
• A working, spec-correct manual decrypt path (DecryptFileManual) that parses the ASN.1 structure directly and handles the real-world encoding variations between openssl and BC (tagging quirks, absent vs. present curve parameters) — validated in all four combinations (openssl/BC encrypt × openssl/BC decrypt equivalent coverage).
• Every primitive in the chain confirmed correct: ECDH agreement, X9.63 KDF with the right SharedInfo construction, RFC 3394 key unwrap, AES-256-CBC content decryption.

CmsEcdhInteropTest.cs.txt