SeriCrypt: LLM-Driven Crypto Protocol Serialization
Abstract
Constructing syntactically correct and cryptographically valid message sequences is essential for protocol state machine learning, conformance testing, and fuzzing. Unlike plaintext protocols, cryptographic protocols involve complex cross-message state dependencies and cryptographic computation constraints. Existing automated message construction approaches predominantly target text-based or plaintext protocols, lacking support for deep semantic dependencies and cryptographic operations, rendering cryptographic protocol message construction largely manual. We present SeriCrypt, an LLM-driven, context-aware serialization framework for cryptographic protocols. SeriCrypt uses a large language model (LLM) to extract protocol field constraints, contextual state dependencies, and cryptographic computation rules from unstructured protocol specifications, representing them as a unified structured intermediate representation. To formally characterize this representation, we design a domain-specific language for cryptographic protocols (CDSL) that explicitly captures semantic constraints and computational dependencies in protocol interactions. Building upon CDSL, we implement a protocol-agnostic execution engine that parses CDSL declarations and automates field value resolution, cryptographic primitive invocation, and byte-stream serialization. As case studies of SeriCrypt in protocol security testing, we employ the framework to construct violation messages targeting specification-defined security constraints and validate its application in protocol fuzzing scenarios, with evaluation on multiple mainstream implementations of TLS 1.2/1.3, IKEv1/v2, SSH, and TLCP. Results show SeriCrypt can generate valid message sequences accepted by real implementations and successfully complete handshakes across all evaluated scenarios. Security constraint testing revealed five specification violations, and fuzzing reached deeper protocol states with higher code coverage than mainstream fuzzers under the same time budget, demonstrating the frameworks practical value for cryptographic protocol security testing.