Cryptography in Computer Science: Algorithms, Keys, and Protocols

Cryptography is the technical discipline that enables confidential communication, data integrity verification, and authenticated identity over untrusted channels. This page covers the foundational algorithms, key structures, and protocol architectures that practitioners and researchers encounter across computer science, from symmetric block ciphers to post-quantum standards. The treatment is reference-grade, addressing classification boundaries, design tradeoffs, and the formal bodies — primarily NIST — that define production standards.

• Definition and scope
• Core mechanics or structure
• Causal relationships or drivers
• Classification boundaries
• Tradeoffs and tensions
• Common misconceptions
• Checklist or steps
• Reference table or matrix
• References

Definition and scope

Cryptography in computer science is the formal study and application of techniques for securing information against adversarial access, modification, or forgery. It operates across three functional axes: confidentiality (preventing unauthorized reading), integrity (detecting unauthorized modification), and authenticity (verifying origin or identity). A fourth property, non-repudiation, is derived from authenticated schemes and prevents a sender from denying a signed transmission.

The scope within computer science spans theoretical foundations in computational complexity — directly connected to the field of computational complexity theory — and practical protocol engineering. The hardness assumptions underlying most cryptographic constructions (integer factorization, discrete logarithm, lattice shortest-vector) are unproven reductions, meaning that cryptographic security ultimately rests on the conjectured intractability of specific mathematical problems.

Cryptography is a core subdomain of cybersecurity fundamentals, but its theoretical reach extends into algorithms, number theory, and information theory. NIST's Cryptographic Technology Group at the Computer Security Resource Center (CSRC) serves as the primary US federal body for standardizing cryptographic algorithms and protocols.

Core mechanics or structure

Symmetric-key cryptography

Symmetric schemes use a single shared secret key for both encryption and decryption. The dominant production standard is AES (Advanced Encryption Standard), specified in NIST FIPS 197. AES operates on 128-bit blocks with key lengths of 128, 192, or 256 bits. Its structure is a substitution-permutation network executing 10, 12, or 14 rounds depending on key length.

Block cipher modes of operation — CBC, CTR, GCM — determine how AES processes data larger than one block. GCM (Galois/Counter Mode) is the authenticated encryption mode specified in NIST SP 800-38D; it combines confidentiality with a 128-bit authentication tag.

Asymmetric (public-key) cryptography

Asymmetric schemes use mathematically linked key pairs: a public key for encryption or signature verification, and a private key for decryption or signing. RSA (Rivest–Shamir–Adleman) relies on the difficulty of factoring large integers; standard deployments use 2048-bit or 3072-bit moduli per NIST SP 800-131A Rev 2.

Elliptic Curve Cryptography (ECC) achieves equivalent security to RSA at substantially shorter key lengths — a 256-bit ECC key provides roughly the same security level as a 3072-bit RSA key, as documented in NIST SP 800-57 Part 1. ECDSA (Elliptic Curve Digital Signature Algorithm) and ECDH (Elliptic Curve Diffie–Hellman) are standardized in NIST FIPS 186-5.

Hash functions

Cryptographic hash functions produce a fixed-length digest from arbitrary input data. SHA-256 and SHA-3 (Keccak), both standardized under NIST FIPS 180-4 and NIST FIPS 202 respectively, are the primary production choices. Hash functions are foundational to digital signatures, message authentication codes (HMAC), and password-storage schemes.

Key exchange protocols

The Diffie–Hellman key exchange, and its elliptic curve variant ECDH, allow two parties to establish a shared secret over an insecure channel without prior key distribution. TLS 1.3 — defined in IETF RFC 8446 — mandates forward-secret key exchange via ECDHE, eliminating static RSA key exchange entirely.

Causal relationships or drivers

Several converging forces drive the structure and evolution of cryptographic standards:

Computational power growth directly erodes security margins. As processor performance increases, brute-force key search becomes cheaper. NIST's key-length recommendations in SP 800-57 reflect projections that 80-bit security is inadequate beyond 2010, and that 112-bit security is the minimum acceptable through 2030.

Quantum computing presents a categorical threat to asymmetric schemes based on integer factorization and discrete logarithm. Shor's algorithm, running on a sufficiently capable quantum computer, solves these problems in polynomial time. This threat drove NIST's post-quantum cryptography (PQC) standardization project, which in 2024 published FIPS 203 (ML-KEM, based on CRYSTALS-Kyber), FIPS 204 (ML-DSA, based on CRYSTALS-Dilithium), and FIPS 205 (SLH-DSA, based on SPHINCS+) as the initial post-quantum standards. The topic of quantum-resistant algorithm design connects directly to quantum computing fundamentals.

Protocol complexity introduces implementation vulnerabilities independent of algorithmic strength. The majority of real-world cryptographic failures trace to incorrect padding, nonce reuse, or misconfigured protocol parameters rather than to breaks in the underlying algorithm.

Classification boundaries

Cryptographic constructs divide along several technical dimensions:

By key structure: Symmetric (shared secret), asymmetric (public/private pair), and hybrid (asymmetric key exchange + symmetric bulk encryption).

By security property provided: Confidentiality-only (stream ciphers), integrity-only (HMAC, hash), authenticated encryption with associated data (AEAD — AES-GCM, ChaCha20-Poly1305), and digital signature (RSA-PSS, ECDSA, EdDSA).

By hardness assumption: Integer factorization (RSA), discrete logarithm over finite fields (DH, DSA), elliptic curve discrete logarithm (ECDH, ECDSA), lattice problems (ML-KEM, ML-DSA), hash-based constructions (SLH-DSA).

By deployment context: Transport-layer security (TLS), storage encryption, code signing, certificate infrastructure (X.509 PKI), and end-to-end messaging (Signal Protocol). The boundary between cryptographic primitives and full protocol stacks is significant: a sound primitive can be deployed insecurely inside a flawed protocol.

Cryptography intersects with blockchain technology at hash chaining and public-key signature verification, and with network security principles at TLS and IPsec protocol layers.

Tradeoffs and tensions

Security level vs. performance: Longer keys and higher round counts increase computational cost. AES-256 requires 14 rounds vs. 10 for AES-128; the security margin difference is significant for threat models involving quantum adversaries, but the performance cost is measurable in latency-sensitive applications. ECC addresses this partially by achieving 128-bit security with 256-bit keys rather than the 3072-bit RSA keys required for equivalent strength.

Forward secrecy vs. auditability: Ephemeral key exchange (as mandated in TLS 1.3) provides forward secrecy — compromise of long-term keys does not expose past sessions — but also means that recorded traffic cannot be decrypted later even by authorized parties. Enterprise network monitoring tools that rely on decrypting TLS traffic face operational conflict with forward-secrecy requirements.

Standardization vs. agility: NIST-standardized algorithms provide interoperability and third-party validation but impose migration lag. When SHA-1 was deprecated following practical collision demonstrations, systems dependent on SHA-1 certificates required multi-year migration cycles. Cryptographic agility — designing systems to swap algorithms without full re-architecture — is an explicit NIST recommendation in SP 800-131A but adds protocol complexity.

Authenticated encryption complexity: AEAD modes like AES-GCM require nonce uniqueness for a given key. Nonce reuse in GCM completely breaks both confidentiality and integrity. ChaCha20-Poly1305 (standardized in IETF RFC 8439) is often preferred in contexts where nonce-reuse resistance is harder to guarantee.

Common misconceptions

"Encryption guarantees integrity." Confidentiality and integrity are distinct properties. A cipher-text encrypted under AES-CBC without an authentication tag can be modified in ways that produce predictable plaintext changes — a property exploited by padding oracle attacks. Only authenticated encryption schemes (AEAD) provide both properties simultaneously.

"Longer keys always mean stronger security." Security depends on the algorithm, its mode of operation, and implementation correctness, not key length alone. A 4096-bit RSA key implemented with PKCS#1 v1.5 padding is vulnerable to Bleichenbacher-class attacks; a 256-bit ECC key with correct implementation is more secure in practice.

"HTTPS means the site is trustworthy." HTTPS indicates that the transport channel is encrypted and that the server certificate was issued by a recognized CA. It does not verify the legitimacy of the site's content or operator. Certificate issuance by public CAs does not require verification of organizational intent.

"Hashing a password is sufficient for storage security." Unsalted hashing of passwords allows precomputed dictionary (rainbow table) attacks. Production password storage requires adaptive hashing algorithms — bcrypt, scrypt, or Argon2 (winner of the Password Hashing Competition in 2015) — that are computationally expensive and incorporate per-user salts.

"Post-quantum algorithms are only relevant for future quantum computers." Adversaries can record encrypted traffic now and decrypt it when quantum capability is available — a "harvest now, decrypt later" threat. This makes PQC migration relevant for data with long-term sensitivity, independent of when large-scale quantum computers become operational.

Checklist or steps

The following sequence describes the phases of a cryptographic scheme evaluation as practiced against NIST and IETF criteria. This is a descriptive process model, not prescriptive advice.

1. Define the security goal — identify which properties are required: confidentiality, integrity, authenticity, non-repudiation, forward secrecy.
2. Identify the threat model — characterize adversary capabilities: passive eavesdropping, active tampering, quantum adversary, insider access.
3. Select algorithm primitives — match the required security level (e.g., 128-bit, 192-bit, 256-bit) to NIST-approved algorithms per SP 800-57 Part 1 Rev 5.
4. Select a mode of operation — for symmetric encryption, choose an authenticated encryption mode (AES-GCM, ChaCha20-Poly1305) unless a separate MAC layer is explicitly provided.
5. Establish key management architecture — define key generation, distribution, rotation, and destruction procedures per NIST SP 800-57 Part 2 Rev 1.
6. Define nonce/IV generation strategy — deterministic counters or random generation; document the maximum number of encryptions before key rotation.
7. Validate implementation against test vectors — NIST publishes Cryptographic Algorithm Validation Program (CAVP) test vectors at csrc.nist.gov/projects/cryptographic-algorithm-validation-program.
8. Review for protocol-level vulnerabilities — check for padding oracle exposure, downgrade attack surfaces, and certificate validation logic.
9. Plan for cryptographic agility — abstract algorithm identifiers to allow substitution without full re-architecture.
10. Document PQC migration timeline — assess which components handle data requiring long-term confidentiality and prioritize ML-KEM or SLH-DSA integration per NIST FIPS 203/205.

The algorithms and data structures domain provides relevant background on the computational complexity properties that underpin these algorithm selections.

Broader context on how cryptography fits within the full computer science discipline is available at the Computer Science Authority index.

Reference table or matrix

Cryptographic algorithm comparison matrix

Security level figures are classical-bit estimates drawn from NIST SP 800-57 Part 1 Rev 5 and FIPS publication documentation. Post-quantum security levels reflect NIST's security category definitions (Category 1 = AES-128 equivalent, Category 3 = AES-192 equivalent, Category 5 = AES-256 equivalent).

References

• [NIST FIPS 197 — Advanced