What are Encryption Protocols and How Do They Work?

Read time: 30 minutes

Encryption is used every day to secure online communications between two individuals or between clients and servers. Though you may not see it in action, encryption obscures your data-at-rest and data-in-transit from outside attackers who are potentially monitoring your communications. Encryption works by taking in plaintext data, or data that is unobscured, and turning that plaintext into ciphertext. Ciphertext is a random assortment of letters, numbers, and sometimes symbols, that obscures sensitive data from unwanted viewers. Ciphertext can be reversed back into its sensitive data form, as long as either a key was used to encrypt the data, or a pattern is found in the ciphertext to decrypt it. The ability to decrypt data is vital in the online communication process, as the recipient of the information should be able to decrypt the data, which is usually done via key usage. Encryption is vital to ensure sensitive data stays secret from unwanted attackers, and going along with encryption are encryption protocols. 

What is an Encryption Protocol?

Encryption is done through encryption algorithms. These algorithms do all the cryptographic operations, using the encryption key, on the plaintext data. These algorithms are then utilized within encryption protocols to protect data for different usages. The point of an encryption protocol is to fulfill a specific function. The functions encryption protocols can perform vary, from communications with TLS/SSL to remote connections to computers with SSH. We will take a look at some of the more well-known encryption protocols later on in our article. Before going in-depth on encryption protocols, there are a few terms we should learn first, starting with asymmetric and symmetric encryption.

Symmetric and Asymmetric Encryption

Symmetric encryption is the much simpler form of encryption. Symmetric encryption utilizes one key to encrypt data, whether that data is in-transit or at-rest. In reference to encrypting data-in-motion, the key is created and shared with both the sender and the recipient of the message. The data in the message is encrypted with the symmetric key, meaning the only person who can read this data is someone who owns the encryption key. Once the message reaches the recipient, they can use the symmetric key to decrypt the data. Using symmetric encryption alone is not recommended, as it is much more insecure compared to asymmetric encryption. This is due to the fact that with symmetric encryption, the key created must at some point be delivered to the data recipient. If this transfer is not done securely, the key could be intercepted during delivery, meaning any encryption done with that key is now irrelevant. An example of data-in-transit encrypted with a symmetric key can be seen below.

Asymmetric encryption, as I mentioned previously, is the more secure of the two types of encryption. With asymmetric encryption, a key pair is created which consists of a public and private key. The public key is kept available for anyone to see, while the private key is known only by the key pair creator. To asymmetrically encrypt data, the key pair creator encrypts the message with their private key, sends the encrypted message to the recipient, and the recipient can then use the public key, generally found from a public key repository, to decrypt the message. By decrypting the message with the public key, the data recipient can tell that the message is from who they think it is from and that the data in the message has not been changed. If the data in the message had been changed, the decryption with the public key will not produce a readable message, as the data would have been encrypted to a different value. Though asymmetric encryption is more secure than symmetric encryption, they tend to be used in tandem for communications encryption. The initial connection will be created with asymmetric encryption, a symmetric session key will be created, and the session key will then be used to encrypt messages in the session. Below is a diagram of the asymmetric encryption process.

Public Key Infrastructure (PKI)

Working hand in hand with asymmetric encryption and encryption protocols is Public Key Infrastructures, or PKI. A PKI Infrastructure utilizes digital certificates and asymmetric key pairs to authenticate users and devices within a network. When someone wants to use a network that is utilizing a PKI Infrastructure, they must request certificate from a Certificate Authority (CA) within the PKI. The request, also known as a Certificate Signing Request or CSR, which contains information about the requestor, as well as the public key of an asymmetric key pair they are the owner of. The information within the request is verified by the CA and, if it is valid, a certificate is issued to the requestor containing their public key along with a number of other components. Now, when a connection is made between that certificate holder and a server or other user, they can look at their digital certificate’s Chain of Trust to verify that the certificate is still valid.  A certificate’s Chain of Trust is a path from the current certificate leading all the way back to the Root CA’s certificate. Each certificate in this chain is checked itself for validity, to ensure the certificate holder is not using an expired or revoked certificate. If this is the case for every certificate in the chain, then the certificate is validated and a connection can occur. For a better understanding of Public Key Infrastructures, I will go more in-depth into how a PKI is built and what makes it up.

Public Key Infrastructures always contain a Root CA. This is the core of trust in the PKI Infrastructure, issuing certificates to the Issuing CAs to ensure they, in turn, can issue certificates to requestors. The compromise of a Root CA invalidates every single certificate issued within that PKI Infrastructure, and so the Root CA is kept offline at all times. Issuing CAs are also involved in every type of PKI Infrastructure, since the Root CA is offline and cannot issue certificates to users. Issuing CAs, of which there can be any number, do as their name suggest: issue certificates. These CAs are the link from the issued certificate to the Root CA in their Chain of Trust. The compromise of an Issuing CA is not as devastating as the compromise of the Root CA, but it still has harsh consequences. When an Issuing CA is compromised, all of its issued certificates are also compromised. This means a large section of the PKI is now unusable. Similar to an Issuing CA is an Intermediate CA. The Intermediate CA is not used in most Public Key Infrastructures, it is only used in a three tier PKI Infrastructure. These add another layer to the Certification Path, or Chain of Trust. They tend to issue certificates for Issuing CAs, and serve as the link from a Root CA to an Issuing CA. Along with the different CAs, a PKI also includes Certificate Revocation Lists. Certificate Revocation Lists, or CRLs, are lists containing the information of certificates that have been revoked for one reason or another.  This should be sufficient to answer the question of: How does a PKI work? Now, let us take a look at the most common encryption protocols and what they do.

Common Encryption Protocols

• TLS/SSL: TLS/SSL is the most common encryption protocol, which is used every day on the Internet. TLS/SSL stands for Transport Layer Security/ Secure Sockets Layer, which is an encryption protocol that ensures communications between a client and server are kept secure. When your web browser connects to a website, if the connection is secured with TLS/SSL, then a padlock and the word “https” is shown in the search bar. TLS/SSL does not do the encryption itself, it instead uses a variety of encryption algorithms, like RSA or AES, to encrypt the communications. This is why SSL/TLS is considered an encryption protocol. Using TLS/SSL to encrypt communications is very common, as a number of different encryption algorithms are used with it. TLS/SSL can be used for user authentication, traffic encryption, and showing that data has not been modified in transit.
  The way TLS/SSL works is that an asymmetric key pair is utilized in a “Handshake” process to secure the initial connection between the client and server. That “Handshake” is where the specific protocol version to be used is selected, the TLS/SSL certificates of both server and client are verified, the algorithm for the “Record” process is selected, and the shared key is generated with symmetric encryption. The shared key is then used in the next step of the communication, the “Record” protocol. In this, packets shared between the two users are encrypted with the shared key to ensure the safest form of communications.
• IPsec: IPsec, or Internet Protocol Security, is an encryption protocol which utilizes encryption algorithms like 3DES, AES, SHA, and CBC to encrypt data in applications, routing, or Virtual Private Networks, most commonly. Using its two modes, tunneling and transport mode, IPsec protects data moving from one location to another. Transport mode encrypts only the payload of the message, not the header. As some information can be gained from the header, this is only used for simple data transfer situations such as connecting to a server or workstation. Tunneling mode, on the other hand encrypts and authenticates both the payload and header. Tunneling mode is most often used with Virtual Private Networks, or VPNs. Though using VPNs with IPsec is generally faster, as IPsec is quicker to setup a connection, other parts of TLS/SSL make it the preferred method of data-in-transit encryption and authentication.
• SSH: Secure Shell, also known as SSH, is another type of encryption protocol. The way SSH works is similar to a VPN. By creating an encrypted tunnel, users can use SSH to securely and remotely connect to computers, transfer files, port forward, and more. SSH works on 3 different levels: the transport level, the user authentication level, and the connection level. The transport level is the layer which securely connects two parties, securely encrypts any data sent between them, authenticates the users to each other, and ensures that the data shared between the users is not changed in any way in-transit. To exchange keys, the two parties in the SSH connection are connected and the keys of the client and server are traded via the Diffie-Hellman key exchange. During this phase of SSH, the symmetric algorithm, asymmetric algorithm, message authentication algorithm, and the hash algorithm to be used in the transfer of data and messages are chosen. In the authentication level, the client authenticates its identity via a supported authentication method specified by the server from the transport layer. The authentication method in question can be anything, from a password to a digital signature. The connection level handles all of the connections created between the server and client. A different channel is opened for every communication between the server and client. An example of this is if multiple sessions are created to the same server, then for each session a different communication channel is opened. Either the client or the server can open a new communication channel, as long as the parameters for the channel are available for use by both the client and server. 
• PGP: OpenPGP, also referred to as PGP, is an encryption protocol which allows users to encrypt their messages and digitally sign them, giving the message sender a stronger method of both authentication and data integrity protection. Mainly, PGP is used for the purpose of protecting sensitive email information. PGP was developed in the 90’s in an attempt to make it a globally used and interoperable system. PGP is free to use and integrate into a number of different email clients. Different encryption algorithms are available for use with PGP, such as RSA and DSA for asymmetric encryption, AES, 3DES, and Twofish for symmetric encryption, and SHA for hashing. Different vulnerabilities have been found for PGP throughout the years, but these flaws have always been addressed with updates or recommendations. 
• S/MIME: Secure/Multipurpose Internet Mail Extensions, or S/MIME, is a competitor to OpenPGP as an email based encryption protocol. Just like PGP, S/MIME allows users to encrypt and sign email data to further protect it from attackers. The difference with PGP and S/MIME is that S/MIME uses different encryption algorithms to secure data. 
• Kerberos: The encryption protocol Kerberos works by acting as a single sign-on authentication protocol. The protocol authenticates its users against a central authentication and key distribution server. Users of the protocol are given “tickets”, once authenticated, allowing them to use the different services within the network. When a client with a “ticket” reaches out to a server, that server verifies the “ticket and grants the user access. Kerberos’ main use is on Local Area Networks (LANs) and for establishing shared secrets. Kerberos is a well-known and often used encryption protocol, but both the client and server must include code to utilize Kerberos, which turns some organizations away from its usage. 

Though many other encryption protocols exist, these are the most well-known and most widely used encryption protocols. Many of the protocols are may seem the same, as they fulfill the same purpose, but they use many different encryption algorithms, so it is vital to research the algorithms in use when choosing the correct encryption protocol for your organization.

Is it Safe to use Encryption Protocols?

You may note that throughout our discussion of different types of encryption protocols that some of the protocols had vulnerabilities found within them. This begs the question: are these encryption protocols safe to use? The answer is yes. Though oftentimes vulnerabilities are found within encryption protocols, as soon as they are detected security patches, upgrades, or requirements are put in place to protect users from those who would exploit such gaps in security. Even such things that seem like they would never be vulnerable to attacks, like Operating Systems, also have to patch or update to fix vulnerabilities, so as you can see nothing is one hundred percent secure on the Internet. Encryption protocols and encryption algorithms in general are also backed by the National Institute of Science and Technology. The National Institute of Science and Technology (NIST) is an institution designed to provide recommendations for online security for government organizations. The NIST also approves of the newest encryption algorithms and protocols through their recommendations. What I mean by this is that if an encryption protocol or encryption algorithm is recommended for use by the NIST for government organizations, then you know that it has the highest level of security and thus can be used by anyone.

Why should your organization use encryption protocols?

Even for all of the vulnerabilities that can be found in encryption protocols, they are still one of the most secure tools on the Internet for securing sensitive data. Rather than relying on a single method to secure data, encryption protocols like PGP or Kerberos utilize asymmetric encryption, symmetric encryption, and digital signatures to protect the security, integrity, and authenticity of data and the data handlers. These encryption protocols are used in almost every Internet interaction to keep data safe. From email, remote desktop connections, Wi-fi network connections, and more, you use encryption protocols every day. Even governments, who face much more serious threats than the average person, use encryption protocols to keep communications and connections secure. Along with this, as threats grow throughout time, more advanced methods of data protection are developed, including more secure encryption algorithms. These encryption algorithms and security methods are implemented into existing and new encryption protocols to ensure the user of these protocols have the best possible protections in place. Another thing to note is that as newer encryption protocols are created, these protocols will likely do different tasks. Next year’s newest encryption protocols could make the security of databases, Cloud systems, or even self-driving cars much safer than they are today. The updating of existing encryption protocols with newly created encryption algorithms ensures that if an encryption algorithm has been found to be vulnerable, there is another option for users to implement.

Conclusion

In our article, we addressed several questions about encryption protocols, including how does a PKI work, what does an encryption protocol do, are encryption protocols safe to use, and many more. As is readily obvious, encryption protocols permeate every part of the Internet, from emails to website connections. These are vital pieces to a secure Internet for everyone, even big companies and governments. Encryption protocols can be made up of encryption algorithms, digital signing algorithms, hashing algorithms, digital signing code, and more. Encryption protocols are also a big part of Public Key Infrastructures. PKIs need to utilize encryption protocols to ensure data, like an asymmetric digital certificate key, is secure. There are a number of encryption protocols to choose from as well, which all tend to have their own purpose and method of accomplishing that purpose. Protocols like S/MIME or PGP focus on the protection of email messages, while other protocols, like SSH, work to securely connect users to remote computers or outside servers. It is worth noting that encryption protocols can contain security gaps that go unnoticed, but as soon as these gaps are detected, the encryption protocol creators release a fix or configuration option which fixes that vulnerability. Encryption protocols are one of the best tools an organization or user can implement to keep communications and data, both in-transit and at-rest, secure.