Guest postsTips and tricks

False Sense of Security: Instant Messaging Apps Tricks Exposed

If you are considering a secure IM add-on for your CS-Cart eCommerce website, read this post to know which messenger app to select. Let’s see how end-to-end (E2EE) is handled by popular instant messaging apps, such as Telegram web app, Signal, Viber, WhatsApp, and others.

E2EE is considered a panacea when dealing with persistent attempts of hackers or government agencies to intercept online correspondence. The essence of E2EE often boils down to the fact that keys are stored only on interlocutors’ devices and do not get to the server… but this is not always true.

Messaging apps and encryption

The study called Obstacles to the Adoption of Secure Communication Tools (PDF) inspired me to write this article. Scientists found that:

“The vast majority of interviewed participants did not understand the essential concept of end-to-end encryption.”

Simply put, people usually choose the messenger app with their heart and not their brain.

To begin with, in each particular messenger, E2EE has its own features and characteristics. In Signal, E2EE is almost perfect. WhatsApp is practically the same as Signal, except for one critical point: changing the main subscriber’s key does not block sending him messages. In Viber, end-to-end encryption is inactive by default. In Telegram web app, E2EE is used only in secret chats, and it is implemented strangely.

It is sad but a lot of mathematically perfect ideas get spoiled once implemented. Convenience is put above privacy and security.

Initially, instant messaging applications utilized the Off-the-Record (OTR) protocol. It makes use of the AES symmetric encryption in CTR mode, the Diffie–Hellman key exchange method, and the SHA-1 hash function.

This AES – CTR method provides the so-called “disputable” encryption and the opportunity to deny the text authorship in case it is intercepted. You can always claim that the party which intercepted your communication modified the ciphertext.

The OTR protocol authenticates the conversation participants and encrypts information traveling between them. It is a safe scheme as long as participants check each other’s public key fingerprints regularly and try to resist attacks coming from other vectors (that are constantly growing.) 

The main drawback of OTR is that after sending a key, you have to wait for confirmation from another end. If your opponent is offline, the communication process is temporarily impossible. 

The Double Ratchet algorithm was introduced to solve this issue. DR is now used in Signal, Viber, WhatsApp, and other IM apps.

We know that the end-to-end encryption scheme relies on a combination of public and private keys. As usual, the devil is in the detail. The scheme uses several interconnected keys. Some of the keys should obligatorily go to the server before the correspondence begins so that the conversation can be started at any moment.

The beginning of this process is standard for all asymmetric encryption systems—you generate a pair of keys. This is a necessary step since the single-key systems (for example, AES) are too problematic to use for communication. 

Just like in PGP, there are two chat participants Alice and Bob. Each of them generates his own pair of keys. After that, they exchange their public keys, keeping their private keys secret. Public keys needed to encrypt a message and also to verify its signature. Private keys are used for decrypting a message and generating a signature.

This cryptosystem works well in email. To use email, communicating parties are not required to be simultaneously online. Messages are collected on the server and get downloaded from it once the user passes authorization. Decryption happens locally with the help of private keys that do not need to be transmitted anywhere. 

Unfortunately, the asymmetric encryption method is not suitable for IM apps because these services deal with active online communication that involves numerous chains of short and long messages. Besides, one (or both) chat participants may get offline at any moment.

It is not a good idea to encrypt plenty of short messages with only one key. Hundreds of messages can be created in a single day. In many of them, the amount of ciphertext is very tiny and can be predicted (emoji, stickers.) Moreover, they have standard headers that make the cryptanalysis easier.

Because of this typical metadata, an attacker may intercept a significant amount of predictable ciphertext in a short time. The biggest part of the ciphertext will correspond to the well-known text. And in case it is encrypted with only one key, then during a successful attack, all previously sent messages will be compromised, and actually, even those that chat participants will create in the future.

To prevent this, IMs use such things as Forward secrecy and Backward secrecy. This implies the impossibility to read messages sent earlier, or that will be written in the future, having only the current encryption key on hand. For this, multilayer encryption is used with the transition from asymmetric to symmetric cryptography and additional keys with different lifetimes.

IM security and Diffie–Hellman key exchange

Let’s look at Telegram encryption. The classic Diffie–Hellman (DH) key exchange is used in Telegram for an authenticated key distribution. It creates a bridge between asymmetric (RSA) and symmetric (AES) encryption, enabling a certain number of users to carry out encrypted correspondence. For this, session keys are generated, which represent a shared ephemeral key. It is calculated based on the private key of the recipient and the public key of the sender. Long-term public keys authenticate ephemeral keys.

In DH, the transmission channel may not be protected from passive monitoring, but it must be protected from spoofing attacks. If the attacker can spoof traffic (during an MITM attack), then the whole scheme is useless.

Therefore, for its Signal messenger, Open Whisper Systems uses the Extended Triple Diffie-Hellman method— X3DH.

Extended Triple Diffie-Hellman protocol establishes a shared secret between two parties that mutually authenticate each other based on public keys. Additionally, the keys are verified immediately after setting up the session and before starting the transmission of messages.

X3DH uses four types of keys, three of which are constantly changing:

Identity keys—private keys that are created once and serve as the basis for the formation of all the other keys.

Ephemeral key is needed to verify the identity of the user (without divulging the true identity.) 

Signed public key \ signed proof of knowledge—in fact, this is an ephemeral key signed by a private key. Usually, in messaging applications, it gets changed every day, sometimes up to a week. Sometimes, instead of time frames, the number of messages is used. 

The one-time public key is created by the sender before setting up a communication session and is deleted immediately after a successful handshake.

The changing of auxiliary keys in X3DH occurs based on the Double Ratchet algorithm. It replaced the OTR and introduced the concept of a keyring. Anticipating the situation when one of the conversation participants goes offline, several one-time ephemeral keys are uploaded to the server in advance and are consumed one by one as users communicate. This allows the server to receive encrypted messages, each time authenticating their sender with a new key pair, even when the recipient is offline. 

The DR method is used to generate new ephemeral keys that encrypt the next message (or a small portion of messages.) These ephemeral keys will always differ and cannot be predicted within a reasonable time.

Security of group chats

This paper (PDF) describes in detail how group chats are implemented in WhatsApp, Signal, and Threema. The main part here is:

“Our systematic analysis reveals that (1) the communications’ integrity – represented by the integrity of all exchanged messages – and (2) the groups’ closeness – represented by the members’ ability of managing the group – are not end-to-end protected. We additionally show that strong security properties, such as Future Secrecy which is a core part of the one-to-one communication in the Signal protocol, do not hold for its group communication.”

The cornerstone of end-to-end encryption is the authenticated key distribution using the classic or enhanced DH protocol. It works only for two participants, forming a shared secret. In IM messenger, DH is not used in group chats, and the principle of exchanging messages in them is devoid of basic cryptographic properties. This research is based on Signal and WhatsApp, but it is unlikely that other IM apps have solved this problem.

Telegram security

A veil of secrecy covers the Telegram Web app. There is only partial information about the MTProto 2.0 protocol. Its external audit has never been performed. The open-source model is used in a very distorted form, and it seems for marketing purposes only. 

Judging by the official documentation, all undelivered messages are temporarily (we hope) stored on Telegram servers. Servers are synchronized with each other in order to structure and deliver messages to one or several (group chat) participants in a specific order as soon as users appear online. Therefore, encryption is divided into two parts: client-server and server-server. This is a standard scheme, but the strange thing about it is that direct connections (client-client) are never used.

In Telegram, traffic is transmitted through servers even when a secret chat is created. However, it would be more logical to build a P2P connection for this purpose. Consequently, without the constant use of Telegram servers, communication may not happen. Telegram clearly resembles the MITM scheme by design.

All arguments about the strength of MTProto 2.0 focus on the fact that the DH algorithm reliably protects against interception. This is not true. The Diffie-Hellman algorithm is actually vulnerable to a MITM attack. Moreover, in this case, it is probably further weakened at the PRNG level.

The problem is that the Telegram client is guided by a very slurred entropy estimation. Instead of generating pseudo-random numbers locally, the client requests them from the server. There are many questions left without answers:

  • What kind of PRNG is used by the server?
  • How are prime numbers generated?
  • Is there any mechanism on the server for the selective sending of prime numbers with certain properties to specific clients?

The only thing the client application does is performing a simplified verification of the received random number.

Another common argument advocating Telegram security issues is its open-source nature. However, only the client-side code is open-source software. Moreover, the source code is published with long delays and is usually outdated. There is no opportunity to check whether the source code fully corresponds currently distributed product. The server-side code is closed-source and proprietary.

Therefore, talking about this IM messenger audit is pointless. While security specialists dig around the old code for several months, a dozen new versions of Telegram may get released, where huge pieces of code are rewritten. And we know that to make the entire encryption system vulnerable, it is enough to replace a single byte. That’s how work Telegram encryption.

The hackathons arranged by Durov may not replace the audit as they prove nothing. Durov’s hackathons offer an artificial situation where the attacking side has only one encrypted message. In real life, there are no such restrictions, and there are many other vectors for attacking the instant messenger.

IM apps vulnerabilities

Signal is one of the few IM apps that has undergone an external audit (PDF.) Auditors conclude:

“As with many real-world security protocols, there are no detailed security goals specified for the protocol, so it is ultimately impossible to say if Signal achieves its goals. However, our analysis proves that several standard security properties are satisfied by the protocol, and we have found no major flaws in its design, which is very encouraging.”

It is important to note that the analysis of the messaging protocol is an essential stage of the audit, but there are other security factors too. Any IM app works in a real and very vulnerable environment. Many people may not have the latest version of Android. Some messaging apps installed on the phone may abuse permissions or even contain Trojans, and the account itself is tied to a phone number. In many cases, the attacker does not need to hack Signal (or any other protocol) in order to gain access to the entire correspondence. 

Push notifications represent one more flaw. Without push notifications, you cannot tell if you received a message or not. To find it, you would have to launch the messenger app. Using push notifications, you turn the corresponding server into a legalized man-in-the-middle.

For example, for notifications to work in iMessage, it sends encryption keys to Apple servers. Attackers may restore Apple account on another device and get everything from messages to Wi-Fi passwords. Almost the same situation happens when dealing with Google and Microsoft. It is some kind of a paradox, we are talking about the security of end-to-end encryption in a messenger but still have the phone number attached to it as well the main smartphone account.

All current IM apps have to deal with problems associated with unsafe key management and a large attack surface. WhatsApp, Viber, and other messaging applications allow you to create backups of your messages (including cloud backups) and do not encrypt the metadata. And sometimes the fact that the conversation has taken place is more important than its content. 

Signal seems to be the best choice, but still, it is not ideal for several reasons:

  • Signal also uses the Google push notification service. Therefore, it does not work on a smartphone that does not have Google services, for example, all Chinese models for the domestic market.
  • Signal uses a closed RedPhone server for voice communication.
  • Signal allows you to launch a parallel session on another device, simply by scanning a QR code.
  • There was a presentation at HITBSecConf 2017 (PDF) about a number of Signal conceptual problems. Researchers were able to demonstrate a successful attack.

XMPP security

As you can see, it is difficult to trust third-party and especially proprietary IM apps. Therefore, some people build their own solutions. The OTR plugin is suitable for a simple chat, but it does not support group chats. There are some other related protocols like mpOTR and GOTR, where this function is added.

One way or another, for group communication, it is more convenient to use the Extensible Messaging and Presence Protocol (XMPP) that was formerly called Jabber. It is fully open-source. You can set up your XMPP security server without depending on anyone. There are also a lot of ready-made solutions, for example, Pidgin for desktops and Xabber for Android.

The OMEMO (an XMPP Extension Protocol) was created not so long ago. It uses the same DR from Open Whisper Systems (as in Signal and WhatsApp) but is devoid of their flaws.


Modern IM applications claim they support the end-to-end encryption, but it often turns out that it is implemented weirdly. In addition, there are many other holes in their code that were left accidentally or intentionally. The latter is much more likely, especially if you think about how much money and hard work of professionals were invested in their development.

False Sense of Security: Instant Messaging Apps Tricks Exposed: photo 2 - CS-Cart Blog David Balaban is a computer security researcher with over 15 years of experience in malware analysis and antivirus software evaluation. David runs the project which presents expert opinions on the contemporary information security matters, including social engineering, penetration testing, threat intelligence, online privacy and white hat hacking.
Head of Content Marketing at CS-Cart | Website

Yan Anderson is the Head of Content Marketing at CS-Cart with over 10 years of experience in the eCommerce industry. He's passionate about explaining complicated things in simple terms. Yan has expertise in building, running and growing eCommerce marketplaces. He loves to educate people about best practices, new technologies, and trends in the global eCommerce industry.