Alright, I'll bite.
First, let me propose bounding the discussion to just the core of the protocol.
In particular, let's not get hung up on:
- Social engineering attacks
- How broadly the end-to-end encryption is applied (i.e. are all conversations in the app encrypted?)
- The backgrounds of the inventors and reasoning for inventing the protocol
- Metadata leakage (i.e. who is talking to whom, which is pretty much the same between the two protocols)
There's a surprising amount of vitriol floating around the internet on those points, NONE of which is germane to the discussion at hand.
MTProto
EDIT: the following is specific to MTProto 1.0. MTProto 2.0 has since been released, in late 2017. Some of the potential problems discussed below have been fixed (ex. they migrated from sha1 -> sha256), while others have not seen updates (ex. MTProto 2.0 still uses IGV). I encourage people to read the new spec (the url is the same; the content has been updated). I would also encourage any sufficiently-motivated, enterprising individual to write a new answer that takes into account updates to the protocols. :)
Let's quickly review the basics of MTProto 1.0:
(Full specification for those interested: spec v1 )
The protocol starts with a standard Diffie-Hellman key exchange. Users can compare a hash of the generated key, exposed as either an image and hex code, for the two parties to exchange over an existing secure channel (preferably, in person). There was a theoretical vulnerability where the original image wasn't large enough (and hence the key-exchange could be man-in-the-middled by someone with sufficient resources), but that's reportedly been fixed by increasing the size of the images / hex printout. The session key ("auth_key") is derived from this original key (see the spec for particulars).
The meat of the specification hinges on the message envelope. Briefly, it takes the payload, prepends a salt and session ID, hashes that (SHA-1), and uses that as the "msg key" (MAC). This also gets mixed with the auth_key (SHA-1 based KDF) to produce a per-message encryption key (256 bit) and initialization vector (256 bit). The salt, session ID, payload, and 0-15 bytes of arbitrary padding are then encrypted with AES in IGE mode. Finally, the auth_key_id, "msg key" (MAC) and encrypted data are concatenated to produce the final message.
Whew. Actually, the diagram from the spec makes much more sense.
MTProto makes what many cryptographers would consider odd choices. You pointed out several in your question.
- A hash algorithm that's reaching the tail-end of its life (SHA1)
- An old attempt at authenticated encryption (IGE), of which the authentication part has long-since been broken, but which MTProto doesn't actually use to provide authentication.
- A discouraged mode (MAC-then-encrypt)
- Vulnerability to a chosen-ciphertext attack (which may not have been a choice as such).
- A non-standard padding algorithm (which is to say, "append whatever bytes catch your fancy")
Those are the sorts of decisions that give cryptographers the willies - and rightfully so. They've all been broken in various ways.
However - I'll point out that, just like choosing good primitives doesn't necessarily guarantee a secure protocol, choosing bad primitives doesn't necessarily guarantee an insecure protocol. Anecdotally, bad primitives have probably lead to insecure protocols more often than good primitives have lead to secure protocols, but that's beside the point. Bad primitives can still be secure as long as you side-step the parts of them that are broken.
Let's examine that:
- Would a break in SHA1 lead to a break in MTProto? (it would probably allow an attacker to forge a single chat message per collision; but we're not quite there yet as of 2016)
- Does the known problem with IGE lead to a break in MTProto? (no; they're not depending on the authentication)
- Is MAC-then-encrypt used improperly in MTProto? (no)
- Does the IND-CCA attack invalidate the security guarantees of MTProto? (the authors admit not, at least tentatively)
- Does the padding algorithm make MTProto susceptible to any attacks? (technically yes, since it's used in the IND-CCA attack, but again, it doesn't lead to known plaintext-recovery attacks)
Then why do those choices give cryptographers the willies? First because of experience: even if two choices are not problems on the surface, they can sometimes interact in non-intuitive ways to result in real problems. In a sense, problematic choices are anecdotally multiplicative in complexity and probability of breakage. (By analogy, that's an inherent property of high-dimensional design spaces.) Also, cryptographers have to think about lots of unknowns: are nation-state attackers using unpublished vulnerabilities in any of those underlying algorithms? Those are particularly concerning since, again, they can interact badly with other design decisions.
Axolotl / Signal Protocol
The Signal Protocol (formerly known as Axolotl) is, to some degree, an orange to MTProto's apple. It solves different problems, and is concerned with slightly different security model.
For example, Signal is primarily concerned with key derivation and updates/ratcheting (the Axolotl Ratchet, now called the Double Ratchet Algorithm; DRA for short), while MTProto is primarily concerned with the message envelope format, assuming that key exchange has already taken place via fairly generic Diffie-Hellman.
The goal of the Double Ratchet algorithm is to minimize the impact of compromised keys. This article explains how it works better than I ever could, but the gist is this:
DRA takes the best properties of DH ratcheting (used in OTR - the "Off The Record" IM protocol) and symmetric key ratcheting (used in SCIMP, the Silent Circle Instant Messaging Protocol).
Breaking that down:
- SCIMP uses a (one-way) KDF to derive a new key after every message is sent, and the previous key is promptly forgotten. This way, if the key is ever compromised, it can only be used to decrypt future messages, not past messages.
- OTR renegotiates DH keys for every round-trip of messages between parties, and thus self-heals compromised keys after a round-trip has taken place. This limits the exposure window, but the window can extend into the past in this case, unlike with SCIMP.
DRA on the other hand, regenerates per-message keys like SCIMP, but also renegotiates the DH keys for every round-trip.
The envelope used in the Signal Protocol is unremarkable (in a good way), when compared with MTProto:
- Encrypt-then-MAC (using HMAC-SHA256)
- AES in CTR mode, PKCS5 padding (>= version 3, at least)
No red flags there.
A formal analysis showed that the Signal Protocol (then called TextSecure) is vulnerable to an Unknown Key-Share attack, whereby B can trick A into sending message M to C, such that A believes it has actually sent M to B. The paper suggests a way to mitigate that attack, but I wasn't able to find any information as to whether that change has actually been applied.
Conclusion
In the envelope, Signal uses a simple Authenticated-Encryption (AE) system, whereas MTProto uses a custom wrapper that, through a series of odd choices, doesn't qualify as AE or IND-CCA. Although there are no known plaintext-recovery attacks in MTProto (yet), those properties could well lead to theoretical (or practical) attacks.
For key derivation, Signal has a good story to tell: compromising the current key will only compromise a set of future messages (possibly zero; until the next message from the other party arrives). However, in MTProto, compromising the authorization key (which is the closest equivalent key) allows all past and future messages to be compromised.
Personally, I (as a developer) wouldn't use MTProto except if I needed interoperability with Telegram. The folks behind the Signal Protocol (Open Whisper Systems) have fairly universally been praised for the security of their protocol, which makes it an obvious choice.
