Back

Mysteries of Telegram Data Centers (2022)

187 points6 hoursdev.moe
Peanuts992 hours ago

The more I learn about telegram, the sketchier it seems.

wronex2 hours ago

Agreed. I don’t see the appeal compared to Signal. Although, Signal is also sketchy with an operating cost in the tens of millions of dollars per year. Where does the money come from?

isatty2 hours ago

- Telegram has exemplary fast, native clients on most platforms I’ve used it on

- Cat stickers

- Did I mention it has the best native clients out of all the messaging apps? It boggles the mind why other companies can’t get this done.

mdasen33 minutes ago

I'll add:

- Telegram had usernames in 2014 before Signal added them a decade later, allowing people to chat without sharing their phone number

- Telegram has unencrypted chats which allow for giant chat rooms of 200,000+ and channels with millions of subscribers. Signal warns about performance issues when you have more than 150 people in a group. Telegram isn't just a messenger - it's often used as a social publishing platform like Instagram.

I don't use Telegram and use Signal a lot, but I also understand why other people use Telegram: the same reason they use Instagram.

SXX1 hour ago

Durov was smart enough to let community build open source clients and use them. And to make internally built clients open source.

josephg2 hours ago

I’ve never used the telegram app. What do you like about it compared to signal / WhatsApp?

tjohns1 hour ago

- Messages send quickly and reliably, even under poor and sometimes hostile network conditions. Telegram just seems to work even when other chat apps struggle.

- Telegram uses usernames instead of phone numbers by default, which is good if you're using it as an IRC replacement instead of an SMS replacement.

- You can have the client open on essentially unlimited devices simultaneously, including a web app if you need it.

- Messages can be edited at any point after sending with no expiry.

- You can schedule messages to send later, or send a message silently so it doesn't wake people up.

- Different group types - announcement channels, Discord-style groups with sub-channels, flexible moderator roles, etc. (I believe WhatsApp has some of this.)

- Support for bots, which is also very helpful for managing large communities.

- Community-created, sharable stickers. Seriously, people underestimate how nice these are.

The downside is that a lot of this requires state to be stored on the Telegram servers, so most chat's aren't E2E encrypted. (They do have an option for E2E encrypted private 1:1 chats, but you lose most of the polish by using that.)

Also, the official apps are open source, so you can modify them if needed.

+1
isatty1 hour ago
asdff1 hour ago

Probably still doesn't beat ripcord

snailmailman1 hour ago

I’ve always been under the impression that Signal is for secure, private chats and group chats amongst friends and small groups. While telegram is often used more like discord or irc, with “secure” and “private” group chats that are extremely large and semi-public. “Invite only” but invites are handed out easily. Those chats are pretty obviously not as secure, as basically anybody can join them and decrypt the chat. On the surface its somewhat more secure than discord, where discord will be scanning all chat content.

johnnyApplePRNG1 hour ago

Apparently it's funded by your friendly neighbor.

I've used and promoted Signal for years and I've recently become suspicious of them and their funding as well after looking into starting my own encrypted communications app.

It's not cheap sending dozens or hundreds of megabytes of video files or whatever ... whenever the user feels like it, mind you ... with a monetization strategy that's literally simply hoping that donations will cover it?

That's insane.

jacobgold53 minutes ago

Signal users who want to use it with their agents are running an unofficial extracted-and-patched `signal-cli` off GitHub. It's based on an archived official Signal repo and then patched for years by some random accounts. It looks incredibly untrustworthy.

Meanwhile Telegram has bot support and added features specifically for interacting w/agents. It's incredibly easy to write clients and work with it. No one should use it, and I never would, but you can see why it's winning.

Signal's lack of features (like an official Signal CLI) and bots (even attached to existing phone numbers and limited to the owner) is making people less secure than they could be. And unfortunately there are no great alternatives.

flexagoon3 hours ago

Just like DC5 is often down to the discontent of Chinese users, DC2 is the one serving all Russian and Ukrainian users, so in the more technical Russian-speaking communities "dc2 down" is also a pretty common saying

SpaceL10n3 hours ago

For a hot second there I was really excited to learn about historical telegram "data centers".

It's a capital T.

Terr_1 hour ago

As a tangential niche-tourism shoutout: https://www.telcomhistory.org/ConnectionsSeattle.html

Granted, most of it is a bit newer than the heyday of telegrams, but there's a little bit of overlap.

sophacles3 hours ago

You may have already read it, but portions of this sprawling (and very very good) essay by Neal Stephenson get into the topic: Mother Earth Mother Board - https://www.wired.com/1996/12/ffglass/

glaslong4 hours ago

The DC3 gap is interesting. I wonder whether they deprecated it because the other EU server had plenty of capacity, or still keep it but only for... "special" account data flow.

Also, it looks like it's easy enough to ID your DC on their API, though I haven't tried it yet (more of a Matrix Stan personally): https://core.telegram.org/method/help.getConfig

AntronX5 hours ago

DC in Miami, explains why Telegram app is snappy fast for me. I notice similar speed improvement with Meta and other big tech apps when I'm on the west coast. I guess latency matters when your app is making tons of requests.

Anon10964 hours ago

Most of big tech's major data centers are in Loudoun County, VA on the east coast not the west coast. It's centrally located to be great latency for the east coast and OK latency for west coast and Europe. Plus a friendly regulatory environment and lots of existing DCs (AWS us-east-1, Azure East US 1/2)

If you're feeling any better latency on the west coast it's more likely to be placebo than real.

tobinfekkes4 hours ago

On the contrary, Big tech famously has plenty of data centers on the west coast:

• Quincy, WA (Microsoft)

• The Dalles, Oregon (Google)

• Prineville, Oregon (Facebook, Apple)

• Hillsboro, Oregon (Cloudflare, others)

• Boardman, Oregon (AWS)

Anon10964 hours ago

By traffic load east coast datacenters dwarfs these.

+1
tobinfekkes4 hours ago
wronex2 hours ago

Are people still using Telegram? What is the upside compared to Signal?

crazybonkersai2 hours ago

Good ux, pleasant to use. A large community and lots of channels with all kind of content. Api is also great for spinning a bot for whatever purpose. I have for example a critical error bot for a production server running. If a critical error occurs, I get an immediate telegram message.

wronex1 hour ago

Sounds a lot like we might as well use Discord?

arkaic53 minutes ago

Discord is annoying af to use, slow and bloated like the rest of them. Really it all comes down to UX when anyone asks why use Telegram.

gf00049 minutes ago

How is that any better? Still no e2ee, proprietary, arguably worse UX for chat purposes.

Aachen58 minutes ago

Calling/ringing works reliably. My gf usually calls via Telegram to notify me and then we call via Signal

    Code formatting was the first thing my friend ran into after switching last week
In general, formatting on Telegram is a lot more versatile with >quote blocks that show up formatted as quotes and inline links without needing to have a giant message because of the giant URL

Many more image features, like you can do yellow highlighting and not only ugly shades of olive, you can put a message above an image instead of below, you can mark an image as spoiler, you can send it uncompressed, etc.

Jumping to a date was one of the first things I ran into

Web client lets you temporarily use someone else's computer without needing to install software

Making a Telegram bot takes 10 minutes if you just want a simple http callback and know roughly how it works. Signal has no support at all. Some third parties have made libraries but we keep having issues at work with our internal Signal bot to the point where I think we just gave up altogether now

Many people use Telegram to check if internet works because when opening the app it does a lightning quick check and it's very robust (I think it has no problem if DNS is down, for example) Much faster than trying to do a random web search

You can have memorable/pretty group URLs like telegram.me/OpenStreetMapOrg

Large files can be sent

Scheduled messages send at the right time when you're not online & the desktop app supports scheduled messages

Chats take a lot less space. The Signal authors probably don't have many contacts that they use Signal with because you can view the message markers of 4 chats before having to scroll down. In Telegram 9 fit in view. Similar on desktop but both numbers are bigger. Messages are also more space efficient in Telegram

Telegram so so so much faster than Signal's web, ahem, electron client

Literally the only advantage of Signal is the intangible property of privacy. I'm nevertheless switching everything over to Signal but it's often very hard to explain to people why they should give all this up for encryption. My family doesn't use all the advanced features but some techy friends do

St0n3d1 hour ago

If you’re ok with abandoning all security and privacy, then Telegram has some nice extra features that can come in handy.

strebz5 hours ago

DC2 is the first connection point of all MTProto clients.

Any DC may refuse a request and force the client to switch DC.

Profile URL doesn't show where messages/chats/channels are stored, as telegram has two dedicated DCs mostly for media. The rest DCs allow media with bandwidth being throttled.

londons_explore6 hours ago

This strikes me as a huge amount of custom code and technical debt. Every new software dev probably has to learn this.

Why not a sticky master election per user, and have no special data centers?

codedokode5 hours ago

It makes sense: European users are assigned to EU data center, and Chinese to the one closer to them. The "custom code" should not be complicated, just a map of country to DC.

You are suggesting to develop a compicated solution (spend money) when current simple one is working ok without any elections.

inigyou4 hours ago

If you've ever actually tried to implement server clustering you quickly find there's no magic cloud, except in specific cases like blockchains. A privately operated cluster system is mostly about directing requests to the correct server out of a finite set of servers.

fullstop6 hours ago

From what I have read, they only have ~30 employees. They're not exactly onboarding a lot of new people here.

nurumaik4 hours ago

Learn what? How to count to 5?

_ink_5 hours ago

They claim that they store user data on different servers in different jurisdictions so it becomes more difficult for authorities to gain access [1]. Maybe that's true and it has something to do with these DCs that seem to be unused.

[1] - https://telegram.org/privacy

amima4 hours ago

They do not claim that. They do claim that they store specifically encryption keys in several data centers in different jurisdictions. Here is the exact quote: "All data is stored heavily encrypted and the encryption keys in each case are stored in several other data centers in different jurisdictions". So only keys are distributed.

hkpack4 hours ago

What does heavily encrypted even mean? Fully encrypted? Slightly encrypted? Encrypted enough to call it “heavily encrypted” but not enough to be protected from whoever is interested?

Perz1val3 hours ago

Heavily means the key is large so it takes longer to crack, but also longer to encrypt/decrypt, so the service is more costly to run and slower. At least I've seen it used that way

maqp3 hours ago

There's nothing slow about AES.

In this context "heavily" means "we can't legally claim it's end-to-end encrypted because it's not".

Also it's not even post quantum, so it's not heavy. Telegram's Diffie-Hellman breaks instantly with a quantum computer large enough to run Shor against it.

Also, the keys sit on the servers' RAM, no matter what they lie. There is no global distributed RAM system, especially one that encrypts data in distributed fashion and works at the negligible latencies that Telegram boasts.

dakolli3 hours ago

telegram is the safest encrypted messaging app. Period, full stop.

+1
maqp3 hours ago
bflesch4 hours ago

It's more about the fact that five eyes intelligence services prefer to officially spy on each other's countries so they don't have to answer to their respective bureaucrats. They prefer plausible deniability.

Something like this:

   DC1 politically belongs to UK which "spies" on CA/US but physical servers are located in US so US ultimately retains control.

   DC2 politically belongs to France which "spies" on RUS/UKR/DE but physical servers are located in NL (e.g. in UK because one wouldn't be able to spot difference in ping). Maybe it's politically owned by UK/NZ or UK/AUS because France can't be publicly caught spying on Germany. But France wouldn't risk public arrest of Telegram CEO and the spectacle with russia if there is nothing to gain.

   DC4 politically belongs to USA which "spies" on UK/Israel but physical servers are in NL/UK

   DC5 politically belongs to UK/USA which "spies" on AUS/China/India but physical servers are in Singapore (e.g. former UK colony)

I love mentioning the UK in these kind of discussions because the pushback is biggest every time the Crown is mentioned, and ultimately US/CA/NZ/AUS are all colonies under the King.

Really cool to see realpolitik mapped out like this. It also highlights the problem of metadata with these kind of topics.

orbital-decay3 hours ago

The reason it's in Singapore is that Telegram can't operate in China, and Singapore-washing is the closest thing to doing it. A ton of VPNs and other services targeting mainland users but not allowed in the mainland are hosted there, it's a huge hub for companies and networks.

bwunkhaus2 hours ago

I've never listened about that but also im not a big telegram user... but that completely explains why mine is so slow.... I'm on latam and my account is on singapur..

bflesch4 hours ago

Beautiful analysis. It really looks like the country distribution [1] follows the geographical split between five eyes intelligence services, and maybe a small slice for France after they imprisoned the Telegram CEO [2] in order to take over data ownership from russia.

[1] https://dev.moe/wp-content/uploads/2022/05/image-14.png

[2] https://en.wikipedia.org/wiki/Arrest_and_indictment_of_Pavel...

DOGMATICA6 hours ago

i'm far from an authority on content delivery or whatever, but the first thing I thought of was what a bizarre way to setup your infrastructure!

Ghoelian5 hours ago

idk, they probably tried to get people on DC's as close to their location as possible. Using your phone number's country code might seem like a good way to do this at first, and they probably didn't give it much more thought before building the whole thing on this idea.

486sx336 hours ago

[dead]

hocuspocus5 hours ago

I'm on DC5 since I lived in Korea when signing up, but I cannot say I've noticed many outages.

hashtag-til5 hours ago
hhh6 hours ago

something smells suspicious about this kind of data routing

overallduka4 hours ago

The Lex Fridman podcast episode with Pavel Durov is worth listening to. Their servers are built to be very secure — of course, it would be different for others, and they use some clever tricks

dewey2 hours ago

That has "military grade encryption" vibes.

dgroshev5 hours ago
overallduka4 hours ago

Good story, I yet believe the guy is trying to do the right thing. In the lex Friedman podcast he talks about banning extremist channels in both sides always, the story focus more on the Nalvani's block, but accordingly him he also bans other sides depending on the content. I do follow a number of Telegram channels about the Ukraine war, and the pro Ukraine channels are there together with pro Russian channels.

dgroshev3 hours ago

Of course he would say that. Even besides the fallaciousness of this argument (there is no equivalence between an aggressor and a victim), Pavel Durov is completely untrustworthy.

There is product stuff, like misleading claims about Telegram's encryption and comparisons to Signal. In reality, for the vast majority of chats they have plaintext, unlike Signal.

There are more subtle positioning claims. Durov made a huge deal of his "exile", but I saw Telegram's office in St Petersburg with my own two eyes a year after Durov "fled the country". It definitely wasn't shut. There were even local news of Pavel assaulting a guy in St Petersburg for trying to make a photo of him a couple years later.

And then there are just completely unnecessary lifestyle claims. He said multiple times how he doesn't take any "pills" or medications. It only takes a minute to find his old photos. Male pattern baldness doesn't stop progressing without DHT suppression, and last time I checked, finasteride comes in pills. I don't understand why would he make misleading statements about something so visible, but it doesn't make him any more trustworthy.

vvpan4 hours ago

It's a great an telling investigation. Dropped in to share it as well. Telegram deserves no trust from us.

dubbel4 hours ago

The article is from May 2022, just fyi.

ameliaquining4 hours ago

I think it was reposted today because https://core.telegram.org/bots/serverless was just announced, which prompted some curiosity on HN about Telegram's architecture (https://news.ycombinator.com/item?id=48918534). But yes, the title should be updated to indicate this.