Blocking of Cloudflare ECH in Russia, 2024-11-05

[wkrp]

[Discussion moved from #393 (comment). NTC threads are https://ntc.party/t/12837 (technical information) and https://ntc.party/t/12732 (discussion).]

Cloudflare's deployment of Encrypted Client Hello (ECH) is blocked in multiple networks in Russia since 2024-11-05. The blocking trigger is the presence of both of the following two elements in the Client Hello:

1. An SNI extension with the value cloudflare-ech.com.
2. An ECH extension.

Neither of these elements on its own is sufficient. That is, an SNI of cloudflare-ech.com without an ECH extension is not blocked, and ECH extensions that use an SNI other than cloudflare-ech.com are not blocked. In particular, you can still make ECH connections to servers that use a different public_name, such as defo.ie and tls-ech.dev; and GREASE ECH with SNI different from cloudflare-ech.com is not blocked.

Both TCP-based HTTP/2 and UDP-based HTTP/3 (QUIC) are affected. The blocking mechanism is packet dropping on the connection after the signature is detected (i.e., not a TCP RST or other overt teardown).

It seems that Firefox, at least, will retry the connection without ECH after a long delay (about a minute). Such a fallback to plaintext SNI apparently violates the ECH specification: "the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI."

The blocking of ECH was officially acknowledged in a notice from the Public Communications Network Monitoring and Control Center (ЦМУ ССОП):

https://cmu.gov.ru/ru/news/2024/11/07/рекомендуем-отказаться-от-cdn-сервиса-cloudflare/ (archive)

Рекомендуем отказаться от CDN-сервиса CloudFlare

Американская компания CloudFlare, поставщик услуг CDN, включила в октябре применение по умолчанию на своих серверах расширение TLS ECH (Encrypted Client Hello). Эта технология – средство обхода ограничений доступа к запрещенной в России информации. Его использование нарушает российское законодательство и ограничивается техническими средствами противодействия угрозам (ТСПУ).

Рекомендуем владельцам информационных ресурсов отключить расширение TLS ECH или, что правильнее, использовать отечественные CDN-сервисы, которые обеспечивают надежное и безопасное функционирование ресурсов и защиту от компьютерных атак.

В частности, защиту от DDoS-атак может обеспечить Национальная система противодействия DDoS-атакам (НСПА). За время ее работы (с марта 2024 года) отражено более 10,5 тыс. DDoS-атак на различные организации страны.

Обращаем внимание, что CloudFlare была одной из компаний BigTech, которые собирал Госдеп США в сентябре для обсуждения комплексного и организованного противодействия странам, активно защищающим свой информационный суверенитет (источник).

It is recommended to opt out of CloudFlare's CDN service

The American company CloudFlare, a provider of CDN services, in October enabled the default use of the TLS ECH (Encrypted Client Hello) extension on its servers. This technology is a means of circumventing restrictions on access to information banned in Russia. Its use violates Russian law and is restricted by the Technical Measure to Combat Threats (TSPU).

We recommend that owners of information resources disable the TLS ECH extension or, more correctly, use domestic CDN services that ensure reliable and secure functioning of resources and protection from computer attacks.

In particular, protection from DDoS attacks can be provided by the National System for Countering DDoS Attacks (NSPA). During its operation (since March 2024), more than 10.5 thousand DDoS-attacks on various organizations of the country have been reflected.

Note that CloudFlare was one of the BigTech companies that the U.S. State Department gathered in September to discuss a comprehensive and organized response to countries actively defending their information sovereignty (source).

OONI tests web connectivity to the SNI cloudflare-ech.com, and similarly GlobalCheck, but the measurements do not show blocking. That is because these tests do not have the other necessary part of the signature, the ECH extension. OONI is working on a dedicated ECH test.

#393 (comment)

Can you share some more details and possibly data supporting the claim that "All Cloudflare ECH-enabled services are blocked"? The cloudflare-ech.com domain has been added to OONI testing, however we do not see it blocked in the 23 networks it's been tested in so far: https://explorer.ooni.org/chart/mat?probe_cc=RU&since=2024-10-08&until=2024-11-08&time_grain=day&axis_x=measurement_start_day&test_name=web_connectivity&domain=cloudflare-ech.com.

…

We have an upcoming test in OONI Probe that should be able to test this, so it would be useful to collect some information on that once we have it out: ooni/probe#1453.

[wkrp]

Other reporting and discussion.

• Habr: Роскомнадзор, вероятно, начал блокировать подключение к CloudFlare с использованием TLS ECH (Encrypted Client Hello) Roskomnadzor has probably started blocking connections to CloudFlare using TLS ECH (Encrypted Client Hello) (archive)

  Show text

  Роскомнадзор, вероятно, начал блокировать подключение к CloudFlare с использованием TLS ECH (Encrypted Client Hello)

  С полуночи сразу в нескольких российских локациях наблюдается следующая ситуация:

  Chrome отваливается по таймауту при попытке зайти на любой из множетсва сайтов, проксируемых через CloudFlare с включенным TLS 1.3 (ECH это расширение TLS 1.3). При этом те же самые сайты с той же самой машины можно скачать с использованием wget, который не поддерживает ECH. После выключения TLS 1.3 на стороне CloudFlare сайты становятся доступны через несколько минут (отключение TLS 1.3 отключает и ECH тоже).

  Сайты с TLS 1.2 и ниже работают как обычно.

  Воспроизводится из нескольких локаций у разных операторов на примерно сотне сайтов.

  Через VPN те же сайты всё это время работают нормально.

  Среди попавших под раздачу оказались:

  https://openstreetmap.org
  https://diary.ru
  https://kopilkaurokov.ru
  https://uchitelya.com
  https://opensubtitles.org

  Из 10000 самых трафиковых сайтов по версии рейтинга alexa.com на CloudFlare заведены примерно 2500. Из них примерно 700 имеют в DNS запись, информирующую о поддержке ECH.

  Roskomnadzor has probably started blocking connections to CloudFlare using TLS ECH (Encrypted Client Hello)

  Since midnight the following situation is observed in several Russian locations at once:

  Chrome crashes on timeout when trying to access any of the many sites proxied through CloudFlare with TLS 1.3 enabled (ECH is an extension of TLS 1.3). At the same time, the same sites from the same machine can be downloaded using wget, which does not support ECH. After disabling TLS 1.3 on the CloudFlare side, the sites become available in a few minutes (disabling TLS 1.3 disables ECH too).

  Sites with TLS 1.2 and below work as normal.

  Reproduced from several locations at different operators on about a hundred sites.

  The same sites have been working fine via VPN all this time.

  Among the sites that got hit were:

  https://openstreetmap.org
  https://diary.ru
  https://kopilkaurokov.ru
  https://uchitelya.com
  https://opensubtitles.org

  Of the 10,000 highest traffic sites according to alexa.com rankings, about 2,500 are hosted on CloudFlare. Of these, about 700 have a DNS record informing about ECH support.

• Cloudflare Community: Cannot access sites from Russia

  Hello. In Russia, websites through cloudflare have not been working, for several hours now

  In Russia, because of the activities of the RKN, sites are inaccessible due to the ECH, but the fact is that on free tariffs it cannot be turned off, which means that a huge number of sites are inaccessible.

[jesowozapoc]

Since they are "suggesting" people to move to local alternatives to cloudflare. How likely it is that they gonna block all of the ip addresses of cloudflare?

[hellais]

Thanks for summarizing this information.

The upcoming ECH test in OONI, although it only supports the GREASEd ECH extension, based on what has been reported so far should be enough to test for this behaviour.

Based on the input in this thread we are adding support for testing a different server_name values in the OuterClientHello that are not cloudflare-ech.com to check if it would be enough for cloudflare to deploy a different echconfig in order to avoid the block.

as per: https://datatracker.ietf.org/doc/html/draft-ietf-tls-esni-22#section-6.1-6

It SHOULD place the value of ECHConfig.contents.public_name in the "server_name" extension. Clients that do not follow this step, or place a different value in the "server_name" extension, risk breaking the retry mechanism described in Section 6.1.6 or failing to interoperate with servers that require this step to be done; see Section 7.1.

the value inside of the OuterClientHello server_name should be determined by the public_name value of the echconfig and so it can be changed just at the DNS level.

Here is the diff of the changes to the test: ooni/probe-cli#1658 and specification: ooni/spec#297

If there are any thoughts on what we might want to do additionally to this, let us know. We are planning to ship this release of the measurement engine next week.

[wkrp]

It seems that Firefox, at least, will retry the connection without ECH after a long delay (about a minute). Such a fallback to plaintext SNI apparently violates the ECH specification: "the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI."

From what I've been able to gather from asking on NTC, it seems that both Firefox and Chrome have a fallback to retry an ECH connection without ECH. Firefox falls back automatically after a certain amount of time. Chrome falls back after clicking the Reload button multiple times, but the fallback is disabled if DoH is configured. Treat these observations as preliminary.

Is anyone an expert at searching Bugzilla or the Chromium Issue Tracker, or exploring the source code of Firefox or Chromium, who can find where these apparent non-ECH fallback policies might be enacted?

• In Firefox, the browser internally retries the connection (without the user clicking the Reload button), and eventually reconnects without ECH, after about 40 to 60 seconds. This happens whether or not the browser is configured to use DoH.

  https://ntc.party/t/12837/1

  В Firefox 131 сайты с ECH открываются без ECH через минуту «загрузки» — открывается новое соединение с plain SNI, а старое закрывается. Причину не знаю, возможно, это из-за того, что у меня отключён DNS-over-HTTPS, но ECH всё равно используется.

  In Firefox 131 sites with ECH open without ECH after a minute of "loading" — a new connection with plain SNI is opened, and the old one is closed. I don't know the reason, maybe it's because I have DNS-over-HTTPS disabled, but ECH is still used.

  https://ntc.party/t/12732/310

  for example : https://www.hwinfo.com/download

  windows 10 ltsc / firefox / cf dns on router / cf doh in browser
  first attempt browser tries to open with ech:

  Client Hello (SNI=cloudflare-ech.com)

  then after few retransmissions it gets RST

  [RST, ACK]

  then second attempt with same actions

  Client Hello (SNI=cloudflare-ech.com)

  and then it opens with hwinfo client hello after 40 seconds

  Client Hello (SNI=www.hwinfo.com)

  Attempts mean attempts by browser , I didnt click refresh button

  https://ntc.party/t/12732/312

  DoH в Firefox принудительно или дефолт?

  у меня trr mode = 3 использовать только TRR (если ты про это спрашиваешь)

  DoH in Firefox forced or default?

  I have TRR mode = 3 use only TRR (if you ask about it)

• In Chrome, if you click the Reload button enough times (around 6 times), eventually the browser will retry without ECH. However, if the browser is configured to use DoH, the fallback does not occur.

  https://ntc.party/t/12732/65

  Проверил сайты (в последнем ungoogled chromum 130 на линуксе):

  С DoH и ECH под Йотой все не открываются.
  Без DoH и с провайдеровским DNS тоже почти все не открываются, но может иногда один начать открываться (sakhtv.ru) после чистки профиля или наоборот перестать открываться.

  Checked the sites (in the latest ungoogled chromum 130 on linux):

  With DoH and ECH under Yota, everyone does not open.
  Without DoH and with ISP DNS, almost all of them do not open either, but sometimes one of them starts to open (sakhtv.ru) after clearing the profile or vice versa stops opening.

  https://ntc.party/t/12732/311

  Chrome 130.0.6723.116 (официальная сборка) без DoH в браузере на Linux после нескольких нажатий “Обновить” отключает ECH для домена.

  Chrome 130.0.6723.116 (official build) without DoH in the browser on Linux after a few clicks of "Reload" disables ECH for the domain.

  https://ntc.party/t/12732/309

  Chrome 130.0.6723.116 (официальная сборка от Google) с DoH в браузере (1.1.1.1) на Linux не пытается установить соединение без ECH даже после нескольких нажатий "Обновить" (кроме того, и без кнопки "Обновить" он сам безуспешно пытается восстановить коннект). Таймаут после 1 минуты, а потом ошибка.

  Chrome 130.0.6723.116 (official build from Google) with DoH in the browser (1.1.1.1) on Linux does not try to establish a connection without ECH even after a few clicks of "Reload" (in addition, even without the "Reload" button, it unsuccessfully tries to restore the connection itself). Timeout after 1 minute, and then an error.

[FazziCLAY]

disable-ech.sh
https://gist.github.com/FazziCLAY/75f72acc8b728530a637121fdee4dfb5

disable-ech.bat https://gist.github.com/FazziCLAY/38f56ab423a0e0a2f864985cf3ce21be

[vanyasem]

To disable ECH client-side on Chromium based browsers you can take advantage of Enterprise Chromium Policies (the name is quite misleading, the policies can be applied to any Chromium-based browsers, and do not require an account, or an enterprise subscription)

The policy in question is called EncryptedClientHelloEnabled

Google's docs on Chromium policies are very confusing, so I suggest to read Brave Browser's documentation on that topic instead: https://support.brave.com/hc/en-us/articles/360039248271-Group-Policy

I will be providing examples for Brave Browser, but the same principle applies to any Chromium browser. You just have to substitute brave's paths to whatever browser's paths you're using.

Instructions differ based on the OS you're using:

macOS:

Run in the terminal:

defaults write com.brave.Browser EncryptedClientHelloEnabled -bool false

Windows:

Uncompress and apply this reg file brave.reg.zip

Under the hood it modifies the registry in [HKEY_LOCAL_MACHINE\Software\Policies\BraveSoftware\Brave], creating an entry EncryptedClientHelloEnabled with the value of dword:00000000

GNU/Linux:

Run in the terminal:

sudo mkdir -p /etc/brave/policies/managed/ && echo '{"EncryptedClientHelloEnabled": false}' | sudo tee /etc/brave/policies/managed/managed_policies.json

You can confirm that the policy is applied by checking chrome://policy/:

[ValeZAA]

The reason it works on Windows 10 vs 11 is that on Windows 10 there is a bug in WIndows API and it cannot request HTTPS RR that containts ECH key.
https://phabricator.services.mozilla.com/D193656

// For some reason, the DNSQuery_A API doesn't work on Windows 10. 
// It returns a success code, but no records. We only allow 
// native HTTPS records on Win 11 for now.
 

[KershawChang]

It seems that Firefox, at least, will retry the connection without ECH after a long delay (about a minute). Such a fallback to plaintext SNI apparently violates the ECH specification: "the client MUST NOT fall back to using unencrypted ClientHellos, as this allows a network attacker to disclose the contents of this ClientHello, including the SNI."

From what I've been able to gather from asking on NTC, it seems that both Firefox and Chrome have a fallback to retry an ECH connection without ECH. Firefox falls back automatically after a certain amount of time. Chrome falls back after clicking the Reload button multiple times, but the fallback is disabled if DoH is configured. Treat these observations as preliminary.

Is anyone an expert at searching Bugzilla or the Chromium Issue Tracker, or exploring the source code of Firefox or Chromium, who can find where these apparent non-ECH fallback policies might be enacted?

In Firefox, the fallback to a non-ECH connection is controlled by the preference network.dns.echconfig.fallback_to_origin_when_all_failed, which is set to false by default.

When DoH is configured in Firefox, the browser waits for the DNS lookup of the HTTPS record to complete before establishing connections to ensure that ECH is used. If DoH is not configured, Firefox attempts to look up the HTTPS record using the native resolver but does not wait for the DNS lookup to complete. In this scenario, Firefox may fallback to a non-ECH connection because the HTTPS record has not been resolved during the connection establishment process.

If anyone can reproduce a case where Firefox makes a non-ECH connection with DoH enabled, please file a bug in Bugzilla and include an HTTP log. We will investigate the issue.

Thanks.

[ckcr4lyf]

Checking for both the ECH extension AND SNI as cloudflare-ech.com seems just as an initial sanity check.

If one really wanted to stop ECH, they would block all traffic with the ECH extension. Sure, this would block GREASE traffic as well, but the protocol is new so there is no real pressure for them to need to allow it, instead just suggest that clients use e.g. TLSv1.2 instead (or don't set this extension).

[ValeZAA]

wanted to stop ECH, they would block all traffic with the ECH extension

There is no any ECH extension. It looks like there is no SNI, which is what happens when you go to https://1.1.1.1 e.g.

Why do you think there is a fake SNI field there? Because in the future it will be randomised and thus cannot be used to block.

[wkrp]

There is no any ECH extension. It looks like there is no SNI, which is what happens when you go to https://1.1.1.1 e.g.

I beg your pardon, but you are wrong about that.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-23.html#section-1

This document specifies a new TLS extension, called Encrypted Client Hello (ECH), that allows clients to encrypt their ClientHello to the TLS server.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-23.html#section-5

To offer ECH, the client sends an "encrypted_client_hello" extension in the ClientHelloOuter. When it does, it MUST also send the extension in ClientHelloInner.

enum {
  encrypted_client_hello(0xfe0d), (65535)
} ExtensionType;

Why do you think there is a fake SNI field there?

With ECH, the client sends two ClientHellos: a plaintext ClientHelloOuter and an encrypted ClientHelloInner. The ClientHelloInner is the "real" ClientHello, and it is stored inside the ECH extension.

https://www.ietf.org/archive/id/draft-ietf-tls-esni-23.html#section-3.2

When a client wants to establish a TLS session with some backend server, it constructs a private ClientHello, referred to as the ClientHelloInner. The client then constructs a public ClientHello, referred to as the ClientHelloOuter. The ClientHelloOuter contains innocuous values for sensitive extensions and an "encrypted_client_hello" extension (Section 5), which carries the encrypted ClientHelloInner. Finally, the client sends ClientHelloOuter to the server.

[ValeZAA]

This document specifies a new TLS extension, called Encrypted Client Hello (ECH), that allows clients to encrypt their ClientHello to the TLS server.

So? I was right, there is no any extension. There is only a extension inside of the encryption... If there were not any difference between ECH and TLS it would be just using DNS keys, which is certainly possible soon.

offer ECH, the client sends an "encrypted_client_hello" extension in the ClientHelloOuter

If there is HTTPS RR it will tell client what the ipv6, ipv4 and http/3 and ech status and its key

The ClientHelloOuter contains innocuous values for sensitive extensions and an "encrypted_client_hello" extension

How is SNI of domain that has ech in it is inoccent?

[wkrp]

This document specifies a new TLS extension, called Encrypted Client Hello (ECH), that allows clients to encrypt their ClientHello to the TLS server.

So? I was right, there is no any extension. There is only a extension inside of the encryption... If there were not any difference between ECH and TLS it would be just using DNS keys, which is certainly possible soon.

I don't know what to tell you. Try looking at a pcap; the extension number is 0xfe0d.

The ClientHelloOuter contains innocuous values for sensitive extensions and an "encrypted_client_hello" extension

How is SNI of domain that has ech in it is inoccent?

You ought to read the ECH draft if you really want to understand it. I recommend at least reading the "Do Not Stick Out" section, which explains the rationale for much of the design. One may dispute whether the GREASE and outer SNI arguments are convincing, but it is a fact that a TLS connection using ECH has both an ECH extension and an SNI extension.

As a means of reducing the impact of network ossification, [RFC8744] recommends SNI-protection mechanisms be designed in such a way that network operators do not differentiate connections using the mechanism from connections not using the mechanism. To that end, ECH is designed to resemble a standard TLS handshake as much as possible. The most obvious difference is the extension itself: as long as middleboxes ignore it, as required by [RFC8446], the rest of the handshake is designed to look very much as usual.

The GREASE ECH protocol described in Section 6.2 provides a low-risk way to evaluate the deployability of ECH. It is designed to mimic the real ECH protocol (Section 6.1) without changing the security properties of the handshake. The underlying theory is that if GREASE ECH is deployable without triggering middlebox misbehavior, and real ECH looks enough like GREASE ECH, then ECH should be deployable as well. Thus, the strategy for mitigating network ossification is to deploy GREASE ECH widely enough to disincentivize differential treatment of the real ECH protocol by the network.

For HTTPS RRs, see RFC 9640 and the Bootstrapping TLS Encrypted ClientHello with DNS Service Bindings draft. HTTPS records in DNS are used to construct the ECH extension; they are not a replacement for it.

[ValeZAA]

Oh

[Motophan]

The ECH request isnt suppose to fallback to unencrypted SNI

That would be like if you had https took > 30 seconds you wouldnt want to resent your username and password over plaintext in http. This would be poised for all kinds of abuse...