Download speed tests (2020-04-30)

Original post.

I did some experiments of download performance of the DNS tunnel. tl;dr a DNS tunnel can go faster than you may think, but the choice of resolver matters a lot.

I tried downloading a 10 MB file through the tunnel, using a selection of resolvers and DNS transports. I cut off the download after 10 minutes. "none" is the special case of no intermediate recursive resolver (the tunnel client sends queries directly to the tunnel server). The server was located in Fremont, US and the client in Tokyo, JP. There was about 100 ms of latency between the two hosts. Download rates are the median of 5 trials. The dnstt tag was v0.20200430.0.

Cloudflare's DoH and DoT resolvers are both fast. Google's DoH resolvers is much faster than its DoT server (I noticed the DoT server terminating TCP connections every 200 KB or so). Comcast's DoH and DoT resolvers have about the same middling performance. Quad9's DoT resolver is notably slow; there's clearly something wrong there, whether it's the resolver or how the tunnel uses it. For comparison, the download rate of an untunnelled, direct TCP transfer was 4666.3 KB/s.

I repeated the experiment with iodine, an existing DNS tunnel. iodine works over plaintext UDP only. dnstt is faster than iodine in every case, except for the Quad9 DoT resolver. It is possible to run iodine over a DoH proxy; I didn't try that myself but Sebastian Neef reports 4–12 KB/s when tunneling iodine through dnscrypt-proxy.

This graph shows the 5 trials under each experimental condition and gives an idea of the variance. Steeper lines are better.

Download speed tests (2021-08-02)

I ran another set of tests to evaluate the effects of tuning certain performance parameters. While working on Champa, I found that increasing the size of smux Config.MaxStreamBuffer and tweaking other parameters could improve download speeds. For background, see these links:

• Increase default send and receive windows to 1024 (Champa)
• Higher defaults for smux MaxReceiveBuffer and MaxStreamBuffer (Champa)
• Increase turbotunnel queueSize to 256 (Champa)
• Improving Snowflake performance by adjusting smux parameters (Snowflake)
• Increase kcp and smux window and buffer sizes (Snowflake)

I sought to see if a similar improvement in performance was possible in dnstt. As it turns out, it's possible to greatly speed up a UDP transport directly to the dnstt server (i.e., using -udp and giving the address of the server itself, without a recursive resolver in the middle)—but this is a discouraged configuration. When using the recommended configuration of recursive resolver in the middle, I was able to achieve modest improvements in download speed in only a small number of configurations. I re-ran the download test using v0.20200430.0, the same version used in the previous round of testing, and compared it to v1.20210803.0, which has the performance tweaks.

As in the previous tests, I tried downloading a 10 MB file for up to 10 minutes. The server was located in Fremont, US and the client in Tokyo, JP. There was about 100 ms of latency between the two hosts. The following table summarizes the results. The rates shown are the median of 3 trials.

For a few more tables, see the commit messages of:

• Make pollChan buffered, and send on it only once.
• Performance tuning: MaxStreamBuffer, SetWindowSize, QueueSize.

Some observations on the results:

• The speed of a direct UDP connection, without a recursive resolver, nearly doubles. In fact, I was able to increase the speed of this configuration to 1 MB/s or more by really increasing some parameters, but it did not proportionally help other configurations, and could even slightly hurt them.

• Comcast (Xfinity) apparently no longer runs a DoT server at dot.xfinity.com:853 (connection times out).

• The Cloudflare/UDP configuration was one of the fastest in 2020; now it is the slowest. I did not dig into this much, but I did see that the Cloudflare DNS over UDP resolver would sometimes send responses to queries without actually forwarding them to the authoritative resolver (i.e., the dnstt server). These responses contained 0 Answers, and had the TC (truncation) bit set, despite being the same size as the corresponding queries.

The Quad9/UDP and Google/DoT configurations (marked with a * in the table above) look like they have decreased in speed by more than half. Taking a closer look at the amount of data downloaded over time, we can see some unusual features that help explain these changes. One of the three Google/DoT trials in the new version finished faster than in the old version (3.5 minutes versus 4 minutes), but the other two trials, the TLS connection was broken after a minute (which in itself is not unusual), but re-establishing the TLS connection failed because of a DNS error. Compare the files 2021-08-02/data/2-de15c5a5/google_dot_1.dnstt.client.log and 2021-08-02/data/2-de15c5a5/google_dot_2.dnstt.client.log with 2021-08-02/data/2-de15c5a5/google_dot_3.dnstt.client.log. In two of the three Quad9/UDP trials, there is an obvious stairstep pattern: the download quickly makes progress, then stalls, then makes progress, then stalls, and so on. I don't know what is going on here, except that it may be some kind of rate limiting at the server. A similar phenomenon is visible with Quad9/DoH in the old version. There, the cause of the stalls is visible in the logs: a server response status of "500 Internal Server Error" or "502 Bad Gateway" causes dnstt-client to self-throttle for 10 seconds. See the files data/0-3254c1c8/quad9_doh_1.dnstt.client.log and data/0-3254c1c8/quad9_doh_2.dnstt.client.log.