Turbo Tunnel candidate protocol evaluation

This report evaluates selected reliable-transport protocol libraries for their suitability as an intermediate layer in a censorship circumvention protocol. (The Turbo Tunnel idea.) The three libraries tested are:

• quic-go
• kcp-go paired with smux
• pion/sctp

The evaluation is mainly about functionality and usability. It does not specifically consider security, efficiency, and wire-format stability, which are also important considerations. It is not based on a lot of real-world experience, only the sample tunnel implementations discussed below. For the most part, I used default settings and did not explore the various configuration parameters that exist.

The core requirement for a library is that it must provide the option to abstract its network operations—to do all its sends and receives through a programmer-supplied interface, rather than by directly accessing the network. All three libraries meet this requirement: quic-go and kcp-go using the Go net.PacketConn interface, and pion/sctp using net.Conn. Another requirement is that the protocols have active Go implementations, because Go is currently the closest thing to a common language among circumvention implementers. A non-requirement but nice-to-have feature is multiplexing: multiple independent, reliable streams within one notional connection. All three evaluated libraries also provide some form of multiplexing.

Sample tunnel implementations

As part of the evaluation, I wrote three implementations of a custom client–server tunnel protocol, one for each candidate library. The tunnel protocol works over HTTP—kind of like meek, except each HTTP body contains a reliable-transport datagram rather than a raw chunk of a bytestream. I chose this kind of protocol because it has some non-trivial complications that I think will be characteristic of the situations in which the Turbo Tunnel design will be useful. In particular, the server cannot just send out packets whenever it wishes, but must wait for a client to make a request that the server may respond to. Tunnelling through an HTTP server also prevents the implementation from "cheating" by peeking at IP addresses or other metadata outside the tunnel itself.

turbo-tunnel-protocol-evaluation.zip

All three implementations provide the same external interface, a forwarding TCP proxy. The client receives local TCP connections and forwards their contents, as packets, through the HTTP tunnel. The server receives packets, reassembles them into a stream, and forwards the stream to some other TCP address. The client may accept multiple incoming TCP connections, which results in multiple outgoing TCP connections from the server. Simultaneous clients are multiplexed as independent streams within the same reliable-transport connection ("session" in QUIC and KCP; "association" in SCTP).

An easy way to test the sample tunnel implementations is with an Ncat chat server, which implements a simple chat room between multiple TCP connections. Configure the server to talk to a single instance of ncat --chat, and then connect multiple ncats to the client. Then end result will be as if each ncat had connected directly to the ncat --chat: the tunnel acts like a TCP proxy.

run server:
	ncat -l -v --chat 127.0.0.1 31337
	server 127.0.0.1:8000 127.0.0.1:31337

run client:
        client 127.0.0.1:2000 http://127.0.0.1:8000
        ncat -v 127.0.0.1 2000 # as many times as desired

.-------.
:0 ncat |-TCP-.
'-------'     |
              | .------------.        .------------.       .------------------.
.-------.     '-:2000        |        |            |--TCP--:31337             |
:0 ncat |-TCP---:2000 client |--HTTP--:8000 server |--TCP--:31337 ncat --chat |
'-------'     .-:2000        |        |            |--TCP--:31337             |
              | '------------'        '------------'       '------------------'
.-------.     |
:0 ncat |-TCP-'
'-------'

As a more circumvention-oriented example, you could put the tunnel server on a remote host and have it forward to a SOCKS proxy—then configure applications to use the tunnel client's local TCP port as a local SOCKS proxy. The HTTP-based tunnelling protocol is just for demonstration and is not covert, but it would not take much effort to add support for HTTPS and domain fronting, for example. Or you could replace the HTTP tunnel with anything else, just by replacing the net.PacketConn or net.Conn abstractions in the programs.

quic-go

• Home page
• API documentation
• Version used: v0.12.0 with go1.12.9

quic-go is an implementation of QUIC, meant to interoperate with other implementations, such as those in web browsers.

The network abstraction that quic-go relies on is net.PacketConn. In my opinion, this is the right abstraction. PacketConn is the same interface you would get with an unconnected UDP socket: you can WriteTo to send a packet to a particular address, and ReadFrom to receive a packet along with its source address. "Address" in this case is an abstract net.Addr, not necessarily something like an IP address. In the sample tunnel implementations, the server "address" is hardcoded to a web server URL, and a client "address" is just a random string, unique to a single tunnel client connection.

On the client side, you create a quic.Session by calling quic.Dial or quic.DialContext. (quic.DialContext just allows you to cancel the operation if wanted.) The dial functions accept your custom implementation of net.PacketConn (pconn in the listing below). raddr is what will be passed to your custom WriteTo implementation. QUIC has obligatory use of TLS for connection establishment, so you must also provide a tls.Config, an ALPN string, and a hostname for SNI. In the sample implementation, we disable certificate verification, but you could hard-code a trust root specific to your application. Note that this is the TLS configuration, for QUIC only, inside the tunnel—it's completely independent from any TLS (e.g. HTTPS) you may use on the outside.

tlsConfig := &tls.Config{
	InsecureSkipVerify: true,
	NextProtos:         []string{"quichttp"},
}
sess, err := quic.Dial(pconn, raddr, "", tlsConfig, &quic.Config{})

Once the quic.Session exists, you open streams using OpenStream. quic.Stream implements net.Conn and works basically like a TCP connection: you can Read, Write, Close it, etc. In the sample tunnel implementation, we open a stream for each incoming TCP connection.

stream, err := sess.OpenStream()

On the server side, you get a quic.Session by calling quic.Listen and then Accept. Here you must provide your custom net.PacketConn implementation, along with a TLS certificate and an ALPN string. The Accept call takes a context.Context that allows you to cancel the operation.

tlsConfig := &tls.Config{
	Certificates: []tls.Certificate{*cert},
	NextProtos:   []string{"quichttp"},
}
ln, err := quic.Listen(pconn, tlsConfig, &quic.Config{})
sess, err := ln.Accept(context.TODO())

Once you have a quic.Session, you get streams by calling AcceptStream in a loop. Notice a difference from writing a TCP server: in TCP you call Listen and then Accept, which gives you a net.Conn. That's because there's only one stream per TCP connection. With QUIC, we are multiplexing several streams, so you call Listen, then Accept (to get a quic.Session), then AcceptStream (to get a net.Conn).

for {
	stream, err := sess.AcceptStream(context.TODO())
	go func() {
		defer stream.Close()
		// stream.Read, stream.Write, etc.
	}()
}

Notes on quic-go:

• The library is coupled to specific (recent) versions of the Go language and its crypto/tls library. It uses a fork of crypto/tls called qtls, because crypto/tls does not support the custom TLS encoding used in QUIC. If you compile a program with the wrong version of Go, it will crash at runtime with an error like panic: qtls.ClientSessionState not compatible with tls.ClientSessionState.
  • The need for a forked crypto/tls is a bit concerning, but in some cases you're tunnelling traffic (like Tor) that implements its own security features, so you're still secure even if there's a failure of qtls.
• The quic-go API is marked unstable.
• The on-wire format of QUIC is still unstable. quic-go provides a VersionNumber configuration parameter that may allow locking in a specific wire format.
• The client can open a stream, but there's no way for the server to become aware of it until the client sends some data. So it's not suitable for tunnelling server-sends-first protocols, unless you layer on an additional meta-protocol that ignores the client's first sent byte, or something.
• QUIC automatically terminates idle connections. The default idle timeout of 30 seconds is aggressive, but you can adjust it using an IdleTimeout parameter.
• The ability to interrupt blocking operations using context.Context is a nice feature.

kcp-go and smux

• Home page: kcp-go, smux
• API documentation: kcp-go, smux
• Version used: v5.4.11 (kcp-go), v2.0.11 (smux)

This pair of libraries separates reliability and multiplexing. kcp-go implements a reliable, in-order channel over an unreliable datagram transport. smux multiplexes streams inside a reliable, in-order channel.

Like quic-go, the network abstraction used by kcp-go is net.PacketConn. I've said already that I think this is the right design and it's easy to work with.

The API is functionally almost identical to quic-go's. On the client side, first you call kcp.NewConn2 with your custom net.PacketConn to get a so-called kcp.UDPSession (it actually uses your net.PacketConn, not UDP). kcp.UDPSession is a single-stream, reliable, in-order net.Conn. Then you call smux.Client on the kcp.UDPSession to get a multiplexed smux.Session on which you can call OpenStream, just like in quic-go.

kcpConn, err := kcp.NewConn2(raddr, nil, 0, 0, pconn)
sess, err := smux.Client(kcpConn, smux.DefaultConfig())
stream, err := sess.OpenStream()

On the server side, you call kcp.ServeConn (with your custom net.PacketConn) to get a kcp.Listener, then Accept to get a kcp.UDPSession. Then you turn the kcp.UDPSession into a smux.Session by calling smux.Server. Then you can AcceptStream for each incoming stream.

ln, err := kcp.ServeConn(nil, 0, 0, pconn)
conn, err := ln.Accept()
sess, err := smux.Server(conn, smux.DefaultConfig())
for {
	stream, err := sess.AcceptStream()
	go func() {
		defer stream.Close()
		// stream.Read, stream.Write, etc.
	}()
}

Notes on kcp-go and smux:

• kcp-go has optional crypto and error-correction features. The crypto layer is questionable and I wouldn't trust it as much as quic-go's TLS. For example, it seems to only do confidentiality, not integrity or authentication; it only uses a single shared key; and it supports a variety of ciphers.
• kcp-go is up to v5 and I don't know if that means the wire format has changed in the past. There are two versions of smux, v1 and v2, which are presumably incompatible.
  • I don't think there are formal specifications of the KCP and smux protocols, and the upstream documentation on its own does not appear sufficient to reimplement it.
• There is no need to send data when opening a stream, unlike quic-go and pion/sctp.
• The separation of kcp-go and smux into two layers could be useful for efficiency in some cases. For example, Tor does its own multiplexing and in most cases only makes a single, long-lived connection through the pluggable transport. In that case, you could omit smux and only use kcp-go.

pion/sctp

• Home page
• API documentation
• Version used: v1.6.10

pion/sctp is a partial implementation of SCTP (Stream Control Transmission Protocol). Its raison d'être is to implement DataChannels in the pion/webrtc WebRTC stack (WebRTC DataChannels are SCTP inside DTLS).

Unlike quic-go and kcp-go, the network abstraction used by pion/sctp is net.Conn, not net.PacketConn. To me, this seems like a type mismatch of sorts. SCTP is logically composed of discrete packets, like IP datagrams, which is the interface net.PacketConn offers. The code does seem to preserve packet boundaries when sending; i.e., multiple sends at the SCTP stream layer do not coalesce at the net.Conn layer. The code seems to rely on this property for reading as well, assuming that one read equals one packet. So it seems to be using net.Conn in a specific way to work similarly to net.PacketConn, with the main difference being that the source and dest net.Addrs are fixed for the lifetime of the net.Conn. This is just based on my cursory reading of the code and could be mistaken.

On the client side, usage is not too different from the other two libraries. You provide a custom net.Conn implementation to sctp.Client, which returns an sctp.Association. Then you can call OpenStream to get a sctp.Stream, which doesn't implement net.Conn exactly, but io.ReadWriteCloser. One catch is that the library does not automatically keep track of stream identifiers, so you manually have to assign each new stream a unique identifier.

config := sctp.Config{
	NetConn:       conn,
	LoggerFactory: logging.NewDefaultLoggerFactory(),
}
assoc, err := sctp.Client(config)
var streamID uint16
stream, err := assoc.OpenStream(streamID, 0)
streamID++

Usage on the server side is substantially different. There's no equivalent to the Accept calls of the other libraries. Instead, you call sctp.Server on an already existing net.Conn. What this means is that your application must do the work of tracking client addresses on incoming packets and mapping them to net.Conns (instantiating new ones if needed). The sample implementation has a connMap type that acts as an adapter between the net.PacketConn-like interface provided by the HTTP server, and the net.Conn interface expected by sctp.Association. It shunts incoming packets, which are tagged by client address, into appropriate net.Conns, which are implemented simply as in-memory send and receive queues. connMap also provides an Accept function that provides notification of each new net.Conn it creates.

So it's a bit awkward, but with some manual state tracking you get an sctp.Association made with a net.Conn. After that, usage is similar, with an AcceptStream function to accept new streams.

for {
	stream, err := assoc.AcceptStream()
	go func() {
		defer stream.Close()
		// stream.Read, stream.Write, etc.
	}()
}

Notes on pion/sctp:

• In SCTP, you must declare the maximum number of streams you will use (up to 65,535) during the handshake. This is a restriction of the protocol, not the library. It looks like pion/sctp hardcodes the number to the maximum. I'm not sure what happens if you allow the stream identifiers to wrap.
  • If SCTP's native streams are too limiting, one could layer smux on top of it instead (put multiple smux streams onto a single SCTP stream).
• There's no crypto in the library, nor any provision for it in SCTP.
• Like quic-go, the client cannot open a stream without sending at least 1 byte on it.