Blind In/On-Path Attack Disclosure FAQ

In November of last year we reported a vulnerability that allowed to a network adjacent attacker to make inferences about active connections inside VPN-tunneled connections and inject data to reset or hijack these connections. This vulnerability was assigned CVE-2019-14899 and affected Apple, Android, and many Linux and BSD systems. You can read about this in detail in our post from May. This post describes a new attack which takes advantage of the insight gained from the previous attack, but expands the threat to include any in-path router, such as your internet service provider. We have included a section with the most commonly asked questions and will update this post as we receive more correspondence.

In the previous attack, a network adjacent attacker can see the traffic as it passes from the VPN client to the VPN server (and vice versa), and although it is encrypted, they can learn information about the TCP headers by making some good educated guesses. This attack first required the attacker to learn the private, virtual IP that has been assigned to the client by spoofing packets to the client across the private address space. When the attacker guesses correctly, they will see a response from the client (an encrypted RST packet), otherwise, the client will not respond.

Once the attacker knows the virtual IP address, they then can test a connection to an arbitrary website by spoofing packets to the client with the destination address of the virtual interface and the source address of the website. The process is similar to the first step and the attacker will see a response from the client if there is a connection, and no response if the attacker has guessed incorrectly. After the attacker has established that there is an active connection, they can continue to probe the client and learn enough to find an in-window sequence and acknowledgment number. They can then use this to inject arbitrary data into the connection.

This attack works because without some kind of source address validation, such as reverse path filtering, the kernel will process packets received on an incorrect interface as legitimate. This attack can be mitigated by enabling reverse path filtering or adding an iptables/nftables rule to drop packets that are received on an incorrect interface. Some vendors and developers, such as WireGuard, Private Internet Acess, ProtonVPN, and Mullvad have patched their software to protect clients from this attack.

For this new kind of attack, however, we show how one can attack “the other end of the tunnel” from the perspective of an in-path router and assume that reverse path filtering has been enabled. In this scenario the attacker is still able to perform the same attack, as well as inject malicious DNS responses. We disclosed this vulnerability to a number of mailing lists, including OSS private, Linux Kernel Security, OpenVPN, WireGuard, Android, Apple, and Microsoft. The most important aspect of this attack is that it expands the vulnerable client systems to include any operating system and VPN software, including Windows and policy-based VPNs.

The previous attack considered the threat of using a VPN on public WiFi to add an additional layer of security, but this attack includes any machine between the VPN client and the VPN server with the ability to spoof packets. We have created a virtual lab to test this vulnerability, as shown in the figure below. In this scenario, the attacker is spoofing packets from Router 1. The packets are being sent to the VPN server with the source address of the website at the other end of the connection. In a real world scenario, this threat would be the ISP being used by the victim or any machine the packets traverse on their way to the VPN server.

FAQ

In this section we will provide common questions received from developers and kernel maintainers in hopes to clear up any misunderstandings. If you have any questions, feel free to email us at vpn-research@breakpointingbad.com.

Q. Doesn’t enabling reverse path filtering or martian filtering prevent this attack?

Asked by Anthony Liguori and Colm MacCárthaigh

The reverse path filtering settings of the VPN client or server can be anything and the attack will still work. We have reverse path filtering enabled on both client and server in our experiments.

Martian filtering probably is not relevant because the attacker never sends to or from non-routable Internet addresses (except when we use local IPs as Internet addresses in our virtual network). The spoofed packets have the source IP of the web server and the destination IP of the VPN server.

The issue here is that even with reverse path filtering enabled on the VPN client and the gateway, we can perform this attack from any router that is in-path between the client and the VPN server. The mitigation that was previously offered was for the client to enable reverse path filtering, or some other form of source address validation, but this illustrates that this problem requires more than a client-side mitigation.

The routers on the internet do not have source address validation, so this attack also makes policy-based VPNs (with the configuration suggested by Noel Kuntze in the conversation from last December) and operating systems that enforce the strong host model, including Windows, vulnerable. This isn’t the case with the vulnerability as reported in November.

The reverse path filtering we’re talking about is the RFC 3704 kind, but specifically we’re talking about the hosts (VPN server and VPN client). So it has to do with which interfaces they allow packets to come in on. The packets we spoof for the attack enter the VPN server on the same Interface as legitimate packets, and the source IP address is identical to legitimate traffic. So if reverse path filtering is set to strict mode on the VPN server, the incoming spoofed packet will be checked to make sure that any replies to it would be routed out the same interface it’s coming in. Since it’s the Internet-facing interface in all cases (spoofed or legitimate, coming or going) reverse path filtering ala RFC 3704 never comes into play. Then the VPN server NATs the packets into the VPN tunnel and the VPN client receives them through the tunnel on the same virtual interface where legitimate packets come from. So strict reverse path filtering on the client doesn’t make a difference.

Strict reverse path filtering can be carried out by edge networks but if you get deep enough into the Internet then asymmetrical routing makes it impossible (as far as we know, none of us have had the opportunity to play with non-edge networks on the Internet before). Referring to our setup, there’s an opportunity for router 2 to block the spoofed packets using the reasoning that router 1 would never route packets from the website, but that would be some kind of special case (like the website and VPN server are both customers of the same ISP, or router 1 and router 2 are owned by different ISPs that collaborate to validate source IP addresses in some way).

Doesn’t BCP38 prevent this attack? Is this an issue of a misconfigured network?

Asked by Colm MacCárthaigh

BCP38 also seems like it isn’t a major factor. For our own research (which is largely focused on journalists and activists in Asia) we’re specifically interested in attacks that the network infrastructure could carry out on users, the kinds of attacks where it is sometimes assumed that VPNs will protect against them. So if a router is carrying out the attack an upstream router would need to apply the BCP38 filtering, which we would expect to see only for smaller ISPs that only get their Internet through one gateway and don’t route any traffic for others. And any router that was limited by such filtering could easily spoof the packets from another part of the Internet where spoofing is possible. Put another way, we expect that most nations and most major ISPs would have the ability to spoof packets.

Just to be clear, in our new attacks the VPN client never sees spoofed packets anywhere else other than coming through the VPN tunnel on the same virtual interface as legitimate packets. And, as for the VPN server, spoofed packets come in the same interface and have IP and TCP headers identical to legitimate packets (except for whatever field we’re guessing, such as the ephemeral source port).

We mostly have experience with edge networks and would be curious to find out more about how BCP38 is deployed in practice on the parts of the Internet where asymmetrical routing comes into play. We recently moved from the University of New Mexico to Arizona State University. While at UNM we had a VLAN that took us all the way to the Albuqueruque Gigapop, and we never had any trouble spoofing any return IP address we wanted to from there. As far as we know there is source address validation on a national level in some places (e.g., China does it on a national level for IPv6), but mostly networks reason about reverse routes only in local ways. Here are a couple of papers where we used source IP spoofing:

Detecting Intentional Packet Drops on theInternet via TCP/IP Side Channels

Original SYN: Finding Machines Hidden Behind Firewalls

The second one also has an interesting result that only 23.9% of the networks we measured stopped us from spoofing packets with source IPs on the same /24 as the destination. In general, we’ve never had major problems with any kind of filtering which prevented us for being able to spoof source IP addresses. UMich has the same basic network setup we had at UNM, and we plan to have the same thing at ASU very soon. CAIDA’s results show that about a quarter of the world’s ISPs don’t even do the simple kind of BCP38 filtering…

Network Hygiene, Incentives, and Regulation - Caida

Maybe to make it concrete, could you give us an example of how the spoofed packets would be filtered if, e.g., the VPN client was in Ecuador using an Ecuadorian ISP, the VPN server were in the U.S. using a U.S. ISP, and the website were in Brazil using a Brazilian ISP, and an Ecuadorian ISP was the attacker?

How is this related to VPNs? How is it different than TCP attacks on the unencrypted internet?

Asked by Jason Donenfeld and Colm MacCárthaigh

*This example was provided by Jason*

Client accesses web pages by sending TCP packets:

0. vpn client makes a packet: [inner]
1. vpn client encrypts and sends to AP: [outer {encrypted inner}]
2. AP does nat: [outer' {encrypted inner}]
3. AP to router 1: [outer' {encrypted inner}]
4. router 1 to router 2: [outer' {encrypted inner}]
5. router 2 to vpn server: [outer' {encrypted inner}]
6. vpn server decrypts and does nat: [inner']
7. vpn server to router 2: [inner']
8. router 2 to router 3: [inner']
9. router 3 to website: [inner']
10. website replies: [inner2]
11. website to router 3: [inner2]
12. router 3 to router 2: [inner2]
13. router 2 to vpn server: [inner2]
14. vpn server does nat and encrypts: [outer2 {encrypted inner2'}]
15. vpn server to router 2: [outer2 {encrypted inner2'}]
16. router 2 to router 1: [outer2 {encrypted inner2'}]
17. router 1 to AP: [outer2 {encrypted inner2'}]
18. AP does nat: [outer2' {encrypted inner2'}]
19. AP to vpn client: [outer2' {encrypted inner2'}]
20. vpn client does NAT and decrypts: [inner2']

We perform attacks from router 1, which never sees any unencrypted traffic between the VPN server and the website/DNS server. It might be instructive to separate the vulnerability into two separate parts:

1. It’s possible to infer the 4-tuple of a connection being NATed by the VPN server, because only packets with the correct 4-tuple get NATed and tunneled. So if we assume we know what server and port the client is connecting to (e.g., a DNS server on port 53), and we know the TCP connection will appear to come from the VPN server, then we know three of the four parts of the 4-tuple (and we assume we know the protocol, UDP in the case of DNS if people prefer to think in terms of 5-tuples). The part of the tuple we don’t know is the source port used by the VPN server for the connection (which may or may not match the source port the client chose, doesn’t matter for the attack). But if we try every possible source port only the correct one will get NATed by the VPN server and sent through the tunnel. For our attack that spoofs a DNS reply, to find the right source port we put size markers and fill the packets from /dev/urandom so the VPN server can’t compress them. Then we see what size come through the VPN tunnel (we can see the size of encrypted packets at router 1) and keep sorting into buckets and guessing until we’re sure we have the correct source port. Again, all packets we spoof without the correct source port are dropped by the VPN server and never NATed. We’re basically taking advantage of the fact that there’s a conntrack entry, and using that like a very simple side channel.

2. It might be more clear to think of this as a separate vulnerability… Once we know the source port the VPN server is NATing, then any packets we want to spoof to the client can be sent directly to the server. The server will NAT them, encrypt them, and send them to the client through the VPN tunnel. From the client’s perspective the spoofed packets will be identical to legitimate packets for the tunneled connection. For spoofing replies to a DNS query from the client, we assume that we start when the DNS query is sent by the client (we can fingerprint it by size as it traverses the VPN tunnel from client to VPN server), and we also assume that no legitimate reply is coming from the DNS server because we have triggered per-IP rate limiting to the VPN server’s IP, by basically DoSing the DNS server with spoofed packets from that IP. So if we can get the right source port in, e.g., 2 seconds, and the DNS resolver on the client waits for 5 or 10 seconds (0 or 1 DNS retransmissions with 5 second lookup timeout), then we have 3-8 seconds to spoof replies with guessed TXIDs. We spoof them to the VPN server from the IP of the DNS server, and the VPN server NATs them. In practice we can get the TXID right and inject a malicious DNS reply ~10% of the time, but we’re working on improving that. For TCP, we’ve confirmed that we can get the client to respond to in-window spoofed packets, but haven’t built that out into a full TCP hijacking/spoofing attack yet.

In some ways, this can be seen as a vulnerability in conntrack and the related NAT functionality in netfilter, but really it’s only performing NAT the way it’s supposed to and the attack would probably work with the VPN server running, e.g., a BSD with something other than netfilter. Conntrack could start reasoning about things in TCP headers or DNS records—like sequence numbers or TXIDs—to make NAT decisions instead of only for its own state machine, but that could lead to other side channels. (Note: we haven’t completely studied the netfilter source code, but we assume it just NATs packets when the 5-tuple is correct because that’s what we’ve observed.)

So, your statement about where we inject packets and your model are correct, but our attack separates guessing the source port from guessing the TXID (or sequence number or whatever), so that we’re searching two 2^16 spaces instead of a 2^32 space (in the case of DNS and TXIDs, as an example). One way that VPNs are involved is that we use the existence of a tunneled packet to infer something about the conntrack state on the VPN server. That’s how we’re able to separate the source port from other fields we have to figure out for correct spoofing. Another way that VPNs are involved is that the most common mitigation for CVE-2019-14899 was to add a firewall rule or something similar to replace what reverse path filtering would have done, which is to stop spoofed packets from coming from non-VPN interfaces. But in this case both the VPN server and VPN client see the packets in the same format and on the same interface as legitimate packets.

Funding Information