Mitigating Server-Side Request Forgery

Server-Side Request Forgery (SSRF) vulnerabilities allow an attacker to cause a server application to perform an unintended request. When exploited, the server could leak sensitive internal information or perform dangerous actions. Because this vulnerability depends on the capabilities of the server application, the potential impact of an attack can vary.

Webhooks are among the most common features that introduce SSRF vulnerabilities to applications. They combine arbitrary user input (the webhook URL) with the ability to make requests from the backend. It’s important to consider this threat when building and operating webhook systems.

The attack

For the purposes of this post, imagine we have a web application that is able to perform outbound requests to a user-configured endpoint.

Cloud credential leak

Every AWS compute resource has a special internal service, called the Instance Metadata Service (IMDS). This endpoint provides network information, initialization scripts, and even temporary AWS access keys. All of this is conveniently hosted at http://169.254.169.254 . The feature is great for avoiding hard-coded keys but comes at a cost—by default, it is available to every process on the server.

The scenario for exploiting this is simple: the user provides a URL of http://169.254.169.254/latest/meta-data/iam/security-credentials/role-name to the application, then look at the response. It should look something like this:

{
  "Code": "Success",
  "LastUpdated": "2012-04-26T16:39:16Z",
  "Type": "AWS-HMAC",
  "AccessKeyId": "ASIAIOSFODNN7EXAMPLE",
  "SecretAccessKey": "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
  "Token": "token",
  "Expiration": "2017-05-17T15:09:54Z"
}

With the key in hand, the attacker can perform any action that the compute instance is permitted to do. Because AWS IAM can be complicated, some organizations provide more permission than is the server needs—using * wildcards. Occasionally, the same IAM role will be used across staging and production environments.

Even if the server isn’t hosted in AWS, there are similar endpoints in all major cloud providers. This includes Google Cloud, Azure, and DigitalOcean.

Internal endpoints

Mature organizations might rely on private networking to protect internal resources, asserting that a VPN connection is required for access. Unfortunately, web servers tend to sit on the same network.

Imagine there’s an intranet site containing trade secrets sitting on the internal network, available for all employees to access. By convincing an application server to perform requests against the site, an attacker can steal all of those secrets.

Filesystem access

Web application servers tend to contain secrets of their own: source code, configuration files, and user uploads. An attacker providing a request URL with the file:// scheme may be able to access any file visible to the web server process. This might already be locked down depending on the library being used to perform requests from the server.

Mitigating the vulnerability

Deliver, the webhook monitoring system I’ve been working on, is written in Elixir. Naturally, the examples will also use Elixir. The same mitigations will likely be available in any technology, but specific package recommendations may not apply.

Preventing non-HTTP(S) URIs

The following example shows how the URI.parse/1 method breaks a URI into a struct.

iex> URI.parse("https://elixir-lang.org/")

%URI{
	authority: "elixir-lang.org",
	fragment: nil,
	host: "elixir-lang.org",
	path: "/",
	port: 443,
	query: nil,
	scheme: "https",
	userinfo: nil
}

A first pass at SSRF protection might look like this:

def permit_uri?(uri) do
	URI.parse(uri).scheme in ["http", "https"]
end

By locking out unacceptable schemes, we ensure that the filesystem can’t directly be accessed.

Preventing the instance metadata IP address

Another layer of protection that could be valuable is excluding 169.254.169.254, as follows:

def permit_uri?(uri) do
  parsed_uri = URI.parse(uri)

  parsed_uri.scheme in ["http", "https"] and
    parsed_uri.host != "169.254.169.254"
end

One interesting problem with this solution is that public DNS endpoints can return any IP address they’d like in response to an A record query. For example, this would be completely legal in DNS:

> nslookup malicious.example.com
Server:		1.1.1.1
Address:	1.1.1.1#53

Non-authoritative answer:
Name:	malicious.example.com
Address: 169.254.169.254

If an attacker arranged for a request to http://malicious.example.com/latest/meta-data/iam/security-credentials/role-name, it would be treated as though the IMDS IP address were provided to begin with.

Looking up DNS hostnames

Erlang’s built-in :inet_dns module can be used directly in order to perform DNS lookups. On the Erlang side, IPv4 addresses are expressed as tuples with four elements, and strings used for networking have to be of type charlist instead of binary. Clearly, this implementation is getting complicated.

defmodule SSRFProtection do
  @disallowed_hosts [
    {169, 254, 169, 254}
  ]

  defp resolve_host(host) do
    host = to_charlist(host)
    case :inet.parse_address(host) do
      {:ok, addr} ->
        [addr]
      _ ->
        :inet_res.lookup(host, :in, :a)
    end
  end

  def permit_uri?(uri) do
    parsed_uri = URI.parse(uri)

    parsed_uri.scheme in ["http", "https"] and
      not Enum.any?(
        resolve_host(parsed_uri.host),
        &(&1 in @disallowed_hosts)
      )
  end
end

By running the user-provided URIs through the permit_uri?/1 method of SSRFProtection, we can prevent this class of requests.

iex> SSRFProtection.permit_uri?("https://google.com/something")
true

iex> SSRFProtection.permit_uri?("https://169.254.169.254/something")
false

iex> SSRFProtection.permit_uri?("https://malicious.example.com/something")
false

iex> SSRFProtection.permit_uri?("file://C:\\inetpub\\wwwroot\\secret.html")
false

Perform validation on every request

While there is some level of protection provided by validating a user-provided URI when it’s first submitted, it’s not the best we can do. DNS is not static. An attacker could submit a URI that previously resolved to 123.123.123.123, but update the record to 169.254.169.254 after the application validates it.

This is why it’s important to run permit_uri?/1 (or its equivalent) for each request the server makes.

Using SafeURL

As indicated above, this implementation is messy and not great. It describes the concepts, but it’s probably not the most production-ready solution. Luckily, there’s an open-source Elixir package called SafeURL that encapsulates SSRF-protecting validation into a single module.

Wherever SSRFProtection.permit_uri?/1 was used, we can instead use SafeURL.allowed?/1. This library has a built-in list of IP addresses, ranges, and schemes that are a great default. Depending on your use case, it can even perform GET requests for your application through an optional HTTPoison integration.

Similar libraries

There are similar libraries for SSRF protection in most programming languages. A couple that appear to be reputable are:

Taking it further

Don’t follow redirects

Some request libraries automatically follow HTTP 301 and 302 redirects. Ensure this behavior is disabled when performing requests against user-provided URLs. A malicious actor could host https://happy-looking-site.com, where it redirects to 169.254.169.254. If the web application follows the redirect, the attack will succeed.

Security groups

Make sure to configure security groups or the cloud firewall for the hosting provider of the web application in a way that limits access to what’s strictly necessary. If it doesn’t make sense for the application to reach a specific internal service, don’t allow it.

Proxy

An additional layer of security that some companies choose to implement is an HTTP egress proxy. These typically are instances of Squid, Envoy, or a cloud provider hosted solution. When outbound requests are made by the web application, they go through the proxy. This proxy should be configured with no permission to reach back into the private network and ideally will prevent requests to bad IPs.

Wrapping up

Security is tricky to get right and can have disastrous side-effects when done incorrectly. Attackers will poke holes in every line of defense, which is why it’s important to consider your threat model and establish a defense-in-depth strategy.

While writing a solution in-house can seem reasonable, there are tricky edge cases that might be solved in a well-accepted open-source package. Look to the community for the best defense.

Go and protect your application from SSRF vulnerabilities!

Adding soft delete to a Phoenix Commanded (CQRS) API

Part two in my series on Elixir’s Commanded library. Part one can be found here.

Context

In the previous post, I converted a vanilla Phoenix API to CQRS with Commanded.

This application writes to the database using the commanded_ecto_projections hex package, which subscribes to events and projects their state into tables managed by the Ecto database library.

Since the core data model of this application is an append-only (immutable) log, the events can be replayed and the read model can be dramatically changed using existing data.

Goal

Implement a soft delete in the API, allowing items to be restored after deletion.

Follow along with what I learned while iterating on a project named todo_backend_commanded. Its git history shows the steps involved to implement soft delete and restore functionality.

Read more

Using CQRS in a simple Phoenix API with Commanded

Despite being a fan of event sourcing and seeing the clear benefits of the approach, I never built anything from scratch. This weekend, I finally decided to break this study cycle and do something practical.

Follow along with what I learned while implementing a project named todo_backend_commanded. Its git history reflects the process of migrating from a vanilla Phoenix API to an event sourced solution.

Read more

Firestore Backup and Restore

TL;DR

Check out FirestoreRestore. It uses a brand new API to backup and restore your Firestore database!

Background

There have been many requests for Google to add backup and restore functionality to their Cloud Firestore offering. Until this week, the only method for backing up a Firestore database has been to run utilities that traverse the tree as an admin client, saving each document’s JSON representation on disk.

Discovery

This morning, I opened the Google APIs Explorer and found two new methods listed under Firestore v1beta1: exportDocuments and importDocuments.

APIs Explorer Screenshot

Read more