How to test RESTful APIs

RESTful APIs have become a fundamental part of modern web application development in recent years. The RESTful approach is far more simple and scalable than the legacy variants of web API that preceded it – such as SOAP (Simple Object Access Protocol). 

The only implementation of REST is on top of HTTP – the protocol that powers the web. This means that vulnerable REST APIs expose similar risks to traditional web sites and applications, while being more challenging to test with automated web security scanners. 

What is a REST API?

Before we discuss the challenges of effective security testing of REST APIs, we should clarify what we’re talking about. 

An API is a mechanism of transferring information between two computer systems. An API can be implemented either at the code level or at the network level, depending on whether or not the two systems are running on the same machine. 

In a commercial context, an API almost always refers to an interface across the web, which is the most common way of connecting disparate computer systems. 

Modern Web APIs are usually implemented using REST (REpresentational State Transfer). REST is an architectural style in which all of the information necessary to access or change the ‘state’ of a web service can be made in a single API call – such as getting a data record or updating a database. 

RESTful APIs offer a clean separation of concerns between the front-end (presentation layer) and the back-end (data-access layer). The RESTful style has been recognised as the international standard because a single REST API can be consumed simultaneously by mobile devices, web applications and IoT devices without any alterations, making it the cheapest and most flexible way to build modern applications.

Principles of RESTful API Security Testing

There are only four core principles to performing security tests on RESTful APIs. As is often the case however, these principles can be difficult to put into practice. 

The simple principles are as follows, and can be implemented trivially into a web server:

1. Inputs of an incorrect type must be rejected.

a. Corollary: Inputs that are null (empty), when a null is unacceptable, must be rejected.

2. Inputs of an incorrect size must be rejected.

The more difficult principles require an intimate understanding in the range of acceptable values and users, which can be hard to infer without understanding how a REST API will be consumed.

3. For a given input value, the API must provide the expected output.

This can be easy to test when the input domain and the output range are simple (e.g integers or phone numbers). This becomes extremely difficult when building permissive RESTful APIs that enable users to submit their own content (e.g in a chat application). 

4. Input values outside the expected domain must be rejected.

Once again, this is easy when the domain is simple (e.g input values should be integers above zero), but becomes complex when users can supply content (e.g a file upload endpoint could present a significant challenge to secure). 

5. For a given user, the API must provide only the data that they are authorized to access.

If permissions are already defined and are resources stratified in accordance with their permission level, this can be easy to implement. In practice however, authorization is a hard problem – with several multi-billion dollar companies (like Okta) around to solve it.

Most APIs aren’t properly tested to ensure they meet this criteria. Because of this, breaches occur frequently and entire industries exist to offer a protection layer on top of APIs. 

A well designed APIs should present the first-line of defense against attack, and so effective testing should be a top priority.

API Security Tests

There are three main types of testing that compose the security auditing process, designed to secure an API against external threats. 

Security Testing

Security testing validates whether basic security requirements have been met. These include the following questions:

1. What kind of authentication is necessary to consume the API, i.e how do you evaluate the identity of an end user?

2. What sort of encryption is used on the stored data, and at which points are the data decrypted for transmission?

3. Under what conditions are users allowed to access resources? 

This stage of the audit process comes first, and will help prevent the major vulnerabilities.

Penetration Testing

Penetration testing enables you to harden the external surface of your application from vulnerabilities that may have crept in during development. 

In this step, external aspects of the API are attacked in a deliberate fashion in a controlled environment. This can be done using automated tools such as Netspark or Acunetix. 

When organising a Penetration Test, the following steps should be taken:

1. Identify a list of potential vulnerabilities applicable to the application (e.g does it have resources like images which could expose a directory traversal attack?)

2. Order the items in accordance with their risk. You can use the OWASP Top 10 website to get a better understanding of the risk associated with each type of vulnerability.

3. Engineer requests and sessions that incorporate the attacks, and send them at the system – ideally from within the network as well from outside.

4. If unauthorised access to the system is made, file a vulnerability report and go back to patch the issue.

Fuzz Testing

Fuzz testing is the final aspect of a security auditing process, in which an API is pushed to its limits. This can be done by sending vast request volumes at it, attempting to vary the data in as many creative ways as possible to cover the possibilities of vulnerabilities emerging at high volume which could compromise security. 

Such vulnerabilities could be exploited by Denial Of Service or Overflow attacks.  

How to perform a Security Test on an API 

Testing an API means submitting requests using client software to an endpoint of the application that is being evaluated. This is almost always a HTTP client, and there are many free options available. 

The most popular clients are Postman or Insomnia. Insomnia is the best choice for smaller APIs, as it is easy to work with and requires little configuration. Postman is better for more complex APIs, as it stores authentication parameters and enables you to create collections of requests. Postman also has the capacity to automate testing through ‘monitors’, which is useful if the underlying application is constantly changing.

Automating parts of the Security Audit process can speed up the DevOps lifecycle. The two parts that are easiest to automate are the Fuzz Test, and the Security Test that was discussed in the previous section.  

Step 1: Determine Security Requirements. In order to plan a security test on an API, you must first understand the general requirements. This means asking questions like: 

a. Should the API use a TLS/SSL certificate, and be accessed over HTTPS? 

b. What permission groups exist for different resources in the application?

c. What is the authentication flow? Is an external OAUTH provider used? 

d. What is the attack surface of the API? Where could a malicious actor subvert the application? 

As part of asking the above questions, it is important to have a good understanding of what constitutes pass vs failure of your test. 

Step 2: Set up a testing environment. Once the scope of the test has been developed, it is time to prepare an application environment for testing. For smaller applications it’s reasonable to use the standard staging environment. For larger applications with a lot of internal state, it is better to set up a separate environment for the test – either by replicating all resources in the staging environment, or by using a tool such as WireMock to mock them out.

Step 3: Sanity check your API. Send a few requests at the API to ensure that everything has been set up correctly. 

Step 4: Define the input domain. Before developing individual test cases, it is important to understand what each parameter does, and the different combinations that each parameter is allowed to be. This enables you to define edge-cases (values that are barely valid), and determine the parameters which are most vulnerable to injection attacks (like SQL injections).

Step 5: Develop and execute the test cases. Once you have prepared the test environment, and understand possible edge-cases, you can create and execute tests – comparing the actual output with the expected output. As a matter of best practise, you should group these depending on the type of test that is being undertaken. Some examples are as follows:

a. Can resources be accessed using HTTP as well as HTTPS?  

b. Do all endpoints require authentication?

c. If you support file upload, what happens if you upload a potentially malicious file, with the mimetype that is expected by the application?

d. If the web-app that consumes the API embeds user-supplied information (e.g a name) on the page, what happens if you supply a HTML/JS element instead? 

e. Can you access resources that your token isn’t authorized to access?

If you follow these instructions, you should have a good understanding of the security posture of your application, and a toolkit for ensuring that no significant security issues end up in a production deployment.

Jack Hopkins, CEO

jack@sphericaldefense.com