Testing Authorizers in SaaS production environment

About Authorizers

Cloudentity provides the possibility to integrate API Gateways and Service Meshes into its authorization platform in order to secure them with policies. To that end, it uses authorizers - system components responsible for discovering APIs and Services and synchronizing policies with those APIs and Services.

An effective authorization policy that controls resource access is usually distributed between an authorization server and a resource server. In a distributed application system, resource servers are composed of microservices that expose certain APIs for external consumption. Cloudentity provides comprehensive and easy way to govern authorization for APIs and service meshes.

Testing core functionalities is a critical process in all successful companies. When it comes to the Cloudentity, one of such functionalities is Authorizers.

Why Should We Test Authorizers In SaaS Infrastructure

Since Authorizers are one of the core features, we decided to give them as much attention as possible. Many customers rely on our solutions every day and their experience must be the best possible. Our Authorizers are being constantly tested in multiple ways, including their installation, packaging correctness, and others, but none of these methods cover the below needs:

• We need to make sure that Authorizer’s behavior is stable and does not change over time.

• We also need to make sure that any plane, Gateway or Authorizer version bumps do not break anything in the real cluster.

Running some dedicated tests in our SaaS helps us to cover that hole. On the other hand, it takes lots of resources, so we need to select our test scenarios carefully. Various gateways need various setups. Some Authorizers can run as containers, some can run only in Kubernetes cluster, some need external cloud providers, etc.

But the main goal of testing any Authorizer is the same - to make sure that its core responsibilities work:

• The authorizer is registered in Cloudentity (new authorizers only). This can be achieved by creating a new authorizer in Cloudentity and the following manual from the “Quickstart” tab

• APIs are protected with selected policies. This can be achieved by calling the protected endpoint and checking the response.

• Policy synchronization with Cloudentity works and changes are visible after a specific period. This can be achieved by changing the policy for the given endpoint and checking the response again.

If any of those checkpoints fail, we can be sure that some kind of bug has been introduced.

To avoid problems, we should test those responsibilities. Of course, there is no need to register a new authorizer every minute - most of our customers create them only once. It is worth focusing on policies and synchronization instead.

SaaS Authorizer Smoke Tests - How They Work

Let’s start with an example. For demonstration purposes, we will present the Istio Test Suite which is a part of our SaaS smoke testing solution.

• Istio is a Service Mesh that runs as a part of the Kubernetes cluster. Its functionalities help with traffic management, security, observability, and policy management within distributed architectures.

• Istio Authorizer is a dedicated authorizer developed by Cloudentity that helps with policy management within Istio Service Mesh.

• Kubernetes is an open-source solution that helps to manage, automate and scale containerized applications.

• Cloudentity is a powerful authorization server and all-in-one security platform.

To run Istio and Istio Authorizer in the cluster we need to install all of the above components. First of all, we need to set up a Kubernetes cluster. Once it’s up, we can install Istio and configure it to use a custom authorizer for its policy decision-making. Also, we need to choose which services should be protected by Istio. The last step is to deploy Istio Authorizer - the instruction how to do it can be found in Cloudentity platform, as well as our documentation.

Within our Stage and Production clusters, the main goal of testing is to make sure that the cluster is functional 24/7. Our test suite runs in a loop with 60 seconds wait period, so all the components mentioned in this section are already installed and registered in our SaaS, to speed up continuous test execution. It is also the most common production setup.

Once all the components are configured properly, we should be able to see our Istio Authorizer in Cloudentity > Enforcement > Authorizers

API Configuration

For testing purposes, we use httpbin, a tool that provides simple REST API with multiple endpoints. It fits our needs perfectly.

Istio allows marking the whole namespace to be protected. It makes each pod in a given namespace part of Istio Service Mesh and allows us to protect it with policies. It is easy to determine if the pod is protected by Istio - it has got an additional container istio-proxy.

Once httpbin pod gets deployed to a protected namespace istio-tests-current, it gets instantly visible in Cloudentity.

Backward Compatibility

To secure backward compatibility and to make sure that version bumps of any dependent application do not break anything, we configured 3 different Istio Authorizer pods that represent 3 different versions - old, current and new.

• “Old” represents the previous publicly-released Istio Authorizer version.

• “Current” represents the latest publicly-released stable version.

• “New” is the latest build from the version under development.

Our tests are being executed on all three environments at the same time. The results are well described in Allure so it is easy to find the flaky environment in case of any trouble.

Test Scenarios

As a prerequisite, we have pre-configured 3 endpoints in the following way:

• endpoint /get without any policy

  

• endpoint /deny with “Block API” policy

  

• endpoint /ip that changes its policy during the test to make sure that synchronization works.

  

All the above conditions lead to the following test cases:

• Scenario 1 - No policy

  1. Call httpbin /get endpoint
     Expected result: HTTP 200

  Since there is no policy assigned to the /get endpoint, the request is expected to be successful. Nothing prevents requests from reaching their destination.

• Scenario 2 - “Block All” policy

  1. Call httpbin /deny endpoint\

  Expected result: HTTP 403

  Since there is a “Block API” policy assigned to the /deny endpoint, the request is expected to fail with HTTP 403 Forbidden.

• Scenario 3 - Switch policy, synchronization test

  1. Call httpbin /ip endpoint

     Expected result: HTTP 200

  2. As Cloudentity Admin, sign in to Cloudentity and switch /ip policy to “Block API”

  3. Wait for synchronization

  4. Call httpbin /ip endpoint

     Expected result: HTTP 403

  Initially, there is no policy assigned to the /ip endpoint, so the request is expected to be successful. As a next step, the policy “Block API” is assigned to /ip endpoint. After such a change, Istio Authorizer needs a few seconds to refresh its configuration. When it’s finished, another call to /ip endpoint is made. This time the request is expected to fail with HTTP 403.

Monitoring

All the tests end with automated Allure report creation. Allure report contains all the data needed to find the root cause of potential problems. Current and previous reports are stored in our Amazon S3 instance.

Test metrics are automatically sent to Prometheus and results can be watched in Grafana real-time.

In case of any severe issue in our Production, thanks to our support, our engineers can receive alerts and react immediately 24/7.

Conclusion

Cloudentity is reliably monitoring authorizers. We are running a set of quality checks in our SaaS production environment to make sure everything runs smoothly.

Like what you see? Register for free to get access to a Cloudentity tenant and start exploring our platform!

try now