The Cost of Ignorance: Why You Need a Chain of Custody

You may have seen our post on how East/West is the New North/South. The bottom line is that traditional API Gateway models simply don’t provide the level of security we need in modern microservice architecture. The problem is that only 20 percent of the traffic – that is the inbound traffic up until the gateway – is secure, and everything inside the data center is “trusted.”

Sure, it may be secured by two-way certs, IP whitelisting, or other network models, but there is no identity. We don’t know WHY “Service A” called to “Service B.” We might not even know that “Service A” was what called to “Service B,” just that something made a call.

But why do we care? We trust “Service A” — don’t we?

In our previous blog post, we looked at the three reasons why we care about cloud security. We looked at things we can do to lock down those services, and what we mean by “fine-grained access control.” And then, in an offhand way, I said “log everything,” which is a gross oversimplification.

You see, simple logs aren’t enough. If you have a breach, you’re going to need to provide a complete record from what kicked off the original request, what other requests it kicked off, and what data was exposed.

Let’s look at our three reasons we secure data, and what we need to know about those transactions.

Infrastructure – Root Cause Analysis Paralysis

Even if you your services are secure, if you don’t know why “Service A” called to “Service B,” then you have a lot of trouble determining your RCA (root cause of the problem). I think most of us have been forced to answer, “Well, it stopped, so I think we’re okay, but keep an eye on it.” If you’re lucky, the problem doesn’t come back, but relying on luck isn’t really a “best practice.”

If you can look at the logs, determine which transactions are suspicious, and then drill your way back up the chain to see that a very special user is able to start a cascading series of requests in just such a way, you can close that door. Without the upstream requests that caused Service A to fire off an odd request, you may never know who was responsible or adequately secure the environment.

Legal – Just How Bad is the Problem?

You may feel that you’re in an industry not covered by the new California Privacy laws, or by Europe’s GDPR, or even the State of New York’s Financial Services laws, but, honestly, if you deal with money, then you probably are. And then there’s HIPAA…

All of these require an immutable chain of custody. Note that word “immutable.” This is where syslogs aren’t enough. Not only do you need to know that User A called App B, which called to Service A, which needed to make a callout to Services B and C, but you also need more than a few lines of logfiles stitching those requests together. You have to be able to show that every request was signed and that the record was not altered.

There aren’t a lot of ways to do that, unless you’re registering each application with its own set of keys and using those keys to send and receive data on each and every transaction. It’s not about trusted and untrusted services anymore — it’s about an untrusted logging system now because, you know, lawyers.

The other thing is that we get charged per exposure. The average cost per exposure varies anywhere from $110 to $336 per record, but if you can’t determine exactly what was exposed, you could be on the hook for your entire database…

Cost per exposure of personal data by industry

Life Science$259Retail$144
Education$240Public Sector$110

So even at $123 per record in the “Research” business, a Cambridge Analytica breach of 87 million records could have cost over $10 billion in fines.

Business – We Demand Answers

The Cambridge Analytica exposure wasn’t a breach, so it wasn’t something that could be fined, but it definitely put Facebook in a tough place, and part of the reason it was tough was because they couldn’t definitively answer what seemed like a simple question: “How many records were exposed?”

This stuff is complicated. It’s hard enough talking to the DevOps team about the problem, but talking to your shareholders, a news outlet, or lawmakers makes it even harder. Saying, “we don’t really know what data was exposed” makes you appear to either be lying or incompetent. Either way, it’s not a good day.

So to sum up, service-to-service security isn’t just about securing the flow, it’s also about creating a full chain of custody so your DevOps team can fix problems, you can limit your legal exposure, and you can simplify business explanations when something goes wrong.