Due diligence, great eh?

Many of you will have seen security questionnaires that are sent by a Buying department, usually in a large corporate and which are not only an annoying distraction, but I think I really harmful to the whole intention of having them in the first place.

So here is the scenario. I am a large customer and want to buy a "solution". If I am going to put my data into it, and even if I am not, I cannot simply take a vendor's word for the solution being secure, I am responsible legally (in many countries) for having due diligence in the purchasing process. I come up with a wonderful idea, I will create a large questionnaire covering all sorts of things related to security including availability, confidentially and integrity, all of the things I learned in my CISSP qualification. What could go wrong?

Pretty much everything.

First of all, these are extremely burdensome on the vendor:

  1. Even if you know the answers, it takes a long time to fill them in, 1 to 2 hours minimum and sometimes longer
  2. The questions are never standardized so I can't simply document the previous answers to copy and paste
  3. The questions are often not written to identify the intention of the question. If a question is, "Do you use AES256 encryption", is the intention that it must absolutely be AES256? If it is and we don't use it, we might as well tell you thanks but no thanks. Is it actually, "Is your data encrypted at rest using AES256 or an equivalent secure algorithm"?
  4. The people who manage the survey for the customer are rarely technically savvy so requests for clarification are either slow or sometimes not even forthcoming.
  5. The technical team are inevitably disturbed for answers, even though these questionnaires are a sales team function.

Secondly, the questions assume a simplicity that does not exist even in the simplest organisations. 

  1. It only takes a Buyer seconds to add a question like "Describe how you manage encryption keys". How long would it take you to answer that with any useful level of detail? An hour?
  2. If I answer that all of our data is encrypted using XYZ or all of our routers are type ABC, what happens if I want to change this in the future? I don't know whether people are looking for general sense of quality/security or whether they have whitelists of manufacturers and suppliers.
  3. The questionnaires almost always demonstrate a complete lack of understanding how systems exist within the eco-system of an integrated network. In the old days, if you were buying a "system" it was likely standalone and could be described independently of any other system and it would be up to the customer to install it securely on-premises. In these days of integrated office networks and cloud systems, the complexity is orders of magnitudes higher. "How are the networks integrated?" I don't know, ask Azure.
  4. The questionnaires are rarely tied to any objective list, if they were, it would be much easier to answer them. "Does your system conform to the encryption requirements of section 1.2.3 of the PCI requirements?" can easily be answerd yes/no/don't know.

Thirdly, they don't actually work

How many of these questionnaires do you think contain the truth, the whole truth  and nothing but the truth? None? I don't know chapter and verse about single thing I use in my system, I base a lot of it on trust and reputation. I am not qualified (and probably not allowed) to ask Azure for a chapter-and-verse description of their network protocols so I can assure my customers that our Office 365 is secure or that a weird zero-day exploit couldn't be set against an Azure instance to break something.

Also, there is a pressure to get a sale and the questionnaire is a cold hard barrier to the sale. Even if the sales team do not lie, they are not likely to put any more detail than necessary that could upset the sale. What about that random piece of software I use on my desktop that isn't strictly PCI compliant or isn't on an approved list. Am I going to mention that? Nope. What about the other 200 applications I use as a developer? Are they all 100% legit? It's impossible to know. In other words, the most dubious attack vectors are the ones that are not likely to make it onto the list of software that is used by the vendor.

The truth is that they are used as an objective measure in a world that is much more subjective. I shouldn't be penalised because my network routers aren't certified to some milspec setting (unless I am selling to the military!) just because some professional certified security manager thinks it would be a good idea. There are not many objective measures in industry for security and it is unfair to burden vendors with very arbitrary requirements and even the work required to specify everything, even if they do comply.

So what should we do?

It's very simple. We should use, and create/amend if necessary, specific industry standards, whether ISO27001, some NIST spec, Cyber Essentials and/or a combination of them. The questionnaire can then be one simple question: "Are you certified to ISO27001 and if not, please explain why?". 

If I am already doing something that is pretty much mandatory to work in the SaaS world like ISO27001, it should be enough to demonstrate that we have been accredited. In most cases, the people who writes these specs are likely to be either more intelligent or more pragmatic than some random security team at an Enterprise who might be simply justifying the 100s of 1000s they cost to employ by generating bureaucracy where it is not useful.