Testing, in the black box (ATV), Security & Privacy

How Automate Testing Vehicles (ATV) should include Pentesting.

Why should privacy officers get involved in development, regression testing process?

Why does IT need to improve their testing strategies?

Pitfalls in Testing, Security/Privacy concerns is what drives people to have nightmares. Privacy officers need to have a better understanding of the environment they work in. The IT people need to embrace the notion that Privacy/Security starts from the beginning. So in that way the chances of being on a front page of a newspaper because of a breach and/or a failure will be minimized. NO ONE wants to phone the CIO about a problem like this. It is a team effort.

I do have to warn you, the reader, that some of the material may be a  little IT oriented. But in an organization where one needs to satisfy a number of different objectives, I would suggest at least a basic knowledge of the IT process is needed. And that the IT personnel need to understand the present compliance/regulator landscape.

Some definitions are warranted before I begin.

ATV or Automated Testing Vehicle. What is it? Why do I care? And is it a 'best practice'? (one of the most over used phrase at present).

The idea is fairly simple. Having a set of scripts (automated) that can be run to test the system in question. The objective is to test the system before any changes are implemented. The process should set up  the files that will be used for testing(see one of my previous blog posts concerning using data for testing),  then run the test scripts, and afterwards run the comparison reports and highlight items of concern from the test just executed. All this is done in an automated fashion. Rather simple concept, but one that can be 'processes' changing in a good way.

Well there is more to this. But let me define another term or two first.

IT systems that are down cost money in lost revenue, and good will to the enterprise.  As an example, in 2012 Google had an outage.
Google June 2012 down for 10 min.

The ball park figure cost that Google suffered was calculated at about $750,000. And that was for 10 minutes.  Now I am not suggesting all downtime costs are that much. It depends on the circumstances, but I am sure no one would like to find out for their own companies.

Another good example of the costs is sited at costs of web down time per industry

This site allows you to calculate the cost of a web site being down per industry/application. Its an eye opener to say the least.

In another 'word', downtime is BAD/EXPENSIVE *Yea  I know that is two words*. But joking aside we need to reduce unavailability as much as possible.

PenTesting. Wikipedia link  The Information Systems Audit and Control Association(ISACA) defines Penetration Testing as  "A test of the effectiveness of security defences through mimicking the actions of real-life attackers."

(For the reader who is more concerned with Privacy/Security, please read on)

So now let's proceed. When an application change happens IT personnel (or a designated organization) tests the changes (IE regression testing). They test the change to see if it works. Now depending on the process that is followed, a user may also test/approve the same series of changes to the application for user approval. Fine, right? Do you notice something missing in the above? In fact, there is more then one item here that needs to be defined/explored.

For many organizations testing to maintain the basic functions within an application does happen in a haphazardly way.  Sure the change is tested and to get to the enhancements, some basic functions are tested as well, But, based on my anecdotal experiences, on many occasions, the entire core functions of the changed application are not testing on a consistent bases.  A test of the all the basic core functions should also be completely tested whenever there is a change.

As an example, if the application in question is some public facing web application (a web store as an example), basic function testing should also be done. Test for example, the ability to add/change a Credit card information and make sure that the update still works. Test adding an item to the shopping cart etc.

So if the new function within the application fails, you have verified that the basic core functions, the one you need to keep the doors open, will still operate.

Imagine if an error occurs at your bank, yet the basic functions were tested successfully with the 'improved mobile bank portal' (the change that will be implemented).  Then logic would dictate that the basic functions should still work (you can still pay bills) even if the enhancement of the bank's mobile app does not. Corrections can be retested and implemented with minimal cost/embarrassment to the organization.

I am therefore advocating that there should be standard testing scripts that confirm, even with the changes that are going to be implemented,  that ALL the core functions still are accessible.

So to implement a process like this, you first need to map out the basic functions that you can not live without. Once that is done and scripts are created, an automated process should be created. When ready, a series of script can be executed with little human intervention. (less change for human error). The 'Best Practice' (there is that phase again) would be something along the lines of submitting the scripts and going home. When you get into the office the following day the results are ready for analysis/correction etc.

This should ensure that at even if the new change fails. You, the customer, can still do business with the organization in question. This is what some people call a ATV (see above). This process can be called your insurance policy.

However, lets' takes this further. Why just test  the basic functionality of the application? Should we also test for Security/Privacy issues?  Should the company's Privacy/Security office ensure that this type of testing, verification is also included within an ATV and executed whenever anything changes?

Absolutely!

A process that includes PenTesting (see above) is something one should consider adding to the above mentioned ATV. With any change there is always a chance that a vulnerability is created that may not have been there before.

Any failure can by it's very nature, cause the potential to expose sensitive information. It can be business secrets, and/or Personnel Identifiable Information (PII) to name but two potential headaches.

There is software in the marketplace that has the capability to engage/test/analyze applications for vulnerabilities. Some of the software I have previously mentioned as well as others which are available with the capabilities needed.

So I suggest that one creates an ATV process that includes the basic functionality of the application/system in question as well as additional testing for security/privacy. All  this should be automated so that more extensive testing can be executed as well as reducing the chance for human error.

Privacy officers need to ensure that any changes that are implemented will not cause exposure that may be costly. IT people need to make sure that the basic systems functions still run, no matter what is changed.

Finally, while no one can claim in absolute terms that there will be no issues, following these basic concepts can help reduce the chance that the CIO needs to be called because of an issue.

Privacy and Security sometimes don't talk the same language.

 

In this blog, which may seem as a little rambling of sorts, I will try to first explain where I see the disjointedness between Security (where the IT people see it) and Privacy (where the Audit Privacy officers see it). I will then try to guide the reader to various resources on the web to offer help.

This will allow the reader, no matter which side of the fence they sit on, to at least understand the other side and understand what they are talking about. Some of these resources quoted below will be targeted for the IT techies, and the others for the Privacy gurus. By putting  them in one central location I hope to bring together, in some small part, the two groups so they can better understand each other.

So let's begin.


Privacy. According to the Webster online Dictionary: 'freedom from unauthorized intrusion <one's right to privacy> '

Security. According to the Webster online Dictionary: 'measures taken to guard against espionage or sabotage, crime, attack, or escape '


An explanation. Security is the process that is put in place to protect the Privacy of the information, whether it is Personnel Identifiable Information (PII), company's intellectual property etc.

We have a problem. The 'WE' in the previous sentence belongs to the IT personnel as well as  the Privacy Officers of an organization. Many times the Computer guys speaks in 'techy talk' (subroutines, C#, apache configuration etc) and the Compliance personnel talk in legalese (jurisprudence, PIPEDA, Opt in, Office of the Privacy Commissioner etc). So, it is no wonder that many organizations seem to have a disconnect between the two.

To address the need for privacy and security in our day to day computer lives, some measures were/are developed by people who may look at a issue  but see it in two different ways.

As an analogy, let's take a look at a square drawn on a piece of paper. The IT people see it as four lines connected at the corners, and the Privacy people see it as four corners connect by some lines. (I hope you get my meaning in this analogy I just presented). Both are right, but both don't see the entire picture either. And thus, this illustrates the issues that many organizations face.

Yes this disconnect is evolving. There are now certification/training sessions for people who are responsible for privacy policies and are not  technical (for more info go to https://www.privacyassociation.org/) but try to bridge the gap (CIPP/IT) And there is various integrated development environment (IDE)  to try to ensure that the code written can be tested for security (IE PENetration testing etc.) But as much as these two groups are trying to work and understand each other there can be some areas where they are miles apart.

If you have followed this series from the beginning, you would have remembered at least one very common example where there is a security/privacy hole big enough to drive a tractor trailer through. (see my previous blogs for more information) And I would bet my two weeks of pay(jar of peanuts) that most Compliance/office of privacy departments have yet to investigate the arena. This is a clear example where the lack of understanding of one department operations by another can lead to some very ominous problems.

Privacy Officials, for the most part do not understand the nuisances of coding, testing , developing applications etc. for the current market place. They do know the laws of the land, and do create compliance rules that all have to abide by.

IT professionals, again for the most part, do not understand the rules that governs Privacy. What is a Opt in or Opt out option? Why must Credit card numbers  be treated under some externally developed standards? What are those standards? (Well, maybe they do, but this is used only as a simplistic example). IT professionals know how to create a automated process to sell, bill, retrieve the widgets that the company makes, Yet the problem is that IT people (the techies) more often then not are not involved, nor understand the Privacy Realm.

Education on both sides is the only real answer. So in the following I will try to give some resources to the reader with some comments that may help understand the other side.

Please note I do not have any financial relationships with the organizations listed below. Nor do I recommend or agree with the statements contained within, though I have found these sites to contain valuable information. Whether you are a techie or privacy person I strongly suggest you take a look at all these resources to better understand the world we have to work in, so to speak.

The first resource that you may or may not be aware is the Privacy Rights Clearing House. (https://www.privacyrights.org/). A very useful web site, where among other items, is a list of all publicly disclosed data breaches since 2005. In fact according to the web site, as of when I started writing this blog, 607,472,154 DATA RECORDS WERE BREACHED. The number of breaches were 3,678 DATA BREACHES made public since 2005.

The type of breaches  that you will find there include 'dumpster diving', laptop/hard drive being misplaced, and SQL injection to name but three. Chances are that you or someone you know was a victim of at least one, if not more, of a data breach. In fact if you do the math, the number of records is about twice the entire population of the US. And this site is very light on breaches outside of the US.

The next resource  is Ponemon Institute (http://www.ponemon.org/). This site has  a wealth of research on the  who and how of privacy. Its stated purpose is 'to enable organizations in both the private and public sectors to have a clearer understanding of the trends in practices, perceptions and potential threats that will affect the collection, management and safeguarding of personal and confidential information about individuals and organizations.'


The next site I would like to point the readers to is the Verizon Security Blog. (http://securityblog.verizonbusiness.com). Basically a week by week analysis of what security issues are out there. Also, they have teamed up with various
International Law Enforcement organizations to issue a yearly report investigating the causes of data breaches. And the information would be a surprise to some. For example

97%  OF ATTACKS ARE AVOIDABLE BY SIMPLE OR INTERMEDIATE
CONTROL.

The latest report on this web site is a review of 2012, but the updated 2013 report is expected out very shortly.

Next is an organization called International Association of Privacy Professionals (https://www.privacyassociation.org).

I happen to have two certifications from them. They have two items I would recommend the reader to investigate. One is a blog they call Privacy Perspective, an interesting blog where various people talk about issues of the day. The other item  is their 'DASHBOARD' (They have one for US, Canada, Europe and ANZ). They gleam information from various sources and present them in a concise 'executive' brief type.


The above resources are just a tip of the iceberg. The problem with Privacy professionals and IT gurus understanding each other and thus being able to frame the issues/requirements/concerns, etc taking into account  each other's prospective is not something that can be done within a simple blog. But I hope that it will open some people's minds on what the issues are and some resources that will bridge the gap. Or at least have each side gain a better understanding of the other.

Next blog will continue along these same lines.

However the next blog will be in two weeks time.

Till then, if you have any comments or feel like you want to touch base, drop me a line at rgalambos@gmail.com

Testing and Data Privacy, is there an issue (final post or is it)?

In the previous posts I covered the issue of testing data and privacy. What options are  generally available to 'address' the issue, and a description of what each of the options are.

This time I will wrap up this portion of the discussion and then further delve into related issues that may be of interest.

If you have read the previous post, you may have surmised that the option I favour is analysis of the data structure/elements. Then applying intelligent business savvy masking rules to the copied data This entails designing a process that would obfuscate the Personnel Identifiable Information(PII) data by applying rules that take into account the business logic  to the information that is retained within the organization for testing purposes.

But at the same time there is never an all or nothing answer to these issues. It all depends on the situation, the company culture and requirements to name but a few mitigating circumstances. But let me explain.

And by the way, I will try to stop myself from going down the techy talk that most NON IT people get lost in.

So let's assume the company we work for has surmised that the testing environment(s) that exists presently needs to be scrubbed  to ensure that there is no real PII information. Yet the CIO also insists that one of the requirements to ensure quality work is the ability to copy real data for testing from time to time.

So the requirement is to copy data, when needed, but removing PII at the same time. That the removal of the sensitive information will still retain the quality that is needed.We need to develop a  process that scrubs the data consistently and have it executed whenever a data copy is to be done. Right?.... But how much data do we scrub? Do we need more then one copy? Who is going to be responsible to maintain the obfuscating rules etc.?

These are just some of the other factors that need to be considered.

As you start your analysis  you may come up with a question along these lines. Will there be  a need to sub-set the data while copying the real data for testing?

The more revealing question may be, what are we going to be doing with this data after it is scrubbed? You might think that testing is the response  and you would be right. But what kind of testing? You see in most medium to large companies there are more then one kind of testing that is done before any changes are put into the real world.

There is the testing that the coder/programmer does to help make changes to the code to ensure that the program works and produces the anticipated results. Generally speaking, this is called unit testing. In this case there may not even be a need for real data just some made up stuff. So we might not need to consider this type of testing in our requirement analysis.

Then there is what I call kernel testing. To run a logical unit/series of 'programs' (yes they can be stored procedures, scripts  etc, but I am trying to keep the terminology simple and it really means the same thing) to see if it runs with the changes successfully. Usually this is where a small sample of real data would be used. The data used here does not have to be related to any other application/data, so the masking process would be rather easy to implement. There would be no need to ensure that the same rules that are applied here would be applied to another application within the organization.

Next is some form of regression testing. Simply put, this is to make sure the application still works with the changes done to code. However, you will probably not want the same number of records as production data. Otherwise each test would take the same amount of resources/time as production. Remember, you are testing to make sure everything works, and if it doesn't you need to correct the issue and retest.  The old adage goes like this, time is money. The quicker the programers/coders can turn around the testing the better.  That means you will need to sub-set the data in question. AN example would be take a single branch's data as a test versus the entire companies branches. However this is not as easy as it sounds.

For example, if we have a banking application that we are going to be testing, we may decide to use only branch 'A' as the testing branch. This branch has a wide variation of customers etc. and it fits very nicely in the testing that needs to be done. We will need to copy only those customers within that branch (this most likely will be in some other location database). We will then need to copy only those accounts of those customers within that particular branch. In other words copy all the related information and only the related information for that branch, sub-setting the data. Oh, don't forget that we will need to mask the data as it is being moved over from production to avoid any potential issues further down the line.

Next maybe a user acceptance test allowing the users of the application in question to test the change(s) to ensure it is what they asked for and it works are required. While a complete copy of data can be used,  a sub-set of data can also be used in most cases.

And then in the next order of business there may be  a volume test. This test is normally done to ensure that the application can take the real world volume. (all the branches), the final kick of the tires, you can say.


Now while I have generalized, and each company/requirements are different, I hope you can see complexity that is involved.  The type of testing and the data used for that testing is extremely important, and it is just as important to analyze each testing requirement and come up with a solution that meets  all the needs.

So lets assume that we have all the answers to the questions posed above. We know what kind of data we need, the various versions/copies and the  other parameters that may have been discovered. What is next?

The next  post will cover the how to. The components of privacy project, the pitfalls, the bumps on the road, and the elephant in the room (and yes there is a BIG elehpant that needs to be fed)

While is may not be directly related to a privacy role, anyone in privacy needs to understand the complexity inherent in the process that a company needs to go through, so the project will come to a successful completion.

So I strongly suggest you stay tuned for the next installement. Till then if you have any questions feel free in contacting me at the email address below

rgalambos@gmail.com

Testing and Data Privacy, is there an iIssue, (PART III out of IV)?

Let's recap. In the previous posts I discussed why we should be aware of how application changes are tested within your IT department etc. or we may have a data breach before you know it. Then I explained how to mitigate some of the risks  with different processes/choices and listed the pros and con for each of them.
I will now continue the discussion about the various options, and which ones are the best etc.

So lets get started

The four choices that I presented previously are
1) Create your own test data
2) Copy production data into the test environment
3) Same as #2 but have everyone sign Non-disclosure agreements
4) Same as number #2 but obfuscate(scrub) the data

Looking at the obvious option #2. that is clearly a TABOO or is it?. The reason that we should not do this is obvious, right?  Copying Data  is what happens in the real world today. As far as I know there are not studies along these lines, (most companies would not want to share this type of information) but experience tells me that you would be surprised at the number of companies which have at least some areas where this practice is done regularly. While it can be argued that this would happen only within smaller companies, experience would say otherwise. Remember that you may have a policy in place forbidding this, but in some corner area of IT that has been around for years, they may be practicing "copy the data" because that is how it was always done. That being said, you may be surprised to hear me say that there can be times when there is a legitimate reason (fooled you) to copy production data within a testing environment.

This will be a topic for a future post concerning (and this is a BIG hint) testing, cost, risk  and support issues that revolve around data and data privacy.

For now let's just say this is not a good option and should only be considered in specific areas and reasons.

Option #3 in my opinion is slightly different then just 'saying no'. It should be standard policy that all individuals, no matter who they are, employees, consultants or outsourceers need to sign a non disclosure agreement.  But let me clear, this will not help in preventing any data breaches. And just to remind you why, there are studies concerning data breaches that state that more than 70% of all data breaches are non malicious. If the breach is malicious (disgruntled employee, criminal activity etc.) it will not stop data from be exposed either. So if it does not prevent breaches, why bother? What this  does is make it easier for legal remedies in case there is a need.

Option #1 is a viable option. Many companies I worked with have policies along those lines And in fact chances are that your testers will have to make up some data to test things that should not happen in real life. IE testing for error checking/handling. But is it be all to end all? No. One can never make up all the permutations and combinations one would need to test  to ensure that, first the change worked, and two that it did not break anything else. Now there are processes that mitigate the risks (for another post) involved. However there are no guarantees.

Last but not least there is  Option #4. This option states that all product data copied over to testing should have the Personnel Identifiable Information (PII) scrubbed. There are problems even with this option. To do a good job in scrubbing the data (it took me two years to be able to even pronounce obfuscate, never mind to spell it, so scrub is the term that describes the option as well, and easier to roll off my tongue) takes time, money, expertise and some risk.

So what does the process entail. How does one go about scrubbing data? The first step is to identify all the fields that have PII. Easy, right?. Nope. In this complex world we live in, I can assure you in saying,  No 'data' is an island entire of itself' (to Paraphrase John Donne)

Programs (applications, process etc) work together. The bill that is entered in the Accounts receivable system needs to be posted into the GL (as an example). etc. The bill also has a purchaser's Credit Card Number that feeds the Credit Card processor etc. The address on the bill is entered in the customer information system.

This interaction can be complex to say the least. One application has edits in place to verify a Zip/Postal code matches the address because the program that sends out mail needs to make sure the combinations make sense. But the application that is used for analyzing buying habits may not even look at this.

 Once all the PII fields are discovered and how they are related between applications/files/databases, the next step is to figure out what method should be used to scrub the data given the interaction I just described. Do we scramble the values, or should we generate new ones. Does the data need to follow certain business rules? Are there home made systems that need to be used to mask the data (IE. account number generator).

There are basically four differnet types of scrubbing methods. 

#1 A simple scrambling method. Taking wherever the letter 'A' appears and changing it to 'X' as an example. (there are variations of this to make it harder to reverse it the results).

#2 Looking up a translation table. by various methods  using the original value as a key to find an entry within the translation table. So if that value appears in another location, the same scrubbed value is returned.

#3 Generating new data. Basically either randomly or with some guidelines. This is an issue because every time the same value will be scrubbed, the result will be different this losing consistency.

#4 Replace the data with a 'string' or blank etc. As an example putting 'N/A' in each free form field because no processing is done to that data.

And there are other techniques that I did not mention, such as, date aging, flip flopping of real data, mathematically manipulating the values etc.

After it is determined what techniques are to be used, the next step is 'coding' the rules to be applied. and then testing them. Expect that this is an iterative process because the more you do, the more will appear that you may have missed something.

And finally the implementation of the process.

This is not an easy task, nor is it something that should be taken on lightly. But if you don't want to have your company in the cross hairs of journalists, bureaucrats, courts, general public. You need to do due diligence (making sure you do the best you can to prevent data leakages).

In the next chapter I will talk about how this  fits together in the overall picture, how one needs to  consider other factors when talking about testing.

Testing and Data Privacy, Is there an issue(Part II)??

So here we are now. Lets  recap some of the major points about the subject that we covered previously before we go on.

IT departments maintain and use both 'Production' (what is used to run the business) and Testing environments. They need data to test with. And where do you think most of the testing data comes from? In the 'real world', it is most likely 'real' Credit cards numbers ( PCI DSS does not allow this (Payment Card Industry Data Security Standard), Tax Identification numbers etc.

And to further complicate the matters, testing by it's very nature, means easier access to the data by Developers, Testers, IT operations etc. And this gives us the exposure that business try  so hard to avoid. And you may not even know about it.

So lets take a look at some legal ramifications of this matter. 

An example is in  Canada, where one of the principle laws governing Privacy is Personal Information Protection and Electronic Documents Act (PIPEDA). Basically (and this is an over simplification but is good enough for this discussion) the Company will use the Personal Identifiable Information (PII) it gathers solely for the intent that 'advises'  the user. So if a user goes into a bank to open an account as an example , he/she has to sign a 'whole bunch' of papers, and more often then not get a copy of them to take home to wall paper the house (I know its a bad joke.) Realistically these statements are only read by a lawyer or a privacy specialist). 

But in all seriousness, at least one of these documents(Best practices) is basically an agreement made with the bank that allows the bank  to gather the information they need, to provide the service you are requesting from them. It also states who they may share that information with, and how they will protect it, and hopefully list a  Department/Person in case one has any questions about the Privacy Policy of the Company. 

I guarantee that there is no place in that document that states the company may use the information for testing purposes. And don't forget the looser criteria requirements of the testing world.

If you think that this is only for Canada you will be mistaken, big time. As an another example in the EU one of the applicable 'laws' is called Directive 95/46/EC (Or more commonly known as the The EU Directive on Data Protection). It is one of the most stringent laws pertaining to Privacy there is. And don't be fooled by thinking that just because you do not have any offices in the EU or Canada etc, you don't have to worry about that. In fact if you have any customers from the EU, or collect some information while they are on your website, you may still be under their Privacy jurisdiction.

Now this particular aspect is worth a book in itself, but let's just leave it for now, and if you, the reader agrees, we can try to figure out what needs to be done, and the benefits/cost of each solution.

1) Well, lets create the test material needed and not rely on ANY real data.

The Pros:

Will not need to worry about relaxed security restrictions because the information does not represent any real person.

The data is  'easy' to create. So  even if the printed reports are found in the trash bin there will be no worries.

The Cons:

 'Quality' of the made up data. Is the data,  a good sampling of the various permutations and combinations of different aspects of your customers. I.E. do you have  customers who live in NYC (Hong Kong, Budapest, Montreal etc) and who have a chequeing account in the spouses name as well as two children's accounts, etc. If you do not cover all the different variations that exist, how do you know that your testing is complete and will be able to  discover failures before implementation?

2) Copy Real Data for use in testing

The Pros:

You will be testing with real data, and if there are a issues, they will be discovered before the change is put into 'production. If the tests work then there is no reason why it will not work during productions

The Cons:

As previously discussed, chances are that you are close to breaking some laws  (if any of the information in question is PII).

The data volumes, is another concern. Who nowadays has the capacity, large or small business, to be able to copy the entire production data to be used for testing. And if we are talking about most major companies they may have many testing environments to help them to move forward.

Then there is extra time you will need for multiple testing to be done with large amounts of data. (another topic in my series of Blogs in the future will be about volumes of data and testing types, etc. and issues/solutions).

The reduced Security (see above) around the testing will allow increased access. This could increase the chances of a Data Breach.

If there is a  Data Breach, your company's reputation would suffer and its name may appear on the front page of the local/national newspaper etc. The  cost of loss of customer confidence with your organization may also effect the bottom line. This can cost millions of dollars and loss of business. (All depending on the number of records exposed).

3) Copy Real Data For use in testing and have everyone sign non disclosure agreements.

The Pro:

You now use real data, with all its different combinations, to test with and the legal protection of a non disclosure agreement.

The Cons:

According to some studies, over 70% of all Data Breeches are non malicious and therefore agreements of this sort would not stop a breach.

We are also still looking at large volume issues.


Real data may not have all the information you need for testing properly (testing for error handling as an example)

4) Copy and  obfuscate(scrub)  the PII data so no one can figure out who the real data record represents

The Pros:

You get real data to work with and thus even if a reports ends up in a trash bin, no one can figure out who the data identifies, belongs to.

The Con:

You will need to have a full understanding your data

You will have to do  analysis work on how to scrub the data.

You will need to understand how the PII data  work together within your environment/application.

In my next blog I will further investigate all of the above options and discuss which option maybe the most suitable for your situation. Maybe a hybrid solution could be the answer.


If you have any comments or questions, feel free in dropping me a line


As a note, this blog is not attended to be legal advice.

Robert