It happens time to time that I need to do penetration test on a production system because the costumer doesn’t have any test environment. That is always a bad start. The explanation is usually that the target system is not a separate application or appliance but it is tightly integrated in the company’s infrastructure. Nevertheless it is very unfortunate from the penetration tester’s point of view. This brings the following limitations in the pentest:
- no insertion in the database that cannot be removed
- no agressive tests
- careful with business logic
For example in a production system it is difficult to decide whether clicking on a submit button will immediately order something for €1000, send out 10000 e-mails or first pop up a confirmation window which might be vulnerable to Cross Site Scripting.
Although if you are a pentester I must have already convinced you that not having a test system is bad. Still as you can imagine there must be some reason why companies don’t create test systems. Of course everything can be tracked back to budget. A test system has to be installed, maintained and upgraded so that it is always functional. This costs physical, computational and human resources and all of them cost money. Nothing can be done with this part, but after examining the values that the test environment could bring everybody has to decide themselves whether it’s worth it. An even bigger obstacle then money is the complexity. In a complex infrastructure the different services and applications are tightly integrated with each other. That gives the impression that if a test system for an application is to be created then the whole infrastructure must be duplicated as a test environment just to give input to the testable application. That is a little misleading because drivers and stubs can be created with a medium amount of effort. Of course it must be designed and planned as every other development in the system. But let’s see the values that are brought by a test system which in my opinion overcome the time, effort and money investments:
- Creating a test system will force people to understand and document the system and it’s relationship with other systems. I think everybody who works in IT knows the feeling when one tries to solve a problem and tries to find a person who actually knows how the particular system works and at the end there is nobody who could help. This is very typical in complex systems that evolved during the years. But when the test system must be created there is no other option then finding out the dependencies between systems and how the infrastructure works. This is a great opportunity to document it.
- Understanding the system can lead to future development and optimisation ideas.
- As the main goal of the whole thing, development and IT will be able to test new features or software upgrades in the test system. This can be a huge step forward because many times companies have to immediately go live with new features taken from development because there is no place to do integration/regression/acceptance tests.
- More thorough penetration tests could be done. I had to mention it since that was the whole motivation behind this blog :).
- Due to proper testing there will be less downtime because of software failure.
- Trust in the system will increase. Many times questions come up like ‘Can we do this or that..’ and the answer is ‘Hmm I don’t know. You should try it.’. The only problem is that there are lots of things one cannot try in production system. If there is an up-and-running test system one can just simply try and see whether it works or not.
- Test system could be used for education purposes. I think everybody experienced when the application that one was supposed to use day and night was introduce in a ‘read-only’ way with the comment that ‘you must be really careful because…’. Test systems could allow exercise oriented introduction of an application.
As a last idea I just want to mention that using virtualization for the test environment could reduce the costs and virtual machine snapshot could really help in maintaining the system. Also this way the whole test infrastructure can be easily copied and reproduced.
I hope I gave some good arguments to get the decision makers to invest in a test system. If you see any other advantages or disadvantages or have experienced it while building your test environment, then you are welcome to write a comment about it.