In the past week I helped a client whose programming staff had just caused a business disruption for the fifth time in two months because of the changes they made in the program code of their online service. The programmers, and so many of my other clients, have expressed the opinion that they can just code something and plop it out into production, without testing. And then they try to tell me that is “agile programming.” No, it is not. It is unsecure and, quite frankly, lazy programming.
Over the years in the security classes I’ve taught, I’ve covered the importance of program change control management, and I wanted to continue the discussion here because as important as it is, it typically does not get the attention it deserves in most organizations. And most of my clients that are start-ups, mobile apps developers and cloud service providers did not have any change control processes in place. Too many of them started their business with basic programming knowledge, but absolutely none of the very important knowledge necessary for secure programs that will not fail, cause disruptions, or even leak personal information and cause privacy breaches. It seems from the many continuing exploits of online applications that there is still work to be done around application program change controls.
Looking back to program code better going forward
I actually got onto the information security, privacy and compliance path way back at the beginning of my career as a result of creating and maintaining the change control system at a large multinational financial/healthcare organization.
The programs were all housed in an IBM 390 mainframe (where most of them still are today; mainframes now seem to be high-speed application servers) divided into four regions for each of the several business unit regions.
My change control system was used to move a program from the development region to test region to the pilot/beta region, and finally to the production region within each of the applicable business unit regions. It was an online system that required authorizations for each of the moves. A manager had to approve, through the online system, of the move from development to test to pilot. A director had to approve, through the online system, of the move of a program from pilot to production. The documented procedures required the managers and directors to carefully review the change documentation, and proof of thorough testing as signed off by the program team leader or manager, respectively, before they would provide their approval within the system.
The concept was good. The system was good. The procedures were good. Unfortunately many of the individuals using my change control system were not so good.
It was a real frustration for me to walk through the many different programming areas (we had around 700 programmers at the time) on Thursdays (the last day of the week for directors to approve of program changes to be moved into production on Friday) and see so many of the directors with their terminals logged on and open to access (no PCs were used in the programming area at the time…that actually didn’t change until the mid-1990s), and not even at their desks or in their offices, so that the programmers could go in and make the online approvals themselves!
That bothered me for a couple of reasons…
- At a personal level, I wondered why I put so much time and effort into creating a sound, tightly controlled change control system, only to have the people authorized to use it defeat those controls. Many of you will think, “Whatever; get over it.” Fair enough. But then…
- At a business level, I saw how dangerous this was. As a result of these managers and directors not really doing the reviews, each week we had a large number of production moves that had to be backed out on Friday afternoons because of the problems they caused. Many were very minor problems, but some brought the system to a standstill or even messed up the customer databases significantly before the problems were noticed.
After being responsible for this online change control system for almost two years, there was an opening in the IT Audit area. Working on the change control system helped me to see firsthand the importance of controls, so I applied for, and got, the IT Audit opening to learn more about how controls impact business.
Program change controls are needed now more than ever
One important lesson, then, is that even with the greatest systems and procedures in place, if the individuals who are authorized to use the systems, and make the authorizations to move code from development to test to pilot/beta to production, do not follow procedures (because it is too inconvenient for them to do so, too time consuming from their perspective, not worthwhile in their consideration, or whatever) the controls will be defeated and incidents and problems will occur.
With even more diverse and complex systems and programs/applications running on the current business networks, it is more important than ever to follow effective change control processes. Especially considering almost every type of program today involves some type of personal information, in some way. There are exponentially more risks with program code today than 20 years ago, when it was also vitally important.
Managers and Programmers, do these things
Create change control procedures! You need to follow change control procedures to consistently make:
- Changes involving new program code implementations
- Installations of new code into your existing environment
- Changes within existing codes to change, improve, etc., a feature or process
- A fix to a discovered error or bug
- Technical changes and enhancements
- Emergency changes
- Configuration and parameter code changes
Some things to keep in mind:
- Be sure to look beyond just the documentation and the systems capabilities within any change control system; also observe how well individuals are following those procedures.
- Technology tools are necessary and good to support information security and privacy, but they cannot, by themselves, provide effective safeguards for business information. Well written and correct code is necessary for ensuring incidents and breaches do not occur.
- Personnel must receive effective training and ongoing awareness communications to know not only what they must do to build program code to safeguard information, but also why.
- Noncompliance with change control policies and procedures must be consistently enforced, or the policies and procedures will not be effective.
Oh, and as a side note, after I went to the IT Audit area, the common practice for leaving unattended terminals and PCs logged in and unsecured, allowing others to use them, changed soon afterwards.
This post was written as part of the Dell Insight Partners program, which provides news and analysis about the evolving world of tech. For more on these topics, visit Dell’s thought leadership site PowerMore. Dell sponsored this article, but the opinions are my own and don’t necessarily represent Dell’s positions or strategies.
Tags: awareness and training, change controls, Dell, Information Security, IT compliance, policies and procedures, power more, privacy, privacy compliance, privacy professor, privacyprof, program changes, risk management, security awareness, security training Rebecca Herold, toprank