A recent 1E survey of 1000 U.S. IT operations pros found that they spend, on average, about one-third of their time dealing with emergencies. That’s about 14 full weeks a year. Of course, some amount of firefighting comes with the job, but one-third of their day surprised me.
For those of you involved in staving off the WannaCry attacks, this dynamic is probably still fresh in your minds. In another 1E survey, 10% of respondents said that their organizations were infected by WannaCry, and 86% said they dedicated significant resources to protecting their organization against the attack.
Not coincidentally, 86% also reported that their organization doesn’t or can’t rollout security updates immediately – exactly the same proportion as had to divert resources to safeguarding against the attack.
I’ve spent my entire 20 year career working in Windows environments. In that time, we’ve seen a plethora of new tools that promised to help simplify and fortify IT infrastructure, yet we are spending more and more of our time reacting to emergencies, and less time innovating. IT operations teams are constantly interrupted with emergencies such as hacks, security updates, configuration changes and software audits. They field requests from all over the enterprise, which can lead to conflicts and make it difficult to prioritize. So, what should IT ops teams be doing to get out of the fire drill cycle?
The first step is to automate all the easy things, especially repetitive, commodity tasks. For IT ops, these might include provisioning new devices and keeping their software current, including patches. Those are all tasks that are ripe for automation.
Many IT teams feel they are already doing that, by using tools like Microsoft SCCM. SCCM is the best Windows management solution and is used by most organizations. It is however a general purpose tool and does not provide real-time visibility and end-to-end automation, as was evidenced by how slowly organizations responded to the need to patch their Windows machines during the WannaCry and Petya outbreaks.
According to our survey, 71% took more than one day to apply patches and a quarter took between one and four weeks. Given the extremely destructive nature of these viruses, organizations must respond faster.
To do that, organizations require a combination of tools and processes that have a single vision – to keep the organization current with technology. To achieve this, security and IT infrastructure and operations (I&O) must learn to work together. In a recent blog post, Forrester Senior Analyst Chris Sherman made the analogy of manning toll booths versus building automated toll lanes; Sherman believes I&O teams should be focused on the latter, building the appropriate infrastructure and guidelines and then letting other teams – like security – do their thing within that structure.
Presently, there is a tendency for security to keep buy and implement tools that effectively compete with I&O solution already in place. The fact is that security is not best placed to appreciate the wider impact of these tools, to oversee the change management processes, or to help resolve issues caused by the change. I&O and security teams need to work together to stay current. Some steps they can take:
- Start reporting on whether all software used in your organization is current. Very few organizations look at the vendor updates available and compare them to their installed versions.
- Build processes and automation so IT can update all software without touching each system, as this is very costly. This is important especially for Windows upgrades, such as the migration to Windows 10. Rather than earmark this as a huge one-off project, instruct IT to build automation so they can always keep their systems current. Microsoft SCCM itself does a good job and there are third parties like 1E who leverage the Microsoft solution to complete the automation steps and provide a practice solution to staying current.
- Invest in a system that enables your IT team to check if you have a problem or are being hacked and deal with it in real-time. Your organization is constantly under attack; you need to be able to react in seconds and your current systems probably respond in days. Ideally, companies need tools that integrate with SCCM so that better decisions can be made.
The hardest step is the first one: the decision to stay current. Most organizations have yet to make this commitment. However, 70% of our survey respondents said that, post-WannaCry, the will to stay current in their organization has increased.