As the IT support business we want to keep our users happy, they need to use the software with no interruptions. Though the share of the incidents in the tickets is still 30% no matter what.

So why is IT support is so hard these days? Why do we still have the incidents despite all the progress IT industry did so far?

Long story short this is because of the software complexity, that causes incidents and security issues.

Let’s discuss here why do we still have incidents, and what to do to decrease their likelihood and improve our users happiness.

Planning is hard

Normally IT department has two types of work - planned and unplanned. Planned are mostly projects and requests for changes.

Unplanned work consists of incidents. An incident with big impact sucks out resources. Unplanned work disrupts work rythm, increases anxiety and leads to burnouts.

So unplanned work should be avoided as much as possible.

Incident impacts planned work

Definition of the incident is when something doesn’t work as expected, i.e. either it doesn’t work what was inteded to work, or it works with degraded quiality.

Normally incoming incident has more high priority than planned work, therefore it affects the schedule of the planned work.

So why do we still have incidents?

For a number of reasons with the leading one. Growing complexity.

Increased application state complexity

Combinatoric burst of the system state complexity. “Reboot your system” - this means to recreate the application state from scratch. This is because right now your system is in the state that wasn’t foreseen by the application developers.

If you have lots of applications set on the system for support, there is no way to ensure all applications will collaborate nicely since they share the same system components and no one conducted testing for such cases.

You are a tester.

Increased software complexity

Increased software complexity leads to incidents.

In order to be competitive on the market vendors forced to add new features. Large codebases lead to the little understanding of the all parts of the application and their interconnections. Increased complexity, in turn, leads to bugs.

Applications use shared system resources

Docker tries to address this issue and isolate all parts of the system from applications, but there is no such thing as docker for UI rich applications so far.

This is why we have raise of the Chromium based web applications pretending to be native (Slack, etc.). Which is greate waste of resources and fallback from progress industry made so far.

Viruses exploit this concept in a hard way.

Security issues

Software industry demand for workforce is huge. So we have a big wave of newcomers with no or little experience.

Normally PRs are reviewed by senior developers, but sometimes I personally witnessed cases when PR from junior developer for whatever reasons was approved by another junior developer. And here we go, bad code landed the base with possibly new security hole.

Cost cutting and outsourcing also normally do not increase the quality of the software they produce. Moreover this effect present even in critical things, remember Boeing issues with 787?

Windows 7 or Windows XP still exist, and they don’t receive security updates.

Unstable environment

Networks are unreliable, think about WiFi in a crowded space. Disks have limited lifespan. Batteries loose capacity.

What to do?

Split planned work and unplanned work to the separate teams. If not possible, then develop on-call schedule with rotation. This aspecially effective for small teams.

Reduce amount of supported applications on the user workstation.

Prefer running web applications over native applications. Use Gmail, not Microsoft Outlook.

Prefer running applications using terminal access, Citrix, etc.

Harden security measures. Use the rule of thumb to provide as less access to the sytems as needed. Perform security audits from time to time.

Provide users with standardized hardware and software.

Perform automates security updates.