Incident response and the false sense of security: Some time ago I was asked to help with incident response for a small company. While the incident itself was not very exciting, the lessons learned were a bit more than a surprise. The victim was shocked how spectacularly they failed even though they considered themselves to follow good security practices or at least to be above the “low hanging fruit” category. This is classic example of false sense of security.
Key lessons learned:
Running a hardened web server as reverse proxy to protect the actual application is a great idea, however if the actual web application also listens on publicly available TCP port, there is nothing to stop the attacker from going after the application directly, bypassing the proxy.
(If possible always bind the applications to localhost only or at least use the firewall to limit access to the application. This is how the attacker got a foothold on the system – known vulnerable web application and bypassing simple but efficient virtual patching done by the reverse proxy.)
Hard-coded passwords and password reuse – as it turned out, all of the IT systems and components used the same administrator password. The original password could be found in a publicly readable backup script on a compromised server located in the DMZ.
(Backup process is one of the most sensitive elements of the system – should everything else fail, backup is all you have. If privilege separation was implemented and properly used the attacker wouldn’t get the administrator’s password. Finally there is no excuse for password reuse – password management applications are widely available and really easy to use.)
Centralised logging can be very useful, especially if it’s used with some kind of log monitoring solution. Unfortunately it can also create extra work if you try to review logs from the incident and notice large portion of the systems having their clocks off by minutes or hours.
(Keeping all your system clocks in sync is really important. NTP clients do the job very well and are already built into most if not all of the network equipment and general purpose operating systems. Another thing to keep in mind are time zones – make sure all systems use the same time zone and if possible pick one that doesn’t observe Daylight Saving Time (DST) as this has great potential to create additional issues or delays if the incident spans systems located in more than one country, especially if it happened around DST time change. Remember – simplicity is your friend.)