Perimeter security solutions are like the walls of a fort: organisations have been trying to strengthen these proverbial walls, building them as tall as and as thick as possible, hoping to stop all intrusions.

But in reality, no matter how high the walls are, they still are not able to stop all intrusions, simply because many of the unknown threats actually loom from within the organisation.

As Sun Tzu said in The Art of War: "If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle."

The best intelligence we have on potential threats is found directly at the endpoints – but this can be like finding a needle in a haystack given the vast reams of data involved. So how best can organisations manage these data mountains to gain a better insight into threats from within, and avoid succumbing to the battle?

The threat within

Known threats can be tracked and are typically blocked by antivirus software and firewalls, while security incident and event management or SIEM tools help to identify and flag potential threats. While these are necessary measures and can be effective in stopping some threats, they can only stop a threat once it has found its way beyond the ‘estate’. However, what about the threats that have already infiltrated the walls but which may go undetected for weeks or months? Although employees are often cited as "the weakest links", whether as a result of intentional behaviour (such as retaliation from a disgruntled employee) or accidental, when they inadvertently click on malicious links embedded in e-mails, threats may come from targeted attacks or spear phishing as well, and likely lurk concealed for a long period of time.

Finding the needle in a haystack

Organisations today generate large volumes of data. In fact, every day we create 2.5 quintillion bytes of data – this means that around 90% of the data in the world today has been created in the last two years alone. Every minute of every day, email users send 204,166,667 emails, Google receive over 2,000,000 search queries, and brand/organisations receive 34,722 likes on Facebook.

With the massive amount of endpoint data being generated, coupled with the fact that sophisticated cyber threats often bypass traditional perimeter security and target the rich data stores residing on endpoints and servers, many IT departments are left in the dark. How can organisations best aggregate data, find meaningful correlations, and more importantly, pinpoint and remediate risks? It can be like finding the proverbial needle in a haystack.

"Big Data" – what does it mean?

‘Big Data’ does not necessarily equal ‘Big Intelligence’. Data collection and preparation is a mammoth, time-consuming task. It is very laborious to collect a vast amount of unstructured data, and requires a deep understanding of the data sources, data models, and metadata to be able to cleanse the data for analysis.

Further, specialist and experienced data engineers needed to gain meaningful insight from the data are scarce and often are not available to the security operations teams.

However, once data has been put in place and cleansed, organisations will need to derive insights from these data, using predictive analytics and machine learning statistical models, such as the logistic regression "Naïve-Baysian classifier", where "k- means clustering" for example. However, non-technical stakeholders will struggle to comprehend what this entails and how this translates into reputational and/or financial damage.

Satellite view of Data

Forget the dashboard. Organisations need to zoom out much further than that to analyse Big Data and find correlations within disparate data sets that may indicate security risks. Big Data Analytics can be applied to the mountains of the data generated. This will help provide organisations with better insight into endpoint activity and thus help them get through the ‘noise’ to gain a big picture view to improve security risks and threats. Consider the following:

1. Typically, due to resource constraints, organisations generally sample smaller data sets, but must be mindful of the data sample sizes, to ensure the tests performed can generate statistically significant results. However, the tests would be more useful if they were able to look at the entire data population to gain a holistic assessment.

2. To gain intelligence and insights in an organisation’s security posture, IT departments should not only inspect machine logs, network packets, and end-user activity to perform Big Data Analytics, but also investigate encrypted data, slack space, registry and RAM on the kernel level. Although extremely difficult to obtain, this is important because it avoids making false assessments by analysing information from compromised operating systems.

3. How can organisations spot an anomaly when they don’t know what "normal" is? One of the critical dimensions of security intelligence is to be able to determine a baseline of normal activities over a period of time. Cases such as sudden spikes of unique processes running across the endpoints could then indicate something that requires further investigation, but it may not be visible as "anomalous behaviour" unless it was tracked over a period of time.

4. Verizon’s 2013 Data Breach Investigations Report revealed that approximately 70% of cyber breaches go completely undetected by organisations’ security teams, and are instead discovered by external parties like the authorities, FBI, or even the attackers themselves. Revenue and time losses aside, such publicly known cyber breaches have a huge impact on the affected organisation’s reputation.

To obtain insights into unknown threats, organisations typically focused on log files or network packets. However, this approach alone is no longer sufficient to detect the anomalous behaviour of new and unknown threats. Today, organisations need a complete security intelligence solution that leverages endpoint analytics to produce a clear picture of security risk and exposure to unknown threats before they do irreparable damage to the business.