Microsoft Talk Machine Learning for Cyber Security

Microsoft Talk Machine Learning for Cyber Security

The growing volume and sophistication of cyber-attacks has made cyber security a major cause for concern for enterprises of all sizes. These cyber threats disrupt IT systems and hack sensitive business and personal information by using malware, spyware or by phishing.

A significant increase in the number of connected devices has made it difficult for organizations to monitor, manage and maintain data traversing through wireless networks. With these attacks growing in scale and complexity, organizations are spending more and more on digital security solutions to actively manage and reduce security risks.

At last month's RSA Conference in San Francisco, Microsoft’s Chief Technology Officer Mark Russinovich spoke about cybersecurity and how the company is using machine learning for its anomaly detection. The following blog will look at how Microsoft is protecting its infrastructure by applying data mining and machine learning algorithms.

Working on a Massive Scale

It can be difficult for companies like Microsoft to protect its data when working on such a massive scale. They have over 300 million active users, and security analysts must sift through alerts coming from multiple event streams. Whether it is unrecognised credentials or a suspect e-mail address, an independent alert is created for the analyst to deal with. Russinovich says they have had breaches in the past where the post-mortem shows the attacker was active and alerts were generated, but the analysts were overwhelmed by information.

Another problem can be the interpretability of alerts. A security analyst working on a triage might decide that something is not anomalous, based off the rules generated by that event. But this might not get fed back into the rule system, so the information is effectively lost. They needed a system that automatically feeds this information back into the rule system, and that’s where machine learning comes in.

Machine learning highlights patterns in the data and makes anomalies visible, giving the security analysts a better chance of detecting malware. You feed in the data and output, labels and features, and machine learning will create the best program to generate this result. When there is a miss on the output, this is fed back into the algorithm so it can improve.

Security is just one of many application areas for machine learning. Machine learning algorithms have been a feature of online advertising for some time now. It is working in the background compiling information about our usage patterns and what sites we are visiting. According to a recent report on AI technology, there is an increased demand for machine learning in the retail, healthcare, law and finance sectors.

The problem in security is that there is no room for error with machine learning. Unlike with advertising or any of these other sectors, if the algorithm isn’t extremely accurate, it could cost you enormously. Russinovich makes the point that, in security, you could have one-hundred thousand successful log-ins and one bad one. The task for the machine algorithm is to find a model that can pick out that one in a hundred thousand, every time.

Anomaly Detection

Microsoft say machine learning has helped reduce the triage of burden by prioritizing alerts based on the most suspicious and by ranking them. Instead of getting a flat list of suspect credentials and anonymous IP addresses, machine learning takes all these signals together and decides which the analyst should focus on first.

The company also uses user feedback to improve the system signal. If machine learning detects something it deems suspicious, the user will be contacted. If it was them accessing their account, this information will be fed back into the algorithm to improve its anomaly detection.

One of the main problems Microsoft had in the past was they had no idea what generated the alert or why it was deemed ‘suspicious’. The machine learning algorithm can tell you exactly why it ranks something as a high-priority anomalous. Instead of analysts being overwhelmed by contextual information, they can immediately focus on what is going on and what the correlating factors are.

So, now we know why they use machine learning, but how exactly have they put it into practice? Russinovich gave two examples in his presentation - DevOps anomalies and Geo-Anomaly Detection.

Machine Learning Algorithms

First up is DevOps anomalies, which means infrastructure. Machine learning is used to identify rogue users, users that have been compromised or a service account that has been hacked. Microsoft needs to know when these accounts are performing unusual operations.

This is done by taking all the data from Microsoft’s logs and featurizing it. For example, a typical event log has an operation, a correlation ID, a time of day, an API that was called and a target system that was interacted with. A model is built based on this data, with a real-time scoring system. The machine learning algorithm will process and flag anything deemed suspicious. Finally, there is an auto-triage system and report for the data analysts.

In his speech, Mark Russinovich gives a more detailed explanation of this algorithm, known as the Principal Component Analysis (PCA). He talks about how they find the vectors that capture the most variation in the data and why they use random projection. You can find this on the RSA Conference YouTube account.

The second example he used was for geo-anomaly detection. One of the easiest ways to determine if an account has been compromised is to look at suspicious data in their log-in patterns. This machine learning algorithm caches the last 10 locations of the user. If the location used is not on this list, it will challenge the user. If this turns out to be a false positive, the location will be added to the cache. According to Microsoft, the false positive rate is currently 0.001 percent.

But there were some problems that needed to be addressed, including company proxies, cellphone networks and holidays/travel. The solution was to create simple rules for determining suspicious log-ins and a machine learning approach for determining normal behaviour. This was done by incorporating the behaviour of similar users, travel heuristics and device familiarity requirements. The algorithm would flag the unexplainable remainder.


The difficult task for machine learning is that Microsoft is operating in a constantly changing environment. What looked normal a month ago, won’t necessarily look normal now and the machine learning algorithm must adapt.

It is also inevitable that cyber criminals will find a way to con machine learning and avoid detection. Russinovich describes it is as “a cat and mouse game” and says they will have to get more robust as more people begin using machine learning.

Stay up-to-date with the latest market developments, trending news stories and industry advances with the Research and Markets blog. Don’t forget to join our mailing list to receive alerts for the latest blog plus information about new products.


Published by Research and Markets


Artificial Intelligence (AI) Market by Technology (Machine Learning, Natural Language Processing (NLP), Image Processing, and Speech Recognition), Application & Geography - Global Forecast to 2020 - Product Image

Artificial Intelligence (AI) Market by Technology (Machine Learning, Natural Language Processing (NLP), Image Processing, and Speech Recognition), Application & Geography - Global Forecast to 2020

  • Report
  • 151 Pages
  • Region: Global
Global Digital Security Market 2016-2020 - Product Image

Global Digital Security Market 2016-2020

  • Report
  • Region: Global