Skip to content

Pattern Recognition

Threat hunting in Microsoft Sentinel often requires identifying subtle patterns in log data that indicate adversarial activity. By leveraging KQL (Kusto Query Language), defenders can detect anomalies, correlate events, and identify indicators of compromise (IoCs) that align with known threat intelligence. This section explores techniques for pattern recognition, focusing on both signature-based detection and behavioral analysis.

Identifying Anomalies via Baseline Deviation

Anomalies often emerge when system behavior deviates from established baselines. For example, unexpected network traffic or unusual user activity can signal a compromise. Use KQL to calculate baselines and flag outliers.

Example: Detecting abnormal outbound traffic

// Baseline: Average number of outbound connections per host  
OutboundConnections
| summarize avgConnections = avg(numConnections) by host  
| extend baseline = avgConnections  

// Anomaly detection: Hosts with > 2x baseline connections  
OutboundConnections
| join kind=inner (baseline) on host
| where numConnections > 2 * baseline
| project host, numConnections, baseline

Example: Flagging unusual login times

// Baseline: Average login time of users  
UserLogins
| summarize avgLoginTime = avg(todatetime(loginTime)) by userAccount  
| extend baseline = avgLoginTime  

// Anomaly detection: Logins outside typical hours (e.g., 2 AM)  
UserLogins
| join kind=inner (baseline) on userAccount
| where loginTime > baseline - 1h and loginTime < baseline + 1h
| where loginTime < datetime(2023-01-01T02:00:00) or loginTime > datetime(2023-01-01T06:00:00)
| project userAccount, loginTime, baseline

Detecting IoCs with Signature-Based Patterns

Signature-based detection involves matching log data to known IoCs, such as hashes, IP addresses, or process executions. Use where clauses and join operations to correlate events against threat intelligence feeds.

Example: Matching suspicious hashes to process executions

// Known malicious hashes (example list)  
KnownHashes
| where hash in ("5f4dcc3b5aa765d61d8327deb882cf99", "d41d8cd98f00b204e9800998ecf8427e")  

// Correlate with process execution logs  
ProcessExecutions
| join kind=inner (KnownHashes) on hash
| project processName, hash, timestamp

Example: Detecting command-line arguments matching IoCs

// Search for suspicious command-line arguments  
SecurityEvents
| where EventID == 4688 and (CommandLine contains "cmd.exe" or CommandLine contains "powershell.exe")
| where CommandLine contains " -encodedcommand" or CommandLine contains "Invoke-Command"
| project EventID, Timestamp, Computer, CommandLine

Temporal and Spatial Correlation

Threat actors often exhibit patterns across time and systems. Use timeslice and mv-expand to analyze sequences of events or cross-system activity.

Example: Detecting multi-stage attacks via time-based correlation

// Identify sequences of events within a 10-minute window  
SecurityEvents
| where EventID in (4688, 4689, 4672)
| project EventID, Timestamp, Computer, ProcessName, User
| sort by Timestamp  
| timeslice 10m  
| summarize EventIDs = make_set(EventID), Computers = make_set(Computer), Users = make_set(User) by bin(Timestamp, 10m)
| where EventIDs has 4688 and EventIDs has 4689

Example: Cross-system lateral movement

// Correlate logins across systems to identify lateral movement  
UserLogins
| where EventID == 4624
| project Computer, User, Timestamp
| join kind=inner (UserLogins) on User
| where Computer != targetComputer
| project sourceComputer = Computer, targetComputer = Computer2, Timestamp

Custom Pattern Recognition with KQL Functions

For complex patterns, combine KQL functions like mv-expand, summarize, and parse to extract and analyze structured data.

Example: Detecting PowerShell script execution with encoded commands

// Extract and analyze PowerShell command-line arguments  
SecurityEvents
| where EventID == 4688 and CommandLine contains "powershell.exe"
| parse kind=regex CommandLine with 
    "powershell.exe" 
    ".*?-Command\"?\"?([^\"\\)]+)\"?\"?" 
    ".*?-EncodedCommand\"?\"?([^\"\\)]+)\"?\"?" 
    ".*?-File\"?\"?([^\"\\)]+)\"?\"?" 
| mv-expand Command, EncodedCommand, File
| where EncodedCommand != "" or File != ""
| project Computer, Timestamp, CommandLine, Command, EncodedCommand, File

Key takeaways

  • Use baselines and statistical analysis to identify anomalies in log data.
  • Leverage signature-based queries to match IoCs against known threat intelligence.
  • Combine temporal and spatial correlation to detect multi-stage attacks.
  • Customize KQL with advanced functions to analyze complex patterns in structured logs.
  • Iterate and refine queries based on false positives and evolving threat tactics.