What Is an Intrusion Prevention System (IPS)?

What Is an Intrusion Prevention System (IPS)?

An intrusion prevention system (IPS) detects intrusion attempts and malicious activity on a network or system and blocks the associated traffic based on predefined security policies.

An IPS is commonly deployed inline in the traffic between a protected asset and the systems communicating with it, making traffic pass through the IPS inspection engine. It enforces preventive action by dropping malicious packets or immediately terminating the affected session.

An IPS inspects source and destination addresses, packet headers, connection state, application protocol structure, payloads for attack indicators, session behavior, and communication frequency. These capabilities detect exploit attempts against public-facing servers, malformed protocol traffic, connections from compromised endpoints to C2 infrastructure, and internal network scanning. When traffic is classified as malicious, the IPS drops packets, resets sessions, or blocks the source. In an integrated security architecture, it can also instruct other controls to update or enforce policy.

Why IPS Is Important

A traditional firewall enforces access control based on network address, port, protocol, and connection status. Adversaries, however, routinely deliver attacks over approved ports and legitimate protocols. This makes it difficult for firewalls to fully distinguish a valid application transaction from a malicious request carried over the same connection.

An IPS performs deeper inspection of traffic. It identifies attack patterns and anomalous behavior within approved sessions and interrupts malicious communication immediately after detection, limiting exploit execution, malware delivery, privilege escalation, and lateral movement.

For systems that cannot be patched immediately, an IPS also works as a virtual patching control. It blocks traffic patterns associated with a specific vulnerability, or invalid protocol behavior used to trigger that vulnerability, before the traffic reaches the target system. This reduces exposure until the organization can deploy patches.

IPS events are also important sources of security telemetry. They show which vulnerabilities are targeted repeatedly and which network segments receive concentrated attack activity. Correlating IPS events with firewall, authentication, and endpoint logs in a Security Information and Event Management (SIEM) or security monitoring platform helps analysts prioritize vulnerability remediation and assess the scope of a compromise.

How an IPS Analyzes and Blocks Traffic

When an IPS receives traffic, it reconstructs the session, decodes the protocol, and evaluates the resulting communication for malicious activity. If the traffic meets the criteria for an attack, the IPS applies the configured prevention policy and records the detection evidence and enforcement action.

The inspection process begins by identifying traffic direction and tracking connection state. The IPS reassembles IP fragments and data streams divided across multiple packets, then inspects the content for malicious payloads concealed within otherwise legitimate looking traffic. Accurate handling of packet order, retransmissions, and overlapping data is critical as reconstruction errors either allow attacks to evade inspection or cause legitimate traffic to be classified as malicious.

During inspection, the IPS normalizes traffic to reduce variations that can be used to evade detection. The same content can appear in different forms due to encoding, fragmentation, or nonstandard transmission methods. Normalization converts these variations into a consistent form and reduces discrepancies during inspection.

Protocol decoders then interpret application-layer traffic such as HTTP, DNS, SMB, and database protocols. The IPS examines request and response structures, field values, commands, and payload content, while identifying the protocol and application from observed traffic rather than relying only on port assignments. It compares anomalous requests, invalid commands, and known attack indicators against detection rules and behavioral models to determine whether the communication constitutes a threat.

The IPS applies a predefined response policy when traffic matches a detection rule or behavioral model. When the IPS detects clear exploit activity, it drops the affected packets or terminates the session. Events that closely resemble legitimate business traffic, or where a false positive would have a significant operational impact, can remain in alert-only mode. Organizations can assign different enforcement levels to the same threat category based on asset criticality, service characteristics, traffic direction, and the probability of successful exploitation.

The IPS applies different enforcement actions depending on the detected threat and the type of traffic involved. The most direct action is to withhold the malicious packet from its destination. For TCP traffic, the IPS can inject reset packets to tear down the session. It can also block subsequent traffic to or from a source for a defined period or extend enforcement through integration with firewalls and network access control systems. Blocking an IP address for an extended period can disrupt legitimate users, so block criteria and expiration periods must reflect operational risk.

IPS Threat Detection

IPS platforms use signature-based detection, anomaly and behavior-based detection, policy-based detection, and stateful protocol analysis. Effective deployment requires tuning each method to the protected assets and the production environment.

Signature-Based Detection

Signature-based detection converts recurring characteristics of known attacks into detection rules and searches network traffic for corresponding indicators. Rules inspect elements such as specific commands and strings, exploit-code byte sequences, invalid data lengths, combinations of protocol fields, and packet sequences associated with vulnerability exploitation. When traffic matches a signature, the IPS drops the malicious packet or terminates the connection. This method is effective for rapidly identifying threats with well-defined characteristics, including attacks against known vulnerabilities, malware delivery, and connections to known C2 infrastructure.

Anomaly and Behavior-Based Detection

Anomaly-based detection focuses on deviations from established communication patterns rather than fixed attack signatures. Using network activity observed over time or an administrator defined baseline, the IPS establishes normal connection frequency, port usage, transfer volume, communication time, session duration, and destinations. When anomalous activity occurs, the IPS evaluates how much the current traffic deviates from the expected pattern. This approach supports the detection of attacks that do not precisely match known signatures, as well as lateral movement, automated scanning, and unusual outbound communication.

For example, an endpoint that connects to multiple ports across several internal servers within a short period indicates possible internal reconnaissance or malware propagation. Small transfers sent at regular intervals to a previously unseen external address are characteristic of C2 beaconing. A business server transferring a large volume of data externally late at night can indicate data exfiltration.

Policy Based Detection

Policy based detection identifies network activity that violates an organization's defined security policies. Detection rules can use conditions such as source, destination, user, device, service, time, and traffic direction to determine whether communication is permitted.

For example, an organization can allow a database server to accept requests only from designated application servers while blocking direct access from general user devices. It can also restrict critical servers from transferring files to unauthorized external services or communicating with destinations that are not required for business operations. Effective policy based detection depends on accurate asset information, clearly defined authorized communication flows, and regular policy updates as systems and services change.

Stateful Protocol Analysis

Stateful protocol analysis determines whether network communication follows the expected structure and sequence of a protocol. The IPS tracks session state and analyzes request and response relationships, command order, field lengths, permitted values, and transaction sequences across the communication flow.

Attackers may attempt to exploit vulnerabilities by sending abnormally long values, issuing commands that are invalid for the current protocol state, or manipulating protocol fields. By reconstructing transactions across multiple packets, the IPS can identify protocol violations and invalid state transitions. Signature based detection searches for known patterns of specific attacks, while stateful protocol analysis evaluates whether the protocol exchange follows expected behavior.

IPS Types and Deployment

IPS controls can be deployed across network, host, and wireless environments according to the protected assets and traffic types. Each model provides a distinct visibility scope and operating method, so deployment planning must begin with actual traffic flows and asset characteristics.

Network-Based IPS

A network-based IPS (NIPS) inspects packets and sessions traversing a network segment. It combines signature matching, protocol-state validation, and traffic-behavior analysis to detect and block vulnerability exploitation, malware communication, scanning, and related network threats.

Common deployment points include the internet perimeter, the ingress to a data center, boundaries between user and server networks, and critical internal segments. Inspection at these points provides visibility into inbound north-south traffic and selected east-west communication between internal systems. A NIPS has no visibility into traffic that does not go through its inspection point. In cloud and multi-branch environments, security architects must therefore map actual traffic paths before selecting deployment locations.

Host-Based IPS

A host based IPS (HIPS) runs on an individual server or endpoint and monitors activity within that host. It can analyze network activity for that host, system logs, running processes, file modifications, and application behavior. Organizations use HIPS to prevent malicious code execution, unauthorized file changes, privilege escalation attempts, and network communication initiated by unapproved applications on critical servers and business endpoints.

Because HIPS operates directly on the host, it can monitor system and application activity that network based security controls may not be able to observe. NIPS provides broad visibility across traffic for multiple systems. HIPS provides granular visibility into the internal activity of a single system. Deploying both on critical assets provides complementary enforcement at the network and host levels.

Wireless Intrusion Prevention System

A wireless intrusion prevention system (WIPS) continuously monitors the radio frequency environment and connection state of wireless networks. Its primary functions are to identify unauthorized access points, unapproved client devices, insecure wireless configurations, and anomalous association attempts. Dedicated sensors or managed wireless access points collect nearby signals and compare discovered devices against the authorized asset inventory. WIPS alerts administrators when it detects an access point impersonating the corporate network, an unauthorized personal router, or repeated authentication attacks.

Depending on product capabilities and policy, WIPS also blocks suspicious wireless connections or supports device location tracking. Wireless containment can disrupt nearby communications, so organizations must validate the action against internal security policy and applicable legal or regulatory requirements before enabling it.

Differences Between IPS, IDS, Firewalls, WAF, and NDR

Technology	Primary Objective	Inspection Scope	Response
IPS	Prevent attacks and intrusion attempts in real time	Packets, sessions, protocols, payload indicators, and traffic behavior	Drop packets, terminate sessions, and block sources
IDS	Detect suspicious activity and generate alerts	Mirrored network traffic or host events	Generate alerts and logs for investigation
Firewall	Enforce network access control	Addresses, ports, protocols, and connection state	Permit or deny traffic according to policy
WAF	Protect web applications	HTTP and HTTPS requests and responses within web-application context	Block malicious web requests and control sessions
NDR	Detect and investigate anomalous network behavior	Traffic flows, metadata, and behavioral relationships	Generate alerts, support investigation, and orchestrate other controls

A traditional firewall determines whether communication is permitted based on addresses, ports, protocols, connection state, and related policy conditions. An IPS examines traffic that has already passed those access control checks and blocks attack patterns or invalid protocol behavior within the permitted communication. The firewall defines which connections are allowed. The IPS evaluates whether the content and behavior of those connections are malicious.

IDS and IPS share the objective of detecting suspicious network or system activity, but their enforcement models differ. An IDS generally analyzes mirrored traffic or host events and generates alerts without affecting the production data path. An IPS processes live traffic inline and therefore drops malicious packets or terminates sessions as the attack occurs.

A web application firewall (WAF) specializes in protecting web applications. It inspects HTTP and HTTPS traffic at the application layer, including URLs, headers, parameters, cookies, and message bodies, to detect and block attacks such as SQL injection and cross site scripting (XSS). A NIPS can detect some web attacks, while a WAF provides more specialized inspection of web application traffic using HTTP and application specific rules.

Network detection and response (NDR) analyzes traffic flows and communication relationships over time to identify anomalous behavior and indicators of compromise. An IPS focuses on preventing an attack as it traverses the enforcement point. NDR correlates observations across systems and time periods to reconstruct threat context and movement through the environment. In an integrated architecture, NDR can enrich and investigate IPS events, while threat intelligence derived from NDR analysis can inform IPS policy.

Prevention Scenarios Across the Attack Flow

IPS detection and enforcement vary according to whether an attack originates externally, propagates within the internal network, or involves outbound communication from a compromised system.

Exploitation of an Internet Facing Server

Attackers use legitimate HTTPS connections to deliver requests that exploit web server vulnerabilities. Port and connection metadata alone cannot distinguish a valid HTTPS transaction from a malicious request. Where the security architecture provides decrypted HTTPS traffic to the IPS, it inspects request content, identifies known exploit patterns, and blocks the malicious session. The resulting event also enables the security team to verify the target's patch status and search for related attempts against other systems.

Lateral Movement from a Compromised Endpoint

An endpoint compromised through phishing or another initial access technique can scan neighboring systems or repeatedly exploit vulnerabilities in file sharing protocols. An IPS positioned in the internal traffic path identifies rapid connections to multiple addresses and associated exploit patterns, then terminates the sessions. If internal communication bypasses the IPS, or systems within the same network segment communicate directly, this activity remains outside IPS visibility. Network segmentation, deliberate traffic path design, and complementary security controls reduce these monitoring gaps.

Outbound C2 Communication

An IPS uses integrated threat intelligence, protocol indicators, and recurring beacon patterns to detect or block connections to C2 infrastructure. A confirmed C2 event requires investigation of the internal system for compromise. After blocking the connection, the security team should isolate the endpoint and use endpoint detection and response (EDR) telemetry to investigate malicious processes and execution artifacts. Related accounts and indicators of compromise on other systems must also be reviewed.

Traffic visibility is the principal condition governing prevention effectiveness in all three scenarios. Detection coverage degrades when traffic bypasses the IPS or when encryption prevents payload inspection. IPS placement, traffic routing, and the scope of encrypted traffic inspection must therefore be designed together during implementation, followed by detection and enforcement policies calibrated to the production environment.

IPS Limitations and Operational Considerations

An IPS does not identify every threat with complete accuracy. Stable operation requires controls for false positives, false negatives, performance degradation, device failure, and encrypted traffic, as well as operating standards that compensate for these limitations.

Managing False Positives and False Negatives

IPS policy must balance attack prevention against the risk of blocking legitimate traffic. Detection criteria that are too broad or poorly aligned with actual business flows generate false positives. Conversely, the IPS misses attacks when signatures do not cover a new technique or when encryption and network routing prevent adequate inspection.

Security teams should continuously tune detection policies against the operating systems, applications, vulnerabilities, and observed traffic characteristics of protected assets. A new signature should enter blocking mode only after its business impact has been assessed. Exceptions created for false positives should apply only to the minimum required scope. Regular review of signature currency and traffic visibility prevents persistent detection gaps.

Measuring Performance Under Production Conditions

Maximum throughput is not a sufficient measure of IPS performance. Effective capacity depends on concurrent session count, new connection rate, packet size, the number of enabled signatures, application control functions, and TLS decryption. When multiple security functions operate concurrently, production throughput is often lower than the vendor's stated maximum.

Capacity planning must account for average load, peak business period traffic, and projected growth. After deployment, operators should monitor throughput, latency, session count, packet loss, and CPU and memory utilization. As the platform approaches its capacity limit, inspection latency increases and packets can be dropped. Organizations should define scaling and traffic distribution thresholds before these conditions affect service.

Traffic Handling During an IPS Failure

Because an inline IPS processes all traffic on its network path, hardware failure, power loss, or software malfunction can affect every connection in that segment. The organization must define in advance whether traffic will continue to pass or be blocked during an IPS failure.

Fail open behavior preserves service availability but creates an inspection gap until the IPS is restored. Fail closed behavior maintains enforcement by blocking uninspected traffic, but it can interrupt legitimate business services. Internet access paths and critical server segments have different asset sensitivities and availability requirements and therefore require different failure policies. High availability pairs, bypass paths, failure alerts, and documented manual failover procedures should form part of the deployment design.

Encrypted Traffic and TLS Decryption

Without TLS decryption, an IPS cannot directly inspect request and response payloads. This limits its ability to detect exploits and malicious commands embedded in encrypted content. TLS decryption restores payload visibility but exposes sensitive personal data and authentication information to the inspection platform. Deployment must therefore comply with privacy policy and applicable legal and regulatory requirements.

Organizations need explicit criteria for traffic excluded from decryption for business or policy reasons. The design must also account for communications that restrict intermediary decryption through certificate pinning or mutual TLS. Where the IPS lacks payload visibility, endpoint controls and application or server logs provide compensating visibility.

Security Activity Outside IPS Visibility

An IPS does not provide direct visibility into malicious processes already executing on an endpoint, host file modifications, misuse of cloud administrative accounts, or privilege abuse through legitimate credentials. It also cannot inspect communication within the same network segment when the traffic does not traverse the IPS.

Complementary controls must address these gaps. EDR monitors endpoint process and file activity, while strong authentication and least privilege access reduce account misuse risk. Correlating cloud audit logs, authentication logs, NDR data, and SIEM records with IPS events reveals attack sequences that no single control can observe independently.

Criteria for Effective IPS Adoption and Operation

Before selecting an IPS product, an organization must define its protected assets and security objectives. Planning should identify critical assets and services, relevant attack types, actual traffic paths, encrypted segments, peak traffic volume, and availability requirements. These inputs determine appropriate enforcement points, processing capacity, high availability architecture, and policy scope.

Detection policy must reflect the operating systems, applications, known vulnerabilities, and asset criticality of the protected environment. Enabling irrelevant signatures indiscriminately in blocking mode increases false positives and operational overhead. New signatures require a business impact assessment before enforcement. Every exception should record the affected asset, justification, scope, and expiration date. Organizations also need an expedited process for critical vulnerabilities that accounts for asset importance and exploitability.

Operational reviews should cover signature and engine update status, policy change history, and active exceptions. Integration with SIEM, security orchestration, automation, and response (SOAR), and EDR platforms enables analysts to correlate network detections with other telemetry and initiate follow up response. Automated blocking and endpoint isolation directly affect business operations, so detection confidence and enforcement scope must be validated before these actions are enabled.

Block count alone does not demonstrate IPS effectiveness. Meaningful evaluation determines whether blocked events represented actual attacks, measures the operational impact of false positives, and identifies traffic that escaped inspection. Platform latency and availability are also core performance indicators.

Article

Securing the Future: Our Next Generation Network Security

FAQ

AhnLab IPS Solution

AhnLab AIPS is a next generation network intrusion prevention system that detects and blocks advanced network attacks targeting organizations. It uses packet inspection, signature and behavior based detection, and malware detection to monitor network traffic and malicious activities. AhnLab AIPS thoroughly inspects packet headers and payloads and detects vulnerability based attacks through signature pattern matching. It also provides SSL traffic decryption for detecting threats in encrypted traffic, along with Application Control and Zone Settings for applying different security policies and controls.