How do you monitor “East — West” Network Traffic?
If your organization does not have a strategy for monitoring network communications between each of the network nodes, you are potentially missing a wide variety of malicious lateral movement and not collecting forensics which could be analyzed after an attack. In this post I will examine what east-west traffic monitoring means, how various organizations are dealing with this problem.
What are East-West traffic flows?
Over the past decade the terms “east-west” and “north-south” emerged to describe network flows that were within an enclave and those that crossed the boundary of an enclave.
Consider a basic firewall protecting 100 systems in an office. There is only one firewall and monitoring it means collecting logs from that firewall which includes network connection activity. We could even deploy a network IDS/IPS or some sort of network security monitoring solution to collect packets at that point for more analysis.
Basic Logging and Netflow Collection
However, if we want to collect traffic flowing between each of the endpoints, there is no single place to easily plug in and get the data. Instead, there is likely a combination of multiple switches, multiple wireless access points, potentially multiple VLANs and possibly a set of security controls such as additional firewalls, network access control systems and dual homed servers.
Because of these complexities, there are a variety of approaches, each with pros and cons we will discuss.
Before you spend any money and time adding more security to your network, you should consider what you can get for free based on included features in your endpoints and your network gear.
Deploying a Dedicated Monitoring Network
If you can collect system logs from your endpoints, you can make an argument that system logs have higher fidelity than doing actual packet capture. Purists will point out that probes, lateral movement attempts and many attacks don’t show up in OS logs. You can counter by offering to configure the basic firewall that comes with your OS and then looking for denied connection attempts between individual hosts. In the same manner, if your switches and routers also have the ability to collect NetFlow, its very possible to set up monitoring which looks for east-west communications.
Many organizations do not do this for a variety of reasons. The top reason is complexity. Enabling logging, collecting it to a log storage and search place and doing the analytics on it is a messy and non-trivial exercise, especially at scale and across multiple organizations. Another is performance. Enabling robust network endpoint monitoring with firewall logs generates a lot of logs. This creates performance impacts and potential IT tickets when disk drives fill up. Enabling NetFlow on core switches and routers can also impact network performance.
Many organizations opt to deploy a secondary network to perform packet collection and analysis. In a network made up of routers and switches, you can only get packets to your collection point from a few places — inline proxies, taps and span ports. Rather than deploying one network IDS or packet vault for each tap, the “sniffed” network traffic is collected with one or more switches to an aggregator and monitored with a single solution. Commercial solutions exist to help with this such as GigaMon.
However, with east-west traffic, this approach has a hard time scaling due to math. If my network had three 100 Mb connections to the Internet, you could imagine aggregating this too one point over a 1Gb network, even if each link had 100 Mb up and 100 Mb down. As long as my packet collection solution could keep up with the 600 Mb/s we are good. Internally though, the packet rates for east-west traffic can be much higher. File sharing, internal email, chat, database, and many other applications which don’t exit out of the egress point run as fast as the internal switch can handle. For 100 Mb networks, the aggregate data of the switch can be in the multiple gigabits. It can also cause performance issues. Switches are designed to move packets from one port to another as fast as possible, not copy them all to one side.
Because of these limitations, I’ve seen organizations choose to only span or tap the up links on the switch — If you have multiple switches, you may find it more palatable to only monitor traffic on packets to or from the core. You will miss port to port attacks here, but will have more visibility than just looking at egress for the entire network.
Another approach is to deploy more firewalls. Firewalls are great at enforcing access, but can also be very good at logging where connections are originating from and where they are going.
A new approach from a Gula Tech Adventures portfolio company called CryptoniteNXT offers organizations the ability to limit east-west traffic and monitor it simultaneously. Their solution works with existing switching networks, but adds a device that white lists all traffic at the switching layer preventing recon and lateral movement. The hardware device also has enough horsepower to give both a user annotated log each network connection as well as a full packet span port. Below is an example network diagram from their “How it works” page.
Deploying Dedicated Agents
For OS X, Windows and Linux, a wide variety of software can be added to endpoints to collect network connectivity, network traffic and many other types of network forensics. Ever since the late 1990s, solutions like Network ICE ran full network IDS/IPS algorithms on the endpoints. Today there are a wide range of realtime, query based, open source and commercial solutions such as Carbon Black, Tanium, Tenable, OS Query, running SysMon on Windows systems and much more.
In large numbers of endpoints, running agents at scale can be a struggle. The more data that is collected, the more it has to be moved and stored some place. Agents need to exist within the environment of pre-existing anti-virus, patch management, backup, authentication that are already installed. The SIM, orchestration tool, data-lake, cloud SOC or log aggregation tool needs to be able to collect this volume of data and be able to ingest it, store it and search it later.
What about home users?
I’m seeing a variety of solutions come into play with the desire to monitor home user traffic.
- Assume the user is going to Salesforce, O365, Box, .etc and then monitor their behavior with the audit trails from those places. For example, as an investor in Eastwind Networks, I use it to monitor my office network and all the telemetry from my Dropbox and Gmail cloud activity.
- Force the remote users to use a known DNS server and monitor that. This was made popular with OpenDNS (now part of Cisco). With one DNS entry, you can observe a great deal of remote user’s activity. This is completely by-passable by malware and applications that use their own DNS server but many companies are very happy with this level of monitoring.
- Force a captive connection and pull traffic back through your corporate stack. Many companies also have a policy where the remote device is captive. Through the use of a VPN or a dedicated wireless provider tunnel, all remote traffic gets pulled back to the corporate network for inspection. Performance can be an issue here as the latency of this solution could be less than desirable.
Unless you have specifically engineered a solution to monitor east-west network traffic, you likely have a variety of blind spots where adversaries and malicious insiders could be hiding. If you have engineered a solution to monitor this type of traffic, it is likely complex enough to warrant monitoring on its own. I’ve sold a wide variety of passive network analysis solutions over the years and amount of organizations who realized collection was broken as they went to install a new solution was stunningly high.
If you do deploy this type of monitoring you will be able to have actual network forensics, be able to detect and respond to incidents better and IT will also able to troubleshoot problems easier.