By Zane Blomgren, Belden
Periodic active network scanning is generally essential to maintain an accurate picture of the network, as significant information is only available upon request and otherwise never present in normal traffic.
The reason for passive traffic analysis is that many industrial control systems (ICS) devices were really only designed to function as expected and are often not tested to maintain function when they receive traffic other than designed. For example, I’ve seen RFID controllers and Anybus modules lock up, simply by receiving packets that confused them.
Active network scanning can deliver much more information than passive scanning, and can be an incredibly valuable tool in any industrial environment. However, as I noted in a previous blog post, devices in the industrial environment – including VFD s , PLCs, I/O blocks, actuators and sensors – can be more sensitive than those in the office environment.
Standard IT methods for network scanning cannot be used in an industrial environment without planning and forethought.
Precautions need to be taken; for example, scanning should be done delicately and while machines are not operating due to the potential that the added traffic could add latencies and other issues. Yes, some machines could run just fine during an active scan, but it requires an additional study to verify that.
Operators can get all of the benefits of an active scan without concerns for their network by incorporating the proper proactive, responsible and knowledgeable planning before performing an active scan. Usually.
I say “usually” because my colleagues and I were recently involved in a situation that did not go 100% as planned. However, it had a fascinating outcome.
Example: Active network scanning achieved through partnership
We were brought in to perform an active scan on a network operated by a large automotive manufacturer. This client is very sophisticated and we have a close partnership with them.
They knew just what they wanted – an active network scan that would quickly and efficiently go deep into their network to identify every device and provide extremely detailed information. Their goal was to receive rich data about all of their systems along with thorough analysis and recommendations.
Through their relationship with Belden, the client chose to work with Tripwire specifically due to our experience and knowledge in this space. We scheduled the scan activity during a period that the line was down for scheduled maintenance. The scan was designed to be performed in a slow, gentle manner throughout the network.
We expected the impact on the network to be the equivalent of a light breeze. Yet, it was soon reported that a VFD tripped. Of course, we were concerned. Could our scan have caused this? Fortunately, everyone involved looked at the situation objectively and quickly concluded that the extraordinarily gentle scan could not – or should not – have caused the failure.
As it turns out, the source was an existing, hidden vulnerability in the VFD that could have been triggered by a multitude of disruptive situations, including a broadcast storm or a series of malformed packets. It was incredibly fortunate that it was triggered harmlessly in this circumstance – if it had been triggered while the line was up, it could have been a serious issue, potentially shutting down production.
Our analysis confirmed for the client where the issue was stemming from and they approached the VFD manufacturer. To their credit, the manufacturer was grateful for the knowledge and agreed that what happened during our scanning activity should not have occurred. They tested the VFDs in their labs and addressed the issues, proactively correcting other reliability issues with modifications and firmware.
The end result is a better product and more reliable operation for all.
What would you do?
The reason I am discussing this situation is that active network scanning still has a bad reputation due to situations where IT professionals have applied the active monitoring methods common in the office environment without adapting to the sensitive nature of the OT environment, causing adverse device interactions.
With this reputation still lingering, it could have been natural for the automotive network operator to assume that when the VFD tripped, it was the result of a poorly executed active scan. Fortunately, they were involved in and knowledgeable about the very careful precautions taken to alleviate any potential negative impacts while getting all the benefits of active network scanning.
With open eyes, they investigated the situation and accurately identified the source of the issue. And that was the first step towards solving the problem and ensuring that it doesn’t happen again. I must also give credit to the drive manufacturer, who took the opportunity to address the situation and improve further upon a quality product. Truth is, this could have been a story where everyone was angry and finger pointing. But instead, it was a partnership where everyone looked in the right direction and ultimately benefited from the situation.
So I’d like to ask—if this happened in your facility, what would you have done? Would you have jumped to conclusions, or investigated the situation? I welcome a dialog.
Related Links
Check out our free e-newsletters
to read more great articles.
By Zane Blomgren, Belden
Periodic active network scanning is generally essential to maintain an accurate picture of the network, as significant information is only available upon request and otherwise never present in normal traffic.
The reason for passive traffic analysis is that many industrial control systems (ICS) devices were really only designed to function as expected and are often not tested to maintain function when they receive traffic other than designed. For example, I’ve seen RFID controllers and Anybus modules lock up, simply by receiving packets that confused them.
Active network scanning can deliver much more information than passive scanning, and can be an incredibly valuable tool in any industrial environment. However, as I noted in a previous blog post, devices in the industrial environment – including VFD s , PLCs, I/O blocks, actuators and sensors – can be more sensitive than those in the office environment.
Standard IT methods for network scanning cannot be used in an industrial environment without planning and forethought.
Precautions need to be taken; for example, scanning should be done delicately and while machines are not operating due to the potential that the added traffic could add latencies and other issues. Yes, some machines could run just fine during an active scan, but it requires an additional study to verify that.
Operators can get all of the benefits of an active scan without concerns for their network by incorporating the proper proactive, responsible and knowledgeable planning before performing an active scan. Usually.
I say “usually” because my colleagues and I were recently involved in a situation that did not go 100% as planned. However, it had a fascinating outcome.
Example: Active network scanning achieved through partnership
We were brought in to perform an active scan on a network operated by a large automotive manufacturer. This client is very sophisticated and we have a close partnership with them.
They knew just what they wanted – an active network scan that would quickly and efficiently go deep into their network to identify every device and provide extremely detailed information. Their goal was to receive rich data about all of their systems along with thorough analysis and recommendations.
Through their relationship with Belden, the client chose to work with Tripwire specifically due to our experience and knowledge in this space. We scheduled the scan activity during a period that the line was down for scheduled maintenance. The scan was designed to be performed in a slow, gentle manner throughout the network.
We expected the impact on the network to be the equivalent of a light breeze. Yet, it was soon reported that a VFD tripped. Of course, we were concerned. Could our scan have caused this? Fortunately, everyone involved looked at the situation objectively and quickly concluded that the extraordinarily gentle scan could not – or should not – have caused the failure.
As it turns out, the source was an existing, hidden vulnerability in the VFD that could have been triggered by a multitude of disruptive situations, including a broadcast storm or a series of malformed packets. It was incredibly fortunate that it was triggered harmlessly in this circumstance – if it had been triggered while the line was up, it could have been a serious issue, potentially shutting down production.
Our analysis confirmed for the client where the issue was stemming from and they approached the VFD manufacturer. To their credit, the manufacturer was grateful for the knowledge and agreed that what happened during our scanning activity should not have occurred. They tested the VFDs in their labs and addressed the issues, proactively correcting other reliability issues with modifications and firmware.
The end result is a better product and more reliable operation for all.
What would you do?
The reason I am discussing this situation is that active network scanning still has a bad reputation due to situations where IT professionals have applied the active monitoring methods common in the office environment without adapting to the sensitive nature of the OT environment, causing adverse device interactions.
With this reputation still lingering, it could have been natural for the automotive network operator to assume that when the VFD tripped, it was the result of a poorly executed active scan. Fortunately, they were involved in and knowledgeable about the very careful precautions taken to alleviate any potential negative impacts while getting all the benefits of active network scanning.
With open eyes, they investigated the situation and accurately identified the source of the issue. And that was the first step towards solving the problem and ensuring that it doesn’t happen again. I must also give credit to the drive manufacturer, who took the opportunity to address the situation and improve further upon a quality product. Truth is, this could have been a story where everyone was angry and finger pointing. But instead, it was a partnership where everyone looked in the right direction and ultimately benefited from the situation.
So I’d like to ask—if this happened in your facility, what would you have done? Would you have jumped to conclusions, or investigated the situation? I welcome a dialog.
Related Links
Check out our free e-newsletters
to read more great articles.
© 2019, Best Brothers Group. All rights reserved.
