List members: I've read numerous messages over the last couple weeks about people plugging their PLCs and associated doo-dads into the Internet, and I have a question. Have you guys ever thought about: 1) Security Holes 2) Vulnerability to denial-of-service attacks -- Security Holes Most all software that has been used on the internet does indeed have some security holes. In the case of open-source software, like Apache, bugs are fixed by people looking at the code and going "Hey! That's not right." In the case of closed-source software, like Microsoft's IIS, bugs are fixed by people telling Microsoft, and them fixing it. Given that we don't have access to the code running in these web modules or what have you attached to the Internet, we're all stuck in the "hope no one finds any bugs" mode, and after that the "hope no one finds my device" mode. There's a risk there, but I would say that there's such a miniscule amount of these devices on the web, that the chance of one being hacked into is minimal, because no one will bother. -- Vulnerability to denial-of-service attacks This is one that I think is much more critical, because I actually managed to do this. A motion controller from a company that shall remain nameless has an ethernet adapter, and TCP/IP support. Being the inquisitive type that I am, I fired up a linux box, and port scanned the motion controller using nmap. The result: The motion controller's interface hung. It stopped. I couldn't ping the thing anymore. I couldn't access the thing through the serial port anymore either. I didn't have any motors plugged in, so I couldn't tell whether or not that part was still functioning. I contacted the company the makes the controller, told them what I did, and their response was... "We don't support Linux" This was despite the fact that Linux wasn't the problem... the port scan that hit the ports on the controller crashed it. I can get port scanners for every OS, and its not uncommon for machines on the internet to be portscanned. My dynamic dialup connection gets port scanned on a regular basis! So now the questions are... What type of testing are you any of you guys doing on devices that you're hooking up to the internet? What type of testing does the manufacturer of the equipment do? Do they realize that plugging a device into the Internet means that its going to have to have a much more robust communication stack than before? I would be extremely interested to hear your thoughts on this matter. Alex Pavloff Software Engineer Eason Technology

 

> What type of testing are you any of you guys doing on devices that you're hooking up to the internet? What type of testing does the manufacturer of the equipment do? Do they realize that plugging a device into the Internet > means that its going to have to have a much more robust communication stack than before? I am not connecting anything to the internet. People I work with typically turn white at the thought of connecting their control ethernet to a business ethernet. IP-ready devices should hardly be called internet-ready. I believe they should be carefully nursed by experienced systems administrators and control techs, who understand the needs and vulnerabilities, to properly segment and firewall the networks. > port scanned the motion controller using nmap. > > The result: The motion controller's interface hung. It stopped. I couldn't ping the thing anymore. I couldn't access the thing through the serial port anymore either. I didn't have any motors plugged in, so I couldn't tell whether or not that part was still functioning. > > I contacted the company the makes the controller, told them what I did, and their response was... > > "We don't support Linux" And the vendors must learn to be sympathetic to this engineering effort. They should be willing to provide support for their products to implementors who talk about (and practice) intrusion detection, penetration testing, and of course, firewall configuration. BTW did your instrument crash so badly it needed to be locally power cycled - or worse? David, in a personal capacity [email protected]

 

> Have you guys ever thought about: > 1) Security Holes > 2) Vulnerability to denial-of-service attacks These problems can be overcome with a decent firewall. If you're looking for an open firewall solution, look no further than OpenBSD or Linux. While I am not familiar with BSD (I'm a Linux guy as you will shortly see), it is a more mature technology and claims to be more secure. I do a small bit of work around the <a href="http://www.uClinux.org">uClinux</a> (Microcontroller Linux) operating system which is a port of Linux to devices without Memory Management Units. The reason I mention this is that <a href="http://www.moretonbay.com">Moreton Bay</a>, an Australlian based company, has developed a standalone firewall using uClinux on a Motorola ColdFire processor. It will do dialup, DSL or cable and provide you with an efficient and configurable firewall to protect your network. There is also something called the <a href="http://www.linuxrouter.org">Linux Router Project</a>, which takes a single floppy disk and converts any PC to a Linux firewall. While I'm not affiliated with either of these projects, I feel that they both make excellent products. There are many other firewall options with Linux: it's just a matter of deciding what you want and how you want it. The second part of your concerns, denial of service attacks, are more difficult to control. A well-designed firewall can prevent a DoS from killing your control network but it cannot control the DoS from tearing apart your connection to the internet which will obviously interfere with monitoring and control of the devices behind the firewall. Things like TCP syncookies and rate limiting your responses helps, but if you've got 100MB worth of traffic coming at you through a 56k link you're toast no matter what you do. My personal opinion is that as long as you have a strong firewall set up and a seperate dialup line connected to a modem for emergencies, hooking up your control network to the internet isn't a bad idea at all. The key, as I've mentioned, is having a strong firewall. The modem comes in handy if your network connection is dead; you can simply call the modem and establish a PPP link or just a simple shell session to the firewall and act as if nothing were wrong. Regards, Andrew Kohlsmith

 

Hi Alex, The simple answer is that you would not (should not) permanently connect such a device directly to the Internet unless you were sure that it could withstand such attacks. The prefered solution would be to place the devices behind a well configured firewall which would filter out such denial of service attacks. For those interested in setting up such a device, have a look at http://www.ecst.csuchico.edu/~dranch/LINUX/index-linux.html#TrinityOS . Regards Peter Whalley [email protected] Managing Director Magenta Communications Pty Ltd 121 King Street, Melbourne, VIC 3000 Australia.

 

1. I don't know about other PLC's, but I expect that Modicon would provide (web downloadable) flash upgrades for their modules. 2. I don't think Ethernet modules were intended to get PLC's onto anything other than an intranet. In fact, if you look at past conversations on Ethernet in the plant, you would find recommendations against mixing plant and office equipment on the same network. Just because something is possible, does not mean it should be done. Anthony Kerstens P.Eng.

 

Anthony wrote: >1. I don't know about other PLC's, but I expect that Modicon >would provide (web downloadable) flash upgrades for their >modules. Yes, upgrades new features etc. etc. >2. I don't think Ethernet modules were intended to get PLC's >onto anything other than an intranet. In fact, if you look at >past conversations on Ethernet in the plant, you would find >recommendations against mixing plant and office equipment on >the same network. Exactly, all that I am saying is, you can connect direct to the PLC via HTTP to send/retrieve data in one form or another without requiring special interfaces. the OS on the module is VXworks, comms is TCPIP(UTP or fiber) everything is standard off the shelf. Rob E.

 

>---------- Forwarded message ---------- >From: Andrew Kohlsmith <[email protected]> >> Have you guys ever thought about: >> 1) Security Holes >> 2) Vulnerability to denial-of-service attacks >These problems can be overcome with a decent firewall. If you're looking >for an open firewall solution, look no further than OpenBSD or Linux. >While I am not familiar with BSD (I'm a Linux guy as you will shortly see), >it is a more mature technology and claims to be more secure. The few times I have placed an Ethernet enabled PLC (PLC-5 in the most recent case) on a network was via a dedicated PC running RSLinx. That way I had access to what I wanted but took advantage of the LAN and WAN firewalls. More importantly, it isolated all of my control traffic from the business network. To program or monitor the PLC we used Timbuktu - other than the slight jerkiness for mouse movements while operating through the LAN, it gave me all the capability of standing next to the local PC. It was password protected and had a exclusive hardware plus user account login (if we hadn't set-up an account, a casual user could try signing on even with a valid user ID and password to no avail if they were at a different PC). Our motivation was more to limit traffic than anything else as the LAN was an internal network with firewalls already in place to the Internet. However, it did limit any business network problems from impacting the controls. Russ Kinner AVCA Corporation Maumee, OH USA

 

Hi Bill & Automation list (I'm responding to a question from Bill Townsley and am posting the response to the list also) Some more info about port scanning: Each machine using TCP/IP has an IP address. On each IP address, there are multiple ports (1 through 65535, I think). Each port is a point for an outgoing connection or an incoming connection. For example, HTTP servers are generally on port 80, FTP servers on ports 20 and 21, etc etc. What a port scan is where, via one of a few methods, someone tries to figure out what ports are open to receive data. For example, I scan my NT4 server here at work with my Linux box, and I see: ---------- Starting nmap V. 2.53 by [email protected] ( www.insecure.org/nmap/ ) Interesting ports on nt4-server.eason.com (10.0.0.1): (The 1513 ports scanned but not shown below are in state: closed) Port State Service 21/tcp open ftp 25/tcp open smtp 27/tcp open nsw-fe 42/tcp open nameserver 80/tcp open http 135/tcp open loc-srv 139/tcp open netbios-ssn 443/tcp open https 593/tcp open http-rpc-epmap 2042/tcp open isis Nmap run completed -- 1 IP address (1 host up) scanned in 2 seconds ------------- This tells me that my server is running an ftp server, a web server, some mail server stuff, netbios (for windows networking), and a couple other miscellaneous things. Now, a port scan isn't inherently evil. Its the computer equivalent of walking around someone's house and counting the number of doors and writing down where they're at. However, one of the major uses for a port scan is for crackers to find out if your system has anything that they can attempt to break into. My Linux box, acting as a firewall (even on a dialup line!), gets many of these scans. I pulled these from my portsentry (which blocks people trying to do port scans) log file, and you can see people trying to scan ports 21,23 and 110. --- 980690554 - 01/28/2001 06:02:34 Host: 209.10.176.98/209.10.176.98 Port: 110 TCP Blocked 980697143 - 01/28/2001 07:52:23 Host: 216.218.52.225/216.218.52.225 Port: 23 TCP Blocked 980699624 - 01/28/2001 08:33:44 Host: lp.snaptek.com/207.246.52.40 Port: 21 TCP Blocked --- People are looking for specific ports that are running pieces of software that are known to have bugs that would allow them to gain access. So basically, the moment a computer is on the internet, there's going to be the computer equivalent of people constantly walking around and checking out the doors and windows. You've got to sit inside your house and make sure all the doors and windows are properly sealed so that people don't get inside. The main way to do that is a firewall. As I said, I've got a Linux server acting as the firewall between all the computers on my network here at work, and the internet in general. Everyone here at work can connect through the firewall to get out to the internet, but people outside on the internet can't get at my internal network. What I could do, if I so desired, is forward several ports on the Linux server to ports on a local machine, say if I wanted to allow ONLY web connections to my server, I could do that. People would connect to my Linux server on port 80 and it would just appear to be a web server. The other thing you can do is have machines still connected to the internet, but behind a firewall, so that only authorized traffic on certain ports get through. But when it comes to internet security, I think the main thing that you've got to have is a slightly-paranoid mindset and the understanding that just because something was secure yesterday... it might not be today. If you're still interested after my long pedantic ranting, you might want to read... http://www.w3.org/Security/Faq/www-security-faq.html There's lots of good info there. Cheers! (and if someone out there wants to correct me for a mistake, please do so!) Alex Pavloff Software Engineer Eason Technology > -----Original Message----- > From: Townsley, Bill [mailto:[email protected]] > Sent: Tuesday, January 30, 2001 9:02 AM > To: '[email protected]' > Alex, > > You bring up a couple of good points. My question (sorry for > being naive on the subject) - what exactly is port scanning? I was planning > to connect a control system to the internet to enable remote monitoring and limited > control. I can't do this yet because of limited availability > of high speed access. In fact the best speed I can get right now is ~28.8 using the > telephone lines. Cable and DSL are about two years away and > Satellite is too expensive for two-way communication. > > As for the port scanning, how can you block access to an external user > enabling or disabling the ports? > > Thanks for any info. > > Bill. > > PS. Post you response to the list server if you like.

 

On Fri, Jan 26, 2001 at 02:18:37PM -0800, Alex Pavloff wrote: > > I've read numerous messages over the last couple weeks about people plugging > their PLCs and associated doo-dads into the Internet, and I have a question. > > Have you guys ever thought about: > > 1) Security Holes > 2) Vulnerability to denial-of-service attacks Lonworks seems to be making headway in our area, and some contractors, with little notion of any potential problems, are happily connecting nodes to the Internet for remote configuration and operation. I don't know what measures are taken by the makers of the connectivity devices, but these things are going online, and not behind any firewall. In this case the answer to your question is a definite "No!". Ken -- Ken Irving <[email protected]>

 

> 1. I don't know about other PLC's, but I expect that Modicon > would provide (web downloadable) flash upgrades for their > modules. > > 2. I don't think Ethernet modules were intended to get PLC's > onto anything other than an intranet. In fact, if you look at > past conversations on Ethernet in the plant, you would find > recommendations against mixing plant and office equipment on > the same network. It is extremely possible to have your plant equipment on its own network that can access the internet. > Just because something is possible, does not mean it should > be done. I know what you're saying... but you give customers an inch, and they'll take a mile. This happens with the products that I do here. We added a very inaccurate millisecond timer, mainly for benchmark purposes. The first thing people did, despite it specifically saying in the documentation "DO NOT USE THIS FOR CONTROL" was... attempt to use it for control. Now with these TCP/IP on ethernet devices popping up, I see people wanting to take advantage of TCP/IP and remotely access the factory much simpler than they could previously. Set your plant up with a smart ethernet switch, add a gateway top the internet, and you're done. You're right... this wasn't what any of these devices were designed to do. But they're going to be used that way, and with companies promoting TCP/IP! access on everything, I just want to: 1) Make sure that people understand about the security concerns about these devices 2) Ask if the companies put out these devices are taking security concerns into account I think that the answers are "No and No", and that worries the hell out of me.

 

> > Have you guys ever thought about:
> > 1) Security Holes
> > 2) Vulnerability to denial-of-service attacks > > These problems can be overcome with a decent firewall.
If you're looking
> for an open firewall solution, look no further than OpenBSD or Linux.
> While I am not familiar with BSD (I'm a Linux guy as you will
> shortly see), it is a more mature technology and claims to be more secure.

I would disagree. A firewall is not a be-all/end-all to solve all security needs. It'll solve many problems, mainly those involving people accessing stuff on your network that they're not supposed to. However, if I have a service that I want to provide to the outside world, like a web server, and the web server has some huge gaping security holes (like buffer overflows), the firewall isn't help with that at all. In my case, the motion controller that I was using was unable to accept a simple port scan. If a system administrator decided to do a port scan of all the machines on the factory floor in an attempt to figure out if any of them are providing network services... *POOF* there goes the motion controller's interface. What good is a TCP/IP interface if it can't coexist with things that can go on on a TCP/IP network? This is bad. Even if a TCP/IP device isn't connected to the internet, it's still connected to a much larger world that it was before, and I, as you may realize, am concerned that the companies putting TCP/IP on industrial automation devices haven't rigorously tested their products for reliability. (I'm choosing to ignore the fact that many companies in the computer industry as a whole don't bother testing their products for reliablility). Alex Pavloff Software Engineer Eason Technology

 

List members: First I agree that a good fire wall where the Internet connection enters your plant (and maybe another between the manufacturing and business networks which are *certainly* separated) should solve most of the problems of external attack from most hackers. A knowledgeable insider (former disgruntled employee, etc.) can probably kill you no matter what. But there is another side that our support people ran into recently with a web browser enabled device. A customer of ours has had TCP/IP/Ethernet running in the factory with complete success for a couple of years. (Our product is NOT web enabled.) Then one day, things when to.... well, a lot of network problems started to occur. All the standard things were considered to no avail. Finally, our customer support folks gave me a capture of the network traffic that was being passed around on the manufacturing network. After mushing through bunches of Ethernet packets and scratching my head (and other places) a lot, I found the problem. The events that lead to the problem were: The customer had recently added a controls device that was web enabled. Of course, Internet browsers were installed on plant floor computers. All was well, until one knowledgeable operator pointed the browser to the Internet and started listening to a favorite radio station over the Internet. (In this plant, having a regular radio is OK, so no company rules were violated.) This put a heavy stream of audio TCP/IP/Ethernet broadcast packets on the manufacturing network which at times became over loaded. Essentially, a Denial of Service attack generated by an employee listening to the (Internet) radio! I have some real concern about web browsers on the plant floor. I can see sexual harassment law suits when one work breaks through the protection schemes (all of which can be broken) and has pornography on the computer and another employee sees it..... A hacker with a web browser wanders in from the Internet and finds the web enabled devices. A short time later breaks the password and starts playing around with the device...... Note that a device that uses a unique protocol (most hackers have not heard about MODBUS or any industrial protocols) is immune from this sort of hacker attack. Figuring out a typical plant floor protocol from an external hacker location is a *lot* harder than figuring out a password on a browser page. Maybe dedicated PC programs for configuring and accessing plant floor devices is not such a bad idea after all...... Has anyone heard of any other actual cases of problems from web browsers on the plant floor? Anyone have any comments on this-- is it a real problem? Cheers! Steve Cliff (My opinions are mine and not those of my employer, etc etc)

 

Standard practice should be to have multiple backbones running on your fiber optic backbone. (Fiber is relatively cheap, compared to a down-time of mfg equipment). Anyways, install both your process-control backbone, your enterprise backbone, and a MES backbone through-out the plant. Use hubs that support multiple segments, so you can quickly patch systems from one system to another. (Follow BICSI wiring guidelines, so you can quickly change your mind as your requirements evolve). There is nothing wrong with Web and Internet enabled devices, providing you have good network management practices in force at your company. Firewalls are merely one aspect of a good network management practice, documenting failure modes is another, use of open standards is a another practice, change-control, etc. In short, to be successful these days, you need strong collaboration between engineering and the IT to quickly solve problems. In other words, we are talking about issues that IT has addressed for the last 20 years (Access Control). I am looking forward to the day, where our PLC's can use 128-bit encryption, and access the LDAP (or Active Directory), for security tokens. Hopefully, built into a PCI card like my 3Com unit at home. Oh well, stop dreaming and go back to work. - Billiam -

 

> The customer had recently added a controls device that was > web enabled. Of course, Internet browsers were installed on plant floor > computers. All was well, until one knowledgeable operator pointed the browser to > the Internet and started listening to a favorite radio station over the > Internet. (In this plant, having a regular radio is OK, so no company rules were > violated.) This put a heavy stream of audio TCP/IP/Ethernet broadcast > packets on the manufacturing network which at times became > over loaded. Essentially, a Denial of Service attack generated by an > employee listening to the (Internet) radio! Well, this is where technological solutions go into play... it would be pretty easy to restrict outgoing traffic via one of numerous methods. However, this just goes into my original point of "know exactly what you're getting yourself into." > I have some real concern about web browsers on the plant > floor. I can see sexual harassment law suits when one work breaks through the > protection schemes (all of which can be broken) and has pornography on > the computer and another employee sees it..... Yup. If I was in charge of PCs on the factory floor, I would make sure that the PCs were "locked down" to the point where and operator could just, well... operate. > Note that a device that uses a unique protocol (most hackers > have not heard about MODBUS or any industrial protocols) is immune from this > sort of hacker attack. Figuring out a typical plant floor protocol from an > external hacker location is a *lot* harder than figuring out a password on a > browser page. Maybe dedicated PC programs for configuring and accessing plant floor > devices is not such a bad idea after all...... I disagree completely. Most hack attacks are NOT done by "password guessing", they're done by exploiting vulnerabilities in software via scripts. The reason you won't see a bunch of "script kiddies" (those with no real computer ability that have downloaded these scripts from the internet) attacking factory automation equipment is, well, because they comprise a tiny fraction of the machines out there on the internet (compared to servers running Microsoft's IIS or Apache or some standard web server). Automation equipment inherently more secure than any other piece of internet software, and my concern is that automation vendors, lulled into a false sense of security aren't paying any attention to standard security practices (no hardcoded passwords, avoiding buffer overflows, etc etc etc.) What security measures have those of you running TCP/IP on any sort of network connected in anyway to the internet taken?

 

Hello, List,

I apologize for the length but thought this useful to all. This is with regard to Alex
Pavloff's original posting of 26 January 2001...

A local corporation has reloaded the programs in fifty Allen-Bradley PLC-20e in one block of ethernet LAN TCP/IP addresses (not altogether
contiguous).

All fifty PLCs stopped communicating with their HMIs at the same datetime (within seconds) and dumped their programs completely. The HMIs and many Siemens S7 (perhaps S5) PLCs in the same block of addresses showed no sign of distress.

The PLC-20e processor switches were all in the "remote" position - enabling remote programming via the LAN. DO NOT DO THIS! The corporation has outside firewalls but they do *not* have routers between the process network(s) and the information network. USE ROUTERS!

Alex's original posting and the subsequent discussion point to a possibly innocent incident promulgated by internal sniffer or port scanning software. One can not discount malicious activity, internal or external.

Have you had any thoughts, experiences, and countermeasures since the original discussion?

Best,

John G. Boland, president
VisiBit Corporation
www.visibit.com
One Parker Square Suite 408
2525 Kell Boulevard
Wichita Falls, Texas 76308
940.322.9922
940.723.1478 fax

 

> Alex's original posting and the subsequent discussion point to a
> possibly innocent incident promulgated by internal sniffer or port
> scanning software. One can not discount malicious activity, internal or
> external.
>
> Have you had any thoughts, experiences, and countermeasures since the
> original discussion?

Sadly, I am just now reaching the point where I'm using and have access to more TCP/IP enabled devices. So far, 2 out of 2 ethernet motion controllers from different companies fail horribly when subject to an ordinary nmap
scan.

I too would be very curious if those out there with ethernet devices would try a port scan (something like nmap -sS from *nix) on these devices to see how well they perform. I suspect poorly. While I haven't seen a faulted PLC yet, my two pieces of equipment stopped responding to serial and TCP/IP requests.

Alex Pavloff
Software Engineer
Eason Technology

 

Hi Alex and list,

Alex Pavloff wrote:
<clip>
So far, 2 out of 2 ethernet motion controllers from different companies fail horribly when subject to an ordinary nmap scan.
<clip>
While I haven't seen a faulted PLC yet, my two pieces of equipment stopped responding to serial and TCP/IP requests.
<clip>

after John G. Boland wrote:
<clip>
fifty PLCs stopped communicating with their HMIs at the same datetime (within seconds) and dumped their programs completely.
<clip>

17:50 24 September 2002
Sorting things out takes a while...

The firmware in the local corporation's ethernet LAN (not internet) connected PLCs apparently expected an ethernet header packet type (header followed by contents). Some DEC VMS era ethernet used trailer encapsulated packet types. A Berkeley trailer encapsulated ethernet packet (perhaps in multicast mode) fooled the PLC firmware, which offset a header's length number of bytes into the packet and attempted to use
the contents. Oops, no header, wrong offset...

Unfortunately the incorrectly offset instruction / data packet contents were *almost* valid. The PLC puked. Actually fifty PLCs puked.
Simultaneously. (There is a PLC firmware upgrade.)

...and the LAN-connected PLCs had run continuously for a couple of years without a hiccup

...and there had been no publicized changes to HMI or other LANned PC software.

Make mine flambe'... In my opinion, interconnectedness and openess initiatives *must* increase the probability of this kind of accident.
Compare and contrast with a traditional DCS wherein proprietary hardware and protocols and hierarchical design have thwarted access to the
process controls. Those barriers to data access and creativity have, nonetheless, mitigated the risk of desktop BrandX shareware, upgrades, or new and improved data access software tools poisoning the well.

This demonstrated risk should be of great concern... again, respectfully, in my opinion.

Best,

John G. Boland, president
[email protected]
VisiBit Corporation
www.visibit.com
One Parker Square Suite 408
2525 Kell Boulevard
Wichita Falls, Texas 76308
940.322.9922
940.723.1478 fax

 

You're right, a good firewall is essential to protect your environment, and there are many, many out there. Don't forget the good ones have many additional security features since hackers have moved well beyond raw connectivity. They now use buffer overflows and other types of legit traffic that a firewall lets through. To protect against this there is a new class of perimeter device called integrated perimeter appliance, Unified Threat Manager and other such things. They provide firewall, intrusion prevention, anti-virus and a whole lot of networking features like VPN encrption and routing. In the industrial world a company like Verano offers that capability in their Industrial Defender line.

Dr.John