Verizon DSL Hacked into my Home Network

(Rev 1.2.1 updated 5/2011)

 Brian Mork - Fall 2010
© 2010-12 Increa Technology

Site Index  •  Wiki  • Blog

Tweet


Verizon DSL Router Caught Red-Handed !


 My in-home network was working fine with a Netgear WGR614 ver 7 wireless 4-port Ethernet router using a 192.168.1.1/8 subnet LAN.  I  signed up for the Verizon DSL service.  Verizon's ads show up many on this page in the ad bar on this page.  You can compare DSL vs. Cable vs. MegaPath Ethernet options.  T.J. Nelson used to have a good DSL vs. Cable modem comparison web page, but the page seems dead summer 2012; any good replacement suggestions?

I installed Verizon's Westell DSL modem on the WAN side of my router following the instruction that came with the modem.  Within a day after the DSL service went live, every computer on the LAN side of my network was switched from 192.168.1.1/8 to 10.0.0.1/8.  Additionally, all the DHCP ranges, and reserved addresses inside the Netgear router were reprogrammed to 10.0.0.1/8 address.  The Westell modem took the 192.168.1.1/8 network, and by taking this address, any 192.168.1.1 gateway requests from my computers (intended for my Netgear to handle) were now routed to the Westell modem.  How in the world did the new modem, on the WAN side of my network take over the LAN side of my network?

westell modemPrevious experience with the Helix distribution of Linux, and the Ethereal program (now the Wireshark program) taught me how to probe and log network traffic.  I reset all the computers and routers the way I wanted and removed the DSL modem (I disconnected the Ethernet cable between the Westell and the Netgear).  When everything was stable logging, I plugged the Netgear modem into the DSL modem and went to Thanksgiving dinner.  I left Wireshark, on my computer, logging all the packets it could hear from its connection to the Netgear router.

When I got home, I perused the logs.  Sure enough!  My 10.0.0.101 computer had been switched to a 192.168.1.18 number!  It was easy to scroll down the Source column and find where the IP address switched about 21632 seconds into the log.  Verizon performed a classic ARP attack to steal my computer's IP, swapped addresses inside my LAN network, executed a perfect Man-in-the-middle (MITM) attack on my network.  It appears they culled admin passwords to the router, and modified my router settings.   Unbelievably, their DSL User Agreement paragraph 10(4) actually says they will "access and adjust your computer" settings!  Do you know what your internet provider is doing?!

 Here is one security expert write-up on why MITM attacks are threatening.  I used to do a lot of work with PLC and SCADA devices (computers used to control hardware in the real physical world).  Here is a description of how the  Stuxnet MITM attack is used to modify behavior of a Siemens PLC.  The best Windows OS analysis of Stuxnet that I know of is from Mark Russinovich, author of the SysInternals suite of tools. If you're interested, you can read about the murder of an Iranian computer expert trying to purge this worm out of their nuclear power plants.


Ethernet Log Documents Verizon Hack into My Network


Below is a screen-shot of the Wireshark log of my network traffic while I was out of the house on a holiday trip to friends house.  Wireshark is the same program as Ethereal, which apparently quit development in about November 2006, and became Wireshark. Wireshark runs on Windows, OSX, and Linux. You can even get wireshark on the USB-drive bootable U3 operating system.  I recommend obtaining and playing with this program for a weekend.  You will never see your internet connection in the same way again!

In this case, Wireshark was running on my Mandriva OS computer on the LAN-side of the Netgear in promiscuous mode.  None of the activity logged below was initiated by human activity.  Topology was the DSL modem hooked into the "uplink" port of a Netgear router.  The only computer powered up was a Mandriva Linux OS plugged into the LAN side of the Netgear router.  The DSL modem owned the 192.168.1.1/8 network, and the Netgear router owned the internal 10.0.0.1/8 network.  Missing packet numbers in the sequence are because many 127.0.0.1 housekeeping transmissions are not shown.  More specfically, the packet filter on the Wireshark display was "! (ip.addr == 127.0.0.1)".  This log picks up 6:00 hours (21611 seconds) after logging was started...

verizon modem steals my ip address

What you see in the above Wireshark log, on the third blue line, is my computer (10.0.0.101 Elitegro network card) announcing its CUPS printer server available on port 631.  The logs show this occuring every 30 seconds or so for a few hours.  Verizon is using the Westell modem to sequentially ARP-scan 192.168.1.1/8 addresses to find anybody on the network.  This is done at a rate (every few seconds) just below the threshold that most firewall programs will report a scan attack.  Perhaps this is a way to find rogue DHCP servers that someone may have put on the net to conflict with the Verizon's services.  I don't know why Verizon does this, whether from remote servers, or from Jungo DSL middleware installed on the modem (as suggested by one reader).  Notice the Westell modem doesn't do this type of thing.  It's a "stupid shell" through which the logic and intent of Verizon's scripts execute.

When Verizon found my Netgear router at 192.168.1.18, the my Netgear router the Linux box responded with the five purple packets.  What happens?  About 0.01 seconds later, the modem sends a bogus attack packet followed by an immediate DHCP offer of a new IP address, and a DHCP ACK.  Notice there are no missing packets in this sequence.  All three are shotgun out in quick sequence.  You can duplicate this type of behavior using the Ettercap program if you'd like to experiment yourself.  In fact, acquire a copy of the entire Helix Linux distribution, and you can have all sorts of fun.  I have successfully captured other computers on a network, and forced them to send all their traffic through me.  Caution: you do this where you're not welcome and it is criminal activity.

This fast triple sequence (NAK OFFER ACK) is intended to snatch the attention of the target computer before any ~other~ DHCP server can figure out what happened.  It worked.  Notice the packet was sent to 255.255.255.255 - this means all other networks, not just the subnetwork that the Westell is on.  There is no reason to do this except an intent to reach into other subnets and hack into them.  Remember, the Westell was on the WAN side of the Netgear; this data is being logged on the LAN side of the Netgear. I need to look into whether or not the Netgear router can be configured to block "broadcast all" packets from WAN side to LAN side.

As an aside, maintaining TOTAL silence on the network (replying not at all) would have caused the Westell modem to pass me by and go searching at other addresses.  This is the theme strongly advocated by the grc.com ShieldsUP! web pages in order to make your computer less attacked while on the Internet. However, your computer can not remain silent about DHCP ARP packets, because that's how your computer gets an IP address to do all the rest of the required network communication.

You'll see that the attack worked because 0.08 seconds later, my computer (Elitegro MAC address) announced its new IP address with the bright tan-colored packet.  It has been hacked!  Simultaneously, it's netmask and gateway has been changed to point it toward the Westell modem (not visible in this log).  The Westell has inserted itself in the middle of any communication between my computer and my LAN router.  This is a Man-In-The-Middle attack. In other words, if my computer now tries to send to the Netgear router at 10.0.0.1, the packet will be sent to the Westell modem first because this address is off its newly assigned subnet.  When I sent a username and password to the Netgear router, it appears the Westell modem stole it, and used it to do whatever it wanted with the Netgear router.  I believe that's how it succeeded in reprogramming my internal LAN network away from 192.168.1.1/8 to 10.0.0.1/8.

Verizon Hacks and Collects your LAN Information

Below is the main page of the web server inside the Westell modem.  Only the top panel exists when you first install the modem, before it has ever communicated on a DSL line; the bottom 3 panels of data (with the red banners) do not show up.  Instead, a panel saying something like "Hook up your DSL cable" is displayed.  Whenever the DSL cable was disconnected, the data for the panels was unavailable.  The point here is that displaying these pages requires communication with the "mother home ship" back inside the Verizon network somewhere.  The Westell modem is not intelligent enough to do these things itself.

It appears that any information about my internal network is traveling over the DSL lines back to Verizon. Of note, the modem home page displayed below, at 192.1681.1, was recorded after I fixed the IP-snatching problem.  The one listed computer on "My Network" below is one device Verizon should see: the Netgear router.  When I had other computers inside my LAN, they also appeared on this page until I tuned them to refuse DHCP ARP attacks.  Realize that all this internal LAN information is being collected and used by Verizon, available to them because they hack into the network on the LAN side of your firewalled router!  See Verizon's DSL User Agreement, paragraphs 10(2).

westell g90 homepage


Solution


I turned off DHCP acquisition of IP addresses from every hard-wired omputer inside the LAN.  It seems to be okay to let the Netgear issue IP addresses to wireless computers. In other words, manually set your computer addresses to 10.0.0.x, netmask to 255.255.255.0, and your gateway to 10.0.0.1 (the router).  This requires you to also go into the Netgear router and specify each of the 10.0.0.x addresses used by one of your computers as a reserve assignment, so the router doesn't attempt to give it to any other computers that happen to log onto your LAN network and request an IP from the normal DHCP pool.  Also, go into the Netgear router and set its WAN side IP address to a fixed value (I use 192.168.1.19).  It seems there should be some way to stop passing "broadcast all" messages from one side of the router to the other, but I haven't found it. If you know of any other details or better ideas, please email!

Update December 2010 - I added another hardwired DHCP Windows computer on the downstream side of the Netgear router and issue a DHCP request.  Boy, did that gum up the works!  Somehow the DSL modem issued a 192.168.1.x address to it before the Netgear could issue a 10.0.0.x address.  This situation got my Netgear modem into some sort of routing loop or reprogramming state it would not come out of (check light lit up on the front).  Even pressing the hardware button on the Netgear router would not reset it!  It cost me 5 hours of dancing configuration to get this fixed.  I was hopping mad.

The only way I found to fix this was to let the Netgear router get a DHCP address from the DSL modem (192.168.1.0/8) and let the Netgear router issue DHCP addresses to the LAN (10.0.0.0/8).  Then, within seconds of all the addresses being issued, I had to separate the DSL modem and the Netgear modem so that they would not go back into the gum it up mode.  Once they were separated, I admined into the Netgear with a computer manually set to a 10.0.0.0/8 number, and locked it down to a fixed IP address on the WAN side.  I admined into the DSL modem using a computer manually set to a 192.168.1.0/8 number and turned off it's network scanning, so it wouldn't try to look inside my network.

I also wanted to use Tor, so I admined into the DSL modem and set up a port forward from 9100-9100 to a new base of 9100 (doing nothing, but giving me a port forward rule to use).  However, the port forwarding rule could not be completed by manually specifying the 192.168.1.19 Netgear IP.  Instead, I had to turn network scanning back on, let the DSL modem find my Netgear, and then it would allow me to select the Netgear IP as part of the port forwarding rule.  Lastly, I entered in a port forward inside the Netgear to pass packets to my Tor computer.

Update April 2011 - This page was referenced on DSLReports.com, and I've received a number of thoughtful emails, suggesting further tests, or questioning things.  Below are some thoughts and additional information.  I'll continue to do tests and post results (as this activity fits in an already busy schedule of family, work, life).

I should have explained up front that I didn't go hunting for these problems. Instead, I had a stable local area net running on the LAN side of the Netgear router, using 192.168.1.1/8 addresses issued from the Netgear router's internal DHCP server.  I purchased Internet service, and hooked up the DSL modem to the WAN port on the Netgear router to provide Internet access.  Problems began.

Within hours, all my computers on the LAN-side of the Netgear were reprogrammed to 10.0.0.1/8 IP addresses.  There was no reason the computers should have asked for / obtained new IP addresses, and nothing *I* had configured would even make 10.0.0.1/8 addresses available.  Everything was reset to the way I wanted it, and it happened again.  I looked deeper and found that the Netgear DHCP server was now programmed to hand out 10.0.0.1/8 numbers.  The third time, I logged packets to see what happened. Seeing the issues, I locked everything down with static IP addresses and things have gone well for several months.

Whether or not this fits your definition of an attack, I cannot decide.  This much I know: without the DSL modem, things were stable with my choice of IP addresses.  With the DSL modem, everything got re-programmed three separate times.  I publish these results to learn all I can (and part of what I'm learning is there are a lot of smart people reading DSLReports that are willing to help!), but in the end, the presence of the DSL modem was undeniably a causal factor.

I don't want to engage with syntactical debates about what exactly constitutes an "attack".  If an un-welcomed WAN-side identity reprograms my router and my LAN-side computers, and sets themselves up as the DHCP and DNS server/gateway, that is an attack in my dictionary.  Sorry, disbelieving guys and gals, that's what happened.  If you don't believe me, please suggest what experiments you'd like to do and document. I assure you I didn't have have time to sit down and fake a Wireshark log.

Specific clarifications:
  1. Someone asked, "What makes you think material posted on your page is credible?"  As with any research, I would say documentation, data, and community discussion.
  2. How was Wireshark being run?  The Mandriva Linux OS was the only computer running.  I added an additional sentence to clarify that Wireshark was running on it.  Anything on the ethernet cable plugged into the Linux computer would be available to the Wireshark log.  This includes traffice sourced from anywhere on the LAN or WAN side of the Netgear (including the world-wide Internet) provided the Netgear put it on the LAN-side ethernet cable to the Linux box.
  3. Wireshark was set to filter packets as explained in the main text, using the rule "! (ip.addr == 127.0.0.1)".  All other packets are shown.  I believe this filter changes none of the conclusions.
  4. At least one readers bristled at my claim of "bogus attack packet", quoting sources describing how a NAK packet is suppose to be used, and pointing out that my claim doesn't fit the protocol.  I apologize for not being clear: I know this is not normal NAK usage, and using a NAK packet in an incorrect way is why I called it bogus!
  5. Network topology came up in some dicussions.  A clarification is in order for those not familiar with the terminology: a Man-In-The-Middle attack doesn't mean the attacker is physically in the middle.  I was critiqued for not recognizing that my Netgar was in the middle.  Of course.  With a MITM attack, the "middle man" can be on either side of any network or firewall.  I've done MITM attacks using a wireless laptop link into a WAP, and was able to reach any other computer on the network.  By "reach", I mean cause that other computer to route all traffic through me, and accept traffic through me as if it came from elsewhere.  The concept of being "in the middle" is an operational concept, not an attempt to describe physical connection topology.
  6. I was critiqued for not recognizing a modem can detect if there is a DSL signal, and asked to provide "proof" that the data displayed on the Westell's configuration pages was being accessed from elswhere.  I clarified the relevant passage in the main text.
  7. To the folks pointing out the problems are impossible with static IP addresses, I'm sorry for being unclear.  Static IP addresses were used as a solution to fix the problems, they weren't part of the problem.
  8. One interesting discussion highlighted that if my claims were really happening, 99% of the people would never know, but the remaining 1% would be vocal about it. I agree! Well, I'm being vocal, and I'm all ears for whatever data or documentation polite readers would like to cooperatively collect. I propose: setup your own 192.168.1.1/8 network, plug in your own DSL modem, network log what happens, and let's see if your local network computers are reprogrammed to 10.0.0.1/8.
  9. I received rather violent disagreement and personal slams that no outside network can "hack into my wireless Netgear router".  I can only refer that reader back to the text above starting with "This much I know: ..."  I do have ideas of how this can be done.  I'll document that it can be done with the assistance of some people behind more friendly emails, and the maybe we'll spill the beans of how.
Update May 2011 - I've received a number of suggestions that "double NAT" arrangments are unstable or unpredictable or "may cause problems".  I've learned that those phrases often mean I don't understand what is really happening yet.  By double-NAT, I mean that the Westell had different sub-nets on the WAN and LAN side, the LAN side of the Westell was on a different subnet connected to the WAN side of the Netgear, and the Netgear had a third sub-net on it's own LAN side. For me, once I locked things down with static IPs, the double-NAT setup has been stable for 6 months or more.  However, I like the simplicity of simply a bridge and my router, so I decided to give it a try.

I changed the Westell to a Bridge mode instead of Routed Bridge and reset it. Looking at the modem, you can tell it's in Bridge mode because the Internet light stays off. In this mode, the Westell obtains no IP address, and issues no IP addresses.  It behaves as a simple modem, converting DSL signals and voltages to ethernet packets. The Verizon documentation doesn't say you have to then change your local router MAC (Netgear in my case) to adopt what the Westell MAC used to be.  In my case, I found Verizon would not issue an IP address set to the router without doing this, so I did the MAC cloning according to other Westell 6100 instructions.  After cloning the Westell MAC onto the Negear WAN side, the Netgear router was assigned a WAN IP of 71.x.y.z, mask 255.255.255.0, and gateway 71.x.y.1.  DNS servers were assigned 68.238.64.12 and 68.238.128.12.

Only one problem: the assigned gateway at 71.x.y.1 is unresponsive.  It gives no response to any protocol I can send it, including simple pings.  The gateway is necessary because it has to handle all traffic to/from anywhere other than my local network!  In simple terms, I could not access the Internet. Looks like the Bridge mode has problems, or I'm still missing something.

Back to the double-NAT or Routed Bridge mode...

Unplug the Netgear from power.  Poke the reset button on the Westell for 12 or so seconds.  About 20 seconds later, the Westell Internet light comes back on, meaning it's back on the net with an IP.  Power back up the Netgear.  I manually put the Netgear DNS servers back to Google's numbers, WAN side of the Netgear set to static 192.168.1.19, and gateway set to 192.168.1.1, and MAC number back to default.

WWW through the Netgear to the Westell and change the default password (admin/password) to something a bit more obscure.  Main page reports "Routed Bridge" mode. Main tab no changes. Advanced tab select Private LAN and un-select DHCP services.  Strangely the Westell responds with "Warning:  Enabling Private LAN will disable Public LAN. Your Modem will reboot automatically due to IP address modifications. After the reboot you may need to release and renew your IP address to communicate with the modem."  I note that I am not enabling Private LAN, but rather disabling the DHCP server.  I don't know what this has to do with the Public LAN (aka WAN) side of the Westell. Click through and the Westell provided a reset banner page to my web browser for about 40 seconds. Reset date and time to my timezone. Put back my custom port forwards.

Browsing the Westell logs under the Advanced System Log tab shows a normal reset and bootup, except a continuing error gets logged over and over: "daemon.err statsd[140]: supplyDnsLocalDev: failed to open /var/etc/dnsmasq.leases: No such file or directory", and "user.err syslog: addDHCPDevicesToARP: failed to open /var/etc/dnsmasq.leases."  Appears the on-board firmware is reporting this error several times per minute.  Anybody else seeing these?  Google reports 1 or 2 hits of others with the same errors, but no replies.




Bibliography / Additional Resources

Wikipedia on ARP Spoofing -  describes the technique used to steal an IP address from your computer.  Once your IP address taken, the other computer can be you on the network.

Wikipedia on DHCP and ARP Snooping.

Dartmouth College Enumeration of DHCP Flaws- describes how the DHCP protocol lets another computer (or DSL modem) steal the IP address of your router gateway and penetrate your network to intercept whatever it wants!  This includes the password to your router/gateway, which it can then use to log-in and do secret administration of your router settings!

http://www.tekbar.net/hackers-and-security/seven-easy-way-to-help-you-defend-against-arp.html - How to defend against ARP attacks.

Valid HTML 4.01 Transitional


Thes skeleton of this document was originally created using AbiWord under a Gnome desktop.  It was subsequently edited by Konquerer to become the web page you are reading.  Created November 2010. Last updated May 2011.