Growing pains with AT&T Gigabit Fiber

We live in a truly glorious world. A world where consumers can get a symmetric gigabit of fiber optic in their homes for a completely reasonable price. There are a number of competing providers out there, and I will happily hand my money to the first one offering service in my area (looking at you Google Fiber). While excited, I’ve also been extremely curious about the implications for prosumer and business class services. This new wave of broadband fiber plans are offered for literally a fraction of the cost of an equivalent T1/Dedicated connection.

After walking through a painful, multi day debugging process with AT&T’s “Small Business” fiber solution at work, I realized what this new class of service really can do, and what it really cannot.

First, let me say that I am more excited than ever to have gigabit fiber in my home. You really do get whats advertised: stupidly snappy webpages, giggle inducing download speeds, and that oh-so-sweet XBox Live host advantage.

However, if you happen to be a business, there are a few very serious drawbacks. The first, and most obvious, is the lack of an SLA. A web-based software company cannot afford to have the internet go out without a real support contract.

This particular problem is easily remedied by upgrading to the business version of consumer fiber, as oxymoronic as that sounds. Generally, you end up paying twice as much for asymmetric speeds (~1000×200) but walk away with some legal agreements and a static IP. Sounds perfect, right? Wrong…

ATT U-verse GigaPower

Our corporate network crashed a mere 3 minutes after coming online with AT&T.

After 3 days of on-and-off debugging and not a peep of helpful advice from AT&T, I noticed something in the configuration of the modem. AT&T has imposed a limit on the size of the internal NAT table of the provided modem (Arris NVG595). This is not some limit so astronomically high as to be meaningless (as exists in every modern router). No, this modem locks up after a measly 2500 NAT entries.

For those of you playing at home, opening a single webpage that contains some ads, a couple of analytics tools, and a third party font provider will consume half a dozen or more NAT table entires for the next 15 minutes. Now, multiply that out for every page change in every tab on every computer in an office full of Software Engineers and tell me if the number you get is less, or MUCH greater, than 2500 entries. That doesn’t even take into account  people who programmatically SSH into dozens of servers at a time or use torrents to download Linux ISOs. Now, I completely understand that we are “power users”, but only 2500 sessions? Really?!

The solution to this problem, of course, is just to enable bridge mode. Their NAT table doesn’t matter if I just fall back to the one in my real router. I looked for the setting once when I did my initial configuration pass, but couldn’t find it at the time. I figured it was buried deep in the GUI, and since it wasn’t pressing, I just moved on. Turns out, the reason I couldn’t find the option for bridge mode is because it just isn’t there. There are a bunch of fancy sounding features like “IP Passthrough” and “Cascade routing” which almost add up to bridge mode. But not one of them allows you to disable Network Address Translation.

Arris NVG595 Modem

Absolutely dumbstruck, I looked in to it, and yes, the Arris NVG595 does indeed support bridge mode. At the time, I couldn’t decide if I was more relieved that Arris hadn’t manufactured a brand new modem that didn’t support bridge mode, or more disappointed in AT&T for condemning us to this fate for truly unfathomable reasons.

Feeling defeated, I figured we would just have to buy an SFP+ laser and drop AT&T’s fiber line directly into the router. Not the end of the world, just inconvenient. Plot Twist! AT&T actually authenticates their entire broadband network using 802.1X certs burned in to the modem’s firmware. Now, of course, my inner hacker wants me to whip up a quick 802.1X filter using an SFP enabled Edgerouter and bypass the modem for everything except authentication traffic. If I were at home, I probably would have. But in an enterprise setting, there is a very real time-cost for hacks like that.

When I finally got a hold of an engineer from AT&T, he essentially said “Oh yea, we know about that, sorry. We are reviewing the policy and should have a fix out by Q3”. Their only suggestion in the mean time was to switch to their Business Ethernet (T1-esque) service.

Needless to say, we switched back to our previous service from One Ring, and have been much happier. We upgraded our speeds a fair bit, and while it is a hell of a lot more expensive than broadband, it is still cheaper than the AT&T alternative. You really do get what you pay for.

15 thoughts on “Growing pains with AT&T Gigabit Fiber

  1. Outstanding write up. THANK YOU FOR THIS!!

    We ran in to this exact situation as well, when setting up a client on this same AT&T service (ABF, or, AT&T Business Fiber: $650.00 a month, no contract, 1 gigabit symmetrical, static IPs.). It had worked perfectly for smaller under-30-user offices, including our own. As soon as it was installed in a business with 100 users, it collapsed in a few minutes. Neither AT&T nor Arris were of any real use, other than saying “Yeah. we’ve heard of this. Donno really.”

    This post details everything one needs to know about the limitations of AT&T “Business Fiber”.

    As a general rule, we no longer recommend this service for any office that has more than 20 people.

    Thanks again for your hard work and detailed post.

  2. Pingback: Homepage
  3. We have this problem as well and no longer use this shared fiber as our primary circuit. Our sales rep said they should have a new modem that is capable of possibly 7-8k sessions in March or April, 2018. However, there are no guarantees on which modem a customer will get because “it’s going to be tricky. There is no guarantee as to what modem the customer will get; it’s just based on each garages inventory”.

  4. We had the same thing – we were so happy about getting ATT Fiber-Optic 1G connection, until during the install the tech worker from ATT said: “By the way, do you know that this modem support about 2500 sessions? It starts dying once it reaches 2000 sessions”. It simply could not support our 90+ people office. I checked our firewall stats – we had somewhere between 3500-7000 sessions during the day.

    I researched a lot, and implemented a bridge based on Ubiquiti Edgerouter Pro router. Session limitation was gone. But the device could not support full 1Gb speed – it was maxing out on 400Mb up and down. Plus there was a memory leak in router OS and in 2 weeks memory usage went from 29% to 85%. CPU usages also stayed 95-100 %. Maybe with additional tweaks I could disable all unnecessary services and make it work, but I decided to try another route.

    Next, I found descriptions about using Linux and a computer with 3 networks cards. Took me several weeks in my spare time to learn Linux, configure and make it work. Success!!! Full 1Gb is supported (actually we are getting about 940-970 Mbps up and down). CPU usage (Xeon E3-1230) is always less than 1%, memory usage is 109MB. I used a server level motherboard (SuperMicro) and a server level 4 port network card. Works with zero problems since started for the first time – more than 6 month.

    The Linux bridge is configured to simply split traffic into two routes – authentication goes to the modem, our public IP range is redirected to a separate network port. Modem is serving as an authentication device, the rest of the traffic is handled by Linux box. Actual change to base Linux is to add several modules and about 20-30 lines of configuration. The only current limitation is that this configuration supports IPv4 only. We simply did not need IPv6 yet.

    1. It is good to hear that you managed to get it working! I am actually curious how you managed to get the modem in bridge mode. Our modem absolutely refused to enter true bridge mode – it would always try and run its own NAT, resulting in crashes.

      1. I could not switch the modem to a true bridge mode. Did not work. It always had to use NAT table, no matter what I tried.

        The new device (basically a Linux server with 4-port Ethernet card) – I call it an “ATT Fiber Optic bridge” sits between Internet fiber port (on the wall) and the modem.

        This is the schema of how everything is connected (I am not putting media converters here):

        [ATT Fiber Optic Port]
        [ATT F.O. bridge]—[ATT FO modem]
        [Office Firewall]
        [Office Network]

        In order to enable communication, the user end modem has to be authenticated when establishing the connection and then re-authenticated every 24 hours.

        Authentication traffic is sent to the ATT FO modem only. Rules on Linux box – ATT FO bridge enforce proper traffic routing.

        The rest of the traffic is directed through another port to Office Firewall.

        In solution #1 ATT FO bridge was Ubiquiti Edgerouter Pro.
        In solution #2 ATT FO bridge is a server based on SuperMicro MB with E3-1230 CPU, 16 GB RAM and 500GB HDD. This is a real overkill. I am working on building a bridge based on an Atom-based server board with 4 Ethernet ports.

        Solution #2 was running since August 2017. ATT did not complain 😉

  5. Our 7 person office was able to reach our NAT sessions limit of 2560 last week. AT&T field and call center techs have no idea about this stuff. It took us eight days of being up and down when finally our senior IT consultant engineer said, you might want to take a look at your NAT sessions, then we found your post. Thanks. Our business bank had us install a “Detect Safe Browsing” program that put us over the top. But this also explains why we sometimes hit a wall with our internet service and phones. During this process we also concluded that our staff’s use of p2p music services contributes to more sessions per normal user.

    Q: Since we can’t switch the modem to bridge mode, can’t we just use multiple fiber modems and split the traffic and double our available sessions?

  6. Hi Vlad K,
    Is there a way to communicate with you more specifically about your setup? I would like to know how you connected from the fiber to the linux modem.

    I bought a Checkpoint 730 firewall and I am wondering if I can use it instead of the Ubiquiti router you got

  7. Regarding using multiple modems in parallel – in my opinion they will not work in this configuration: splitting one service in parallel threads. Each modem has a built-in authentication module, which uses some sort of unique identifier. ATT will identify multiple modems as devices for separate services, not for one service. Technically it is possible, but you will end up paying for multiple connections.

    Regarding Checkpoint 730 firewall – unfortunately I have no information about this firewall. If it has Linux as base OS, then it might be possible to modify config files to make it work – similar to Ubiquiti.

    There is one trick that Linux is capable to do – allow authentication traffic (EAP 801.1X) forwarding through two network interfaces configured as a bridge. Regular switches are blocking this traffic. I tried putting a regular switch between the modem and FO port on the wall. Connection is not getting established.

    I will be really busy next two-three weeks – need to finish one important project for our customer. I will start working on a write-up after that.

    If you have patience, you will have to wait until I have the write-up. If you have no patience, you can search Inet for Linux setup for ATT FO Inet bypassing session limit.

    I used this link to setup Ubiquiti router:

    This link gives some details (not everything unfortunately) about how to setup Linux box to work as bridge:

    Here is one more Linux setup info link:

    The last one helped me the most. You still will need more than just this config.

    This is all I have time for at this moment.

  8. Hi Vlad,

    I’m wondering if you had time to document your setup yet – I am due to get ATT FO in the next several weeks – I like your workaround

  9. Pingback: essayforme
  10. Pingback: essayforme
  11. AT&T just replaced our gateway with the new model that supposedly has an 8000 NAT entry limit. However, the BGW210-700 does not support fiber so it now needs a fiber to ethernet converter so they’ve also installed a TP-Link MC200L media converter.

    I think it might be time to pony up the extra $500/month for the Enterprise fiber =(.

  12. We went through the issue of maxing out NAT sessions on AT&T’s older model and spent weeks troubleshooting the issue because it appeared to be packet loss. Ultimately we got the newer model and that helped a lot.

    Everyone needs to beware of a confirmed recent firmware upgrade that was applied between 12/11/18-12/13/18. It made a few settings static that were once dynamic and now our site-to-site IPSEC VPN connections make their gateway freak out and results in about 40% packet loss, 1000+ ms latency, and ultimately DoS’s their gateway to the point of it rebooting.

    I just spent the last week and a half troubleshooting and finally found out about the firmware upgrade. They tell me the upgrade is automatic and can’t be reverted. Even a gateway that has older firmware that is still in the box can’t be “preserved” in its current state. I find that hard to swallow since this is essentially a DoS vulnerability.

    The AT&T field technicians are all professional people and are typically really great people. Most of the ones I talk to are seasoned technicians and many have been in the telecom industry for over 10 years. It really is a joy to talk to some of them face to face. The bad part is that they are seriously undertrained. ABF was a big reaction to Google Fiber in my metro area, and

    All this time troubleshooting, I learned some more of AT&T’s acronyms. They call the gateway the RG. I thought it stands for Remote Gateway, but the emotional low point was reached when I learned it really stands for RESIDENTIAL Gateway. Coincidentally, I was at a friend’s house last night who has AT&T fiber and what do ya know, he has the exact model I have for business.

    Thankfully we have a Spectrum cable circuit still up and running and so we are using that for site-to-site VPN tunnels for the time being. Unfortunately that circuit speed is only 200/20.

    Soon, we are leaving AT&T to go with Century Link (formerly Level 3) and begin life with an SLA. I am thrilled to think of what it will be like to simply call a phone number and talk to someone who doesn’t read script from a screen to troubleshoot a problem. Someone who really knows what they are talking about and is empowered to identify and fix a problem.

    At the end of all this, I feel ripped off after seeing the same RG in my friend’s house. I’m paying a LOT more for my ABF, and his circuit speed is faster. Ultimately we get the same level of “support”.

Leave a Reply