We live in a truly glorious world. A world where consumers can get a symmetric gigabit of fiber optic in their homes for a completely reasonable price. There are a number of competing providers out there, and I will happily hand my money to the first one offering service in my area (looking at you Google Fiber). While excited, I’ve also been extremely curious about the implications for prosumer and business class services. This new wave of broadband fiber plans are offered for literally a fraction of the cost of an equivalent T1/Dedicated connection.
After walking through a painful, multi day debugging process with AT&T’s “Small Business” fiber solution at work, I realized what this new class of service really can do, and what it really cannot.
First, let me say that I am more excited than ever to have gigabit fiber in my home. You really do get whats advertised: stupidly snappy webpages, giggle inducing download speeds, and that oh-so-sweet XBox Live host advantage.
However, if you happen to be a business, there are a few very serious drawbacks. The first, and most obvious, is the lack of an SLA. A web-based software company cannot afford to have the internet go out without a real support contract.
This particular problem is easily remedied by upgrading to the business version of consumer fiber, as oxymoronic as that sounds. Generally, you end up paying twice as much for asymmetric speeds (~1000×200) but walk away with some legal agreements and a static IP. Sounds perfect, right? Wrong…
Our corporate network crashed a mere 3 minutes after coming online with AT&T.
After 3 days of on-and-off debugging and not a peep of helpful advice from AT&T, I noticed something in the configuration of the modem. AT&T has imposed a limit on the size of the internal NAT table of the provided modem (Arris NVG595). This is not some limit so astronomically high as to be meaningless (as exists in every modern router). No, this modem locks up after a measly 2500 NAT entries.
For those of you playing at home, opening a single webpage that contains some ads, a couple of analytics tools, and a third party font provider will consume half a dozen or more NAT table entires for the next 15 minutes. Now, multiply that out for every page change in every tab on every computer in an office full of Software Engineers and tell me if the number you get is less, or MUCH greater, than 2500 entries. That doesn’t even take into account people who programmatically SSH into dozens of servers at a time or use torrents to download Linux ISOs. Now, I completely understand that we are “power users”, but only 2500 sessions? Really?!
The solution to this problem, of course, is just to enable bridge mode. Their NAT table doesn’t matter if I just fall back to the one in my real router. I looked for the setting once when I did my initial configuration pass, but couldn’t find it at the time. I figured it was buried deep in the GUI, and since it wasn’t pressing, I just moved on. Turns out, the reason I couldn’t find the option for bridge mode is because it just isn’t there. There are a bunch of fancy sounding features like “IP Passthrough” and “Cascade routing” which almost add up to bridge mode. But not one of them allows you to disable Network Address Translation.
Absolutely dumbstruck, I looked in to it, and yes, the Arris NVG595 does indeed support bridge mode. At the time, I couldn’t decide if I was more relieved that Arris hadn’t manufactured a brand new modem that didn’t support bridge mode, or more disappointed in AT&T for condemning us to this fate for truly unfathomable reasons.
Feeling defeated, I figured we would just have to buy an SFP+ laser and drop AT&T’s fiber line directly into the router. Not the end of the world, just inconvenient. Plot Twist! AT&T actually authenticates their entire broadband network using 802.1X certs burned in to the modem’s firmware. Now, of course, my inner hacker wants me to whip up a quick 802.1X filter using an SFP enabled Edgerouter and bypass the modem for everything except authentication traffic. If I were at home, I probably would have. But in an enterprise setting, there is a very real time-cost for hacks like that.
When I finally got a hold of an engineer from AT&T, he essentially said “Oh yea, we know about that, sorry. We are reviewing the policy and should have a fix out by Q3”. Their only suggestion in the mean time was to switch to their Business Ethernet (T1-esque) service.
Needless to say, we switched back to our previous service from One Ring, and have been much happier. We upgraded our speeds a fair bit, and while it is a hell of a lot more expensive than broadband, it is still cheaper than the AT&T alternative. You really do get what you pay for.