HE Peering Issue

Question

I have been in contact with some admins from NFO servers recently as well as a few months back in October.
I sent them a few trace-routes from my IP address to one of the servers they Host for Battlefield 4 because of some unusually high latency I was getting. Long story short the issue came back recently and they referred to it as a HE Peering Issue that they are attempting to route around so that I can still get reasonable latency. For the most part they have been successful but they wanted me to inform Cox of these issues.
I'm not exactly sure what information will be beneficial for the Cox technicians so I am posting here so that they can tell me who to inform and what information they need.
Here is a summary of one of the responses the NFO admins gave me in an email:
"Previously, Cox was overloading its connection to Level3 in Phoenix, so I had to force our system to avoid using Level3 to reach Cox. That rule is still in place.  Tonight, Cox was overloading its connections to HE, Tata, and some other providers in Chicago, and that was causing high latencies."
Hopefully that makes sense to somebody and I can post more information if necessary. I still have some old trace-route logs from when the problems were occurring that I can post as well.

tecknowhelp · Answer

First, it may be helpful to include the relevant data. I am curious how they can tell it's not congestion.
Second, as good intentioned as they are, it's unlikely a moderator would be able to escalate the data to anyone that would have enough pull to change routing rules on a national level.&nbsp;
Last, this page shows latency and packet loss between Level3 and SBC as well as Saavis, AKA Century Link, so I think other providers are having issues with Level3 too. On top of that, this page has alot of people posting issues with Battlefield servers, including different states and different ISPs. This feels like a EA issue.

guardor · Answer

I have included an example below of my most recent trace-route during the period I was experience high latency. I have at least 1 or 2 older trace routes I can add as well if needed.
As far as the moderators go I was using that term to refer to several people who responded to me. The problem was actually escalated to a founder/president of the company not just basic server admins.
I'm not going to pretend I know whats going on with the routing and the problem with Level3 but that seems like the center of the issue. I don't know if Cox has control of that or can do anything about it but perhaps you could enlighten me. Based on what you said it sounds like its in everyone interest to figure out whats going on with this "Level3" thing, whatever it is.
If more information is required the best I can do is try to get you in contact with the person who was helping me and you guys can try to sort this out.

Tracing route to c-216-52-143-163.managed-ded.premium-chicago.nfoservers.com [216.52.143.163]over a maximum of 30 hops:&nbsp; 1&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp; 192.168.0.1&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp;&nbsp;&nbsp;&nbsp; 8 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp; 10.32.0.1&nbsp; 3&nbsp;&nbsp;&nbsp;&nbsp; 9 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp;&nbsp;&nbsp;&nbsp; 8 ms&nbsp; 100.127.68.170&nbsp; 4&nbsp;&nbsp;&nbsp; 12 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp; 70.169.74.146&nbsp; 5&nbsp;&nbsp;&nbsp; 41 ms&nbsp;&nbsp;&nbsp; 22 ms&nbsp;&nbsp;&nbsp; 19 ms&nbsp; langbprj01-ae1.rd.la.cox.net [68.1.1.13]&nbsp; 6&nbsp;&nbsp;&nbsp; 21 ms&nbsp;&nbsp;&nbsp; 24 ms&nbsp;&nbsp;&nbsp; 21 ms&nbsp; be-204-pe02.600wseventh.ca.ibone.comcast.net [173.167.57.109]&nbsp; 7&nbsp;&nbsp;&nbsp; 22 ms&nbsp;&nbsp;&nbsp; 22 ms&nbsp;&nbsp;&nbsp; 30 ms&nbsp; hu-0-5-0-3-cr02.losangeles.ca.ibone.comcast.net [68.86.83.29]&nbsp; 8&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp; Request timed out.&nbsp; 9&nbsp;&nbsp;&nbsp; 55 ms&nbsp;&nbsp;&nbsp; 49 ms&nbsp;&nbsp;&nbsp; 51 ms&nbsp; be-10919-cr01.1601milehigh.co.ibone.comcast.net [68.86.85.154]&nbsp;10&nbsp;&nbsp;&nbsp; 60 ms&nbsp;&nbsp;&nbsp; 51 ms&nbsp;&nbsp;&nbsp; 51 ms&nbsp; be-11719-cr02.denver.co.ibone.comcast.net [68.86.86.77]&nbsp;11&nbsp;&nbsp;&nbsp; 68 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp; be-10517-cr02.350ecermak.il.ibone.comcast.net [68.86.85.169]&nbsp;12&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 64 ms&nbsp; be-10563-pe01.350ecermak.il.ibone.comcast.net [68.86.82.158]&nbsp;13&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 64 ms&nbsp;&nbsp;&nbsp; 67 ms&nbsp; 96-87-8-110-static.hfc.comcastbusiness.net [96.87.8.110]&nbsp;14&nbsp;&nbsp;&nbsp; 69 ms&nbsp;&nbsp;&nbsp; 64 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp; border6.po2-bbnet2.chg.pnap.net [64.94.32.75]&nbsp;15&nbsp;&nbsp;&nbsp; 74 ms&nbsp;&nbsp;&nbsp; 77 ms&nbsp;&nbsp;&nbsp; 67 ms&nbsp; inap-b6.e2.router1.chicago.nfoservers.com [216.52.148.254]&nbsp;16&nbsp;&nbsp; 157 ms&nbsp;&nbsp; 155 ms&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp; c-216-52-143-163.managed-ded.premium-chicago.nfoservers.com [216.52.143.163]
17&nbsp;&nbsp; 153 ms&nbsp;&nbsp; 153 ms&nbsp;&nbsp; 153 ms&nbsp; c-216-52-143-163.managed-ded.premium-chicago.nfoservers.com [216.52.143.163]
Trace complete.

tecknowhelp · Answer

Where did they say the Cox problem was? I don't see Level3 or any of the other mentioned groups in the tracert. &nbsp;Also, most of the latency occurs between hop 15 and 16, which is on NFO network, the people who host the servers. Can you show the other tracert?
Also, as per here, NFO says:
"Our&nbsp;Premium&nbsp;locations are based on a strong foundation of Internap bandwidth, but we use additional peering and transit as well as our own optimizations to further enhance it and bring latencies down to the lowest that we can make them for our customers. Premium locations should offer the best possible latencies in their respective cities -- not just lower than Internap alone, but lower than anyone else out there."
It looks like they have a Metro-E connection between a Cox Business account and a Comcast Business account and they are using that to route traffic. I wonder if this is part of their "own optimazation" when their peers get congested. It's complex, but there is two kinds of bandwidth, the bandwidth your ISP pays for to put the traffic onto the internet, and what the content provider(NFO) pays to upload their traffic to send it back to you. Inbetween the two, there is usually bottlenecks of the public internet. Your ISP can control how the traffic gets routed inside their network, where it goes to get to the internet, and problems between the two, but once it hits the a network they can't control, all they can do is complain to those that do control it. However complaining to a provider about routing problems is like calling the fire station to report the fire station is on fire....they probably already know.
Last, it looks like this has been happening for a while. I see reports going back to 2014 showing issues specifically getting to Chicago servers. See forum post here. Here is one from 2016 that goes into more depth about the problem.

guardor · Answer

[*I had to make several edits to this post due to formatting issues, please refresh your window for latest response.]Below I have included the older trace routes as well as a couple relevant email responses I received from the NFO team.If you can dissect it and determine a course of action then that would be amazing. I've made a few notes in [*Brackets like this].As a side note to everything so far I have recently experience serious packet loss while playing on these servers. So much so that it has kicked me from the game about 3 times now. All the while my internet connection remains intact but only well enough browse websites.\\\\\\\\[*Early e-mail response in regards to the trace-routes below.]It's strange that your ISP is using Comcast to reach us. I haven't seen an ISP besides Comcast do that before, and we prepend Comcast to discourage it. But, that doesn't seem to be the real problem. The real problem seems to be on the outbound, in that your ISP is overloading its connection to Level3 in or around Phoenix, and they are advertising their prefixes in an unusual waythat is preventing our route optimizer from being able to correctly see that the Telia-&gt;Level3 path is poor. I've made a manual adjustment now to our system and it is re-optimizing the path to you. You should see animprovement within just a few minutes.[*Note: I replied to them that the fix worked and they made a mention that they hope Cox upgrades its peering link.]
\\\\\\\\[*A few months later I posted the newer trace-route and they had this to say]This one isn't the same problem as before, actually. It's another problem with Cox, but it's a different one. Previously, Cox was overloading its connection to Level3 in Phoenix, so I had to force our system to avoid using Level3 to reach Cox. That rule is still in place. Tonight, Cox was overloading its connections to HE, Tata, and some other providers in Chicago, and that wascausing high latencies. This is a new problem that fundamentally isn't on our end, but I'm going to see what I can do to try to work around it. I'll have to get more data and do further testing while the problem is occurring. I also encourage you to reach out to Cox for an ETR on a fix.[*In their latest response to me they mentioned that this issue had something to do with an HE peering link.]\\\\\\\\Tracing route to c-216-52-148-245.managed-ded.premium-chicago.nfoservers.com [216.52.148.245]over a maximum of 30 hops:&nbsp; 1&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp; 192.168.0.1&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp; 8 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp;&nbsp;&nbsp; 13 ms&nbsp; 10.32.0.1&nbsp; 3&nbsp;&nbsp;&nbsp; 26 ms&nbsp;&nbsp;&nbsp; 16 ms&nbsp;&nbsp;&nbsp;&nbsp; 8 ms&nbsp; 100.127.68.168&nbsp; 4&nbsp;&nbsp;&nbsp;&nbsp; 8 ms&nbsp;&nbsp;&nbsp; 12 ms&nbsp;&nbsp;&nbsp;&nbsp; 9 ms&nbsp; 70.169.74.52&nbsp; 5&nbsp;&nbsp;&nbsp; 24 ms&nbsp;&nbsp;&nbsp; 19 ms&nbsp;&nbsp;&nbsp; 40 ms&nbsp; langbprj01-ae1.rd.la.cox.net [68.1.1.13]&nbsp; 6&nbsp;&nbsp;&nbsp; 21 ms&nbsp;&nbsp;&nbsp; 21 ms&nbsp;&nbsp;&nbsp; 22 ms&nbsp; be-204-pe02.600wseventh.ca.ibone.comcast.net [173.167.57.109]&nbsp; 7&nbsp;&nbsp;&nbsp; 23 ms&nbsp;&nbsp;&nbsp; 24 ms&nbsp;&nbsp;&nbsp; 24 ms&nbsp; hu-0-5-0-1-cr02.losangeles.ca.ibone.comcast.net [68.86.83.21]&nbsp; 8&nbsp;&nbsp;&nbsp;&nbsp; *&nbsp;&nbsp;&nbsp;&nbsp; 2672 ms&nbsp; 2704 ms&nbsp; be-10915-cr01.sunnyvale.ca.ibone.comcast.net [68.86.86.97]&nbsp; 9&nbsp;&nbsp;&nbsp; 50 ms&nbsp;&nbsp;&nbsp; 50 ms&nbsp;&nbsp;&nbsp; 52 ms&nbsp; be-10919-cr01.1601milehigh.co.ibone.comcast.net [68.86.85.154]&nbsp;10&nbsp;&nbsp;&nbsp; 51 ms&nbsp;&nbsp;&nbsp; 63 ms&nbsp;&nbsp;&nbsp; 53 ms&nbsp; be-11719-cr02.denver.co.ibone.comcast.net [68.86.86.77]&nbsp;11&nbsp;&nbsp;&nbsp; 67 ms&nbsp;&nbsp;&nbsp; 68 ms&nbsp;&nbsp;&nbsp; 69 ms&nbsp; be-10517-cr02.350ecermak.il.ibone.comcast.net [68.86.85.169]&nbsp;12&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp;&nbsp;&nbsp; 67 ms&nbsp; hu-0-13-0-4-pe01.350ecermak.il.ibone.comcast.net [68.86.85.2]&nbsp;13&nbsp;&nbsp;&nbsp; 67 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 67 ms&nbsp; 96-87-8-110-static.hfc.comcastbusiness.net [96.87.8.110]&nbsp;14&nbsp;&nbsp;&nbsp; 70 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp; border5.po2-bbnet2.chg.pnap.net [64.94.32.74]&nbsp;15&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp; inap-b5.e3.router1.chicago.nfoservers.com [216.52.143.254]&nbsp;16&nbsp;&nbsp; 150 ms&nbsp;&nbsp; 152 ms&nbsp;&nbsp; 154 ms&nbsp; c-216-52-148-245.managed-ded.premium-chicago.nfoservers.com [216.52.148.245]
Trace complete.\\\\\\\\Tracing route to c-216-52-143-163.managed-ded.premium-chicago.nfoservers.com [216.52.143.163]over a maximum of 30 hops:&nbsp; 1&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp;&nbsp;&nbsp; &lt;1 ms&nbsp; 192.168.0.1&nbsp; 2&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp;&nbsp;&nbsp;&nbsp; 8 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp; 10.32.0.1&nbsp; 3&nbsp;&nbsp;&nbsp;&nbsp; 9 ms&nbsp;&nbsp;&nbsp;&nbsp; 9 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp; 100.127.68.168&nbsp; 4&nbsp;&nbsp;&nbsp; 11 ms&nbsp;&nbsp;&nbsp; 10 ms&nbsp;&nbsp;&nbsp;&nbsp; 7 ms&nbsp; 70.169.74.52&nbsp; 5&nbsp;&nbsp;&nbsp; 23 ms&nbsp;&nbsp;&nbsp; 20 ms&nbsp;&nbsp;&nbsp; 21 ms&nbsp; langbprj01-ae1.rd.la.cox.net [68.1.1.13]&nbsp; 6&nbsp;&nbsp;&nbsp; 22 ms&nbsp;&nbsp;&nbsp; 23 ms&nbsp;&nbsp;&nbsp; 24 ms&nbsp; be-204-pe02.600wseventh.ca.ibone.comcast.net [173.167.57.109]&nbsp; 7&nbsp;&nbsp;&nbsp; 24 ms&nbsp;&nbsp;&nbsp; 23 ms&nbsp;&nbsp;&nbsp; 22 ms&nbsp; hu-1-0-0-0-cr02.losangeles.ca.ibone.comcast.net [68.86.88.9]&nbsp; 8&nbsp;&nbsp;&nbsp; 37 ms&nbsp;&nbsp;&nbsp; 37 ms&nbsp;&nbsp;&nbsp; 36 ms&nbsp; be-10915-cr01.sunnyvale.ca.ibone.comcast.net [68.86.86.97]&nbsp; 9&nbsp;&nbsp;&nbsp; 52 ms&nbsp;&nbsp;&nbsp; 52 ms&nbsp;&nbsp;&nbsp; 50 ms&nbsp; be-10919-cr01.1601milehigh.co.ibone.comcast.net [68.86.85.154]&nbsp;10&nbsp;&nbsp;&nbsp; 52 ms&nbsp;&nbsp;&nbsp; 55 ms&nbsp;&nbsp;&nbsp; 52 ms&nbsp; be-11719-cr02.denver.co.ibone.comcast.net [68.86.86.77]&nbsp;11&nbsp;&nbsp;&nbsp; 66 ms&nbsp;&nbsp;&nbsp; 68 ms&nbsp;&nbsp;&nbsp; 68 ms&nbsp; be-10517-cr02.350ecermak.il.ibone.comcast.net [68.86.85.169]&nbsp;12&nbsp;&nbsp;&nbsp; 64 ms&nbsp;&nbsp;&nbsp; 68 ms&nbsp;&nbsp;&nbsp; 65 ms&nbsp; hu-0-14-0-6-pe01.350ecermak.il.ibone.comcast.net&nbsp; [68.86.86.122]&nbsp;13&nbsp;&nbsp;&nbsp; 68 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp; 96-87-8-110-static.hfc.comcastbusiness.net [96.87.8.110]&nbsp;14&nbsp;&nbsp;&nbsp; 75 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp; border10.po1-bbnet1.chg.pnap.net [64.94.32.22]&nbsp;15&nbsp;&nbsp;&nbsp; 65 ms&nbsp;&nbsp;&nbsp; 68 ms&nbsp;&nbsp;&nbsp; 66 ms&nbsp; inap-b10.e5.router1.chicago.nfoservers.com [64.74.97.254]&nbsp;16&nbsp;&nbsp; 156 ms&nbsp;&nbsp; 157 ms&nbsp;&nbsp; 159 ms&nbsp; c-216-52-143-163.managed-ded.premium-chicago.nfoservers.com [216.52.143.163]Trace complete.

tecknowhelp · Answer

Haven't had time to read the entire post yet, but what concerns me most is when they say "The real problem seems to be on the outbound,. What outbound do they mean? Outbound from them? If so, that route is decided by them. It seems like the engineer is seeing congestion between them and the different routes they use for outbound and blaming that on Cox. Why isn't he commenting about the 100ms latency that is happening in their network?
Also, they say there was latency between Cox and Level 3, but none of the tracert show Level3 peering. Can you show the tracert that the engineer was referring to when stating what he said?
Last, I think it would help to get different people's opinion on this so it's not just me (as Cox) pointing the finger at NFO and NFO pointing the finger at Cox. Try posting a thread on DSLReports and link to this thread. There are engineers, both official Cox registered ones and anom. floaters so I think they will be able to comment and possible fix &nbsp;the issue if there IS a issue on Cox side. Not saying it isn't possible there is a issue on Cox end, I just haven't seen the data for it yet.

Forum Discussion

HE Peering Issue

9 Replies

Related Content

Peering Partners

Cox having peering issues with GTT and Tata?

Connection Issues

Major Latency Issues

Disney + Access Issue

Recent Discussions

Welcome to the Archives

Gigablast

Beeping Noise

mic/status led on contour 2 remote

Internet connectivity frustrations