YouTube connectivity issues

How long of a answer do you want. ::chuckle:: This is a issue quite a few of us have been having. Most of the people have been in AZ or West Coast at least, but the reports are varied. I can tell you quite confidently that its not throttling. Simply because I have seen reports of the issue across multiple ISP's, and its unlikely multiple ISP would throttle the same content at the same time. So it must be something that all have in common, which suggest its not a peering issue either, like they sometimes have with Blizzard. Several ISP, including TWC and Verizon had the problem, but some how fixed it, according to the forum posts. So that tells me it is something the ISP can fix in some way.

What I think it is Cox offers Google to host their content servers on Cox's network. This is mutual beneficial to both Google and Cox. Google gets their content (and thus their ad revenue) closer to the viewer, and Cox saves on bandwidth over the backbone. Think of these Content Delivery Servers (CDN) like convenience stores in your town. I think Cox offers these CDN a certain amount of bandwidth, and past that Google should load balance around them to Google's main servers. These CDN servers are run by a company called Akamai. How everything works and who gets paid what, I am not sure of, but I think that is where the bottle neck is. In some area's Cox is forcing a workaround by changing their DNS to use different IP's for content, but that longer route can have it's own problem. That is why your VPN fixes the problem too. Also, some areas don't the user concentration for it to cause a bottle neck at the CDN's. I think that is why AZ is effected. If I had to guess, that is one of Cox's markets with the most amount of people per square mile.

I don't think it's a Cox issue, I just think Cox knows what the problem is, but doesn't want to publicly blame such a large company like Google. Don't get me wrong, I still think it warrants alot of debate.

PS. In the linked screenshot, I approve of the SAB. 

Thanks for the reply, this sheds more light on the issue.  I just wish that Cox would do something about it so that I don't have to VPN into another country to watch Youtube vids...

Well if you can run Netstat in a command prompt while streaming one the problem videos, it should show you which server you are using when hitting the bottleneck. If you can find that range, and then block it (either your HOST, registry, DNS,etc) there, you won't have to use a VPN atleast. Still not the elegant solution though. I wish the other ISP would announce what they did to fix the problem. The fact that its all kind of hush hush probably means there is alot of money involved. But thats just a guess.

Hi Freakintoddles,

I have experienced this issue myself and understand how frustrating it is! Trikein is correct; we have been getting reports from folks regarding YouTube streaming problems for the last month or so. While this issue is happening to folks connecting through varied ISPs, we have a team engaged so we can make sure we are doing everything we to address the problem on the Cox side.

I know you mentioned that other video streaming sites work without the same difficulties; do you notice any other internet issues? Have you run any speed tests to confirm your download speed?  Access the Cox Internet Tools Speed Check Tool HERE.

Thanks for your post and for the information you have provided. Every new piece of data gets us closer to putting all the puzzle pieces together! As soon as we have further info, we will let you know.

I'll chime in.  Ran the speed check tool:

Start	Cox Net	Download Speed	Upload Speed	Latency	Your IP Address	Cox Market
Mon Sep 23 7:58 PM	Yes	91 Mbps	33 Mbps	24 millisec	70.162.82.140	Phoenix

On some other forums people seemed interested in running netstat in dos during youtube streaming.

It seems that most of my youtube-related connections are a variant of lax************.1e100.net, such as:

TCP 192.168.0.104:62492 lax02s01-in-f15.1e100.net:https ESTABLISHED
TCP 192.168.0.104:62493 lax04s08-in-f8.1e100.net:https ESTABLISHED
TCP 192.168.0.104:62494 lax04s08-in-f8.1e100.net:https ESTABLISHED

That being said, like freakintoddles, I ran my traffic through a VPN/proxy (mines in England) and 1080p streams beautifully, showing similar netstat results.

I'm not complaining here... I just want to help!  Thanks!

And if this means anything... tracert: first via normal connection and then via proxy:

Microsoft Windows [Version 6.1.7601]
Copyright (c) 2009 Microsoft Corporation. All rights reserved.

C:\Windows\system32>tracert lax04s08-in-f8.1e100.net

Tracing route to lax04s08-in-f8.1e100.net [74.125.224.232]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms dlinkrouter.ph.cox.net [192.168.0.1]
2 9 ms 9 ms 7 ms 10.35.192.1
3 10 ms 8 ms 7 ms 172.21.0.82
4 12 ms 11 ms 11 ms 70.169.73.196
5 37 ms 10 ms 9 ms 70.169.75.157
6 * * * Request timed out.
7 44 ms 43 ms 46 ms 72.14.215.221
8 23 ms 21 ms 23 ms 209.85.248.187
9 31 ms 23 ms 23 ms 216.239.43.76
10 76 ms 75 ms 72 ms lax04s08-in-f8.1e100.net [74.125.224.232]

Trace complete.

C:\Windows\system32>tracert lax04s08-in-f8.1e100.net

Tracing route to lax04s08-in-f8.1e100.net [74.125.224.232]
over a maximum of 30 hops:

1 160 ms 167 ms 158 ms *.*.*.1
2 162 ms 169 ms 161 ms *.*.*.129
3 162 ms 160 ms 160 ms *.*.*.77
4 164 ms 159 ms 166 ms te0-6-0-0.ccr21.lon01.atlas.cogentco.com [154.54
.72.185]
5 161 ms 159 ms 160 ms te1-1.ccr01.lon18.atlas.cogentco.com [130.117.51
.162]
6 161 ms 159 ms 165 ms 149.6.146.30
7 159 ms 159 ms 178 ms 209.85.255.76
8 164 ms 160 ms 163 ms 209.85.253.92
9 246 ms 255 ms 242 ms 66.249.94.246
10 242 ms 243 ms 250 ms 72.14.239.93
11 261 ms 250 ms 250 ms 66.249.95.231
12 256 ms 257 ms 297 ms 216.239.48.4
13 317 ms 314 ms 323 ms 72.14.239.83
14 312 ms 311 ms 315 ms 64.233.174.191
15 316 ms 319 ms 319 ms 216.239.43.76
16 314 ms 314 ms 315 ms lax04s08-in-f8.1e100.net [74.125.224.232]

Trace complete.

According to one Cox employee, the problem is with Google's servers.  HOWEVER, I agree with you.  I have Cox high-speed internet at home AND at work, and almost can't bother trying to watch youtube videos at either site.  Youtube via my Verizon 4G LTE connection is much much faster.

I'm having the same issue at home using my computer and my cell phone (over WiFi). If I turn off the WiFi on my phone I can stream HD videos over my Sprint 3G connection just fine (I live close to a tower). There is obviously a problem if I can stream video over 3G and not my high speed preferred connection.

Becky, can you comment further based on these responses?  Would love to know what Cox is doing to correct the issue and what a reasonable ETA is for a fix.

The problem is Cox DNS servers automatically route everyone to the slow/overloaded youtube CDNs when there are plenty of faster youtube servers available. Sometimes it is so bad that you can't even watch 240p without buffering but if you use a proxy site like http://www.proxfree.com/youtube-proxy.php you can load videos 1080p no problem because you can get routed to a faster server. So yes the fault is with Cox cause it is their DNS servers sending us to bad youtube CDNs. Also these bad CDNs usually are in the cox network and it's their responsibility to upgrade them to handle the demand if they are going to keep sending everyone to them. I think the problem is you get routed to the closest server or something which will be assumed to be the fastest. It's like a GPS that only give you the shortest road, but if there's a traffic jam on that road it doesn't take it into account, sometimes taking the longer less traveled road will be faster. I guess the technology still needs improvement if its sending us to bad server when hundreds of better ones are available, all because cox wants to save on traffic leaving their network.

While you bring up some very good points, I disagree with your overall conclusion. I don't think it's so much an DNS issue as it is a bottleneck issue, as you stated, at the CDN. Exactly who's responsibility is that depends on why the bottleneck exists. Im pretty sure Cox only offers so much bandwidth to them for free (probably proportionate to the bandwidth they save Cox from going over the backbone?) and past that I think it's Akamai's responsibility to buy more. Akamai is probably routing as much traffic to them as possible since they kind of paid based on how much content they host. I say this because I actually hard the problem worst when I was using Google DNS. I think Google controls what goes to the CDN servers, and what doesn't, and how much bandwidth is in each area, and who controls the servers (probably not always Akamai) and a bunch of other factors.

In short, what I think Cox needs to do is politly hint for Google to buy more bandwidth from Cox for Akamai to host their CDN servers on Cox's network. Also, Google needs to get better at load balancing between the global cache and the ISP cache.

I can confirm that cox is throttling the Home market.

I'm a Sr Systems Administrator for a software company in Chandler, AZ... We use cox Fiber at our office with a 100mbit circuit. at Home I have a 50mbit cable connection.

All services to google (youtube, search, image search, etc) are effected from this throttling.  

If I try and load a youtube video from my house, it will buffer and load very slow, If I try and view image search and scroll thru the list, all images will load slow.
Now if I route my traffic over VPN to use my Cox office connection all the same actions work perfectly, youtube is instant, image search works faster than I can scroll.

Checking the IP's my machine took while on VPN and off VPN, they take the same route to (74.125.X.X) <-- googles CDN's

So this shows that the CDN's are not overloaded, Cox network is not overloaded, only the Home market is throttled.

Regards,

This is very good information. Could you provide some tracert from both your Fiber and cable connection so we can compare? I would suggest removing any personal information of course. 

What do you suspect is going on? If its simply just a throttling issue, why has the problem been happening to different ISP? And who not in all areas? 

Sure thing, Traceroutes Below:
Shortly after posting this, my Ping times to 74.125.224.199 have dropped significantly from 130-180ms to around 20-30ms
Testing youtube, and google image search and everything is working as it should... Fast load times now.

Maybe someone saw this post :)

Keep in mind that ping times over cox fiber are still 10-15ms even tho it has 15 hops
Also the original route my home router took has changed... I was not 9 hops like it is now.. too bad I didnt get that traceroute :(

As far as why different ISPs are having the same issue, it boils down to network saturation of these types of services.  I'm in a sense an ISP for my users at work.. I will throttle different services like Netflix, Pandora, YouTube, etc because if many users do these things during business hrs it would kill our network, and then no one would be able to work.  Its the same thing with Cox, and other ISPs... they will choose to slow down specific traffic to help save the rest of it. Mainly most people don't mind if a YouTube video takes a while to load, but if Netflix does not work, that would drive them crazy and the ISP and Netflix would get calls... but you notice Netflix usually does not have any streaming issues the same time YouTube does.  Kinda hard to call Google and say hey... YouTube is slow, fix it! :)

My point is, I totally understand why they do it, but the fact that I could run a speed test at full 50+mbit, download an Itunes movie or stream netflix no problems at full speed, but have an issue loading even 240p clip on youtube is crazy.

From Cox Home

1 <1 ms <1 ms <1 ms [192.168.1.1]
2 10 ms 6 ms 7 ms *
3 9 ms 10 ms 8 ms 172.21.1.80
4 10 ms 10 ms 11 ms 70.169.75.153
5 30 ms 23 ms 21 ms langbprj02-ae2.rd.la.cox.net [68.1.1.19]
6 43 ms 43 ms 45 ms 72.14.215.221
7 23 ms 24 ms 101 ms 209.85.248.185
8 24 ms 23 ms 22 ms 72.14.236.13
9 21 ms 23 ms 32 ms lax02s02-in-f7.1e100.net [74.125.224.199]

VPN and Cox Fiber:

1 <1 ms <1 ms <1 ms 10.10.10.1
2 1 ms <1 ms <1 ms *
3 <1 ms <1 ms <1 ms *
4 2 ms 2 ms 1 ms *
5 1 ms <1 ms <1 *
6 1 ms 1 ms 1 ms *
7 1 ms 1 ms 1 ms *
8 1 ms 1 ms 1 ms *
9 2 ms 3 ms 3 ms 70.169.73.90
10 1 ms 1 ms 1 ms 70.169.75.153
11 14 ms 14 ms 14 ms langbprj02-ae2.rd.la.cox.net [68.1.1.19]
12 14 ms 39 ms 14 ms 72.14.215.221
13 14 ms 14 ms 14 ms 209.85.248.185
14 14 ms 14 ms 14 ms 72.14.236.13
15 14 ms 13 ms 14 ms lax02s02-in-f7.1e100.net [74.125.224.199]

The fact that the route changes and then it gets better says to me it's not throttling, since you wouldn't need to change the route to throttle a service. Also, I find it unlikely that all the ISP would start throttling Youtube around the same time, unless they met up and decided to all do it at once, but thats a stretch. I mean your technical data is good, I just don't understand your conclusions. Could you explain?

Also, what do you know about CDN servers and do you think it's possible they are playing a role?

I can't speak for your particular connection, but here has been my experience with Cox & YouTube. For the past year and a half or so I wasn't able to stream 1080p, 720 was about as good as I could get, but even then I would get buffering from time to time. About a month and a half ago something changed, my friend and I assumed it was on YouTube's side as I wasn't able to stream 1080 at home OR at work (which uses a completely different provider.) After whatever upgrade we assumed happened on google's end, the speeds have dramatically improved at both home and work.

So no, I don't think Cox was throttling me in my case, but the fact that you are able to proxy around the speed issue AND stream just fine with Verizon makes me think there may be something afoot here. Are you still experiencing the slowness now?