Here. There. Everywhere.
High-quality streaming for hard-to-reach subscribers.
Tech Talk Introductions
Really pleased to be here. I’m really pleased for Lou to join me on stage to talk about some of the challenges with getting what we’ve been talking about all day actually delivered to people out on the internet.
So we’re going to talk about what it means to get the content that you’re all pushing over the top down to people who live far out in the networks and are served by ISPs like Lou here.
So my name’s Steve Miller-Jones. I am the VP of product strategy for a company called Netskrt.
We’re an Edge-Native CDN. So we have a CDN that is designed to work in the far outreaches, the outskirts, if you like, of the internet. We put our CDN into places like airplanes, passenger environments, trains as well. And we also put our CDN into Tier 3 ISPs out in the rural and more remote or even extra urban parts of the internet.
And I’m really pleased to be joined by Lou here. Lou, maybe you could just introduce yourself.
Hi, I’m Lou Gutzman. I’m Director of Network Engineering and Operations at Greenlight Networks. We’re based out of Rochester, New York. We are a fiber-to-the-home internet provider, providing primarily data services only. So everything is delivered over the IP network.
The Challenge of Delivering Live Sports Streaming Over the Top
Cool. So what we’re going to do, I’m going to talk for a little bit about the problem, the challenge of delivering sports over the top. And then we’re going to have a chat with Lou about his network, how it works, what kinds of challenges does a large sports event introduce and what does that mean.
So what we know, of course, is that there is an enormous amount being spent on sports rights in the last year and the last couple of years and going forward. Whether or not they get repeated each year is kind of moot. There’s an enormous amount of money being spent on sports rights that are going into OTT streaming.
Record-breaking Live Sports Events Over the Internet
Last year, 2023, we saw some of the peak internet events. The IPL out in India drove, I think, the top five biggest streaming events ever in history. Certainly the NFL in the USA, more close to home, drove some pretty significant spikes of traffic as well. And as we’ve been hearing today, online sports is also bringing younger audiences to market as well. And [so] we’re ending up with a series of rights out in the marketplace that are going to drive live streaming down into the networks that have to deliver them to end-users.
[What] sort of challenge does this present to us? The real challenge that we have here is that on the right-hand side […] we’ve got our consumer in their home watching on a TV or a tablet or some kind of device. And, essentially, on the left-hand side, the content platforms are up in the cloud environment. And [they’re] today likely to be delivering their content to you using a CDN, which is located at the same location as carrier networks, maybe transit networks, IXs, and peering points in the internet.
And [depending] on where you are in the country and where you live, there’s either a 20 milliseconds to over 100 milliseconds gap between you and the consumer. And there’s all sorts of networks in between the CDN and the end consumer.
In the ISP case, the ISP is operating some kind of access network that people are connected to. I have my home router. It’s connected to something. I’m connected to an access network. And they’re typically less than five milliseconds away from the end consumer. And so this is an obvious place to serve the user from, because it’s close to them. But it’s not necessarily where this is happening today.
And so it really sort of presents a set of challenges to the content owner:
- [Are] they getting to all of their audience?
- Can they validate that they’re reaching people?
- So what sort of numbers are we talking about?
13-15% of the US population are served by Tier 3 ISPs
In the USA today, there are just about 112 million broadband subscribers. This was data from last spring 2023. There’s just shy of 3,000 ISPs in the USA. You can see the sort of breakdown between cable, wireline, and fixed wireless. An interesting stat that’s not on here is the fixed wireless and the wireline piece is kind of growing. The cable is a bit less, but the growth is in fiber-to-the-home opportunities and fixed wireless broadband as well. The real interesting thing here and what we’re looking at is that about 13% to 15% of the population of the USA live in remote areas and are served by so-called Tier 3 ISPs.
And they have to not only provide the access to their home, but they have to get their network connected to the upstream parts of the internet as well. And so this is where some of the challenges lie in ensuring high quality of experience (QoE as we know it) to those subscribers, as well as people who live in a highly populated, more densely populated area.
You know, if you’re looking at your sort of histogram of performance, everyone’s up and to the left, that’s most of the population. There’s still a long tail of people who experience errors or rebuffering events, or who have a lower than ideal quality delivered to them, and the bit rate they’re getting is low versus what they can feasibly get through their connectivity.
And so these are the sets of subscribers that we’re trying to solve the challenge for. How do we make sure and ensure with some form of validity and validation that we can get the high-value content to these underserved locations with scale and efficiency? That’s the problem that Netskrt are trying to solve.
The Edge-Native CDN and Greenlight’s Network
Lou here is one of our ISP customers. We have our Edge-Native CDN deployed into Lou’s network. We have three of our caches deployed out there, and we’re serving his end-users. We’ve been doing that over the last (sort of eight) to ten months.
Lou, as a starting point, maybe, if you don’t mind, you could sort of talk a bit about the kind of offering that you have to your customers? Talk me through what they’re getting, and also then what’s your network connected to as well?
Sure. So our end-users or our subscribers are primarily residential. And the offering that we have is symmetrical speeds up to 8 Gbps to that end-user. Challenges to get them the content is really cost-driven, right? So, because we’re a Tier 3 provider, we’re not peering directly with a lot of the content providers. To go into a carrier hotel, it’s very costly. So a lot of times we’re just paying for bandwidth from our tier one peering partners. So that drives a lot of the cost.
Managing Peak Traffic Hours at Greenlight Networks
Right, right. So the challenge becomes live content, right? So if you have live content, like we use NFL Prime on Thursday nights, for example, that’s specifically streaming only, right? So there’s no over-the-air, there’s no satellite. It’s Amazon Prime streaming only.
So you have to account and try and forecast what that bandwidth is going to bring in, right? So it presents some challenges because you really don’t know. Like, some games are a little bit jump, and then, you know, being from the Rochester, Buffalo, Binghamton, Western New York market, you know, when the Bills are on TV, it’s a 30% to 40% increase in your typical night bandwidth so you know it drives cost right because you have to account for for bandwidth so it’s it’s not only is it paying for you know subscribing to additional bandwidth from your your tier one provider but it’s also hardware costs to to build the network to sustain that bandwidth.
Well, yeah. So when COVID hit, a lot of students, the kids in school, worked from home. A lot of the parents were working from home. So at the time, Greenlight was offering asymmetrical speed. So our base product was 500 by 50, 500 bag by 50. And as a courtesy to our subscribers, when that happened, we basically just offered them symmetrical speed at no cost.
From a data perspective, you saw that maybe bandwidth trends changed a little bit. You saw an increment in daytime bandwidth, but it still didn’t touch the content-driven bandwidth at night, right?
The Impacts of Live Sports Events on Tier 3 ISPs
Okay, so we’ve got a bit of a background of what’s happening in Lou’s network. I’ll sort of leave this up while we talk through a few more things.
You kind of touched on it briefly there, but how do you feel and what do you see from live sporting events, in particular, in your network, what kind of impacts do they have for you?
It’s a lot of stress. Right? Being responsible for that end-user experience. I mean nothing’s going to impact or give you negative reviews faster than not being able to provide the bandwidth. Then the next step is that latency – how many times have you had an app on your phone and the score gets updated on your phone before the play happens on the TV?
It’s frustrating for our end-users.
As an ISP, you’re kind of not in control of what the streaming platforms are doing, but everyone’s going out of your network. They’re all connecting to something that’s upstream from you, and you’ve got to handle all of that coming in. I think one of the games in the last season was pretty significant. What kind of difference did you see? One of the Peacock games was pretty big. How did that impact what users were doing? How did it impact how you had to manage it that day?
If you’re efficient at your job, you’re really managing that in advance. The night of the game, it was kind of like, okay, what are we going to get to?
It’s really kind of like, wow, what limit are we going to push to? The planning and the preparation is done in advance. You look at those types of games and those types of events, you’re really planning those months in advance or weeks in advance because you have to.
You can’t just add 100 Gbps links to your peering partners at the drop of a hat.
So a long lead time, if you wanted to go, I just need to lay in an extra 10 Gbps today. You need to have thought about that a few months before.
Other Streaming Events That Drive Traffic Spikes
You do. We talked earlier and certainly like major TV events, I’ll say like Game of Thrones, right? Game of Thrones drives a lot of bandwidth, right?
The difference between Game of Thrones and say like the Peacock streaming or NFL exclusive and the Thursday Night Prime is those are streaming platforms only, whereas HBO, you can get over satellite. You can get over some different things. But you still see that spike, right? You still see that incremental bandwidth because a lot of people are streaming. They are paying for that app.
And so those big events, I think we talked earlier a bit about the numbers, like on a normal Peacock day or a normal football game day, like Monday nights, you’re seeing 300 Gbps or something upstream?
Normal bandwidth for us across our environment is around 300 Gbps. On an average lately, with a Peacock playoff game streaming, it was over 400 Gbps.
Okay, so adding a third during a significant event. And again, Lou’s not alone here. There are 3,000 two or three ISPs doing the same thing. So it’s individually, okay, we’re just managing this, but collectively these numbers start to be meaningful, significant as well for the sports programs.
So, how do you strategize dealing with this? I mean, there’s the business side, the financial thing. Your population isn’t going to grow by 50% in 10 years, right? So you don’t need to over-provision enormously, but I’m sure you have […] options for how you might approach this kind of problem.
Right. We’re at the outskirts of the Internet, so that’s what that slide tells us. And how does that fit into your sort of business cost, operational if you have to go and lay in some wavelengths, like symmetrical wavelengths? Instead of just having some direct connectivity, you’re going to have to increase your prices to your consumers.
So what kind of business challenges does that give you?
So bringing that CDN appliance in, you’re really downloading that content once, right? And then you’re distributing it from the inside of your network. So the bandwidth itself doesn’t change on the inside. Because the end-user is still using that bandwidth. But where it saves you, as an ISP who wants to make money, It really saves you on buying that external bandwidth. It adds up when you’re talking, you’re paying overcharges on 100 Gbps three, four times a month.
So it really does save a lot of the cost. And it really helps simplify some of the bandwidth planning that you’re doing. You’re really managing the inside of your network and making sure that your internal rings and your links have enough bandwidth to sustain this consumer request, but maybe not so much on the outside. Maybe you have the aggregate bandwidth there, but you’re really only subscribed to maybe 40 or 50% of that so that you’re not really expending that cost.
And are you seeing, everyone probably knows Netflix do a thing with a box. We’re obviously doing what we call this Edge-Native CDN, which is designed to manage content effectively, and we do a lot of work to understand what’s happening in your network so we know how to connect to origins effectively to get the content.
Are you seeing others come in with so-called Edge CDNs as well? What do you see?
I mean, live is difficult. Everyone’s using a lot of different devices to stream, so yeah, absolutely.
My timer tells me we’re out of time. That one says we have three minutes.
Any questions out for the floor while we’re still here? From the floor, even. I can’t see anything, so if anyone’s got their hand up, I’m sure there’s a mic in the room somewhere.
And if there aren’t any questions in the room, then I will say thank you, Lou, for [coming here] to talk to us about what you’re doing in your network and how the business of sports streaming is a challenge for you and one of the solutions that together we’re using to try and help solve that.
[Conclusion]
