These are unedited transcripts and may contain errors.
Plenary Session
9 a.m. - 14 May, 2013
CHAIR: Good morning everyone, and welcome to the second day of the RIPE Meeting. First up this morning, we have got Dave Wilson from HEAnet of this fair land and he is going to talk about virtualising our CPE with open ??
DAVE WILSON: Welcome to Dublin, I'm delighted to see you all here. Delighted to finally welcome you. I am Dave Wilson, I work in the network and development department of HEAnet and NRENs have been up to stuff for a while and I'd like to tell you about one of the things we have been doing recently, which is a piece of software called OpenNaaS, we have been doing this with a number of partners including [Idocas] in Barcelona.
The problem we're trying to solve is this: We're a national research network and while we have a reputation for, you know, forward looking and research networking, we're also, most importantly, the ISP to our clients and our clients universities these days, and further education institutions, are basically enterprises, they have enterprise needs, they need an enterprise reliability, they use enterprise style equipment. We have to make sure that that works and that works well.
So, we're under the same sort of reliability pressures and the same sort of technology requirements that you see elsewhere.
So we took a look at how we deliver our service, we took a look at each piece of equipment that we use, and we went what are they all for? What's a firewall for? Okay, that's an easy one, the fair wall implements a security see. You make a policy, you implement it. What's a switch for? These littering the coms rooms of our clients as well. The switch connects a network at a given speed and if you want a fast are network you need a faster switch. Very well. Transmission equipment, what's that for? That's for speed over a given distance and if you need to go a greater distance you need more powerful lasers, all very well. What's a router for? Exactly in the sense does it do?
It roots packets. What does that mean? What's the purpose of that? I started thinking, there must be an answer for this because you have quite a lot of them and they are quite expensive. Bandwidth. They don't well provide bandwidth because that's what the switch does. The connectivity. That's really what the transmission equipment is doing. It must be resilience, right? Not entirely. They do run the protocols that ensure that a network that is resilient operates that way, but if you want resilience you need two other things.
What they really do, when you come down to it, they provide separation. The clue is in the I of IOS, it's inter?networking and they ensure that we can manage our network, our clients can manage their network and the peers can manage their networks and we do not impact each other unduly eve. Which is fine. But all those jobs are being done by separate pieces of equipment and like to really see, it I pulled the trolly into our test lab there a while ago, and I started loading them up and I went okay, what equipment do we deploy at a typical client site in we put out our router, fine, and the client has to match that with their router to match the edge of their network, very well, fine so far. But over the last, you know, decade, 15 years, we have also seen the need for resilience in these connections, so we put another router out there and that must be matched by in our router. And these days, we deliver service that is aren't necessarily IP services, we want deliver layer 2 connections so we put a switch out there and they are important so let's put two out. Then we need a firewall put in place and if you don't want everything go down when that breaks, you get two of them. That's a lot of kit. That's too much. There must be something we can do about this, because this is against health and safety fire regulations so there must be something we can do about this.
So, we thought the problem we have here is threefold. The initial cost of these things just to buy the things, they are very expensive. And then when you buy them, operation maintenance is the kind of of gift that keeps on giving and you are paying some percentage of your capital cost every year in maintaining them. And then simply the operational management of this network that used to be quite simple and that has grown quite complex is a problem that needs to be solved.
But if all the routers are doing is separation ?? oh sorry, if all they are doing is separation, why can't we do that inside one router? Why can't we get one big router and put everyone on that? I think we all know the answer to this one because we have been through this, we have tried to reduce it, and you just run into trouble because everyone has different needs for how they connect to the network and what they run on the network and you can't collapse that into one box with one particular config, it runs into too much trouble trying to manage it and it ends up in a state where one client might get the services they need on their own internal network because that becomes a part of the income, but the other doesn't. We have to do something about that.
But you can virtualise software. And all these separation functions are just software. Why not virtualise it? We have been virtualising things for years. At the end of the day, virtualisation is give me a PC, make it to appear as several devices to several people. We can do that with routers, an obvious one is JUnos, allowed for logical routers directly on its config. And if you can do that, then we thought maybe we can do this with our CPE. Now this isn't a wholly new idea. Certainly, I mean SUNet have been doing this for years but we were thinking, we have these switches and routers on site. Let's take away the routers on site, let's get rid of the links between them and relays them with something aggregated closer to the core, serving the same purpose but virtualise and providing different function to say different clients clients.
Cool...
So, that means if you look at a single client, what we're trying to pull off mere is I have two physical routers, one of them is just down the road in our offices in the IFSC, the other is in the data centre in Park West. I want to create two logical routers, and then I need five things to connect them together. I need obviously two upstream link to say get them to the Internet, I need links in between them in order to manage fail over and just as obviously, we need links to the downstream networks.
I'll show you what we have done. I have a what you are looking at here is the web interface that we have built for this particular purpose to log into the software that we have. And let me just go through this quite quickly. We have a number of use cases we have been working on, I'm going to show you the virtual CPE one that I have been just been talking about here. This is a representation of the physical network that I just showed you, two physical routers with particular links in between them. Let me go to the good part.
This is a view of the logical network I'm about to create. This is going to use purely virtual resources. The software we're looking at here it's actually smart enough to suggest things like IP addresses and VLANs that it understands not to have been used by other systems. I have particular setups in my lab at the moment so I'm going to go through some of these manually, but if I create ?? I have here two upstream links. Two interfaces, fine, which connect to my core routers. What's the next thing you need to do in order to make a network work? If it's a client speaking to the provider, you need to set a BGP and we have the concept here that it should be possible to specify something once and only once. One of my most common mistakes is I have to type the same thing in three or four or eight places and I'll get it wrong at least once and perhaps seven times and we want to be able to avoid that. It provides a client's AS number and the upstream's AS number. Then we get into the logical routers themselves. The topology here is I always like to have back interface once my devices, so we'll set those up. This is going to be the primary virtual CPE, and this on the right, is going to be the backup virtual CPE. I'll show how those are connected together now.
They are each going to need, like I said, downstream links, and links in between. So I happen to know that we are prepared to provide VLANs here. These VLANs are not currently set up by the way, we'll be be setting this up in a moment and I'll show you how. But that's the topology we're going for here. How exactly are they connected? Let me expand this out.
This is what's going to make those connections between, those circuits don't currently exist. We have a downstream link for the primary, to the local LAN, we have a down stream link for the backup to the local LAN and we have a link in between. And we're going to provision those as well.
And then that's all very well from my point of view as a provider. But the client also, like I said, has configuration that is they need to set up. They are probably going to want to set VRP or some sort of stand by protocol. And they are probably going to have for example firewall filters that that they want to apply.
That takes a few minutes now to run and let me explain exactly what it's doing. I have ?? let me explain what it's doing now. Let me explain why we do it in this way. You saw the ?? we have two routers, and we use some sort of system which is quite mysterious and I haven't explained yet in order to set up the virtual resources. Well, there are three parts to this. We think, in order to do this well. And the first is on demand provisioning. Where we came from with this is, we got religion awhile ago when we set up a point to point network. When we have on demand provision, it solves the initial cost because we are able to set these things up in a virtualised way with sharing resources and in a very reliable way. So we get the cost is reduced and the maintenance cost because we are able to put less equipment in is also less reduced. Operational management is still recomplex. Setting up is fine but this network is still sitting there and even though you have all these perhaps fewer boxes in place, you still have to manage all these logical devices and it's no longer even the case that you can start pulling out cables and find out which one is going wrong because it's something inside a device. So you don't even have that comfort any more.
So, like I said, we got religion on this a few years ago when we set up our ethernet point to point network in HEAnet. This is so simple even I can use it, having long forgotten how to provision one of these things. It is a bunch of drop?down menus where I say I have a switch port over here and there, oh networks that one is already in use, I'll just use a different one. And what this got for us was a bunch of benefits.
In particular, we have these really easy changes which means change control as much more lightweight than it was before. You don't have to rely on someone like me to forget to update the inventory, it can do it itself. Same for monitoring. The best part is, we can expose an API. If we want to give someone else access to our network, we can. We'll come back to that in a moment.
We looked at this and we thought, thinks great, wouldn't it be great if we could extend these benefits through layer 2 but also to layer 3. And that's where we got involved in developing the tool that we have been working on. What this is, it's a framework for network as a service. What's a framework? The idea here is that you have this open source piece of software, we have both a command line interface and a web services API that you can use to access and the gooey you just saw is actually built on the web services API and is running off that. And the ?? it's integrated with our point to point network. But more than that. We have to do more than just provide on demand provisioning. There also needs to be sensible abstractions here. There is a lot we could do if we could, you know ?? there is a lot you can get away with by setting up a simple provisioning system like that, but I think when you bring in two layers at once, suddenly have two networks which are depending on each other which need awareness of each other and it's important when you are providing services off that to be able to abstract that out. Abstractions for who and who to? To whoever you want. If you want to provide abstracted service to your customers, that's who you provide the abstraction to. If you want the network to be able done figure these without worrying about the the nitty?gritty day to day, they can do that too. It's entirely up to you.
So, what we end up up with, when I say framework, we have these parts of the system but the really important thing from my point of view when I go in on a Monday morning is I have my physical equipment, I have these green resources which are exposed, I pull out the ones that I want to use for my service, which are going to be a subset of this and I expose that to the user as virtual CPE or whatever it is that I'm trying to do.
So like I said, you need your layer 3, but you also need your connectivity. How do we get the connectivity? We already had as I said the point?to?point network inside in HEAnet which is able to expose an API, they without just use that. Except we have something a little bit cool going on here because it's not just our own network, we also have a system inside [Jayant], where we are able to interact with other people's layer 2 networks. They are their own networks, I have mine and we operate off our own technology but we can accept requests from a sort of met a provisioning system if you like, where I want to make a connection not just between Dublin and Park West or between Galway and Limerick, but want to make a connection between Dublin and Copenhagen I can make that request and it will iterate through the different networks I'll need to reach that end point.
When it came to layer 2 for this system we thought we don't need to build layer 2, it's being done, we'll just provide and interface towards that and we'll use that to connect our equipment.
So, that's awesome. What's the third part? If we are providing this to clients, like I said, a little bit earlier, we find ourselves in a state where we are running equipment that has effectively on their network and they have particular needs which are different from client to client because it's it's propose appropriate to their network, so you must be able to delegate part of these resources to the client in such a way they can operate them independently. Things like for my own personal example, like we are using here. If we are operating CPE, even physical ones, never mind virtual CPE, even the ones we have out there it had, the problem we have to solve is there are parts of the job that that does, which are appropriate for the provider and which are specified by the provider and if you change these and the provider doesn't change the core router, it's only going to break. On the other hand, the client also has configurations that they need to apply to the same device because it's really sitting on the border of the two networks which, if I change, it's only going to break their internal network and we should be able to accommodate that.
So, let's see how that went. If I can ask to you switch back to the laptop for a moment. That operation is completed and I now have a network setup. To prove that is nothing up my sleeve, if I go to my ?? this is the logs for the provisioning system that was creating the point point to point circuits. This one here ?? I can't prove to you that the config is in there but there is a config in there. The way I'll prove it, in fact, I have a webcam set up on my desk and this is connected only through the network that I just created and I'm going to tear this down in a few minutes and that will start working. But that's the weather about 500 metres north of here. But ?? so what have we got there? That's all very well.
I'm able to, as a user, sitting on a stage in a hotel on a Tuesday morning, make configuration changes to my live network, because that's my live network recollect point to point network there is the same one that our customers is using for production today, one of the routers is in my test lab but the other is providing production services with other logical routers already on it. Not just, I'm able to configure that, but also, if I go up here and log out, I can log back in. Not as a network user but as an abstracted client and let's see what they can see? Again here you see my list of virtual networks which I have access, to that has one. And I can update that. You see a very different interface to what you saw before because these are the things appropriate to the client. My favourite part of this, by the way, being that, would you like OSPF? Yes or no? Remember what I said earlier about being able to specify everything once and only once, by the time we get down to this point we know what the prefixs are we know what the interfaces are, and we can really make an operation not in each individual interface but on the network, which is would you like an IGP? Yes or no? And only at that point if you need to make further detailed configurations that's where you start to make them.
So, if I can switch back to the slides.
Where are we? Well that's all very well, that's demos nicely, but the problem is why on earth should you deploy it and how do you get there from here? And I kind of have sort of two points of view on this. One of them is it's a quote from a guy who knew about automation, it's Henry Ford, he said if you need a machine and don't spend the money and buy it you will ultimately find that you have paid for your machine and you still don't have it. On the other hand, we have all seen a lot of things that demo pretty well, and you know, which make for great pilots but it's another story when it's trying to get them into real life. There is a senior executive in Visa I think was quoted on NFC as saying he has seen more pilots in the RAF. How do you get over that?
Well, the parts to this are, I think, you need to see some real deployment. That's what we are doing now in HEAnet, we are putting this in place and trying to get a hold of how well it's working and what we need to do to make a production. But we also have to be able to deploy it incrementally, that's something you must be able to do alongside our existing services, not throw them away, just move people to it as they get excited for it and if they are interested in itnd appropriate. If something goes wrong you have to be able to move them back. And just to be able to get people's hands on it and try it.
So, what we have today, and what I'd love to be able to get from you guys if at all possible, is this: We have this generalised software, with a flexible command line but with a sing will gooey which we have developed so far and my expectation is that people with different needs will either adapt a gooey that's there or develop their own one. But right now, if you go to the website today and hit download you'll get something that does virtual CPE. Has a simple OSPF, config, does simple BGP, and VRRP we thought this is a great place to start. People can understand it and get their heads around it, there is clear benefits to do that go in that that particular case. It's a great place to expand from, because we can go from here very easily to a point where if a client suddenly goes, phones us up and says, Dave, thank you for my virtual CPE but I need to extend my network, I have a site that suddenly needs to be connected in another city in Europe, can you help me out? Today I'd have to go, hang on a moment, I'll go, design a network for you, order some resources and I'll send you the bill. I'm able to go give me a moment, I'll draw a picture, will this do? Okay, I'll implement that. We see ?? I see this as kind of a trojan horse: once you have the virtual CPE working and that's got its own single benefit, you can expand out from there easily.
There's plenty of work to do. VIX is in there but we're not done with that. Things like the IGPs, OSPF, you are going to need to account for more complex topology than you necessarily find in our pilot projects. So, I'd really love to get this in people's hands. And I'd really love to help ?? get your help developing this as we go on with this.
What we have is, it's actually an FP7 funded project that's coming to an end, but [Jaunt] is going to take over the development of this. But the way we ran this from day one, it wasn't as a three year project which starts and finishes and that's it. But the guys in [IJCAT] know what they are doing here and they set this up properly. So anyone can subscribe to the mailing lists, can download and develop it. From the day we first posted the code, that has been the case.
So with that, I'd like to invite you all to check this out. Feel free to download and try, it join the mailing list because we'll we'd like the help with that, and thank you very much.
(Applause)
CHAIR: Do we have any questions for Dave?
AUDIENCE SPEAKER: Hi there. Niall Murphy, Google. I am curious to hear you use the example of create another port in another European city, for example, surely that requires permission on behalf of the place which is hosting the ports, or do you extend network infrastructure out there yourselves everywhere you might want to create a port?
DAVE WILSON: No, you are right in the first place. What we do, and this is not actually a part of the OpenNaaS code. You only get this if you download, it but the point to point network that we use is one of the resource that is we are able to interface with in the OpenNaaS software, so we went it's not just enough to make routers, we need connectivity. We have a resource out there that does that, that has all these benefits, these characteristics. So let's take advantage of that. Which leads to the sort of weird state where I'm using, for a very specific purpose, which is creating a circuit inside my own network, I'm using this European system which is capable of interfaceing with three our four different provisioning systems at the same time and requesting them. I do see that that's the right way around. I have I want to have something that's capable of something very flexible but I'll use it for my specific purpose each time, instead of having something specific and you have to build something new for the more general case. What we have there is in HEAnet, we expose a certain amount, to just limited part of our network to the others who might wish to use them. They are available for use and the rest are not. That exposure is always up to us as the network providing the resources, the same police to every other network that participates.
CHAIR: Any more questions? Looks like you get off lightly, Dave. Thanks very much.
So, our next ?? Wolfgang Tremmel is going to talk about how DE?CIX selected and will build their new exchange platform.
WOLFGANG TREMMEL: Good morning. Hope you had a good evening in Dublin and not too much of the black stuff last night.
I would like to talk about what DE?CIX plans to make sure that you can still switch packets within the next decade in Frankfurt. And there is one marketing slide on the presentation, but I will skip over it very quickly.
So the first thing you might notice is that we call this DE?CIX Apollon, so why do we tall it Apollon? The thing is we called the old platform, and we didn't want to call the new platform the so?and?so vendor with so?and?so optical platform. So the marketing guys came up with the a nice name and we came up with the DE?CIX Apollon platform.
So why we doing this? This is the current DE?CIX topology and up one of the things you might easily notice first is that we have, in the middle, we have these core switches, we have one to one redundancy with three core switches each, so, we have three core switches at any time doing nothing, just standing around and costing money. That was fine when we had only two of them. So one standing around, one doing the work. But now we have six of them and three doing nothing and just costing money isn't too efficient.
Also, it causes some kind of problems with Mac learning, so if a packet comes in over one of them, and the one wants so send it back over another one, this one doesn't know the Mac address and there is a flatting on all the ports. We run into some problems there.
We also have no ?? we do not do a full monitoring on the links, we cannot run LSCP, and in the case of a failure, about 400 10?gig circuits need to be switched and need to work from the first millisecond on, and the backup links, cannot monitor them while they are not active. So, this is some of our motivation why we do look for a complete redesign and a complete new platform. We also have no 100?gig ports on the stands, and quite some other stuff. The last point ?? we sell our parts, we are only half?way looping, I could only rephrase that also, no VLAN rewriting, so if you wanted to change the VLAN from an incoming connection what we do is it goes in with the tacked port, looped out and untacked port into another untacked port and is done in another VLAN so that's also not very sophisticated for doing our reseller ports.
What do we want to achieve? We want to have ?? we want to be on the cutting edge again. We want to have up to date modern platform with 100 gig for customers, we want to support traffic growth for the next five years. Scaleability, we want to replace the 121 redundancy with N plus 1 redundancy, which is much more efficient, also money wise, to run, and of course we want to also reduce the number of links using 10 gig links in the core right now, we do want to do 100 gig links in the core only to reduce the number of links we are running across our network.
So, what did we do? Very, very early on we decided that we do no longer want to use a passive of the platform. We had in the old four ten switches we were using DWM receivers with a passive multiplexer and with these little boxes which was the mirrors which switched from active to stand by course, and we want to use an active optical platform between the switching platform just to make sure that we can use standard transceivers, we don't use special optics any more, we are independent of any form factors of the optics on top of that. We have better optical monitoring of the link qualities. And that was the main reason we said, okay, get an active optical platform and on top that have we get a routing and switching platform.
So, first step, that decision. Then we started gathering information, having a look who is on the market? Who can offer what? Doing a short list with three vendors on the optical and three vendors on the switching, or routing platform, and of course, plan really extensive lab testing. Daniel, who co?wrote this presentation and who is, I guess, watching me now, via video stream, did about a year travelling and doing lab tests with the vendors, and we put really a lot of effort into that to make sure that we have the selected the right platform.
Optical platform, main requirements of course 100 gig. No doubt about that. That's where we're going. 80 DWDM channels. 28 gig each. So if you do the maths, you come up with a slightly higher number than 100 gig, if you put four of them together because because they are also running some protocols of the platform over the channels. We want to have very fast protection switching, we want to have scaleability of course and we also want to have compact size. The rooms these boxes are in are not getting big, sorry we need to make space and find space where to put them and also keep the power requirements and cooling requirements to what we have.
So, finally we decided on the Adva, the Adva system, but I have another slide about that later.
The switching platform also here we wanted to have 100 gig, we want of to have real, real, real high port dense see because the higher the port dense see we have in the edge switch, the more traffic simply stays local and doesn't have to go over the core. Each traffic which goes over the core uses backbone capacity and the less we use of that, the better. We want to use multi?passing via MPLS, we want to have port security like we have it now, so customers doing strange stuff with Mac addresses or sending us a loop or whatever, I don't know who of you was a customer about eight years ago when we had the very old platform, we have a lot of pain with loops from customers and bad stuff coming in. We got rid of that with the current platform and we would like to keep it that way, and would like to keep it stable. So we don't have any problems with that.
Of course we want to have the VLAN translation functionality, so, VLAN tack comes in and we want to rewrite it to whatever we like. If the customer comes and uses the tack port and other customers can use the same tagging and whatever tagging they want to and we still put it into the right VLAN of the platform.
So, we came down to a short list. And I'm sure you recognise these three. And as I said, Daniel started his travels. He went to the labs with the vendors, did a lot of tests. The lab tests basically consisted of emulating what we wanted to achieve and also emulate the migration scenario, because, you know, we would have to do a migration of the current platform and with it needed to be as painless as possible for the customers.
In the lab tests, we used up to four course, two minimum, if you wanted to emulate multiple course, you need to have multiple course, we emulated two new access switches and two old access switches. We had a customer emulation, we used ?? tried to really to emulate all scenarios, all failure scenarios, all switch?over scenarios we had and also assimilated of course the migration from old to new, having both kind of switches in the infrastructure running at the same time.
We came to a really big decision matrix. You might notice some of this is blurred because we didn't tell the vendors what we waited and how we waited, and I didn't want to learn from that from a RIPE presentation, but you can really make sure we really, really make educated the best decision we could make to have the right platform selected.
And the winner is, on the optical platform, we decided for the Adva system. It has a very compact size, it's quite small. It has all the features we need. It has up to 80, 20?gig links, so 2 terabits per fibre pair. Switching layer we picked the Alcatel?Lucent. The port density, I think, is pretty impressive, up to 80, 100 gig per chassis. It's multi?chassis. We can use ?? if it gets ?? if we need more we can pull the second chassis beside it and we have 160. We are going to implement ten of them. What we call Apollon super nodes, basically it replaces the old core switches so the one in the core of the network and six edge switches.
As I said, it's ready for multi?chassis, it has the best and most complete and elegant implementation of the features we want. It has excellent hardware performance and it supports our migration scenario. On the con side, it's only a DC chassis so we need some external retrospect fires in the data centres which can not supply DC power and it does not support the S flow we are using currently so our software department will have to put in some over time so put in some new software so we can get out of it.
It supports IP fix, which is kind of a NetFlow, so I think we can get out what we want.
That's the marketing slide. You will see that on T?shirts and stuff and things we put into circulation within the next month.
So this is the high level design we are having. Basically, what we do is we run across the core, across the core from each edge to each edge a circuit, you see the wire on MPLS, so if one core fails, we have still three left, we do not do any rerouting if a core fails, we still go with the remaining three and testing show this scenario works and we are pretty confident that it will cope also with the traffic levels we are expecting in the next five years.
We are re?running VPLS, MPLS; nothing special, but for us it's new since we were running only switches, ethernet switching in the past.
This is the physical topology. The coloured ones are the core switches in four different locations. Then we have the optical equipment here, with kind of backup path. We have edge switches at the same location as the core switches. Which makes things a bit easier. Each edge is, like you see here with the coloured lines, is connected to each core. We are one optical system. And some of the sides where we do not have a lot of customers at back hole, we are an optical system to the nearest edge switch. So, we will have less switch sites than we have before, but still customers get all the same service and all the same SLA as they did. We opened a few more satellite sites, or optical connected sites. We are now in E shelter here as well, we are in two Equinix sites in Frankfurt, and we are, of course, where we were before at Interaction, Telecity and so on.
Migration. Main goal: As painless as possible for the customers. It's painful for us getting up early in the morning over a few weeks, so it should be painless for you, you shouldn't notice your connection going down for a long time. We will move you one by one and, highlighted, we have one ?? a few decisions we made. We hired a dedicated project manager for that. So, he will only take care of the implementation of the DE?CIX Apollon and the migration of the customers. He will be running the project and of course the usual stuff, will do the work and we'll do the actual migration. We involve every department. My guys in customer support will be there for each migration, so if anything ?? if you have any questions during migration, you can send us an e?mail, you can call us, we will be there, we will be answering your questions.
And I'm pretty confident ?? I done one migration already from the old Cisco to the current platform and we learned a lot during that time a few years ago, and we also talked to some other exchanges who might make migrations recently and we are pretty confident that it will work like we planned.
So that's basically the migration steps. Don't be confused by too many diagrams here. The first step is we put in one new core and this will happen next week, next week on Wednesday we will put in the first new core switch just into the existing platform. And then we basically could start putting in edge switches, which we will do, but they will just be connected and waiting for the first customers. Then we go on, put in another core switch, take out an old core switch, put in another core switch, take out and old core switch until we have only new core switches, and old and new edge switches connected and then we start migrating customers from here to here and here to here.
That's what I said. And lots of getting up early in the morning.
This is basically shows how the old and the new world react together. This is a bit complicated slide. I'm not the most technical guy, but Daniel said what we do basically is we run instances in the course, one VPLS ethernet core for the old, one MPLS core for the new ones, and they are interconnected here with a loop outside. If you want to have more details on that, send us an e?mail and we are happy to explain it. As I said, I'm not the most technical guy on the migration, so, I'm pretty sure it works. We tested it and... yeah...
Edge migration. Keep the old edge switch running, move customer fibre by fibre, test them, make sure it works and we expect, depending on the customer, from three to 30 minutes down time during the migration, if you have to move a customer with a lot of connections, it will take a bit longer. Also, what we do once the new platform is in place, all upgrades and new customers will of course go immediately to the new platform, so, if any of you guys is connected to DE?CIX and want to upgrade to 100 meg immediately, just talk to me.
So, that is it. We are pretty confident, this is what will last us for the next five to six years. The newer platform offers possibilities of course for new products, so we keep you posted there, we haven't announced anything yet. We are working on that. And... yeah... any questions?
AUDIENCE SPEAKER: Blake Lillis with Neo Telecoms. Just a quick question for you. Do you anticipate that this platform change will reduce your cost per port to the end customer?
WOLFGANG TREMMEL: In the long run, yes, definitely yes.
BLAKE LILLIS: And the other question I would add is: Was there a driver that led you specifically to picking a separate DWDM platform versus DWDM line cards in a platform?
WOLFGANG TREMMEL: Yes, we wanted to be independent of the optics in the switching or routing platform, and we wanted to be form factor independent. We never know what's coming in the next five to six years. And of course, we wanted to use also it for back holing, so, and we wanted to have protection and monitoring and everything, that's basically ?? you only get these things if you use something active in between.
AUDIENCE SPEAKER: Okay. Not with all the vendors, but...
WOLFGANG TREMMEL: Yes.
CHAIR: Are there any more questions for Wolfgang? Now is your time. Wolfgang, I'd like to thank you for an excellent presentation.
(Applause)
Our next presenter will be Alan, who will be talking a little bit about submarine cables.
ALAN MAULDIN: Good morning, happy to be here, it's my first time at a RIPE event so it's been a fun experience so far. I am going to talk about international submarine cable trends, so as we are on an island, obviously having cable connectivity is an important way to reach the rest of the world.
I'm going to start today by looking at what we have seen in terms of supply growth around the world and the role we're seeing new technology is having in enabling us to boost the supply even further.
Then I'm going to look at some trends with transport prices, and then I'll look at a review of new cables that are being built around the world.
Cable capacity. This chart here is showing some of the major submarine cables around the world. It's a nice long time frame. You can see what's nice here, we had the supply glut, right, it's very flat here after 2001, but since then, the glut has been eaten up and consumed, operators are adding capacity rapidly to all the cables that are in service and all these routes here, so you can see the transatlantic route now has 19 terabits of capacity. That's since the end of last year. Some of the most rapid growth has been in Latin America and to the Middle East and to Europe and Middle East and Asia. Those bottom two lines there, the yellow and brown are you can see are really surging and catching up to the other routes.
So when you look at the capacity among these major inter continental routes in different time frames, you can relate to this trend very, very clearly. The first group of lines here is '97 to 2002, and the middle one is 2002 to 2007. The final one is 2007 to 2012. And what this is really showing you here is that initially, the capacity across the Atlantic was, that was added in that first time frame, that was more than all the our routes combined, the Atlantic was clearly the dominant route for submarine cable capacity in the world. That's been changing a lot. If you look at the past five years, the capacity added to these major routes, all the routes have had over ten terabits of new capacity added to them. So, this is partially a reflection of the Atlantic's growth slowing over time. But what it also shows really is the rapid growth we're seeing in Latin America, Asia, Middle East as well.
So how are we able to meet the demand growth? So, building cables is a very expensive thing to do, as you may know, there are hundreds of million dollars each for some of the bigger cables.
Upgrades to cables is ?? has been the primary way to add new capacity. Maybe you don't know, cables usually enter service with a fraction of their capacity is length over time. You can invest just on the shore ends, upgrade, it's very easy to add capacity over time. So on several of these routes there's been no new cables for the past few years. So, in the Atlantic, also Latin America, there's been no new cables. All this capacity is coming from upgrades to cables that are over a decade old. So these were cables that were built during the boom days are serving us quite well and are going strong.
There is some areas where we are seeing new cables play bigger role in the contribution to the new capacity. So, one of the most prominent examples of that you can see here, is the Europe to Sub?Saharan Africa route, where we see new cables over both coast over the past five years.
So, with all this growth in capacity being added to these old cables, you'd think maybe we're going to run out of capacity. Maybe there is going to be a shortage. People say you can't add capacity to the submarine cables. At some point you are going to have add new cables. That's true. But how close are we to that threshold of having to add new capacity. This chart here is showing the percentage of potential capacity as Lit. The trend was very clear, it was going up for many years on all these routes. Something weird happened then just in 2009.
It started going down. It's kind of odd because we saw in the first slide, this sharp uptake of Lit capacity being added to obvious routes, but yet it's less utilised. It's a very interesting phenomenon we have been seeing. So, what's really behind that? And cheats behind that is the ability of the older cables that were built in 1999, 2000, 2001, that era, to be able to accommodate higher bit rate wavelengths which was not thought possible initially. When this ?? in this example here of a fibre pair on the Tata TGN?Atlantic cable. When it was in first service, it could do 64 times ten tonne a pair. That was the view they had for the first you know six years of the cable's life. Then they said we can do maybe 100 times ten. We have tighter spacing of the wavelengths, and here is where the big change happens is, there was uncertainty whether or not they could do 40 over old cables. They find out they can. They can do 40 times 40 in 2010 and then two years ago we said we can do 72 times 40, and then the biggest leap obviously is now going to 100 gigs, this is over a decade?old cable. So, right now it can do 48 times 100, they believe, per fibre pair, so there is four fibre pairs in each span of cable. This is just one system. So, each cable has a different capability for capacity. This is a good example to show you what we have been seeing in terms old cables, their design capacity to be able to be increased over time.
Let's look at the trend in pricing now. We gather data on prices from a cable operators and market participants and what this shows here is the medium monthly lease price for a 10 gigabit wavelength on various routes, and I think everybody is aware that the prices across the Atlantic are rather cheap to say the least. The end of last year I think they were about, there is a median, 9,000 US dollars, this is for a single wave, most people are buying blocks or you are getting well below that if you are buying multiple ones, in contrast, in Asia, it's a lot more expensive. It's about 20 thousand dollars. Latin America has become actually cheaper, this is actually good for the price. It was about $75,000 last year, and that's come up quite a bit over the past few years, we are seeing a lot of pressure there with prices and possible new cables coming there to drive the prices down in that region. One of the highest areas for capacity price is the Middle East, so London to ?? one of the main cables on the east coast, you seen there it's about 85 thousand dollars. Also that's been coming down a lot as well. Been a lot of new cables entering the service the past few years and when you have new cables, you have new capacity. That tends to lead to lower prices.
As I mentioned we gather prices from a large array of operators and so there is this, there is a very large range of the price that is exist in the market, so this is showing here that price of a LA to Tokyo wavelength for the past four years, you can see there is a rather large range of prices, so different operators are inclined to be aggressive or non aggressive, depending on price and business strategy. Others are trying to seek a small premium for having the lowest latency path. There is always going to be range of price that is exist in the market.
I want to look at Africa now because this was touched on yesterday in the talk by Michele, so, the new cables to Africa have totally just transformed the cost there for international connectivity, so looking here, this is just STM 1, and seeing in 2009, it's over $140,000 a month, that's rather costly for an STM 1. And now it has come down to 21,000 and we have heard this year it's already dropping well below that due to the wax cable being added last year and that's having a lot of operators dropping prices as well.
So, this type of scenario we see, you know, around the world with ?? in the Middle East, in Africa, we see a lot of new cables coming in service, you get new players to the market, and it's leading to a rapid drop of international capacity prices.
So, if more cables lead to lower prices, more cables on the way is going to be great for you guys, maybe not for the people who sell bandwidth. So, you can see here this is showing the number of new cables being added to the world for the past decade or so. You can see here is the boom days, 1999, 2000, you can see 15, 20 new cables added each year. There's been a relatively boom in new cables the past few years even, we saw about 17 cables last year in service. They are not all big Trans?Atlantic cables, some of them are small say from Ireland to the UK, for instance, but still it's a real investment, it's happening in new cables. Despite all of the spare capacity, there is a need for new cables for route diversity, some operators who don't want to buy capacity, they want to own capacity. So you are seeing a lot of those drivers driving the case for new cables and so, this trend is going to continue, it looks like, for the next few years, given what's been announced. I say more cables on the way maybe because a lot of the projects that have been announced are rather speculative in nature, some of them are privately funded and it's quite hard to get money lined up for cable projects.
So, I think that this year there are several that we're going to see, at least eight that we know of that are going to happen for sure and next year there is a couple. There could be many, many more. It's a bit too far outright now to know how many will happen in 2014.
So, where is the investment coming? What seeing most of it. As I mentioned before most of the main growth we have been seeing is in the Middle East, Africa, Asia, so what this chart is showing is the amount of investment in new cable construction around the world. We have seen over, or nearly one?and?a?half billion invested in Africa in just the past couple of years. The Europe to Asia route going through Egypt and the Red Sea also has been a commonplace for many new cables.
But you also see where there has not been a new investment. Not much in Latin America, the Atlantic, the Pacific at all in the past few years.
What's happening is that's shifting, so the focus of investment of new cables is moving, and so, here is what's happening is some happening in Africa still, but a lot is happening in Asia, so these larger intra?Asian cable systems are having two new ones being built in the next couple can of years there. Latin America is just teeming with new cables that are going to be in service there. Oceania, mainly these are cables, possible cables that would link the west coast of Australia to Indonesia and Singapore, also plans for cables on the east coast as well which, are a bit more speculative. And the Atlantic also, there is a pair of new cables that are proposed there, Hibernia express cable and the Emerald express cable which both hope to be ready by the end of next year if they proceed. There is the other category up there, that's a lot of things there, but a couple of those that are speculative are the cables across the arc particular, there is two cables that want to go, one is I guess over a Canada, one is over Russia, so with the polarised caps thawing, you can get cables there during part of the year, they are going to do that possibly, we'll see if they happen or not.
Let's look at a couple of new ones coming into service this year, this is from ?? if you want to see these maps better, go to that website ?? but this is America Movil's AMX?1, it's coming into service in August of this year. This is a massive cable hitting a lot of countries in central America going down to Brazil. This is a mobile operator putting in a cable. They are gotten tired for paying for capacity, you know, paying at level 3 and Telefonica and these guys, they want to have their own cable to serve some of their supply subsidiaries in these various locations.
One of the new cables coming, I believe it's this summer perhaps, this SJC cable. This is a consortium, and it's notable because it does involve traditional members in the region, it's KDD and SingTel, it also involves China Mobile, who is trying to be become more active. Also Google is involved in this cable, so this is a member of the unity cable which links California to Japan, this will link them down through all of southeast Asia. So, a very typical ?? the design here is a typical now where no more ring cables being built ever, they've gone away. They are all these truck and branch cables. So you have an express path from Singapore to Japan and you have branches off that to various places, so here the branches are to Taiwan, Thailand and Brunei.
So, free cable maps. Well, for those who can answer a few trivia questions, so if you haven't been watching and have been tweeting or looking at videos or whatever you guys do out there, it's okay, because this quiz will not involve what I discussed. It's trivia. So, I'm going to ask some questions, and whoever raises their hands first and gets the right answer can get one of our new maps the this is a map we designed this past year, it's an antique style map but it shows the current take of the cables around the world. So let's see how this goes with the quiz here.
What is the diameter of deep sea cable most similar to?
A pint of Guinness?
A garden water hose?
Or a World War 2 air torpedo?
Answer: B.
Correct, a garden watering hose.
So, the cables are a little thicker near to shore where they're armoured and have more clutting on them but in the deep sea they are rather small.
How are cables loaded on to ships? So whenever you are going to load a cable on to a ship to lay it somewhere, how do you get the cable onto the ship.
A mechanised winch?
A robotic arm?
Or manually, by a team of guys walking around in a circle?
Answer: C:
C is correct. I know you wouldn't think it, but they have teams of guys who walk in circles in eight hour shifts and they do it. They have yet to invent ?? they have yet to invent a machine that can lay the cable in the bays as accurately as the human hand, apparently. So...
One last one here:
Which of these is not a major cause of cable faults?
Ships' anchors?
Commercial fishing.
These three guys?
This is kind of easy.
Answer: B...
Wrong...
C. Correct. This photo you guys made me aware of, this was the photo that the Egyptian navy took in March of some guys who were apparently trying to cut some cables off the coast of Egypt. The reason why I have this here is not that to make a lot of these guys apply for what happened to them or the fact that cables do get cut sometimes by ?? but I think there is this impression that there is this huge risk of people going around and cutting up the world's cables when in fact the real risks is fishermen who don't know where they are putting their nets or trawling, or it is ships who will set sail with their anchors down. It happens a lot. They drag their anger through the port and are dragging cables all the time. So there is you know a couple of hundred cable faults a year and I think over 80% of them are due to the first two items listed here.
So I don't have any more questions for you guys, but that's all for today. We have time for questions I guess, if we have ?? we have time.
AUDIENCE SPEAKER: Hi, Marty an began from Akamai. What percentage of the Trans?Atlantic cables are either near E O L or you know, more than 50%.
ALAN MAULDIN: That's a very good question. So cables are designed for a 25 year life?span. So they are designed for. Most people think they can go beyond that. It's an open question, no one really knows. Some cables like the tad 14 cable have had some bad problems with diodes in their peter, so, that's kind of a unique example there. Other cables, you know, the latest one or the newest one that was built in 2003, that's a poll owe, that's ten years old. That will have another 15 years still. So the rest then would be, you know, about 12 years old, 13 years old. So they are probably about half?way through their life?spans, but as I said, no one knows if you can push them beyond 25 years, because it hasn't actually happened before, cables are always turned off well before we get to that point because they couldn't push the capacity higher. They are economically obsolete before they were actually physically obsolete. No one really knows for sure yet.
AUDIENCE SPEAKER: Can you clarify a little bit more with respect to the Emerald express cable. My understanding is that they have actually, they are actually at IT P, they are not kind of hopeful, they are going towability that thing.
ALAN MAULDIN: I can't comment ?? I'm not fully aware of where they stand.
AUDIENCE SPEAKER: Okay. Thank you.
AUDIENCE SPEAKER: I'd like to thank you guys for tracking all of this, because it's not an area ?? but I have watched your blog and your information. I'd like to say thanks because it affects all of us but there aren't that many people out there tracking to the degree you guys are. So thanks.
ALAN MAULDIN: Thank you. I appreciate the comment.
AUDIENCE SPEAKER: Mike Hughes. One of the things I noticed you were talking about the new Singapore?Japan cable and again it's going through the Luzon Strait with pretty much every other cable there and we are not seeing a great deal to deal with these hot spots like Suez and Luzon Strait. There is the new path coming out of the Middle East going via Russia but there is very little else that I have seen anyway of paths that can avoid these hot spots. What do you know.
ALAN MAULDIN: You are right with Asia. Between Taiwan and the Philippines. I think the ASC cable does try to take a different path and avoid the area so that cable at least, which was new, is ?? has made an effort to try and avoid that. But I think, you know, one of the challenges is the permitting as well where you can get cables in certain places, it's a huge issue right now. Especially in Asia with China claiming larger waters around trying to get multiple permits to put cables. You might see a path you want to use but the reality of the environmental, legal stuff can prevent it from happening. But there is definitely a need to have, thinking of different paths you can take.
AUDIENCE SPEAKER: It would be good because we become constrained by things like Suez and Luzon. It's not just the symmetry and the undersea environment; it's a legal environment that's an issue.
ALAN MAULDIN: Yeah, environmental, everything.
AUDIENCE SPEAKER: Donal Cunningham from AirSpeed Telecom. More of a comment than a question. But for those of you who are lucky enough to never have to worry about this, smile and wave, because we have seen a double primary and backup cable failure from a major provider twice in the last two months going from Ireland to the UK. And it's even more fun when it turns out that alternate providers suddenly turn out to be using those same cables to provide their resiliency as well. Now, luckily not 100% of the same cables but a disturbingly higher number, i.e. greater than zero turn out to be sharing the same routes. So if you live in a small island republic off the edge of Europe, please for the love of God check which fibre paths your vendors are using.
ALAN MAULDIN: That's a very good point. You can multihome all you want but they all use the same cable it doesn't really do anything for you. Yes.
BEN: What's the main reason to roll out the new cable? It's commercial or just to be independent from other providers?
ALAN MAULDIN: It's a tough business case, but it's, as you said, it's trying to ?? if you can get enough market share and you want to invest in owning the capacity. Certain companies just want to own the cable, it's a strategic, they want to have it as part of their company. They want to don't want to buy from a competing carrier, the need for diversity, the point that you need to have cables that go different places, you might have two or three cables, it isn't enough. Maybe the business case is crappy, you need to have that to make sure you have a high level of service to your country.
FREDY KUENZLER: Do you have a rational explanation why the Middle East route is still so much more expensive than transatlantic? You probably could go back to that slide where we have the graph where the price comparison is. I mean, to my understanding, the Middle East route still has quite a bit of competition, meanwhile, and we still see about ten times more than transatlantic
ALAN MAULDIN: Yeah, I think it's a scale issue as well. You have got so many cables. In the Atlantic, it's really an exception here, because cables went out of business, and so the pricing in the Atlantic is a reflection of the incremental cost newspaper grading, basically. So it's not really I think a good ?? it is very low.
AUDIENCE SPEAKER: So you mean that the transatlantic cables are already written off kind of.
ALAN MAULDIN: Yeah. In the Middle East, you know, it's ?? the demand has been growing fast I think with we are seeing the pricey rogues come as well so I think we are going to see, it I don't think you are going to see it as being the same price as the Atlantic, but I think we are seeing this difference is narrowing, right. So, that the difference between what we saw the price in, you know, Brazil to the US versus is the Atlantic, I think it used to be 16 times higher, now I think it's only 10, so it's getting better.
AUDIENCE SPEAKER: Dan, a question for you this time. To the person who talked previously about the two cables coming out of here, out of Ireland. Are there other like single points of failure that worry you with the overall global submarine cable infrastructure?
ALAN MAULDIN: I think the common spot people identify is Egypt still because of the terrestrial crossing across Egypt. They have been trying to build an extra terrestrial path across Egypt to provide some diversity there. So that's supposed to be done in a few months, but I think that's where like the most common areas, besides the issue with the anchor problems in the Mediterranean, the terrestrial as well in Egypt is an issue. But it's going to stay a problem. There's been so much money that's been put into cables that go across that route, you can't ?? you are not going to see people switching to use the route now that goes through Russia. There is a new path that goes from Oman to Iran, to Azerbaijan to Russia to Ukraine, slow vac a, Frankfurt so, that entered service a few months ago is a great new option. The problems it costs five times the other route cost through Egypt. So, yes, you want to have a diverse path but how much are you going to pay for it? And so if you can pay, you know, you'll have five different paths across Egypt for the price of one path that's terrestrial through to Russia so, it's...
AUDIENCE SPEAKER: Any other spots beyond Egypt and Asia or anywhere else?
ALAN MAULDIN: People cite Singapore a lot. A lot of cables that are going from India up to China, some go through Malaysia and you are seeing some of the new cables that are being built are trying to avoid that, so there is a new cable announced called the Bay of Bengal Gateway, I believe, announced a few weeks ago and it will go from India to Malaysia actually and not go down the Singapore, so trying to take a different path. So the operators are well aware of what these problems are and they are trying to take measures to avoid them. At some point geography is what is it is and there is politics as well and certain countries you just can't run cables through or there is too many mountains or deserts, or whatever...
CHAIR: Thank you very much for your presentation regarding cables.
(Applause)
Although I was a little BIS disappointed that the cables in my garden were not on the map. Ladies and gentlemen, we'll have a coffee break now and we'll be back at 11, we ran ahead a bit of schedule. We'll see you in 40 minutes. Thank you.
(Coffee break)
