Olev Kartau, HUT

October 1998

Abstract

This paper gives a brief overview of who is implementing so-called gigabit and terabit routers for IP traffic.

According to Figure 1, the supported line speed just cannot keep up with demand in backbones.

Figure 1. (This slide is from [Pluris-Presentation])

Using a higher-speed router is a way to scale IP network bottlenecks, without changing the way packets are processed.

Most (all?) new innovative designs to combat the problem of Internet scalability are done in small, dynamic router start-up companies, often funded by venture capital.

An overview of what is happening in new router projects is in [NewRouters].

Scaling a router does not require changes to other parts, but using IP switching is effective only if many nodes do that; quality vs. quantity leap; routing vs. switching, etc.

A multigigabit router needs to achieve three goals [RFC-1953]

• internal bandwidth
• packet processing power
• a set of protocol standards

Basic components of multigigabit router [IP- Switching]

line card (physical layer components)

forwarding engine (determines to which line card packets should be sent)

network processor (runs the routing protocols and computes the routing tables)

switch fabric (interconnects the components of a router)

OC=Optical Carrier, used to specify the speed of fiber optic networks conforming to the SONET standard.

MP = Massively Parallel

Background:

BBN has made many technological advances that made Internet possible (designed and implemented ARPANET, Defense Data Network, first routers, first TCP for Unix).

GTE, one of the largest telecom companies, acquired BBN in Aug. 1997

The router is research-only project [BBN-Project] with technical overview at [BBN-Tech]. It started as an independent research and development project in 1992-93. Two years later, the Defense Advanced Research Projects Agency awarded BBN a two-year contract to begin development of the prototype. In 1996, the project was still "on the drawing board".

The aim was/is to design an IP router with 50 Gbps total throughput capable of forwarding 6 to 30 million packets/second.

Initial interfaces: 2 x ATM-over-SONET OC-12c

Initial maximum fan-out: 24 interfaces

Future interfaces: OC-48c, HIPPI, FIBRE CHANNEL, 100 Mbps ethernet, FDDI, etc.

Should also be usable as an ATM bridge.

Key innovations:

• Forwarding performance and functionality are independent of network architecture and IP traffic characteristics
• Can keep up with optical link speeds, even for small packets.
• All forwarding & routing functions implemented in software for flexibility, future performance will increase with the availability of faster processors
• Supports experimentation (e.g., with new forwarding algorithms)
• Can provide IP quality-of-service processing at optical link speeds
• Two separate internal connection fabrics to isolate control traffic from the data path
• Routing updates don't interfere with packet data transfer
• Packet transfer within router is pipelined in interrupt-free hardware
• Forwarding speed is enhanced via abstraction of link-level details into a uniform format
• Built entirely from off-the-shelf components.

Technical Approach:

• High performance by separating packet routing process into two systems
• Multiple Forwarding Engines
• High-speed connection fabric between network interfaces

Cisco 12000 series include models 12004, 12008, 12012 [Cisco-12000]

IP switching capacity 5 to 60 Gbps.

A router includes hardware-assisted packet-forwarding engine with distributed buffering and queuing mechanisms, supporting Cisco's Tag Switching, class-of-service (CoS) techniques, and providing interfaces at speeds OC-3/STM1 (155 Mbps), OC-12/STM4 (622 Mbps),OC-48/STM-16 (2.4 Gbps)

Neo Networks [Neo] uses massively parallel router architecture. Neo claims that other architectures that employ fast switching fabrics with modular or separate route processors can become bottlenecks by forcing packets to take multiple trips through the switch fabric. [NewRouters]

Neo’s StreamProcessor 2400 with fully configured chassis includes 7 ASICs with over 1000 embedded RISC processors. It uses "layer-less" forwarding – it both forwards and characterizes data in parallel. While traditional forwarding decisions are made at layers 2 and 3 of the OSI model, the parallel architecture simultaneously applies "rules" to the data. Rules are event-driven actions that are based on anything unique within the data stream (including payload) at any layer of the OSI model.

In other words, the StreamProcessor views incoming packets as a stream of bits that become an instruction set for the massively parallel architecture. Incoming data is parsed and directed to the RISC processors for application and protocol processing. While one processor handles application and protocol processing for the data stream, another performs look-up, queuing and prioritization for that stream. This architecture is claimed to yield forwarding performance of more than 400 million frames/sec. Each interface adapter connections with a native port rate of 2.5 gigabits full duplex. System latency is claimed to be under 10 microseconds to transverse the entire system.

Streamprocessor 2400 has over 1 million storage entries for routes, MAC addresses, filters/rules/labels.

There is also downgraded model StreamProcessor 1000.

Neo’s solution to terabit router is to link multiple chassis.

Torrent [Torrent] is another venture-based startup from April 1996 situated in Silver Spring, Maryland.

Torrent’s IP9000 is a router in 8-slot and 16-slot modular chassis, having shared memory switch fabric, forwarding engines with route look-up and packet classification ASICs on each port and a general purpose CPU for routing table maintenance and system configuration. The maintenance of routing tables is moved out of the packet-forwarding path. The route processor uses a private channel to update tables on forwarding engines.

Torrent does not believe in massively parallel processing, they think their model is sufficient for gigabit routers.

Avici Systems, Inc. [Avici] is working on Terabit Switch Router (TSR) which is claimed to scale from 600M bit/sec to multiple terabits.

They use hardware-based routing, forwarding, multicasting and QoS features. They have powerful line cards - each (of 20) line card has a "Direct Connect Fabric" that serves as a 70G bit/sec router. So the fully loaded rack will have 1.4 Tbps of switching capacity.

Pluris, Inc. [Pluris] proposes massively parallel processing model.

Their Massively Parallel Router (MPR) is a collection of single-board computers. Each node has power to route IP packets at OC-12 or 622Mbit/sec speeds. One or more dedicated processing nodes are used for processing routing protocols. Every processing node has a copy of the forwarding table.

The data interconnect is a patent-pending "self-healing" butterfly switch based on multiple multigigabit serial communication lines. The butterfly switch was chosen to achieve linear scalability. Antonov at Pluris: "Crossbar switches are great for telephony because you have long-lived connections. If you try to use crossbar switches for packets you soon discover that scaling the crossbar kills you".

Switch capacity: 5 terabits per second; Port capacity: 2.5 terabits per second; supports up to 2,048 line cards.

Supported IP traffic over single fiber: 1.25 Tbps

Prototype in Feb 1998, release (planned) the second half of 1998

Argon Networks, Inc. [Argon]

Argon Networks, Inc. [Argon] has project "GigaPacket Node" (GPN) router which will have a 150 Gbps ATM switch at its core, port-level route look-up and support for heterogeneous network and data link protocols. The GPN family is designed to scale from 20 Gbps capacity up to 160 Gbps.

Nexabit Networks [Nexabit], located in Marborough, MA, uses 2.5T bit/sec switch fabric and supports 32 OC-48 interfaces in the NX64000 Multi-Terabit Switch/Router, which will provide an internal switching bandwidth of 6.4 Tbps per chassis. They use "patented breakthrough switch technology".

NX64000 will support OC-3, OC-12, OC-48 and OC-192, with "future scalability" to support OC-768 & OC-3072.

Beta trials were planned to June 1998, Nexabit’s website invites to meet at NetWorld+Interop 98, Oct. 21 to 23

Vision (Nexabit):

Performance: Router platforms that cannot scale 10x each year, or those made up from blocks of 100 Gbps or lower, will quickly run out of bandwidth.

Multi-service: core router must support ATM, IP and Frame Relay. IP natively routed, ATM natively switched.

QoS: router must guarantee multiple levels of traffic priority. End-to-end delay must be low and predictable. Must support multicast traffic without affect to other (multicast or unicast) traffic.

References

[RFC-1953] "Ipsilon Flow Management Protocol Specification for IPv4 Version 1.0", P Newman et. al., IETF RFC1953, May 1996 ]

[IP-Switching] "IP-Switching and Gigabit Routers", Peter Newman, Greg Minshall and Tom Lyon, Ipsilon Networks Inc.]

[BBN-Project] http://www.net-tech.bbn.com/

[BBN-Tech] http://www.net-tech.bbn.com/mgr/pp-mgr-ovw/

[Avici] http://www.avici.com/

[Argon] http://www.gigapacket.com

[Nexabit] http://www.nexabit.com/