Reliable transport vs. reliable transport

One of the most contradictory uses of terminology in communications concerns the word transport as used by the IETF and the ITU-T communities. To make matters worse, the term’s prevalent modifier reliable leads to even further divergence in meaning.

To the Internet community, transport refers to the fourth layer of the OSI layer stack (a layer stack known to the ITU-T as X.200, but largely assumed to have been superseded by the more flexible G.80x layering model). The transport layer sits above the network layer (IP), and is responsible (or not) for a range of end-to-end path functionalities. The IETF has defined four transport protocols – the two celebrated ones being UDP (unreliable) and TCP (reliable), and their less fêted brethren are SCTP (highly reliable and message-oriented), and DCCP (unreliable but TCP-friendly).

Since the IP stack does not extend below OSI layer 3, the basic tenet is that nothing can be done about defects in lower layers (congestion, lost or misordered packets) and the best strategy is to compensate for any such problems using the layer over IP (e.g., retransmission, packet reordering). Reliability in this context thus means employing such compensation.

To the communications infrastructure community a transport network is a communications network that serves no function other than transport of user information between the network’s ingress and its egress. In particular, no complex processing such as packet retransmission of reordering is in scope. Reliability in this context means monitoring the functionality and performance of the lowest accessible layer (OAM), and bypassing defective elements in as short a time as possible (protection switching).

For many years the Rubicon between L2 and L3 so effectively separated the two communities that the disparate usages could continue un-noticed. But then came MPLS.

MPLS was originally invented as a method of accelerating the processing of IP packets, by parsing the IP header only at network ingress, and attaching a short label to be looked-up from then on. With advances in header parsing hardware and algorithms this acceleration became less and less significant. However, the possibility of treating a packet consistently throughout the network, and thus performing traffic engineering under layer three instead of compensation over it, kept MPLS from becoming marginalized. While IntServ at the IP level (implemented by RSVP) never caught on, traffic engineering at the MPLS layer (RSVP-TE) starting gaining momentum.

Of course, for MPLS to truly make IP more reliable, it required more “transport” functionality (in the ITU sense), such as stronger OAM and protection switching. This lead to the introduction of “Transport-MPLS” (T-MPLS), later to be renamed “MPLS-Transport Profile” (MPLS-TP).
Thus it became impossible to suppress the conflict between the two transports. In the IETF Work on MPLS Transport Profile in the IETF is obviously not performed in the “Transport Area” (having nothing to do with transport), but in the “Routing Area” (although it has very little to do with routing).

So what does the future hold for the words “transport” and “reliable” ? In theory it would be possible to adopt synonyms (such “carriage”, and “resilient”), although I doubt that either community would be willing to abandon its traditions. At a high enough level of abstraction the meanings coalesce, so perhaps the best tactic today (whenever there is room for error) is to say sub-IP transport and super-IP transport. Reliability can be left for the super-IP case (where it is not really apt) since there are multiple alternatives for the sub-IP case.

Y(J)S

OAM for flows

Continuing my coverage of the recent joint IESG/IAB design team on OAM, this time I want to discuss the issue of OAM for flows in Packet Switched Networks (PSNs).

From a pure topology standpoint any communications network is imply a set of source ports (i.e., interfaces into which we may input information), a set of destination ports (i.e., interfaces from which we may receive information), and a set of links connecting source ports to destination ports. Of course, the destination ports will may be located very far from the source ports (and this is the reason we use the network in the first place), but this geometry is irrelevant from a topological point of view.

PSNs are communications networks that transfer information in units of packets. They can be classified as either Connection Oriented (CO) , or ConnectLess (CL). In a CO network the end-to-end path from source port to destination port needs to be set up (by physical connection, or by manual/management-system configuration, or by control-plane signaling) before information can be sent down the path. Once set up, it makes sense to call this end-to-end path a “connection”. A connection is essentially an association of a source port with a destination port that has been prepared to carry information.

In a CL PSN each packet of information is individually sent towards its destination. No set up is required as each packet is identified by a unique destination address. When we send a data packet from a source port to the desired destination we can still think about an association of the source and destination ports, but as this association is ephemeral, this association does not constitute a connection. If, however, many similar packets are sent from source to destination, it may be useful to speak of a “flow” of packets. Of course, there is no guarantee that all the packets travel from source to destination over precisely the same path through the network, but in many cases this is the case for substantial periods of time until a reroute event takes place. When load balancing is used the definition (and its consequences) becomes truly problematic.

OAM mechanisms were originally designed for Circuit Switched (CS) or CO communications systems, such as PDH, SDH, and ATM networks. For such networks it makes perfect sense to ask about continuity of the connection, or its performance parameters (e.g., delay). Thus Continuity Check (CC) OAM functions became standard, and PM functions recommended for CO networks. The issue is more complex for CL networks. Continuity doesn’t mean very much if every packet is sent to a different destination! It means somewhat more when there is a prolonged flow; but even then packet loss and delay are statistical combinations, since consecutive packets may traverse different network elements on their route from source to destination.

When packets are delivered to an incorrect destination we say a misconnection has occurred, and Connectivity Verification (CV) monitors for such events. (There is often confusion between CC – a functionality needed for CO and CL networks, and CV – which is only for CL.)

For some types of prolonged flows it makes sense to introduce OAM mechanisms to monitor continuity and performance parameters. Pseudostreaming of video over the Internet may involve hundreds of packets per second for many minutes. IPTV flows are even higher in rate and can last for hours. Ethernet Virtual Connections (EVCs) between customer sites last indefinitely.

In a future entry I will discuss when it doesn’t make sense to talk about flows, and what OAM means for such cases.

Y(J)S

IETF and OAM

On October 12nd and 13th the IESG (Internet Engineering Steering Group) and IAB (Internet Architecture Board), the two IETF management bodies, held a joint design session on OAM. I was a bit surprised that the IETF leadership would be interested in devoting a separate meeting (not coinciding with an IETF conference) to the subject of OAM; OAM has never been an area of IETF expertise. Indeed, when the meeting was first announced on the IETF main discussion email list several long-time IETF participants asked for the acronym OAM to be spelled out! Of course, the ICMP (ping) was defined in RFC 792 circa 1981, and BFD that runs between routers has its own BFD Working Group (WG) in the IETF, but the overall concept of OAM has never been central to the IETF world view.

However, just as the interests of the ITU-T have been migrating up from synchronous networks to ones based on Ethernet, IP and MPLS, so have the interests of the IETF been migrating down from applications, end-to-end transport, and routing to pure transport functionality. And OAM is a crucial element of transport networks.

The physical meeting was held at George Mason University in Fairfax Virginia, but was also WebEx’ed. Thus I managed to actively participate, and even present slides, without having to travel; but did find myself jet-lagged due to shifting my work day by 6 hours. Unfortunately, the Internet connectivity at the conference site was not completely solid, and the remote attendees frequently found themselves talking to themselves on how to alert those on-site that the connection had failed. Some connectivity OAM would definitely have been useful …

So where does the IETF want to use OAM ? The main interest is now MPLS-TP, but there are still open issues regarding PW OAM.

The IETF PWE3 WG standardized an associated channel that shares fate with the PW traffic, which is mostly employed for OAM. This OAM is misnamed VCCV for Virtual Channel (an old name for PW) Connectivity Verification (which should be Continuity Check). VCCV presently allows IP ping, LSP ping, and BFD protocols to run inside the associated channel in order to provide FM. Back at IETF-67 (November 2006) I proposed using Y.1731 inside the associated channel. This idea was later developed into draft-mohan-pwe3-vccv-eth, backed by Nortel, RAD, France Telecom, KDDI, Huawei, NTT and Sprint, but was rejected by the larger community due to confusion as to its use of Ethernet (it was never intended to be limited to MPLS over Ethernet, or Ethernet PWs).

As I am sure all my readers know, MPLS-TP is a transport technology, being jointly developed by the ITU-T and IETF. The ITU-T views MPLS-TP as yet another transport network, which needs the same OAM functionality as all the other transport networks developed to date (SDH, OTN, carrier-grade Ethernet). In particular, the generic research on OAM for packet-based networks, and the protocol development (in cooperation with IEEE 802.1) of Y.1731, is seen by the ITU-T community as directly relevant to MPLS-TP. Work in the ITU, and Internet Draft draft-bhh-mpls-tp-oam-y1731 submitted to the IETF, proposed maximizing re-use of Y.1731 formats. This approach is strongly advocated by Alcatel-Lucent and Huawei, and is being backed by many operators, including China Mobile, China Telecom, Telecom Italia, France Telecom, Deutche Telekom, Telstra, and KPN. The idea expands on my earlier proposal, solves both FM and PM with a single OAM protocol, and is expected to undergo major deployment in the near future.

IETF participants from Cisco, Juniper, and Ericsson produced an alternative OAM FM proposal, based on the IETF’s own BFD instead of the ITU-T’s Y.1731. The IETF MPLS WG could not reach consensus as to which mechanism to prefer (in an email poll the community was split about 50/50). The MPLS WG chairs decided to exercise their authority to break such ties, and elevated the BFD-based draft to WG status as draft-ietf-mpls-tp-cc-cv-rdi, thus effectively killing draft-bhh for FM. The issue of PM was open for a while, but a Cisco draft has recently been elevated to become draft-ietf-mpls-tp-loss-delay, thus blocking draft-bhh from the PM function as well. This draft is not fully fleshed out, but it uses the MPLS-TP G-ACh mechanism, and allows either NTP-style or 1588-style timestamps.

So we see that OAM has become a hot (and contentious) topic in the IETF.

After this long introduction, my next entry will delve into a few of the subjects discussed at the joint design session.

Y(J)S

OAM for FM and PM

The Operations, Administration, and Maintenance (OAM) functionality provided in all modern communications systems supports two distinguishable functions, namely Fault Management (FM) and Performance Management (PM).
It is important to remember that despite the use of the word “management” here, OAM is a user-plane function. OAM may trigger control plane procedures (e.g., protection switching) or management plane actions (such as alarms), but the OAM itself is data that runs along with the user data.

FM deals with the detection and reporting of malfunctions. ITU-T Recommendation G.806 defines a scale of such malfunctions :

• anomaly (n): smallest observable discrepancy between desired and actual characteristics
• defect (d): sufficient density of anomalies that interrupts some required function
• fault cause (c): root cause behind multiple defects
• failure (f): persistent fault cause such that the ability to perform the function is terminated

The main FM functions include :

• Continuity Check (CC): checking that data sent from A to B indeed arrives at B
• Connectivity Verification (CV): checking that data set from A to B does not incorrectly arrive at C
• Loopback (LB): checking that data can be sent from A to B can be returned from B and received at A
• Forward Defect Indication (FDI) also called Alarm Indication Signal (AIS): when data sent from A to B is destined for C, B reports to C that it did not receive data from A
• Backward Defect Indication (BDI) also called Reverse Defect Indication (RDI): when data is sent from A to B, B reports to A that it did not receive the data.

PM deals with monitoring of parameters such as end-to-end delay, Bit Error Rate (BER), and Packet Loss Ratio (PLR). While there may not be loss of basic connectivity if performance parameters are not maintained within their desired realms, the ability to provide specific services may be compromised, even to the extent that there is a loss of service. For example, excessive round-trip delay makes it difficult to hold interactive audio conferences, and excessive PLR may lead to loss of an IPTV service. For this reason, Service Providers (SPs) commit to Service Level Agreements (SLAs) that specify the acceptable PM parameters.

A partial list of PM parameters that may appear in an SLA is :

• BER or PLR (for packet oriented networks)
• 1-way delay (1DM) also called latency: the amount of time it takes for data to go between two points of interest (this measurement requires clock synchronization between endpoints)
• 2-way delay also called roundtrip delay (RTD): the amount of time it takes for data to go to a point of interest and return (does not require clock synchronization)
• Packet Delay Variation (PDV): the variation of delay (may be 1-way or 2-way, but even 1-way does not require time synchronization, although frequency synchronization may be required for highly accurate measurements)
• Availability: percentage of time that the service can be provided
• Throughput or Bandwidth profile (for packet oriented networks): methods of quantifying the sustainable data rate (will generally be needed for each direction separately)

While certain FM functions, in particular Continuity Check (CC), are usually run periodically, PM functions are frequently called on an ad-hoc basis. However, with an SLA in effect, the SP needs to periodically monitor the PM parameters, and the customer may want to do so as well. In fact, while customers typically trust legacy SPs to provide the promised service level (after all, a 2.048 Mbps leased line is never going to deliver only 1.9 Mbps!), they have much less trust for newer services (it is relatively easy for a SP to cheat and provide 8 Mbps Ethernet throughput instead of the promised 10 Mbps).

In future entries I will deal with questions such as what parameter levels are needed for particular applications, how PM impacts user experience, and how SPs and customers should monitor performance.

Y(J)S

Deployment, R&D, and protocols

In my last entry I discussed why the last mile is a bandwidth bottleneck while the backhaul network is a utilization bottleneck. Since I was discussing the access network I did not delve into the core, but it is clear that the core is where the rates are highest, and where the traffic is the most diverse in nature.

Based on these facts, we can enumerate the critical issues for deployment and R&D investment in each of these segments. For the last mile the most important deployment issue is maximizing the data-rate over existing infrastructures, and the area for technology improvement is data-rate enhancement for these infrastructures.

For the backhaul network the deployment imperative is congestion control, while development focuses on OAM and control plane protocols to minimize congestion and manage performance and faults.

For the core network the most costly deployment issue is large-capacity, fast and redundant network forwarding elements, along with rich connectivity. Future developments involve a huge range of topics, from optimized packet formats (MPLS) through routing protocols, to management plane functionality.

A further consequence of these different critical issues is the preference of protocols used in each of these segments. In the last mile efficiency is critical, but there no little need for complex connectivity. So physical-layer framing protocols rule. As there may be the need for multiplexing or inverse multiplexing, one sometimes sees non-trivial use of higher-layer protocols. However, these are usually avoided. For example, Ethernet has long had an inefficient inverse multiplexing mechanism (LAG), but this is being replaced with the more efficient sub-Ethernet PAF (EFM bonding) alongside physical layer (m-pair) bonding for DSL links.

In the backhaul network carrier-grade Ethernet has replaced ATM as the dominant protocol, although MPLS-TP advocates are proposing it for this segment. Carrier-grade Ethernet acquired all the required fault and performance mechanisms with the adoption of Y.1731, while the MEF has worked hard in developing the needed shaping, policing, and scheduling mechanisms.

In the core the IP suite is sovereign. MPLS was originally developed to accelerate IP forwarding, but advances in algorithms and hardware have made IPv4 forwarding remarkably fast. IP caters to a diverse set of traffic types, and the large number of RFCs attests to the richness of available functionality.

Of course it is sometimes useful to use different protocols. A service provider that requires out-of-footprint connectivity might prefer IP backhaul to Ethernet. An operator with regulatory constraints might prefer a pure Ethernet (PBBN) core to an IP one. Yet, understanding the nature and constraints of each of the segments helps us weigh the possibilities.

Y(J)S