Subscription to Multiple Stream Originators
draft-zhou-netconf-multi-stream-originators-02
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft whose latest revision state is "Replaced".
|
|
|---|---|---|---|
| Authors | Tianran Zhou , Guangying Zheng , Eric Voit , Alexander Clemm , Andy Bierman | ||
| Last updated | 2018-05-16 | ||
| Replaced by | draft-unyte-netconf-distributed-notif | ||
| RFC stream | (None) | ||
| Formats | |||
| Additional resources | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-zhou-netconf-multi-stream-originators-02
NETCONF T. Zhou
Internet-Draft G. Zheng
Intended status: Standards Track Huawei
Expires: November 17, 2018 E. Voit
Cisco Systems
A. Clemm
Huawei
A. Bierman
YumaWorks
May 16, 2018
Subscription to Multiple Stream Originators
draft-zhou-netconf-multi-stream-originators-02
Abstract
This document describes the distributed data collection mechanism
that allows multiple data streams to be managed using a single
subscription. Specifically, multiple data streams are pushed
directly to the collector without passing through a broker for
internal consolidation.
Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in RFC 2119 [RFC2119].
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on November 17, 2018.
Zhou, et al. Expires November 17, 2018 [Page 1]
Internet-Draft Multiple Steam Originators May 2018
Copyright Notice
Copyright (c) 2018 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(https://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Use Cases . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2.1. Use Case 1: Data Collection from Devices with Main-board
and Line-cards . . . . . . . . . . . . . . . . . . . . . 3
2.2. Use Case 2: IoT Data Collection . . . . . . . . . . . . . 4
3. Solution Overview . . . . . . . . . . . . . . . . . . . . . . 5
4. Subscription Decomposition . . . . . . . . . . . . . . . . . 7
5. Publication Composition . . . . . . . . . . . . . . . . . . . 9
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 9
7. Security Considerations . . . . . . . . . . . . . . . . . . . 9
8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 9
9. References . . . . . . . . . . . . . . . . . . . . . . . . . 9
9.1. Normative References . . . . . . . . . . . . . . . . . . 9
9.2. Informative References . . . . . . . . . . . . . . . . . 10
Appendix A. Change Log . . . . . . . . . . . . . . . . . . . . . 10
Appendix B. Subscription Management . . . . . . . . . . . . . . 11
Appendix C. Notifications on Subscription State Changes . . . . 11
Appendix D. Configured Subscription and Call Home . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11
1. Introduction
Streaming telemetry refers to sending a continuous stream of
operational data from a device to a remote receiver. This provides
an ability to monitor a network from remote and to provide network
analytics. Devices generate telemetry data and push that data to a
collector for further analysis. By streaming the data, much better
performance, finer-grained sampling, monitoring accuracy, and
bandwidth utilization can be achieved than with polling-based
alternatives.
Zhou, et al. Expires November 17, 2018 [Page 2]
Internet-Draft Multiple Steam Originators May 2018
YANG-Push [I-D.ietf-netconf-yang-push] defines a transport-
independent subscription mechanism for datastore updates, in which a
subscriber can subscribe to a stream of datastore updates from a
server, or update provider. The current design involves subscription
to a single push server. This conceptually centralized model
encounters efficiency limitations in cases where the data sources are
themselves distributed, such as line cards in a piece of network
equipment. In such cases, it will be a lot more efficient to have
each data source (e.g., each line card) originate its own stream of
updates, rather than requiring updates to be tunneled through a
central server where they are combined. What is needed is a
distributed mechanism that allows to directly push multiple
individual data substreams, without needing to first pass them
through an additional processing stage for internal consolidation,
but still allowing those substreams to be managed and controlled via
a single subscription.
This document will describe such distributed data collection
mechanism and how it can work by extending existing YANG-Push
mechanism. The proposal is general enough to fit many scenarios.
2. Use Cases
2.1. Use Case 1: Data Collection from Devices with Main-board and Line-
cards
For data collection from devices with main-board and line-cards,
existing YANG-Push solutions consider only one push server typically
reside in the main board. As shown in the following figure, data are
collected from line cards and aggregate to the main board as one
consolidated stream. So the main board can easily become the
performance bottle-neck. The optimization is to apply the
distributed data collection mechanism which can directly push data
from line cards to a collector. On one hand, this will reduce the
cost of scarce compute and memory resources on the main board for
data processing and assembling. On the other hand, distributed data
push can off-load the streaming traffic to multiple interface
Zhou, et al. Expires November 17, 2018 [Page 3]
Internet-Draft Multiple Steam Originators May 2018
+-------------------------------------+
| collector |
+------^-----------^-----------^------+
| | |
| | |
+-------------------------------------+
| | | | |
| | +-----+------+ | |
| | | main board | | |
| | +--^-----^---+ | |
| | | | | |
| | +---+ +---+ | |
| | | | | |
| +----+----+---+ +---+----+----+ |
| | line card 1 | | line card 2 | |
| +-------------+ +-------------+ |
| device |
+-------------------------------------+
Fig. 1 Data Collection from Devices with Main-board and Line-cards
2.2. Use Case 2: IoT Data Collection
In the IoT data collection scenario, as shown in the following
figure, collector usually cannot access to IoT nodes directly, but is
isolated by the border router. So the collector subscribes data from
the border router, and let the border router to disassemble the
subscription to corresponding IoT nodes. The border router is
typically the traffic convergence point. It's intuitive to treat the
border router as a broker assembling the data collected from the IoT
nodes and forwarding to the collector[I-D.ietf-core-coap-pubsub].
However, the border router is not so powerful on data assembling as a
network device. It's more efficient for the collector, which may be
a server or even a cluster, to assemble the subscribed data if
possible. In this case, push servers that reside in IoT nodes can
stream data to the collector directly while traffic only passes
through the border router.
Zhou, et al. Expires November 17, 2018 [Page 4]
Internet-Draft Multiple Steam Originators May 2018
+-------------------------------+
| collector |
+---^-----------^------------^--+
| | |
| | |
| | |
| +-------+--------+ |
| | border router | |
| +----^------^----+ |
| | | |
| | | |
| +---+ +---+ |
| | | |
+---+----+---+ +---+----+---+
| IoT node 1 | | IoT node 2 |
+------------+ +------------+
Fig. 2 IoT Data Collection
3. Solution Overview
All the use cases described in the previous section are very similar
on the data subscription and publication mode, hence can be
abstracted to the following generic distributed data collection
framework, as shown in the following figure.
A Collector usually includes two components,
o the Subscriber generates the subscription instructions to express
what and how the collector want to receive the data;
o the Receiver is the target for the data publication.
For one subscription, there may be one to many receivers. And the
subscriber does not necessarily share the same address with
receivers.
In this framework, the stream originators have the Master role and
the Agent role. Both the Master and the Agent include two
components,
o the Subscription Server manages capabilities that it can provide
to the subscriber.
o the Publisher pushes data to the receiver according to the
subscription information.
Zhou, et al. Expires November 17, 2018 [Page 5]
Internet-Draft Multiple Steam Originators May 2018
The Master knows all the capabilities that the attached Agents and
itself can provide, and exposes the Global Capability to the
Collector. The Collector cannot see the Agents directly, so it will
only send the Global Subscription information to the Master. The
Master disassembles the Global Subscription to multiple Component
Subscriptions, each involving data from a separate telemetry source.
The Component Subscriptions are then distributed to the corresponding
Agents.
When data streaming, the Publisher located in each stream originator
collects and encapsulates the packets per the Component Subscription,
and pushes the piece of data which can serve directly to the
designated data Collector. The Collector is able to assemble many
pieces of data associated with one Global Subscription, and can also
deduce the missing pieces of data.
+-------------------------------------+
| Collector +-------------+ |
| +|-----------+ | |
| +------------+ || Receiver | <-----+
| | Subscriber | |+-------------+ | |
| +-^----+-----+ +------------+ | |
| | | | | |
+-------------------------------------+ |
Global | |Global | push |
Capability | |Subscription | |
+-------------------------------------+ |
| | | Master | | |
| +--+----v------+ +------+------+ | |
| | Subscription | | Publisher | | |
| | Server | | | | |
| +--^----+------+ +-------------+ | |
| | | | |
+-------------------------------------+ |
Component | | Component push |
Capability | | Subscription |
+-------------------------------------+ |
| | | Agent | |
| +--+----v------+ +-------------+ | |
| | Component | | Publisher | | |
| | Subscription | | +------+
| | Server | +-------------+ |
| +--------------+ |
+-------------------------------------+
Fig. 3 The Generic Distributed Data Collection Framework
Master and Agents may interact with each other in several ways:
Zhou, et al. Expires November 17, 2018 [Page 6]
Internet-Draft Multiple Steam Originators May 2018
o Agents need to have a registration or announcement handshake with
the Master, so the Master is aware of them and of life-cycle
events (such as Agent appearing and disappearing).
o Contracts are needed between the Master and each Agent on the
Component Capability, and the format for streaming data structure.
o The Master relays the component subscriptions to the Agents.
o The Agents indicate status of Component Subscriptions to the
Master. The status of the overall subscription is maintained by
the Master. The Master is also responsible for notifying the
subscriber in case of any problems of Component Subscriptions.
Any technical mechanisms or protocols used for the coordination of
operational information between Master and Agent is out-of-scope of
the solution. We will need to instrument the results of this
coordination on the Master Node.
Note: Some preliminary considerations on the solution details are now
listed in the appendix for reference. The detailed solution need to
be discussed and will be added if the WG accepts the problem
statement.
4. Subscription Decomposition
Since Agents are invisible to the Collector, the Collector can only
subscribe to the Master. This requires the Master to:
1. expose the Global Capability that can be served by multiple
stream originators;
2. disassemble the Global Subscription to multiple Component
Subscriptions, and distribute them to the corresponding telemetry
sources;
3. notify on changes between portions of a subscription moving
between different Agents over time.
To achieve the above requirements, the Master need a Global
Capability description which is typically the YANG [RFC7950] data
model. This global YANG model is provided as the contract between
the Master and the Collector. Each Agent associating with the Master
owns a local YANG model to describe the Component Capabilities which
it can serve as part of the Global Capability. All the Agents need
to know the namespace associated with the Master.
Zhou, et al. Expires November 17, 2018 [Page 7]
Internet-Draft Multiple Steam Originators May 2018
The Master also need a data structure, typically a Resource-Location
Table, to keep track of the mapping between the resource and the
corresponding location of the Subscription Server which commits to
serve the data. When a Global Subscription request arrives, the
Master will firstly extract the filter information from the request.
Consequently, according to the Resource-Location Table, the Global
Subscription can be disassembled into multiple Component
Subscriptions, and the corresponding location can be associated.
The decision whether to decompose a Global Subscription into multiple
Component Subscriptions rests with the Resource-Location Table. A
Master can decide to not decompose a Global Subscription at all and
push a single stream to the receiver, because the location
information indicates the Global Subscription can be served locally
by the Master. Similarly, it can decide to entirely decompose a
Global Subscription into multiple Component Subscriptions that each
push their own streams, but not from the Master. It can also decide
to decompose the Global Subscription into several Component
Subscriptions and retain some aspects of the Global Subscription
itself, also pushing its own stream.
Component Subscriptions belonging to the same Global Subscription
MUST NOT overlap. The combination of all Component Subscriptions
MUST cover the same range of nodes as the Global Subscription. Also,
the same subscription settings apply to each Component Subscription,
i.e., the same receivers, the same time periods, the same encodings
are applied to each Component Subscription per the settings of the
Global Subscription.
Each Component Subscription in effect constitutes a full-fledged
subscription, with the following constraints:
o Component subscriptions are system-controlled, i.e. managed by the
Master Node, not by the subscriber.
o Component subscription settings such as time periods, dampening
periods, encodings, receivers adopt the settings of their Global
Subscription.
o The life-cycle of the Component Subscription is tied to the life-
cycle of the Global Subscription. Specifically, terminating/
removing the Global Subscription results in termination/removal of
Component Subscriptions.
o The Component Subscriptions share the same Subscription ID as the
Global Subscription.
Zhou, et al. Expires November 17, 2018 [Page 8]
Internet-Draft Multiple Steam Originators May 2018
5. Publication Composition
The Publisher collects data and encapsulates the packets per the
component subscription. There are several potential encodings,
including XML, JSON, CBOR and GPB. The format and structure of the
data records are defined by the YANG schema, so that the composition
at the Receiver can benefit from the structured and hierarchical data
instance.
The Receiver is able to assemble many pieces of data associated with
one subscription, and can also deduce the missing pieces of data.
The Receiver recognizes data records associated with one subscription
according the Subscription ID. Data records generated per one
subscription are assigned with the same Subscription ID.
For the time series data stream, records are produced periodically
from each stream originator. The message arrival time varies because
of the distributed nature of the publication. The Receiver assembles
data generated at the same time period based on the recording time
consisted in each data record. In this case, time synchronization is
required for all the steam originators.
6. IANA Considerations
This document makes no request of IANA.
Note to RFC Editor: this section may be removed on publication as an
RFC.
7. Security Considerations
It's expected to reuse the existing secure transport layer protocols,
such as TLS [RFC5246] and DTLS [RFC6347], to secure the telemetry
stream. The Collector cannot access the Agent directly but to
negotiate the security parameters with the Master. However the data
streams are actually generated by the Agents which are invisible to
the Collector. So mechanisms may need to consider when adapting
secure transport layer protocols here. the detailed solution is TBD.
8. Acknowledgements
9. References
9.1. Normative References
Zhou, et al. Expires November 17, 2018 [Page 9]
Internet-Draft Multiple Steam Originators May 2018
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC5246] Dierks, T. and E. Rescorla, "The Transport Layer Security
(TLS) Protocol Version 1.2", RFC 5246,
DOI 10.17487/RFC5246, August 2008,
<https://www.rfc-editor.org/info/rfc5246>.
[RFC6347] Rescorla, E. and N. Modadugu, "Datagram Transport Layer
Security Version 1.2", RFC 6347, DOI 10.17487/RFC6347,
January 2012, <https://www.rfc-editor.org/info/rfc6347>.
[RFC7950] Bjorklund, M., Ed., "The YANG 1.1 Data Modeling Language",
RFC 7950, DOI 10.17487/RFC7950, August 2016,
<https://www.rfc-editor.org/info/rfc7950>.
9.2. Informative References
[I-D.ietf-core-coap-pubsub]
Koster, M., Keranen, A., and J. Jimenez, "Publish-
Subscribe Broker for the Constrained Application Protocol
(CoAP)", draft-ietf-core-coap-pubsub-04 (work in
progress), March 2018.
[I-D.ietf-netconf-yang-push]
Clemm, A., Voit, E., Prieto, A., Tripathy, A., Nilsen-
Nygaard, E., Bierman, A., and B. Lengyel, "YANG Datastore
Subscription", draft-ietf-netconf-yang-push-15 (work in
progress), February 2018.
Appendix A. Change Log
(To be removed by RFC editor prior to publication)
v01
o Minor revision on Subscription Decomposition
o Revised terminologies
o Removed most implementation related text
o Place holder of two sections: Subscription Management, and
Notifications on Subscription State Changes
v02
Zhou, et al. Expires November 17, 2018 [Page 10]
Internet-Draft Multiple Steam Originators May 2018
o Revised section 4 and 5. Moved them from apendix to the main
text.
Appendix B. Subscription Management
A Global Subscription can be rejected for multiple reasons. Some are
related to the Subscription Decomposition and Component Subscription.
New error codes are defined to indicate why a datastore subscription
attempt has failed. The subscription result with the failure reason
is returned as part of the RPC response.
Appendix C. Notifications on Subscription State Changes
Each component subscription maintains its own subscription state and
is responsible for sending its own OAM notifications (for example,
when the component subscription is suspended or when it can resume).
TBD.
Appendix D. Configured Subscription and Call Home
TBD. Only about the message layer which is transport independent.
Authors' Addresses
Tianran Zhou
Huawei
156 Beiqing Rd., Haidian District
Beijing
China
Email: zhoutianran@huawei.com
Guangying Zheng
Huawei
101 Yu-Hua-Tai Software Road
Nanjing, Jiangsu
China
Email: zhengguangying@huawei.com
Eric Voit
Cisco Systems
United States of America
Email: evoit@cisco.com
Zhou, et al. Expires November 17, 2018 [Page 11]
Internet-Draft Multiple Steam Originators May 2018
Alexander Clemm
Huawei
2330 Central Expressway
Santa Clara, California
United States of America
Email: alexander.clemm@huawei.com
Andy Bierman
YumaWorks
United States of America
Email: andy@yumaworks.com
Zhou, et al. Expires November 17, 2018 [Page 12]