NFS Protocol Extension: Retrospect and Prospect
draft-dnoveck-nfs-extension-02
The information below is for an old version of the document.
| Document | Type |
This is an older version of an Internet-Draft whose latest revision state is "Expired".
|
|
|---|---|---|---|
| Author | David Noveck | ||
| Last updated | 2015-03-08 | ||
| RFC stream | (None) | ||
| Formats | |||
| Additional resources | |||
| Stream | Stream state | (No stream defined) | |
| Consensus boilerplate | Unknown | ||
| RFC Editor Note | (None) | ||
| IESG | IESG state | I-D Exists | |
| Telechat date | (None) | ||
| Responsible AD | (None) | ||
| Send notices to | (None) |
draft-dnoveck-nfs-extension-02
NFSv4 D. Noveck
Internet-Draft Dell
Intended status: Informational March 8, 2015
Expires: September 9, 2015
NFS Protocol Extension: Retrospect and Prospect
draft-dnoveck-nfs-extension-02
Abstract
This document surveys the processes by which the NFS protocol has
been extended in the past, including all of the NFS major and minor
versions. A particular focus is on how the minor versioning model of
NFSv4 has worked and what might be done to enhance version management
within NFSv4.
The work already done in the new NFSv4 version management document is
summarized, and there is a discussion of further issues the working
group will need to address in moving that work forward.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 9, 2015.
Copyright Notice
Copyright (c) 2015 IETF Trust and the persons identified as the
document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents
carefully, as they describe your rights and restrictions with respect
Noveck Expires September 9, 2015 [Page 1]
Internet-Draft nfs-extension March 2015
to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3
1.1. Use of Key words defined in RFC2119 . . . . . . . . . . . 3
2. Protocol Extension . . . . . . . . . . . . . . . . . . . . . 4
3. Protocol Extension Mechanisms . . . . . . . . . . . . . . . . 5
3.1. Specific Protocol Mechanisms Designed for Extension . . . 6
3.2. Protocol Extension by XDR Replacement . . . . . . . . . . 6
3.3. Protocol Extension by XDR Extension . . . . . . . . . . . 6
3.4. Combination of Protocol Extension Mechanisms . . . . . . 7
4. Pre-IETF NFS Versioning . . . . . . . . . . . . . . . . . . . 8
4.1. The Pre-IETF NFS Environment . . . . . . . . . . . . . . 8
4.2. Transition from NFSv2 to NFSv3 . . . . . . . . . . . . . 8
5. Initial Approach to NFS Versioning Within IETF . . . . . . . 9
5.1. Transition from NFSv3 to NFSv4 . . . . . . . . . . . . . 9
5.2. Initial Minor Versioning Model for NFSv4 . . . . . . . . 10
5.3. Transition from NFSv4.0 to NFSv4.1 . . . . . . . . . . . 12
5.4. Transition from NFSv4.1 to NFSv4.2 . . . . . . . . . . . 14
5.5. Evolution of Minor Versioning Model within NFSv4 . . . . 15
5.6. The Role of Minor Version Number in NFSv4 . . . . . . . . 16
5.7. Review of NFSv4 Versioning through NFSv4.2 . . . . . . . 17
6. Inherited NFSv4 Versioning Approach . . . . . . . . . . . . . 18
6.1. Non-XDR-based Changes . . . . . . . . . . . . . . . . . . 18
6.2. Inherited NFS Versioning Practices . . . . . . . . . . . 18
6.3. Problems with Inherited NFS Versioning Approach . . . . . 19
7. Formulating a New NFSv4 Extension Approach . . . . . . . . . 22
8. Current Versioning-related Work . . . . . . . . . . . . . . . 22
8.1. New Working Group Versioning Document . . . . . . . . . . 22
8.2. Related Changes to Other Documents . . . . . . . . . . . 23
9. Issues Going Forward . . . . . . . . . . . . . . . . . . . . 24
9.1. Dealing with Minor Version Scope Issues . . . . . . . . . 24
9.2. Issues with Versioning Rules . . . . . . . . . . . . . . 25
10. Security Considerations . . . . . . . . . . . . . . . . . . . 26
11. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 26
12. References . . . . . . . . . . . . . . . . . . . . . . . . . 26
12.1. Normative References . . . . . . . . . . . . . . . . . . 26
12.2. Informative References . . . . . . . . . . . . . . . . . 26
Appendix A. Acknowledgements . . . . . . . . . . . . . . . . . . 28
Author's Address . . . . . . . . . . . . . . . . . . . . . . . . 28
Noveck Expires September 9, 2015 [Page 2]
Internet-Draft nfs-extension March 2015
1. Introduction
This document examines the subject of protocol extension within the
NFS family of protocols. In order to better understand the issues
that exist going forward with NFSv4, we examine the history of
protocol extension throughout the development of NFS including
o The pre-IETF period
o The development of successive NFSv4 minor versions, under minor
versioning rules first defined in [RFC3530] and subsequently
modified based on the working group's then-current needs.
o The consolidation of minor versioning rules in [NFSv4-vers] along
with the other changes made in that document to make NFSv4
versioning more flexible.
With this history in mind, we discuss the issues that the working
group needs to address to create an NFSv4 versioning RFC that will
serve to define a comprehensive approach to version management for
the NFSv4 family of protocols.
1.1. Use of Key words defined in RFC2119
It is intended that these key words be used in this document in a way
consistent with their definition in [RFC2119].
However, because of the considerations noted below, there are some
issues that readers need to be aware of:
o It is not altogether clear whether these keywords are to be
limited to requirements regarding protocol implementations, or
whether they can reasonably be used in specifying guidance
directed at future potential RFCs.
o The nature of this document is such that it does not directly
specify implementation requirements. In some cases it discusses
potential requirements that an RFC derived from [NFSv4-vers])
might direct to implementations. In other cases, requirements are
one step farther removed from protocol implementations. For
example, the versioning RFC might specify the kinds of
requirements that future minor version definition documents and
feature definition documents might specify.
o The documents that are discussed herein, have at times used these
key words in a way that seems to be at variance with the
definitions in [RFC2119]. For details, see Sections 5.3 and 8.2.
Noveck Expires September 9, 2015 [Page 3]
Internet-Draft nfs-extension March 2015
The reader should be aware of our basic practices regarding these
terms.
o We only apply these upper-case key words to directions regarding
protocol implementations, rather than to guidance intended for
those writing RFCs.
o As a result, these keywords will often appear as quotations from
existing documents, either direct or indirect. Readers should be
aware that it is possible that some such uses might not be in
accord with [RFC2119].
o In other contexts, these keywords will be in the context of
proposals/suggestions of what future RFCs could or might say
regarding certain issues.
The use of these terms as part of minor versioning rules poses
troublesome issues in the context of this document.
o These rules have appeared in [RFC3530], [RFC5661], and
[NFSv4-vers], and also in various drafts of [RFC3530bis] and
[NFSv42]. During this period there were multiple switches between
the RFC2119 keywords and corresponding lower-case terms.
o Use of the terms "OPTIONAL" and "REQUIRED" for feature status
would conform to our general practice since a feature status is
essentially a way in which a minor version specification RFC might
REQUIRE (or not) implementations to support a given feature.
Unfortunately, "RECOMMENDED" would not fit that model very well.
o In this document, in the interests of uniformity, we will use
lower-case terms for feature statuses. except when we are
referring to what is said in a particular document which used the
upper-case keywords.
2. Protocol Extension
Protocols often require means by which they can be extended. Such
extensions may be needed to meet new requirements, to correct
protocol weaknesses exposed by experience, or even to correct
protocol bugs (as can happen when protocols are published as RFC's
without fully fleshed-out implementations).
We need to distinguish here between protocol "extension" and
"versioning". Versioning is a form of protocol extension but not
every form of protocol extension can be accommodated within a
versioning paradigm.
Noveck Expires September 9, 2015 [Page 4]
Internet-Draft nfs-extension March 2015
When a versioning paradigm is in place, groups of extensions are
conceived of as ordered, allowing extensions in subsequent versions
to build upon those in previous versions. When multiple extensions
are combined into a single version, each of the extensions may be
built assuming that the others will be present as well. In such
cases, there can be the opportunity to make design changes in the
protocol, allowing elements of the protocol to be restructured,
sometimes in major ways.
When a versioning paradigm is in effect and extensions are optional,
extensions cannot build upon one another, since the presence of any
particular extension cannot be assumed. In such cases, the ability
to restructure the protocol is reduced, but smaller changes may be
introduced more easily.
In this latter case, it is not clear that the word "versioning" is
appropriate. Nevertheless, in this document, we will, as in the
phrase "NFSv4 minor versioning" use the existing terminology without
necessarily subscribing to the view that "versioning" is the
appropriate description.
3. Protocol Extension Mechanisms
Some factors that are often relevant in deciding on the means by
which a protocol will be extended.
o Whether extensions are to be individually selectable (i.e.
optional) or assumed to be always present, allowing one to build
upon another earlier one.
o The size and scope of extensions that will be made.
o Compatibility issues with existing implementations.
o Issues that relate to ensuring that when individual extensions,
separately arrived at, are each optionally allowed, that the ones
used are compatible and can be used effectively together.
o The overall implementation framework. For example, RPC-based
protocols may do extension by means of the RPC version mechanism.
While it is possible to use different sorts of extension mechanisms
for different sorts of extensions, protocols typically do not take
advantage of that flexibility.
On the other hand, protocols do, as NFS has done, change their
preferred extension mechanisms in response to long-term changes in
requirements. However, once having done so, they rarely switch back.
Noveck Expires September 9, 2015 [Page 5]
Internet-Draft nfs-extension March 2015
Changing extension mechanisms is a big step, both conceptually and in
implementation terms, and is not frequently repeated.
3.1. Specific Protocol Mechanisms Designed for Extension
Often, protocols will be designed with specific mechanisms, designed
to allow protocol extension. An example is the provision for TCP
options (see [RFC0793] and [RFC2780].) Most often, such mechanisms
are designed to allow individual extensions to be designed and
implemented independently, with any dependency relations between
extensions specified separately and not enforced by the extension
mechanism itself.
3.2. Protocol Extension by XDR Replacement
RPC-based protocols may, and often do, provide for protocol extension
by replacing the XDR for one version with that for another and using
the RPC versioning mechanism to manage selection of the proper
protocol variant. The use of the RPC versioning mechanism enforces a
versioning paradigm of this sort on protocols using this extension
mechanism.
This extension mechanism allows very extensive protocol changes, up
to and including the replacement of one protocol by an entirely
different one. For some kinds of protocol extensions, this seems the
only way to effect the change.
3.3. Protocol Extension by XDR Extension
It is possible to replace an XDR definition by one which is an
extension in the sense that
o The set of messages described by the second definition is a
superset of that described by the first.
o Each message within the set of messages described by the first
definition is recognized as having exact the same structure/
interpretation by the second definition.
o Each message within the set of messages described by the second
definition but not the first definition must be recognized as part
of an unsupported extension.
Within an XDR/RPC framework, extensions can be arrived at by:
o Addition of previously unspecified RPC operation codes.
o Addition of new, previously unused, values to existing enums.
Noveck Expires September 9, 2015 [Page 6]
Internet-Draft nfs-extension March 2015
o Addition of previously unassigned bit values to a flag word.
o Addition of new cases to existing switches, provided that the
existing switch did not contain a default case.
Such an extension relation between XDR descriptions is reflexive and
transitive and thus defines a partial order. In practice, provisions
have to be made to make sure that two extensions of the same
description are compatible (i.e. either one is an extension of the
other, or there is a another description that is a valid extension of
both).
To put things in concrete terms, such compatibility can be assured if
measures are taken to ensure:
o That the same operation code is not used for two different RPC
operations.
o That the same enum value is not assigned two different meanings.
o That the same bit flag value is not assigned two different
meanings.
o That whenever the same case value is added to the same switch in
two different extensions, the content assigned to the two matching
added cases is the same.
o That default cases are never added to existing switches.
3.4. Combination of Protocol Extension Mechanisms
It is possible to use multiple of these means of protocol extension
simultaneously. When this is done, the result is a composite
extension mechanism. For example, if the XDR replacement or XDR
extension mechanism is adopted, a protocol-specific mechanism can be
added to it, if the protocol-specific mechanism is built on objects
whose XDR definition is sufficiently generic. (e.g. opaque arrays or
feature bitmasks).
It can be argued that the NFSv4 attribute model provides such an
embedded protocol-specific extension mechanism, since sets of
attribute values are specified as XDR opaque arrays and attribute
sets are specified as variable-length arrays of 32-bit integers
allowing new attribute bits to be defined outside of the XDR
definition framework.
Note that there exists specification text that suggests that
attributes are part of the XDR specification, making it hard to reach
Noveck Expires September 9, 2015 [Page 7]
Internet-Draft nfs-extension March 2015
a firm conclusion on the matter. However, the resolution of this
question does not affect the other matters discussed below, since, in
either case, we have an extension mechanism that allows optional
extensions.
4. Pre-IETF NFS Versioning
4.1. The Pre-IETF NFS Environment
NFSv2 and NFSv3 were described by the informational RFC's [RFC1094]
and [RFC1813]. These documents each described existing
interoperating client and server implementations. Thus they started
with running code. If there was a rough consensus in effect, it was
that these were useful protocols to use and thus that someone
building a client or server had to interoperate with the existing
implementations.
The following characteristics of protocol development during that
period are noteworthy.
o Most client implementations were implemented on very similar
systems, in terms of the API's supported and many specifics of the
local filesystems exported by servers.
o Often, the important client and server implementations were done
by the same organization.
As a result of these commonalities, specifications tended to avoid a
lot of detail that would have been required in a more diverse
environment. New features were thought of in terms of generally
understood client and server implementation frameworks and it was
generally clear which of those could be implemented without markedly
changing that framework.
4.2. Transition from NFSv2 to NFSv3
There were a number of significant changes involved, but only the
first two were of major importance.
o Converting file sizes and offsets from 32-bit to 64-bit unsigned
integers.
o Support for uncommitted WRITEs and the COMMIT request.
o Provision for WRITE to return atomic pre- and post-write file
attributes, in order to allow a client to determine whether
another client was writing the file.
Noveck Expires September 9, 2015 [Page 8]
Internet-Draft nfs-extension March 2015
Interestingly enough, this feature was not carried over into
NFSv4.
o The READDIRPLUS request.
o The addition of NFS3ERR_JUKEBOX (the precursor of NFS4ERR_DELAY).
Of these only the first actually needed something as drastic as the
XDR replacement model. The others could have been handled simply by
adding new RPC requests and an enum value to an existing NFSv2 XDR.
Since, NFS's extension mechanism was then XDR replacement, such
choices were not available.
5. Initial Approach to NFS Versioning Within IETF
5.1. Transition from NFSv3 to NFSv4
NFSv4 was the first NFS version published as a Standards track
document. Although an initial [RFC3010], entitled "NFS version 4
protocol" was published as a Proposed Standard, it was never
implemented due to issues with the design of locking.
Subsequently, [RFC3530], entitled "Network File System (NFS) version
4 Protocol" was published as a Proposed Standard, obsoleting
[RFC3010]. Currently, there are bis documents, [RFC3530bis] and
[RFC3530bis-dotx], nearing publication.
The set of changes made to create NFSv4 was larger by far than that
for NFSv3. A partial list follows.
o Conversion to a stateful protocol, including support for locking.
Locking facilities included support for OPENs (with share
reservations), byte-range locking (optionally including mandatory
byte-range locks) and delegations.
o The COMPOUND operation.
o Conversion to a bi-directional protocol, by the addition of
(optional) callbacks.
o Internationalization.
o Support for filesystems doing case-insensitive name matching.
o A new, extensible file attribute model, including support for
acls, and conversion of user and group to a string model, opening
the way for multi-domain support.
Noveck Expires September 9, 2015 [Page 9]
Internet-Draft nfs-extension March 2015
o Support for named attributes.
o Merger of ancillary protocols (e.g. MOUNT, NSM, NLM) into the NFS
Protocol proper.
o Support for crossing of filesystem boundaries within a server's
file name space (originally done for the incorporation of MOUNT
functionality).
o Support for such multi-server operations as migration,
replication, and referral.
Referrals were not explicitly mentioned in [RFC3530] and are
explained in [RFC3530bis].
o Creation of a minor versioning model (to be discussed in
Section 5.2) to allow further protocol extension.
These features/extensions were implemented via the XDR replacement
model. This was the only realistic alternative, not only because of
the size of the list, but also because some of the changes undercut
some of the central design elements of the pre-IETF NFS protocol.
5.2. Initial Minor Versioning Model for NFSv4
The minor versioning approach for NFSv4 uses an XDR extension model.
It was presented within a versioning paradigm but the fact that all
the added features were to be (at least initially) optional indicated
that features were intended to be built independently, and that
clients were expected to deal with their presence or absence. Note
that the term "features" is not explicitly defined. We assume that
the definition includes operations within COMPOUND or CB_COMPOUND,
attributes, added flag bits and enum values, and new cases of XDR
switch definitions.
Now let's look at some specifics of the minor version rules
established for NFSv4 in [RFC3530]. Note that some of these were
significantly modified by [RFC5661] and [NFSv42], as discussed in
Section 5.6.
o No RPC requests may be added. Thus COMPOUND (and NULL) are to be
the only requests within all NFSv4 minor versions.
Similarly for callbacks, CB_COMPOUND and CB_NULL are the only
requests within the callback program.
o The arguments for COMPOUND and CB_COMPOUND contain a 32-bit
minorversion field.
Noveck Expires September 9, 2015 [Page 10]
Internet-Draft nfs-extension March 2015
Although this field is part of the minor versioning paradigm, it
is not clear how useful it is, as long as all extensions are
optional. For a more detailed discussion of this issue, see
Section 5.6.
o Features may be defined as optional, recommended, or required.
Later, these designations were converted to use the corresponding
upper-case terms defined in [RFC2119]. See Sections 5.3 and 8.2
for details.
These designations apply to implementation by the server. For
clients, no operations are defined as required, although it is
hard to imagine an NFSv4.0 client that does not use PUTFH or
SETCLIENTID, for example.
o Features may be upgraded or downgraded along the
optional/recommended/required scale.
o Features may be declared "mandatory to not implement". This
allows the deletion of a feature while retaining as reserved the
value previously assigned.
o Clients and servers that support a particular minor version must
support all previous minor versions as well.
o New features may not be designated as required in the minor
version in which they are introduced.
o Clients are not allowed to use stateids, filehandles, or similar
returned objects from the COMPOUND procedure with one minor within
another COMPOUND procedure with a different value of the
minorversion field.
This model was subsequently modified in [RFC5661] and in [NFSv42].
See Sections 5.3 and 5.4 for details.
Many of the events anticipated in the model presented above have
never been realized and it may be that they never will be realized.
See Section 5.5 for some details. Examples are:
o There have never been recommended operations.
o There have never been optional attributes.
o Features have never been upgraded or downgraded in the transition
between minor versions.
Noveck Expires September 9, 2015 [Page 11]
Internet-Draft nfs-extension March 2015
5.3. Transition from NFSv4.0 to NFSv4.1
NFSv4.1 was described in [RFC5661] and [RFC5662], each of which was
published as a Proposed Standard.
NFSv4.1 made a major change to NFSv4.0. It was able to do primarily
using an XDR extension model although it did not follow the rules
laid out in Section 5.2. Instead, [RFC5661] presented its own set of
minor versioning rules.
The following major changes in the rules were made.
o Uses of the terms "recommended", "required", "should", etc. were
switched to use the corresponding upper-case words defined in
[RFC2119]. This included conversion of "recommended" attributes
to be "RECOMMENDED". The reasons for this change remain unclear.
Unlike the changes noted below, this change was incorporated in
[RFC3530bis]. For a discussion of how this situation was dealt
with during the IESG review of that document, see Section 8.2
o The rule requiring that features be introduced initially as
optional was modified to enable features described as
"infrastructural" to be required upon introduction.
o Also, the requirement that clients and servers support previous
minor versions changed from a "must" to a "SHOULD". Presumably,
this change reflects the fact that a minor version with
substantial infrastructural changes is essentially a new protocol,
making the "must" seem dubious. Whether the "SHOULD" here is in
line with [RFC2119] needs to be explored.
NFSv4.1 also bypassed the versioning rules by making non-XDR-based
protocol changes. See Section 6.1 for a discussion of the logic
behind such changes.
The following features were added as infrastructural features.
o Support for a sessions model including support for EOS (exactly-
once semantics). Implementation of this feature included some
non-XDR-based changes in operation behavior.
Note that COMPOUND was taken advantage of to avoid adding slot and
sequence information to the request header. Instead this
information is packaged in a SEQUENCE or CB_SEQUENCE operation at
the start of the COMPOUND or CB_COMPOUND.
Noveck Expires September 9, 2015 [Page 12]
Internet-Draft nfs-extension March 2015
o A new set of operations were added which enable the client and
server to identify themselves to one another.
Although these are often thought of as part of the sessions model,
in fact they are logically distinct.
o RECLAIM_COMPLETE was added to allow better server sequencing of
lock reclaim operations.
In addition to these required features, there were a number of non-
XDR-based changes made in NFSv4.1. Although not thought of as
"features" at the time, they are described in [RFC5661], which gives
no indication that support is optional. These include:
o Addition of a new special stateid to represent the current
stateid.
o New rules for the interpretation of a stateid seqid of zero.
o New rules regarding the interaction of the MODE and acl-related
attributes.
There also a number of non-required features. While such features
are optional in the sense that a server may or may not support them,
it is not clear that all are optional in terms of the existing minor
versioning rules. Given that all attributes are specified as
REQUIRED OR RECOMMENDED, it may be that attributes that may or may
not be supported are considered recommended from the viewpoint of the
minor versioning rules. Here and in the future we will use "non-
required" instead of "optional or recommended".
The following non-required features were added.
o Parallel NFS
o WANT_DELEGATION to allow delegations to be obtained apart from
opens.
o Directory delegations and notifications.
o The MODE_SET_MASKED, SACL, DACL attributes.
o The FS_LOCATIONS_INFO and FS_STATUS attributes.
Note that there has been little implementation work on the last two
of these.
Noveck Expires September 9, 2015 [Page 13]
Internet-Draft nfs-extension March 2015
Parallel NFS created an alternate protocol extension mechanism for
NFS. New pNFS mapping types could be added. Existing mapping types
might have their own extension mechanisms. There also exists the
possibility that features might be added within the NFSv4 protocol
proper, designed to, or capable of, interacting with particular
mapping types. This document will not these issues but eventually,
the NFSv4 Versioning RFC will have to address them.
5.4. Transition from NFSv4.1 to NFSv4.2
While NFSv4.2 has not been defined in an RFC, its development has
been going on for some time and it is unlikely to experience
additions to or deletions from its feature set before completion.
The descriptions in [NFSv42] and [NFSv42-dotx] can serve as useful
references for this analysis. This is despite the fact that some of
the features may later experience changes.
Because of the lengthy process involved in producing NFSv4.1, the
working group decided that NFSv4.2 was to be a relatively small minor
version consisting only of optional features which would be
documented by specifying changes from NFSv4.1.
The following features (all optional) are provided for in NFSv4.2:
o Support for labeled NFS.
o Server-side copy.
o An operation fence option on EXCHANGE_ID.
o Application data holes (formerly application data blocks).
o Disk-space reservation (nominally "recommended" since it is
implemented by an attribute and attributes have never been
declared "optional").
o Hole-punching operations.
o READ_PLUS.
Note that there are two pieces of infrastructure that are used by
multiple features above. These are not "infrastructural" in the
sense mentioned in Section 5.3 (i.e. they are not required), but they
do serve an infrastructural role in that are required to be present
if one of the optional features that use them are supported.
o WRITE_PLUS used to implement (ordinary) hole punching and
application data holes.
Noveck Expires September 9, 2015 [Page 14]
Internet-Draft nfs-extension March 2015
o OFFLOAD operations/callbacks used to support WRITE_PLUS and
server-side copy.
5.5. Evolution of Minor Versioning Model within NFSv4
As noted above, there have been changes made by [RFC5661] and
[NFSv42] in the NFSV4 minor versioning model.
o NFSv4.1 (in [RFC5661]) introduced the concept of "infrastructural"
features (i.e. those defined as required at initial introduction).
o NFSV4.1 made additional non-XDR-based changes, which, although not
explicitly defined as such, were effectively required.
Taking note of these changes, we can classify potential minor
versions, starting with those that currently exist, based on how they
affect inter-version compatibility.
o Minor version zero which introduced a new (major) version of the
NFS protocol. All of the operations within it are new and a
subset are effectively required.
o Minor versions which make required changes in the protocol that
affect all implementations of the previous minor version.
Such changes may include addition of required/infrastructural
feature, incorporation of an effectively required non-XDR-based
change, or the deletion of a required feature by making it
mandatory to not implement.
Currently, the only such version is minor version one, although it
is possible that there could be others in the future.
o Features that only add optional/recommended features, each present
or absent on the server with clients needing to be able to deal
individually with their presence or absence.
Currently, the only such version is minor version two. It is
likely there will be others in the future and it is possible that
all future minor versions will be of this character.
o Minor versions which make required changes in the protocol that
will affect some implementations of the previous minor version.
These can result from feature status changes. Changing a non-
required feature to required affects only implementations that do
not support the feature. Conversely, deleting a non-required
Noveck Expires September 9, 2015 [Page 15]
Internet-Draft nfs-extension March 2015
feature by making it mandatory to not implement. affects only
implementations that do support the feature.
No such minor versions currently exist.
5.6. The Role of Minor Version Number in NFSv4
Minor versions which may affect inter-version compatibility, form an
ascending sequence in which we also have a versioning paradigm,
implemented principally using XDR extension.
Optional-feature-only versions are fundamentally different. Each
NFSv4.2 server implements the same protocol as NFSv4.1 with a
particular set of optional or recommended features beyond those that
are required. This set may range from the null set all the way to
all of the non-required features. Here, it appears that the
versioning paradigm is not appropriate to the reality of the
extension mechanism.
As a way of illustrating the basic point here, let us consider two
servers each of which only supports operations within NFSv4.1:
o The first server "supports" NFSv4.2 but none of the non-required
features added in [NFSv42]. In this case, any attempt by a client
to use one of those features will result in an NFS4ERR_OPNOTSUPP
being returned.
o Let us say that the second server does not support NFSv4.2 and
supports precisely the same set of features. In this case, a
request will be rejected (with error NFS4RR_MINOR_VERS_MISMATCH)
if its COMPOUND minorversion field is two and if the field is one,
any unsupported NFSv4.2 operation will be rejected with
NFS4ERR_OP_ILLEGAL.
Although this obeys the rules as they stand, there is no obvious
value for the client, the server, or the protocol in making these
artificial distinctions. Optional-feature-only minor versions such
as NFSv4.2 are not minor versions in the same sense that NFSv4.0 and
NFSv4.1 are. In this case the minorversion field is not providing
any information, while the set of operations supported is the
important thing that the server implementer chooses and the client
needs to know.
It is not clear how thus mismatch will be resolved. Nevertheless, it
is an indication that the focus previously placed on the construction
of minor versions was misplaced and that managing the addition of
features is in most cases the fundamental issue. How minor versions
Noveck Expires September 9, 2015 [Page 16]
Internet-Draft nfs-extension March 2015
are to fit within that framework is something that the group needs to
decide.
5.7. Review of NFSv4 Versioning through NFSv4.2
To summarize protocol extension as it has been applied to the NFSv4
protocols:
o NFSV4.0 was implemented using the XDR replacement approach
inherited from NFSv2 and NFSv3. As was to be expected given the
nature and scope of the changes, its development took considerable
time.
It defined a protocol extension approach based on the XDR
extension mechanism which was designed to enable future
development of minor versions. However, this mechanism was not
used as part of the implementation of NFSv4.0.
o NFSV4.1 was primarily implemented using the XDR extension
mechanism. To implement sessions, it was forced to modify the
extension approach in the only way that was viable in the
circumstances. As a result, the specification process took a long
time, since it made significant structural changes to the protocol
and also because it specified the entire protocol in a new RFC,
rather than documenting a set of extensions.
NFSv4.1 also made a number of modifications to the NFSv4 protocol
which did not involve any XDR-based changes at all. These are
discussed in Section 6.1. While not considered infrastructural,
or even thought of as "features" at the time, these changes are
effectively required and the possibility of such changes needs to
be considered as the working group formulates its approach to the
problems of NFSv4 version management.
o NFSV4.2 returned to the original XDR extension mechanism and was
intended to be a small incremental update with a one-hundred page
(or less) specification. The fact that this turned out to be a
multi-year effort has occasioned concern and we will discuss how
the process can be streamlined and otherwise made more effective.
The history of NFSv4.2 development serves to illustrate some of
the inherent problems in the then-current approach to minor
versioning. Since similar problems can be expected to recur
unless that approach is changed, attention needs to be paid to
understanding why such difficulties were experienced.
It is important to note that NFSv4.0 (in [RFC3530] and [RFC3530bis]),
both about 300 pages and NFSV4.1 (in [RFC5661]), about 600 pages
Noveck Expires September 9, 2015 [Page 17]
Internet-Draft nfs-extension March 2015
documented the entire minor version. On the other hand, the NFSv4.2
specification document (in [NFSv42]) simply specified the changes
from the previous minor version, in about 100 pages.
Given the difficulties of writing very large specifications, it has
to be considered extremely unlikely, that any future minor versions
will be documented other than as a set of changes from the previous
minor version.
6. Inherited NFSv4 Versioning Approach
6.1. Non-XDR-based Changes
Although not recognized at the time, the existence of non-XDR-based
protocol changes is part of the existing NFSv4 versioning approach
and must be addressed in any revision of NFSv4 versioning.
Such changes have been of two types:
o Changes in field in interpretation and use. Although the
intention has always been that all such matters were to be defined
in XDR, there are areas where this is not true. The
interpretation of certain opaque fields and of strings are cases
in which the field interpretation and use is defined in protocol
specification text.
o Changes in operation semantics. These may apply to a few
operations or to most or all operations.
All such changes have been implemented as required with
implementations using the minor version field of the COMPOUND to
determine whether the change applies to a particular request.
6.2. Inherited NFS Versioning Practices
The following pattern was followed for NFSv4.2, and, unless changes
are made, seems likely to persist.
o Various features are sketched out in individual drafts
o The working group reaches a decision (i.e. by rough consensus) as
to the extensions/features to be included in the minor version.
At the time this decision is made, the features may vary as to
their maturity. Some have individual drafts which sketch them out
while others do not.
o Any existing individual drafts are combined into a draft of a
working group document intended to eventually evolve into an RFC
Noveck Expires September 9, 2015 [Page 18]
Internet-Draft nfs-extension March 2015
describing the new minor version. These are then supplemented
with descriptions of the other features chosen for inclusion.
o This document goes though further refinement and cycles of working
group document review. At some point a companion -dot-x document
is prepared and reviewed by the working group as well.
o The two documents go through working group last call, IESG review,
and RFC publication.
This pattern of development is not a good fit for the kind of minor
version that NFSv4.2 is and many future such minor versions will be.
Such versions consist of a set of mostly unrelated features, each
individually selectable or not by implementers, artificially yoked
together. In essence, we have a "feature batch" rather than a minor
version.
6.3. Problems with Inherited NFS Versioning Approach
A number of issues have been noted with the existing process for
NFSv4.2, leading to the conclusion that the process needs to be
revised in some way for future minor versions, of the same sort.
o It takes too long to get a minor version drafted and through
working group and IESG review. Despite the fact that NFSv4.2 was
intended to be a fairly minimal minor version, describable in a
one-hundred-page spec, it looks like the pace of development is
such that there will be nearly a four-year delay from the time
that the first NFSv4.2 draft was submitted to the point at which
the final draft is ready to be submitted to the IESG.
o The timeline for some of the features within NFSv4.2 is even
longer. It will have taken about six years for the inter-server
copy feature to go from initial draft to IESG submission of the
minor version of which it is a part.
o Only very recently have we had significant active implementations
in which proposed last-minute protocol changes can be tested for
validity. As an example of the problem, consider the decision to
pass source stateids to the COPY op. If we had had prototype
implementations of inter-server copy, the problems that this
created (since stateids are tied to clientids in NFSv4.1 and
beyond) would have quickly become manifest.
o Many features within NFSv4.2 did not received the kind of
searching review appropriate to later stages of specification
development.
Noveck Expires September 9, 2015 [Page 19]
Internet-Draft nfs-extension March 2015
Some instances of problems/issues ascribable to a lack of searching
document review are described below. Rather than requiring the
necessary review prior to feature acceptance, a common pattern has
been that important issues are only discovered on those occasions in
which it appears that work on the minor version is coming to a close
and that there are issues that have to be addressed before they
create difficulties for prospective implementers.
o The state of the IO hints feature is most unsatisfactory. It is
not clear how, or even if, it is possible to specify this in a way
that interoperable clients and servers can be written which
respond appropriately to such hints so that useful performance
improvements can be demonstrated.
o It was the general understanding within the group that labeled NFS
required use of RPCSEC_GSSv3, when in fact, only one case of
labelled NFS, the server-enforced one, required it. When it
became clear that RPCSEC_GSSV3 had not been worked on for a long
time, the working group had to address that issue seriously.
o The security for inter-server copy was specified to be dependent
on RPCSEC_GSSv3, yet, when it was found that RPCSEC_GSSv3 seemed
not to be on the horizon, it turned out that a simple alternative
was available. After a great deal of uncertainty about whether
the functionality needed for inter-server copy would really be
doable in RPCSEC_GSSv3, the working group had a range of choices
for inter-server copy security.
Later, after much work was done on RPCSEC_GSSv3, the working group
has the option of going back to an approach dependent on
RPCSEC_GSSv3.
o Application data blocks and related features have had a
complicated history within NFSv4.2. Application data blocks were
superseded by application data holes. There was little interest
in implementing the feature since, as specified, a server
implementation would require extensive filesystem extensions.
Also, client-side API's were lacking, meaning that the only
possible client-side implementations would be in special-purpose
clients tightly bound to a particular implementation.
Nevertheless, the feature continued in this unsatisfactory state
for a long while.
Eventually, it became clear that, as the feature was defined, it
imposed significant implementation constraints even on NFSv4.2
clients not implementing the feature. At this point, it became
clear that significant changes had to be made.
Noveck Expires September 9, 2015 [Page 20]
Internet-Draft nfs-extension March 2015
This issue was resolved by limiting the scope of the proposed
feature so that servers were not required to remember the
parameters relevant to application data holes, but only the data
that they represent.
If we look at the problems above, we can understand better how such
problems can arise. In short, the decision as to what features to
include within a minor version, is not a good use of the rough
consensus model and in proceeding on that basis, the group created a
set of perverse incentives that undercut the process. Also, as the
process goes on for a long time, as is likely, these perverse
incentives are intensified. Consider the following points:
o It is not clear exactly what the consensus as to proposed minor
version contents actually means. Working group members might
interpret it as meaning "These features are worth pursuing and
they should be pursued". However, if they thought the definition
was more like, "each of these features is so important that, if it
is not ready, any other feature, including the one I'm interested
in, should be delayed also", then it is hard to imagine any such
rough consensus existing. Note that, given the minor versioning
implementation laid out above (in Section 6.2), the latter
definition is, for functional purposes, the effective meaning, of
the minor version content consensus.
o Given that many features are linked together, any delay in one
feature, once it is accepted as part of the feature batch, delays
all of the features, making it hard for people to comment
forthrightly on any significant specification inadequacies. Not
only will it delay your preferred feature, but if the problems are
not fixed, the only recourse is an extreme penalty. As a result,
it often seems not worth pursuing these sorts of issues.
o As the version turnaround cycle is so long, it is very difficult
to remove a feature from a minor version feature batch. Given
that these are all features that have enough interest to be in the
minor version, it is hard to transfer the feature into the next
minor version, given that doing so will certainly result in a
multi-year delay, at a minimum.
As a result, features, once accepted, have an implicit guarantee
of inclusion in the minor version, undercutting the motivation of
the proposer and others to work to move the feature forward.
On the other hand, uncertainty about the time of specification
completion, often makes it is hard to plan for and allocate
resources to development of client and server implementations.
Noveck Expires September 9, 2015 [Page 21]
Internet-Draft nfs-extension March 2015
o Given that responsibility for a minor version is transferred to
the editor of the minor version definition document at an early
stage, we have a process in which it is not clear who has
responsibility to follow up on the work necessary other than the
minor version editor who may not have the required time and
expertise in all cases. There is not a specific feature owner
with the responsibility to make the feature happen.
7. Formulating a New NFSv4 Extension Approach
The following issues led the working group to consider formulation of
a new approach to NFSv4 versioning:
o The experience of NFSv4.2 showed that, although we needed to make
our documents smaller, doing so without other changes in how we
did things left important issues unaddressed.
o The lack of implementation experience for many features led to a
general feeling that the working group needed to do more to
encourage/require development of feature implementations before
they were published as Proposed Standards.
o The publication of multiple minor versioning rules came to seem to
variance with the idea of a rule-based approach. Eventually, it
was decided that a single version management document was needed
for NFSv4 as a whole.
8. Current Versioning-related Work
8.1. New Working Group Versioning Document
The working group has started work on a standards-track document
([NFSv4-vers]) defining an approach to version management that is
intended to apply to NFSv4 as a whole. Noteworthy advances in that
document include
o Creation of a set of minor versioning rules to form a basis for a
unified of set versioning rules. Such a set of rules would
supersede the per-minor-version rules that had previously been
present.
o Creation of the concept of extensible minor version with an
associated workflow that avoids many of the problems noted above
with regard to the batching of features.
o Explicitly allowing XDR changes to correct protocol errors. This
addresses a situation in which there was sufficient uncertainty
Noveck Expires September 9, 2015 [Page 22]
Internet-Draft nfs-extension March 2015
about such changes to prevent them from being considered, even
though they were not explicitly disallowed.
8.2. Related Changes to Other Documents
During this period, changes were made to some related documents, as
they moved toward publication.
Some of these changes were made to simplify handling of the
transition in versioning models that would occur upon publication of
[NFSv4-vers] as an RFC.
o During the IESG review process, [RFC3530bis] was modified to
eliminate specific minor versioning rules.
This made sense because the rules in question did not apply to
NFSv4.0, and were, with regard to NFSv4.1 and beyond, superseded
by those in [RFC5661].
o The restatement of minor versioning rules in [NFSv42] was
eliminated. In its place was left a short statement of
v4.2-relevant exceptions from existing rules. The text is
compatible with the base rules being taken from either [RFC5661]
or [NFSv4-vers].
As a result of these changes, upon publication of an NFSv4 version
management RFC, it will only need to be marked as updating [RFC5661].
Another relevant change concerned the use of the term "RECOMMENDED"
regarding attributes which are not REQUIRED. As it appeared that
this usage is inconsistent with [RFC2119], [RFC3530bis] was modified
to include the following statement:
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL"
in this document are to be interpreted as described in RFC 2119
except where "REQUIRED" and "RECOMMENDED" are used as qualifiers
to distinguish classes of attributes as described in
Section 1.3.3.2 and Section 5.
A reasonable conclusion to draw from this is that while certain
attributes are indeed REQUIRED, the other ones cannot be designated
as "RECOMMENDED" as that term is defined in [RFC2119]. The following
issues remain to be resolved:
o How to deal with the analogous issue in [RFC5661].
Noveck Expires September 9, 2015 [Page 23]
Internet-Draft nfs-extension March 2015
o How this affects the use of "RECOMMENDED" (and possibly of
"recommended") as a feature status.
9. Issues Going Forward
As of now, the versioning document incorporates two conceptual models
which need to be better integrated as the document moves toward RFC
status
o The versioning rules inherited from [RFC5661] are focused on minor
versions and individual feature elements (which are referred to as
"features"). Even though the addition of incremental features is
now at the core of our prospective versioning practice, the rules
relate to that only by implication.
o The version extensibility and protocol fix work is focused on
(non-required) features, but has no real places for minor versions
other than as a passive recipient of those features. It was
decided to focus on this aspect of version management for good
reasons but if the working group wants to leave open the
possibility of minor versions that might refactor things or
otherwise allow other sorts of minor-version-level change, the
versioning RFC should be written to enable that.
9.1. Dealing with Minor Version Scope Issues
With the exception of the incorporation of minor versioning rules
(based on those in [RFC5661]), the current NFSv4 versioning document
is focused on addressing the problems that have been seen with minor
versions that are like NFSv4.2, in that they add non-required
features and do nothing else. Such versions may not make any of the
following types of changes:
o Addition of required features.
o Changes in the status of existing features, either to upgrade or
downgrade them. In this context, eliminating a feature, i.e.
making it mandatory to not implement, is considered a form of
downgrading.
o Non-XDR changes in the protocol. Instances of such changes, which
include changes in field interpretation and use and changes to
operation semantics have occurred in NFSv4.1 and are discussed in
Section 6.1. Such changes are not provided for in the existing
minor versioning rules.
Each of the above types of change has occurred as part of NFSv4.1.
While it is quite likely that the working group is prepared to
Noveck Expires September 9, 2015 [Page 24]
Internet-Draft nfs-extension March 2015
foreclose future minor versions with as large a scope as NFSv4.1, it
is not clear that the working group is prepared to decide that only
changes like those in NFSv4.2 be made in all future minor versions.
As work proceeds on [NFSv4-vers], the working group will have to
decide which, if any of the above sorts of changes to allow in minor
versions, even if most minor versions do not use these additional
forms of change. As part of that, the working group will have to
decide:
o Whether to impose restrictions on use of certain of these
additional forms of change.
Such restrictions could include periods of experimental use or
obsolescence windows. They might also limit the combination of
otherwise valid sorts of changes. For example, some changes might
be allowed to be required at initial introduction and non-XDR-
based changes might be allowed as well but required non-XDR-based
changes might be excluded.
o To determine how these additional kinds of change can be
integrated in the documentation model within [NFSv4-vers], which
is now focused on non-required features that only consist of XDR-
based extensions.
9.2. Issues with Versioning Rules
One way of dealing with the issues surrounding these rules is to re-
organize them so as to move away from a single set of minor
versioning rules, while adapting them to the evolving versioning
model in [NFSv4-vers].
Although the result may not have the format of a numbered list of
rules, there will be sets of guidelines that serve the function. One
possible organization is the following:
o Rules defining the types of protocol changes that may be made.
These would include the XDR extensions mentioned in the existing
rules, but there would also have to be provision for at least some
of the non-XDR-based changes that have been useful in the past.
o Rules governing the construction and organization of features.
o Rules governing the incorporation of features in the protocol,
whether this as part of a published minor version, an extension to
a published minor version, or as a correction to a published minor
version.
Noveck Expires September 9, 2015 [Page 25]
Internet-Draft nfs-extension March 2015
o Rules governing the changes of feature statuses in a minor
version. Such rules would also deal with possibility of post
facto changes in the composition of features or of inter-feature
compatibility constraints. In this same category would be
handling of feature re-organization including split and merger.
o Rules that deal with the interaction of multiple minor versions,
on the model of rules 11 and 13 in [RFC5661]. These are the only
rules directed to implementers, as opposed to those creating minor
versions.
Depending on the decisions the working group makes, some of the above
rule categories may not exist.
10. Security Considerations
Since no substantive protocol changes are proposed here, no security
considerations apply.
As features and minor versions designed and specified in standards-
track documents, their security issues will be addressed and each RFC
candidate will receive the appropriate security review from the NFSv4
working group and IESG.
11. IANA Considerations
The current document does not require any actions by IANA.
Depending on decisions that the working group makes about how to
address the issues raised in this document, future documents may
require actions by IANA.
12. References
12.1. Normative References
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
12.2. Informative References
[NFSv4-vers]
Haynes, T. and D. Noveck, "NFSv4 Version Management",
November 2014, <http://www.ietf.org/id/
draft-ietf-nfsv4-versioning-00.txt>.
Work in progress.
Noveck Expires September 9, 2015 [Page 26]
Internet-Draft nfs-extension March 2015
[NFSv42] Haynes, T., Ed., "NFS Version 4 Minor Version 2", March
2015, <http://www.ietf.org/id/
draft-ietf-nfsv4-minorversion2-33.txt>.
Work in progress.
[NFSv42-dotx]
Haynes, T., Ed., "NFS Version 4 Minor Version 2 External
Data Representation Standard (XDR) Description", March
2015, <http://www.ietf.org/id/
draft-ietf-nfsv4-minorversion2-dot-x-33.txt>.
Work in progress.
[RFC0793] Postel, J., "Transmission Control Protocol", STD 7, RFC
793, September 1981.
[RFC1094] Nowicki, B., "NFS: Network File System Protocol
specification", RFC 1094, March 1989.
[RFC1813] Callaghan, B., Pawlowski, B., and P. Staubach, "NFS
Version 3 Protocol Specification", RFC 1813, June 1995.
[RFC2780] Bradner, S. and V. Paxson, "IANA Allocation Guidelines For
Values In the Internet Protocol and Related Headers", BCP
37, RFC 2780, March 2000.
[RFC3010] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "NFS version 4
Protocol", RFC 3010, December 2000.
[RFC3530] Shepler, S., Callaghan, B., Robinson, D., Thurlow, R.,
Beame, C., Eisler, M., and D. Noveck, "Network File System
(NFS) version 4 Protocol", RFC 3530, April 2003.
[RFC3530bis]
Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 Protocol", December 2014,
<http://www.ietf.org/id/
draft-ietf-nfsv4-rfc3530bis-35.txt>.
Work in progress.
Noveck Expires September 9, 2015 [Page 27]
Internet-Draft nfs-extension March 2015
[RFC3530bis-dotx]
Haynes, T., Ed. and D. Noveck, Ed., "Network File System
(NFS) Version 4 Protocol External Data Representation
Standard (XDR) Description", December 2014,
<http://www.ietf.org/id/
draft-ietf-nfsv4-rfc3530bis-dot-x-24.txt>.
Work in progress.
[RFC5661] Shepler, S., Eisler, M., and D. Noveck, "Network File
System (NFS) Version 4 Minor Version 1 Protocol", RFC
5661, January 2010.
[RFC5662] Shepler, S., Eisler, M., and D. Noveck, "Network File
System (NFS) Version 4 Minor Version 1 External Data
Representation Standard (XDR) Description", RFC 5662,
January 2010.
Appendix A. Acknowledgements
The author wishes to thank Chuck Lever of Oracle for his helpful
document review and many important suggestions.
Author's Address
David Noveck
Dell
300 Innovative Way
Nashua, NH 03062
US
Phone: +1 781 572 8038
Email: davenoveck@gmail.com
Noveck Expires September 9, 2015 [Page 28]