Network Working Group M. Chadalapaka
Request for Comments: 5047 HP
Category: Informational J. Hufferd
Brocade Inc.
J. Satran
IBM
H. Shah
Broadcom Corporation
October 2007
DA: Datamover Architecture for
the Internet Small Computer System Interface (iSCSI)
Status of This Memo
This memo provides information for the Internet community. It does
not specify an Internet standard of any kind. Distribution of this
memo is unlimited.
Abstract
The Internet Small Computer System Interface (iSCSI) is a SCSI
transport protocol that maps the SCSI family of application protocols
onto TCP/IP. Datamover Architecture for iSCSI (DA) defines an
abstract model in which the movement of data between iSCSI end nodes
is logically separated from the rest of the iSCSI protocol in order
to allow iSCSI to adapt to innovations available in new IP
transports. While DA defines the architectural functions required of
the class of Datamover protocols, it does not define any specific
Datamover protocols. Each such Datamover protocol, defined in a
separate document, provides a reliable transport for all iSCSI PDUs,
but actually moves the data required for certain iSCSI PDUs without
involving the remote iSCSI layer itself. This document begins with
an introduction of a few new abstractions, defines a layered
architecture for iSCSI and Datamover protocols, and then models the
interactions within an iSCSI end node between the iSCSI layer and the
Datamover layer that happen in order to transparently perform remote
data movement within an IP fabric. It is intended that this
definition will help map iSCSI to generic Remote Direct Memory Access
(RDMA)-capable IP fabrics in the future comprising TCP, the Stream
Control Transmission Protocol (SCTP), and possibly other underlying
network transport layers, such as InfiniBand.
Chadalapaka, et al. Informational [Page 1]
RFC 5047 DA October 2007
Table of Contents
1. Motivation ......................................................4
1.1. Intent .....................................................4
1.2. Interpretation of Requirements .............................5
2. Definitions and Acronyms ........................................5
2.1. Definitions ................................................5
2.2. Acronyms ...................................................6
3. Architectural Layering of iSCSI and Datamover Layers ............7
4. Design Overview .................................................9
5. Architectural Concepts .........................................10
5.1. iSCSI PDU Types ...........................................10
5.1.1. iSCSI Data-Type PDUs ...............................10
5.1.2. iSCSI Control-Type PDUs ............................11
5.2. Data_Descriptor ...........................................11
5.3. Connection_Handle .........................................11
5.4. Operational Primitive .....................................12
5.5. Transport Connection ......................................13
6. Datamover Layer and Datamover Protocol .........................13
7. Functional Overview ............................................14
7.1. Startup ...................................................14
7.2. Full Feature Phase ........................................15
7.3. Wrap-up ...................................................15
8. Operational Primitives Provided by the Datamover Layer .........16
8.1. Send_Control ..............................................16
8.2. Put_Data ..................................................17
8.3. Get_Data ..................................................17
8.4. Allocate_Connection_Resources .............................18
8.5. Deallocate_Connection_Resources ...........................19
8.6. Enable_Datamover ..........................................19
8.7. Connection_Terminate ......................................20
8.8. Notice_Key_Values .........................................20
8.9. Deallocate_Task_Resources .................................20
9. Operational Primitives Provided by the iSCSI Layer .............21
9.1. Control_Notify ............................................21
9.2. Connection_Terminate_Notify ...............................22
9.3. Data_Completion_Notify ....................................22
9.4. Data_ACK_Notify ...........................................23
10. Datamover Interface (DI) ......................................23
10.1. Overview .................................................23
10.2. Interactions for Handling Asynchronous Notifications .....24
10.2.1. Connection Termination ............................24
10.2.2. Data Transfer Completion ..........................24
10.2.3. Data Acknowledgement ..............................25
10.3. Interactions for Sending an iSCSI PDU ....................25
10.3.1. SCSI Command ......................................26
10.3.2. SCSI Response .....................................26
10.3.3. Task Management Function Request ..................26
Chadalapaka, et al. Informational [Page 2]
RFC 5047 DA October 2007
10.3.4. Task Management Function Response .................27
10.3.5. SCSI Data-Out and SCSI Data-In ....................27
10.3.6. Ready To Transfer (R2T) ...........................28
10.3.7. Asynchronous Message ..............................28
10.3.8. Text Request ......................................28
10.3.9. Text Response .....................................28
10.3.10. Login Request ....................................29
10.3.11. Login Response ...................................29
10.3.12. Logout Command ...................................29
10.3.13. Logout Response ..................................30
10.3.14. SNACK Request ....................................30
10.3.15. Reject ...........................................30
10.3.16. NOP-Out ..........................................30
10.3.17. NOP-In ...........................................30
10.4. Interactions for Receiving an iSCSI PDU ..................31
10.4.1. General Control-Type PDU Notification .............31
10.4.2. SCSI Data Transfer PDUs ...........................31
10.4.3. Login Request .....................................32
10.4.4. Login Response ....................................32
11. Security Considerations .......................................33
11.1. Architectural Considerations .............................33
11.2. Wire Protocol Considerations .............................33
12. References ....................................................34
12.1. Normative References .....................................34
12.2. Informative References ...................................34
Appendix A. Design Considerations and Examples ....................35
A.1. Design Considerations for a Datamover Protocol ............35
A.2. Examples of Datamover Interactions ........................35
Acknowledgements ..................................................44
Table of Figures
Figure 1. Datamover Architecture Diagram, with the RDMAP Example ...8
Figure 2. A Successful iSCSI Login on Initiator ...................37
Figure 3. A Successful iSCSI Login on Target ......................37
Figure 4. A Failed iSCSI Login on Initiator .......................38
Figure 5. A Failed iSCSI Login on Target ..........................38
Figure 6. iSCSI Does Not Enable the Datamover .....................39
Figure 7. A Normal iSCSI Connection Termination ...................40
Figure 8. An Abnormal iSCSI Connection Termination ................40
Figure 9. A SCSI Write Data Transfer ..............................41
Figure 10. A SCSI Read Data Transfer ..............................42
Figure 11. A SCSI Read Data Acknowledgement .......................43
Figure 12. Task Resource Cleanup on Abort .........................44
Chadalapaka, et al. Informational [Page 3]
RFC 5047 DA October 2007
1. Motivation
1.1. Intent
There are relatively new standard protocols that enable Remote Direct
Memory Access (RDMA) and Remote Direct Data Placement (RDDP)
technologies to work over IP fabrics. The principal value
proposition of these technologies is that they enable one end node to
place data in the final intended buffer on the remote end node, thus
eliminating the need for a receive path data copy that moves the data
to its final location. The data copy avoidance in turn eliminates
unnecessary memory bandwidth consumption, substantially decreases the
reassembly buffer size requirements, and preserves CPU cycles that
would otherwise be spent in copying.
The iSCSI specification [RFC3720] defines a very detailed data
transfer model that employs SCSI Data-In PDUs, SCSI Data-Out PDUs,
and R2T PDUs, in addition to the SCSI Command and SCSI Response PDUs
that respectively create and conclude the task context for the data
transfer. In the traditional iSCSI model, the iSCSI protocol layer
plays the central role in pacing the data transfer and carrying out
the ensuing data transfer itself. An alternative architecture would
be for iSCSI to delegate a large part of this data transfer role to a
separate protocol layer exclusively designed to move data, which in
turn is possibly aided by a data movement and placement technology
such as RDMA.
If iSCSI were operating in such RDMA environments, iSCSI would be
shielded from the low-level data transfer mechanics but would only be
privy to the conclusion of the requested data transfer. Thus, there
would be an effective "off-loading" of the work that an iSCSI
protocol layer is expected to perform, compared to today's iSCSI end
nodes. For such RDMA environments, it is highly desirable that there
be a standard architecture to separate the data movement part of the
iSCSI protocol definition from the rest of the iSCSI functionality.
This architecture precisely defines what a Datamover layer is and
also describes the model of interactions between the iSCSI layer and
the Datamover layer (Section 6). In order to satisfy this need, this
document presents a Datamover Architecture for iSCSI (DA) and
summarizes a reasonable model for interactions between the iSCSI
layer and the Datamover layer for each of the iSCSI PDUs that are
defined in [RFC3720]. Note that while DA is motivated by the advent
of RDMA over TCP/IP technology, the architecture is not dependent on
RDMA in its design. DA is intended to be a generic architectural
framework for allowing different types of Datamovers based on
different types of RDMA and transport protocols. Adoption of this
model will help iSCSI proliferate into more environments.
Chadalapaka, et al. Informational [Page 4]
RFC 5047 DA October 2007
1.2. Interpretation of Requirements
This document introduces certain architectural abstractions and
builds an abstract functional interface model between iSCSI and
Datamover protocol layers based on those abstractions. This
architectural style is motivated by the following desires:
a) Provide guidance to Datamover protocol designers with respect
to the functional boundary between iSCSI and the Datamover
protocols. This guidance is critical since a significant part
of the [RFC3720] protocol definition is left unchanged by DA
architecture and the iSCSI notions from [RFC3720] (e.g., tasks,
ITTs) are leveraged by the Datamover protocol.
b) Aid existing iSCSI implementations to rapidly adapt to DA
architecture, largely by leveraging the architectural
abstractions into implementation constructs -- e.g., functions,
APIs, modules.
However, note that DA architecture does not intend to impose any
implementation specifics per se. When a DA architectural concept
(e.g., Operational Primitive) is described as mandatory ("MUST") or
recommended ("SHOULD") of a layer (iSCSI or Datamover) in this
document, the intent is that an implementation respectively MUST or
SHOULD produce the same protocol action as what the model describes.
Specifically, no implementation compliance in terms of names, modules
or API arguments etc. is implied by this Architecture by such use of
[RFC2119] terms, only a functional compliance is sought.
2. Definitions and Acronyms
2.1. Definitions
I/O Buffer - A buffer that is used in a SCSI Read or Write operation
so that SCSI data may be sent from or received by the buffer.
Datamover protocol - A Datamover protocol is a data transfer wire
protocol for iSCSI that meets the requirements stated in Section
6.
Datamover layer - A Datamover layer is a protocol layer within an end
node that implements the Datamover protocol.
Datamover-assisted - An iSCSI connection is said to be "Datamover-
assisted" when a Datamover layer is enabled for moving control and
data information on that iSCSI connection.
Chadalapaka, et al. Informational [Page 5]
RFC 5047 DA October 2007
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this
document are to be interpreted as described in [RFC2119].
2.2. Acronyms
Acronym Definition
-------------------------------------------------------------
DA Datamover Architecture for iSCSI
DDP Direct Data Placement Protocol
DI Datamover Interface
IANA Internet Assigned Numbers Authority
IETF Internet Engineering Task Force
I/O Input - Output
IP Internet Protocol
iSCSI Internet SCSI
iSER iSCSI Extensions for RDMA
ITT Initiator Task Tag
LO Leading Only
MPA Marker PDU Aligned Framing for TCP
PDU Protocol Data Unit
RDDP Remote Direct Data Placement
RDMA Remote Direct Memory Access
R2T Ready To Transfer
R2TSN Ready To Transfer Sequence Number
RDMA Remote Direct Memory Access
RDMAP Remote Direct Memory Access Protocol
RFC Request For Comments
Chadalapaka, et al. Informational [Page 6]
RFC 5047 DA October 2007
SAM SCSI Architecture Model
SCSI Small Computer Systems Interface
SN Sequence Number
SNACK Selective Negative Acknowledgment - also
Sequence Number Acknowledgement for Data
TCP Transmission Control Protocol
TTT Target Transfer Tag
3. Architectural Layering of iSCSI and Datamover Layers
Figure 1 illustrates an example of the architectural layering of
iSCSI and Datamover layers, in conjunction with a TCP/IP
implementation of RDMAP/DDP ([DDP]) layers in an iSCSI end node.
Note that RDMAP/DDP/MPA and TCP protocol layers are shown here only
as an example, and in reality, DA is completely oblivious to protocol
layers below the Datamover layer. The RDMAP/DDP/MPA protocol stack
provides a generic transport service with direct data placement.
There is no need to tailor the implementation of this protocol stack
to the specific ULP to benefit from these services.
Chadalapaka, et al. Informational [Page 7]
RFC 5047 DA October 2007
Initiator stack Target stack
+----------------+ SCSI application +----------------+
| SCSI Layer | protocols | SCSI Layer |
+----------------+ +----------------+
^ ^
| |
v v
+----------------+ iSCSI protocol +----------------+
| iSCSI Layer | (excluding data | iSCSI Layer |
+----------------+ movement) +----------------+
^ ^
-- ---+-- ---- DI (Datamover Interface)--- ----+--- ----
v v
+----------------+ a Datamover +----------------+
| Datamover Layer| protocol | Datamover Layer|
+----------------+ +----------------+
^ ^
+-------+----------+ +---------+-----------+
| v | | v |
|+---------------+ | | +-----------------+ |
|| RDMAP/DDP/MPA | | RDMAP/DDP/MPA | | RDMAP/DDP/MPA | |
|| Layers | | protocols | | Layers | |
|+---------------+ | | +-----------------+ |
| ^ | | ^ |
| | network | | | network |
| | transport| | | transport |
| v | | v |
|+---------------+ | | +----------------+ |
|| TCP Layer | | TCP protocol | | TCP Layer | |
|+---------------+ | | +----------------+ |
| ^ | | ^ |
+-------+----------+ +---------+-----------+
+------------------------------------------+
Figure 1. Datamover Architecture Diagram,
with the RDMAP Example
The scope of this document is limited to:
1. Defining the notion of a Datamover layer and a Datamover
protocol (Section 6).
2. Defining the functionality distribution between the iSCSI layer
and the Datamover layer, along with the communication model
between the two (Operational Primitives).
Chadalapaka, et al. Informational [Page 8]
RFC 5047 DA October 2007
3. Modeling the interactions between the blocks labeled as "iSCSI
Layer" and "Datamover Layer" in Figure 1 -- i.e., defining the
interface labeled "DI" in the figure -- for each defined iSCSI
PDU, based on the Operational Primitives.
4. Design Overview
This document discusses and defines a model for interactions between
the iSCSI layer and a "Datamover layer" (see Section 6) operating
within an iSCSI end node, presumably communicating with one or more
iSCSI end nodes with similar layering. The model for interactions
for handling different iSCSI operations is called the "Datamover
Interface" (DI, Section 10), while the architecture itself is called
the "Datamover Architecture for iSCSI" (DA). It is likely that the
architecture will have implications on the Datamover wire protocols
as DA places certain requirements and functionality expectations on
the Datamover layer. However, this document itself neither defines
any new wire protocol for the Datamover layer, nor any potential
modifications to the iSCSI wire protocol to employ the Datamover
layer. The scope of this document is strictly limited to specifying
the architectural framework and the minimally required interactions
that happen within an iSCSI end node to leverage the Datamover layer.
The design ideas behind DA can be summarized as follows:
1) DA defines an abstract functional interface model of the iSCSI
layer's interactions with a Datamover layer below -- i.e., DA
models the interactions between the logical "bottom" interface
of iSCSI and the logical "top" interface of a Datamover.
2) DA guides the wire protocol for a Datamover layer by defining
the iSCSI knowledge that the Datamover layer may utilize in its
protocol definition (as an example, this document completely
limits the notion of "iSCSI session" to the iSCSI layer).
3) DA is designed to allow implementation of the Datamover layer
either in hardware or in software.
4) DA is not a wire protocol spec, but an architecture that also
models the interactions between iSCSI and Datamover layers
operating within an iSCSI end node.
5) DA by design seeks to model the iSCSI-Datamover interactions in
a way that the modeling is independent of the specifics of
either a particular iSCSI revision or an instantiation of a
Datamover layer.
Chadalapaka, et al. Informational [Page 9]
RFC 5047 DA October 2007
6) DA introduces and relies on the notion of a defined set of
Operational Primitives (could be seen as entry point
definitions in implementation terms) provided by each layer to
the other to carry out the request-response interactions.
7) DA is intended to allow Datamover protocol definitions with
minimal changes to existing iSCSI implementations.
8) DA is designed to allow the iSCSI layer to completely rely on
the Datamover layer for all data transport needs.
9) DA models the architecturally required minimal interactions
between an operational iSCSI layer and a Datamover layer to
realize the iSCSI-transparent data movement. There may be
several other interactions in a typical implementation in order
to bootstrap a Datamover layer (or an iSCSI layer) into
operation, but they are outside the scope of this document.
Note that in summary, DA is architected to support many different
Datamover protocols operating under the iSCSI layer. One such
example of a Datamover protocol is iSER [iSER].
5. Architectural Concepts
5.1. iSCSI PDU Types
This section defines the iSCSI PDU classification terminology, as
defined and used in this document. Out of the set of legal iSCSI
PDUs defined in [RFC3720], as we will see in Section 5.1.1, the iSCSI
layer does not request a SCSI Data-Out PDU carrying solicited data
for transmission across the Datamover Interface per this
architecture. For this reason, the SCSI Data-Out PDU carrying
solicited data is excluded in the iSCSI PDU classification we
introduce in this section (for SCSI Data-Out PDUs for unsolicited
Data, see Section 5.1.2). The rest of the legal iSCSI PDUs that may
be exchanged across the Datamover Interface are defined to consist of
two classes:
1) iSCSI data-type PDUs
2) iSCSI control-type PDUs
5.1.1. iSCSI Data-Type PDUs
An iSCSI data-type PDU is defined as an iSCSI PDU that causes data
transfer, transparent to the remote iSCSI layer, to take place
between the peer iSCSI nodes on a Full Feature Phase iSCSI
connection. A data-type PDU, when requested for transmission by the
Chadalapaka, et al. Informational [Page 10]
RFC 5047 DA October 2007
sender iSCSI layer, results in the associated data transfer without
the participation of the remote iSCSI layer, i.e., the PDU itself is
not delivered as-is to the remote iSCSI layer. The following iSCSI
PDUs constitute the set of iSCSI data-type PDUs:
1) SCSI Data-In PDU
2) R2T PDU
In an iSCSI end node structured as an iSCSI layer and a Datamover
layer as defined in this document, the solicitation for Data-Out
(i.e., R2T PDU) is not delivered to the initiator iSCSI layer, per
the definition of an iSCSI data-type PDU. The data transfer is
instead performed via the mechanisms known to the Datamover layer
(e.g., RDMA Read). This in turn implies that a SCSI Data-Out PDU for
solicited data is never requested for transmission across the
Datamover Interface at the initiator.
5.1.2. iSCSI Control-Type PDUs
Any iSCSI PDU that is not an iSCSI data-type PDU and also not a
solicited SCSI Data-Out PDU is defined as an iSCSI control-type PDU.
Specifically, note that SCSI Data-Out PDUs for unsolicited Data are
defined as iSCSI control-type PDUs.
5.2. Data_Descriptor
A Data_Descriptor is an information element that describes an
iSCSI/SCSI data buffer, provided by the iSCSI layer to its local
Datamover layer or provided by the Datamover layer to its local iSCSI
layer for identifying the data associated respectively with the
requested or completed operation.
In implementation terms, a Data_Descriptor may be a scatter-gather
list describing a local buffer, the exact structure of which is
subject to the constraints imposed by the operating environment on
the local iSCSI node.
5.3. Connection_Handle
A Connection_Handle is an information element that identifies the
particular iSCSI connection for which an inbound or outbound iSCSI
PDU is intended. A connection handle is unique for a given pair of
an iSCSI layer instance and a Datamover layer instance. The
Connection_Handle qualifier is used in all invocations of any
Operational Primitive for connection identification.
Chadalapaka, et al. Informational [Page 11]
RFC 5047 DA October 2007
Note that the Connection_Handle is conceptually different from the
Connection Identifier (CID) defined by the iSCSI specification.
While the CID is a unique identifier of an iSCSI connection within an
iSCSI session, the uniqueness of the Connection_Handle extends to the
entire iSCSI layer instance coupled with the Datamover layer
instance, across possibly multiple iSCSI sessions.
In implementation terms, a Connection_Handle could be an opaque
identifier exchanged between the iSCSI layer and the Datamover layer
at the connection login time. One may also consider it to be similar
in scope of uniqueness to a socket identifier. The exact structure
and modalities of exchange of a Connection_Handle between the two
layers is implementation-specific.
5.4. Operational Primitive
An Operational Primitive, in this document, is an abstract functional
interface procedure that requests another layer perform a specific
action on the requestor's behalf or notifies the other layer of some
event. The Datamover Interface between an iSCSI layer instance and a
Datamover layer instance within an iSCSI end node uses a set of
Operational Primitives to define the functional interface between the
two layers. Note that not every invocation of an Operational
Primitive may elicit a response from the requested layer. This
document describes the types of Operational Primitives that are
implicitly required and provided by the iSCSI protocol layer as
defined in [RFC3720], and the semantics of these Primitives.
Note that ownership of buffers and data structures is likely to be
exchanged between the iSCSI layer and its local Datamover layer in
invoking the Operational Primitives defined in this architecture.
The buffer management details, including how buffers are allocated
and released, are implementation-specific and thus are outside the
scope of this document.
Each Operational Primitive invocation needs a certain "information
context" (e.g., Connection_Handle) for performing the specific action
being requested. The required information context is described in
this document by a listing of "qualifiers" on each invocation, in the
style of function call arguments. There is no specific
implementation implied in this notation. The "qualifiers" of any
Operational Primitive invocation specified in this document thus
represent the mandatory information context that the Operational
Primitive invocation MUST consider in performing the action. While
the qualifiers are required, the method of realizing the qualifiers
(passed synchronously with invocation, or retrieved from task
context, or retrieved from shared memory etc.) is really up to the
implementations.
Chadalapaka, et al. Informational [Page 12]
RFC 5047 DA October 2007
When an Operational Primitive implementation is described as
mandatory ("MUST") or recommended ("SHOULD") of a layer (iSCSI or
Datamover) in this document, the intent is that an implementation
respectively MUST or SHOULD produce the same protocol action as what
the model describes.
5.5. Transport Connection
The term "Transport Connection" is used in this document as a generic
term to represent the end-to-end logical connection as defined by the
underlying reliable transport protocol. For this document, all
instances of Transport Connection refer to a TCP connection.
6. Datamover Layer and Datamover Protocol
This section introduces the notion of a "Datamover layer" and
"Datamover protocol" as meant in this document, and defines the
requirements on a Datamover protocol.
A Datamover layer is the implementation component that realizes a
Datamover protocol functionality in an iSCSI-capable end node in
communicating with other iSCSI end nodes with similar capabilities.
More specifically, a "Datamover layer" MUST provide the following
functionality and the "Datamover protocol" MUST consist of the wire
protocol required to realize the following functionality:
1) guarantee that all the necessary data transfers take place when
the local iSCSI layer requests transmitting a command (in order
to complete a SCSI command, for an initiator), or
sending/receiving an iSCSI data sequence (in order to complete
part of a SCSI command for a target).
2) transport an iSCSI control-type PDU as-is to the peer Datamover
layer when requested to do so by the local iSCSI layer.
3) provide notification and delivery to the iSCSI layer upon
arrival of an iSCSI control-type PDU.
4) provide an initiator-to-target data acknowledgement of SCSI
read data back to the target iSCSI layer, when requested.
5) provide an asynchronous notification upon completion of a
requested data transfer operation that moved data without
involving the iSCSI layer.
6) place the SCSI data into the I/O buffers or pick up the SCSI
data for transmission out of the data buffers that the iSCSI
layer had requested to be used for a SCSI I/O.
Chadalapaka, et al. Informational [Page 13]
RFC 5047 DA October 2007
7) provide an error-free (i.e., must have at least the same level
of assurance of data integrity as the CRC32C iSCSI data
digest), reliable, in-order delivery transport mechanism over
IP networks in performing the data transfer, and asynchronously
notify the iSCSI layer upon iSCSI connection termination.
Note that this architecture expects that each compliant Datamover
protocol will define the precise means of satisfying the requirements
specified in this section.
In order to meet the functional requirements listed in this section,
certain Datamover protocols may require pre-posted buffers from the
local iSCSI protocol layer via mechanisms outside the scope of this
document. In some implementations, the absence of such buffers may
result in a connection failure. Datamover protocols may also realize
these functional requirements via methods not explicitly listed in
this document.
7. Functional Overview
This section presents an overview of the functional interactions
between the iSCSI layer and the Datamover layer as intended by this
Architecture.
7.1. Startup
The iSCSI Login Phase on an iSCSI connection occurs as defined in
[RFC3720]. The Architecture assumes that at the end of the Login
Phase, both the initiator and target, if they had so decided,
transition the connection to being Datamover-assisted. The precise
means of how an iSCSI initiator and an iSCSI target agree on having
the connection Datamover-assisted is defined by the Datamover
protocol. The only architectural requirement is that all iSCSI
interactions in the iSCSI Full Feature Phase MUST be Datamover-
assisted subject to the prior agreement, meaning that the Datamover
protocol is in the iSCSI-to-iSCSI communication path below the iSCSI
layer on either side as shown in Figure 1. DA defines the
Enable_Datamover Operational Primitive (Section 8.6) to bring about
this transition to a Datamover-assisted connection.
The Architecture also assumes that the Datamover layer may require a
certain number of opaque local resources for making a connection
Datamover-assisted. DA thus defines the
Allocate_Connection_Resources Operational Primitive (Section 8.4) to
model this interaction. This Primitive is intended to be invoked on
each side once the two sides decide (as previously noted) to have the
connection be Datamover-assisted. The expected sequence of Primitive
invocations is depicted in Figures 2 and 3 in Section 13.2. Figures
Chadalapaka, et al. Informational [Page 14]
RFC 5047 DA October 2007
4, 5, and 6 illustrate how the Primitives may be employed to deal
with various legal login outcomes.
7.2. Full Feature Phase
All iSCSI peer communication in the Full Feature Phase happens
through the Datamover layers if the iSCSI connection is Datamover-
assisted. The Architecture assumes that a Datamover layer may
require a certain number of opaque local resources for each new iSCSI
task. In the normal course of execution, these task-level resources
in the Datamover layer are assumed to be transparently allocated on
each task initiation and deallocated on the conclusion of each task
as appropriate. In exception scenarios however -- scenarios that do
not yield a SCSI Response for each task such as ABORT TASK operation
-- the Architecture assumes that the Datamover layer needs to be
notified of the individual task terminations to aid its task-level
resource management. DA thus defines the Deallocate_Task_Resources
Operational Primitive (Section 8.9) to model this task-resource
management. In specifying the ITT qualifier for the
Deallocate_Task_Resources Primitive, the Architecture further assumes
that the Datamover layer tracks its opaque task-level local resources
by the iSCSI ITT. DA also defines Send_Control (Section 8.1),
Put_Data (Section 8.2), Get_Data (Section 8.3),
Data_Completion_Notify (Section 9.3), Data_ACK_Notify (Section 9.4),
and Control_Notify (Section 9.1) Operational Primitives to model the
various Full Feature Phase interactions.
Figures 9, 10, and 11 in Section 13.2 show some Full Feature Phase
interactions -- SCSI Write task, SCSI Read task, and a SCSI Read Data
acknowledgement, respectively. Figure 12 in Section 13.2 illustrates
how an ABORT TASK operation can be modeled leading to deterministic
resource cleanup on the Datamover layer.
7.3. Wrap-up
Once an iSCSI connection becomes Datamover-assisted, the connection
continues in that state until the end of the Full Feature Phase,
i.e., the termination of the connection. The Architecture assumes
that when a connection is normally logged out, the Datamover layer
needs to be notified so that its connection-level opaque resources
(see Section 7.1) may be freed up. DA thus defines a
Connection_Terminate Operational Primitive (Section 8.7) to model
this interaction. The Architecture further assumes that when a
connection termination happens without iSCSI layer's involvement
(e.g., TCP RST), the Datamover layer is capable of locally cleaning
up its task-level and connection-level resources before notifying the
iSCSI layer of the fact. DA thus defines the
Chadalapaka, et al. Informational [Page 15]
RFC 5047 DA October 2007
Connection_Terminate_Notify Operational Primitive (Section 9.2) to
model this interaction.
Figures 7 and 8 in Section 13.2 illustrate the interactions between
the iSCSI and Datamover layers in normal and unexpected connection
termination scenarios.
8. Operational Primitives Provided by the Datamover Layer
While the iSCSI specification itself does not have a notion of
Operational Primitives, any iSCSI layer implementing the iSCSI
specification functionally requires the following Operational
Primitives from its Datamover layer. Thus, any Datamover protocol
compliant with this architecture MUST implement the Operational
Primitives described in this section. These Operational Primitives
are invoked by the iSCSI layer as appropriate. Unless otherwise
stated, all the following Operational Primitives may be used both on
the initiator side and the target side. In general programming
terminology, this set of Operational Primitives may be construed as
"down calls".
1) Send_Control
2) Put_Data
3) Get_Data
4) Allocate_Connection_Resources
5) Deallocate_Connection_Resources
6) Enable_Datamover
7) Connection_Terminate
8) Notice_Key_Values
9) Deallocate_Task_Resources
8.1. Send_Control
Input qualifiers: Connection_Handle, iSCSI PDU-specific qualifiers
Return Results: Not specified.
An iSCSI layer requests that its local Datamover layer transmit an
iSCSI control-type PDU to the peer iSCSI layer operating in the
remote iSCSI node by this Operational Primitive. The Datamover layer
Chadalapaka, et al. Informational [Page 16]
RFC 5047 DA October 2007
performs the requested operation, and may add its own protocol
headers in doing so. The iSCSI layer MUST NOT invoke the
Send_Control Operational Primitive on an iSCSI connection that is not
yet Datamover-assisted.
An initiator iSCSI layer requesting the transfer of a SCSI Command
PDU or a target iSCSI layer requesting the transfer of a SCSI
response PDU are examples of invoking the Send_Control Operational
Primitive. As Section 10.3.1 illustrates later on, the iSCSI PDU-
specific qualifiers in this example are: BHS and AHS,
DataDescriptorOut, DataDescriptorIn, ImmediateDataSize, and
UnsolicitedDataSize.
8.2. Put_Data
Input qualifiers: Connection_Handle, contents of a SCSI Data-In PDU
header, Data_Descriptor, Notify_Enable
Return Results: Not specified.
An iSCSI layer requests that its local Datamover layer transmit the
data identified by the Data_Descriptor for the SCSI Data-In PDU to
the peer iSCSI layer on the remote iSCSI node by this Operational
Primitive. The Datamover layer performs the operation by using its
own protocol means, completely transparent to the remote iSCSI layer.
The iSCSI layer MUST NOT invoke the Put_Data Operational Primitive on
an iSCSI connection that is not yet Datamover-assisted.
The Notify_Enable qualifier is used to request the local Datamover
layer to generate or not generate the eventual local completion
notification to the iSCSI layer for this Put_Data invocation. For
detailed semantics of this qualifier, see Section 9.3.
A Put_Data Primitive may only be invoked by an iSCSI layer on the
target to its local Datamover layer.
A target iSCSI layer requesting the transfer of an iSCSI read data
sequence (also known as a read burst) is an example of invoking the
Put_Data Operational Primitive.
8.3. Get_Data
Input qualifiers: Connection_Handle, contents of an R2T PDU,
Data_Descriptor, Notify_Enable
Return Results: Not specified.
Chadalapaka, et al. Informational [Page 17]
RFC 5047 DA October 2007
An iSCSI layer requests that its local Datamover layer retrieve
certain data identified by the R2T PDU from the peer iSCSI layer on
the remote iSCSI node and place it into the buffer identified by the
Data_Descriptor by invoking this Operational Primitive. The
Datamover layer performs the operation by using its own protocol
means, completely transparent to the remote iSCSI layer. The iSCSI
layer MUST NOT invoke the Get_Data Operational Primitive on an iSCSI
connection that is not yet Datamover-assisted.
The Notify_Enable qualifier is used to request that the local
Datamover layer generate or not generate the eventual local
completion notification to the iSCSI layer for this Get_Data
invocation. For detailed semantics of this qualifier, see Section
9.3.
A Get_Data Primitive may only be invoked by an iSCSI layer on the
target to its local Datamover layer.
A target iSCSI layer requesting the transfer of an iSCSI write data
sequence (also known as a write burst) is an example of invoking the
Get_Data Operational Primitive.
8.4. Allocate_Connection_Resources
Input qualifiers: Connection_Handle[, Resource_Descriptor ]
Return Results: Status.
By invoking this Operational Primitive, an iSCSI layer requests that
its local Datamover layer perform all the Datamover-specific resource
allocations required for the Full Feature Phase of an iSCSI
connection. The Connection_Handle identifies the connection for
which the iSCSI layer is requesting resources to be allocated.
Allocation of these resources is a step towards eventually
transitioning the connection to become a Datamover-assisted iSCSI
connection. Note that the Datamover layer however does not allocate
any Datamover-specific task-level resources upon invocation of this
Primitive.
An iSCSI layer, in addition, optionally specifies the
implementation-specific resource requirements for the iSCSI
connection to the Datamover layer by passing an input qualifier
called Resource_Descriptor. The exact structure of a
Resource_Descriptor is implementation-dependent, and hence
structurally opaque to DA.
A return result of Status=success means that the
Allocate_Connection_Resources invocation corresponding to that
Chadalapaka, et al. Informational [Page 18]
RFC 5047 DA October 2007
Connection_Handle succeeded. If an Allocate_Connection_Resources
invocation is made for a Connection_Handle for which an earlier
invocation succeeded, the return Status must be success and the
request will be ignored by the Datamover layer. A return result of
Status=failure means that the Allocate_Connection_Resources
invocation corresponding to that Connection_Handle failed. There
MUST NOT be more than one Allocate_Connection_Resources Primitive
invocation outstanding for a given Connection_Handle at any time.
The iSCSI layer must invoke the Allocate_Connection_Resources
Primitive before the invocation of the Enable_Datamover Primitive.
8.5. Deallocate_Connection_Resources
Input qualifiers: Connection_Handle
Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer requests that
its local Datamover layer deallocate all the Datamover-specific
resources that may have been allocated earlier for the Transport
Connection identified by the Connection_Handle. The iSCSI layer may
invoke this Operational Primitive when the Datamover-specific
resources associated with the Connection_Handle are no longer
necessary (such as the Login failure of the corresponding iSCSI
connection).
8.6. Enable_Datamover
Input qualifiers: Connection_Handle, Transport_Connection_Descriptor
[, Final_Login_Response_PDU]
Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer requests that
its local Datamover layer assist all further iSCSI exchanges on the
iSCSI connection (i.e., to make the connection Datamover-assisted)
identified by the Connection_Handle, for which the Datamover-specific
resource allocation was earlier made. The iSCSI layer MUST NOT
invoke the Enable_Datamover Operational Primitive for an iSCSI
connection unless there is a corresponding prior resource allocation.
The Final_Login_Response_PDU input qualifier is applicable only for a
target, and contains the final Login Response that concludes the
iSCSI Login Phase and which must be sent as a byte stream as expected
by the initiator iSCSI layer. When this qualifier is used, the
target-Datamover layer MUST transmit this final Login Response before
Datamover assistance is enabled for the Transport Connection.
Chadalapaka, et al. Informational [Page 19]
RFC 5047 DA October 2007
The iSCSI layer identifies the specific Transport Connection
associated with the Connection_Handle to the Datamover layer by
specifying the Transport_Connection_Descriptor. The exact structure
of this Descriptor is implementation-dependent.
8.7. Connection_Terminate
Input qualifiers: Connection_Handle
Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer requests that
its local Datamover layer terminate the Transport Connection and
deallocate all the connection and task resources associated with the
Connection_Handle. When this Operational Primitive invocation
returns to the iSCSI layer, the iSCSI layer may assume the full
ownership of all the iSCSI-level resources, e.g., I/O Buffers,
associated with the connection. This Operational Primitive may be
invoked only with a valid Connection_Handle, and the Transport
Connection associated with the Connection_Handle must already be
Datamover-assisted.
8.8. Notice_Key_Values
Input qualifiers: Connection_Handle, Number of keys, a list of Key-
Value pairs.
Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer requests that
its local Datamover layer take note of the negotiated values of the
listed keys for the Transport Connection. This Operational Primitive
may be invoked only with a valid Connection_Handle, and the Key-Value
pairs MUST be the current values that were successfully agreed upon
by the iSCSI peers for the connection. The Datamover layer may use
the values of the keys to aid the Datamover operation as it deems
appropriate. The specific keys to be passed as input qualifiers and
the point(s) in time this Operational Primitive is invoked are
implementation-dependent.
8.9. Deallocate_Task_Resources
Input qualifiers: Connection_Handle, ITT
Return Results: Not specified.
By invoking this Operational Primitive, an iSCSI layer requests that
its local Datamover layer deallocate all Datamover-specific resources
Chadalapaka, et al. Informational [Page 20]
RFC 5047 DA October 2007
that earlier may have been allocated for the task identified by the
ITT qualifier. The iSCSI layer uses this Operational Primitive
during exception processing when one or more active tasks are to be
terminated without corresponding SCSI Response PDUs. This Primitive
MUST be invoked for each active task terminated without a SCSI
Response PDU. This Primitive MUST NOT be invoked by the iSCSI layer
when a SCSI Response PDU normally concludes a task. When a SCSI
Response PDU normally concludes a task (even if the SCSI Status was
not a success), the Datamover layer is assumed to have automatically
deallocated all Datamover-specific task resources for that task.
Refer to Section 7.2 for a related discussion on the Architectural
assumptions on the task-level Datamover resource management,
especially with respect to when the resources are assumed to be
allocated.
9. Operational Primitives Provided by the iSCSI Layer
While the iSCSI specification itself does not have a notion of
Operational Primitives, any iSCSI layer implementing the iSCSI
specification would have to provide the following Operational
Primitives to its local Datamover layer. Thus, any iSCSI protocol
implementation compliant with this architecture MUST implement the
Operational Primitives described in this section. These Operational
Primitives are invoked by the Datamover layer as appropriate and when
the iSCSI connection is Datamover-assisted. Unless otherwise stated,
all the following Operational Primitives may be used both on the
initiator side and the target side. In general programming
terminology, this set of Operational Primitives may be construed as
"up calls".
1) Control_Notify
2) Connection_Terminate_Notify
3) Data_Completion_Notify
4) Data_ACK_Notify
9.1. Control_Notify
Input qualifiers: Connection_Handle, an iSCSI control-type PDU.
Return Results: Not specified.
A Datamover layer notifies its local iSCSI layer, via this
Operational Primitive, of the arrival of an iSCSI control-type PDU
from the peer Datamover layer on the remote iSCSI node. The iSCSI
layer processes the control-type PDU as defined in [RFC3720].
Chadalapaka, et al. Informational [Page 21]
RFC 5047 DA October 2007
A target iSCSI layer being notified of the arrival of a SCSI command
is an example of invoking the Control_Notify Operational Primitive.
Note that implementations may choose to describe the "iSCSI control-
type PDU" qualifier in this notification using a Data_Descriptor
(Section 5.2) and not necessarily one contiguous buffer.
9.2. Connection_Terminate_Notify
Input qualifiers: Connection_Handle
Return Results: Not specified.
A Datamover layer notifies its local iSCSI layer on an unsolicited
termination or failure of an iSCSI connection providing the
Connection_Handle associated with the iSCSI Connection. The iSCSI
layer MUST consider the Connection_Handle to be invalid upon being so
notified. The iSCSI layer processes the connection termination as
defined in [RFC3720]. The Datamover layer MUST deallocate the
connection and task resources associated with the terminated
connection before notifying the iSCSI layer of the termination via
this Operational Primitive.
A target iSCSI layer is notified of an ungraceful connection
termination by the Datamover layer when the underlying Transport
Connection is torn down. Such a Connection_Terminate_Notify
Operational Primitive may be triggered, for example, by a TCP RESET
in cases where the underlying Transport Connection uses TCP.
9.3. Data_Completion_Notify
Input qualifiers: Connection_Handle, ITT, SN
Return Results: Not specified.
A Datamover layer notifies its local iSCSI layer on completing the
retrieval of the data or upon sending the data, as requested in a
prior iSCSI data-type PDU, from/to the peer Datamover layer on the
remote iSCSI node via this Operational Primitive. The iSCSI layer
processes the operation as defined in [RFC3720].
SN may be either the DataSN associated with the SCSI Data-In PDU or
R2TSN associated with the R2T PDU depending on the SCSI operation.
Note that, for targets, a TTT (see [RFC3720]) could have been
specified instead of an SN. However, the considered choice was to
leave the SN to be the qualifier for two reasons -- a) it is generic
and applicable to initiators and targets as well as Data-In and
Data-Out, and b) having both SN and TTT qualifiers for the
Chadalapaka, et al. Informational [Page 22]
RFC 5047 DA October 2007
notification is considered onerous on the Datamover layer, in terms
of state maintenance for each completion notification. The
implication of this choice is that iSCSI target implementations will
have to adapt to using the ITT-SN tuple in associating the solicited
data to the appropriate task, rather than the ITT-TTT tuple for doing
the same.
If Notify_Enable is set in either a Put_Data or a Get_Data
invocation, the Datamover layer MUST invoke the
Data_Completion_Notify Operational Primitive upon completing that
requested data transfer. If the Notify_Enable was cleared in either
a Put_Data or a Get_Data invocation, the Datamover layer MUST NOT
invoke the Data_Completion_Notify Operational Primitive upon
completing that requested data transfer.
A Data_Completion_Notify invocation serves to notify the iSCSI layer
of the Put_Data or Get_Data completion, respectively. As earlier
noted in Sections 8.2 and 8.3, specific Datamover protocol
definitions may restrict the usage scope of Put_Data and Get_Data,
and thus implicitly the usage scope of Data_Completion_Notify.
A target iSCSI layer being notified of the retrieval of a write data
sequence is an example of invoking the Data_Completion_Notify
Operational Primitive.
9.4. Data_ACK_Notify
Input qualifiers: Connection_Handle, ITT, DataSN
Return Results: Not specified.
A target Datamover layer notifies its local iSCSI layer of the
arrival of a previously requested data acknowledgement from the peer
Datamover layer on the remote (initiator) iSCSI node via this
Operational Primitive. The iSCSI layer processes the data
acknowledgement notification as defined in [RFC3720].
A target iSCSI layer being notified of the arrival of a data
acknowledgement for a certain SCSI Read data PDU is the only example
of invoking the Data_ACK_Notify Operational Primitive.
10. Datamover Interface (DI)
10.1. Overview
This section describes the model of interactions between iSCSI and
Datamover layers when the iSCSI connection is Datamover-assisted so
the iSCSI layer may carry out the following:
Chadalapaka, et al. Informational [Page 23]
RFC 5047 DA October 2007
- send iSCSI data-type PDUs and exchange iSCSI control-type PDUs,
and
- handle asynchronous notifications such as completion of data
sequence transfer and connection failure.
This chapter relies on the notion of Operational Primitives (Section
5.4) to define DI.
10.2. Interactions for Handling Asynchronous Notifications
10.2.1. Connection Termination
As stated in Section 9.2, the Datamover layer notifies the iSCSI
layer of a failed or terminated connection via the
Connection_Terminate_Notify Operational Primitive. The iSCSI layer
MUST consider the connection unusable upon the invocation of this
Primitive and handle the connection termination as specified in
[RFC3720].
10.2.2. Data Transfer Completion
As stated in Section 9.3, the Datamover layer notifies the iSCSI
layer of a completed data transfer operation via the
Data_Completion_Notify Operational Primitive. The iSCSI layer
processes the transfer completion as specified in [RFC3720].
10.2.2.1. Completion of a Requested SCSI Data Transfer
To notify the iSCSI layer of the completion of a requested iSCSI
data-type PDU transfer, the Datamover layer uses the
Data_Completion_Notify Operational Primitive with the following input
qualifiers.
a) Connection_Handle.
b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
c) SN: DataSN for a SCSI Data-in/Data-out PDU, and R2TSN for an
iSCSI R2T PDU. The semantics for both types of sequence
numbers are as defined in [RFC3720].
The rationale for choosing SN is explained in Section 9.3.
Every invocation of the Data_Completion_Notify Operational Primitive
MUST be preceded by an invocation of the Put_Data or Get_Data
Operational Primitive with the Notify_Enable qualifier set by the
iSCSI layer at an earlier point in time.
Chadalapaka, et al. Informational [Page 24]
RFC 5047 DA October 2007
10.2.3. Data Acknowledgement
[RFC3720] allows the iSCSI targets to optionally solicit data
acknowledgement from the initiator for one or more Data-In PDUs, via
setting of the A-bit on a Data-In PDU. The Data_ACK_Notify
Operational Primitive with the following input qualifiers is used by
the target Datamover layer to notify the local iSCSI layer of the
arrival of data acknowledgement of a previously solicited iSCSI read
data acknowledgement. This Operational Primitive thus is applicable
only to iSCSI targets.
a) Connection_Handle.
b) ITT: Initiator Task Tag semantics as defined in [RFC3720].
c) DataSN: of the next SCSI Data-In PDU, which immediately follows
the SCSI Data-In PDU with the A-bit set to which this
notification corresponds, with semantics as defined in
[RFC3720].
Every invocation of the Data_ACK_Notify Operational Primitive MUST be
preceded by an invocation of the Put_Data Operational Primitive by
the iSCSI target layer with the A-bit set to 1 at an earlier point in
time.
10.3. Interactions for Sending an iSCSI PDU
This section discusses the model of interactions for sending each of
the iSCSI PDUs defined in [RFC3720]. A Connection_Handle (see
Section 5.3) is assumed to qualify each of these interactions so that
the Datamover layer can route it to the appropriate Transport
Connection. The qualifying Connection_Handle is not explicitly
listed in the subsequent sections.
Note that the defined list of input qualifiers represents the
semantically required set for the Datamover layer to consider in
implementing the Primitive in each interaction described in this
section (see Section 5.4 for an elaboration). Implementations may
choose to deduce the qualifiers in ways that are optimized for the
implementation specifics. Two examples of this are:
1. For SCSI command (Section 10.3.1), deducing the
ImmediateDataSize input qualifier from the DataSegmentLength
field of the SCSI Command PDU.
2. For SCSI Data-Out (Section 10.3.5.1), deducing the
DataDescriptorOut input qualifier from the associated SCSI
command invocation qualifiers (assuming such state is
Chadalapaka, et al. Informational [Page 25]
RFC 5047 DA October 2007
maintained) in conjunction with BHS fields of the SCSI Data-Out
PDU.
10.3.1. SCSI Command
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a SCSI Command
PDU.
a) BHS and AHS, if any, of the SCSI Command PDU as defined in
[RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
Out for the entire command, in the case of a write or
bidirectional command.
c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
for the entire command, in the case of a read or bidirectional
command.
d) ImmediateDataSize: that defines the number of octets of
immediate unsolicited data for a write/bidirectional command.
e) UnsolicitedDataSize: that defines the number of octets of
immediate and non-immediate unsolicited data for a
write/bidirectional command.
10.3.2. SCSI Response
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a SCSI Response
PDU.
a) BHS of the SCSI Response PDU as defined in [RFC3720].
b) DataDescriptorStatus: that defines the iSCSI buffer that
contains the sense and response information for the command.
10.3.3. Task Management Function Request
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Task
Management Function Request PDU.
a) BHS of the Task Management Function Request PDU as defined in
[RFC3720].
Chadalapaka, et al. Informational [Page 26]
RFC 5047 DA October 2007
b) DataDescriptorOut: that defines the I/O Buffer meant for Data-
Out for the entire command, in the case of a write or
bidirectional command. (Only valid if Function="TASK REASSIGN"
- [RFC3720].)
c) DataDescriptorIn: that defines the I/O Buffer meant for Data-In
for the entire command, in the case of a read or bidirectional
command. (Only valid if Function="TASK REASSIGN" - [RFC3720].)
10.3.4. Task Management Function Response
The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a Task
Management Function Response PDU.
a) BHS of the Task Management Function Response PDU as defined in
[RFC3720].
10.3.5. SCSI Data-Out and SCSI Data-In
10.3.5.1. SCSI Data-Out
The Send_Control Operational Primitive with the following input
qualifiers is used by the initiator iSCSI layer for requesting the
transmission of a SCSI Data-Out PDU carrying the non-immediate
unsolicited data.
a) BHS of the SCSI Data-Out PDU as defined in [RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer with the Data-
Out to be carried in the iSCSI data segment of the PDU.
10.3.5.2. SCSI Data-In
The Put_Data Operational Primitive with the following input
qualifiers is used by the target iSCSI layer for requesting the
transmission of the data carried by a SCSI Data-In PDU.
a) BHS of the SCSI Data-In PDU as defined in [RFC3720].
b) DataDescriptorIn: that defines the I/O Buffer with the Data-In
being requested for transmission.
Chadalapaka, et al. Informational [Page 27]
RFC 5047 DA October 2007
10.3.6. Ready To Transfer (R2T)
The Get_Data Operational Primitive with the following input
qualifiers is used by the target iSCSI layer for requesting the
retrieval of the data as specified by the semantic content of an R2T
PDU.
a) BHS of the Ready To Transfer PDU as defined in [RFC3720].
b) DataDescriptorOut: that defines the I/O Buffer for the Data-Out
being requested for retrieval.
10.3.7. Asynchronous Message
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of an Asynchronous
Message PDU.
a) BHS of the Asynchronous Message PDU as defined in [RFC3720].
b) DataDescriptorSense: that defines an iSCSI buffer that contains
the sense and iSCSI Event information.
10.3.8. Text Request
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Text Request
PDU.
a) BHS of the Text Request PDU as defined in [RFC3720].
b) DataDescriptorTextOut: that defines the iSCSI Text Request
buffer.
10.3.9. Text Response
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Text Response
PDU.
a) BHS of the Text Response PDU as defined in [RFC3720].
b) DataDescriptorTextIn: that defines the iSCSI Text Response
buffer.
Chadalapaka, et al. Informational [Page 28]
RFC 5047 DA October 2007
10.3.10. Login Request
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Login Request
PDU.
a) BHS of the Login Request PDU as defined in [RFC3720].
b) DataDescriptorLoginRequest: that defines the iSCSI Login
Request buffer.
Note that specific Datamover protocols may choose to disallow the
standard DA Primitives from being used for the iSCSI Login Phase.
When used in conjunction with such Datamover protocols, an attempt to
send a Login Request via the Send_Control Operational Primitive
invocation is clearly an error scenario, as the Login Request PDU is
being sent while the connection is in the iSCSI Full Feature Phase.
It is outside the scope of this document to specify the resulting
implementation behavior in this case -- [RFC3720] already defines the
error handling for this error scenario.
10.3.11. Login Response
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Login
Response PDU.
a) BHS of the Login Response PDU as defined in [RFC3720].
b) DataDescriptorLoginResponse: that defines the iSCSI Login
Response buffer.
Note that specific Datamover protocols may choose to disallow the
standard DA Primitives from being used for the iSCSI Login Phase.
When used in conjunction with such Datamover protocols, an attempt to
send a Login Response via the Send_Control Operational Primitive
invocation is clearly an error scenario, as the Login Response PDU is
being sent while in the iSCSI Full Feature Phase. It is outside the
scope of this document to specify the resulting implementation
behavior in this case -- [RFC3720] already defines the error handling
for this error scenario.
10.3.12. Logout Command
The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a Logout Command
PDU.
Chadalapaka, et al. Informational [Page 29]
RFC 5047 DA October 2007
a) BHS of the Logout Command PDU as defined in [RFC3720].
10.3.13. Logout Response
The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a Logout
Response PDU.
a) BHS of the Logout Response PDU as defined in [RFC3720].
10.3.14. SNACK Request
The Send_Control Operational Primitive with the following input
qualifier is used for requesting the transmission of a SNACK Request
PDU.
a) BHS of the SNACK Request PDU as defined in [RFC3720].
10.3.15. Reject
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a Reject PDU.
a) BHS of the Reject PDU as defined in [RFC3720].
b) DataDescriptorReject: that defines the iSCSI Reject buffer.
10.3.16. NOP-Out
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a NOP-Out PDU.
a) BHS of the NOP-Out PDU as defined in [RFC3720].
b) DataDescriptorNOPOut: that defines the iSCSI Ping data buffer.
10.3.17. NOP-In
The Send_Control Operational Primitive with the following input
qualifiers is used for requesting the transmission of a NOP-In PDU.
a) BHS of the NOP-In PDU as defined in [RFC3720].
b) DataDescriptorNOPIn: that defines the iSCSI Return Ping data
buffer.
Chadalapaka, et al. Informational [Page 30]
RFC 5047 DA October 2007
10.4. Interactions for Receiving an iSCSI PDU
The only PDUs that are received by an iSCSI layer operating on a
Datamover layer are the iSCSI control-type PDUs. The Datamover layer
delivers the iSCSI control-type PDUs as they arrive, qualifying each
with the Connection_Handle (see Section 5.3) that identifies the
iSCSI connection for which the PDU is meant. The subsequent
processing of the iSCSI control-type PDUs proceeds as defined in
[RFC3720].
10.4.1. General Control-Type PDU Notification
This sub-section describes the general mechanics applicable to
several control-type PDUs. The following sub-sections note
additional considerations for control-type PDUs that are not covered
in this sub-section.
The Control_Notify Operational Primitive is used to notify the iSCSI
layer of the arrival of the following iSCSI control-type PDUs: SCSI
Command, SCSI Response, Task Management Function Request, Task
Management Function Response, Asynchronous Message, Text Request,
Text Response, Logout Command, Logout Response, SNACK, Reject, NOP-
Out, NOP-In.
10.4.2. SCSI Data Transfer PDUs
10.4.2.1. SCSI Data-Out
The Control_Notify Operational Primitive is used to notify the iSCSI
layer of the arrival of a SCSI Data-Out PDU carrying the non-
immediate unsolicited data. Note however that the solicited SCSI
Data-Out arriving on the target does not cause a notification to the
iSCSI layer using the Control_Notify Primitive because the solicited
SCSI Data-Out was not sent by the initiator iSCSI layer as control-
type PDUs.
10.4.2.2. SCSI Data-In
The Datamover layer does not notify the iSCSI layer of the arrival of
the SCSI Data-in at the initiator, because SCSI Data-in is an iSCSI
data-type PDU (see section 5.1). The iSCSI layer at the initiator
however may infer the arrival of the SCSI Data-In when it receives a
subsequent notification of the SCSI Response PDU via a Control_Notify
invocation.
While this document does not contemplate the possibility of a Data-In
PDU being received at the initiator iSCSI layer, specific Datamover
protocols may define how to deal with an unexpected inbound SCSI
Chadalapaka, et al. Informational [Page 31]
RFC 5047 DA October 2007
Data-In PDU that may result in the initiator iSCSI layer receiving
the Data-In PDU. This document leaves the details of handling this
error scenario to the specific Datamover protocols, so each may
define the appropriate error handling specific to the Datamover
environment.
10.4.2.3. Ready To Transfer (R2T)
Because an R2T PDU is an iSCSI data-type PDU (see Section 5.1) that
is not delivered as-is to the initiator iSCSI layer, the Datamover
layer does not notify the iSCSI layer of the arrival of an R2T PDU.
When an iSCSI node sends an R2T PDU to its local Datamover layer, the
local and remote Datamover layers transparently bring about the data
transfer requested by the R2T PDU.
While this document does not contemplate the possibility of an R2T
PDU being received at the initiator iSCSI layer, specific Datamover
protocols may define how to deal with an unexpected inbound R2T PDU
that may result in the initiator iSCSI layer receiving the R2T PDU.
This document leaves the details of handling this error scenario to
the specific Datamover protocols, so each may define the appropriate
error handling specific to the Datamover environment.
10.4.3. Login Request
The Control_Notify Operational Primitive is used for notifying the
target iSCSI layer of the arrival of a Login Request PDU. Note that
specific Datamover protocols may choose to disallow the standard DA
Primitives from being used for the iSCSI Login Phase. When used in
conjunction with such Datamover protocols, the arrival of a Login
Request necessitating the Control_Notify Operational Primitive
invocation is clearly an error scenario, as the Login Request PDU is
arriving in the iSCSI Full Feature Phase. It is outside the scope of
this document to specify the resulting implementation behavior in
this case -- [RFC3720] already defines the error handling in this
error scenario.
10.4.4. Login Response
The Control_Notify Operational Primitive is used to notify the
initiator iSCSI layer of the arrival of a Login Response PDU. Note
that specific Datamover protocols may choose to disallow the standard
DA Primitives from being used for the iSCSI Login Phase. When used
in conjunction with such Datamover protocols, the arrival of a Login
Response necessitating the Control_Notify Operational Primitive
invocation is clearly an error scenario, as the Login Response PDU is
arriving in the iSCSI Full Feature Phase. It is outside the scope of
this document to specify the resulting implementation behavior in
Chadalapaka, et al. Informational [Page 32]
RFC 5047 DA October 2007
this case -- [RFC3720] already defines the error handling in this
error scenario.
11. Security Considerations
11.1. Architectural Considerations
DA enables compliant iSCSI implementations to realize a control and
data separation in the way they interact with their Datamover
protocols. Note however that this separation does not imply a
separation in transport mediums between control traffic and data
traffic -- the basic iSCSI architecture with respect to tasks and PDU
relationships to tasks remains unchanged. [RFC3720] defines several
MUST requirements on ordering relationships across control and data
for a given task besides a mandatory deterministic task allegiance
model -- DA does not change this basic architecture (DA has a
normative reference to [RFC3720]) for allow any additional
flexibility in compliance in this area. To summarize, sending bulk
data transfers (prompted by Put_Data and Get_Data Primitive
invocations) on a different transport medium would be as ill-advised
as sending just the Data-Out/Data-In PDUs on a different TCP
connection in RFC 3720-based iSCSI implementations. Consequently,
all the iSCSI-related security text in [RFC3723] is directly
applicable to a DA-enabled iSCSI implementation.
Another area with security implications is the Datamover connection
resource management model, which DA defines -- particularly the
Allocate_Connection_Resources Primitive. An inadvertent realization
of this model could leave an iSCSI implementation exposed to denial-
of-service attacks. As Figures 2 and 3 in Section 13.2 illustrate,
the most effective countermeasure to this potential attack consists
of performing the Datamover resource allocation when the iSCSI layer
is sufficiently far along in the iSCSI Login Phase that it is
reasonably certain that the peer side is not an attacker. In
particular, if the Login Phase includes a SecurityNegotiation stage,
an iSCSI end node MUST defer the Datamover connection resource
allocation (i.e., invoking the Allocate_Connection_Resources
Primitive) to the LoginOperationalNegotiation stage [RFC3720] so that
the resource allocation happens post-authentication. This
considerably minimizes the potential for a denial-of service attack.
11.2. Wire Protocol Considerations
In view of the fact that the DA architecture itself does not define
any new wire protocol or propose modifications to the existing
protocols, there are no additional wire protocol security
considerations in employing DA itself. However, a DA-compliant iSCSI
implementation MUST comply with all the iSCSI-related requirements
Chadalapaka, et al. Informational [Page 33]
RFC 5047 DA October 2007
stipulated in [RFC3723] and [RFC3720]. Note further that in
realizing DA, each Datamover protocol must define and elaborate as
appropriate on any additional security considerations resulting from
the use of that Datamover protocol.
All Datamover protocol designers are strongly recommended to refer to
[RDDPSEC] for the types of security issues to consider. While
[RDDPSEC] elaborates on the security considerations applicable to an
RDDP-based Datamover [iSER], the document is representative of the
type of analysis of resource exhaustion and the application of
countermeasures that need to be done for any Datamover protocol.
12. References
12.1. Normative References
[RFC3720] Satran, J., Meth, K., Sapuntzakis, C., Chadalapaka, M., and
E. Zeidner, "Internet Small Computer Systems Interface
(iSCSI)", RFC 3720, April 2004.
[RFC3723] Aboba, B., Tseng, J., Walker, J., Rangan, V., and F.
Travostino, "Securing Block Storage Protocols over IP", RFC
3723, April 2004.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, March 1997.
12.2. Informative References
[DDP] Shah, H., Pinkerton, J., Recio, R., and P. Culley, "Direct
Data Placement over Reliable Transports", RFC 5041, October
2007.
[iSER] Ko, M., Chadalapaka, M., Hufferd, J., Elzur, U., Shah, H.,
and P. Thaler, "Internet Small Computer System Interface
(iSCSI) Extensions for Remote Direct Memory Access (RDMA)",
RFC 5046, October 2007.
[RDDPSEC] Pinkerton, J. and E. Deleganes, "Direct Data Placement
Protocol (DDP) / Remote Direct Memory Access Protocol
(RDMAP) Security", RFC 5042, October 2007.
Chadalapaka, et al. Informational [Page 34]
RFC 5047 DA October 2007
Appendix A. Design Considerations and Examples
A.1. Design Considerations for a Datamover Protocol
This section discusses the specific considerations for RDMA-based and
RDDP-based Datamover protocols.
a) Note that the modeling of interactions for SCSI Data-Out
(Section 10.3.5.1) is only used for unsolicited data transfer.
b) The modeling of interactions for SNACK (Sections 10.3.14 and
10.4.1) is not expected to be used given that one of the design
requirements on the Datamover is that it "guarantees an error-
free, reliable, in-order transport mechanism" (Section 6). The
interactions for sending and receiving a SNACK are nevertheless
modeled in this document because the receiving iSCSI layer can
deterministically deal with an inadvertent SNACK. This also
shows the DA designers' intent that DI is not meant to filter
certain types of PDUs.
c) The onus is on a reliable Datamover (per requirements stated in
Section 6) to realize end-to-end data acknowledgements via
Datamover-specific means. In view of this, even use of data-
ACK-type SNACKs are unnecessary. Consequently, an initiator
may never request sending a SNACK Request in this model
assuming that the proactive (timeout-driven) SNACK
functionality is turned off in the legacy iSCSI code.
d) Note that the current DA model for bootstrapping a
Connection_Handle into service -- i.e., associating a new iSCSI
connection with a Connection_Handle -- clearly implies that the
iSCSI connection must already be in Full Feature Phase when the
Datamover layer comes into the stack. This further implies
that the iSCSI Login Phase must be carried out in the
traditional "Byte streaming mode" with no assistance or
involvement from the Datamover layer.
A.2. Examples of Datamover Interactions
The figures described in this section provide some examples of the
usage of Operational Primitives in interactions between the iSCSI
layer and the Datamover layer. The following abbreviations are used
in this section.
Avail - Available
Abted - Aborted
Chadalapaka, et al. Informational [Page 35]
RFC 5047 DA October 2007
Buf - I/O Buffer
Cmd - Command
Compl - Complete
Conn - Connection
Ctrl_Ntfy - Control_Notify
Dal_Tk_Res - Deallocate_Task_Resources
Data_Cmp_Nfy - Data_Completion_Notify
Data_ACK_Nfy - Data_ACK_Notify
DM - Datamover
Imm - Immediate
Snd_Ctrl - Send_Control
Msg - Message
Resp - Response
Sol - Solicited
TMF Req - Task Management Function Request
TMF Res - Task Management Function Response
Trans - Transfer
Unsol - Unsolicited
Chadalapaka, et al. Informational [Page 36]
RFC 5047 DA October 2007
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | | Login Phase
| L | Final Login Response (success) v succeeds
| a |<----------------------------------------^
| y | | L | | iSCSI
| e | Enable_Datamover | a | | Full
| r |------------------------------->| y | | Feature
| | Datamover is enabled | e | | Phase
| | | r | |
| | Full Feature Phase | | |
| | control and data Transfer | | v
Figure 2. A Successful iSCSI Login on Initiator
| | Notice_Key_Values | | |
| |------------------------------->| | |
| | Datamover layer is notified | | |
| | of the negotiated key values | | |
| | | | |
| | Allocate_Connection_Resources | | |
| |------------------------------->| D | |
| | Connection resources are | a | |
| i | successfully allocated | t | | iSCSI
| S | | a | | Login
| C | | m |Final | Phase
| S | | o |Login |
| I |Enable_Datamover(Login Response)| v |Resp |
| |------------------------------->| e |---->vLogin Phase
| L | Datamover is enabled | r | ^ succeeds
| a | | | |
| y | | L | | iSCSI
| e | | a | | Full
| r | | y | | Feature
| | | e | | Phase
| | Full Feature Phase | r | |
| | control and data Transfer | | |
| | | | v
Figure 3. A Successful iSCSI Login on Target
Chadalapaka, et al. Informational [Page 37]
RFC 5047 DA October 2007
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | | Login
| | | | | Phase
| L | Final Login Response (failure) v fails
| a |<------------------------------------------
| y | | L |
| e | Deallocate_Connection_Resources| a |
| r |------------------------------->| y |
| | Datamover-specific | e |
| | connection resources freed | r |
| | | |
| |
| | Connection terminated by standard means
| |--------------------------------------------->
Figure 4. A Failed iSCSI Login on Initiator
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | | Login
| | | | | Phase
| L | Final Login Response (failure) v fails
| a |---------------------------------------------->
| y | | L |
| e | Deallocate_Connection_Resources| a |
| r |------------------------------->| y |
| | Datamover-specific | e |
| | connection resources freed | r |
| | | |
| |
| | Connection terminated by standard means
| |-------------------------------------------->
Figure 5. A Failed iSCSI Login on Target
Chadalapaka, et al. Informational [Page 38]
RFC 5047 DA October 2007
| | Allocate_Connection_Resources | D | ^
| |------------------------------->| a | |
| | Connection resources are | t | |
| i | successfully allocated | a | | iSCSI
| S | | m | | Login
| C | | o | | Phase
| S | | v | |
| I | | e | |
| | | r | |
| L | Login non-Final Request/Response |
| a |<-----------------------------------------|
| y | iSCSI layer decides not to | L | |
| e | enable Datamover for this | a | |
| r | connection | y | |
| | | e | |
| | Deallocate_Connection_Resources| r | |
| |------------------------------->| | |
| | All Datamover-specific | | |
| | resources deallocated | | |
| | | | | Login
| | | | | Phase
| | | continues
| | Regular Login negotiation continues |
| |<---------------------------------------->|
| | .
| | .
| | .
Figure 6. iSCSI Does Not Enable the Datamover
Chadalapaka, et al. Informational [Page 39]
RFC 5047 DA October 2007
| | | | ^
| | Full Feature Phase Control & | | |
| | Data Transfer Using DM | D | | iSCSI
| | | a | | Full Feature
| i | | t | | Phase
| S | | a | | (DM Enabled)
| C | | m | |
| S | Successful iSCSI Logout | o | |
| I | | v | v
| | Connection_Terminate | e |
| L |------------------------------->| r |
| a | Connection is terminated | |
| y | Datamover-specific resources | L | Transport
| e | deallocated, both connection | a | Connection
| r | level & task level | y | is terminated
| | | e |
| | | r |
| | | |
| | | |
Figure 7. A Normal iSCSI Connection Termination
| | | | ^
| | Full Feature Phase Control & | D | | iSCSI
| | Data Transfer Using DM | a | | Full Feature
| i | | t | | Phase
| S | | a | | (DM Enabled)
| C | | m | v
| S | | o |<--Transport
| I | Datamover-specific resources | v | Connection
| | deallocated, both connection | e | Terminated (e.g.
| L | level & task level | r | unexpected
| a | | | FIN/RESET)
| y | | L |
| e | Connection_Terminate_Notify | a |
| r |<-------------------------------| y |
| | | e |
| | | r |
| | | |
Figure 8. An Abnormal iSCSI Connection Termination
Chadalapaka, et al. Informational [Page 40]
RFC 5047 DA October 2007
<-----Initiator-----> <-------Target------->
| | | | DM Msg holding | | | |
SCSI | | | | SCSI Cmd PDU & | | | |SCSI
Cmd | | Snd_Ctrl | |Unsol Imm Data | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |--->
| | | | | | | |
| | | | DM Msg holding | | | |
| | Snd_Ctrl | |SCSI Dataout PDU| |Ctrl_Notify | |
| |--------->| |--------------->| |----------->| |
| | . | | . | | . | |Unsol
| | . | D| . | D| . | |Data
| | . | a| DM Msg holding | a| . | |Trans
| i| Snd_Ctrl | t|SCSI Dataout PDU| t|Ctrl_Notify | i|
| S|--------->| a|--------------->| a|----------->| S|
| C| | m| | m| | C|Buf
| S| | o| | o| | S|Avail
| I| | v| | v| Get_Data | I|(R2T)
| | | e|----------------| e|<-----------| |<----
| L| | r||Solicited Data | r| | L| .
| a| | || Transfer | | | a| .
| y| | L|--------------->| L| . | y|Buf
| e| | a| . | a| . | e|Avail
| r| | y| . | y| Get_Data | r|(R2T)
| | | e|----------------| e|<-----------| |<----
| | | r||Solicited Data | r| | |
| | | || Transfer | | | |
| | | |--------------->| |Data_Cmp_Nfy| |Data
| | | | | |----------->| |Trans
| | | | | | | |Compl
| | | | DM Msg holding | | | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | |
Figure 9. A SCSI Write Data Transfer
Chadalapaka, et al. Informational [Page 41]
RFC 5047 DA October 2007
<-----Initiator-----> <-------Target------->
| | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |--->
| | | | | | | |
| | | D| SCSI Read | D| | |Buf
| | | a| Data Transfer | a| Put_Data | |Avail
| i| | t|<---------------| t|<-----------| i|<----
| S| | a| . | a| . | S| .
| C| | m| . | m| . | C| .
| S| | o| . | o| . | S| .
| I| | v| SCSI Read | v| . | I|Buf
| | | e| Data Transfer | e| Put_Data | |Avail
| L| | r|<---------------| r|<-----------| L|<----
| a| | | | | | a|
| y| | L| | L| | y|
| e| | a| | a|Data_Cmp_Nfy| e|Data
| r| | y| | y|----------->| r|Trans
| | | e| | e| | |Compl
| | | r| DM Msg holding | r| | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | |
Figure 10. A SCSI Read Data Transfer
Chadalapaka, et al. Informational [Page 42]
RFC 5047 DA October 2007
<-----Initiator-----> <-------Target------->
| | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |---->
| | | | | | | |
| | | D| SCSI Read | D| Put_Data | |Buf
| | | a| Data Transfer | a|Data_in.A=1 | |Avail
| i| | t|<---------------| t|<-----------| i|<----
| S| | a| . | a| . | S| .
| C| | m| . | m|Data_ACK_Nfy| C| .
| S| | o| | o|----------->| S| .
| I| | v| | v| . | I|
| | | e| | e| . | |
| L| | r| | r| | L|
| a| | | | | | a|
| y| | L| | L| | y|
| e| | a| | a| | e|Data
| r| | y| | y| | r|Trans
| | | e| | e| | |Compl
| | | r| DM Msg holding | r| | |
SCSI | | | |SCSI Resp PDU & | | | |SCSI
Resp | |Ctrl_Ntfy | | Sense Data | | Snd_Ctrl | |Resp
<----| |<---------| |<---------------| |<-----------| |<----
| | | | | | | |
Figure 11. A SCSI Read Data Acknowledgement
Chadalapaka, et al. Informational [Page 43]
RFC 5047 DA October 2007
<-----Initiator-----> <-------Target------->
| | | | | | | |
SCSI | | | | DM Msg holding | | | |SCSI
Cmd | | Snd_Ctrl | | SCSI Cmd PDU | |Ctrl_Notify | |Cmd
---->| |--------->| |--------------->| |----------->| |---->
| | | | | | | |
| | | D| SCSI Read | D| | |Buf
| | | a| Data Transfer | a| Put_Data | |Avail
| i| | t|<---------------| t|<-----------| i|<----
| S| | a| . | a| . | S| .
Abort| C| | m| DM Msg holding | m| . | C|Abort
Task | S| Snd_Ctrl | o| Abort TMF Req | o|Ctrl_Notify | S|Task
---->| I|--------->| v|--------------->| v|----------->| I|---->
| | | e| . | e| . | |
Abort| L| | r| DM Msg holding| r| | L| .
Done | a|Ctrl_Ntfy | | Abort TMF Res| | Snd_Ctrl | |Abted
<----| y|<---------| L|<---------------| L|<-----------| y|<----
| e| | a| | a| | e|
| r| | y| | y| | r|
| | | e| | e| | |
| | | r| | r| | |
| | | | | | | |
| |Dal_Tk_Res| | | |Dal_Tk_Res | |
| |--------->| | | |<-----------| |
| | | | | | | |
Figure 12. Task Resource Cleanup on Abort
Acknowledgements
The IP Storage (IPS) Working Group in the Transport Area of
IETF has been responsible for defining the iSCSI protocol
(apart from a host of other relevant IP Storage protocols).
The authors are grateful to the entire working group, whose
work allowed this document to build on the concepts and
details of the iSCSI protocol.
In addition, the following individuals reviewed and
contributed to the improvement of this document. The authors
are grateful for their contribution.
John Carrier
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (360) 378-8526
EMail: john_carrier@adaptec.com
Chadalapaka, et al. Informational [Page 44]
RFC 5047 DA October 2007
Hari Ghadia
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (408) 957-5608
EMail: hari_ghadia@adaptec.com
Hari Mudaliar
Adaptec, Inc.
691 S. Milpitas Blvd., Milpitas, CA 95035, USA
Phone: +1 (408) 957-6012
EMail: hari_mudaliar@adaptec.com
Patricia Thaler
Agilent Technologies, Inc.
1101 Creekside Ridge Drive, #100, M/S-RG10,
Roseville, CA 95678, USA
Phone: +1-916-788-5662
EMail: pat_thaler@agilent.com
Uri Elzur
Broadcom Corporation
16215 Alton Parkway, Irvine, CA 92619-7013, USA
Phone: +1 (949) 585-6432
EMail: Uri@Broadcom.com
Mike Penna
Broadcom Corporation
16215 Alton Parkway,Irvine, CA 92619-7013, USA
Phone: +1 (949) 926-7149
EMail: MPenna@Broadcom.com
David Black
EMC Corporation
176 South St., Hopkinton, MA 01748, USA
Phone: +1 (508) 293-7953
EMail: black_david@emc.com
Ted Compton
EMC Corporation
Research Triangle Park, NC 27709, USA
Phone: +1-919-248-6075
EMail: compton_ted@emc.com
Dwight Barron
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-2769
EMail: Dwight.Barron@Hp.com
Chadalapaka, et al. Informational [Page 45]
RFC 5047 DA October 2007
Paul R. Culley
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-5543
EMail: paul.culley@hp.com
Dave Garcia
Hewlett-Packard Company
19333 Vallco Parkway, Cupertino, CA 95014, USA
Phone: +1 (408) 285-6116
EMail: dave.garcia@hp.com
Randy Haagens
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA, USA
Phone: +1-916-785-4578
EMail: randy_haagens@hp.com
Jeff Hilland
Hewlett-Packard Company
20555 SH 249, Houston, TX 77070-2698, USA
Phone: +1 (281) 514-9489
EMail: jeff.hilland@hp.com
Mike Krause
Hewlett-Packard Company, 43LN
19410 Homestead Road, Cupertino, CA 95014, USA
Phone: +1 (408) 447-3191
EMail: krause@cup.hp.com
Jim Wendt
Hewlett-Packard Company
8000 Foothills Blvd, MS 5668, Roseville CA, USA
Phone: +1-916-785-5198
EMail: jim_wendt@hp.com
Mike Ko
IBM
650 Harry Rd, San Jose, CA 95120, USA
Phone: +1 (408) 927-2085
EMail: mako@us.ibm.com
Renato Recio
IBM Corporation
11501 Burnett Road, Austin, TX 78758, USA
Phone: +1 (512) 838-1365
EMail: recio@us.ibm.com
Chadalapaka, et al. Informational [Page 46]
RFC 5047 DA October 2007
Howard C. Herbert
Intel Corporation
MS CH7-404,5000 West Chandler Blvd., Chandler, AZ 85226, USA
Phone: +1 (480) 554-3116
EMail: howard.c.herbert@intel.com
Dave Minturn
Intel Corporation
MS JF1-210, 5200 North East Elam Young Parkway
Hillsboro, OR 97124, USA
Phone: +1 (503) 712-4106
EMail: dave.b.minturn@intel.com
James Pinkerton
Microsoft Corporation
One Microsoft Way, Redmond, WA 98052, USA
Phone: +1 (425) 705-5442
EMail: jpink@microsoft.com
Tom Talpey
Network Appliance
375 Totten Pond Road, Waltham, MA 02451, USA
Phone: +1 (781) 768-5329
EMail: thomas.talpey@netapp.com
Chadalapaka, et al. Informational [Page 47]
RFC 5047 DA October 2007
Authors' Addresses
Mallikarjun Chadalapaka
Hewlett-Packard Company
8000 Foothills Blvd.
Roseville, CA 95747-5668, USA
Phone: +1-916-785-5621
EMail: cbm@rose.hp.com
John L. Hufferd
Brocade, Inc.
1745 Technology Drive
San Jose, CA 95110, USA
Phone: +1-408-333-5244
EMail: jhufferd@brocade.com
Julian Satran
IBM, Haifa Research Lab
Haifa University Campus - Mount Carmel
Haifa 31905, Israel
Phone +972-4-829-6264
EMail: Julian_Satran@il.ibm.com
Hemal Shah
Broadcom Corporation
5300 California Avenue
Irvine, California 92617, USA
Phone: +1-949-926-6941
EMail: hemal@broadcom.com
Comments may be sent to Mallikarjun Chadalapaka.
Chadalapaka, et al. Informational [Page 48]
RFC 5047 DA October 2007
Full Copyright Statement
Copyright (C) The IETF Trust (2007).
This document is subject to the rights, licenses and restrictions
contained in BCP 78, and except as set forth therein, the authors
retain all their rights.
This document and the information contained herein are provided on an
"AS IS" basis and THE CONTRIBUTOR, THE ORGANIZATION HE/SHE REPRESENTS
OR IS SPONSORED BY (IF ANY), THE INTERNET SOCIETY, THE IETF TRUST AND
THE INTERNET ENGINEERING TASK FORCE DISCLAIM ALL WARRANTIES, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF
THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED
WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.
Intellectual Property
The IETF takes no position regarding the validity or scope of any
Intellectual Property Rights or other rights that might be claimed to
pertain to the implementation or use of the technology described in
this document or the extent to which any license under such rights
might or might not be available; nor does it represent that it has
made any independent effort to identify any such rights. Information
on the procedures with respect to rights in RFC documents can be
found in BCP 78 and BCP 79.
Copies of IPR disclosures made to the IETF Secretariat and any
assurances of licenses to be made available, or the result of an
attempt made to obtain a general license or permission for the use of
such proprietary rights by implementers or users of this
specification can be obtained from the IETF on-line IPR repository at
http://www.ietf.org/ipr.
The IETF invites any interested party to bring to its attention any
copyrights, patents or patent applications, or other proprietary
rights that may cover technology that may be required to implement
this standard. Please address the information to the IETF at
ietf-ipr@ietf.org.
Chadalapaka, et al. Informational [Page 49]
RFC, FYI, BCP