(DISCLAIMER: I am a UCS-focused Consulting SE at Cisco.  I am NOT an EMC VNX expert by any stretch of the imagination.) 

Jonathan, this is a great reference!  However, your last point is mixing up three distinct concepts in UCS solution design that may or may not be involved in making the upstream network part of the critical path for iSCSI traffic: 

1) VLAN matching between Appliances and LAN Cloud
    Ethernet Appliances will, by definition, only be connected to a subset of the VLANs defined on a UCS fabric.  Therefore there is a separate section under the LAN tab called Appliances where these specific VLANs are defined specifically for consumption by Appliance ports.  These VLANs must still be defined on the UCS LAN Cloud overall in order to forward traffic either internally to UCS hosts or egress the fabric on a soft- or hard-pinned uplink.  So if you create a VLAN only in the Appliances section, you cannot configure it in a hosts' vNIC configuration.  But having those VLANs in existence in both the LAN Cloud and under Appliances does not imply that the storage traffic must leave the Fabric Interconnect to reach the host port or appliance; that is up to the way the hosts and appliance are communicating in normal operation or failover scenarios.

2) Uplink Pinning and Network Control Policies
   It is assumed by design that whatever you plug in downstream of the UCS Fabric Interconnect operating in End Host Mode, either on a Server Port or Appliance Port, will probably need to talk to something outside of its home UCS fabric at some point, so all vNICs and Appliance Ports are pinned to an Uplink port or port channel either automatically (soft-pinning) or based on user designated pairs of individual ports/port channels on the two fabric interconnects (hard-pinning).  In order to make sure a host or appliance knows when it becomes isolated from the rest of the data center Ethernet fabric, the default behavior is to administratively down the vEth that device is connected to if either the specific port it is pinned to (in the case of hard-pinning) or all uplinks to the same L2 domain (in the case of soft-pinning) lose physical link.  So by default, if those upstream switches go away, the storage appliance and any other device talking to them will lose connectivity (this is also where Fabric Failover kicks in on the Virtual Interface Card).  This behavior can be overridden by specifying "Warn only" under the Action On Uplink Fail value of a Network Control Policy.  The ability to apply a custom Network Control Policy to Appliance Ports was added in the most recent version of UCS firmware, v2.0.  This means that the loss of the uplink(s) which an appliance port or storage-oriented vNIC is pinned to no longer requires those downstream interfaces to be disabled, and traffic can continue to be switched between vNICs and Appliance Ports on the same UCS fabric if the upstream switch goes down.

3) Storage Traffic Pathing
    Given the configuration you provide above, with completely isolated redundant iSCSI storage VLANs, a healthy environment would not require a UCS server to send its storage traffic through the upstream switch to the other fabric.  Both SPs have IP addresses in both VLANs, both SPs have links in each VLAN connected directly to the UCS fabric that hosts that VLAN, host iSCSI interfaces are connected only to Fabric A or B as appropriate, and tagged only for iSCSI-A or B.  In this design, not only would storage traffic not need to pass between the UCS fabric via the upstream switches, but they couldn't possibly do so, since the vLANs are effectively isolated on each fabric.  Nothing is listening in the iSCSI-B VLAN on Fabric A, and vice versa.

So while your observation that upstream switches are required for iSCSI storage availability to UCS hosts is accurate in a UCS v1.4 environment due to inability to specify a "Warn only" Network Control Policy for an Appliance Port, and while VLANs must be defined in both the LAN Cloud and for Appliances in order for UCS host servers' storage vNICs to talk to Appliance ports, those VLANs do not necessarily have to be allowed on the upstream switch ports 9Unless you need things outside of UCS to tak to the SP iSCSI ports).  Given the design you laid out here, turning on a "warn only" NCP for Appliance ports and storage-specific vNICs in 2.0 would also prevent the loss of the upstream switch from affecting your iSCSI traffic between UCS hosts and the VNX, thus eliminating it from the critical path for iSCSI.

Hope this helps to clarify things,

BT