Sunday, February 10, 2013

Device Mapping and Logical System Board Numbers in SPARC Enterprise M3000/M4000/M5000/M8000/M9000 servers from Oracle and Fujitsu



Mapping Device Path Names
This appendix describes how to map device path names to physical system devices in SPARC Enterprise M3000/M4000/M5000/M8000/M9000 servers from Oracle and Fujitsu. It contains these sections:


Device Mapping and Logical System Board Numbers

The physical address represents a physical characteristic that is unique to the device. Examples of physical addresses include the bus address and the slot number. The slot number indicates where the device is installed.
The logical system board (LSB) number affects both the processor numbering and the I/O device paths in the server. Physical resources are assigned to LSBs in the specified domain using the setdcl command. It is the LSB numbers that the Oracle Solaris OS uses.


CPU Mapping

Each LSB on a midrange or high-end server has a bank of 32 processor numbers assigned to it. For entry-level servers, the bank holds eight processors. The SPARC64 VI processor has two cores, each with two threads (also known as virtual processors). The SPARC64 VII+ and SPARC64 VII processor has two cores or four cores, each with two threads.
An LSB on a midrange or high-end server has up to four processors (when a Uni-XSB is assigned to the LSB); therefore, the LSB needs 32 processor IDs. An LSB on an entry-level server, which supports only a single processor, requires only eight processor IDs.
TABLE A-1 shows the relationship between LSB numbers and starting processor (proc) numbers, in hexadecimal/decimal format. The Oracle Solaris prtdiag(IM) command provides the LSB numbers and CPU chip numbers in decimal format for components that are part of the domain.

TABLE A-1 LSB Numbers and Starting Processor Numbers
LSB Number
CPU Chip 0
CPU Chip 1
CPU Chip 2
CPU Chip 3
Entry-Level Servers
00
00/00
N/A
N/A
N/A
Midrange and High-End Servers
00
00/00
08/08
10/16
18/24
01
20/32
28/40
30/48
38/56
02
40/64
48/72
50/80
58/88
03
60/96
68/104
70/112
78/120
04
80/128
88/136
90/144
98/152
05
a0/160
a8/168
b0/176
b8/184
06
c0/192
c8/200
d0/208
d8/216
07
e0/224
e8/232
f0/240
f8/248
08
100/256
108/264
110/272
118/280
09
120/288
128/296
130/304
138/312
10
140/320
148/328
150/336
158/344
11
160/352
168/360
170/368
178/376
12
180/384
188/392
190/400
198/408
13
1a0/416
1a8/424
1b0/432
1b8/440
14
1c0/448
1c8/456
1d0/464
1d8/472
15
1e0/480
1e8/488
1f0/496
1f8/504

CPU Numbering Examples

This section contains examples of CPU numbering, using the output of the showboards(8) command on the Service Processor, and the output of the prtdiag(1M) command on the domain.

XSCF> showboards -a 
 
XSB  DID(LSB) Assignment  Pwr  Conn Conf Test    Fault    
---- -------- ----------- ---- ---- ---- ------- -------- 
00-0 00(00)   Assigned    y    y    y    Passed  Normal   
00-1 00(01)   Assigned    y    y    y    Passed  Normal   
00-2 00(04)   Assigned    y    y    y    Passed  Normal   
00-3 00(05)   Assigned    y    n    n    Passed  Normal   
01-0 01(00)   Assigned    y    y    y    Passed  Normal   
01-1 01(09)   Assigned    y    y    y    Passed  Normal   
01-2 01(06)   Assigned    y    n    n    Passed  Normal   
01-3 01(07)   Assigned    y    n    n    Passed  Normal   


domain_0# prtdiag -v
...
==================================== CPUs ====================================
 
       CPU              CPU            Run       L2$       CPU      CPU 
LSB    Chip              ID            MHz        MB       Impl.    Mask 
---    ----      --------------------  ----      ---       -----    ---- 
 00      0          0,   1,   2,   3   2150      4.0          6      129
 01      1         40,  41,  42,  43   2150      4.0          6      129
 04      2        144, 145, 146, 147   2150      4.0          6      129 
 05      3        184, 185, 186, 187   2150      4.0          6      129
 
==============================================================================


domain_1# prtdiag -v
...
==================================== CPUs ====================================
 
       CPU              CPU            Run       L2$       CPU      CPU 
LSB    Chip              ID            MHz        MB       Impl.    Mask 
---    ----      --------------------  ----      ---       -----    ---- 
 00      0          0,   1,   2,   3   2150      4.0          6      129
 09      1        296, 297, 298, 299   2150      4.0          6      129
 06      2        208, 209, 210, 211   2150      4.0          6      129
 07      3        248, 249, 250, 251   2150      4.0          6      129
 
==============================================================================


I/O Device Mapping

I/O device paths are dictated by which LSB the I/O unit is assigned to.
Entry-level servers have one I/O controller. The XSB is assigned four PCIe slots.
Midrange servers have only one I/O controller on the I/O unit (IOU). For an XSB in Uni-XSB mode, all I/O is on XSB#xx-0. For an XSB in Quad-XSB mode, internal resources, the PCI-X slot, and two PCIe slots are on XSB#xx-0, and two PCIe slots are on XSB#xx-1.
High-end servers have two I/O controllers; therefore, each XSB can have two PCIe slots assigned to it.
TABLE A-2 shows the LSB numbers and the corresponding device path values that are used in I/O device mapping on the server.

TABLE A-2 LSB Numbers and Device Path Values
LSB Number
Device Path Value
00
No value
01
1
02
2
03
3
04
4
05
5
06
6
07
7
08
8
09
9
10
a
11
b
12
c
13
d
14
e
15
f

I/O Device Mapping on Entry-Level Servers

TABLE A-3 shows the device mapping on an entry-level server.

TABLE A-3 I/O Device Mapping on an Entry-level Server
PCIe Slot
Host Bus Adapter Slot Type
OpenBoot PROM Device Path
Slot 0
PCIe
/pci@0,600000/pci@0/pci@8
Slot 1
PCIe
/pci@1,700000/pci@0/pci@0
Slot 2
PCIe
/pci@1,700000/pci@0/pci@8
Slot 3
PCIe
/pci@1,700000/pci@0/pci@9

Internal Devices on Entry-Level Servers With SPARC64 VII+ Processors

The entry-level server has a single system board, at location XSB 00-0. Internal devices and device paths are shown in TABLE A-4.

TABLE A-4 Internal Devices and Device Paths on an Entry-level Server with SPARC64 VII+ Processors
XSB 00-0/IOU 0
Accessible Internal Devices
Device Physical Location
OpenBoot PROM Device Path
Network Port 0
System
/pci@0,600000/pci@0/pci@1/network@0
Network Port 1
System
/pci@0,600000/pci@0/pci@1/network@0,1
Network Port 2
System
/pci@0,600000/pci@0/pci@2/network@0
Network Port 3
System
/pci@0,600000/pci@0/pci@2/network@0,1
HD0
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@0
HD1
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@1
HD2
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@2
HD3
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@3
CD-RW/DVD-RW
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@4
SAS port
System
/pci@0,600000/pci@0/pci@0/scsi@0/xx@5,z, where xx is the disk when connecting to a disk, or a tape when connecting to a tape drive unit.

Internal Devices on Entry-Level Servers Without SPARC64 VII+ Processors

The entry-level server has a single system board, at location XSB 00-0. Internal devices and device paths are shown in TABLE A-5

TABLE A-5 Internal Devices and Device Paths on an Entry-level Server without SPARC64 VII+ Processors
XSB 00-0/IOU 0
Accessible Internal Devices
Device Physical Location
OpenBoot PROM Device Path
Network Port 0
System
/pci@0,600000/pci@0/pci@1/pci@0/network@4
Network Port 1
System
/pci@0,600000/pci@0/pci@1/pci@0/network@4,1
Network Port 2
System
/pci@0,600000/pci@0/pci@2/pci@0/network@4
Network Port 3
System
/pci@0,600000/pci@0/pci@2/pci@0/network@4,1
HD0
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@0
HD1
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@1
HD2
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@2
HD3
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@3
CD-RW/DVD-RW
System
/pci@0,600000/pci@0/pci@0/scsi@0/disk@4
SAS port
System
/pci@0,600000/pci@0/pci@0/scsi@0/xx@5,z, where xx is the disk when connecting to a disk, or a tape when connecting to a tape drive unit.

I/O Device Mapping on Midrange Servers

TABLE A-6 shows the device mapping on a midrange server. In the device path, x is LSB-dependent, and is assigned a value as shown in TABLE A-2.

TABLE A-6 I/O Device Mapping on a Midrange Server
Slot
Host Bus Adapter Slot Type
OpenBoot PROM Device Path
IOU Slot 0
PCI-X
/pci@x0,600000/pci@0/pci@8/pci@0,1
IOU Slot 1
PCIe
/pci@x0,600000/pci@0/pci@9
IOU Slot 2
PCIe
/pci@x1,700000
IOU Slot 3
PCIe
/pci@x2,600000
IOU Slot 4
PCIe
/pci@x3,700000

Internal Devices on Midrange Servers

The internal midrange server devices, which are located at the XSB location 00-0 or 01-0 (regardless of Uni-XSB or Quad-XSB mode), are shown in TABLE A-7 and TABLE A-8. In the device path, x is LSB-dependent, and is assigned a value as shown in TABLE A-2.

TABLE A-7 Internal Devices and Device Paths on the Midrange Servers, IOU#0
XSB 00-0/IOU 0 Accessible Internal Devices (M4000/M5000)
Device Physical Location
OpenBoot PROM Device Path
Network Port 0
IOU#0
/pci@x0,600000/pci@0/pci@8/pci@0/network@2
Network Port 1
IOU#0
/pci@x0,600000/pci@0/pci@8/pci@0/network@2,1
HD0
System
/pci@x0,600000/pci@0/pci@8/pci@0/scsi@1/disk@0
HD1
System
/pci@x0,600000/pci@0/pci@8/pci@0/scsi@1/disk@1
CD-RW/DVD-RW
System
/pci@x0,600000/pci@0/pci@8/pci@0/scsi@1/disk@3
Tape
System
/pci@x0,600000/pci@0/pci@8/pci@0/scsi@1/tape@2


TABLE A-8 Internal Devices and Device Paths on the M5000 (but not M4000) Server, IOU#1
XSB 01-0/IOU 1 Accessible Internal Device (M5000)
Device Physical Location
OpenBoot PROM Device Path
Network Port 0
IOU#1
/pci@x0,600000/pci@0/pci@8/pci@0/network@2
Network Port 1
IOU#1
/pci@x0,600000/pci@0/pci@8/pci@0/network@2,1
HD2
System
/pci@x0,600000/pci@0/pci@8/pci@0/scsi@1/disk@0
HD3
System
/pci@x0,600000/pci@0/pci@8/pci@0/scsi@1/disk@1

I/O Device Mapping on High-End Servers

TABLE A-9 shows the device mapping on a high-end server. In the PCIe device path, x is LSB-dependent, and is assigned a value as shown in TABLE A-2. xx is the XSB number and is in the range from 00-15.

TABLE A-9 I/O Device Mapping on a High-end Server
PCIe Slot
Uni-XSB[1]
Quad-XSB[2]
OpenBoot PROM PCIe Device Path[3]
IOU Slot 0
xx-0
xx-0
pci@x0,600000
IOU Slot 1
xx-0
xx-0
pci@x1,700000
IOU Slot 2
xx-0
xx-1
pci@x2,600000
IOU Slot 3
xx-0
xx-1
pci@x3,700000
IOU Slot 4
xx-0
xx-2
pci@x4,600000
IOU Slot 5
xx-0
xx-2
pci@x5,700000
IOU Slot 6
xx-0
xx-3
pci@x6,600000
IOU Slot 7
xx-0
xx-3
pci@x7,700000

Internal Devices on High-End Servers

The IOUA is a PCIe Host Bus Adapter that provides access to internal devices when installed at specific locations. The IOUA contains two 1Gb Ethernet ports on the card (“on-board”). When the IOUA is installed at specific locations, it also provides access to storage located on the IOU, as well as platform CD-RW/DVD-RW drive or tape drive resources at the locations shown in TABLE A-10. In the PCIe device path, x is LSB-dependent, and is assigned a value as shown in TABLE A-2. xx is the XSB number and is in the range from 00-15. nn is the number associated with the PSB to which the CD-RW/DVD-RW drive or tape drive is attached, as further explained in the table footnote.
 

TABLE A-10 Internal Devices and Device Paths on a High-end Server
PCIe Slot
Uni-XSB[4]
Quad-XSB[5]
OpenBoot PROM PCIe Device Path[6]
OpenBoot PROM IOUA HBA On-board, IOU, and Platform Accessible Devices[7]
IOU Slot 0
xx-0
xx-0
pci@x0,600000
.../pci@0,1/network@1 (IOUA HBA On-board BGE Port 0)
.../pci@0,1/network@1,1 (IOUA HBA On-board BGE Port 1)
.../pci@0/scsi@1/disk@0 (IOU HD0; SCSI Target 0)
.../pci@0/scsi@1/disk@1 (IOU HD1; SCSI Target 1)
.../pci@0/scsi@1/disk@4 (Platform CD-RW/DVD-RW at cfgdevice port
nn-0; SCSI Target 4)
.../pci@0/scsi@1/tape@5 (Platform tape at cfgdevice port nn-0;
SCSI Target 5)
IOU Slot 1
xx-0
xx-0
pci@x1,700000
 
IOU Slot 2
xx-0
xx-1
pci@x2,600000
.../pci@0,1/network@1 (IOUA HBA On-board BGE Port 0)
.../pci@0,1/network@1,1 (IOUA HBA On-board BGE Port 1)
.../pci@0/scsi@1/disk@4 (Platform CD-RW/DVD-RW at cfgdevice port
nn-2; SCSI Target 4)
.../pci@0/scsi@1/tape@5 (Platform tape at cfgdevice port nn-2;
SCSI Target 5)
IOU Slot 3
xx-0
xx-1
pci@x3,700000
.
IOU Slot 4
xx-0
xx-2
pci@x4,600000
.../pci@0,1/network@1 (IOUA HBA On-board BGE Port 0)
.../pci@0,1/network@1,1 (IOUA HBA On-board BGE Port 1)
.../pci@0/scsi@1/disk@0 (IOU HD2; SCSI Target 0)
.../pci@0/scsi@1/disk@1 (IOU HD3; SCSI Target 1)
.../pci@0/scsi@1/disk@4 (Platform CD-RW/DVD-RW at cfgdevice port
nn-4; SCSI Target 4)
.../pci@0/scsi@1/tape@5 (Platform tape at cfgdevice port nn-4;
SCSI Target 5)
IOU Slot 5
xx-0
xx-2
pci@x5,700000
 
IOU Slot 6
xx-0
xx-3
pci@x6,600000
.../pci@0,1/network@1 (IOUA HBA On-board BGE Port 0)
.../pci@0,1/network@1,1 (IOUA HBA On-board BGE Port 1)
.../pci@0/scsi@1/disk@4 (Platform CD-RW/DVD-RW at cfgdevice port
nn-6; SCSI Target 4)
.../pci@0/scsi@1/tape@5 (Platform tape at cfgdevice port nn-6;
SCSI Target 5)
IOU Slot 7
xx-0
xx-3
pci@x7,700000
 

Sample cfgadm Output

This section contains:
  • Sample output for the command cfgadm -s “select=class(pci)” on an unpopulated server. As you connect devices, the cfgadm output will change to reflect the device type and connection status on your server.
  • The device matrix for midrange and for high-end servers, when the IOU is configured as part of a domain. I/O portions of the IOU resources may be in different domains.

Entry-Level Server

The entry-level server does not support PCI hotplug. Therefore, the concepts of attachment points and classes do not apply, and executing the command

cfgadm -s "select=class(pci)"

either would produce an error or display nothing.

Midrange Servers

M4000 Server sample output:

# cfgadm -s "select=class(pci)"
Ap_Id                          Type         Receptacle   Occupant     Condition
iou#0-pci#0                    unknown      empty        unconfigured unknown
iou#0-pci#1                    unknown      empty        unconfigured unknown
iou#0-pci#2                    unknown      empty        unconfigured unknown
iou#0-pci#3                    unknown      empty        unconfigured unknown
iou#0-pci#4                    unknown      empty        unconfigured unknown

M5000 Server sample output:

# cfgadm -s "select=class(pci)"
Ap_Id                          Type         Receptacle   Occupant     Condition
iou#0-pci#0                    unknown      empty        unconfigured unknown
iou#0-pci#1                    unknown      empty        unconfigured unknown
iou#0-pci#2                    unknown      empty        unconfigured unknown
iou#0-pci#3                    unknown      empty        unconfigured unknown
iou#0-pci#4                    unknown      empty        unconfigured unknown
iou#1-pci#0                    unknown      empty        unconfigured unknown
iou#1-pci#1                    unknown      empty        unconfigured unknown
iou#1-pci#2                    unknown      empty        unconfigured unknown
iou#1-pci#3                    unknown      empty        unconfigured unknown
iou#1-pci#4                    unknown      empty        unconfigured unknown


TABLE A-11 cfgadm Device Matrix for Midrange Servers
PCI Slot #
PCI Slot Type
IOU#0 (M4000/M5000)
IOU#1 (M5000)
0
PCI-X
iou#0-pci#0
iou#1-pci#0
1
PCIe
iou#0-pci#1
iou#1-pci#1
2
PCIe
iou#0-pci#2
iou#1-pci#2
3
PCIe
iou#0-pci#3
iou#1-pci#3
4
PCIe
iou#0-pci#4
iou#1-pci#4

High-End Servers

M8000 Server sample output:

# cfgadm -s "select=class(pci)"
Ap_Id                          Type         Receptacle   Occupant     Condition
iou#1-pci#0                    unknown      empty        unconfigured unknown
iou#1-pci#1                    unknown      empty        unconfigured unknown
iou#1-pci#4                    unknown      empty        unconfigured unknown
iou#1-pci#5                    unknown      empty        unconfigured unknown
iou#1-pci#6                    unknown      empty        unconfigured unknown
iou#1-pci#7                    unknown      empty        unconfigured unknown

M9000 Server sample output:

# cfgadm -s "select=class(pci)"
Ap_Id                          Type         Receptacle   Occupant     Condition
iou#0-pci#0                    unknown      empty        unconfigured unknown
iou#0-pci#1                    unknown      empty        unconfigured unknown
iou#0-pci#2                    unknown      empty        unconfigured unknown
iou#0-pci#3                    unknown      empty        unconfigured unknown
iou#0-pci#4                    unknown      empty        unconfigured unknown
iou#0-pci#5                    unknown      empty        unconfigured unknown
iou#0-pci#6                    unknown      empty        unconfigured unknown
iou#0-pci#7                    unknown      empty        unconfigured unknown
iou#3-pci#0                    unknown      empty        unconfigured unknown
iou#3-pci#1                    unknown      empty        unconfigured unknown
iou#3-pci#2                    unknown      empty        unconfigured unknown
iou#3-pci#3                    unknown      empty        unconfigured unknown


TABLE A-12 cfgadm Device Matrix for High-End Servers
PCI Slot #
PCI Slot Type
IOU#0
IOU#1
IOU#n[8]
0
PCIe
iou#0-pci#0
iou#1-pci#0
iou#n-pci#0
1
PCIe
iou#0-pci#1
iou#1-pci#1
iou#n-pci#1
2
PCIe
iou#0-pci#2
iou#1-pci#2
iou#n-pci#2
3
PCIe
iou#0-pci#3
iou#1-pci#3
iou#n-pci#3
4
PCIe
iou#0-pci#4
iou#1-pci#4
iou#n-pci#4
5
PCIe
iou#0-pci#5
iou#1-pci#5
iou#n-pci#5
6
PCIe
iou#0-pci#6
iou#1-pci#6
iou#n-pci#6
7
PCIe
iou#0-pci#7
iou#1-pci#7
iou#n-pci#7


1 (TableFootnote) xx is the XSB number, 00-15
2 (TableFootnote) xx is the XSB number, 00-15
3 (TableFootnote) x is LSB-dependent, assigned a value as shown in TABLE A-2
4 (TableFootnote) xx is the XSB number, in the range of 00-15.
5 (TableFootnote) xx is the XSB number, in the range of 00-15.
6 (TableFootnote) x is LSB-dependent, and is assigned a value as shown in TABLE A-2.
7 (TableFootnote) nn is the number associated with the PSB to which the CD-RW/DVD-RW drive or tape drive is attached, as follows: for an M8000 server, nn is in the range of 0-3; for an M9000 server, nn is in the range of 0-7; for an M9000 server plus expansion unit, nn is in the range of 0-15.
8 (TableFootnote) n is the IOU number