SGI Origin 3400

The Silicon Graphics Origin 3000 series machines are modular single-system-image (distributed shared memory) supercomputers, released in the year 2000. They're scalable from 2 to 512 processors and from 1GB to 1TB of RAM. The interconnect between nodes is SGI's proprietary NUMAlink3, which offers very low latency (1 microsecond for MPI messages, 50 nanoseconds round-trip for raw NUMAlink packets) and high bandwidth (1.6 GBytes/s in one direction). The topology of the entire system is a fat bristled hypercube.

Origin 3400 rack diagram
Origin 3400 rack cutaway diagram
Origin 3400 32-processor rack diagram

Table of contents

General description ^

Each Origin system consists of multiple different "bricks" (that is the official terminology), the following types are available:

NameDiagramBack panelDescription
C-brickC-brick diagramC-brick back panelContains up to 4 processors and 8GB of RAM. It's possible to mix different CPU types and speeds within the system, but not within a single C-brick.
I-brickI-brick diagramI-brick back panelProvides basic I/O: 10/100 ethernet, two fibre-channel disks, CD-ROM drive (TEAC CD-540E, connected via a FireWire-to-IDE adapter), five hot-pluggable 64-bit PCI slots (on two busses, 2x 66MHz + 3x 33MHz slots), two USB ports, and a FireWire (IEEE-1394) port.
R-brickR-brick diagramR-brick back panelProvides routing between C-bricks in systems with more than two C-bricks (there are 6- and 8-port versions of the router brick).
P-brickP-brick diagramP-brick back panelProvides an additional 12 hot-pluggable PCI slots (on 6 busses).
X-brickX-brick diagramX-brick back panelProvides four XIO expansion slots (XIO is SGI's proprietary expansion interface used in their previous line of Origin 2000, Onyx2, and Octane systems).
G-brickG-brick diagramG-brick back panelProvides graphics (this brick is the only difference between an Origin 3000 and an Onyx 3000). Each G-brick supports one or two InfiniteReality pipes (sets of boards).
V-brickV-brick diagramV-brick back panelProvides lower-end graphics (an Onyx 3000 can have either one or more G-bricks or one or more V-bricks, but not both). Each V-brick supports one or two InfinitePerformance pipes, each pipe consists of a VPro V12 graphics board (also available in workstations).
N-brickN-brick diagramN-brick back panelConnects four C-bricks to four G-bricks (useful if you don't need the extra I/O of I- and X-bricks that would have been used for this purpose instead).
D-brickD-brick diagramD-brick back panelProvides disk storage (twelve hot-pluggable fibre-channel drives, no RAID).
Power bayPower bay diagram (front)Power bay diagram (back)Provides 48V DC power to bricks within a rack. Contains 3 to 6 power supplies that are hot-swappable and N+1 redundant.

Each brick also contains massive hot-swappable fans in a front-to-back airflow configuration and an L1 controller with an illuminated LCD, showing the status of the brick.

In addition to bricks, each system contains one or more L2 controllers (one per rack) with optional monochrome touchscreen LCD (for turning the system and individual bricks on/off) and power bays (with hot-swappable redundant power supplies). The power supplies provide 48V DC to bricks within a rack.

The difference between the Origin 3200, 3400, and 3800 units is purely in the combination of bricks. The 3200 has a maximum of 8 processors and comes in a short rack with two C-bricks, one I-brick, and one power bay (one slot is free for an additional I-, P-, or X-brick). The 3400 has a maximum of 32 processors and comes in a tall rack with two to eight C-bricks, one I-brick, two R-bricks (6-port), and one power bay. The 3800 series is for larger installations and comes in multiple racks.

Block diagram of the Origin 3400 ^

Block diagram of a 16-processor Origin 3400 The above diagram shows a 16-CPU configuration. A 32-CPU configuration contains twice the elements in the diagram and the two routers are linked together with two NUMAlink3 cables.

Common hardware issues ^

The Origin 3000 series machines consist of a lot of parts, so it's not unusual for some of them to start acting up due to old age (but otherwise, it's a very reliable machine).

Replacing batteries ^

There are three kinds of batteries in the Origin 3000 machines:

The L2 controller battery can be replaced with any 1/2AA (half-AA) 3V lithium battery, but you can also use the 3.6V batteries more common in older Apple Macs (e.g. Tadiran SL-750/S) — these normally have a higher capacity than the standard 3V batteries, so they last longer. I use a Tadiran SL-750/S 3.6V battery in my L2 controller.
To get to the battery, remove the L2 controller from the rack and unscrew its top cover — the battery is mounted in a holder in the middle of the board.
After replacing the battery, set the current date and time using the "date" L2 command.

The I-brick battery can be replaced either with the original ST M4T28-BR12SH1 (48 mAh capacity), or even better, with a higher-capacity industrial ST M4T32-BR12SH6 (120 mAh capacity). The last part of the the part number denotes the temperature range — SH1 is commercial (0C to 70C) and SH6 is industrial (-40C to +85C). I use an M4T32-BR12SH6.
To get to the battery, remove the I-brick from the rack and unscrew the top panel located towards the back of the brick — the battery is located in the middle of the board, near the large black heatsink. Use an IC remover tool to gently remove it from the socket and put the new one in its place (make note of the orientation!).
After replacing the battery, set the current date and time in IRIX.

The non-replaceable batteries inside the DS1742W RTC chips don't really pose any problems for the Origin (unlike in some other systems), in my experience. In the case of R-bricks, having flat batteries is even advantageous, as they will happily accept any serial number, so you can mix and match them with any Origin system you want! (See section R-brick serial number mismatches below for more information.)

Dealing with faulty DS1780 chips ^

SGI got a bad batch of DS1780E monitoring chips that seem to fail over time, resulting in a lot of annoying bogus error messages related to environmental monitoring on all systems based on the Origin 3000 design (Fuel, Tezro, etc.).

Typical error messages on the L2 console include:

001i21 ALERT: Error reading monitor POWER 1 interrupt status 1: no acknowledge
001c07 ALERT: Error reading monitor NODE 0 interrupt status 1: no acknowledge
001c30 ALERT: Error reading monitor NODE 2 interrupt status 1: no acknowledge
001c33 ALERT: Error reading monitor NODE 2 interrupt status 1: no acknowledge
001c33 ALERT: Error reading monitor PIMM1 A interrupt status 1: no acknowledge
001c33 ALERT: Error reading monitor PIMM1 B interrupt status 1: no acknowledge
001r25 ALERT: Error reading monitor POWER interrupt status 1: no acknowledge

And so on (these errors repeat every few seconds). What it's trying to tell you is "I can't talk to the DS1780E chip, it's not responding!"

I haven't had the time to pinpoint the exact nature of the error, but in addition to giving bogus temperature and voltage readings, I suspect that the faulty chips drive the I2C bus lines either high or low at inappropriate times, thus preventing communication from taking place on the bus.

This isn't such a big problem on the Origin, as you can just disable environmental monitoring with "* env off" (and "* env temp off") on the L2 controller (you only need to do this once, the bricks will remember the setting). Then to turn off the nagging messages, do "l1dbg printf off" on the L2 controller (you need to do this every time you apply power to the Origin rack). If you do disable environmental monitoring, make sure you at least manually monitor the temperatures, so the system doesn't overheat.

The proper way to fix this is to replace all the DS1780E ICs with new ones (they're still being made). However, in an Origin, each brick has multiple of these (e.g. a C-brick has three on the motherboard plus two on each PIMM, an I-brick has three on the removable power section), so in a larger system, replacing about 65 surface-mount ICs is an incredibly tedious task.

Nevertheless, I have managed to replace all the faulty DS1780E ICs in my Origin (around 50 of them!) and have re-enabled environmental monitoring ("* env on" and "* env temp on"). I left the working DS1780E ICs alone, as it will be interesting to see if they also develop defects with time or not.

"No L1 comm!" error ^

One of the C-bricks in my Origin 3400 will always report this error on its L1 controller's LCD when I first connect it to power. However, after I toggle the power switch on the back of the affected C-brick, it will happily work as if nothing is wrong!

This is a very weird error that I'm not completely sure how to fix and there's nothing about it on the internet (if anyone has any information on this, please let me know).

My best guess is that some part of the reset circuitry for the L1 controller isn't working as it should. I plan to fix this by hacking-in a power-on-reset controller over the L1 reset button at the back of the C-brick — this should monitor the voltage and keep the reset line active until the voltage stabilizes, which will most likely fix the issue.

"enet_ssram: ssram address walk error" ^

This error happens sometimes during boot after powering-on the Origin rack after it has been off for a while. It goes away after doing an "enableall" and "update", then "reset" in the PROM command monitor.

It doesn't appear if the system is put in stand-by mode when not in use instead of turning it completely off or if the system is left to warm-up in stand-by mode for a few hours. Not surprising, as these systems were really meant to be on all the time.

Replacing faulty RAM ^

I haven't had a good DIMM go bad on its own yet, but I have bought bad RAM on eBay.

In case of bad RAM, you will get errors such as these during boot:

MAIN MEMORY ERROR AT: Location DIMMs/bank Addr Offset  subtest
(0x9600000080000000)   001c10  2 & 3/ 0   0x00000000   Base + Walk/Inv
	expected: 0xaa55aa55aa55aa55,0xaa55aa55aa55aa55,0xa55a
	  actual: 0xaa55aa55aa55aa55,0x5a55aa55aa55aa55,0xa55a
	    diff: ^x................,^xf...............,^x....

suspect devices = [Dimm 2 : A4B6]
Diagram of memory slots in an Origin 3000 C-brick

These errors are really useful, since they show you which chip on which DIMM is malfunctioning. If you're so inclined, you can desolder the broken chip and solder-in a replacement. However, the error might not always be directly caused by the given chip — e.g. a damaged decoupling capacitor or a scratched contact on the edge connector (common with DIMMs that were thrown around) can also cause issues that can be misattributed to a chip.

After printing multiple error messages like the one above, the power-on diagnostics will fail with:

DIAG RESULTS:
         /hw/module/001c10/node/mem: MEMBANK(S) 2 3 4 5  disabled
                 Reason:
                         Bank 2: Some DIMMs failed mem test.
                         Bank 3: Some DIMMs failed mem test.
                         Bank 4: Some DIMMs failed mem test.
                         Bank 5: Some DIMMs failed mem test.

In this case, you need to remove the faulty DIMMs, then in the PROM command monitor, do "enableall", then "update", and finally "reset".

If you don't perform these steps, the DIMM slots will remain disabled even if you put good RAM into them!

If you're unlucky and you put the bad RAM into DIMM slots 0 and 1, the system may fail to boot entirely. Put good RAM in the next two banks (2 and 3), do the "enableall+update+reset" steps, move good RAM back to slots 0 and 1, repeat the "enableall+update+reset" steps, and everything should be back to normal.

R-brick serial number mismatches ^

The Origin 3000 series router bricks are locked to the serial number of the system they're in. This means that if you move an R-brick to a system with a different serial number, it will refuse to work!

This serial number is stored in the DS1742W battery-backed NVRAM chip inside the R-brick.

The workaround for this is really simple — replace the DS1742W chip with one that has a flat battery, then the R-brick will always copy the serial number from the system it's in!

You can also replace it with a new chip, in which case it will copy the serial number from the system it's in only the first time, then it will remain locked to that serial.

Alternatively, if you have an old-enough L1 controller firmware on the R-brick, you can try the command "let the carnage begin", after which you can issue the command "serial clear", then after rebooting the L1 controller ("reboot_l1"), the R-brick should copy the system's serial from the L2 controller.

My Origin 3400 ^

My Origin 3400 has 32 processors and 32GB of RAM. It consists of eight C-bricks, two R-bricks, and one I-brick. The processors are mixed-speed — 24 of them are 500MHz R14000 and 8 are 400MHz R12000; both types of processors have 8MB of L2 cache. The root filesystem is on two 200GB SLC solid-state fibre-channel drives (Hitachi HUSSL4020ALF400) running in RAID-1 (mirrored) mode.

This system was put together from two separate 16-CPU Origin 3400 machines.

The first machine came from the UK, without the rack, as it was too heavy to ship — it weighed about 230kg (200kg for the bricks, the rest of the weight were cables). Power consumption for a 16-processor Origin 3400 is about 1320W when idle, about 1400W at full load, and 300W in stand-by mode (system off, but L1 and L2 controllers on). The power factor is an almost-perfect 0.99!

The second machine came from Croatia (thanks Harold!), where it was used at a large pharmaceutical company until it was decommissioned in 2013. This one came with the full rack and was in very nice condition!

Because of the modular nature of the Origin 3000 series, it was possible to merge the two 16-processor machines into a single 32-processor machine — I simply moved my old bricks into the shiny new rack and connected them together. I have no idea how much the final machine weighs, but it must be at least 500kg if not more (the rack is heavier than it looks, even when empty).

The power consumption of the 32-processor machine is around 2330W when idle, around 2400W at full load, and 450W in stand-by mode (system off, but L1 and L2 controllers on). The power factor when the system is on is a respectable 0.96, and a still-good 0.83 in stand-by mode.

Output of hinv

Processor 0: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 1: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 2: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 3: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 4: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 5: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 6: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 7: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 8: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 9: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 10: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 11: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 12: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 13: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 14: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 15: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 16: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 17: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 18: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 19: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 20: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 21: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 22: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 23: 500 MHZ IP35
CPU: MIPS R14000 Processor Chip Revision: 1.4
FPU: MIPS R14010 Floating Point Chip Revision: 1.4
Processor 24: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 25: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 26: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 27: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 28: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 29: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 30: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Processor 31: 400 MHZ IP35
CPU: MIPS R12000 Processor Chip Revision: 3.5
FPU: MIPS R12010 Floating Point Chip Revision: 3.5
Main memory size: 32768 Mbytes
Instruction cache size: 32 Kbytes
Data cache size: 32 Kbytes
Secondary unified instruction/data cache size: 8 Mbytes
Integral SCSI controller 4: Version SAS/SATA LS1068
Integral SCSI controller 0: Version Fibre Channel QL2200A
  Disk drive: unit 1 on SCSI controller 0
  Disk drive: unit 2 on SCSI controller 0
Integral SCSI controller 5: Version IEEE1394 SBP2
  IEEE1394 CDROM: node 1010031001c252 port 0 on SCSI controller 5
IOC3/IOC4 serial port: tty4
IOC3/IOC4 serial port: tty5
IOC3/IOC4 serial port: tty3
Gigabit Ethernet: eg0, module 001c07, pci_bus 2, pci_slot 1, firmware version 12.4.10
Integral Fast Ethernet: ef0, version 1, module 001c07, pci 4
IOC3/IOC4 external interrupts: 1
IEEE 1394 High performance serial bus controller 0: Type: OHCI, Version 0 0
USB controller: type OHCI

Output of scsimon

/dev/scsi/sc0d1l0: [HITACHI HUSSL402 CLAR200C165]       19C/66F [0C/32F]
/dev/scsi/sc0d2l0: [HITACHI HUSSL402 CLAR200C165]       19C/66F [0C/32F]

Pictures ^

SGI Origin 3400 (front view, door closed)
SGI Origin 3400 (front view, door open)
SGI Origin 3400 (side view)
Close-up of the SGI Origin 3400 logo
SGI Origin 3400 (back view, door closed)
SGI Origin 3400 (back view, door open, powered-on)
SGI Origin 3400 (back view, door open, without NUMAlink3 cabling)
Close-up of L1 display on a C-brick
Close-up of the L2 display
CPU activity LEDs on the back of C-bricks
NUMAlink3 cable connected to a C-brick
Removing a C-brick from the rack
Inside of a C-brick
Inside of a C-brick (close-up)
Inside of a C-brick (3/4 view)
Close-up of the Dual 400MHz CPU module
Close-up of the Dual 500MHz CPU module (front)
Close-up of the Dual 500MHz CPU module (back)
Close-up of the CPU module connectors
Close-up of the L1 controller inside a C-brick
Top view of C-brick board
Bottom view of C-brick board
Close-up of L1 controller on the C-brick board
Close-up of the pesky DS1780E chips on the C-brick board
Close-up of an official fix on the C-brick motherboard
Close-up of the power connector on the C-brick motherboard
Close-up of the NUMAlink3 connector on the C-brick motherboard
1GB RAM DIMM (front)
1GB RAM DIMM (back)
I-brick (back view)
Close-up of the I-brick ports
Inside of an I-brick
Inside of an I-brick (close-up)
I-brick CD-ROM adapter (Firewire to IDE)
I-brick CD-ROM
Original I-brick battery (left) and better replacement (right)
I-brick Power Board and Fan module
I-brick Power Board (front)
I-brick Power Board (back)
R-brick (back view)
Inside of an R-brick
Close-up of the router board inside the R-brick
Close-up of the L1 board inside the R-brick
Close-up of the ICs on the L1 board inside the R-brick
Close-up of the DS1780 on the L1 board inside the R-brick
Power bay (front view)
Power bay (back view)
Power bay cabling
Power bay specifications
Ports on the back of the L2 controller
Inside the L2 controller
Power section of the L2 controller
Logic section of the L2 controller
CR 1/2AA 3V lithium battery for L2 controller
SGI Dual Serial PCI card (front)
SGI Dual Serial PCI card (back)
SGI Dual Serial PCI card (ports)
LSI Logic LSI22320-R Ultra320 SCSI card
LSI Logic SAS3800X SAS/SATA card
Thermal view (front)
Thermal view (side)
Thermal view of upper C-bricks (front)
Thermal view of upper C-bricks (back)
Thermal view of the inside of a C-brick

More pictures coming soon!

Documents ^

Acknowledgements ^

All diagrams/drawings on this page were taken from SGI manuals and other documentation.

 

[ View other machines in my hardware collection ]