Half a Petabyte in 4U

OK, so first up, I flat up lied; it’s only 480TB. But that’s how storage marketing goes, right?

I just caught mention of Dell’s MD3060e Dense Enclosure, an insane 4U storage enclosure that accommodates 60x 3.5″ hard drives. (Five trays of 12 disks each.)

Screen Shot 2015-03-16 at 11.39.02 PM

Here’s one on eBay, with 60x 4TB disks for a mere $38,999.

I also learned in the past week about shingled magnetic recording, a new technology for spinning magnetic disks that allows much greater density, at the cost of some stranger semantics on how data is written. It’s how Seagate already has an 8TB drive on the market, at an oddly low price.

I’ve also been playing around with ZFS (on Linux) lately. I’m building a file server with 3x 6TB drives (RAIDz1, akin to RAID 5), and 2x 512GB SSDs for cache/ZIL. I have a lot of kinks to work out still, but the idea is that ‘popular’ content will get auto-cached on the SSDs, backed by magnetic storage. A modest bit of SSD space is set aside for ZIL, where writes forcing a sync can be written to SSD and then later written to spinning disks. I suspect that the “SSD as cache in front of slower magnetic storage” paradigm will grow in popularity until SSDs become cheap enough that magnetic storage is entirely obsolete. Things like shingled drives would seem to make that even more important.

I feel like this would be a really exciting combination to play around with, if only I had the money and the need for almost a half-petabyte of storage.

IPMI trick: Set the boot device

I’ve long known that you can use IPMI (on compatible systems) for some nice lights-out management, like power control or reading system information that you’d see in the BIOS. But one thing that has always annoyed me is having to get a remote console on the box and wait for just the right second to press the right key to bring up the screen where I can change boot order.

Lo and behold, this is an IPMI feature! It’s implemented at least on some IBM servers I’m working with:

mawagner ~ $ ipmitool -H 172.16.10.1 -U USERID chassis bootdev
Password:
bootdev <device> [clear-cmos=yes|no]
bootdev <device> [options=help,...]
none : Do not change boot device order
pxe : Force PXE boot
disk : Force boot from default Hard-drive
safe : Force boot from default Hard-drive, request Safe Mode
diag : Force boot from Diagnostic Partition
cdrom : Force boot from CD/DVD
bios : Force boot into BIOS Setup
floppy: Force boot from Floppy/primary removable media
mawagner ~ $ ipmitool -H 172.16.10.1 -U USERID chassis bootdev cdrom
Password:
Set Boot Device to cdrom

It appears that this will override settings on the next boot, but not make a persistent change. (You can, however, set the ‘persistent’ option.) Using bootdev none options=help lists out the available options, some of which are pretty interesting.

Cisco WS-C2948G-GE-TX Review

I picked up a Cisco WS-C2948G-GE-TX switch on eBay a while back. I’m about a decade late to the party in reviewing one (mine’s dated November 3, 2003!), but they’re showing up in quantity on eBay for short money, so I thought a modern look at the thing might be helpful.

WS-C2948G-GE-TX (on top of a Dell C6100)

So, the executive summary is this:

Pros:

  • 48-port Gigabit switch
  • Managed
  • 4x SFP ports
  • Runs about $100 on eBay

Of course, a 48-port Gigabit managed Cisco for $100 seems must have some catch. Read on!

Cons:

  • Past End of Life
  • CatOS support only (no upgrade to IOS)
  • Oversubscribed switching fabric (12 Gbps)
  • SFPs have no 10Gb support
  • Appears to support SSH v1.0 only?

If you’re looking for a switch for a serious production environment, this isn’t for you. (But why are you even looking at sub-$100 switches on eBay if that’s the case?!) But for certain cases, this is seemingly the perfect switch. I’m building out a small home lab. I want Gigabit ports (and more than a few of them) but I’m not actually moving a ton of traffic through the switch, so the fact that the switch couldn’t support all 48 ports actually pushing 1Gbps simultaneously is of no concern to me. The CatOS bit isn’t great, but I’m not trying to go for my CCIE where I need all the latest features IOS brings, or running this in a production environment where having the latest security patches is critical. (Not that security is ever unimportant, but it’s on a private LAN in my basement so I’m not going to lose sleep if it’s got known vulnerabilities.) Realistically the only “managed” feature I have used is setting up a second VLAN for a few ports.

For the curious, here’s what mine came running:

sw01> (enable) sho ver

WARNING: This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and use.
Delivery of Cisco cryptographic products does not imply third-party authority
to import, export, distribute or use encryption. Importers, exporters,
distributors and users are responsible for compliance with U.S. and local
country laws. By using this product you agree to comply with applicable
laws and regulations. If you are unable to comply with U.S. and local laws,
return this product immediately.

WS-C2948G-GE-TX Software, Version NmpSW: 8.4(11)GLX
Copyright (c) 1995-2006 by Cisco Systems, Inc.
NMP S/W compiled on Apr 27 2006, 12:46:44
GSP S/W compiled on Apr 27 2006, 11:47:52

System Bootstrap Version: 6.1(6)

Hardware Version: 1.1  Model: WS-C2948G-GE-TX  Serial #: [redacted]

Mod Port Model              Serial #              Versions
--- ---- ------------------ -------------------- -------------------------------
1   0    WS-X2948G-GE-TX    [redacted]           Hw : 1.1
                                                 Gsp: 8.4(11.0)
                                                 Nmp: 8.4(11)GLX
2   52   WS-C2948G-GE-TX    [redacted]           Hw : 1.1

       DRAM                    FLASH                   NVRAM
Module Total   Used    Free    Total   Used    Free    Total Used  Free
------ ------- ------- ------- ------- ------- ------- ----- ----- -----
1       65536K  36933K  28603K  16384K  10779K   5605K  480K  320K  160K

Uptime is 94 days, 1 hour, 25 minutes

And then, the modules:

Mod Slot Ports Module-Type               Model               Sub Status
--- ---- ----- ------------------------- ------------------- --- --------
1   1    0     Switching Supervisor      WS-X2948G-GE-TX     no  ok
2   1    52    10/100/1000 Ethernet      WS-C2948G-GE-TX     no  ok

Mod Module-Name          Serial-Num
--- -------------------- --------------------
1                        [redacted]
2                        [redacted]

Mod MAC-Address(es)                        Hw     Fw         Sw
--- -------------------------------------- ------ ---------- -----------------
1   00-0d-bd-b1-XX-00 to 00-0d-bd-b1-XX-3f 1.1    6.1(6)     8.4(11)GLX
2   00-0e-bd-b1-XX-de to 00-0e-bd-b1-XX-3d 1.1

Note that I haven’t tried anything fancy. In theory it supports stuff like CDP, tagged VLANs, 802.1x, LACP, RADIUS, port mirroring, SNMP monitoring, STP, and so on, but I haven’t set any of that up, and frankly wouldn’t know how.

Back of switch

All in all, if you’re looking for a cheap Gigabit switch with some basic management features and don’t mind it running old software and having a heavily-oversubscribed backplane, I’ve found this switch to be a steal.