Log in

No account? Create an account
entries friends calendar profile Previous Previous Next Next
Changes - Ed's journal
I've come to realise the thing I actually like about my job, is stuff I can really get my teeth into. Fiddle-faddle helpdesk calls, from users with trivial issues, logged by people who don't speak English, and therefore can't write a coherent fault report, I find immensely frustrating.

Not that I can't solve the problem, as much as it's immensely frustrating to jump through hoops to get anywhere near being able to troubleshoot. This is, of course, leaving aside the part where things can't be solved, simply because I don't have the right tools or access to do so, which is depressingly common.

But yesterday and today was a config change, on a symmetrix. The good part about doing this, is it's possible to 'almost completely' pre-script it. You can define a set of changes, explicitly, and 'preview' run them on your array, to validate them.
Then, once you've put your 'change request' paperwork in, actually _doing_ the change is very straightforward.

The reason I like doing this, is because these changes are actually fairly complex - in order to get storage onto a server, you need to:

Define the LUN (logical unit. Think 'partition' and you're not far off) size, RAID type (different RAID levels are more or less appropriate for different 'types' of workload) and number, and create them. We mostly use RAID5 (7+1) or RAID 1+0. The former for 'most stuff' and the latter for things like server journal files for performance reasons. Specific workload performance actually varies quite a bit by RAID type, so the 'randomness' of your I/O can be very relevant. A database journal file, for example, is pretty much only written to, and it's written to sequentially. The actual database however, might well only being read from, and read from in a highly random fashion which means you need to be able to fetch data from arbitary points on disk on demand, and you can't really prefetch much data.

Merge the LUNs into metadevices, of the correct size (a single LUN can only be up to 30Gb in size, so you have to stick them together to get bigger disks). Meta layout also is affected a little by data profile, but mostly you're trying to get the maximum number of spindles. (More physical disks involved, means better overall data transfer)

Map the metadevices to a front end director (FA) picking one such that you're optimizing system workload and layout. Not necessarily just distributing mean load, but more matching daily workload with light load time periods - some of our servers run (much) harder overnight, some run harder during the day, so the average isn't always a good indicator.

Zone the host bus adaptor on the host, with the FA you've allocated the LUNs to. This is both for performance, and access control reasons - if you've a 1-1 mapping from HBA to FA, then you don't get as much 'noise' where if you have a many to one, or many to many, you get an overhead of fiber (yes, that is the correct spelling of fiber) channel traffic. However it's also for access control reasons - servers aren't generally 'well behaved' about new disks, and will just 'grab' them. If you don't actually touch them, at all, then you're fine, but windows in particular loves to declare 'found new hardware, would you like to write a signature'. (this is bad when it's a disk allocated to another server).

Each host has two HBAs, because we want no 'single points of failure'. They're connected by fibre to the switches, but ... unfortunately fibres are fragile, and go ping, quite easily. So you use two, so that happens less. And of course, avoid 'errors' by having two, entirely independant data paths.

Then you get to mask the devices to the servers - each FA will typically have multiple servers 'talking' to it, so you provide another layer of access control, such that only one server can access a given device. Sometimes you want multiple servers accessing the same device, but that's really only in some very specific instances (e.g. clusters) in general, it leads to badness. I did try this once, and you get some _really_ interesting behaviour if two windows servers try and do stuff, concurrently, on a single volume. By 'interesting' I mean unpredictable and will just totally screw over any data you're trying to put on there, as one server caches and overwrites what the other is doing.

And last but by no means least, configure WWN aliases. So that 16 char hexidecimal number that's the HBA WWN (Host bus adaptor, world wide name - bit like a MAC address on a network card) actaully has a name associated with it. Working with 50 servers, each with two WWNs, and just working with long sequences of long hexidecimal numbers, is just ... ugh, bad for your brain.

All in all, I think I like the config and implementation part of my job, way more than the 'dealing with Networker and users' part.

But mostly, having a system that I know well enough that I can prepare what I'm going to do, in detail, the day before (I did spend about a day of prep on it), without really having to worry about some random gotchas that screw it all up, is what I really like about working in storage. Unix is somewhat similar, and Windows ... just isn't.


1 comment or Leave a comment
ash1977law From: ash1977law Date: November 9th, 2007 02:03 pm (UTC) (Link)
I just lost 1d4 SAN
1 comment or Leave a comment