?

Log in

No account? Create an account
entries friends calendar profile Previous Previous Next Next
Online volume expansion, SRDF Environment - Ed's journal
sobrique
sobrique
Online volume expansion, SRDF Environment



How to online volume expand.

Conventions:
To avoid confusion, symconfigure commands look like:


create dev count=5, size=21846, config= 2-Way-BCV-Mir, emulation=FBA;




Command line commands.
C:\> diskpart
Most of them start ‘sym...’ but there’s one or two (well, ok, there’s the diskpart commands) that don’t.
Before you start:
This takes time. Quite a lot of time. If you’re looking to do it all in one night, then you will be spending all night doing it. Ideally you’ll be planning on doing in 4 separate stages, as that minimises the amount of time spent watching logs.

Overview:
Stage 1
1.Create devices for expansion.
2.Create BCVs to protect the expansion
3.Establish BCVs to disks to be expanded. (Takes time, expect 20Mb/s)
Stage 2
4.Stop SRDF and delete RDF pairing.
5.Split the BCVs.
6.Expand primary device (R1) ‘protected’, using the BCV created in 2. (Takes time, expect 20Mb/sec, twice)
7.Expand secondary device (R2) ‘unprotected’ (this leaves the BCV created in 2 as a backup copy)
Stage 3
8.Extend the volume on the host using diskpart
9.Re-establish and resume SRDF in adaptive copy as you’ll have to do a full copy. (takes time)
Stage 4
10.Switch SRDF back to Sync.
11.Clean up – delete BCV devices.

This is non disruptive if it all works. If anything does go horribly wrong, it’ll be during Stage 2, and therefore Stage 2 should definitely be done out of hours. The others … well, it depends what you can blag really. Steps 9 and 10 are best done ‘as soon as possible’ in preference to waiting until an outage window, as you’re not really saving any headaches leaving SRDF switched off for longer than necessary.

Before starting:
Check the device to be expanded. It needs to be a meta device already. You can turn a STD into a meta, but in order to do this it needs to be made ‘not ready’ whilst the operation completes.
You will need to know: Meta component size in cylinders. Number of meta components. Disk group of the device on both sides of the SRDF link. (It is often the same, but not always).

Check the backup – you’re messing with the live volume, a good backup is recommended.

Check the RDF device group this device is us (symdev show will tell you)

You will also want to check if this device has BCVs already for backup reasons. If it does, you will need to also increase the size of the BCVs, but you’ll also be able to use the existing BCVs for the expansion.

Create devices for expansion:

Decide amount to expand by - must be same component size as existing volume.
Use symdev show on your existing device if unsure.

Create new devices to expand onto accordingly – don’t forget to check they’re in the right disk group too. Technically they don’t need to be in the same disk group, but it’s preferable that they are.
This doesn’t need to be put into a meta, but does need doing on both sides of the SRDF link. If you use two different ECC servers, you can do this concurrently, as you only need config lock on the one Symmetrix (you cannot run both at once, because the SYMAPI database blocks)

Also, create devices in a BCV configuration, the same track size as the existing volume - this MUST be 'BCV' or 'BCV-2-way-mir', not BCV-R-5. Doesn’t explicitly need to be exactly the same geometry as the primary device, but it doesn’t hurt. Also BCVs shouldn’t use slower disk than the primary device, because that’ll slow both down.

Be careful running the below – it may not create devices in the order you think. Actually, it probably won’t, so you may want to run these separately.

create dev count=5, size=21846, config=RAID-5, emulation=FBA, disk_group=2;
create dev count=5, size=21846, config= 2-Way-BCV-Mir, emulation=FBA;


Then make the BCV into a meta. Note that it’s only coincidence that we’re using the same device count here – the BCV needs to match the existing device, where the expansion LUNs can be any number.


form meta from dev XXX config=striped, stripe_size=1 cyl;
add dev YYY:ZZZ to meta XXX;

This needs doing on both sides of the SRDF link.

Taking the BCV copy for expansion

Take a BCV copy of the R2 device, as a 'in case of emergencies' device.
Add both BCVs, because that makes monitoring progress easier.

symdg create temp_expand –type RDF1
symld –g temp_expand add rimary dev
symbcv –g temp_expand add bcv_dev
symbcv –rdf –g temp_expand add bcv_dev


Establish the remote site BCV to the R2 device.

symmir –g temp_expand –rdf establish DEV001 BCV LD RBCV001 -full

Wait for this to finish, and maybe take note – this gives you a ballpark figure of how long the actual operation is going to take (it has to do this twice, so assume at least twice this amount of time). If you’re so inclined, you can establish the R1 BCV as well. It won’t hurt, and it may speed up the later migration.



Stop/remove RDF pairing and replication.

First off, we need to split the BCVs. The remote BCV will be left untouched throughout this process to give us a ‘known good’ copy of the data.
Assuming you used the ‘temp’ device group above:

symmir –g temp_expand split
symmir –g temp_expand –rdf split

Symmetrixes are clever, but they’re not quite clever enough to cope with changing the device size, and thus the relationship of tracks on both sides of a SRDF link. Therefore we have to stop SRDF replication, and delete the pairing.
Hopefully your SRDF device will already be in an RDF group.

symrdf –g BANANA_3 split
symrdf -f RDF_Pairs.txt -sid SymmID -rdfg 3 deletepair –force

where RDF_Pairs.txt looks like:
1535 1535

Swap to appropriate device numbers in RDF_Pairs.txt


Extend the devices:

We expand the primary (R1) device with data protection, and use the BCV we created as the ‘protection’ device. On the remote device, we do an unprotected expansion – it’ll be out of sync already, and by doing this we preserve our copy of the data on the BCV.


add dev 0233:0334 to meta 1535, protect_data=TRUE, bcv_meta_head=03bd;

And on the R2 side:

add dev 0233:0334 to meta 1535, protect_data=TRUE, bcv_meta_head=03bd;

This operation is time consuming, so expect to leave it running overnight. It has to synchronise the primary volume twice.

Extend the partition on the server
Technically, this is something that’s ‘server team’ responsibility. In practice, it’s much more useful that we do it, both to avoid getting someone in the server team to spend 30 seconds running some commands, and because it provides us verification that the change has actually worked.

You need to use diskpart, which uses the ‘Virtual Disk’ service. You can do a rescan from ‘disk management’ but you can also do it from diskpart:

C:\> diskpart
DISKPART> rescan

Please wait while DiskPart scans your configuration...

DiskPart has finished scanning your configuration.
DISKPART> select disk 1
Disk 1 is now the selected disk.
DISKPART> select partition 1
Partition 1 is now the selected partition.
DISKPART> extend
DiskPart successfully extended the volume.
DISKPART>


Resume RDF
Turn back on SRDF – you’ll have to treat this as a ‘new’ device pairing, because you’ve made substantial alterations to the geometry of the volume, and so the invalid track monitoring mechanism just won’t work.

First off we need to enable dynamic RDF.

set dev 1535 attribute=dyn_rdf;

Then we need to recreate the Symmetrix device pairing. If you’ve been running commands on two separate ECC servers, you’ll need to rediscover the configuration, otherwise it’ll fail. It’s also necessary to write disable the remote device, because an SRDF R2 cannot be writable.

symcfg discover
symdev -sid SymmID write_disable 1535

symrdf createpair -f RDF_Pairs.txt -sid SymmID -type RDF1 -rdfg 3 -invalidate r2

symrdf -g BANANA_3 set mode acp_disk DEV001
symrdf -g BANANA_3 establish -full DEV001


And here you get to wait again, as this will take a significant amount of time.
You can monitor with:

symrdf –g BANANA_3 query

Set RDF back to Synchronous, and tidy up BCVs.
Check that your replication has mostly caught up – use a ‘symrdf query’. If your invalid track count/MB is not too large a number, then you’re good to switch the device into synchronous mode.

symrdf -g BANANA_3 set mode sync DEV001

Then you need to tidy up the expansion BCVs it’s probably worth leaving this until you’re sure your change has worked – if you didn’t end up doing a partition expansion for whatever reason, or couldn’t verify the server could ‘see’ the increased disk space, then it’s worth hanging onto your BCV copies, especially the R2.

dissolve meta dev 03bd;
delete dev 03bd:03AF;


If you ended up with your BCV device numbers being lower than your STD/R1 device numbers then you may want to create some filler luns at this point. We like having contiguous LUNs created in subsequent changes, and a ‘gap’ of 5 gets filled in an odd fashion. (it will skip the gap entirely if it tries to create 6 LUNs at once, but not if you create 4)

create dev count=5, size=3, config=2-way-mir, emulation=FBA;

Having done all this, you should now have devices on both sides of your SRDF link extended and synchronised, and the larger LUN visible to the host, and the partition expanded to take advantage of the available space.
You’ll also have done this with no outage.
2 comments or Leave a comment
Comments
stgpcm From: stgpcm Date: December 11th, 2008 12:45 am (UTC) (Link)

doesn't this leave you exposed?

it seems you're without your RDF for some time after an expansion
sobrique From: sobrique Date: December 11th, 2008 06:31 pm (UTC) (Link)

Re: doesn't this leave you exposed?

It does mean you don't get RDF sync whilst the expansion is happening, or indeed until you've caught up the differential after completing it.
It therefore might be more acceptable to down the services whilst you're doing this.
But it's an either or choice - you can either have your service running 24/7, or your offsite replica synchronised with your primary at all times.
2 comments or Leave a comment