Sunday, January 15, 2012

Recover files from a VMFS virtual volume inside a iSCSI virtual volume


The problem:
Some time ago, I had a real bad RAID failure. Since I use my VMWARE ESXi hardware just to boot from a pen drive and connect to my QNAP NAS via iSCSI, a raid failure means the RAID degrades and enters read-only mode. Linux Virtual Machines keep working with errors on some packages, but windows virtual machines freeze in a couple of seconds.

The Solution:
Since I'm not rich and buying a full NAS system to copy from the read-only to the new one is out of question, I brought 2 2tb USB drives, connected them to my second NAS and moved the files out to the drives making room to move files from one NAS to another.
Now until this part it's all very simple; grab NAS02, make a desktop terminal connect into both iSCSI target on NAS01 and NFS share on NAS02 and move files from one to the other...and you start by installing open-iscsi.
HOWEVER, vmware hypervisors use a specially designed file system called VMFS, so mounting your iSCSI volumes will get you nowhere.
That's when vmfs tools get into action. Install the vmfs tools package and then all you have to do is mount the drives. It's a crazy opp, but it works. Basically, You'll be mounting the virtual iSCSI volume into a virtual directory, then the VMFS virtual volume that the iSCSI virtual volume contains into another virtual directory.
Better have Gigabit Ethernet for the next part: After this mount the NFS from the second NAS, and copy everything...for a couple of days, depending on your network speed :S

Sources I used to study:



How did I mount the iSCSI, then then VMFS virtual volumes in my linux desktop? here goes the command LOG:
administrator@RWS01:~$ su
Password:"SECRET"

root@RWS01:/home/administrator# /etc/init.d/open-iscsi start
 * Setting up iSCSI targets                                              [ OK ]

root@RWS01:/home/administrator# mount 10.0.101.246:/RECOVER /iscsi/rec

root@RWS01:/home/administrator# /etc/init.d/open-iscsi restart
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Starting iSCSI initiator service iscsid                               [ OK ]
 * Setting up iSCSI targets                                              [ OK ]

root@RWS01:/home/administrator# iscsiadm --mode discovery --type sendtargets --portal 10.0.101.247
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
iscsiadm: config file line 42 do not has value
iscsiadm: config file line 43 do not has value
iscsiadm: config file line 56 do not has value
iscsiadm: config file line 57 do not has value
10.0.101.247:3260,1 iqn.2004-04.com.qnap:ts-639:iscsi.vmwareisdata.bd99c2
10.0.101.247:3260,1 iqn.2004-04.com.qnap:ts-639:iscsi.vmwareboots.bd99c2

root@RWS01:/home/administrator# /etc/init.d/open-iscsi restart
 * Disconnecting iSCSI targets                                           [ OK ]
 * Stopping iSCSI initiator service                                      [ OK ]
 * Starting iSCSI initiator service iscsid                               [ OK ]
 * Setting up iSCSI targets                                              [ OK ]

root@RWS01:/home/administrator# tail -f /var/log/messages
Nov 23 01:44:01 RWS01 kernel: [  391.203611] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.203638] sd 22:0:0:0: [sdc] READ CAPACITY failed
Nov 23 01:44:01 RWS01 kernel: [  391.203640] sd 22:0:0:0: [sdc] Result: hostbyte=DID_OK driverbyte=DRIVER_OK
Nov 23 01:44:01 RWS01 kernel: [  391.203643] sd 22:0:0:0: [sdc] Sense not available.
Nov 23 01:44:01 RWS01 kernel: [  391.206527] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.207253] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.208968] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.209013] sd 22:0:0:0: [sdc] Test WP failed, assume Write Enabled
Nov 23 01:44:01 RWS01 kernel: [  391.211881] sd 22:0:0:0: reservation conflict
Nov 23 01:44:01 RWS01 kernel: [  391.211903] sd 22:0:0:0: [sdc] Attached SCSI disk
tail -f /var/log/messages
^C

root@RWS01:/home/administrator# mkdir /iscsimount

root@RWS01:/home/administrator# mount /dev/sdb1 /iscsimount
mount: you must specify the filesystem type

root@RWS01:/home/administrator# sudo vmfs-fuse /dev/sdb1 /iscsi/4

root@RWS01:/home/administrator# sudo ls /iscsi/4 -alh
total 4.0K
drwxr-xr-t 9 root root 2.0K 2010-06-05 21:06 .
drwxr-xr-x 9 root root 4.0K 2010-11-22 20:18 ..
-r-------- 1 root root 2.5M 2010-02-28 18:44 .fbb.sf
-r-------- 1 root root  61M 2010-02-28 18:44 .fdc.sf
-r-------- 1 root root 244M 2010-02-28 18:44 .pbc.sf
-r-------- 1 root root 249M 2010-02-28 18:44 .sbc.sf
-r-------- 1 root root 4.0M 2010-02-28 18:44 .vh.sf
drwxr-xr-x 2 root root  560 2010-11-14 18:19 VNAS01_BackupServer_UNX
drwxr-xr-x 2 root root 2.5K 2010-11-15 19:18 VSRV06_DomainServer_2k8
drwxr-xr-x 2 root root 6.6K 2010-11-15 19:15 VSRV07_LTSDomainServer_2k8
drwxr-xr-x 2 root root 3.1K 2010-10-19 01:31 VSRV07_Sharepoint_2k8
drwxr-xr-x 2 root root 2.8K 2010-11-15 19:07 VSRV08_WebServer_2k8_R2_64
drwxr-xr-x 2 root root 2.5K 2010-08-07 03:20 VWKS03_Private_XP32
drwxr-xr-x 2 root root 3.5K 2010-08-23 16:45 VWKS04_Testbench1_XP32

root@RWS01:/home/administrator# sudo vmfs-fuse /dev/sdd1 /iscsi/5

root@RWS01:/home/administrator# sudo ls /iscsi/5 -alh
total 4.0K
drwxr-xr-t 11 root root 2.2K 2010-03-06 04:29 .
drwxr-xr-x  9 root root 4.0K 2010-11-22 20:18 ..
-r--------  1 root root 2.5M 2010-02-28 04:20 .fbb.sf
-r--------  1 root root  61M 2010-02-28 04:20 .fdc.sf
-r--------  1 root root 244M 2010-02-28 04:20 .pbc.sf
-r--------  1 root root 249M 2010-02-28 04:20 .sbc.sf
-r--------  1 root root 4.0M 2010-02-28 04:20 .vh.sf
drwxr-xr-x  2 root root 2.6K 2010-11-15 19:22 VRV01_DEVServer_2k3
drwxr-xr-x  2 root root  280 2010-02-28 16:32 VSRV01_Development_2k3
drwxr-xr-x  2 root root 2.9K 2010-11-15 19:22 VSRV02_LNX_VPN_WEB_DB
drwxr-xr-x  2 root root  280 2010-02-28 16:16 VSRV02_VPN_MYSQL_APACHE
drwxr-xr-x  2 root root 2.8K 2010-11-15 19:16 VSRV03_DBServer_2k3
drwxr-xr-x  2 root root 3.2K 2010-11-15 19:14 VSRV04_WEBServer_2k8
drwxr-xr-x  2 root root 2.6K 2010-11-15 19:22 VSRV05_DBServer_LNX
drwxr-xr-x  2 root root 2.8K 2010-08-07 03:22 VWKS01_Downloader_XP32
drwxr-xr-x  2 root root 2.5K 2010-11-15 19:22 VWKS02_Monitor

root@RWS01:/home/administrator#
THAT's COPY TIME NOW