I recently setup cygwin on a Windows 2008 server, and when I tried to issue an rsh command, the rshd process was returning a NULL to the rsh client. It turned out that I had several copies of inetd running. Some I had started using cygrunsrv – and others using iu-config.

After killing all the copies of inetd, and starting a new one using cygrunsrv – everything was OK.

The key is to not run cygrunsrv, before setting up the cywgin “user” using iu-config – otherwise you end up with an inetd process owned by SYSTEM, and that – it seems is bad. When all is working, my ps -ef looks like this, where the user cyg_srv is the user setup during iu-config.

$ ps -ef

     UID     PID    PPID TTY     STIME COMMAND

Administ    3948       1 con  17:52:52 /usr/bin/bash

cyg_serv    3492       1   ?  17:55:22 /usr/bin/cygrunsrv

cyg_serv    3652    3492   ?  17:55:22 /usr/sbin/inetd

Administ    1308    3948 con  17:59:49 /usr/bin/ps

Tweet
            

            

Sometimes you might want to change the “type” of a disk within Solaris. The “type” of the disk is stored on the disk itself, along with the partition table (presumably in the VTOC in the first part of the disk). For the purpose of this post, I use disk “name” and “type” interchangeably.

In the past, the name of the disk would be displayed in format, something like “SUN 10G blah blah” but sometimes the disk name is just “DEFAULT” which can be annoying. It turns out that it is quite simple to change the name of the disk, IF you happen to know the correct sequence.

So, in this case, I want to change the name of disk 1, which currently has the name “DEFAULT” followed by some details of the physical layout.

Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c3d0 DEFAULT cyl 4092 alt 2 hd 128 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c3d1 DEFAULT cyl 141 alt 2 hd 64 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0

The first thing I am going to do is to create a file which contains the geometry of this disk. In the case of x86 accessing the disk via a BIOS, the “disk” is actually just the partition which is given over to Solaris, but in principle we think of it as “the disk”

We will use the format “save” option from within the format/disk mentu to generate the file. The file is called format.dat by default, and this will be a freshly created file, similar in syntax to /etc/format.dat, but only containing information about one disk (the one which we want to change).

Specify disk (enter its number): 1
selecting c3d1
Controller working list found
[disk formatted, defect list found]

FORMAT MENU:
        disk       - select a disk
        type       - select (define) a disk type
        partition  - select (define) a partition table
        current    - describe the current disk
        format     - format and analyze the disk
        fdisk      - run the fdisk program
        repair     - repair a defective sector
        show       - translate a disk address
        label      - write label to the disk
        analyze    - surface analysis
        defect     - defect list management
        backup     - search for backup labels
        verify     - read and display labels
        save       - save new disk/partition definitions
        volname    - set 8-character volume name
        !     - execute , then return
        quit
format> save
Saving new disk and partition definitions
Enter file name["./format.dat"]:
format> q

Next we will edit that file, and change the name to whatever it is we want to use. The original content of format.dat

gjl@vbox-sol1:/a# cat format.dat
#
# New disk/partition type  saved on Thu Jan  7 11:14:21 2010
#
disk_type = "DEFAULT" \
         : ctlr = ata : ncyl = 141 : acyl = 2 : pcyl = 143 \
         : nhead = 64 : nsect = 32 : rpm = 3600

partition = "original" \
         : disk = "DEFAULT" : ctlr = ata \
         : 2 =  0, 288768 : 8 =  0, 2048 : 9 =  1, 4096
 

and now the edited version, I have hust change the word “DEFAULT” to “SOLARIS_DATA_VDI”

gjl@vbox-sol1:/a# cat format.dat
#
# New disk/partition type  saved on Thu Jan  7 11:14:21 2010
#
disk_type = "SOLARIS_DATA_VDI" \
         : ctlr = ata : ncyl = 141 : acyl = 2 : pcyl = 143 \
         : nhead = 64 : nsect = 32 : rpm = 3600

partition = "original" \
         : disk = "SOLARIS_DATA_VDI": ctlr = ata \
         : 2 =  0, 288768 : 8 =  0, 2048 : 9 =  1, 4096

So, now I want to use this file to change the values stored on the disk, which is used by format to identify the disk. To do that I will be using the -x option to format, and point to my just edited copy of format.dat

gjl@vbox-sol1:/a# format -x ./format.dat
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c3d0 DEFAULT cyl 4092 alt 2 hd 128 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c3d1 DEFAULT cyl 141 alt 2 hd 64 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0

Notice that nothing has changed in the output of format…. yet. Now I will tell format that I want to change the disk type, and the disk type “SOLARIS_DATA_VDI” will now appear in the menu. It has all the same values as originally, except that I have changed the name.

First I select the disk I am going to change (disk 1)

Specify disk (enter its number): 1
selecting c3d1
Controller working list found
[disk formatted, defect list found]

Then I select “type” to change the type and my new disk type appears as a selectable option.

format> type

AVAILABLE DRIVE TYPES:
        0. SOLARIS_DATA_VDI
        1. DEFAULT
        2. DEFAULT

I select Option ’0′

Specify disk type (enter its number)[2]: 0
selecting c3d1
No current partition list
No defect list found
[disk formatted, no defect list found]

and finally I label the disk.

format> la
Ready to label disk, continue? y

format> q
gjl@vbox-sol1:/a#

Now, when I run format again, I see that the disk type/name has changed, even though I am no longer using my special “format.dat”. Which is because the name is stored on the disk itself.

gjl@vbox-sol1:/a# format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c3d0 DEFAULT cyl 4092 alt 2 hd 128 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c3d1 SOLARIS_DATA_VDI cyl 141 alt 2 hd 64 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0

Of course, a simpler way to mark the disk would be just to add a “Volume name” to the disk. I specifically wanted to get rid of the disk type “DEFAULT” from my disk list, so changing the volume name alone was not enough. In the format output below, I create the volume name “VDISK” which appears in addition to the name “SOLARIS_DATA_VDI” which is actually the ASCII disk name/type”

format> vol
Enter 8-character volume name (remember quotes)[""]:"VDISK"
Ready to label disk, continue? y

format> di

AVAILABLE DISK SELECTIONS:
       0. c3d0 DEFAULT cyl 4092 alt 2 hd 128 sec 32
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@0,0
       1. c3d1 SOLARIS_DATA_VDI cyl 141 alt 2 hd 64 sec 32  VDISK
          /pci@0,0/pci-ide@1,1/ide@0/cmdk@1,0

Not sure how well known this trick is, but it is possible to figure out the vendor of a particular NIC by looking at the first few digits of the MAC address. We used to use this when I worked at Sun to figure out if the customers were using pukka Sun NICs and what might be at the other end. In the modern world there is a web tool that will do the lookup for you. Mac Find

This site has a whole boat load of one-liner unix commands. You can upload and contribute your own

My command line fu
All command line Fu

The script uses the Digest::md5 module to look for duplicate files. It takes as input a list of filenames, and returns a list of all files which share the same md5 hash/signature. Files which share the same md5 signature are very likely to be duplicate files.

lovebox:~ gjl$ cp /etc/hosts file1
lovebox:~ gjl$ cp /etc/hosts file2
lovebox:~ gjl$ dupes file1 file2
cbe7e7eb6480e869bccfa284dc8bd732 : file1 file2

This usage is almost identical to using the standard unix utility cksum, except that the md5 cheksum is less likely to give the same ‘checksum’ for a file which is actually different.

lovebox:~ gjl$ cksum file1 file2
85078130 236 file1
85078130 236 file2

For a 100m file created using ‘dd if=/dev/random’ the cost in time of the two checksum methods is about the same, even though one would exptect that the md5 checksum is more computationally expensive. But the time(1) utility seems to show that the md5 script actually uses less CPU time than the unix cksum tool.

The wall time is greater than the sum of user+sys so there must be some wait event, even though the files are cached by the filesystem.

lovebox-2:~ gjl$ time cksum file1 file2
3512382378 104857600 file1
3512382378 104857600 file2

real	0m1.833s
user	0m1.660s
sys	0m0.144s

lovebox-2:~ gjl$ time dupes file1 file2
ed3b0c7dcfc7192186899a3c63854eb0 : file1 file2

real	0m1.181s
user	0m0.916s
sys	0m0.250s

Here’s a weird one that I’d never seen before. We were finding that at least one of our linux binaries was becoming corrupt (verified using rpm -V) however we didn’t know how the binary was becoming corrupt – because these files should never be written to. On the off-chance that the corruption might tell us something about the corruption I decided to dump the binaries out using ‘od -x’ the ‘octal dump’ command, and allowed me to use ‘diff’ to see where in the file the corruption was and to see what sort of data was in the place where the executable code of the ‘cc1′ binary ought to be.

After using ‘diff’ to find where the corruption was and seeing that the data was not just a bunch of 00′s or FF’s, I switched to ‘od -c’ to dump out the binary character by character. After peering for a while I noticed that there were some filenames embedded in the corrupted data. Interestingly the filenames were not printed out using ‘strings’ because each character was zero terminated (I guess they were stored as multibyte chars rather than single byte chars – and so did not appear in ‘strings’.

The point of interest is here :-
16175560 \f 003 u \0 s \0 e \0 r \0 d \0 i \0 f \0
16175600 f \0 . \0 L \0 O \0 G \0 \0 \0 \0 \0 \0 \0
16175620 \0 \0 \0 \0 \0 \0 \0 \0 020 \0 \0 \0 002 \0 \0 \0


If you look closely, there is what looks like a filename ‘userdiff.LOG’ It turns out that the file ‘userdiff.log’ is a Windows file!

So, how does a windows filename end up embedded inside our linux binary? Here’s what I think happened. This client is a multi-purpose lab client sometimes booted into Windows, sometimes into Linux. The client is SAN booted from a NetApp LUN.

But surely when the Linux LUN is swapped in, the writes immediately stop because the Windows LUN is ripped away? No! the Windows SCSI driver (like all SCSI drivers) just puts the blocks on the SCSI bus with a destination address. There will probably be a SCSI error, such as SCSI_TIMEOUT whilst the new LUN is swapped in, however the SCSI driver WILL RETRY when the new LUN is swapped in, and then successfully writes to the new LUN after one or more retries.

Recently we had an issue on some linux clients that would mysteriously fail to run SFS benchmarks. On closer inspection we found that the binary ‘cpp’ would exit with SIGSEGV.
# cpp
cc1: internal compiler error: Segmentation fault
Please submit a full bug report,
with preprocessed source if appropriate.
See for instructions.

For some reason that I didn’t understand, I was not able to get a core file (tried setting ulimit -c but while I could get cores for other binaries, I could not get one for cpp). Anyhow, since cpp was working on other clients, it looked like something had gone wrong with either the binary or library. So I took a look at the libraries…
# ldd /usr/bin/cpp
libc.so.6 => /lib64/tls/libc.so.6 (0x000000398f800000)
/lib64/ld-linux-x86-64.so.2 (0x000000398f600000)

Nothing peculiar there, and if libc was broken, pretty much nothing would work. So the next step was to verify the cpp binary itself. I’d initially planned to run cksum or md5 on the cpp binary on the broken machine, and on a known good machine. Then I remembered that modern package management systems often keep a checksum or hash of the binaries in the package database.


So first we need to determine which package cpp belongs to;
# rpm -qf /usr/bin/cpp
cpp-3.4.5-2

And then verify the package against the checksum/hash in the package DB using ‘rpm -V package- name, where ‘V’ (uppercase) is for ‘Verify’.
rpm -V cpp-3.4.5-2
..5..... /usr/libexec/gcc/x86_64-redhat-linux/3.4.3/cc1

The output is a bit cryptic, but what it is saying is that the file /usr/libexec/gcc/x86_64-redhat-linux/3.4.3/cc1 failed the md5 check. A more verbose output makes things a little clearer, note the lowercase ‘v’ for verbose.
# rpm -Vv cpp-3.4.5-2
........ /lib/cpp
........ /usr/bin/cpp
........ /usr/libexec/gcc
........ /usr/libexec/gcc/x86_64-redhat-linux
........ /usr/libexec/gcc/x86_64-redhat-linux/3.4.3
..5..... /usr/libexec/gcc/x86_64-redhat-linux/3.4.3/cc1
........ /usr/libexec/gcc/x86_64-redhat-linux/3.4.5
........ d /usr/share/info/cpp.info.gz
........ d /usr/share/info/cppinternals.info.gz
........ d /usr/share/man/man1/cpp.1.gz
So we have a corrupt bianry, but that binary is neither the executed binary or a linked library. Can this really be a cause of the failure? A strace of ‘cpp’ should tell us.

[root@kc95b7 ~]# strace -f cpp 2>&1 | grep cc1
stat("/usr/libexec/gcc/x86_64-redhat-linux/3.4.5/cc1", {st_mode=S_IFREG|0755, st_size=4149432, ...}) = 0
access("/usr/libexec/gcc/x86_64-redhat-linux/3.4.5/cc1", X_OK) = 0
[pid 31323] execve("/usr/libexec/gcc/x86_64-redhat-linux/3.4.5/cc1", ["/usr/libexec/gcc/x86_64-redhat-l"..., "-E", "-quiet", "-", "-mtune=k8"], [/* 27 vars */]) = 0
[pid 31323] write(2, "cc1: internal compiler error: Se"..., 48cc1: internal compiler error: Segmentation fault) = 48

Using the -f switch to strace I see that we call execve() on ‘cc1′ so cpp does call cc1. And using a full strace (without grep) shows that is is indeed cc1 that causes a SEGV


Here we see the ‘cpp’ binary calling cc1. The cpp process has PID 31377, cpp is PID 31378.

[pid 31377] wait4(31378, Process 31377 suspended

[pid 31378] execve("/usr/libexec/gcc/x86_64-redhat-linux/3.4.5/cc1", ["/usr/libexec/gcc/x86_64-redhat-l"..., "-E", "-quiet", "-", "-mtune=k8"], [/* 27 vars */]) = 0
[pid 31378] uname({sys="Linux", node="kc95b7", ...}) = 0
[pid 31378] brk(0) = 0xaf4000

Later on we see that PID 31378 (cc1) generates a segfault, and receives from the kernel a SIGSEGV, BUT the cc1 process catches the SEGV signal and deals with it directly with rt_sigaction,and prints its own error message. It does not dump core – which is why we do not get a core file

[pid 31378] mmap(NULL, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x2a983e1000
[pid 31378] --- SIGSEGV (Segmentation fault) @ 0 (0) ---
[pid 31378] rt_sigaction(SIGSEGV, {SIG_DFL}, {0x572c06, [SEGV], SA_RESTORER|SA_RESTART, 0x398f82e380}, 8) = 0
[pid 31378] open("/usr/share/locale/locale.alias", O_RDONLY) = 3
...
[pid 31378] write(2, "cc1: internal compiler error: Se"..., 48cc1: internal compiler error: Segmentation fault) = 48
[pid 31378] write(2, "\n", 1
) = 1
[pid 31378] write(2, "Please submit a full bug report,"..., 138Please submit a full bug report,
with preprocessed source if appropriate.
See for instructions.
) = 138
[pid 31378] exit_group(27) = ?
Process 31377 resumed
Process 31378 detached
<... wait4 resumed> [{WIFEXITED(s) && WEXITSTATUS(s) == 27}], 0, NULL) = 31378
--- SIGCHLD (Child exited) @ 0 (0) ---
exit_group(1) = ?

So now we just have to figure out how the binary got corrupted….

A simple script to send an email after some (presumably long running) command is done.  I called it simply ‘m’ for ‘mail’ Execute it like this.

lovebox:shell-scripts gjl$ m gzip 10mfile 

Command line is : gzip 10mfile
Result : Success

And an email will be sent when the command completes.

RUNID=`date +"%y%m%d%H%M"`
USER=you.username@domainname.com
RESFILE=$$$RUNID
echo "Command line is : $@" > /tmp/$RESFILE
$@ >> /tmp/$RESFILE 2>&1
RC=$?
if [ $RC == 0 ] ; then
echo "Result : Success" >> /tmp/$RESFILE
RES=Success
else
echo "Result : Command fail" >> /tmp/$RESFILE
RES=Failed
fi
cat /tmp/$RESFILE
mailx -s "$RUNID $@ finished $RES" $USER < /tmp/$RESFILE
rm /tmp/$RESFILE


A simple way to convert a flexible volume to a traditional volume is to use ndmp, within the same filer.

Firstly enable ndmp and turn it on.
filer> options ndmpd.enable on
filer> ndmpd on

Next create the volume that you want to migrate the data to. In this case, I create a traditional volume with 5 disks, and use raid4 protection rather than raid-dp.

filer>vol create vol0_trad -t raid4 5

The output of the command should look similar to this :-

Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.4 Shelf 0 Bay 4 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7A9J2] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.0 Shelf 0 Bay 3 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7KTSY] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.3 Shelf 0 Bay 2 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7LHPR] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0c.6 Shelf 0 Bay 7 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7C91R] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.1 Shelf 0 Bay 1 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7LHVF] to volume vol0_trad has completed successfully
Creation of a volume with 5 disks has completed.
perfdevsrv015> Tue Jul  8 20:33:26 GMT [wafl.vol.add:notice]: Volume vol0_trad has been added to the system.

Next I use ndmpcopy to copy the files to the new volume. I use the -f switch since I am copying the root volume.
ndmpcopy -f /vol/vol0 /vol/vol0_trad
The output looks as below

Ndmpcopy: Starting copy [ 0 ] ...Ndmpcopy: perfdevsrv015: Notify: Connection established
Ndmpcopy: perfdevsrv015: Notify: Connection established
Ndmpcopy: perfdevsrv015: Connect: Authentication successful
Ndmpcopy: perfdevsrv015: Connect: Authentication successful
Ndmpcopy: perfdevsrv015: Log: DUMP: creating "/vol/vol0/../snapshot_for_backup.0" snapshot.
Ndmpcopy: perfdevsrv015: Log: DUMP: Using Full Volume Dump
Ndmpcopy: perfdevsrv015: Log: DUMP: Date of this level 0 dump: Tue Jul  8 20:35:51 2008.
Ndmpcopy: perfdevsrv015: Log: DUMP: Date of last level 0 dump: the epoch.
Ndmpcopy: perfdevsrv015: Log: DUMP: Dumping /vol/vol0 to NDMP connection
Ndmpcopy: perfdevsrv015: Log: DUMP: mapping (Pass I)[regular files]
Ndmpcopy: perfdevsrv015: Log: DUMP: mapping (Pass II)[directories]
Ndmpcopy: perfdevsrv015: Log: DUMP: estimated 235187 KB.Ndmpcopy: perfdevsrv015: Log: DUMP: dumping (Pass III) [directories]
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:40 2008: Begin level 0 restore
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:40 2008: Reading directories from the backup
Ndmpcopy: perfdevsrv015: Log: DUMP: dumping (Pass IV) [regular files]
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:49 2008: Creating files and directories.
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:54 2008: Writing data to files.
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:37:30 2008: Restoring NT ACLs.
Ndmpcopy: perfdevsrv015: Log: DUMP: dumping (Pass V) [ACLs]
Ndmpcopy: perfdevsrv015: Log: DUMP: 238893 KB
Ndmpcopy: perfdevsrv015: Log: RESTORE: RESTORE IS DONE
Ndmpcopy: perfdevsrv015: Log: DUMP: DUMP IS DONE
Ndmpcopy: perfdevsrv015: Log: RESTORE: The destination path is /vol/vol0_trad/
Ndmpcopy: perfdevsrv015: Notify: restore successful
Ndmpcopy: perfdevsrv015: Log: DUMP: Deleting "/vol/vol0/../snapshot_for_backup.0" snapshot.
Ndmpcopy: perfdevsrv015: Notify: dump successful
Ndmpcopy: Transfer successful [ 1 minutes 44 seconds ]
Ndmpcopy: Done

Now since I want to boot from this volume I set the boot option

vol options vol0_trad root

ONTAP tells me that this volume will become the root volume on next boot;

Volume 'vol0_trad' will become root at the next boot.

I can now reboot. I only need to do this step because I am messing with the boot/root volume

filer>reboot

After reboot I can rename the volumes

filer>vol rename vol0 vol0_flex

And now I can safely remove the volume and the aggregate so that I can use the disks in my new traditional volume

filer> vol offline vol0_flex


Tue Jul  8 20:46:01 GMT [wafl.vvol.offline:info]: Volume 'vol0_flex' has been set temporarily offlineVolume 'vol0_flex' is now offline.

Now destroy the volume, so we can get rid of the aggregate

filer> vol destroy vol0_flex

Are you sure you want to destroy this volume? y


Volume 'vol0_flex' destroyed.

filer> aggr offline aggr0

Aggregate 'aggr0' is now offline.

filer> aggr destroy aggr0
Are you sure you want to destroy this aggregate? y


Tue Jul  8 20:46:36 GMT [raid.config.vol.destroyed:info]: Aggregate 'aggr0' destroyed.Aggregate 'aggr0' destroyed.

 Now that I have destroyed the volume and aggegate, I am free to reuse those disks. ONTAP gives a warning that I will have no spare disks, and that this is risky in case of a failure.


filer> vol add vol0 2

WARNING! Continuing with vol add will result in having
no spare disk available for one or more RAID groups.
Are you sure you want to continue with vol add? y


Addition of 2 disks to the volume has been initiated.  The disks need
to be zeroed before addition to the volume.  The process has been initiated
and you will be notified via the system log as disks are added.

Once the disks are zeroed all the disks now belong to the traditional volume.

Recently we needed to transfer an aggregate that was hosted on an external shelf to disks housed internally in our test FAS2020 and the same trick can be used to move an aggregate from one shelf to another.

The trick uses the disk-replace command. It works serially, and so it is a little time consuming, BUT it retains exactly the same layout on disk, since it’s a bit-for-bit copy.

In short the command to use is ‘disk replace start e.g.

 disk replace start 0b.19 0c.00.0

So here’s what my disk layout looks like – I am using traditional volumes

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

Spare disks

RAID Disk       Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

---------       ------          ------------- ---- ---- ---- ----- --------------    --------------

Spare disks for block or zoned checksum traditional volumes or aggregates

spare           0a.16           0a    1   0   FC:A   -  ATA   7200 211377/432901760  211921/434014304(not zeroed)

spare           0a.20           0a    1   4   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.22           0a    1   6   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.24           0a    1   8   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.26           0a    1   10  FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.17           0b    1   1   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0b.25           0b    1   9   FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.27           0b    1   11  FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0c.00.0         0c    0   0   SA:1   -  SATA  7200 211377/432901760  211977/434130816 (not zeroed)

All the disks marked with a ‘CHAN’ type of FC:A or FC:B are actually SATA drives on the other side of a FC->SATA bridge, and are basically SATA disks in an external shelf. The disks marked SA:1 are SATA drive internal to the FAS2020. In the output above I am part-way through migrating the disks. Some of the disks in my volume/aggregate are internal and some are external. On of the disks that is still on the external shelf is disk 0b.19. Using the command below I will transfer the data on that individual disk to an internal SATA disk (0c.00.0). The command I use is.

 disk replace start 0b.19 0c.00.0

Then I receive a warning messge

*** You are about to copy and replace the following file system disk ***  Disk /vol1/plex0/rg0/0b.19

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304

***Really replace disk 0b.19 with 0c.00.0? ydisk replace: Disk 0b.19 was marked for replacing.

The output of sysconfig -r shows something like this

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    -------------- 

     dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (replacing)

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

And then once the replacement gets under way.. I see my copy under way.

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (replacing, copy in progress)      -> copy   0c.00.0         0c    0   0   SA:1   -  SATA  7200 211377/432901760  211977/434130816 (copy 1% completed)

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

And finally when the copy is done, 0c.00.0 is part of the aggregate/volume and 0b.19 is spare.

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.0         0c    0   0   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.2         0c    0   2   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

Spare disks

RAID Disk       Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

---------       ------          ------------- ---- ---- ---- ----- --------------    --------------

Spare disks for block or zoned checksum traditional volumes or aggregates

spare           0a.16           0a    1   0   FC:A   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0a.20           0a    1   4   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.22           0a    1   6   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.24           0a    1   8   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.26           0a    1   10  FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.17           0b    1   1   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0b.25           0b    1   9   FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.27           0b    1   11  FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)