After some connectivity swap-a-roos in the lab, I could no longer see my LUNS from the linux host attached to my filer.

In this case I am using a QLogic HBA – and I am not using any of the NetApp host side tools – just the sanlun tool.

Using the SANsurfer Menu (/opt/QLogic_Corporation/SANsurferCLI) I can tell that this linux host can see the filers’ LUNS over FC. But there are no SCSI /dev/sdX devices for them, and so Linux cannot use them…

Here’s how I checked to see that there was FC connectivity – which also confirms that the FC protocol is working.

	SANsurfer FC/CNA HBA CLI

	v1.7.2 Build 7

    Main Menu

    1:	General Information  <---- Option 1
    2:	HBA Information
    3:	HBA Parameters
    4:	Target/LUN List
    5:	iiDMA Settings
...

    General Information Menu

    1:	Host Information
    2:	Host Topology
    3:	Report     <---- Option 3..
    4:	Refresh
    5:	Return to Previous Menu

	Note: 0 to return to Main Menu
	Enter Selection: 1

   Report Menu

    HBA Model QLE2462
      1: Port   1: WWPN: 21-00-00-E0-8B-9B-C5-36 Online
      2: Port   2: WWPN: 21-01-00-E0-8B-BB-C5-36 Online
      3: All HBAs  <---- Option 3
      4: Return to Previous Menu

	Note: 0 to return to Main Menu
	Enter Selection: 3

I could see that there was connectivity from the Linux host to the filer

---------------------------------------
LUN 1
---------------------------------------
Product Vendor                    : NETAPP
Product ID                        : LUN
Product Revision                  : 811a
LUN                               : 1
Size                              : 17.93 GB
Type                              : SBC-2 Direct access block device
			           (e.g., magnetic disk)
WWULN                             : 4E-45-54-41-50-50-20-20-20-4C-55-4E-20-32-46-68
			           72-53-3F-2D-68-4F-79-6C-33-00-00-00-00-00-00-00
OS LUN Name                       :

From the filer side, I could see that the host's FC adapters had connected to the filer,
and were in the right igroup

filer1*> igroup show
    filer1 (FCP) (ostype: linux):
        21:00:00:e0:8b:9b:c5:36 (logged in on: 0a)
        21:01:00:e0:8b:bb:c5:36 (logged in on: 0b)

The only thing that was missing was that there were no 'sd' devices created in Linux for these devices.

"sanlun" utility was not helpful and just told me that there wer no LUNs mapped.

The solution was to issue this very odd looking command

linuxhost:[/sys/class/scsi_host] $ echo "- - -" > host0/scan

This caused the sd devices to be created, representing the NetApp LUNs which I knew could already be seen over FC. Since I have both ports on the same HBA attached to the filer, host0 scan created my /dev/sdc* devices, and host1/scan created my /dev/sdd* devices.

The shell 'hung' for the duration of the command, and I would expect that Linux was off in kernel land for some time - and so i would NOT recommend issuing the command on a production server.

I'm still puzzled why the linux host did not see the luns even after reboot though.

I am mentoring a new starter at NetApp and so I found a couple of papers which discuss at a high level some of the problems that WAFL and RAID set out to solve. Both papers are quite old, but are interesting in that they discuss the big picture of the original problem.

A Storage Networking Appliance
This is a great paper which discusses the principles of the “filer” concept.

• NetApp Internal (Updated 2006) :

File System Design for an NFS
File Server Appliance

• File System Design for an NFS
  File Server Appliance

• Usenix : Text file (1994 Version):
• ACM Digital Library PDF (1994 Version) :

Just recently an ex-colleague from Sun asked me to send him some background info on NetApp filer performance. Normally the filer can just be plugged in and it ought to be ready to go without much tweaking. The following documents give some background on how to setup a filer in various environments.

• A list of all technical reports
• Database Performance with NFS: Optimizing Oracle on NFS.
  A little dated, but still an interesting introduction to using NFS in an enterprise environment

• Best Practices for File System Alignment in Virtual Environments. Some specific tips for using a filer with hypervisors
• Evaluating the Performance of FCoE, iSCSI, and FC Using DSS Workloads with Microsoft SQL Server 2008
• If You’re Doing This, Then Your Storage Could Be Underutilized
• Introduction to Predictive Cache Statistics
  DataONTAP contains a facility which enables users to determine how much benefit they might get from installing a flash based accelerator card (FlashCache).

• Performance Study of IBM DB2 9 with NFS, iSCSI, and FCP
  The recommendations in this document are also generally applicable across NFS, iCSCI and FCP.

• NetApp and VMware Virtual Infrastructure 3 Storage Best Practices

To setup a default gateway as 10.199.50.1 use the following (note the final ’1′ at the end of the command line.

filer-3170> route add default 10.199.50.1 1

filer*> ifconfig e1b alias 192.168.101.4

Will add the address 192.168.101.4 to my NIC e1b, which already contained the IP 192.168.100.4

filer*> ifconfig e1b
e1b: flags=0x3d48867 mtu 9000
        inet 192.168.100.4 netmask 0xffffff00 broadcast 192.168.100.255
        inet 192.168.101.4 netmask 0xffffff00 broadcast 192.168.101.255
        ether 00:07:43:05:7c:db (auto-10g_sr-fd-up) flowcontrol full

A simple way to convert a flexible volume to a traditional volume is to use ndmp, within the same filer.

Firstly enable ndmp and turn it on.
filer> options ndmpd.enable on
filer> ndmpd on

Next create the volume that you want to migrate the data to. In this case, I create a traditional volume with 5 disks, and use raid4 protection rather than raid-dp.

filer>vol create vol0_trad -t raid4 5

The output of the command should look similar to this :-

Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.4 Shelf 0 Bay 4 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7A9J2] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.0 Shelf 0 Bay 3 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7KTSY] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.3 Shelf 0 Bay 2 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7LHPR] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0c.6 Shelf 0 Bay 7 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7C91R] to volume vol0_trad has completed successfully
Tue Jul  8 20:33:25 GMT [raid.vol.disk.add.done:notice]: Addition of Disk /vol0_trad/plex0/rg0/0d.1 Shelf 0 Bay 1 [SEAGATE  ST3500630NSAH    AQMZ] S/N [9QG7LHVF] to volume vol0_trad has completed successfully
Creation of a volume with 5 disks has completed.
perfdevsrv015> Tue Jul  8 20:33:26 GMT [wafl.vol.add:notice]: Volume vol0_trad has been added to the system.

Next I use ndmpcopy to copy the files to the new volume. I use the -f switch since I am copying the root volume.
ndmpcopy -f /vol/vol0 /vol/vol0_trad
The output looks as below

Ndmpcopy: Starting copy [ 0 ] ...Ndmpcopy: perfdevsrv015: Notify: Connection established
Ndmpcopy: perfdevsrv015: Notify: Connection established
Ndmpcopy: perfdevsrv015: Connect: Authentication successful
Ndmpcopy: perfdevsrv015: Connect: Authentication successful
Ndmpcopy: perfdevsrv015: Log: DUMP: creating "/vol/vol0/../snapshot_for_backup.0" snapshot.
Ndmpcopy: perfdevsrv015: Log: DUMP: Using Full Volume Dump
Ndmpcopy: perfdevsrv015: Log: DUMP: Date of this level 0 dump: Tue Jul  8 20:35:51 2008.
Ndmpcopy: perfdevsrv015: Log: DUMP: Date of last level 0 dump: the epoch.
Ndmpcopy: perfdevsrv015: Log: DUMP: Dumping /vol/vol0 to NDMP connection
Ndmpcopy: perfdevsrv015: Log: DUMP: mapping (Pass I)[regular files]
Ndmpcopy: perfdevsrv015: Log: DUMP: mapping (Pass II)[directories]
Ndmpcopy: perfdevsrv015: Log: DUMP: estimated 235187 KB.Ndmpcopy: perfdevsrv015: Log: DUMP: dumping (Pass III) [directories]
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:40 2008: Begin level 0 restore
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:40 2008: Reading directories from the backup
Ndmpcopy: perfdevsrv015: Log: DUMP: dumping (Pass IV) [regular files]
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:49 2008: Creating files and directories.
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:36:54 2008: Writing data to files.
Ndmpcopy: perfdevsrv015: Log: RESTORE: Tue Jul  8 20:37:30 2008: Restoring NT ACLs.
Ndmpcopy: perfdevsrv015: Log: DUMP: dumping (Pass V) [ACLs]
Ndmpcopy: perfdevsrv015: Log: DUMP: 238893 KB
Ndmpcopy: perfdevsrv015: Log: RESTORE: RESTORE IS DONE
Ndmpcopy: perfdevsrv015: Log: DUMP: DUMP IS DONE
Ndmpcopy: perfdevsrv015: Log: RESTORE: The destination path is /vol/vol0_trad/
Ndmpcopy: perfdevsrv015: Notify: restore successful
Ndmpcopy: perfdevsrv015: Log: DUMP: Deleting "/vol/vol0/../snapshot_for_backup.0" snapshot.
Ndmpcopy: perfdevsrv015: Notify: dump successful
Ndmpcopy: Transfer successful [ 1 minutes 44 seconds ]
Ndmpcopy: Done

Now since I want to boot from this volume I set the boot option

vol options vol0_trad root

ONTAP tells me that this volume will become the root volume on next boot;

Volume 'vol0_trad' will become root at the next boot.

I can now reboot. I only need to do this step because I am messing with the boot/root volume

filer>reboot

After reboot I can rename the volumes

filer>vol rename vol0 vol0_flex

And now I can safely remove the volume and the aggregate so that I can use the disks in my new traditional volume

filer> vol offline vol0_flex


Tue Jul  8 20:46:01 GMT [wafl.vvol.offline:info]: Volume 'vol0_flex' has been set temporarily offlineVolume 'vol0_flex' is now offline.

Now destroy the volume, so we can get rid of the aggregate

filer> vol destroy vol0_flex

Are you sure you want to destroy this volume? y


Volume 'vol0_flex' destroyed.

filer> aggr offline aggr0

Aggregate 'aggr0' is now offline.

filer> aggr destroy aggr0
Are you sure you want to destroy this aggregate? y


Tue Jul  8 20:46:36 GMT [raid.config.vol.destroyed:info]: Aggregate 'aggr0' destroyed.Aggregate 'aggr0' destroyed.

 Now that I have destroyed the volume and aggegate, I am free to reuse those disks. ONTAP gives a warning that I will have no spare disks, and that this is risky in case of a failure.


filer> vol add vol0 2

WARNING! Continuing with vol add will result in having
no spare disk available for one or more RAID groups.
Are you sure you want to continue with vol add? y


Addition of 2 disks to the volume has been initiated.  The disks need
to be zeroed before addition to the volume.  The process has been initiated
and you will be notified via the system log as disks are added.

Once the disks are zeroed all the disks now belong to the traditional volume.

Recently we needed to transfer an aggregate that was hosted on an external shelf to disks housed internally in our test FAS2020 and the same trick can be used to move an aggregate from one shelf to another.

The trick uses the disk-replace command. It works serially, and so it is a little time consuming, BUT it retains exactly the same layout on disk, since it’s a bit-for-bit copy.

In short the command to use is ‘disk replace start e.g.

 disk replace start 0b.19 0c.00.0

So here’s what my disk layout looks like – I am using traditional volumes

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

Spare disks

RAID Disk       Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

---------       ------          ------------- ---- ---- ---- ----- --------------    --------------

Spare disks for block or zoned checksum traditional volumes or aggregates

spare           0a.16           0a    1   0   FC:A   -  ATA   7200 211377/432901760  211921/434014304(not zeroed)

spare           0a.20           0a    1   4   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.22           0a    1   6   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.24           0a    1   8   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.26           0a    1   10  FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.17           0b    1   1   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0b.25           0b    1   9   FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.27           0b    1   11  FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0c.00.0         0c    0   0   SA:1   -  SATA  7200 211377/432901760  211977/434130816 (not zeroed)

All the disks marked with a ‘CHAN’ type of FC:A or FC:B are actually SATA drives on the other side of a FC->SATA bridge, and are basically SATA disks in an external shelf. The disks marked SA:1 are SATA drive internal to the FAS2020. In the output above I am part-way through migrating the disks. Some of the disks in my volume/aggregate are internal and some are external. On of the disks that is still on the external shelf is disk 0b.19. Using the command below I will transfer the data on that individual disk to an internal SATA disk (0c.00.0). The command I use is.

 disk replace start 0b.19 0c.00.0

Then I receive a warning messge

*** You are about to copy and replace the following file system disk ***  Disk /vol1/plex0/rg0/0b.19

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304

***Really replace disk 0b.19 with 0c.00.0? ydisk replace: Disk 0b.19 was marked for replacing.

The output of sysconfig -r shows something like this

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    -------------- 

     dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (replacing)

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

And then once the replacement gets under way.. I see my copy under way.

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (replacing, copy in progress)      -> copy   0c.00.0         0c    0   0   SA:1   -  SATA  7200 211377/432901760  211977/434130816 (copy 1% completed)

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

And finally when the copy is done, 0c.00.0 is part of the aggregate/volume and 0b.19 is spare.

Volume vol1 (online, raid_dp) (block checksums)  Plex /vol1/plex0 (online, normal, active)    RAID group /vol1/plex0/rg0 (normal)

      RAID Disk Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

      --------- ------          ------------- ---- ---- ---- ----- --------------    --------------

      dparity   0c.00.6         0c    0   6   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      parity    0c.00.1         0c    0   1   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.0         0c    0   0   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.3         0c    0   3   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.11        0c    0   11  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.10        0c    0   10  SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.9         0c    0   9   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.8         0c    0   8   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.7         0c    0   7   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.4         0c    0   4   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.2         0c    0   2   SA:1   -  SATA  7200 211377/432901760  211977/434130816

      data      0c.00.5         0c    0   5   SA:1   -  SATA  7200 211377/432901760  211977/434130816

Spare disks

RAID Disk       Device          HA  SHELF BAY CHAN Pool Type  RPM  Used (MB/blks)    Phys (MB/blks)

---------       ------          ------------- ---- ---- ---- ----- --------------    --------------

Spare disks for block or zoned checksum traditional volumes or aggregates

spare           0a.16           0a    1   0   FC:A   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0a.20           0a    1   4   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.22           0a    1   6   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.24           0a    1   8   FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0a.26           0a    1   10  FC:A   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.17           0b    1   1   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0b.19           0b    1   3   FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

spare           0b.25           0b    1   9   FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.27           0b    1   11  FC:B   -  ATA   7200 211377/432901760  211921/434014304

spare           0b.29           0b    1   13  FC:B   -  ATA   7200 211377/432901760  211921/434014304 (not zeroed)

I don’t know if this is a universal syntax for ifconfig, but the other day I needed to add another IP address to a NIC within a filer. The syntax was surprisingly quite intuitive.

ifconfig e0b alias 192.168.1.10

The CP type is displayed in the CP/ty column in the output of sysstat. The CP type column contains two pieces of data. The cause of the CP (the CP type) – the first character, and the ‘phase’. The second character. In the output below, the first row shows a CP type “T” and phase of “f”. The second row shows a cp type of “:” which just means that the same CP was still ongoing when sysstat sampled the internal CP counters the next time. We get a bit more insight into this process in the CP/time column which just represents the amount of the sample time which was spent in the CP. So, in the case where we are sampling every second – the entire CP took about 1/2 a second in the first sample, and then continued over into the second sample for 17% of one second. So, we can assume that the CP started in the second half of the first sample period, and continued a little into the next sample.

filer*> sysstat -x 1
    CP    CP Disk    FCP iSCSI   FCP  kB/s iSCSI  kB/s
    time  ty util                 in   out    in   out

    54%   Tf  92%    183     0     4 145228     0     0
    17%   :   96%    201     0     5 149234     0     0

The table below describes the CP types as per the sysstat manpage. In a system with a small amount of incoming data (or none at all) there will be an artificially generated CP caused by a timer, which fires once every 10 seconds, this is the CP type “T”.

When there is a high incoming data rate, the filer tries to free up resources before they become exhausted, which would mean that the user would see high write latencies. When everything is working well, as is normally the case – the incoming write latency is the time it takes to write the users data into NVRAM (plus the network round trip time). This is how the filer is able to achieve extremely high write rates, even for a random IO pattern. If the filer is not able to keep up with the incoming workload, it will sometimes show CP type “B” which can mean that the filer is continually in a state of CP, and the user workload can see higher latencies as a result.

The log full CP literally means that the NVRAM log is 50% full, and so the filer must write this dirty data out to disk. Whilst the CP is happening, the other 50% of the NVRAM is used to accept more incoming data, and so the clients should see the same low latency as when there is no CP ongoing.

A CP type of “H” indicates that the filer has a large number of dirty buffers in the system, and even though the NRVRAM is not yet 50% full, the filer issues a CP in order to free up RAM. This CP type is sometimes seen on filers with a small amount of RAM, and a large incoming write rate with a small IO size to random offsets.

CP type “Z” is often caused by snapshot creation or deletion. Remember that snapshot deletion can be triggered by several conditions besides a user typing “snap delete” at the console. Examples of ‘automatic’ deletion are.

• Snapshot deletion to recover space on the aggregate
• Snapshot deletion, to maintain a specific snap schedule