Running a network trace from a filer
Data ONTAP has a built in packet trace capability, which can be run straight from the command line. The format of the capture is the standard BPF and so can be imported into pretty much any tool for viewing and analysis. By default the packet trace is stored in memory, but it can be easily switched to write out to a local volume on the filer in order to gather larger traces.
Start a trace on interface eOb and write the file out to ‘/’ with a miximum trace size of 5 MB
rtpperfsrv053*> pktt start e0b -d / -s 5m
Measuring iSCSI traffic with tethereal
tethereal is a command line interface to Ethereal, I am not sure if it comes bundled with Wireshark, but it is available on some linux distros. I find the syntax and functionality nicer than tcpdump. There’s lots of data on the net about using tethereal with NFS but not so much with iSCSI. Below are a few commands and filters I used to analyse some traces of iSCSI traffic.
overview
- In the example below the captured packet data rolled up into the iscsi protocol, and so we see the SCSI writes from the clients (192.168.10.4 & 192.168.10.3) to the filer (192.168.10.5). The LBA: value is the offset into the LUN where the read/write will take place. I am using the ‘-r’ switch to read a file creates by pktt. The filename is interfacename_datetimestamp.trc
$tethereal -r e0b_20080519_102523.trc iscsi
7 0.000450 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0xffff (CDB:0xffff) (Good)8 0.000545 192.168.10.5 -> 192.168.10.4 iSCSI SCSI: Response LUN: 0xffff (CDB:0xffff) (Good)68 0.001521 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x07 (LBA: 0x00228718, Len: 112)SCSI: Data Out LUN: 0x07 (Write(10) Request Data)69 0.002746 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0x07 (Write(10)) (Good)129 0.003225 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x1e (LBA: 0x00228b00, Len: 112)SCSI: Data Out LUN: 0x1e (Write(10) Request Data)180 0.004140 192.168.10.4 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x1a (LBA: 0x03dd12e8, Len: 96)SCSI: Data Out LUN: 0x1a (Write(10) Request Data)236 0.004871 192.168.10.4 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x03 (LBA: 0x03de1060, Len: 104)SCSI: Data Out LUN: 0x03 (Write(10) Request Data)315 0.005210 192.168.10.4 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x1c (LBA: 0x03de1b00, Len: 104)SCSI: Data Out LUN: 0x1c (Write(10) Request Data)331 0.005320 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0x1e (Write(10)) (Good)
Gathering statistics
Although there are some really nice commands for viewing RPC (e.g. NFS) calls including the round trip time (RTT) of each NFS IO, I could not find anything similar for iSCSI.
- tethereal does alow the display of per interval statistics, l, in the example below I tethereal displays stats every 1/2 second. It is possible specify any time value down to millisecond. In this example we are reading a pktt created file using ‘-r’. The output shows that the traffic is somewhat ‘bursty’ in nature.
- I use the switch -tad to give me time as an absolute delta from the first packet. -r to read data from a file (rather than ‘live’ from a NIC). -q to print out just the summary result table. -z io,stat,0.5 tells tethereal to generate statistics on the I/O every 0.5 seconds.
$tethereal -tad -r e0b_20080519_153919.trc -qz io,stat,0.5
IO StatisticsInterval: 0.500 secsColumn #0: | Column #0 Time |frames| bytes000.000-000.500 15249 15114470000.500-001.000 7828 7686548001.000-001.500 9094 8857572001.500-002.000 9774 9650058002.000-002.500 6218 6062802002.500-003.000 15552 15328528003.000-003.500 4803 4643892003.500-004.000 6354 6203928004.000-004.500 10224 10086896004.500-005.000 5312 5169636005.000-005.500 13555 13354682005.500-006.000 6451 6369138006.000-006.500 7528 7393106
Using filters
The filter style used here is the ‘display filter’ syntax of ethereal explained here
Now I want to just see traffic from one of my iSCSI clients. I can filter iSCSI traffic from IP address 192.168.10.3. N.B. Use quotes to achieve boolean logic, not parentheses ‘( )’
tethereal -tad -r e0b_20080519_153919.trc -R 'ip.src==192.168.10.3 && iscsi'
Now I want to filer further because I have several iSCSI sessions on that client. I know that the session I am interested in is using port 4704 as its source port. Now I can filter iSCSI traffic to/from 192.168.10.3 AND the port is 4704 AND traffic is iSCSI – useful for watching one iSCSI session.
tethereal -tad -r e0b_20080519_153919.trc -R 'ip.addr==192.168.10.3 && iscsi &&; tcp.port==4704'
Here is that filter in action. We see that we only see writes to a specific LUN which is what we expect in this example since I know that each session on the client only writes to one iSCSI LUN.
-bash-3.00$ sudo tethereal -tad -r e0b_20080519_153919.trc -R 'ip.addr==192.168.10.3 && iscsi && tcp.port==4704'16620 2008-05-19 15:39:19.894110 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Read(10) LUN: 0x0a (LBA: 0x00000001, Len: 1)17072 2008-05-19 15:39:19.935890 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Data In LUN: 0x0a (Read(10) Response Data) SCSI: Response LUN: 0x0a (Read(10)) (Good)17172 2008-05-19 15:39:19.943567 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x0a (LBA: 0x0146c3a0, Len: 96)SCSI: Data Out LUN: 0x0a (Write(10) Request Data)39667 2008-05-19 15:39:20.957360 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x0a (LBA: 0x0146c400, Len: 96)SCSI: Data Out LUN: 0x0a (Write(10) Request Data)39909 2008-05-19 15:39:21.019261 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0x0a (Write(10)) (Good)40226 2008-05-19 15:39:21.036799 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x0a (LBA: 0x0146c460, Len: 96)SCSI: Data Out LUN: 0x0a (Write(10) Request Data)