lovebox-4:[~/data/traces] $ tshark -r trace.trc -R nfs > trace.out

gnuplot> plot "trace.out" u 2:(stringcolumn(8) eq "WRITE" ? $1:0/0 ) lc 2 t "WRITES" w points, "" u 2:(stringcolumn(8) eq "READ" ? $1:0/0) lc 3 t "READS" w dots
gnuplot> set term png
gnuplot> set output "trace.png"
gnuplot> replot

Sizes larger than rsize result in parallel IO’s

64K Rather than 32K rsize gives ??? improvement (depends on userland read size no?)

Install ttcp on the Mac

lovebox-3:~ gjl$ sudo port install ttcp
--->  Fetching ttcp
--->  Attempting to fetch ttcp.c from http://ykf.ca.distfiles.macports.org/MacPorts/mpdistfiles/ttcp
--->  Attempting to fetch ttcp.1 from http://ykf.ca.distfiles.macports.org/MacPorts/mpdistfiles/ttcp
--->  Attempting to fetch README from http://ykf.ca.distfiles.macports.org/MacPorts/mpdistfiles/ttcp
--->  Verifying checksum(s) for ttcp
--->  Extracting ttcp
--->  Configuring ttcp
--->  Building ttcp
--->  Staging ttcp into destroot
--->  Installing ttcp @1.10_0
--->  Activating ttcp @1.10_0
--->  Cleaning ttcp

Install nttcp on Linux

gjl@gjl-desktop:~$ sudo apt-get install nttcp
[sudo] password for gjl:
Reading package lists... Done
Building dependency tree
Reading state information... Done
Suggested packages:
  openbsd-inetd inet-superserver
The following NEW packages will be installed:
  nttcp
0 upgraded, 1 newly installed, 0 to remove and 51 not upgraded.
Need to get 25.4kB of archives.
After this operation, 98.3kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ lucid/multiverse nttcp 1.47-12 [25.4kB]
Fetched 25.4kB in 0s (56.6kB/s)
Selecting previously deselected package nttcp.
(Reading database ... 243192 files and directories currently installed.)
Unpacking nttcp (from .../nttcp_1.47-12_i386.deb) ...
Processing triggers for man-db ...
Setting up nttcp (1.47-12) ...

Usage

On the server

gjl@gjl-desktop:~$ nttcp -i -p 5001

On the client

lovebox-3:~ gjl$ ttcp -s -t 10.0.1.4 -v -l 4096 -p 5002

Output

ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  -> 10.0.1.4
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 6.41 real seconds = 2554.32 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 3.21, calls/sec = 319.29
ttcp-t: 0.0user 0.2sys 0:06real 3% 0i+0d 202752maxrss 0+2pf 6593+53csw

Wifi

lovebox-3:~ gjl$ ttcp -s -t 10.0.1.4 -v -l 4096 -p 5002
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  -> 10.0.1.4
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 12.77 real seconds = 1282.91 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 6.39, calls/sec = 160.36
ttcp-t: 0.0user 0.2sys 0:12real 1% 0i+0d 202752maxrss 0+3pf 5549+69csw

5Ghz network

lovebox-3:~ gjl$ ttcp -s -t 10.0.1.4 -v -l 4096 -p 5002
ttcp-t: buflen=8192, nbuf=2048, align=16384/0, port=5001  tcp  -> 10.0.1.4
ttcp-t: socket
ttcp-t: connect
ttcp-t: 16777216 bytes in 5.31 real seconds = 3083.91 KB/sec +++
ttcp-t: 2048 I/O calls, msec/call = 2.66, calls/sec = 385.49
ttcp-t: 0.0user 0.1sys 0:05real 3% 0i+0d 202752maxrss 0+2pf 1185+318csw
lovebox-3:~ gjl$ ttcp -s -t 10.0.1.4 -v -l 4096 -p 5002

I cobbled together this example from things I found on the interwebs as well as the demo code. Eventually I want to be able to decode NFS packets. So far, I can get the TCP port numbers – so that’s a start at least.

I used Mac OS X, and compiled both pcapy and impacket from source without any hassle.

from pcapy import open_offline
from impacket.ImpactDecoder import EthDecoder
from impacket.ImpactPacket import IP, TCP, UDP, ICMP

pcap = open_offline("some_packet_trace_file.trc")

decoder = EthDecoder()

def callback(hdr,data):
    print ".",
    packet=decoder.decode(data)
    l2=packet.child()
    if isinstance(l2,IP):
        print "IP",
        l3=l2.child()
        if isinstance(l3,TCP):
           src_ip = l2.get_ip_src()
           dst_ip = l2.get_ip_dst()
           tcp_dst_port = l3.get_th_sport()
           tcp_src_port = l3.get_th_dport()

           print "TCP from %s (%s) to %s(%s) " % (src_ip,tcp_src_port,dst_ip,tcp_dst_port)
        if isinstance(l3,UDP):
            print "UDP"

pcap.loop(0,callback)

print "Done"

The output looks like this

...
. IP TCP from 192.168.10.3 (2049) to 192.168.10.4(947)
. IP TCP from 192.168.10.3 (2049) to 192.168.10.4(947)
Done

To setup a default gateway as 10.199.50.1 use the following (note the final ’1′ at the end of the command line.

filer-3170> route add default 10.199.50.1 1

Not sure how well known this trick is, but it is possible to figure out the vendor of a particular NIC by looking at the first few digits of the MAC address. We used to use this when I worked at Sun to figure out if the customers were using pukka Sun NICs and what might be at the other end. In the modern world there is a web tool that will do the lookup for you. Mac Find

filer*> ifconfig e1b alias 192.168.101.4

Will add the address 192.168.101.4 to my NIC e1b, which already contained the IP 192.168.100.4

filer*> ifconfig e1b
e1b: flags=0x3d48867 mtu 9000
        inet 192.168.100.4 netmask 0xffffff00 broadcast 192.168.100.255
        inet 192.168.101.4 netmask 0xffffff00 broadcast 192.168.101.255
        ether 00:07:43:05:7c:db (auto-10g_sr-fd-up) flowcontrol full

This paper gives throughput and latency figures achieved using 10 GbE using Linux clients. Their tests showed a 300% improvement in latency over 1GbE. This whitepaper was published in 2003 by LANL.
Initial End-to-End Performance Evaluation of 10-Gigabit Ethernet�

I don’t know if this is a universal syntax for ifconfig, but the other day I needed to add another IP address to a NIC within a filer. The syntax was surprisingly quite intuitive.

ifconfig e0b alias 192.168.1.10

Running a network trace from a filer

Data ONTAP has a built in packet trace capability, which can be run straight from the command line. The format of the capture is the standard BPF and so can be imported into pretty much any tool for viewing and analysis. By default the packet trace is stored in memory, but it can be easily switched to write out to a local volume on the filer in order to gather larger traces.

Start a trace on interface eOb and write the file out to ‘/’ with a miximum trace size of 5 MB

rtpperfsrv053*> pktt start e0b -d / -s 5m

Measuring iSCSI traffic with tethereal

tethereal is a command line interface to Ethereal, I am not sure if it comes bundled with Wireshark, but it is available on some linux distros. I find the syntax and functionality nicer than tcpdump. There’s lots of data on the net about using tethereal with NFS but not so much with iSCSI. Below are a few commands and filters I used to analyse some traces of iSCSI traffic.

overview

• In the example below the captured packet data rolled up into the iscsi protocol, and so we see the SCSI writes from the clients (192.168.10.4 & 192.168.10.3) to the filer (192.168.10.5). The LBA: value is the offset into the LUN where the read/write will take place. I am using the ‘-r’ switch to read a file creates by pktt. The filename is interfacename_datetimestamp.trc

$tethereal -r e0b_20080519_102523.trc iscsi

7   0.000450 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0xffff (CDB:0xffff) (Good)8   0.000545 192.168.10.5 -> 192.168.10.4 iSCSI SCSI: Response LUN: 0xffff (CDB:0xffff) (Good)68   0.001521 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x07 (LBA: 0x00228718, Len: 112)SCSI: Data Out LUN: 0x07 (Write(10) Request Data)69   0.002746 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0x07 (Write(10)) (Good)129   0.003225 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x1e (LBA: 0x00228b00, Len: 112)SCSI: Data Out LUN: 0x1e (Write(10) Request Data)180   0.004140 192.168.10.4 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x1a (LBA: 0x03dd12e8, Len: 96)SCSI: Data Out LUN: 0x1a (Write(10) Request Data)236   0.004871 192.168.10.4 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x03 (LBA: 0x03de1060, Len: 104)SCSI: Data Out LUN: 0x03 (Write(10) Request Data)315   0.005210 192.168.10.4 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x1c (LBA: 0x03de1b00, Len: 104)SCSI: Data Out LUN: 0x1c (Write(10) Request Data)331   0.005320 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0x1e (Write(10)) (Good)

Gathering statistics

Although there are some really nice commands for viewing RPC (e.g. NFS) calls including the round trip time (RTT) of each NFS IO, I could not find anything similar for iSCSI.

• tethereal does alow the display of per interval statistics, l, in the example below I tethereal displays stats every 1/2 second. It is possible specify any time value down to millisecond. In this example we are reading a pktt created file using ‘-r’. The output shows that the traffic is somewhat ‘bursty’ in nature.
• I use the switch -tad to give me time as an absolute delta from the first packet. -r to read data from a file (rather than ‘live’ from a NIC). -q to print out just the summary result table. -z io,stat,0.5 tells tethereal to generate statistics on the I/O every 0.5 seconds.

$tethereal -tad -r e0b_20080519_153919.trc -qz io,stat,0.5

IO StatisticsInterval: 0.500 secsColumn #0:              |   Column #0  Time            |frames|  bytes000.000-000.500   15249  15114470000.500-001.000    7828   7686548001.000-001.500    9094   8857572001.500-002.000    9774   9650058002.000-002.500    6218   6062802002.500-003.000   15552  15328528003.000-003.500    4803   4643892003.500-004.000    6354   6203928004.000-004.500   10224  10086896004.500-005.000    5312   5169636005.000-005.500   13555  13354682005.500-006.000    6451   6369138006.000-006.500    7528   7393106 

Using filters

The filter style used here is the ‘display filter’ syntax of ethereal explained here
Now I want to just see traffic from one of my iSCSI clients. I can filter iSCSI traffic from IP address 192.168.10.3. N.B. Use quotes to achieve boolean logic, not parentheses ‘( )’

tethereal -tad -r e0b_20080519_153919.trc -R 'ip.src==192.168.10.3 && iscsi'

Now I want to filer further because I have several iSCSI sessions on that client. I know that the session I am interested in is using port 4704 as its source port. Now I can filter iSCSI traffic to/from 192.168.10.3 AND the port is 4704 AND traffic is iSCSI – useful for watching one iSCSI session.

tethereal -tad -r e0b_20080519_153919.trc -R 'ip.addr==192.168.10.3 && iscsi &&; tcp.port==4704' 

Here is that filter in action. We see that we only see writes to a specific LUN which is what we expect in this example since I know that each session on the client only writes to one iSCSI LUN.

-bash-3.00$ sudo tethereal -tad -r e0b_20080519_153919.trc -R 'ip.addr==192.168.10.3 && iscsi && tcp.port==4704'16620 2008-05-19 15:39:19.894110 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Read(10) LUN: 0x0a (LBA: 0x00000001, Len: 1)17072 2008-05-19 15:39:19.935890 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Data In LUN: 0x0a (Read(10) Response Data) SCSI: Response LUN: 0x0a (Read(10)) (Good)17172 2008-05-19 15:39:19.943567 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x0a (LBA: 0x0146c3a0, Len: 96)SCSI: Data Out LUN: 0x0a (Write(10) Request Data)39667 2008-05-19 15:39:20.957360 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x0a (LBA: 0x0146c400, Len: 96)SCSI: Data Out LUN: 0x0a (Write(10) Request Data)39909 2008-05-19 15:39:21.019261 192.168.10.5 -> 192.168.10.3 iSCSI SCSI: Response LUN: 0x0a (Write(10)) (Good)40226 2008-05-19 15:39:21.036799 192.168.10.3 -> 192.168.10.5 iSCSI SCSI: Write(10) LUN: 0x0a (LBA: 0x0146c460, Len: 96)SCSI: Data Out LUN: 0x0a (Write(10) Request Data)