lovebox-4:[~/data/traces] $ tshark -r trace.trc -R nfs > trace.out

gnuplot> plot "trace.out" u 2:(stringcolumn(8) eq "WRITE" ? $1:0/0 ) lc 2 t "WRITES" w points, "" u 2:(stringcolumn(8) eq "READ" ? $1:0/0) lc 3 t "READS" w dots
gnuplot> set term png
gnuplot> set output "trace.png"
gnuplot> replot

Typically, I use the GUI version of wireshark to see how to specify the read filter, then use tshark at the command line to make use of all the CLI goodness of Unix.

Display NFS_LOOKUP Calls

 tshark -R "nfs.procedure_v3 == 3" -r sometracefile.trc

Display NFS_GETATTR Calls

 tshark -R "nfs.procedure_v3 == 1" -r sometracefile.trc

Display NFS_SETATTR Calls

 tshark -R "nfs.procedure_v3 == 2" -r sometracefile.trc

Display NFS_ACCESS Calls

 tshark -R "nfs.procedure_v3 == 4" -r sometracefile.trc

…
Display NFS_LINK Calls

 tshark -R "nfs.procedure_v3 == 15" -r sometracefile.trc

So, assume I have a git repository on an iDisk, mounted to my mac a /Volumes/garyjlittle. I also have a dropbox account mounted to /Dropbox.

I have a directory called ~/scripts on several machines (home and work) that I keep synchronized and version contolled using git.

Now, I want to move a git repository called “scripts” from iDisk to Dropbox.

Here’s what I do. I have a subdir called ‘git’ on my Dropbox drive, and within there I have a bunch of ‘bare’ repositories – because I always make edits in ~/scripts. The directory /Dropbox/git/scripts.git is just there to be a shared resource to push/pull from.

Make the directory hierarchy in Dropbox.

lovebox:[~] $ mkdir /Dropbox/git
lovebox:[~] $ mkdir /Dropbox/git/scripts.git

Change directory to the Dropbox script directory we just created.

lovebox:[~] $ cd /Dropbox/git/scripts.git
 

Now inititalize that directory as a git bare repository.

lovebox:[/Dropbox/git/scripts.git] $ git init --shared --bare

Now go to the iDisk directory which currently holds the bare repository

lovebox:[~] $ cd /Volumes/garyjlittle/scripts

And push over the repository to the new Dropbox location.

lovebox:[/Volumes/garyjlittle/scripts] $ git push --mirror /Dropbox/git/scripts.git/

Counting objects: 49, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (49/49), done.
Writing objects: 100% (49/49), 21.52 KiB, done.
Total 49 (delta 18), reused 0 (delta 0)
Unpacking objects: 100% (49/49), done.
To /Dropbox/git/scripts.git/
 * [new branch]      master -> master

Now we have to tell each remote location that the shared repository has changed. It’s easiest to just rename the old directory from ~/scripts to ~/old.scripts. And then do a git clone from the new Dropbox git location.

lovebox:[~] $ mv scripts/ old.scripts

lovebox:[~] $ git clone /Dropbox/git/scripts.git/ scripts
Initialized empty Git repository in /Users/gjl/scripts/.git/
lovebox:[~] $ cd scripts

You might notice (as I did) that there are fewer scripts in the new ~/scripts directory. That’s probably because not all of the scripts really were under git control. This could be a good time for housekeeping. Add in the scripts that you really want to keep under git version control, and get rid of the crufty old one-time stuff that inevitably builds up over time.

When collecting network traces to solve performance problems, we need to balance the need to capture enough data to re-construct the upper protocols with the need to limit the amount of trace data collected overall. When capturing NFS traffic, the packet size can grow to be quite large, since the NFS packets will contain the “payload” data i.e. the data that the userland application is reading or writing.

Most of the tools used to collect network trace data support the ‘snaplen’ option which controls how much data to capture per packet. For NFS traces a snaplen of 256 bytes seems to work well. With a snaplen of 256 the trace analysers will have enough data to reconstruct the filehandles, offset and IO size. Tools like tshark will be able to match the NFS response to the request and so you can see the response time from the NFS server.

In this trace below, the total response time is from the first NFS READ Call (packet 13) until the final RPC Continuation in the sequence. In NFS the NFS V3 READ Reply (Packet 14) is just the beginning of the NFS server response. The bulk of the payload is in the “RPC Continuation” packets. The number of Continuation packets will depend on the size of the NFS request and the size of the Ethernet frames. In the trace below, the IO size is 32K, the Ethernet segment size is 1500 and the IO stream is single threaded.

I use the tshark option “-R rpc” to show the “NFS READ calls” as well as the “RPC Continuation” responses from the NFS server.

The command line.

lovebox:test_snaplen gjl$ tshark -r snap256_from_open -R rpc|head -35

The trace.

 13   3.529598 192.168.10.3 -> 192.168.10.4 NFS V3 READ Call, FH:0x44281a4d Offset:0 Len:32768
 14   3.529844 192.168.10.4 -> 192.168.10.3 NFS V3 READ Reply (Call In 13) Len:32768[Unreassembled Packet]
 15   3.529856 192.168.10.4 -> 192.168.10.3 RPC Continuation
 17   3.529872 192.168.10.4 -> 192.168.10.3 RPC Continuation
 18   3.529882 192.168.10.4 -> 192.168.10.3 RPC Continuation
 20   3.529896 192.168.10.4 -> 192.168.10.3 RPC Continuation
 21   3.529905 192.168.10.4 -> 192.168.10.3 RPC Continuation
 23   3.529921 192.168.10.4 -> 192.168.10.3 RPC Continuation
 24   3.529930 192.168.10.4 -> 192.168.10.3 RPC Continuation
 26   3.529944 192.168.10.4 -> 192.168.10.3 RPC Continuation
 27   3.529955 192.168.10.4 -> 192.168.10.3 RPC Continuation
 29   3.529970 192.168.10.4 -> 192.168.10.3 RPC Continuation
 30   3.529980 192.168.10.4 -> 192.168.10.3 RPC Continuation
 32   3.529997 192.168.10.4 -> 192.168.10.3 RPC Continuation
 33   3.530005 192.168.10.4 -> 192.168.10.3 RPC Continuation
 35   3.530019 192.168.10.4 -> 192.168.10.3 RPC Continuation
 36   3.530029 192.168.10.4 -> 192.168.10.3 RPC Continuation
 38   3.530044 192.168.10.4 -> 192.168.10.3 RPC Continuation
 39   3.530053 192.168.10.4 -> 192.168.10.3 RPC Continuation
 41   3.530068 192.168.10.4 -> 192.168.10.3 RPC Continuation
 42   3.530077 192.168.10.4 -> 192.168.10.3 RPC Continuation
 44   3.530092 192.168.10.4 -> 192.168.10.3 RPC Continuation
 45   3.530100 192.168.10.4 -> 192.168.10.3 RPC Continuation
 47   3.530115 192.168.10.4 -> 192.168.10.3 RPC Continuation
 48   3.530197 192.168.10.3 -> 192.168.10.4 NFS V3 READ Call, FH:0x44281a4d Offset:32768 Len:32768

If you are processing someone elses trace data, one clue that the snaplen was too short is that the NFS reads do not contain the offset or Len values. In this example the snaplen was 192. Notice that packet 27 does not show the offset of the read, compare to packet 13 in the above trace, where the snaplen was 256.

 17  30.646984 192.168.10.3 -> 192.168.10.4 NFS V3 ACCESS Call, FH:0x34d96998
 18  30.647233 192.168.10.4 -> 192.168.10.3 NFS V3 ACCESS Reply (Call In 17)
 20  30.647279 192.168.10.3 -> 192.168.10.4 NFS V3 GETATTR Call, FH:0x44281a4d
 21  30.647472 192.168.10.4 -> 192.168.10.3 NFS V3 GETATTR Reply (Call In 20)  Regular File mode:0666 uid:0 gid:0
 22  30.647501 192.168.10.3 -> 192.168.10.4 NFS V3 ACCESS Call, FH:0x44281a4d
 23  30.647727 192.168.10.4 -> 192.168.10.3 NFS V3 ACCESS Reply (Call In 22)
 27  30.648254 192.168.10.3 -> 192.168.10.4 NFS V3 READ Call[Packet size limited during capture]
 28  30.648539 192.168.10.4 -> 192.168.10.3 NFS V3 READ Reply (Call In 27) Len:32768[Packet size limited during capture]

In wireshark, there are a number of settings which control how wireshark/tshark attempts to re-assemble rpc/tcp/nfs packets. Enabling “more aggressive” RPC decoding seemed to mess up the ability to show the RPC continuation packets. Here are the RPC settings from the wireshark preferences used to generate this data. These prefs also affect the way tshark works.

lovebox:test_snaplen gjl$ grep rpc ~/.wireshark/preferences
dcerpc.desegment_dcerpc: TRUE
dcerpc.reassemble_dcerpc: TRUE
rpc.desegment_rpc_over_tcp: TRUE
rpc.defragment_rpc_over_tcp: TRUE
rpc.max_tcp_pdu_size: 4194304
rpc.dissect_unknown_programs: FALSE
rpc.find_fragment_start: FALSE
rpcap.desegment_pdus: TRUE
rpcap.decode_content: TRUE
rpcap.linktype: 0

Often we’ll use awk or cut to spit out a single column of data from a bunch of different files, or maybe from the same file… anyhow once we have these files with one column of data, sometimes we’ll want to splice those files together so we can look at each line (typically a stat of some sort) next to each other. Also it’s very convenient if we want to import the data into a spreadsheet.

Anyhow, let’s say I have three files iscsi_080626.txt greads.out.txt idle.out.txt. I can put them together into one file using the ‘pr’ command like so.

iscsi file

bash-3.00$ head iscsi_080626.txt 32782319.8632793022.9332734064.2332719233.7432652552.7532815570.70

gread file
bash-3.00$ head greads.out.txt

0.000.000.000.000.000.000.000.000.000.00

idle file

bash-3.00$ head idle.out.txt 248.72214217.66291517.19502416.20896915.85643217.06451916.87494817.12295617.19130516.920703

bash-3.00$ pr -m -t iscsi_080626.txt greads.out.txt idle.out.txt > iscsi_greads_idle.txt

And the output file looke like this

bash-3.00$ head iscsi_greads_idle.txt 32782319.86             0.00                    248.72214232793022.93             0.00                    17.66291532734064.23             0.00                    17.19502432719233.74             0.00                    16.20896932652552.75             0.00                    15.85643232815570.70             0.00                    17.06451932801494.63             0.00                    16.87494832696942.10             0.00                    17.12295632715608.64             0.00                    17.19130532835538.38             0.00                    16.920703

I don’t know if this is a universal syntax for ifconfig, but the other day I needed to add another IP address to a NIC within a filer. The syntax was surprisingly quite intuitive.

ifconfig e0b alias 192.168.1.10

SQL> set linesize 120SQL> col file_name for a40SQL> col tablespace_name for a20

select file_name,tablespace_name from dba_data_fileswhere tablespace_name in ( select tablespace_name from dba_tablespaces where contents not in 'TEMPORARY')

FILE_NAME                                TABLESPACE_NAME---------------------------------------- --------------------/oradata/ora9test/system01.dbf           SYSTEM/oradata/ora9test/undotbs01.dbf          UNDOTBS1/oradata/ora9test/drsys01.dbf            DRSYS/oradata/ora9test/example01.dbf          EXAMPLE/oradata/ora9test/indx01.dbf             INDX/oradata/ora9test/tools01.dbf            TOOLS/oradata/ora9test/users01.dbf            USERS/oradata/ora9test/xdb01.dbf              XDB

8 rows selected.

set linesize 120col file_name for a40col tablespace_name for a20

When doing search and replace in vi, there is no need to ‘escape’ forward slash characters ‘/’ when manipulating pathnames. Instead, simply change the character used to delimit the search terms to something other than ‘/’ directly within vi (no .exrc change needed).

e.g.

./.cpan/build/Archive-Tar-1.30./.cpan/build/Archive-Tar-1.30/bin./.cpan/build/Archive-Tar-1.30/bin/ptar./.cpan/build/Archive-Tar-1.30/bin/ptardiff./.cpan/build/Archive-Tar-1.30/blib./.cpan/build/Archive-Tar-1.30/blib/arch

Then use the string below which uses ‘?’ where usually a ‘/’ is used to separate and delimit the srearch & replace terms. In this example I am replacing …cpan/build… with …cpan/build/z/y/z…

:%s?cpan/build?cpan/build/z/y/z?

Which results in

./.cpan/build/x/y/z/Archive-Tar-1.30./.cpan/build/x/y/x/Archive-Tar-1.30/bin./.cpan/build/x/y/x/Archive-Tar-1.30/bin/ptar./.cpan/build/x/y/x/Archive-Tar-1.30/bin/ptardiff./.cpan/build/x/y/x/Archive-Tar-1.30/blib./.cpan/build/x/y/x/Archive-Tar-1.30/blib/arch