set xtics nomirror rotate by -45

I was not able to login to my ubuntu desktop using my unix/nis login which was working fine via su. Here’s how I tracked down the problem to an unsupported ‘set’ command in my .profile.

1) I got this message in /va/log/auth.log

 gdm-session-worker[7336]: pam_ck_connector(gdm:session): nox11 mode

2) But that is actually a red-herring, and not the cause of my problems. I more interesting error was logged in my
~/.xession-errors file

$ cat .xsession-errors
/etc/gdm/Xsession: Beginning session setup...
set: 4: Illegal option -h

So, it is trying to run a ‘set’ command as part of login. I remembered that I had put some ‘set’ commands in my .profile, specifically

set -ha

3) Remove the “set -ha” line from .profile

4) Login via gdm, lightdm etc.

So, it seems that during login to the x session my .profile was being executed in a shell that did note support ‘set -ha’ and generated an error which meant that the X login sequence could not complete, and so I was not able to login even though I could happily ‘su’ to the user in question.

I have a very large file, which is full of directory listings. The file is around 500mb. I want to do a simple replace of “vol2″ to become “vol4″. Using vi or vim would require loading the entire file into memory (and is more difficult to script).

The file is called “vol2_first_40k.out”, and the format of the file looks like this

[root@unix tmp]# head vol2_first_40k.out
/vol2/dirA1/dirA11/file-bCOST@0paeRg`FIbONu]Tn_`n2BNsE\N5HuCaa7cmXw0H[K_n
/vol2/dirA1/dirA11/file-Vzi5oCec@]7cw2bQmBATD4Sf;4[\;EdPWMvO>JX5hvhY58dC@
/vol2/dirA1/dirA11/file-V?4LB<^SrhdT^CfGSod5d0eC8vvPf[QMa1wfcUz]:5O_TIett
/vol2/dirA1/dirA11/file-[tM_^[]dykcavgBtuD6Eo@_>8veEJNsKF5nYJ@^uo4:vXz6dA
/vol2/dirA1/dirA11/file-]cbNkxo_]tS2bTOC;TzxIv^>kk6?mazmN7BPE[jYeYD3<=YQX
/vol2/dirA1/dirA11/file-XyA5eT;KV8^wGsk1eCnDG\@4jPS0?OtP3IPAa[TfXU@jT`cF?
/vol2/dirA1/dirA11/file-O@YLYLL6sDqYXG4L=Sg477CTndn_;wF\GB0MmafH\Na^KEuzW
/vol2/dirA1/dirA11/file-xm<9mzj:MVmKE`IGCzs:lU8]7bWlmPxIkrVUDPPNddGdXCk5S
/vol2/dirA1/dirA11/file-agaj]\8_yMz5V@hLuVFm94e6JD0QYSslQgcHgx^oKLXVovCEt
/vol2/dirA1/dirA11/file-3`PC;tshre0ZzgJAb;xoyDszAHuR76dDAuvp5w@95Y7nFYtWf

Using sed, I can do the search/replace and send the output to a new filename e.g. vol4_first_40k.out I use the simplest sed command ‘s’ which just means substitute. Here I am substituting each instance of the string “vol2″ with the string “vol4″. Often, very simple search/replace is all that’s needed. Anything more complex, is often easier to do (for me at least) in something like Python.

[root@unix tmp]# sed "s/vol2/vol4/" vol2_first_40k.out  > vol4_first_40k.out

Or to test the replacement, before operating on the entire file

[root@unix tmp]# sed "s/vol2/vol4/" vol2_first_40k.out |head
/vol4/dirA1/dirA11/file-bCOST@0paeRg`FIbONu]Tn_`n2BNsE\N5HuCaa7cmXw0H[K_n
/vol4/dirA1/dirA11/file-Vzi5oCec@]7cw2bQmBATD4Sf;4[\;EdPWMvO>JX5hvhY58dC@
/vol4/dirA1/dirA11/file-V?4LB<^SrhdT^CfGSod5d0eC8vvPf[QMa1wfcUz]:5O_TIett
/vol4/dirA1/dirA11/file-[tM_^[]dykcavgBtuD6Eo@_>8veEJNsKF5nYJ@^uo4:vXz6dA
/vol4/dirA1/dirA11/file-]cbNkxo_]tS2bTOC;TzxIv^>kk6?mazmN7BPE[jYeYD3<=YQX
/vol4/dirA1/dirA11/file-XyA5eT;KV8^wGsk1eCnDG\@4jPS0?OtP3IPAa[TfXU@jT`cF?
/vol4/dirA1/dirA11/file-O@YLYLL6sDqYXG4L=Sg477CTndn_;wF\GB0MmafH\Na^KEuzW
/vol4/dirA1/dirA11/file-xm<9mzj:MVmKE`IGCzs:lU8]7bWlmPxIkrVUDPPNddGdXCk5S
/vol4/dirA1/dirA11/file-agaj]\8_yMz5V@hLuVFm94e6JD0QYSslQgcHgx^oKLXVovCEt
/vol4/dirA1/dirA11/file-3`PC;tshre0ZzgJAb;xoyDszAHuR76dDAuvp5w@95Y7nFYtWf

Issue sync first, to flush dirty pages back to the backing store

sync

Issue this command to invalidate the cache

echo 3 > /proc/sys/vm/drop_caches

This kernel.org page has an interesting list of the various settable values for the linux page cache. http://www.kernel.org/doc/Documentation/sysctl/vm.txt

drop_caches

Writing to this will cause the kernel to drop clean caches, dentries and
inodes from memory, causing that memory to become free.

To free pagecache:
	echo 1 > /proc/sys/vm/drop_caches
To free dentries and inodes:
	echo 2 > /proc/sys/vm/drop_caches
To free pagecache, dentries and inodes:
	echo 3 > /proc/sys/vm/drop_caches

I need to send a MIME encoded email ( a png file) to myself. uuencoded files no longer play well with Outlook 2011 on Mac (no idea why). Thankfully we can replace mailx with mutt to do the MIME encoding for us

diskperf-3650-2:[/tmp] $ mutt -s “From mutt” -a /u/little/charts/spc1_csc_cam.png little@netapp.com

Mutt requires the < /dev/null to avoid going into interactive mode. Presumably it checks to see if input is from stdin and goes interactive if it is. The redirection could also point to some boilerplate text “This is the chart you requested” but redirecting

By default, a string like hello_world, sill be translated to hello{subscript}world when org exports to html. This can be annoying. Luckily we can turn the bahavior off, by using the #+OPTIONS “macro”

#+OPTIONS:   H:3 num:t toc:t \n:nil @:t ::t |:t -:t f:t *:t <:t ^:{}

The actual magic to turn of subscripts is

^:{}

Which says, "interpret hello_{world}" as a subscript directive, but not "hello_world"

lovebox-4:[~/data/traces] $ tshark -r trace.trc -R nfs > trace.out

gnuplot> plot "trace.out" u 2:(stringcolumn(8) eq "WRITE" ? $1:0/0 ) lc 2 t "WRITES" w points, "" u 2:(stringcolumn(8) eq "READ" ? $1:0/0) lc 3 t "READS" w dots
gnuplot> set term png
gnuplot> set output "trace.png"
gnuplot> replot

Typically, I use the GUI version of wireshark to see how to specify the read filter, then use tshark at the command line to make use of all the CLI goodness of Unix.

Display NFS_LOOKUP Calls

 tshark -R "nfs.procedure_v3 == 3" -r sometracefile.trc

Display NFS_GETATTR Calls

 tshark -R "nfs.procedure_v3 == 1" -r sometracefile.trc

Display NFS_SETATTR Calls

 tshark -R "nfs.procedure_v3 == 2" -r sometracefile.trc

Display NFS_ACCESS Calls

 tshark -R "nfs.procedure_v3 == 4" -r sometracefile.trc

…
Display NFS_LINK Calls

 tshark -R "nfs.procedure_v3 == 15" -r sometracefile.trc

When running a lot of unix commands via scripts, it’s quite easy to hit the maximum allowable processes per user, or the maximum number of open files per user.

My OS 10.6.8 mac has the following limits defined.

lovebox-4:[~] $ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 266
virtual memory          (kbytes, -v) unlimited

A user can increase these limits directly from the shell using the ulimit command.

ulimit -n

Changes the maximum allowable number of concurrently open files.

ulimit -u

Changes the maximum allowable concurrent processes.

These limits are in place to stop a user from hurting the overall system for other users. Typically on an OS X system – there is only one user, so it’s quite OK to increase the limits. Another thing to know is that the user can only increase the limit up to the maximum allowed by the system. This system maximum can also be changed (by the system administrator / root) – again on most macs that will be the same user. So how to do this?.

First, let’s see what the system limits are using the command

sysctl

 lovebox-4:[~] $ sysctl -a | egrep '(maxfiles|maxproc)'

kern.maxproc = 532
kern.maxfiles = 12288
kern.maxfilesperproc = 102480
kern.maxprocperuid = 266
kern.maxproc: 532
kern.maxfiles: 12288
kern.maxfilesperproc: 20480
kern.maxprocperuid: 512

Now, I want to change those system maximums, so that I can change the user maximums too.

sudo sysctl -w kern.maxfilesperproc=20480
kern.maxfilesperproc: 10240 -> 20480

sudo sysctl -w kern.maxprocperuid=512
kern.maxprocperuid: 266 -> 512

So, now I have higher system limits, I need to tell OS X that I want to use those larger limits. Again, the system allows the user to protect himself by restricting himself to a lower limit.

Next, open .bashrc and put the new limits in there

ulimit -n 1024
ulimit -u 512

You may not be able to set the new limits directly in an existing shell even though the kernel maximums were changed. However a newly executed shell should have the new limits

e.g. In an existing shell

lovebox-4:[~] $ ulimit -n 1024
-bash: ulimit: open files: cannot modify limit: Operation not permitted

In a newly created shell window

lovebox-4:[~] $ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 512
virtual memory          (kbytes, -v) unlimited

So, here’s something non-intuative

lovebox-4:[~] $ ulimit -n 1024  <--- Works because we set 1024 in bashrc
lovebox-4:[~] $ ulimit -n 512  <---- Lower the limit
lovebox-4:[~] $ ulimit -n 1024  <--- Now it cannot be raised in this shell, or any descendents.
-bash: ulimit: open files: cannot modify limit: Operation not permitted

Now, that's sorted out our shells. To do a similar thing but for processes invoked from 'finder' e.g. Web browser or something. We need to mess with launchd...

lovebox-4:[~] $ sudo launchctl limit
	cpu         unlimited      unlimited
	filesize    unlimited      unlimited
	data        unlimited      unlimited
	stack       8388608        67104768
	core        0              unlimited
	rss         unlimited      unlimited
	memlock     unlimited      unlimited
	maxproc     266            532
	maxfiles    256            unlimited

lovebox-4:[~] $ sudo launchctl limit maxproc 1024
lovebox-4:[~] $ sudo launchctl limit
cpu unlimited unlimited
filesize unlimited unlimited
data unlimited unlimited
stack 8388608 67104768
core 0 unlimited
rss unlimited unlimited
memlock unlimited unlimited
maxproc 1024 1024
maxfiles 256 unlimited

^This may not be persistent, and will only be picked up when new processes are launched. In other words, if an already executing process has reached the maximum number of open files – doing the above will not help until the process is re-started.

After some connectivity swap-a-roos in the lab, I could no longer see my LUNS from the linux host attached to my filer.

In this case I am using a QLogic HBA – and I am not using any of the NetApp host side tools – just the sanlun tool.

Using the SANsurfer Menu (/opt/QLogic_Corporation/SANsurferCLI) I can tell that this linux host can see the filers’ LUNS over FC. But there are no SCSI /dev/sdX devices for them, and so Linux cannot use them…

Here’s how I checked to see that there was FC connectivity – which also confirms that the FC protocol is working.

	SANsurfer FC/CNA HBA CLI

	v1.7.2 Build 7

    Main Menu

    1:	General Information  <---- Option 1
    2:	HBA Information
    3:	HBA Parameters
    4:	Target/LUN List
    5:	iiDMA Settings
...

    General Information Menu

    1:	Host Information
    2:	Host Topology
    3:	Report     <---- Option 3..
    4:	Refresh
    5:	Return to Previous Menu

	Note: 0 to return to Main Menu
	Enter Selection: 1

   Report Menu

    HBA Model QLE2462
      1: Port   1: WWPN: 21-00-00-E0-8B-9B-C5-36 Online
      2: Port   2: WWPN: 21-01-00-E0-8B-BB-C5-36 Online
      3: All HBAs  <---- Option 3
      4: Return to Previous Menu

	Note: 0 to return to Main Menu
	Enter Selection: 3

I could see that there was connectivity from the Linux host to the filer

---------------------------------------
LUN 1
---------------------------------------
Product Vendor                    : NETAPP
Product ID                        : LUN
Product Revision                  : 811a
LUN                               : 1
Size                              : 17.93 GB
Type                              : SBC-2 Direct access block device
			           (e.g., magnetic disk)
WWULN                             : 4E-45-54-41-50-50-20-20-20-4C-55-4E-20-32-46-68
			           72-53-3F-2D-68-4F-79-6C-33-00-00-00-00-00-00-00
OS LUN Name                       :

From the filer side, I could see that the host's FC adapters had connected to the filer,
and were in the right igroup

filer1*> igroup show
    filer1 (FCP) (ostype: linux):
        21:00:00:e0:8b:9b:c5:36 (logged in on: 0a)
        21:01:00:e0:8b:bb:c5:36 (logged in on: 0b)

The only thing that was missing was that there were no 'sd' devices created in Linux for these devices.

"sanlun" utility was not helpful and just told me that there wer no LUNs mapped.

The solution was to issue this very odd looking command

linuxhost:[/sys/class/scsi_host] $ echo "- - -" > host0/scan

This caused the sd devices to be created, representing the NetApp LUNs which I knew could already be seen over FC. Since I have both ports on the same HBA attached to the filer, host0 scan created my /dev/sdc* devices, and host1/scan created my /dev/sdd* devices.

The shell 'hung' for the duration of the command, and I would expect that Linux was off in kernel land for some time - and so i would NOT recommend issuing the command on a production server.

I'm still puzzled why the linux host did not see the luns even after reboot though.