Zimbra Attachment Parsing Bugs

After a morning of troubleshooting poor performance on an archive store that was causing backups on the mtas I would bet we’ve hit a regression. Zimbra doesn’t recognize it.

One of the admins IM’ed me a little after 9am yesterday complaining of queue backups on the mtas delivering to the archive store. They have 5 user stores and 1 archive store.

Connecting to port 7025 (lmtp) on the archive store shows sporadic hangs. Some mail was getting in but thousands were backed up on the mtas.

A quick look at the mailbox.log showed errors connecting to the conversion daemon:

2009-03-05 10:25:54,909 WARN  [LmtpServer-20] [name=user-20080704@domain.tld.archive;
mid=70;] ParsedMessage - Unable to parse part 2 (file.pdf, application/pdf) of 
message with Message-ID <000f01c29c4e$3434d3c0$315c200a@user.domain.tld>.
com.zimbra.cs.mime.MimeHandlerException: cannot extract text
    at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(
DefaultHandler.java:117)    
    at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:148)
    at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:1148)    
    at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:426) 
...
Caused by: com.zimbra.cs.convert.ConversionException: Cannot extract text from 
document that previously crashed convertd.
    at com.zimbra.cs.convert.TransformationClient.extract(
TransformationClient.java:196)
    at com.zimbra.cs.convert.SocketTransformationStub.doExtract(
SocketTransformationStub.java:84)
    at com.zimbra.cs.convert.SocketTransformationStub.extract(
SocketTransformationStub.java:116)
    at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(
DefaultHandler.java:111)
    ... 27 more

So on first look convertd died, I restarted it:
su - zimbra -c "zmconvertctl start"

convertd started and the errors in mailbox.log went away. But kvoop processes started to consume cpu. The load rose to almost 30.

This is the kvoop bug we fixed by turning of attachment indexing.. So I checked that and archiving is on again on the archive cos(!)


zimbra@host log]$ zmprov gc default |grep -i attachment
zimbraAttachmentsBlocked: FALSE
zimbraAttachmentsIndexingEnabled: FALSE
zimbraAttachmentsViewInHtmlOnly: FALSE
[zimbra@host log]$ zmprov gc archive |grep -i attachment
zimbraAttachmentsBlocked: FALSE
zimbraAttachmentsIndexingEnabled: TRUE
zimbraAttachmentsViewInHtmlOnly: FALSE
[zimbra@host log]$ 

zimbra@host log]$ zmprov modifyCos archive zimbraAttachmentsIndexingEnabled 
FALSE
zimbra@host log]$

I flushed the queues and all is well.

Background:

http://wiki.zimbra.com/index.php?title=Disable_Indexing_for_Specific_File_Types


http://bugzilla.zimbra.com/show_bug.cgi?id=14016

Look at the bottom of the bugzilla link: It looks to me like we’re hitting a regression: a bug they thought they fixed in 4.5.x is resurfacing in 5.0.x:


------- Comment  #10 From tech 2008-10-06 03:44:08  -------

this bug seems to have started happening intermittently on my 5.04 rhel_x86
installation it hangs the server so i have to reboot

RSI Log: weeks of 2/2 and 2/9

I spent the week of 2/2 I spent on site with a customer. I took my Kinesis keyboard and iFold stand and various hand rests with me. I was in more consistent pain than I can remember for most of the week. I experimented with various heights of wrist wrests by inserting dense foam (designed for motorcycle race seats). Ultimately I think my chair was too low. Sit in your chair, put your elbows at your sides bent 90 degrees and shrug your shoulders–that was about the level of the keyboard.

2/9 I was back to my regular desk in where I am able to raise the chair high enough that the keyboard is right below my hands with my elbows at my side and bent 90 degrees. That and a gel support for my mouse hand is working well and I am maybe even typing faster than normal. I am not using wrist rests at the keyboard: in theory with my elbows at my sides I am suspending only my hands and forearms. Bellis and Damany advocate no wrist rests.

The nerve “twanginess” is all but gone, replaced by some pain in my elbows around the ulnar nerve. Bending my elbow is consistently less painful this week: I can button a shirt, put on a motorcycle helmet, braid my hair etc, with only minimal pain.

I am finding when I sleep I notice very slight numbness in my hands more than usual, about half the time in the ulnar distribution. It’s slight to the point that it might be in my head. I am sleeping with B more nights than not and she often sleeps on one of my arms.. I find I can rotate my arm slightly or shift my weight to a more bony part of my arm to make the numbness go away.

I discovered this weekend that I have traded sleeping on my arm with my elbow bent for leaning on or contorting my wrist. I’ll need to work on that.

Sun pam_ldap/pam_unix and correct subtree LDAP searches

Though addressed in the context JES Messaging this post is really about getting Sun’s pam_ldap or pam_unix to do what some might consider correct subtree searches. Read on:

If you’re using JES messaging with hosted domains you have an ldap tree that looks like this:
o=firstdomain.com,o=isp; o=seconddomain.com,o=isp; etc.

so your user DNs look like this: uid=morgan,ou=people,o=firstdomain,o=isp; uid=matt,ou=people,o=seconddomain,o=isp; etc.

Under normal circumstances you would supply a basedn of “o=isp,” a scope of “sub,” and the application would search all of your hosted domains in search of the username.

Sun’s pam_ldap (and I believe pam_unix) prepend “ou=people” before doing a search. So if you specify “o=isp,” pam_ldap will search within “ou=people, o=isp” which either doesn’t exist or is empty.

After searching for a way to specify multiple base dns in the ldap profile (defaultSearchBase is SINGLE-VALUE in objectclass DUAConfigProfile) it turns out there is a straightforward work-around. From the ldapclient(1) man page:


serviceSearchDescriptor
           Override the default base DN for LDAP searches  for  a
           given  service.  The  format  of  the descriptors also
           allow overriding the default search scope  and  search
           filter  for  each  service. The syntax of serviceSear-
           chDescriptor is defined in  the  profile  IETF  draft.
           The  default value for all services is NULL. This is a
           multivalued attribute. In the example,

           serviceSearchDescriptor=passwd:ou=people,dc=a1,dc=acme,dc=com?one

           the LDAP  client  would  do  a  one  level  search  in
           ou=people,dc=a1,dc=acme,dc=com       rather       than
           ou=people,defaultSearchBase for the passwd service.
 

So set serviceSearchDescriptor=passwd:o=isp and it will search under o=isp, allowing users in all of your hosted domains to authenticate. Of course this does open you up to problems where uids can conflict.

RSI log

Bellis and Damany recommend keeping a log. Here’s a quick two month summary:

Prior to 11/29/08: I remember periodic pain in the palm of my hands below the ring and pinky finger, particularly when using a laptop. My first memory of this is around October, 06. Infrequent pain in my forearms along the path of the ulnar nerve (elbow to base of palm).

11/29/08: I noticed consistent pain from my elbow to the base of my palm for several days straight over the Thanksgiving holiday. I searched the Internet and self diagnosed cubital tunnel syndrome. I immediately began sleeping with my arms straight. When I returned I went to my physician and she referred me to a hand surgeon.

About a week later: my pain changed to consistent pain in my elbow with periodic nerve “twanginess” from my elbow into my hand: imagine the feeling immediately after the initial shock of hitting your funny bone all the time in both arms. This was consistent until mid January.

Bending my arms any more than 30 degrees is a 6-7 in pain and I have to brace myself mentally to button my shirt, touch my face, drink from a glass, braid my hair, wash my hair, etc.

12/9/08: (check date) I met with a surgeon at the Hand Center at Jefferson. She confirmed my diagnosis, told me to continue sleeping with my arms straight, talked a little too much about surgery for my tastes and ordered an EMG to assess the severity of the damage to my nerves.

12/23/08: The EMG verified the diagnosis of cubital tunnel but indicated marginal severity: there was damage but it was light to moderate. It also showed mild carpal tunnel syndrome. I met again with the surgeon and she prescribed physical therapy and asked that I return in six months for another EMG. I remember telling her that I felt I had about a 10% improvement in pain but couldn’t be sure as consistent pain can be hard to gauge. I also gained some confidence in the surgeon that day as she was no longer talking about surgery: she was taking a conservative approach.

It was at about this time that I began taking 400mg of Aleve twice a day: at breakfast and dinner.

1/5/09: I met with the physical therapist, she found no measurable loss of sensation or motor function. Loss of sensation and motor function are the next step in the syndrome and I am not there… I told her I felt I had improved an additional 10% but was still not sure. She gave me median nerve glides which also work the ulnar nerve a little, hand exercises for carpal tunnel syndrome. She also asked I return the following week with photos of where I work so we could evaluate ergonomics.

1/12/08: I returned to physical therapy with photos of me on the couch, feet up, laptop in my lap, slouched in front of my desk, sitting at the kitchen table and so on. She was appalled but not surprised. She showed me the proper seating position and prescribed a split keyboard, a laptop stand and a keyboard tray. I’m tall enough that the key board tray has proven unnecessary but I am now an expert on and believer in split keyboards. I currently own two. I type with almost no hand pain now. If I remember correctly I was still in pain in my elbows more often than not but it was less at times. It is still uncomfortable to bend more than 30 degrees.

1/28/08: I returned to physical therapy with my Goldtouch keyboard and photos of me in a correct position. J, my physical therapist approved. I am now pain free 10% of the time, have nerve twanginess/awareness of the ulnar nerve from my elbow into my hand about 70% of the time and 4-5 pain 20% of the time. Activities requiring extreme arm bending (buttoning my shirt, braiding my hair, washing my hair, rubbing my face) have gone from a 6-7 in pain to a 1-2. I would say I notice these activities now rather than bracing for them.

2/4/08: I am pain free about 60-70% of the time. I seldom have serious pain or awareness of the full run of the ulnar nerve from my elbow into my hand. I have minor pain in my hand, sometimes my forearm. Activities involving extreme elbow bending still hurt about a 1-2.

reading and more insight into RSI

I’m reading Bellis and Damany’s It’s Not Carpal Tunnel Syndrome! RSI Theory & Therapy for Computer Professionals. Their thesis is that many RSIs won’t be solved by surgery, at least not without an understanding of the larger system and it is possible to properly treat an RSI without giving up computer use.

Bellis is a computer professional who had surgery, his symptoms came back only to be resolved by working with an experience physical therapist: Damany. Their advice is in line with what my physical therapist is recommending and it’s working for me. I am now just over two months into my treatment of my RSIs and I am seeing improvement.

Of particular interest is the idea that just fixing an RSI through surgery, though often providing immediate relief will not provide long term relief if you don’t address the underlying cause of the damage. The hands, arms, back, neck, etc. work as a system: just operating on the wrist or the elbow doesn’t address the rest of the system. Further, in many cases RSI can be relieved by just addressing the cause and skipping the surgery altogether. That speaks to me.

Particularly interesting about nerve damage is there is really no way to strengthen or condition nerves. Physical therapy for muscles usually involves strengthening and the result is progressive rebuilding. Physical therapy for nerves involves moving your [arms in my case] in a way to take them through their full range of motion but otherwise just modifying behavior to not put stress on the nerves and hope they heal. And they do heal but over a course of months.

Sensory feedback from muscles is consistent: they hurt when you exercise them and then they heal.. While nerves presumably heal progressively they send confusing, misleading messages about what is going on: one day it will be pain in the hand, another it will be itching “inside” your hand that can’t be scratched. Some days I’ll work most of the day on the keyboard in pain only to have it clear up three quarters of the way through the day and not hurt at all the rest of the day even though I’m working as much as I was in the morning.

I am also a little surprised about how little modern medicine knows about RSIs.. Part of the problem according to Bellis and Damany is that modern medicine treats the body as regions and doctors are trained to focus on one part and so don’t always look at the big picture before treating the location of the pain: damage to the median nerve where it passes through the carpal tunnel for instance is often the result of bad posture and if you operate but don’t’ address the bad posture you will end up with the same symptoms sometimes in a matter of weeks.

It’s your responsibility as a patient to talk to multiple people and try to understand what is going on and help guide your treatment.. Your doctor may well know less about your condition than you do.. Bellis and Damany suggest physical therapy with a focus on the whole system and not just the wrist or just the elbow the most effective way to treat an RSI.

ergonomics

We addressed computer ergonomics today in physical therapy. I am of course a mess. The hardest part of all of this is how tightly the convenience of a laptop is intertwined into my life.. My early feelings on the matter are that I’m going to have have to set up my desk at home and any office I work for more than a day or two. It’s vaguely depressing to drag around a split keyboard and stand everywhere I go but I keep reminding myself it beats surgery..

Here’s an example of what she recommended:
http://www.medicinenet.com/script/main/art.asp?articlekey=22781

Basically: sit back in a chair with lumbar support. It should push your shoulders back. Your knees should be 90 degrees, your ankles should be 90 degrees. Your chair should of course be high enough to allow your knees to be 90 degrees. The top of the monitor should be level with your forehead. Your arms should be at your sides, 90-100 degree bend in your elbows, wrists neutral. My PT says you usually want a keyboard tray to place the keyboard at the right height but in my case I may be able to get away without one because I’m tall enough to reach the keyboard on a table and maintain a neutral wrist position. Hands should bend in naturally.. You want a keyboard that takes minimal finger effort to type on. You should use as little force as possible while typing.

What this means for a traveling laptop user is a portable notebook stand and a split keyboard:
http://store.ergocube.com/gotrnost5.html and
http://www.amazon.com/Goldtouch-Apple-Compatible-Keyboard-Black/dp/B001IIP9UY/ref=sr_1_1?ie=UTF8&s=electronics&qid=1231819997&sr=8-1

You can buy them both at goldtouch.com but their return policy is draconian and I have never seen either in person.

Yes, that’s a Mac keyboard.. draw your own conclusions. No, it hasn’t arrived yet.

A conventional mouse is fine.. so I’m just going to carry one of of the dozen or so Dell mice hanging in my closet.. eventually I’ll pick up a bluetooth mouse.

Traditional molded ergonomic keyboards are apparently bad: they’re one size fits all, ie don’t allow adjustment. Keyboards that tilt toward you are bad. If a keyboard has feet you’re better off folding them down and laying it flat.

I find it surprising that no one makes a bluetooth split keyboard. Goldtouch will have a travel split keyboard in February: http://www.goldtouch.com/p-139-goldtouch-go-travel-keyboard.aspx

Join Casino Free Philadelphia for the launch of their latest campaign tomorrow

Please join us tomorrow, Wednesday evening, from 6-8:30pm at the offices of Liberty Resources, 714 Market Street, as we unveil our newest campaign during this critical phase.

As you know, we have been working for some time to brainstorm and plan. We’ve talked to many of you; we’ve analyzed the status of the various permit applications, the financial situation and the legal landscape. We’ve listened and we’ve even adjusted our game plan. Based on the external events, we believe our movement now has a short but incredibly valuable window of opportunity — about 6 months — to change hearts and minds. And, that’s just what we’re going to do, with your help and dedication.

Join us tomorrow, Wednesday evening, from 6-8:30pm, as we Declare Our Independence from Casinos. Join us to learn about our upcoming search for an elected official who will stand with us. Join us as we prepare to draft our manifesto against casinos in our City. Join us as we set out to organize citizens from all across Philadelphia.

At the meeting we’ll present to you an ambitious, necessary and forceful campaign; and we’ll continue to incorporate your input. We will provide an opportunity for you to plug into our efforts and become responsible for specific tasks. The time is right for you to become even more involved, even more committed and even more protective of your beloved City.

Bring your neighbors, bring your friends and bring your passion.

Warmly,

Amy, Ivan, Jethro and the entire team at Casino-Free Philadelphia

repetitive stress take 1

So I’m 34, a computer consultant and I’ve worked and played on computers since I was in third grade. It should come as no surprise that I’ve been diagnosed with Cubital Tunnel and Carpal Tunnel Syndrome in both arms.

Basically my ulnar nerve which passes from the neck through the two bumps at the base of the elbow and to the hand is damaged at the elbow. I had nearly constant pain in my elbows which was exacerbated by bending my elbows more than 30 or so degrees.

While typing is probably a major contributor I believe sleeping with my arms bent is a significant factor. More on this later.

The therapist pointed out just today that if I tap my wrist just below my palm I get shooting sensations up my index and middle finger. This I believe is the carpal tunnel.

I won’t go into too many more details of cubital or carpal tunnel syndromes as there’s plenty of descriptions of each online but my experience might be helpful. I have read some really great accounts of the surgery but not so many of the early symptoms or conservative approach to treatment. I am of course not a doctor so please take this as my experience.

I have had pain in my hands off and on most of my adult life–I type for a living and it comes with the territory. Around the time I became self employed and thus started working more I began to get pain and numbness in the side of the hand with the pinky and ring fingers. That was almost two years ago. What I didn’t know then is that is the first signs of cubital tunnel syndrome. Knowing what I know now I would have gone to a hand specialist immediately.

Thanksgiving of last year, about 6 weeks ago, I began to have constant pain in my forearms from the palm of my hand up to my elbows. The pain didn’t go away so I called my doctor and got a referral to the Philadelphia Hand Center.

I met with a surgeon I had been to many years ago for a minor hand injury. She called for an EMG to determine the severity of the damage but talked about surgery a little too early and spooked me. After reading up on the surgery and talking to smart people in my life who have been through similar situations I resolved to get a second opinion if she called for surgery. The EMG showed about 43 in my left elbow and 47 in my right. The therapist who gave the EMG was both very good and very patient.

The surgery for Cubital Tunnel syndrome by all accounts very straightforward and generally successful. It does require six or so weeks of no or limited use of the arm and in my case potentially both arms one after the other. There is potential for serious side effects I think mostly because there’s potential for damaging the nerve. Either way I prefer a conservative approach.

The EMG is brutal but manageable–it involves sending electric pulses from different places on your arm and your wrist and measuring the subsequent pulse generated by the nerve. The voltage is benign but it is really surprising. The needle part involves pushing inch and a half needles into the three meaty parts of your hands.. it’s not so bad if you close your eyes and your therapist knows what he’s doing. It’s also key to diagnosing the severity of the problem so do try to be patient.

Early treatment is sleeping with your arms straight, bending your arms as little as possible during the day. This means drinking, eating, showering, putting on a motorcycle helmet, buttoning your shirt, hugging your gf are all painful. It sucks. Anti inflammatories make little difference in the pain but long term may help. Icing my elbow helps a lot. I bought a brace but I was surprised at how quickly I learned to sleep with my arms straight. It still bothers me a little but it only took waking up with half my hand numb once or twice to scare me into straightening my arms. My arms are a little sore after six weeks of minimal bending.

Just from minimizing elbow bending and maybe regular anti inflammatories I have seen a 5-10% improvement in my symptoms. Right now I can button my shirt, braid my hair and eat and drink with little to minor pain. Braiding my hair in particular was a 4 or 5 on a scale of 1-10.

The surgeon says I’m in a gray area and I may benefit from physical therapy. I went in today for my first session. J, my therapist, worked me through various tests to identify the severity: I have no measurable loss of sensation or strength and no signs of atrophy. J feels I am a good candidate for a full recovery from physical therapy. I pressed her for a time frame and she feels I should see a full recover in 8-10 weeks.

I’ll dig up some online examples of the exercises she gave me since I probably shouldn’t publish the paperwork I have here. I absolutely love the carpal tunnel exercises–I am to do them once an hour while I type. They seem work the muscles you use when you type and it feels great on tired hands.

Simple Perl Multi Processing

One of my complaints about programming is that it’s often overcomplicated. Sometimes you need a simple solution with minimal dependencies that gets the job done. This is certainly true of utility scripts designed to run on more than one system or in divergent environments.

I have a section of code that must be run over and over, takes several seconds to run, doesn’t put much load on the system and the overall script run time is too long–a perfect case for multi processing or multi threading. It’s written in perl and the server has plenty of memory and cpu so so I chose multi processing.

Perl multi-processing is implemented on top of UNIX multi processing so the how to reads like a computer science class on UNIX programming. Here’s a working prototype that took a lot of tweaking to develop. Details of problems I encountered are below the code.


#!/usr/bin/perl -w
#
# simple multi processes prototype
# Morgan Jones (mjones7@morganjones.org)
# $Id: multi_proc_proto.pl 125 2008-12-29 21:50:41Z morgan $

use strict;
use POSIX ":sys_wait_h";

my $parallelism = 4;  # number of processes to run simultaneously.
my $pids;  # keep track of PIDs as child processes run

for (my $i=0; $i<100; $i++) {
    my $proc_running = 0;  # flag to indicate a process has been started.

    do {
	my $pidcount = keys %$pids;
	print "pidcount: $pidcount, parallelism: $parallelismn";

	if ($pidcount < $parallelism) {
	    print "starting \"job\" $in";
	    my $pid = fork();

	    if (defined $pid && $pid == 0) {
		# do the meat of the work here..
		my $t;
		do { $t = int (rand(3)); } until ($t>0);
		sleep $t;
		exit 0;
	    } elsif (defined $pid && $pid > 0) {
		# keep track of running processes
		$pids->{$pid} = 1;
		$proc_running++;
	    } else {
		# TODO: count number of failures and exit after too many
		print "problem forking..Sleeping and retrying..n";
		sleep 1;
		next;
	    }
	} else {
	    my $proc_reclaimed = 0;

	    for my $p (keys %$pids) {
		# be careful waiting on $p or substituting'0' for WNOHANG
		my $ret = waitpid(-1, WNOHANG);

		if ($ret<0 || $ret>0) {
		    print "process $ret finished..n";
		    delete $pids->{$ret};
		    $proc_reclaimed = 1;
		}
	    }
	    # only sleep if one or more process didn't finish..  this
	    #   allows us to spin off a new process if one's available
	    #   but keeps us from busy waiting if all the processes
	    #   are busy.

	    unless ($proc_reclaimed) {
		print "sleeping: no processes reclaimed...n";
		sleep 1;
	    }
	}
    } until ($proc_running);
}

My initial version did a waitpid($p, WNOHANG), I also tried waitpid(-1, 0). In both cases it caused the load on the system to go up very quickly, eventually locking the system completely. Searching out result sets less of a few hundred doesn't exhibit the problem.

Running it through the debugger showed the child process finishing. strace showed the process completing. ps would show the process still running. The only symptom was a high sys cpu utilization.

pumpkin pork stew

Leftover ribs from a meat worship party? Pumpkin you don’t know what to do with leftover from a fall stew?

6-10 leftover pork ribs
1/2 a small pumpkin, seeded, skinned and chopped
3-6 small red potatoes chopped
dried marjoram
dried parsley
salt and pepper
4-6 cloves garlic crushed or chopped
16 can of tomatoes including juice
1/2 package of frozen okra (optional)
1/2 cup dry rice

Put the ribs in a pot with just enough water to cover them, simmer until the meat falls off: usually 2-4 hours. The longer you simmer the less the flavor. You may need to add water but only add enough to cover the ribs.

Strain the meat over a bowl. Pick out the bones and cartilage, reserve the meat in a covered dish. Put the broth back in the pot.

Add potatoes, pumpkin, garlic, tomatoes, rice and spices to taste–probably 1/2 – 1 tsp each of the spices. Simmer until the potatoes and pumpkin are slightly cooked, probably 5 minutes. Add okra and reserved meat.

Simmer covered until the rice is done (not crunchy) if you like your vegetables al dente. Really you can simmer as long as like. Add water throughout if it’s too thick.