Zimbra Attachment Parsing Bugs

After a morning of troubleshooting poor performance on an archive store that was causing backups on the mtas I would bet we’ve hit a regression. Zimbra doesn’t recognize it.

One of the admins IM’ed me a little after 9am yesterday complaining of queue backups on the mtas delivering to the archive store. They have 5 user stores and 1 archive store.

Connecting to port 7025 (lmtp) on the archive store shows sporadic hangs. Some mail was getting in but thousands were backed up on the mtas.

A quick look at the mailbox.log showed errors connecting to the conversion daemon:

2009-03-05 10:25:54,909 WARN  [LmtpServer-20] [name=user-20080704@domain.tld.archive;
mid=70;] ParsedMessage - Unable to parse part 2 (file.pdf, application/pdf) of 
message with Message-ID <000f01c29c4e$3434d3c0$315c200a@user.domain.tld>.
com.zimbra.cs.mime.MimeHandlerException: cannot extract text
    at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(
DefaultHandler.java:117)    
    at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:148)
    at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:1148)    
    at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:426) 
...
Caused by: com.zimbra.cs.convert.ConversionException: Cannot extract text from 
document that previously crashed convertd.
    at com.zimbra.cs.convert.TransformationClient.extract(
TransformationClient.java:196)
    at com.zimbra.cs.convert.SocketTransformationStub.doExtract(
SocketTransformationStub.java:84)
    at com.zimbra.cs.convert.SocketTransformationStub.extract(
SocketTransformationStub.java:116)
    at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(
DefaultHandler.java:111)
    ... 27 more

So on first look convertd died, I restarted it:
su - zimbra -c "zmconvertctl start"

convertd started and the errors in mailbox.log went away. But kvoop processes started to consume cpu. The load rose to almost 30.

This is the kvoop bug we fixed by turning of attachment indexing.. So I checked that and archiving is on again on the archive cos(!)


zimbra@host log]$ zmprov gc default |grep -i attachment
zimbraAttachmentsBlocked: FALSE
zimbraAttachmentsIndexingEnabled: FALSE
zimbraAttachmentsViewInHtmlOnly: FALSE
[zimbra@host log]$ zmprov gc archive |grep -i attachment
zimbraAttachmentsBlocked: FALSE
zimbraAttachmentsIndexingEnabled: TRUE
zimbraAttachmentsViewInHtmlOnly: FALSE
[zimbra@host log]$ 

zimbra@host log]$ zmprov modifyCos archive zimbraAttachmentsIndexingEnabled 
FALSE
zimbra@host log]$

I flushed the queues and all is well.

Background:

http://wiki.zimbra.com/index.php?title=Disable_Indexing_for_Specific_File_Types


http://bugzilla.zimbra.com/show_bug.cgi?id=14016

Look at the bottom of the bugzilla link: It looks to me like we’re hitting a regression: a bug they thought they fixed in 4.5.x is resurfacing in 5.0.x:


------- Comment  #10 From tech 2008-10-06 03:44:08  -------

this bug seems to have started happening intermittently on my 5.04 rhel_x86
installation it hangs the server so i have to reboot

RSI Log: weeks of 2/2 and 2/9

I spent the week of 2/2 I spent on site with a customer. I took my Kinesis keyboard and iFold stand and various hand rests with me. I was in more consistent pain than I can remember for most of the week. I experimented with various heights of wrist wrests by inserting dense foam (designed for motorcycle race seats). Ultimately I think my chair was too low. Sit in your chair, put your elbows at your sides bent 90 degrees and shrug your shoulders–that was about the level of the keyboard.

2/9 I was back to my regular desk in where I am able to raise the chair high enough that the keyboard is right below my hands with my elbows at my side and bent 90 degrees. That and a gel support for my mouse hand is working well and I am maybe even typing faster than normal. I am not using wrist rests at the keyboard: in theory with my elbows at my sides I am suspending only my hands and forearms. Bellis and Damany advocate no wrist rests.

The nerve “twanginess” is all but gone, replaced by some pain in my elbows around the ulnar nerve. Bending my elbow is consistently less painful this week: I can button a shirt, put on a motorcycle helmet, braid my hair etc, with only minimal pain.

I am finding when I sleep I notice very slight numbness in my hands more than usual, about half the time in the ulnar distribution. It’s slight to the point that it might be in my head. I am sleeping with B more nights than not and she often sleeps on one of my arms.. I find I can rotate my arm slightly or shift my weight to a more bony part of my arm to make the numbness go away.

I discovered this weekend that I have traded sleeping on my arm with my elbow bent for leaning on or contorting my wrist. I’ll need to work on that.

Sun pam_ldap/pam_unix and correct subtree LDAP searches

Though addressed in the context JES Messaging this post is really about getting Sun’s pam_ldap or pam_unix to do what some might consider correct subtree searches. Read on:

If you’re using JES messaging with hosted domains you have an ldap tree that looks like this:
o=firstdomain.com,o=isp; o=seconddomain.com,o=isp; etc.

so your user DNs look like this: uid=morgan,ou=people,o=firstdomain,o=isp; uid=matt,ou=people,o=seconddomain,o=isp; etc.

Under normal circumstances you would supply a basedn of “o=isp,” a scope of “sub,” and the application would search all of your hosted domains in search of the username.

Sun’s pam_ldap (and I believe pam_unix) prepend “ou=people” before doing a search. So if you specify “o=isp,” pam_ldap will search within “ou=people, o=isp” which either doesn’t exist or is empty.

After searching for a way to specify multiple base dns in the ldap profile (defaultSearchBase is SINGLE-VALUE in objectclass DUAConfigProfile) it turns out there is a straightforward work-around. From the ldapclient(1) man page:


serviceSearchDescriptor
           Override the default base DN for LDAP searches  for  a
           given  service.  The  format  of  the descriptors also
           allow overriding the default search scope  and  search
           filter  for  each  service. The syntax of serviceSear-
           chDescriptor is defined in  the  profile  IETF  draft.
           The  default value for all services is NULL. This is a
           multivalued attribute. In the example,

           serviceSearchDescriptor=passwd:ou=people,dc=a1,dc=acme,dc=com?one

           the LDAP  client  would  do  a  one  level  search  in
           ou=people,dc=a1,dc=acme,dc=com       rather       than
           ou=people,defaultSearchBase for the passwd service.
 

So set serviceSearchDescriptor=passwd:o=isp and it will search under o=isp, allowing users in all of your hosted domains to authenticate. Of course this does open you up to problems where uids can conflict.

RSI log

Bellis and Damany recommend keeping a log. Here’s a quick two month summary:

Prior to 11/29/08: I remember periodic pain in the palm of my hands below the ring and pinky finger, particularly when using a laptop. My first memory of this is around October, 06. Infrequent pain in my forearms along the path of the ulnar nerve (elbow to base of palm).

11/29/08: I noticed consistent pain from my elbow to the base of my palm for several days straight over the Thanksgiving holiday. I searched the Internet and self diagnosed cubital tunnel syndrome. I immediately began sleeping with my arms straight. When I returned I went to my physician and she referred me to a hand surgeon.

About a week later: my pain changed to consistent pain in my elbow with periodic nerve “twanginess” from my elbow into my hand: imagine the feeling immediately after the initial shock of hitting your funny bone all the time in both arms. This was consistent until mid January.

Bending my arms any more than 30 degrees is a 6-7 in pain and I have to brace myself mentally to button my shirt, touch my face, drink from a glass, braid my hair, wash my hair, etc.

12/9/08: (check date) I met with a surgeon at the Hand Center at Jefferson. She confirmed my diagnosis, told me to continue sleeping with my arms straight, talked a little too much about surgery for my tastes and ordered an EMG to assess the severity of the damage to my nerves.

12/23/08: The EMG verified the diagnosis of cubital tunnel but indicated marginal severity: there was damage but it was light to moderate. It also showed mild carpal tunnel syndrome. I met again with the surgeon and she prescribed physical therapy and asked that I return in six months for another EMG. I remember telling her that I felt I had about a 10% improvement in pain but couldn’t be sure as consistent pain can be hard to gauge. I also gained some confidence in the surgeon that day as she was no longer talking about surgery: she was taking a conservative approach.

It was at about this time that I began taking 400mg of Aleve twice a day: at breakfast and dinner.

1/5/09: I met with the physical therapist, she found no measurable loss of sensation or motor function. Loss of sensation and motor function are the next step in the syndrome and I am not there… I told her I felt I had improved an additional 10% but was still not sure. She gave me median nerve glides which also work the ulnar nerve a little, hand exercises for carpal tunnel syndrome. She also asked I return the following week with photos of where I work so we could evaluate ergonomics.

1/12/08: I returned to physical therapy with photos of me on the couch, feet up, laptop in my lap, slouched in front of my desk, sitting at the kitchen table and so on. She was appalled but not surprised. She showed me the proper seating position and prescribed a split keyboard, a laptop stand and a keyboard tray. I’m tall enough that the key board tray has proven unnecessary but I am now an expert on and believer in split keyboards. I currently own two. I type with almost no hand pain now. If I remember correctly I was still in pain in my elbows more often than not but it was less at times. It is still uncomfortable to bend more than 30 degrees.

1/28/08: I returned to physical therapy with my Goldtouch keyboard and photos of me in a correct position. J, my physical therapist approved. I am now pain free 10% of the time, have nerve twanginess/awareness of the ulnar nerve from my elbow into my hand about 70% of the time and 4-5 pain 20% of the time. Activities requiring extreme arm bending (buttoning my shirt, braiding my hair, washing my hair, rubbing my face) have gone from a 6-7 in pain to a 1-2. I would say I notice these activities now rather than bracing for them.

2/4/08: I am pain free about 60-70% of the time. I seldom have serious pain or awareness of the full run of the ulnar nerve from my elbow into my hand. I have minor pain in my hand, sometimes my forearm. Activities involving extreme elbow bending still hurt about a 1-2.

reading and more insight into RSI

I’m reading Bellis and Damany’s It’s Not Carpal Tunnel Syndrome! RSI Theory & Therapy for Computer Professionals. Their thesis is that many RSIs won’t be solved by surgery, at least not without an understanding of the larger system and it is possible to properly treat an RSI without giving up computer use.

Bellis is a computer professional who had surgery, his symptoms came back only to be resolved by working with an experience physical therapist: Damany. Their advice is in line with what my physical therapist is recommending and it’s working for me. I am now just over two months into my treatment of my RSIs and I am seeing improvement.

Of particular interest is the idea that just fixing an RSI through surgery, though often providing immediate relief will not provide long term relief if you don’t address the underlying cause of the damage. The hands, arms, back, neck, etc. work as a system: just operating on the wrist or the elbow doesn’t address the rest of the system. Further, in many cases RSI can be relieved by just addressing the cause and skipping the surgery altogether. That speaks to me.

Particularly interesting about nerve damage is there is really no way to strengthen or condition nerves. Physical therapy for muscles usually involves strengthening and the result is progressive rebuilding. Physical therapy for nerves involves moving your [arms in my case] in a way to take them through their full range of motion but otherwise just modifying behavior to not put stress on the nerves and hope they heal. And they do heal but over a course of months.

Sensory feedback from muscles is consistent: they hurt when you exercise them and then they heal.. While nerves presumably heal progressively they send confusing, misleading messages about what is going on: one day it will be pain in the hand, another it will be itching “inside” your hand that can’t be scratched. Some days I’ll work most of the day on the keyboard in pain only to have it clear up three quarters of the way through the day and not hurt at all the rest of the day even though I’m working as much as I was in the morning.

I am also a little surprised about how little modern medicine knows about RSIs.. Part of the problem according to Bellis and Damany is that modern medicine treats the body as regions and doctors are trained to focus on one part and so don’t always look at the big picture before treating the location of the pain: damage to the median nerve where it passes through the carpal tunnel for instance is often the result of bad posture and if you operate but don’t’ address the bad posture you will end up with the same symptoms sometimes in a matter of weeks.

It’s your responsibility as a patient to talk to multiple people and try to understand what is going on and help guide your treatment.. Your doctor may well know less about your condition than you do.. Bellis and Damany suggest physical therapy with a focus on the whole system and not just the wrist or just the elbow the most effective way to treat an RSI.

ergonomics

We addressed computer ergonomics today in physical therapy. I am of course a mess. The hardest part of all of this is how tightly the convenience of a laptop is intertwined into my life.. My early feelings on the matter are that I’m going to have have to set up my desk at home and any office I work for more than a day or two. It’s vaguely depressing to drag around a split keyboard and stand everywhere I go but I keep reminding myself it beats surgery..

Here’s an example of what she recommended:
http://www.medicinenet.com/script/main/art.asp?articlekey=22781

Basically: sit back in a chair with lumbar support. It should push your shoulders back. Your knees should be 90 degrees, your ankles should be 90 degrees. Your chair should of course be high enough to allow your knees to be 90 degrees. The top of the monitor should be level with your forehead. Your arms should be at your sides, 90-100 degree bend in your elbows, wrists neutral. My PT says you usually want a keyboard tray to place the keyboard at the right height but in my case I may be able to get away without one because I’m tall enough to reach the keyboard on a table and maintain a neutral wrist position. Hands should bend in naturally.. You want a keyboard that takes minimal finger effort to type on. You should use as little force as possible while typing.

What this means for a traveling laptop user is a portable notebook stand and a split keyboard:
http://store.ergocube.com/gotrnost5.html and
http://www.amazon.com/Goldtouch-Apple-Compatible-Keyboard-Black/dp/B001IIP9UY/ref=sr_1_1?ie=UTF8&s=electronics&qid=1231819997&sr=8-1

You can buy them both at goldtouch.com but their return policy is draconian and I have never seen either in person.

Yes, that’s a Mac keyboard.. draw your own conclusions. No, it hasn’t arrived yet.

A conventional mouse is fine.. so I’m just going to carry one of of the dozen or so Dell mice hanging in my closet.. eventually I’ll pick up a bluetooth mouse.

Traditional molded ergonomic keyboards are apparently bad: they’re one size fits all, ie don’t allow adjustment. Keyboards that tilt toward you are bad. If a keyboard has feet you’re better off folding them down and laying it flat.

I find it surprising that no one makes a bluetooth split keyboard. Goldtouch will have a travel split keyboard in February: http://www.goldtouch.com/p-139-goldtouch-go-travel-keyboard.aspx

Join Casino Free Philadelphia for the launch of their latest campaign tomorrow

Please join us tomorrow, Wednesday evening, from 6-8:30pm at the offices of Liberty Resources, 714 Market Street, as we unveil our newest campaign during this critical phase.

As you know, we have been working for some time to brainstorm and plan. We’ve talked to many of you; we’ve analyzed the status of the various permit applications, the financial situation and the legal landscape. We’ve listened and we’ve even adjusted our game plan. Based on the external events, we believe our movement now has a short but incredibly valuable window of opportunity — about 6 months — to change hearts and minds. And, that’s just what we’re going to do, with your help and dedication.

Join us tomorrow, Wednesday evening, from 6-8:30pm, as we Declare Our Independence from Casinos. Join us to learn about our upcoming search for an elected official who will stand with us. Join us as we prepare to draft our manifesto against casinos in our City. Join us as we set out to organize citizens from all across Philadelphia.

At the meeting we’ll present to you an ambitious, necessary and forceful campaign; and we’ll continue to incorporate your input. We will provide an opportunity for you to plug into our efforts and become responsible for specific tasks. The time is right for you to become even more involved, even more committed and even more protective of your beloved City.

Bring your neighbors, bring your friends and bring your passion.

Warmly,

Amy, Ivan, Jethro and the entire team at Casino-Free Philadelphia

Modern consumers

Modern consumers are attracted to online shopping by numerous advantages it offers. In fact, 78% of Millennials already shop online at least once a month, while the number rises to 91% for Millennials aged 35 to 44. As shopping becomes a more permanent experience, this increase in participation may not necessarily signal a drop in demand.

Ecommerce Website E-Commerce Application Development Service, Pan India, |  ID: 19891941688

Online retailers’ combined revenues are predicted to increase from $7.5 trillion in 2015 to $17.1 trillion in 2019, which is a 27% jump over 2014 levels, driven by shopping driven by Internet-based delivery services. Online retailers offer greater choices in offers and delivery options compared to brick and mortar shops. This trend will continue as online retailers develop unique options in everything from options to rent a car to travel insurance to delivery options, creating a more appealing shopping environment.

With the rise of online shopping, consumers are opting for convenience over order accuracy. Millennials prefer buying things online, because they are able to find much more discounts online on websites likeĀ Raise and they also prefer to pay with a mobile wallet such as a smartphone, the increasing convenience and ease of mobile shopping and alluring for Millennials may create an opportunity for online retailers.

Wholesale shoppers are also less likely to put forward an intention to return merchandise after a purchase. On average, Millennials are eight times more likely than other age groups to say they will definitely buy something from the store where they made their purchase. Given these trends, which could reverse in the future, there is a potential for the store-owned sector to benefit from Millennial shoppers’ trends to stay ahead. For example, as Millennials increasingly become the largest group of online shoppers, retailers should look at how to increase their share of Millennial shoppers with discounts and special offers.

When it comes to Millennials, the retail industry has a lot to look forward to. In fact, they make up 48% of the worldwide retail market. If Millennials continue to grow at their current rates, they’ll account for 54% of the global market in 2019. When it comes to shopping online, the Millennial community is so powerful that it should not be underestimated, and retailers in the U.S. are at the forefront of experimenting with online shopping platforms and products, both for increased margins and brand loyalty.

Simple Perl Multi Processing

One of my complaints about programming is that it’s often overcomplicated. Sometimes you need a simple solution with minimal dependencies that gets the job done. This is certainly true of utility scripts designed to run on more than one system or in divergent environments.

I have a section of code that must be run over and over, takes several seconds to run, doesn’t put much load on the system and the overall script run time is too long–a perfect case for multi processing or multi threading. It’s written in perl and the server has plenty of memory and cpu so so I chose multi processing.

Perl multi-processing is implemented on top of UNIX multi processing so the how to reads like a computer science class on UNIX programming. Here’s a working prototype that took a lot of tweaking to develop. Details of problems I encountered are below the code.


#!/usr/bin/perl -w
#
# simple multi processes prototype
# Morgan Jones (mjones7@morganjones.org)
# $Id: multi_proc_proto.pl 125 2008-12-29 21:50:41Z morgan $

use strict;
use POSIX ":sys_wait_h";

my $parallelism = 4;  # number of processes to run simultaneously.
my $pids;  # keep track of PIDs as child processes run

for (my $i=0; $i<100; $i++) {
    my $proc_running = 0;  # flag to indicate a process has been started.

    do {
	my $pidcount = keys %$pids;
	print "pidcount: $pidcount, parallelism: $parallelismn";

	if ($pidcount < $parallelism) {
	    print "starting \"job\" $in";
	    my $pid = fork();

	    if (defined $pid && $pid == 0) {
		# do the meat of the work here..
		my $t;
		do { $t = int (rand(3)); } until ($t>0);
		sleep $t;
		exit 0;
	    } elsif (defined $pid && $pid > 0) {
		# keep track of running processes
		$pids->{$pid} = 1;
		$proc_running++;
	    } else {
		# TODO: count number of failures and exit after too many
		print "problem forking..Sleeping and retrying..n";
		sleep 1;
		next;
	    }
	} else {
	    my $proc_reclaimed = 0;

	    for my $p (keys %$pids) {
		# be careful waiting on $p or substituting'0' for WNOHANG
		my $ret = waitpid(-1, WNOHANG);

		if ($ret<0 || $ret>0) {
		    print "process $ret finished..n";
		    delete $pids->{$ret};
		    $proc_reclaimed = 1;
		}
	    }
	    # only sleep if one or more process didn't finish..  this
	    #   allows us to spin off a new process if one's available
	    #   but keeps us from busy waiting if all the processes
	    #   are busy.

	    unless ($proc_reclaimed) {
		print "sleeping: no processes reclaimed...n";
		sleep 1;
	    }
	}
    } until ($proc_running);
}

My initial version did a waitpid($p, WNOHANG), I also tried waitpid(-1, 0). In both cases it caused the load on the system to go up very quickly, eventually locking the system completely. Searching out result sets less of a few hundred doesn't exhibit the problem.

Running it through the debugger showed the child process finishing. strace showed the process completing. ps would show the process still running. The only symptom was a high sys cpu utilization.