After a morning of troubleshooting poor performance on an archive store that was causing backups on the mtas I would bet we’ve hit a regression. Zimbra doesn’t recognize it.

One of the admins IM’ed me a little after 9am yesterday complaining of queue backups on the mtas delivering to the archive store. They have 5 user stores and 1 archive store.

Connecting to port 7025 (lmtp) on the archive store shows sporadic hangs. Some mail was getting in but thousands were backed up on the mtas.

A quick look at the mailbox.log showed errors connecting to the conversion daemon:

2009-03-05 10:25:54,909 WARN  [LmtpServer-20] [name=user-20080704@domain.tld.archive;
mid=70;] ParsedMessage - Unable to parse part 2 (file.pdf, application/pdf) of 
message with Message-ID <000f01c29c4e$3434d3c0$315c200a@user.domain.tld>.
com.zimbra.cs.mime.MimeHandlerException: cannot extract text
    at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(
DefaultHandler.java:117)    
    at com.zimbra.cs.mime.MimeHandler.getContent(MimeHandler.java:148)
    at com.zimbra.cs.mime.ParsedMessage.analyzePart(ParsedMessage.java:1148)    
    at com.zimbra.cs.mime.ParsedMessage.analyzeNonBodyParts(ParsedMessage.java:426) 
...
Caused by: com.zimbra.cs.convert.ConversionException: Cannot extract text from 
document that previously crashed convertd.
    at com.zimbra.cs.convert.TransformationClient.extract(
TransformationClient.java:196)
    at com.zimbra.cs.convert.SocketTransformationStub.doExtract(
SocketTransformationStub.java:84)
    at com.zimbra.cs.convert.SocketTransformationStub.extract(
SocketTransformationStub.java:116)
    at com.zimbra.cs.mime.handler.DefaultHandler.getContentImpl(
DefaultHandler.java:111)
    ... 27 more

So on first look convertd died, I restarted it:
su - zimbra -c "zmconvertctl start"

convertd started and the errors in mailbox.log went away. But kvoop processes started to consume cpu. The load rose to almost 30.

This is the kvoop bug we fixed by turning of attachment indexing.. So I checked that and archiving is on again on the archive cos(!)

zimbra@host log]$ zmprov gc default |grep -i attachment
zimbraAttachmentsBlocked: FALSE
zimbraAttachmentsIndexingEnabled: FALSE
zimbraAttachmentsViewInHtmlOnly: FALSE
[zimbra@host log]$ zmprov gc archive |grep -i attachment
zimbraAttachmentsBlocked: FALSE
zimbraAttachmentsIndexingEnabled: TRUE
zimbraAttachmentsViewInHtmlOnly: FALSE
[zimbra@host log]$ 

zimbra@host log]$ zmprov modifyCos archive zimbraAttachmentsIndexingEnabled 
FALSE
zimbra@host log]$

I flushed the queues and all is well.

Background:

http://wiki.zimbra.com/index.php?title=Disable_Indexing_for_Specific_File_Types

http://bugzilla.zimbra.com/show_bug.cgi?id=14016

Look at the bottom of the bugzilla link: It looks to me like we’re hitting a regression: a bug they thought they fixed in 4.5.x is resurfacing in 5.0.x:

------- Comment  #10 From tech 2008-10-06 03:44:08  -------

this bug seems to have started happening intermittently on my 5.04 rhel_x86
installation it hangs the server so i have to reboot