October 27, 2011

Disable plain text port in CentOS DS

Filed under: Uncategorized — morgan @ 16:24

aka: disabling Non-Secure Port. Should work the same in Redhat Directory and 389 Directory. This appears to be undocumented:

Validate slapd is listening on 389:

# ps -ef|grep slap
nobody 11206 1 0 16:00 ? 00:00:00 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ldap0 -i /var/run/dirsrv/slapd-ldap0.pid -w /var/run/dirsrv/slapd-ldap0.startpid
root 11514 11392 0 16:20 pts/1 00:00:00 grep slap
# lsof -p 11206 |grep -i ISTEN
ns-slapd 11206 nobody 6u IPv6 23460249 TCP *:ldap (LISTEN)
ns-slapd 11206 nobody 7u IPv6 23460250 TCP *:ldaps (LISTEN)
#

Make the configuration change:

# ldapmodify -xW -D cn=directory\ manager
Enter LDAP Password:
dn: cn=config
changetype: modify
replace: nsslapd-port
nsslapd-port:

modifying entry "cn=config"

#

Restart slapd:

# service dirsrv restart
Shutting down dirsrv:
ldap0... [ OK ]
Starting dirsrv:
ldap0...[27/Oct/2011:16:24:16 -0400] - Information: Non-Secure Port Disabled
[ OK ]
#

Validate slapd is only listening on port 636:

# ps -ef|grep slap
nobody 11570 1 0 16:24 ? 00:00:00 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ldap0 -i /var/run/dirsrv/slapd-ldap0.pid -w /var/run/dirsrv/slapd-ldap0.startpid
root 11659 11392 0 16:26 pts/1 00:00:00 grep slap
# lsof -p 11570|grep LISTE
ns-slapd 11570 nobody 6u IPv6 23463024 TCP *:ldaps (LISTEN)
#

August 31, 2011

Reinitialize OpenLDAP replica

Filed under: Directory/LDAP,linux,Zimbra — morgan @ 14:31

The paths are Zimbra specific but this should apply to OpenLDAP 2.4 or later. It was 2am when were troubleshooting but I am surprised at how little information there is on the topic.

This was Zimbra 7.1.1p2 for reference.

I used this as a baseline:
http://wiki.zimbra.com/index.php?title=LDAP_Replicas_4.5.x_to_5.0.x

This looks simpler but may be slower on a large installation. I haven’t tried it:
http://www.zimbra.com/forums/installation/28199-zimbra-ldap-replication-help-existing-environment.html

Master:

$ /opt/zimbra/openldap/sbin/slapcat -b '' -F /opt/zimbra/data/ldap/config -l /tmp/ldap.ldif

Copy ldap.ldif to replica

Replica:

# su - zimbra -c "ldap stop"
# cd /opt/zimbra/data/ldap
# mv hdb hdb-old
# mkdir -p hdb/logs/
# mkdir hdb/db
# cp hdb-old/db/DB_CONFIG hdb/db
# chown -R zimbra:zimbra hdb
# su - zimbra
$ /opt/zimbra/openldap/sbin/slapadd -w -q -b '' -F /opt/zimbra/data/ldap/config -l /var/tmp/ldap.ldif
*#################### 100.00% eta none elapsed 01m11s spd 1.0 M/s
Closing DB...
$ ldap start

test replication by making a change to the description field of a user's entry:

$ zmprov ma morgan@domain.org description "test of replication 8/30"

Use ldapsearch to make sure it was set on the consumer:

$ ldapsearch -xW -h ldap02.domain.org -D uid=zimbra,cn=admins,cn=zimbra  -Lb "" uid=morgan description

There are of course many reasons you might want to reinitialize replication. In our case though it was due to this error:

Aug 30 01:35:01 ldap02 slapd[25423]: do_syncrep2: rid=100
cookie=rid=100,csn=20110829220813.829835Z#000000#000#000000
Aug 30 01:35:01 ldap02 slapd[25423]: slap_queue_csn: queing 0x1ce24180
20110829220813.829835Z#000000#000#000000
Aug 30 01:35:01 ldap02 slapd[25423]: syncrepl_message_to_op: rid=100 mods check
(zimbraMailStatus: multiple values provided)
Aug 30 01:35:01 ldap02 slapd[25423]: slap_graduate_commit_csn: removing 0x1dceba20
20110829220813.829835Z#000000#000#000000
Aug 30 01:35:01 ldap02 slapd[25423]: do_syncrepl: rid=100 rc 19 retrying

LDAP error 19 is a constraint violation, commonly attempting to add multiple attributes to a single valued attribute. I had just modified zimbraMailStatus for a few users. I checked and each of their zimbraMailStatuses had a single value so my suspicion is this is just a hiccup in replication and reinitializing cleared it.

May 18, 2011

Zimbra mailboxd depends on uid rather than username

Filed under: linux,Messaging,Uncategorized,Zimbra — morgan @ 10:16

We’re refreshing the hardware on a moderate sized multi-server environment. I’m testing in our dev environment:
install the same version of Zimbra on the target host, then:
rm -rf /opt/zimbra
mkdir /opt/zimbra

rsync /opt/zimbra from the source host to the target host.
change the hostname and ip of the target host to that of the source host.
start zimbra

mailboxd started but did not log to /opt/zimbra/log/mailboxd.out
but in zmmailboxd.out:


738  WARN  [main] log - Failed startup of context org.mortbay.jetty.webapp.WebAppContext@35ecfe07{/zimbraAdmin,/opt/zimbra/jetty-6.1.22.z6/webapps/zimbraAdmin}
java.io.FileNotFoundException: /opt/zimbra/jetty-6.1.22.z6/webapps/zimbraAdmin/WEB-INF/web.xml (Permission denied)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.(FileInputStream.java:106)
    at java.io.FileInputStream.(FileInputStream.java:66)
    at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70)
...

This post brought the solution:

http://www.zimbra.com/forums/administrators/37943-jetty-webapps-not-starting-after-server-move.html

change the uid and gid of zimbra (/etc/passwd and /etc/shadow) to what it was on the source system, then do a global chown:

find / -user -type l -exec chown -h zimbra:zimbra {} \;

Thanks to John W. for finding the -h flag of chown. Ownership of symbolic links apparently don’t follow that of the target file.

May 3, 2011

zmrestore, system failure: link (…) No such file or directory

Filed under: Uncategorized — morgan @ 16:01

On first host:


$ zmbackup -z -f -a "user@domain.org"
full-20110503.182532.238

Copy the contents of /opt/zimbra/backup/sessions/full-20110503.182532.238 and accounts.xml to /var/tmp/backup on the remote host.


# chown -R zimbra:zimbra /var/tmp/backup
# su - zimbra
$ zmrestore -rf -t /var/tmp/backup -a user@domain.org -lb full-20110503.182532.238
Error occurred: system failure: link(/opt/zimbra/store/21/86487/msg/20/85569-182399.msg,
/opt/zimbra/store/21/86487/msg/20/85569-182399.msg): No such file or directory

After entirely too much time troubleshooting the cause is still a mystery, however the work-around is relatively simple:


$ zmrestore -ca -pre restored_ -rf -t /var/tmp/backup -a user@domain.org -lb full-20110503.182532.238
$ zmprov ra restored_user@domain.org user@domain.org

In case you’re wondering what the use case for this would be: I’m migrating users from a production 6.0.7 to a dev 6.0.7 wherein we’ll upgrade to 7.1 in preparation for their production upgrade. In short: I am restoring on a different host from which I did the backup.

February 26, 2011

Cyrus imapd to Zimbra via imapsync

Filed under: Messaging,Zimbra — morgan @ 15:05

It seems simple enough, both servers support imap but the devil’s in the details: cyrus puts all mail under INBOX, Zimbra does not by default and cyrus uses ‘.’ as a separator while Zimbra uses ‘/.’ Add –delete2 if you’ve been testing for a few days on an active server and the target server has mail that is getting stale:


$ imapsync --sep1 \. --prefix1 INBOX\. \
    --authmech1 PLAIN --ssl1 --host1 sourcehost.domain.org \
        --user1 morgan --password1 pass \
    --authmech2 PLAIN --ssl2 --host2 targethost.domain.org \
        --user2 morgan --password2 pass \
    --useheader Message-ID --useheader From --useheader To \
        --skipsize  --delete2

Rsync and zimbra single copy message store

Filed under: linux,Messaging,Zimbra — morgan @ 14:45

In the context of a single message store this seems obvious but it took me an hour of looking around to figure out what was going on.

I needed to move several mail store partitions from one SAN volume to another and for various reasons it came down to doing it on the operating system level as opposed to snapshotting or similar within the san.

This is an extremely busy production server and we get limited downtime on it so the process will have to go like this:
- rsync from the production volume to the new volume mounted in a temporary location while Zimbra is running.
- stop zimbra during an outage window and re-run rsync to get messages that have been delivered since
- unmount the current volume, mount the new volume and start Zimbra.

So we started with a small volume, a 317G secondaryMessage volume. We created a new 500G volume, mounted it in a temporary location and started the rsync:


# rsync -avx --delete /opt/zimbra/store1/ /mnt/tmp

Nagios woke me up around 4am as the temp disk was filling. Because Zimbra’s a single message store. How do you verify that?

Cd into each of the directories and compare the sizes of the sub directories. Find size mismatches and do an ls -1i:


# cd /opt/zimbra/store1/1/4139/msg/4
# ls -1i
63148652 17260-86600.msg
63148653 17262-86601.msg
63148654 17280-86700.msg
63148655 17281-86701.msg
...

# cd /mnt/tmp/1/4139/msg/4
# ls -1i
203046917 17260-86600.msg
 85590190 17262-86601.msg
207127555 17280-86700.msg
207127557 17281-86701.msg
...

The solution is to give rsync the -H option:


rsync -avxH --delete /opt/zimbra/store1/ /mnt/tmp

rsync will connect the hard links and disk usage will drop on the target volume.


...
7/30042/msg/11/47149-170909.msg => 9/38784/msg/0/261-101.msg
7/30042/msg/11/47267-171507.msg => 9/38786/msg/0/280-200.msg
7/30042/msg/11/47268-171508.msg => 9/38786/msg/0/281-201.msg
...

Update, 3/2/10: if you’re not using rsync 3.0 or newer you should upgrade! Once we moved to larger filesystems (1.4tb+) rsync 2.83 spent several days collecting a file list before, we believe, eventually consuming enough memory on the system to trigger the OOM killer.

Zimbra Cross Mailbox Search

Filed under: Messaging,Zimbra — morgan @ 13:52

I have a client who has long used Zimbra’s archiving feature: we set it up several versions ago and regularly use it to retrieve messages sent/received on a per-user basis for various purposes.

We recently had a need to search across multiple accounts. We’ve long known that Zimbra offered a tool to do this searching but never got around to looking into it.

http://wiki.zimbra.com/wiki/Ajcody-Notes-Archive-Discovery-Mailstore-Setup

So click Tools->Search Mail in the admin interface. Simple, yes? In both their development and production environments we get a pop-up error like this:

Message: unknown document: GetXMbxSearchesListRequest Error code: service.UNKNOWN_DOCUMENT Method: GetXMboxSearchListRequest Details:soap:Sender

A quick look at the mailbox.log shows the archive store (mail06) is proxying to the Logger store (mail01).

2010-12-21 12:50:56,480 INFO [btpool0-7://mail06.domain.org:7071/service/admin/soap/GetXMbxSearchesListRequest] [name=admin@domain.org;mid=33900;ip=10.0.0.123;ua=ZimbraWebClient - FF3.0 (Mac);] soap – GetXMbxSearchesListRequest (Proxying to ProxyTarget(url=https://mail01.domain.org:7071/service/admin/soap/GetXMbxSearchesListRequest))

I’m reasonably sure it’s a coincidence that it is looking to the Logger store but it is consistent in both of their environments. Incidentally: changing zimbraLogHostname doesn’t fix the problem.

The only fix I’ve found is to re-run the installer and add a zimbra-archiving to the Logger host (mail01.domain.org in this example).

There has to be a setting that configures this.

February 21, 2011

move user mail intra-server with imapsync

Filed under: Messaging,Zimbra — morgan @ 16:30

A seemingly innocuous DNS configuration change caused all of the RBL checks in a client’s Zimbra Spamassassin to trigger mail that would have only had an X-Spam-Score of 2 or 3 to register 7+. This of course lead to legitimate mail to end up in users’ Junk folders.

Our Junk boxes age out in 2 or so weeks and our user base is not in the habit of checking regularly since most SPAM is caught at the gateway and not delivered to Junk. It was decided that the best solution for us was to just move mail delivered to Junk in the last 5 days back into users’ inboxes.

It turns out imapsync can be easily configured to move mail intra-server and intra-account by supplying the same host and user, limiting to –folder Junk and changing –prefix1 and –prefix2. If you have tens of thousands of accounts you may want to parallelize this but for the sake of brevity here’s a short shell script that moves all mail from the last 5 days from Junk to INBOX in a Zimbra environment:


#!/bin/sh
#
p=pass
au=user
h=imap.domain.org

echo starting at `date`
for u in `sudo su - zimbra -c "zmprov -l gaa"|grep -v archive|cut -d @ -f 1`; do
    echo; echo ${u}:
    c="imapsync ${1} --folder Junk --maxage 5 \
        --prefix1 Junk  --ssl1 --host1 ${h} --authuser1 ${au} --user1 ${u} --password1 ${p}\
        --prefix2 INBOX --ssl2 --host2 ${h} --authuser2 ${au} --user2 ${u} --password2 ${p}"
    echo $c
    $c
done
echo finished at `date`

This is Zimbra 6.0.8 but it should work in any version at least up to that:

“user” must be set up as a zimbra admin. I imagine the Zimbra admin user would work.

This script does disclose the password on the command line so it will show up in a ps(1) output. imapsync does include –passfile options to correct this.

If you’re thinking this looks handy but are overwhelmed by the idea of installing imapsync (it has a ton of dependencies) and use CentOS or Redhat I can vouch for epel’s (http://fedoraproject.org/wiki/EPEL) distribution of imapsync. Install the epel repository and imapsync is just a “yum install imapsync” away.

January 6, 2011

command line base64 decoding

Filed under: linux — morgan @ 14:41

Based on instructions here: http://face.centosprime.com/macosxw/base64-decoding

LDAP attributes are sometimes base64 encoded in ldapsearch output:
postalAddress:: MTI1NCBDZW50cmFsIEF2ZS4K

In OS X and presumably any system with openssl installed:
$ openssl base64 -d
MTI1NCBDZW50cmFsIEF2ZS4K
^d
1254 Central Ave.
$

for ‘^d’ you hold the control key and type ‘d.’

If you’re looking for a method that does not involve using stdin:
$ echo MTI1NCBDZW50cmFsIEF2ZS4K | openssl base64 -d
1254 Central Ave.
$

December 21, 2010

simple perl to remove line continuation in ldif

Filed under: Directory/LDAP,Perl,Programming — morgan @ 14:10


ldapsearch -xW -D cn=directory\ manager -LLLb dc=example,dc=org objectclass=\* |perl -0000 -p -e '$_=~s/\n\s//; $_.="\n\n";'

This is also an example of how to set the input record separator to null (with -000, the same as $/=”";) with the -p (or -n) switch.

If you just want a list of DNs add ‘dn’ to the ldapsearch and ‘chop’ before the ‘s///:’

ldapsearch -xW -D cn=directory\ manager -LLLb dc=example,dc=org objectclass=\* dn|perl -0000 -p -e 'chop; $_=~s/\n\s//'

Older Posts »

Powered by WordPress