Validate slapd is listening on 389:

# ps -ef|grep slap
nobody 11206 1 0 16:00 ? 00:00:00 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ldap0 -i /var/run/dirsrv/slapd-ldap0.pid -w /var/run/dirsrv/slapd-ldap0.startpid
root 11514 11392 0 16:20 pts/1 00:00:00 grep slap
# lsof -p 11206 |grep -i ISTEN
ns-slapd 11206 nobody 6u IPv6 23460249 TCP *:ldap (LISTEN)
ns-slapd 11206 nobody 7u IPv6 23460250 TCP *:ldaps (LISTEN)
#


Make the configuration change:

# ldapmodify -xW -D cn=directory\ manager
Enter LDAP Password:
dn: cn=config
changetype: modify
replace: nsslapd-port
nsslapd-port:

#


Restart slapd:

# service dirsrv restart
Shutting down dirsrv:
ldap0... [ OK ]
Starting dirsrv:
ldap0...[27/Oct/2011:16:24:16 -0400] - Information: Non-Secure Port Disabled
[ OK ]
#


Validate slapd is only listening on port 636:

# ps -ef|grep slap
nobody 11570 1 0 16:24 ? 00:00:00 /usr/sbin/ns-slapd -D /etc/dirsrv/slapd-ldap0 -i /var/run/dirsrv/slapd-ldap0.pid -w /var/run/dirsrv/slapd-ldap0.startpid
root 11659 11392 0 16:26 pts/1 00:00:00 grep slap
# lsof -p 11570|grep LISTE
ns-slapd 11570 nobody 6u IPv6 23463024 TCP *:ldaps (LISTEN)
#


This was Zimbra 7.1.1p2 for reference.

I used this as a baseline:
http://wiki.zimbra.com/index.php?title=LDAP_Replicas_4.5.x_to_5.0.x

This looks simpler but may be slower on a large installation. I haven’t tried it:
http://www.zimbra.com/forums/installation/28199-zimbra-ldap-replication-help-existing-environment.html

Master:

$ /opt/zimbra/openldap/sbin/slapcat -b '' -F /opt/zimbra/data/ldap/config -l /tmp/ldap.ldif

Copy ldap.ldif to replica

Replica:

# su - zimbra -c "ldap stop"
# cd /opt/zimbra/data/ldap
# mv hdb hdb-old
# mkdir -p hdb/logs/
# mkdir hdb/db
# cp hdb-old/db/DB_CONFIG hdb/db
# chown -R zimbra:zimbra hdb
# su - zimbra
$ /opt/zimbra/openldap/sbin/slapadd -w -q -b '' -F /opt/zimbra/data/ldap/config -l /var/tmp/ldap.ldif
*#################### 100.00% eta none elapsed 01m11s spd 1.0 M/s
Closing DB...
$ ldap start


test replication by making a change to the description field of a user's entry:

$ zmprov ma morgan@domain.org description "test of replication 8/30"

Use ldapsearch to make sure it was set on the consumer:

$ ldapsearch -xW -h ldap02.domain.org -D uid=zimbra,cn=admins,cn=zimbra  -Lb "" uid=morgan description

There are of course many reasons you might want to reinitialize replication. In our case though it was due to this error:

Aug 30 01:35:01 ldap02 slapd[25423]: do_syncrep2: rid=100
cookie=rid=100,csn=20110829220813.829835Z#000000#000#000000
Aug 30 01:35:01 ldap02 slapd[25423]: slap_queue_csn: queing 0x1ce24180
20110829220813.829835Z#000000#000#000000
Aug 30 01:35:01 ldap02 slapd[25423]: syncrepl_message_to_op: rid=100 mods check
(zimbraMailStatus: multiple values provided)
Aug 30 01:35:01 ldap02 slapd[25423]: slap_graduate_commit_csn: removing 0x1dceba20
20110829220813.829835Z#000000#000#000000
Aug 30 01:35:01 ldap02 slapd[25423]: do_syncrepl: rid=100 rc 19 retrying

LDAP error 19 is a constraint violation, commonly attempting to add multiple attributes to a single valued attribute. I had just modified zimbraMailStatus for a few users. I checked and each of their zimbraMailStatuses had a single value so my suspicion is this is just a hiccup in replication and reinitializing cleared it.

mailboxd started but did not log to /opt/zimbra/log/mailboxd.out
but in zmmailboxd.out:

738  WARN  [main] log - Failed startup of context org.mortbay.jetty.webapp.WebAppContext@35ecfe07{/zimbraAdmin,/opt/zimbra/jetty-6.1.22.z6/webapps/zimbraAdmin}
java.io.FileNotFoundException: /opt/zimbra/jetty-6.1.22.z6/webapps/zimbraAdmin/WEB-INF/web.xml (Permission denied)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.(FileInputStream.java:106)
    at java.io.FileInputStream.(FileInputStream.java:66)
    at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:70)
...

This post brought the solution:

http://www.zimbra.com/forums/administrators/37943-jetty-webapps-not-starting-after-server-move.html

change the uid and gid of zimbra (/etc/passwd and /etc/shadow) to what it was on the source system, then do a global chown:

find / -user -type l -exec chown -h zimbra:zimbra {} \;

Thanks to John W. for finding the -h flag of chown. Ownership of symbolic links apparently don’t follow that of the target file.

$ zmbackup -z -f -a "user@domain.org"
full-20110503.182532.238

Copy the contents of /opt/zimbra/backup/sessions/full-20110503.182532.238 and accounts.xml to /var/tmp/backup on the remote host.

# chown -R zimbra:zimbra /var/tmp/backup
# su - zimbra
$ zmrestore -rf -t /var/tmp/backup -a user@domain.org -lb full-20110503.182532.238
Error occurred: system failure: link(/opt/zimbra/store/21/86487/msg/20/85569-182399.msg,
/opt/zimbra/store/21/86487/msg/20/85569-182399.msg): No such file or directory

After entirely too much time troubleshooting the cause is still a mystery, however the work-around is relatively simple:

$ zmrestore -ca -pre restored_ -rf -t /var/tmp/backup -a user@domain.org -lb full-20110503.182532.238
$ zmprov ra restored_user@domain.org user@domain.org

In case you’re wondering what the use case for this would be: I’m migrating users from a production 6.0.7 to a dev 6.0.7 wherein we’ll upgrade to 7.1 in preparation for their production upgrade. In short: I am restoring on a different host from which I did the backup.

$ imapsync --sep1 \. --prefix1 INBOX\. \
    --authmech1 PLAIN --ssl1 --host1 sourcehost.domain.org \
        --user1 morgan --password1 pass \
    --authmech2 PLAIN --ssl2 --host2 targethost.domain.org \
        --user2 morgan --password2 pass \
    --useheader Message-ID --useheader From --useheader To \
        --skipsize  --delete2

I needed to move several mail store partitions from one SAN volume to another and for various reasons it came down to doing it on the operating system level as opposed to snapshotting or similar within the san.

This is an extremely busy production server and we get limited downtime on it so the process will have to go like this:
- rsync from the production volume to the new volume mounted in a temporary location while Zimbra is running.
- stop zimbra during an outage window and re-run rsync to get messages that have been delivered since
- unmount the current volume, mount the new volume and start Zimbra.

So we started with a small volume, a 317G secondaryMessage volume. We created a new 500G volume, mounted it in a temporary location and started the rsync:

# rsync -avx --delete /opt/zimbra/store1/ /mnt/tmp

Nagios woke me up around 4am as the temp disk was filling. Because Zimbra’s a single message store. How do you verify that?

Cd into each of the directories and compare the sizes of the sub directories. Find size mismatches and do an ls -1i:

# cd /opt/zimbra/store1/1/4139/msg/4
# ls -1i
63148652 17260-86600.msg
63148653 17262-86601.msg
63148654 17280-86700.msg
63148655 17281-86701.msg
...

# cd /mnt/tmp/1/4139/msg/4
# ls -1i
203046917 17260-86600.msg
 85590190 17262-86601.msg
207127555 17280-86700.msg
207127557 17281-86701.msg
...

The solution is to give rsync the -H option:

rsync -avxH --delete /opt/zimbra/store1/ /mnt/tmp

rsync will connect the hard links and disk usage will drop on the target volume.

...
7/30042/msg/11/47149-170909.msg => 9/38784/msg/0/261-101.msg
7/30042/msg/11/47267-171507.msg => 9/38786/msg/0/280-200.msg
7/30042/msg/11/47268-171508.msg => 9/38786/msg/0/281-201.msg
...

Update, 3/2/10: if you’re not using rsync 3.0 or newer you should upgrade! Once we moved to larger filesystems (1.4tb+) rsync 2.83 spent several days collecting a file list before, we believe, eventually consuming enough memory on the system to trigger the OOM killer.

We recently had a need to search across multiple accounts. We’ve long known that Zimbra offered a tool to do this searching but never got around to looking into it.

http://wiki.zimbra.com/wiki/Ajcody-Notes-Archive-Discovery-Mailstore-Setup

So click Tools->Search Mail in the admin interface. Simple, yes? In both their development and production environments we get a pop-up error like this:

Message: unknown document: GetXMbxSearchesListRequest Error code: service.UNKNOWN_DOCUMENT Method: GetXMboxSearchListRequest Details:soap:Sender

A quick look at the mailbox.log shows the archive store (mail06) is proxying to the Logger store (mail01).

2010-12-21 12:50:56,480 INFO [btpool0-7://mail06.domain.org:7071/service/admin/soap/GetXMbxSearchesListRequest] [name=admin@domain.org;mid=33900;ip=10.0.0.123;ua=ZimbraWebClient - FF3.0 (Mac);] soap – GetXMbxSearchesListRequest (Proxying to ProxyTarget(url=https://mail01.domain.org:7071/service/admin/soap/GetXMbxSearchesListRequest))

I’m reasonably sure it’s a coincidence that it is looking to the Logger store but it is consistent in both of their environments. Incidentally: changing zimbraLogHostname doesn’t fix the problem.

The only fix I’ve found is to re-run the installer and add a zimbra-archiving to the Logger host (mail01.domain.org in this example).

There has to be a setting that configures this.

Our Junk boxes age out in 2 or so weeks and our user base is not in the habit of checking regularly since most SPAM is caught at the gateway and not delivered to Junk. It was decided that the best solution for us was to just move mail delivered to Junk in the last 5 days back into users’ inboxes.

It turns out imapsync can be easily configured to move mail intra-server and intra-account by supplying the same host and user, limiting to –folder Junk and changing –prefix1 and –prefix2. If you have tens of thousands of accounts you may want to parallelize this but for the sake of brevity here’s a short shell script that moves all mail from the last 5 days from Junk to INBOX in a Zimbra environment:

#!/bin/sh
#
p=pass
au=user
h=imap.domain.org

echo starting at `date`
for u in `sudo su - zimbra -c "zmprov -l gaa"|grep -v archive|cut -d @ -f 1`; do
    echo; echo ${u}:
    c="imapsync ${1} --folder Junk --maxage 5 \
        --prefix1 Junk  --ssl1 --host1 ${h} --authuser1 ${au} --user1 ${u} --password1 ${p}\
        --prefix2 INBOX --ssl2 --host2 ${h} --authuser2 ${au} --user2 ${u} --password2 ${p}"
    echo $c
    $c
done
echo finished at `date`

This is Zimbra 6.0.8 but it should work in any version at least up to that:

“user” must be set up as a zimbra admin. I imagine the Zimbra admin user would work.

This script does disclose the password on the command line so it will show up in a ps(1) output. imapsync does include –passfile options to correct this.

If you’re thinking this looks handy but are overwhelmed by the idea of installing imapsync (it has a ton of dependencies) and use CentOS or Redhat I can vouch for epel’s (http://fedoraproject.org/wiki/EPEL) distribution of imapsync. Install the epel repository and imapsync is just a “yum install imapsync” away.

LDAP attributes are sometimes base64 encoded in ldapsearch output:
postalAddress:: MTI1NCBDZW50cmFsIEF2ZS4K

In OS X and presumably any system with openssl installed:
$ openssl base64 -d
MTI1NCBDZW50cmFsIEF2ZS4K
^d
1254 Central Ave.
$
for ‘^d’ you hold the control key and type ‘d.’

If you’re looking for a method that does not involve using stdin:
$ echo MTI1NCBDZW50cmFsIEF2ZS4K | openssl base64 -d
1254 Central Ave.
$


This is also an example of how to set the input record separator to null (with -000, the same as $/=”";) with the -p (or -n) switch.

If you just want a list of DNs add ‘dn’ to the ldapsearch and ‘chop’ before the ‘s///:’

ldapsearch -xW -D cn=directory\ manager -LLLb dc=example,dc=org objectclass=\* dn|perl -0000 -p -e 'chop; $_=~s/\n\s//'