#Zimbra Email Migration to #Office365 using #imapsync

Today we'll show you how to use imapsync to migrate email from Zimbra to Exchange 2013 / Office 365.  We got some good feedback from our initial post on Zimbra calendar migration to Microsoft Exchange and wanted to add some value on the issue of email migration.  When we did that for MDaemon everyone was very grateful and it led to much calendar migration.

You can license the technology and purchase the support to migrate your email for 100 Euros. Remember this only moves email.  You will have to budget additional funds to move your calendars.  This gives your end users a complete solution -- email and "live" (with guest lists and responses.) Plus, your conference rooms and resources are fully functional when you're done with your migration.  That's what we do at Sumatra.

We found imapsync to be the most  efficient and cost-effective, email migration product.  Just license imapsync and get support at the same time.

Is this imapsync email migration guide exhaustive?  Hell no!  There is no way to document all the "creative" ways user behavior can wreak havoc -- and some of them are going to be downright pathological.  We are aiming for the 80% side or the 80/20 rule here.  You get that much down you can make a good stab at the rest if you find yourself in difficulties. Barring that imapsync is very forgiving and its model of incremental syncing is laudable in its simplicity and effectiveness.

Migrating your email probably also takes you through 80% of what you need in a migration.  If you're happy with that -- glad to be of service.  If you need the calendaring to come over with fully-functional meetings, guest lists, responses, resources, contact Sumatra and we'll set you up with a trial.

Preliminaries

Zimbra has excellent guide on using imapsync with Zimbra.  We would suggest you add a valid non-self-signed certificate to their requirements.  imapsync runs in Linux and Windows (via the Command Prompt.) We'll demonstrate the Command Prompt since Microsoft Exchange / Office 365 is our target system.  Editorial comment: one of the smartest things Microsoft could do is put Exchange on Linux, but it's more likely some weenie there will recommend porting Windows to the iPad first.

Remember before you test,  you will have to enable IMAP in Exchange 2013 and for your end users in Office 365.  See how to do this in our blog post.

Migrating one user Zimbra to Exchange

In the simplest case migrating from Zimbra to Office 365 looks like this for our user Jimi Hendrix (if you use individual passwords for users).

imapsync.exe ^

--host1 sumatra.com --user1 jimi.hendrix@sumatra.com --password1  "XXXX" ^
--host2 outlook.office365.com --user2 ^
jimi.hendrix@sumatra.onmicrosoft.com --password2 "XXXXX"  ^
--ssl2  --sep2 /

If you set up a service account with FullAccess on Office 365, you accomplish a migration without knowing any password except your service account by using a command like this (where password2 is the service account password):

imapsync.exe ^
--host1 zimbra.sumatra.com  ^
--user1 jimi.hendrix@zimbra.sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --port2 993 --sep2 / ^
--user2 jimi.hendrix@sumatra.onmicrosoft.com ^ 
--authuser2 SERVICE_ACCT@sumatra.onmicrosoft.com ^
--password2 "XXXXX"  --ssl2

Note in  the above, for an Office 365 target system we need to use the "--sep2 /" command. 

Confused about Exchange permissions and setting up your service account?  Read our post The Cookbook Version of Exchange 2013 Migration Rights.

If you want to use a Service Account on Zimbra as well, similar syntax should work using either the "admin" or "zimbra" account.

Note also that this gives you the direct capability to map your user ID, for instance from "jimi.hendrix" on your legacy system to "jhendrix1967"  or "jimi.hendrix1967" on your target system.

Iterating over a user list

In any event you are going to need to generate a user list to migrate email.  Can you keep your migrating user list separate from your migration script?  Answer: YES. This method assumes your legacy ID is the same as your target ID, but allowing for this to change is not a hard extension.

The imapsync ZIP file contains a script for iterating on a user list:

sync_loop_windows.bat

Which also contains an excellent primer on running imapsync in parallel.  

This batch file assumes a text file in the form "User1;Password1;User2;Password2;..."

Zimbra's Guide to imapsync includes a couple of batch processing script examples.

To get  a user list from Zimbra you can use either zmprov (this gets a list of accounts without passwords, so use a Zimbra admin account to get data, and please edit out the spam, A-V, etc. IDs): 

cd /opt/zimbra/bin  

and 
sudo ./zmprov -l gaa >~/accounts.txt

You could also use ldapsearch if you already have scripts for that. 

It has been said that the "death is in the details."  We say, "success is set in the details."

Now come the details

Throttling in Exchange/Office 365: Your Migration Nemesis

Note that I did not write "enemy."   You're more likely to be throttled in Office 365 since the controls to that environment are largely out of your hands.

As per our usual mantra: test everything before you go into production.  

If you get throttled, there are two imapsync switches you can tweak.

One limits the transfer rate to a specific number of messages:  
--maxmessagespersecond

Start at 10 (say) and work up or down from there.  In calendar migrations we start with 25, but our gut experience tells us calendar data is smaller per object on average.

The other limits the transfer rate by byte if that works better for your network environment.

--maxbytespersecond

You can also check out our post Throttling in Exchange 2013.

Really helpful options

• --buffersize 8192000  imapsync has a default I/O buffer of 4 Kb.  Upping this to 8 Mb will probably speed things for you
• --syncinternaldates: some email systems misuse email dates and you therefore run the risk of the receipt dates on your target system (what imapsync refers to as host2) becoming the date of insertion.  This command avoids that unfortunate event. 
• --fast:  this prevents flags from being synced and therefore makes the process (wait for it....) faster.  Not an option to invoke if you want / need to sync flags! 
• --dry:  this is a really useful option for development and debugging,  It just goes through the motions of logging onto both source and target system and displays the status -- a dry run.  Use this or debugging options -debug and -debugimap

Other idiosyncrasies

The imapsync FAQ recommends these additional settings when migrating from Zimbra.

imapsync ... ^
--exclude "Conversation Action Settings" ^
--exclude "Quick Step Settings" ^
--exclude "News Feed"

Although we have not seen those folders in Zimbra in a while, your implementation could be different.

Sent, Junk, and Trash

However, there are differences in Folder names between the two environments that may be relevant to you.  Specifically what Zimbra calls "Sent, Junk, Trash" Office 365 calls "Sent Items, Junk E-Mail, Deleted Items" as seen in this side-by-side comparison.

To successfully migrate these folders, use this command sequence to map the folders: (see Zimbra Documentation):

--regextrans2 's/Sent$/Sent Items/'
--regextrans2 's/Junk$/Junk E-Mail/'

--regextrans2 's/Trash$/Deleted Items/'

Notice how we lined these up so it would be easy to repeat if you needed this for other systems, or different language packs.  These being regular expressions you could also take several related folders and migrate them into a single one on the Office 365 side, but I leave that to your wits and imagination.  If you do find something that works, let us know -- we'll update this post (of course, crediting you!)

Exchange IMAP Prerequisites

Did you configure IMAP?  If not, see how in our blog post:

On the Zimbra side:
Make sure you are enabled for IMAP access. 

Check "Enable clear text login" for the IMAP service via "Global Settings" or under "Servers" under IMAP in the Zimbra Administration Console.

If you need to install Perl modules, this is an excellent tutorial on how to do so.

Sample Scripts

These sample scripts for major migrations / multiple users will help you out a lot.

http://imapsync.lamiral.info/examples/sync_loop_windows.bat

http://imapsync.lamiral.info/examples/file.txt

Final word on really considering what you need in Zimbra migrations Diatribe: Migrating email is about moving tonnage.  Migrating calendars is about preserving responses, recurrences, and resources.

But a full migration methodology has to include more than just moving this data. We have a comprehensive suite of scripts and tools so that before any email or calendars are migrated we take care of:

• Reading the Zimbra user list (and passwords)
• Provisioning users in Exchange
• Re-configuring Outlook to point to your Exchange server and removing the Zimbra Outlook connector. (Note: there are publicly available scripts but we wrote our own after we found they did not work.)
• Pre- and Post-cut over scripting so that legacy emails are moved to the target system, and new emails redirected to the target system.

If you have a few dozen users, it's difficult to make an automated process cost-effective and you might as well do it on your own one at a time.  You can export PSTs or export files from Zimbra, then import those PSTs into Exchange.  Tedious and time-consuming, yes, but free.

If you are a larger site with the need to preserve your meeting guest lists, recurrences, responses, and resources post-migration feel free to contact Sumatra.