Enterprise Calendar Migration @Zimbra to @Office 365 @MSFTExchange

Per request, a video of  migrating calendars from Zimbra into Office 365.

Of course this keeps meetings as genuine, functional, updateable meetings when they are migrated to the Microsoft Exchange side.

We are reading calendar data directly from the Zimbra database, a technique we also use for our full-state Exchange to Exchange migrations and our Oracle Beehive to Exchange migrations.

Microsoft Deprecating the REST API preview endpoint

Microsoft is Deprecating the REST API preview endpoint on October 15.

Now this is the PREVIEW endpoint, not the functionality.

Since we're on something of a roll, we're considering using REST to pull calendar, task, and contact data out of Zimbra at some point.  Our experiments make pulling the data from Zimbra look relatively straight-forward.

Our experiments putting data into Office 365 indicate more trouble and uncertainty than it is worth -- so we're likely to stay with our current methods.

Converting a Meeting Maker server to a relational database

We had someone ask us "Can you guys convert a Meeting Maker server to a relational database?"

Short answer is: Yes - that's how our migration process has always worked.  Raw Meeting Maker data exists in an object-oriented database format that I have described as a cross between a PhD thesis and a junior high school science project. 

Even though we formally ended our MM to Exchange migrations we've been contacted by some Friends of Sumatra who've been through migrations with us before so we've kept our conversion software running.

Time frame: depending on how much history you have and how many users are on your Meeting Maker server we can convert your data into an Microsoft Access database in (at the most) an afternoon.  Usually it's only a few hours but breaking out and spinning up the conversion server is the real hassle.

This converted database alone is insufficient to run a full migration into Exchange.  Do not allow that idea to entertain your mind.

The converted database is completely appropriate for converting / archiving Meeting Maker servers into a standard database format so that you can do searches for legal compliance, history, investigations, or whatever your data-driven needs require.

#Zimbra Email Migration to #Office365 using #imapsync

Today we'll show you how to use imapsync to migrate email from Zimbra to Exchange 2013 / Office 365.  We got some good feedback from our initial post on Zimbra calendar migration to Microsoft Exchange and wanted to add some value on the issue of email migration.  When we did that for MDaemon everyone was very grateful and it led to much calendar migration.

You can license the technology and purchase the support to migrate your email for 100 Euros. Remember this only moves email.  You will have to budget additional funds to move your calendars.  This gives your end users a complete solution -- email and "live" (with guest lists and responses.) Plus, your conference rooms and resources are fully functional when you're done with your migration.  That's what we do at Sumatra.

We found imapsync to be the most  efficient and cost-effective, email migration product.  Just license imapsync and get support at the same time.

Is this imapsync email migration guide exhaustive?  Hell no!  There is no way to document all the "creative" ways user behavior can wreak havoc -- and some of them are going to be downright pathological.  We are aiming for the 80% side or the 80/20 rule here.  You get that much down you can make a good stab at the rest if you find yourself in difficulties. Barring that imapsync is very forgiving and its model of incremental syncing is laudable in its simplicity and effectiveness.

Migrating your email probably also takes you through 80% of what you need in a migration.  If you're happy with that -- glad to be of service.  If you need the calendaring to come over with fully-functional meetings, guest lists, responses, resources, contact Sumatra and we'll set you up with a trial.

Preliminaries

Zimbra has excellent guide on using imapsync with Zimbra.  We would suggest you add a valid non-self-signed certificate to their requirements.  imapsync runs in Linux and Windows (via the Command Prompt.) We'll demonstrate the Command Prompt since Microsoft Exchange / Office 365 is our target system.  Editorial comment: one of the smartest things Microsoft could do is put Exchange on Linux, but it's more likely some weenie there will recommend porting Windows to the iPad first.

Remember before you test,  you will have to enable IMAP in Exchange 2013 and for your end users in Office 365.  See how to do this in our blog post.

Migrating one user Zimbra to Exchange

In the simplest case migrating from Zimbra to Office 365 looks like this for our user Jimi Hendrix (if you use individual passwords for users).

imapsync.exe ^

--host1 sumatra.com --user1 jimi.hendrix@sumatra.com --password1  "XXXX" ^
--host2 outlook.office365.com --user2 ^
jimi.hendrix@sumatra.onmicrosoft.com --password2 "XXXXX"  ^
--ssl2  --sep2 /

If you set up a service account with FullAccess on Office 365, you accomplish a migration without knowing any password except your service account by using a command like this (where password2 is the service account password):

imapsync.exe ^
--host1 zimbra.sumatra.com  ^
--user1 jimi.hendrix@zimbra.sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --port2 993 --sep2 / ^
--user2 jimi.hendrix@sumatra.onmicrosoft.com ^ 
--authuser2 SERVICE_ACCT@sumatra.onmicrosoft.com ^
--password2 "XXXXX"  --ssl2

Note in  the above, for an Office 365 target system we need to use the "--sep2 /" command. 

Confused about Exchange permissions and setting up your service account?  Read our post The Cookbook Version of Exchange 2013 Migration Rights.

If you want to use a Service Account on Zimbra as well, similar syntax should work using either the "admin" or "zimbra" account.

Note also that this gives you the direct capability to map your user ID, for instance from "jimi.hendrix" on your legacy system to "jhendrix1967"  or "jimi.hendrix1967" on your target system.

Iterating over a user list

In any event you are going to need to generate a user list to migrate email.  Can you keep your migrating user list separate from your migration script?  Answer: YES. This method assumes your legacy ID is the same as your target ID, but allowing for this to change is not a hard extension.

The imapsync ZIP file contains a script for iterating on a user list:

sync_loop_windows.bat

Which also contains an excellent primer on running imapsync in parallel.  

This batch file assumes a text file in the form "User1;Password1;User2;Password2;..."

Zimbra's Guide to imapsync includes a couple of batch processing script examples.

To get  a user list from Zimbra you can use either zmprov (this gets a list of accounts without passwords, so use a Zimbra admin account to get data, and please edit out the spam, A-V, etc. IDs): 

cd /opt/zimbra/bin  

and 
sudo ./zmprov -l gaa >~/accounts.txt

You could also use ldapsearch if you already have scripts for that. 

It has been said that the "death is in the details."  We say, "success is set in the details."

Now come the details

Throttling in Exchange/Office 365: Your Migration Nemesis

Note that I did not write "enemy."   You're more likely to be throttled in Office 365 since the controls to that environment are largely out of your hands.

As per our usual mantra: test everything before you go into production.  

If you get throttled, there are two imapsync switches you can tweak.

One limits the transfer rate to a specific number of messages:  
--maxmessagespersecond

Start at 10 (say) and work up or down from there.  In calendar migrations we start with 25, but our gut experience tells us calendar data is smaller per object on average.

The other limits the transfer rate by byte if that works better for your network environment.

--maxbytespersecond

You can also check out our post Throttling in Exchange 2013.

Really helpful options

• --buffersize 8192000  imapsync has a default I/O buffer of 4 Kb.  Upping this to 8 Mb will probably speed things for you
• --syncinternaldates: some email systems misuse email dates and you therefore run the risk of the receipt dates on your target system (what imapsync refers to as host2) becoming the date of insertion.  This command avoids that unfortunate event. 
• --fast:  this prevents flags from being synced and therefore makes the process (wait for it....) faster.  Not an option to invoke if you want / need to sync flags! 
• --dry:  this is a really useful option for development and debugging,  It just goes through the motions of logging onto both source and target system and displays the status -- a dry run.  Use this or debugging options -debug and -debugimap

Other idiosyncrasies

The imapsync FAQ recommends these additional settings when migrating from Zimbra.

imapsync ... ^
--exclude "Conversation Action Settings" ^
--exclude "Quick Step Settings" ^
--exclude "News Feed"

Although we have not seen those folders in Zimbra in a while, your implementation could be different.

Sent, Junk, and Trash

However, there are differences in Folder names between the two environments that may be relevant to you.  Specifically what Zimbra calls "Sent, Junk, Trash" Office 365 calls "Sent Items, Junk E-Mail, Deleted Items" as seen in this side-by-side comparison.

To successfully migrate these folders, use this command sequence to map the folders: (see Zimbra Documentation):

--regextrans2 's/Sent$/Sent Items/'
--regextrans2 's/Junk$/Junk E-Mail/'

--regextrans2 's/Trash$/Deleted Items/'

Notice how we lined these up so it would be easy to repeat if you needed this for other systems, or different language packs.  These being regular expressions you could also take several related folders and migrate them into a single one on the Office 365 side, but I leave that to your wits and imagination.  If you do find something that works, let us know -- we'll update this post (of course, crediting you!)

Exchange IMAP Prerequisites

Did you configure IMAP?  If not, see how in our blog post:

On the Zimbra side:
Make sure you are enabled for IMAP access. 

Check "Enable clear text login" for the IMAP service via "Global Settings" or under "Servers" under IMAP in the Zimbra Administration Console.

If you need to install Perl modules, this is an excellent tutorial on how to do so.

Sample Scripts

These sample scripts for major migrations / multiple users will help you out a lot.

http://imapsync.lamiral.info/examples/sync_loop_windows.bat

http://imapsync.lamiral.info/examples/file.txt

Final word on really considering what you need in Zimbra migrations Diatribe: Migrating email is about moving tonnage.  Migrating calendars is about preserving responses, recurrences, and resources.

But a full migration methodology has to include more than just moving this data. We have a comprehensive suite of scripts and tools so that before any email or calendars are migrated we take care of:

• Reading the Zimbra user list (and passwords)
• Provisioning users in Exchange
• Re-configuring Outlook to point to your Exchange server and removing the Zimbra Outlook connector. (Note: there are publicly available scripts but we wrote our own after we found they did not work.)
• Pre- and Post-cut over scripting so that legacy emails are moved to the target system, and new emails redirected to the target system.

If you have a few dozen users, it's difficult to make an automated process cost-effective and you might as well do it on your own one at a time.  You can export PSTs or export files from Zimbra, then import those PSTs into Exchange.  Tedious and time-consuming, yes, but free.

If you are a larger site with the need to preserve your meeting guest lists, recurrences, responses, and resources post-migration feel free to contact Sumatra.

Quick guide to Enabling IMAP in @Office365 / #MSFTExchange 2013

Let's say you're going to want to start migrating email from a legacy system like Zimbra into Exchange 2013 or Office 365.  You're going to need to get IMAP running on either your on-prem Exchange environment or in O365.

This tells you how to do it.

In a few days we'll show you how to start moving your Zimbra email.  Why are we picking on Zimbra?  We're not, we're responding to market demand to move Zimbra (which is not hard to see since Exchange continues to improve linearly and Zimbra has developed in fits and starts from being independent to being a quizzical part of Yahoo to being remaindered to VMware to passed off to Telligent and now just sold to Synacor). 

There are three pieces to get IMAP working:

1. Start the IMAP service (Exchange 2013.)  Office 365 can skip this step
2. Enable the IMAP Connector
3. Enable the IMAP mailbox feature for user accounts

Start the IMAP Service

The first step is to start IMAP on your Exchange 2013 server (see below).  Once set, you can then enable the IMAP mailbox feature for each user.  (Office 365 customers can skip this step.)  These are the four shell commands:

Set-service msExchangeIMAP4 -startuptype automatic
Set-service msExchangeIMAP4BE -startuptype automatic
Start-service msExchangeIMAP4
Start-service msExchangeIMAP4BE

Step 1: Start two IMAP4 services (and configure those services to automatically start if you wish.)

By default, in Exchange 2013 the IMAP4 service(s) are stopped:

To start those services: On the computer running the Client Access server role:

1. Set the IMAP4 service to automatically start:

     Set-service msExchangeIMAP4 -startuptype automatic

2. Start the Microsoft Exchange IMAP4 service.

     Start-service msExchangeIMAP4

On the computer running the Mailbox server role:

1. Set the Microsoft Exchange IMAP4 Backend service to start automatically.

     Set-service msExchangeIMAP4BE -startuptype automatic

2. Start the Microsoft Exchange IMAP4 Backend service.

             Start-service msExchangeIMAP4BE

In our case, the CAS and Mailbox server roles are on the same box:

Configure Exchange IMAP4 External Connection

This allows users to see (and thus use) the IMAP server.  Here is how via the powershell SET-IMAPSettings cmdlet, e.g.:

     Set-ImapSettings -ExternalConnectionSetting {:993:SSL}.

Note: This requires you restart IIS.

This is true even if you are working within the firewall  Thus in our case, the External Connection is the same as the InternalConnection.

Finally,  Verify things are working, using OWA’s Options select Account, then pick the “Settings for POP or IMAP access” link.

For more information see: http://technet.microsoft.com/en-us/library/bb124489(v=exchg.150).aspx

Enable IMAP for your users
Next, ensure your user mailboxes are enabled for IMAP.
You can do this one-user-at-a-time using the Exchange Admin Center (EAC):

Or via Remote PowerShell for Office 365:

 Set-CASMailbox jimi@sumatra.onmicrosoft.com -MAPIEnabled $True

Which can be set for all users by piping from Get-Mailbox (note that this correctly excludes resource accounts for email migration):

Get-Mailbox -resultsize unlimited ^
  -filter {isResource -eq $false} | set-CASmailbox ^
  -MAPIEnabled $True

You know you are a calendar geek when....

You know you are a calendar geek when you go to the Museum of Modern Art and on seeing this painting:

your first thought is "Out of Facility."  

I'm pretty sure that is not the reaction Ruscha intended.