#CalDAV to #MSExchange calendar migration tool

A quick gut-check from our loyal readers: do you guys want a CalDAV to Microsoft Exchange migration tool?

UPDATE December 2016:  Folks: We did this.

The Cookbook Version of Exchange 2013 Migration Rights

Gentle reader,

No matter what you want to do with calendars server-side in Exchange, you are going to need to become conversant in Exchange permissions.  It's the devil's bargain: if you want to migrate all your calendar data and preserve its utility with guest lists and responses, then you need to be able to manage Permissions on your Exchange users.  

So here we're putting out our best effort at a cookbook guide to Exchange 2013 permissions for migrations.  Even though full-state calendar migrations will never be a purely cookbook operation we think this will get you there (and we have some field experience taking a couple of novices through it this way),

First:  GLOBAL ADMINISTRATOR rights are NOT enough.  These rights give you administration rights over Exchange / Active Directory, but they do not give you the rights to access mailboxes – which is what you will need to move in data and re-create state.

We’re going to take this in stages.

Background. Your ADMINISTRATOR account needs to be able to:

• Use REMOTE POWERSHELL to Log into Office 365
• Create a separate service account (this keeps your ADMIN function separate from your MIGRATION function)
• We call the Service Account EXSU.  When you create it, make sure it is mailbox-enabled (you will be sending email on behalf of this account
• Grant EXSU three rights:
• Impersonation
• No throttling.  This is relevant (i.e., in your control) only for on-premises Exchange.  For Office 365 you will need to contact your Microsoft rep and explain what you are doing and ask throttling turned off for the duration of your migration.
• FULL ACCESS to mailboxes (this makes it easy to use OWA with this account to check the results for individual users in testing and migration)

Execution.  To do this – use the various Exchange PowerShell cmdlets which execute the appropriate actions.

Start POWERSHELL.

REMOTE to your OFFICE365 account

IMPERSONATION: You’re creating a ROLE called “_suImp8” that allows Impersonation and then assigning it to EXSU

THROTTLING: You’re creating a policy called SuThrottling Policy and then assigning it to EXSU.  (Otherwise Exchange 2013 might shut you off mid-migration)

FULL ACCESS:  this grants access to ALL MAILBOXES in your domain to EXSU.

Test.  Can you put the EWS URL in a BROWSER and when prompted for credentials get this?

Log in with your EXSU credentials.
You should see this:

This example shows access to Office 365.  Obviously if you are going into your on-premises or your own hosted domain, your URL and service name will be different.

Now to test FULLACCESS go to the URL box and modify it as I have with a user on your domain:

Hit ENTER

Now you will be prompted for your end user mailbox credentials.   Use the service account (EXSU) and the password to access to the mailbox.  This is where FullAccess comes in – you don’t have to crack all of your end users' passwords!

NOW it should display your test users’ mailboxes in OWA

If all of these are successful, now you can do a test insertion!

Congratulate yourself.  Permissions are historically the worst learning curve in a migration.

Hosted Oracle Beehive to Office 365 Field-Proven

We had an informal bet, Zyg taking the side that Hosted Beehive to Office 365 was not going to work the first time out and Russ figuring that it was.

Russ won. 

Data goes in like a champ in the current version of the Beehive migration code.

Since we're having to go through a proxy server performance is much lower than we see in on-premises migrations -- but that is what multiple instances are for.  Stay tuned for more news.

Burt's Bees calendar Advertising

From the New York Times, Burt's Bees is promoting a new line of anti-aging products via opt-in calendar events.

I first saw calendar spam in Yahoo calendar in 2008.  This at least is an active opt-in which I have no problems with.

Going to be interesting to see if it catches on.

The Curious Case of the Outlook 2013 Contact Notes

UPDATE 9/4/14: Microsoft released a fix for this issue. See our Disappearing Contact/Calendar item body fixed in Exchange 2013 CU6 blog post


Found something weird with Outlook 2013.

This showed up in our Beehive to Exchange migration but it would not be limited to that particular situation.

Let's look at the results first.  All of these clients were accessing the exact same contact record on our Office 365 test system, though we have confirmed the same behavior with Exchange on-premises as well (no surprise there since this looks to be an Outlook 2013 bug).

Contacts and Notes (including some data we capture to the contact notes either because of EWS bugs or there's no other place to put it) show up fine in Outlook 2007 post-migration.

And they show up fine in Outlook 2010.

OWA and the Surface are no problem.

THEN we get to Outlook 2013:

PROBLEM!  Where did the Notes go?  The situation seems to only show up if you have a contact created using EWS.

This gets weirder because you can CREATE a contact with notes in either OWA or Outlook 2013 and it will properly display and edit in either.

This gets EVEN WEIRDER because in Outlook 2013 you get Notes displayed correctly in the PEOPLE view....

but not in any other view (Business Cards for example):

We think this is fairly definitive proof that Outlook 2013 has a bug in it.

Exporting Contacts to a CSV file using the EWS Managed API and Powershell

Exporting Contacts to a CSV file using the EWS Managed API and Powershell by Glen Scales will show you how to get this information OUT of Exchange server-side if you want to.

Normally we see people wanting to put this information INTO Exchange server-side, but there are legit reasons for wanting this capability:  archiving crucial customer data from former employees, saving vital contact data, legal discovery, migrating to Google, etc.

For the same reasons you will want to check out his scripts in Export the GAL or Address list with EWS to Vcards in 2013 MEC sample 1.

Arm yourselves with knowledge.

Mailbox Migration to Microsoft Exchange Performance Analysis

Microsoft recently gave guidelines on Mailbox Migration Performance Analysis.  

The article is a perfect example of how to drown in data before you come up with any concrete solutions -- if you ever come up with solutions.

Basically, read the article BACKWARDS to have it make more sense.

Look at all the reasons Office 365 can be slow (and there are a lot of them)

Then look at the reasons YOUR side might be slow (the suggestion of "get closer to our data center" is REALLY useful, thanks for that, guys!).

THEN consider putting (more of) this in YOUR OWN control with imapsync.  Might as well be frank, the Microsoft data center is a black box and will stay that way.  But you can seize more control over how your data is processed on your side of the wall.  Read our posts on email migrations to see how this works out in the real world.

And OF COURSE the Microsoft analysis and migration tools ignore calendars, tasks, and contacts, but we've all come to expect that from Microsoft.  That is why you have us at Sumatra.

Distribution Lists Now Working in MDaemon Calendar Migration

We've been on a mission to completely automate an MDaemon to Exchange migration.  Email, contacts, calendars, the works.  We'll tell you more about that later after it's fully field-proven.

In this case "the works" also means MDaemon private distribution lists (which in the language of Exchange/Outlook are called "groups") which we have added to our calendar migration application since (as glorified contacts) that is where they belong.

Post-migration in Office 365, it'll look like this:

We do the smart thing in this case:  if a contact on the list is on the legacy domain and needs to be mapped to a new domain or a different email, we automatically do that.  Users outside the legacy domain we leave alone.

Please note: In Exchange, users create distribution groups by linking to existing contacts (or contacts from Active Directory). This way, when a contact name/email address changes, the distribution group gets updated.  However, in the case of this migration, MDaemon does not provide any  indicator if the entry is an existing contact.  We don’t want to automatically create contacts because if they do exist, the user will see double contacts – something that will confuse them.  So communicate to your users that if an email address changes in the private group mailing lists, they will have to either manually edit and fix the name, or delete the name and link it to a previously created contact.

Fully automating an #MDaemon to #MSExchange 2013 migration

One of our customers wanted to migrate from MDaemon to Exchange in stages (vs our typical "big bang".)  A staged migration is more delicate.  It requires you configure (and then re-configure) active directory accounts and Exchange mailboxes pre- and post- migration.  It's not hard, just a lot of steps. This called for a script!  And so, we did.

We have successfully field-proven that an MDaemon to Exchange 2013 on-premises migration can be fully scripted.

We're going to outline our approach:

First some background on the main show: email and calendars: 

• For email we used and recommend imapsync.  It's an excellent, effective, and efficient product.
• For calendars, tasks, contacts, and distribution lists we wrote our own application.

But a full migration methodology has to include more than just moving the data.

So before any email or calendars are migrated we take care of:

• Reading the MDaemon user list (and passwords)
• Provisioning users -- first as Exchange contacts, then as mail-enabled users
• Re-configuring Outlook to point to your Exchange server and removing the MDaemon Outlook Sync. (Note: there are publicly available scripts.  But we found they don't work. After hours of debugging, we gave up and wrote our own.)
• Pre- and Post-cut over scripting so that legacy emails are moved to the target system, and new emails redirected to the target system.

If you have a few dozen users, it's difficult to make this cost-effective and you might as well do it on your own  (Hint:  you can export PSTs from MDaemon, then import those PSTs into Exchange.)  Tedious and time-consuming, yes, but free.  This assumes your time is worth less than the cost of third-party migration tools.  If it takes as long to just DO it (apologies to Nike) as it does to deploy a third-party customized method, just do it.

If you have several hundred users and want to license our tools and methods, you can contact us.

Transferring Palm Desktop Calendar and Contacts Data

Palm Pilots.  

There's nostalgia for you.  

But apparently the data can live on in their Desktop app.

The New York Times had an article on Transferring Old Palm Desktop Data (to Outlook) that is worth having around in case you find yourself in that situation.

Needless to say this is client-to-client.

Why Room 222?

Why we get asked, are our example resources called Room 222?

Millennials....jeesh.

Well, Orwell's Room 101 is a little too downbeat.

But a 1970's-era show that took place in an American public school has some positive connotations. You might recognize Michael Constantine (the principal) from My Big Fat Greek Wedding.

Please nobody ask us who this "Jimi Hendrix" guy is that we also use an example.  

Must Read: Tips for Working Faster in Microsoft Outlook.

The folks at Lifehacker did a fantastic article on 12+ Tips and Tricks to Work Faster in Microsoft Outlook.

Read it and pass it along to everyone else in your organization.

Scripting Sumatra's MDaemon to Exchange calendar migration tool

We've turned the mCalReader application for migrating calendars from MDaemon to Exchange into a scriptable application.

So from the Command Prompt or PowerShell you can issue the commands

mCalReader /usersrc:  /userex:  /cfg: /undo

What each of these mean:

• /usersrc:  the full email address of your source user on MDaemon.  e.g., zyg@sumatra.local
• /userex:  the full email target address on Exchange.  e.g., zyg-furmaniuk@sumatra.onmicrosoft.com
• /cfg:  the full path to the configuration file
• /undo:  remove the data mCalReader inserted for this user.

So a typical scripted insertion for a user would be:

mcalreader 

  /usersrc:zyg@sumatra.local 

  /userex:zyg-furmaniuk@sumatra.onmicrosoft.com 

  /cfg:"C:\Docs and Sets\Administrator\Desktop\mCalReader\_Config_bCalReader.xml"

To remove Sumatra-inserted data:

mcalreader 

  /usersrc:zyg@sumatra.local 

  /userex:zyg-furmaniuk@sumatra.onmicrosoft.com 

  /cfg:"C:\Docs and Sets\Administrator\Desktop\mCalReader\_Config_bCalReader.xml"

  /undo

Each individual user using this method will get their own individual log file:

How do you set the Configuration file?  Simple: Via the user interface to the tool.  You CAN edit the XML if you want, it's pretty straightforward, but the interface exists to make it as simple as possible to test that access to Microsoft Exchange is working.

Create a Company Shared Contacts Folder in Office 365

We got asked to look into shared contacts folders in Office 365.

It turns out we did not need to do anything because some folks at Cogmotive have already done it.

Check out their article Create a company Shared Contacts folder in Office 365.  It's very detailed and thorough.

imapsync vs PST: Tonnage and Speed

Our first lab test of volume for imapsync resulted in an average throughput of 11.8 message/second, or 830 kb/second transferring 3013 messages with 358 skipped messages (I think that had to do with headers and the way we treated them).

We then went into the real world with a Boston-based firm and compared moving PSTs to moving via imapsync.

One PST for a user took 15 minutes to export client-side.  We did not even bother continuing to time after that.

Using imapsync (for email) and our mCalReader (calendars, tasks, contacts) we read that user AND TWO OTHERS and inserted all three in 8 minutes.

Or imapsync has at LEAST 6 times the throughput.

Our conclusion was that imapsync is MUCH faster.  Our further conclusion is that for purposes of migration PSTs fall far short of ideal.  PSTs really suck to tell you the truth.

In going through some of our previous posts, I discovered an interesting factoid that we published years ago when comparing client-side migration speed vs. server-side migration speed.  Server-side was 2-3 time faster on insertion than a client side migration just for calendars.

MDaemon Mail to Exchange 2007 via Microsoft Transporter Suite

Gentle reader,

Today we will use Microsoft Transporter Suite for migrating email from MDaemon to Exchange 2007.

Why?  Obviously because we're going to get to moving calendars for this legacy product, but you only get to calendars if you're also doing email and we get asked about email, so we're documenting it here.

This is for on-premises Exchange migration to Exchange 2007 ONLY.  You want to go into Office 365, the read our previous article How to Migrate Oracle Beehive Email into Office 365 because the method will be the same.  

Why 2007?  Believe it or not Exchange 2007 is still out there, and we still get questions on it.  Might as well deal with reality.

We've also tested email migrations with imapsync (spoiler alert: imapsync is MUCH BETTER).

But it starts with Transporter since it's there, it's free, it's from Microsoft.  What is not to love? Quite a few things, actually, but free is a big draw.

First, download and install it.  But: WHERE should you download it and install it?

You won't even be able to get to the documentation (all in the form of help files) until after you install it -- so I'll cut to the chase:  you must have Exchange Recipient Admin rights and Exchange Impersonation rights on a computer with the Client Access server role installed for Exchange 2007. Later version of Exchange will of course require slightly different methods of setting these permissions (and we've blogged on those enough).

Refer to this section of the in-application help.

For MDaemon we're going to be using the Transporter for Internet Mail. The installer seems to already know we do not have Lotus Notes installed,

but  then very helpfully reminds us in case that was what we were really after.  

We're not in this case, so let's just accept the EULA and get this done.

Now we can actually execute the Transporter

You are now in a position to actually start defining which user mailboxes you will move.

But you need to give it a list in a CSV-- and you notice a distinct lack of a user manual. What should the format of this list be?

In one of the few times I will ever write that the Help Files are actually.... HELPful, I am bidding you to Click on Help and read them.

The format is then readily copied and uploaded.

My file looks something like this:

You'll see something like this and are ready to import.

Now migrate your email.

Corrupt MDaemon meetings in a migration

We found (in the wild) conditions where MDaemon meetings were corrupt, in this case missing their Organizers or Planners.

What to do?  They're corrupt and may or may not display in an MDaemon client -- but they are in the data.

Under our usual rubric of "it is better to recover everything we can and let the user delete if they do not want it -- because it is harder to re-create something not there than delete something that is" we recover the meeting and tag it with the category "Meetings_MissingOrganizer"

So you could search for those in Outlook post-migration and make decisions about them.

MDaemon Mail to Exchange via imapsync

Gentle reader of migration mindedness,

Today we will use imapsync for migrating email from MDaemon to Exchange.

Why?  Obviously because we're going to get to moving calendars for this legacy product, but you only get to calendars if you're also doing email and we get asked about email, so we're documenting it here.

Using imapsync will cost you all of 50 Euros.

Goodness, gracious it is worth it!  

This application has significant advantages over other products:  

1. It is really simple to install and use.
2. The "sync" in the title is serious.  You can upload all the data from a user set during working hours and then cut over the incremental changes starting on a Friday after closing time.
3. You can also use imapsync on a Linux environment as well as a Windows environment.
4. It is all in your control as opposed to run through someone else's data center or through a major integrator looking to run up hours.
5. It is very reasonably priced with the most liberal license I have seen.

We use the Windows version in our testing.

imapsync runs exclusively in the Command Prompt.

First off, please make sure you have enabled IMAP on your Exchange 2013 server.

Migrating from MDaemon to Office 365 looks something like this (if you use individual passwords for users).

imapsync.exe 
--host1 147.1.41.1 --user1 zyg@sumatra.local --password1  "XXXX" 
--host2 outlook.office365.com --user2 zyg@sumatra.onmicrosoft.com --password2 "XXXXX"  
--ssl2

If you set up a service account with FullAccess, you can accomplish a migration with a command like this:


imapsync.exe 
--host1 147.1.41.1  
--user1 jimi.hendrix@sumatra.local --password1  "XXXX" 
--host2 outlook.office365.com --port2 993 --sep2 / 
--user2 jimi.hendrix@sumatra.onmicrosoft.com  
--authuser2 riuliano@sumatra.onmicrosoft.com 
--password2 "XXXXX"  --ssl2

Note in  the above, for an Office 365 target system we need to use the "--sep2 /" command. 

Executing will give you some excellent statistics and feedback.

Other things to be aware of

Exchange Configuration Requirements:

Before you can run imapsync, you will have configure Exchange for IMAP.  This requires two steps.

Step 1: Start two IMAP4 services (and configure those services to automatically start if you wish.)

By default, in Exchange 2013 the IMAP4 service(s) are stopped:

To start those services: On the computer running the Client Access server role:

1. Set the IMAP4 service to automatically start:

     Set-service msExchangeIMAP4 -startuptype automatic

2. Start the Microsoft Exchange IMAP4 service.

     Start-service msExchangeIMAP4

On the computer running the Mailbox server role:

1. Set the Microsoft Exchange IMAP4 Backend service to start automatically.

     Set-service msExchangeIMAP4BE -startuptype automatic

2. Start the Microsoft Exchange IMAP4 Backend service.

             Start-service msExchangeIMAP4BE

In our case, the CAS and Mailbox server roles are on the same box:

Step 2: Configure Exchange IMAP4 External Connection so users can see (and thus use) the IMAP server settings

Use the powershell SET-IMAPSettings cmdlet, e.g.:

     Set-ImapSettings -ExternalConnectionSetting {:993:SSL}.

This requires you restart IIS.

This is true even if you are working within the firewall  Thus in our case, the External Connection is the same as the InternalConnection.

Finally,  Verify things are working, using OWA’s Options select Account, then pick the “Settings for POP or IMAP access” link.

For more information see: http://technet.microsoft.com/en-us/library/bb124489(v=exchg.150).aspx

These sample scripts for major migrations / multiple users will help you out a lot.

http://imapsync.lamiral.info/examples/sync_loop_windows.bat

http://imapsync.lamiral.info/examples/file.txt

I really like the way this user lays out a basic sequencing for migrations including how and when to change your MX records.

http://hadjigeorgiou.com/blog/mdaemon-to-microsoft-exchange-migration/

And on the more than crucial need for advance planning and testing, please read this thread.

#MDaemon Calendar Migration to #MSExchange / #Office365: Video and Documentation

We have calendar migration from MDaemon into Exchange working.

Here's the video.

The migration features are pretty much all dealt with in the video.

The main screen of the application gives you our usual set of options:

And the configuration holds no surprises.

The site we're doing this for has decided it's more important to migrate in a phased sequence of groups of users, so we're taking the calendars over as "flat" but with information on attendees and status in the Agenda/Notes body.  If there's demand for a full-state method we can do it.  But this keeps costs down.

Mapping Users

If you have a user in your legacy domain named "zyg" and in Exchange the ID is "zyg-furmaniuk", you enter the following line in the exceptions.txt folder:

When you validate, you will find the mapped (i.e., correct for the target) address.

Note that in the above example Room.222 did not validate, so either our mapping is wrong, or the account on Office 365 is not set up properly.  

Permissions

You will need a service account on the Exchange side.  This account needs permission to write to all accounts you're mapping into.  We've already blogged on permissions a lot. 

Firewall Requirements 

1. Your fire wall must be configured to allow Time Sync (we need Port 13 and we cycle among time servers, the top two being 64.90.182.55 and 206.246.118.250).
2. Your computer time should not differ from the US Navy Atomic Clock time by more than 1 hour or mCalReader will inform you.

Why do we require this?  Short answer: Digital Rights Management and version control.

Determining your EWS URL
For on-premises Exchange, the EWS URL formula is HTTPS://CAS_server/EWS/Exchange.asmx
In ON-PREMISES you will usually have your IIS set for Windows Authentication (see http://technet.microsoft.com/en-us/library/gg247612.aspx for more details). This is also the default in hosted Exchange. Should you need to change this you may do so in the _Config_XML file by changing the HTTPAuthType parameter (options are Basic, Negotiate, ntlm, and Kerberos)
NB: You hear us talking about Exchange being a moving target in a migration. That’s true here. The default is Negotiate in Exchange 2013, and Basic in Exchange 2007 and 2010.

For Office 365 it is: 
https://outlook.office365.com/EWS/Exchange.asmx

Sumatra and MEC March 30-April 2

Two folks have asked us about us being at MEC  #IamMEC.  One's wanting calendar migration (and at over five thousand users we can talk).  The other about a private label for the holiday cmdlet.

In general we do not do conferences, but we think we're going to head to this one.   Contact us if you want to get together for a beer.  Like everyone else in Exchange programming we certainly owe Glen Scales one.