Oracle calendar to Zimbra via ICS does NOT solve all your problems....

In response to our post: Oracle + Zimbra?  we got at least one person writing in saying "But we can just import the ICS files."

Yes, you can use ICS export/import and it SORT OF works.

You will not have your recurrence patterns and every recurring meeting and appointment will come in as a series of single, disconnected instances.

That is for starters.

Then there are the issues of mapping users throughout the OCS namespace so they map to the new IDs in your Zimbra namespace.

Then there's the issues of conference room / resource behavior that OCS allows but Zimbra does not and how you should deal with that.

@Apple #iCalendar Full-State Migration to Microsoft @MSFTExchange / @Office365

Well we got wind of someone large enough wanting to migrate from Apple Mac OS X Server iCalendar into Microsoft Exchange that we went ahead and actually.... you know.... wrote a conversion method.  It's kind of eerie how much Apple's scheduling solution functions almost exactly like Meeting Maker (down to adopting some down-right goofy functionality like the calendar-only inbox for comments and the "first weekend day" of the month recurrence pattern). But that was probably to be expected.

We did this for El Capitan, so it's based on the iCalendar server being in a back end PostgreSQL database.  Making it work with an Oracle back-end would be no problem.  We read directly from the database and still allow user mappings and create live meetings on the Exchange side when we are done.

This is not yet field-hardened so please do not contact us looking to go into production this weekend.  We won't let you make a decision that egregious.

There's also the issue of how we instantiate Travel Time (it goes in as an appointment with a Free-Busy Status of "Out of Facility").  See our previous post.

We've ignored Contacts for the moment.

Some screen shots.

So this on the Macintosh calendar:

Will become this on Microsoft Exchange:

And of course, meetings are really MEETINGS with guest responses.  So on the Macintosh when Jimi invites Jerry to a meeting and Jerry accepts:

migrating this will carry it through to Exchange or Office 365 as a meeting:

Travel Time, a feature not supported by Outlook or Exchange but loved in Apple calendaring we've implemented by inserting appointments with the appropriate durations in advance of calendar events.

So this on Macintosh

becomes this on Exchange:

The sharp-eyed among you will note that this does something different from our previous post -- it kept the name of the "seed" event in the travel time description.  We experiment a lot with these things.  Any preferences?

Keep an eye on this blog for one of our videos showing this in real-time.  Probably after New Year's.  We've got a lot of migrations going on in December.

To give you an idea what the configuration for this looks like:

You're going to need to map resources to SMTP addresses (not a big surprise I hope).  You'll also notice above we set up the capability to read from a different Postgres database which makes it easier to copy your Apple iCalendar database to a more powerful Windows environment for insertion.

Follow-up to #CalDAV to #MSFTExchange calendar migration tool - round 2

If you have read #CalDAV to #MSFTExchange calendar migration tool - round 2 and said "all very well and good, but I want to LEAVE Microsoft Exchange for some other calendar server" -- WE CAN DO THAT.

We  can already read calendar data directly from the Exchange server and insert it full-state into another Exchange Server or Office 365, as you see here:

Taking all the data out of the source is (for us) simple, inserting it into another target is certainly within our expertise.

If you have at least 1000 users or a realistic budget we can discuss this.

#CalDAV to #MSFTExchange calendar migration tool - round 2

It's been almost exactly a year since we did a sanity check on this #CalDAV to #MSExchange calendar migration tool.

Since then we've field-proven full-state migration of Oracle Calendar Server to Exchange using ICS files -- which means we could insert ICS FULL-STATE for any of the other iCalendar standards based calendars.  You know -- Zimbra, Apple, Google being the only ones that matter these days.

Let us know if you have more than 500 users and want to give it a spin.

New #Oracle Calendar Server to #MSExchange #Office365 migration tools

Lately we have been getting lots of folks (mainly medium businesses and small universities) contacting us looking to migrate their Oracle Calendar Server data into Exchange better than the current crop of low-end solutions can accomplish.  This means three things:

1. Give us recurring meetings and appointments
2. Please please please make the meetings live post-migration
3. We want to choose solutions that reflect our budgets and business goals (i.e., we do not have a lot of money or time)

OK -- we've heard you and we can do all of that.  You could try to write this at home, but we have years of experience that keep us safe.  

So we've created three new options, the first of which we'll let you download and run if you contact us with your work email and size of the migration.

The application allows you to try out what we refer to as our FREE insertion, we think of it as a trial, but if all you need is the next 30 days worth of calendar data from your OCS, hey -- go ahead and use it in production.

The Free version does not re-create recurrence patterns.  You need to license that in what we call the Flat option, which will also include the attendees in the meeting agenda, like this:

The Partial version will actually RE-PROPOSE all current meetings (i.e., not meetings in the past) so that the meetings are real MEETINGS, not static entries in your calendar.  Post-migration your users will need to respond to these meeting invitations (this is part of the deal for lower cost).  But they're live, functioning meetings.  Like this:

Note that now "Attendees" is active because Jimi Hendrix has invited Janis.Joplin to the meeting and that it is remapped to their new domain.

Clicking Attendees shows us that Janis is Required but we have no response from her yet.

Got questions, feel free to contact us.

This also comes with our selective UNDO which will remove ONLY the data we've inserted.

You can see the current version in action in this video:

Want a convenient summary?  Here you go:

If you want your OCS calendar migration…	Choose this Sumatra OCS Migration option
Free, fast, no frills no live meetings, no recurrence patterns.  No history and only 30 days into the future.	FREE -- Inserts ICS export as-is
Inexpensive, fast recurrence patterns and guests in agendas but no live meetings	Flat-- Generates Recurring meetings. Add guests to agendas
Reasonably priced, fast recurrence patterns and current meetings re-proposed (but not responded to automatically)	Partial -- Generates Recurring meetings. Add guests to meetings & sends out proposals
White glove full state-recreation for an enterprise	Full – Recreates Full calendar state recreation for all accounts

Looking to migrate your email as well?

We have found imapsync is an excellent product for email migrations.

Please see our blog postings MDaemon Mail to Exchange via imapsync and imapsync vs PST: Tonnage and Speed (spoiler alert: PSTs are incredibly slow and inefficient) as well as any other recent email migration postings on our blog to give you an idea how easy and fast it is to use imapsync.

Sumatra Decaf: Insert Oracle Calendar ICS into Hosted Exchange

We made our street cred on full-state migrations server-side to server-side.

But there have been folks who either want to try out just ONE user or go for a simpler migration that trades off full-state migration for something faster and simpler.

This is the reason we've done Sumatra Decaf which we're now happy to open for general inspection at http://www.sumatraresourcewatch.com/decaf

A few things to note:

• This works only for hosted Exchange (e.g., Live at Edu, Office365, BPOS)
• We keep track of the domains that insert and limit it to 5 accounts on that domain by default
• We've geared this to the ICS exports from Oracle Calendar System. Others MIGHT work, but they are a lower priority to us. Each ICS upload is limited to 1 Mb and it inserts only from Today (i.e., the date you are inserting) forward.
• In a few days we'll show you how to use this to insert holidays server-side.
• Read the Help PDF at the site for more information.

Web-Based, one-step Oracle Calendar to Hosted Exchange migration

This is an early look at something we call Sumatra Decaf and a call for informed consent volunteers.

We've built a really good business out of migrating calendars full-state into Exchange.

But some folks do not need the whole magilla and are just looking for the simplest way of getting data from your legacy system into Exchange in a "good enough" fashion. So we've done that as well.

So yea, we have server-side ICS insertions to Hosted Exchange.

Let's take a look at Puffy Amiumi's calendar in OCS (also in Japanese Standard Time)

Use UNIICAL to export it and then read it into our web-based application (shown here in Alpha):

and in Outlook Web Access on Live at Edu, the results look like this:

Notice we're prefacing them with _TEST_ for the moment to make sure it does not get into the wild before we're ready.

The 5:00 PM Meeting at which Puffy is a guest looks like this in the agenda so that you know who's organized it and who else is supposed to be there.

Quick, simple, and executable without a lot of prep time.

Now our other option for a migration remains: longer, planned, and full-featured.
We're looking at how we want it to evolve and would welcome some input from you folks.

New xCalReader for using ICS format to migrate Oracle Calendar

Our latest xCalReader now reads .ics files from Oracle Calendar Server for an Exchange migration:

NB: This is because we've been picking up increasing reports of weirdness with the UNICPOUTU format. We've applied all our usual expertise to the iCal exports: including converting RDATE arrays to real recurrence patterns using RTYPE in Outlook / Exchange.
Also this includes support for Zimbra 6 migration.

Oracle Calendar to Exchange using UNIICAL / ICS formats

Yep. We have a version in-house we're using now -- so if any of you out there want to use ICS files to migrate to Exchange and are willing to be informed consent test sites drop us a line.
Oh yeah, same deal: we re-create meetings with responses and have put in tech to turn RDATE strings into real recurrence patterns (your Outlook client will thank you for that).

Oracle Calendar Migration: ICS vs UNICPOUTU

We've been asked why we don't just migrate out of Oracle Calendar using ICS format.

We could, but the results are way better using UNICPOUTU.

Let's look at Jimi Hendrix's meeting with Jerry Garcia and John Lennon in ICS format. The meeting title is "Jimi, Jerry, John"


You'll notice that any of the recurrence pattern syntax is absent, as is any concept of the meeting ATTENDEES. Yes, you can take this and you can insert it as is into Exchange or any other calendar system that reads iCalendar. BUT without recurrences, attendees (and therefore attendee response status) and with no connection to managed conference rooms or resources.

It is very easy to insert this way -- but it is also less functional when it arrives at the target system. Users will need to add back all their guests and send invitations, which the guests then need to respond to. And don't forget the great conference room land grab that's going to happen the second your people get wind the conference rooms are not yet booked on the new system.

And let's look at the same meeting in UNICPOUTU export, Oracle Calendar's own format.

It is CRUCIALLY better because while we are (still) lacking the recurrence patterns (don't worry, we re-create those), we have the attendee list and the attendee responses! This means we can re-create the full state of the meeting and make it live in Exchange. This also allows us to re-create the status in conference rooms.

Yes, this means in a system like Exchange that we need to re-propose the meetings and respond to them. But the results at the end are much more seamless and require less user interaction to get back to its previous state. After doing this for almost nine years though we think we've gotten good at it.

Meeting Maker to Apple iCal via Zinsert Field-Proven

We got a call a from a medical school in Silicon Valley asking if we could migrate 2 users out of a Meeting Maker server of 40 into ICS format.

This is not usually cost effective but they just wanted iCalendar files and they were very polite about the request.

Most times this would not be worth mentioning, but they had a novel approach -- they wanted to obsolete their legacy Meeting Maker server ASAP and move on so they took their data from Meeting Maker into Apple iCal before planning on putting it into Zimbra.

The good news: Seems to have worked fine!

To quote their plan:

...for now it will reside in iCal until Yahoo gets done with the updates that
(we) asked for. Then we will be able to sync iCal to Zimbra.

I really have to give them points for creativity.

Perl script for breaking apart large ICS files for a Zimbra migration

A migration can be really confusing and time-constrained. Since we look at the world through the soda straw of converting calendar data it's easy to sometimes forget there's this whole business of bringing in a new system on (possibly) new hardware, configuring it, provisioning users, training users.... You get the idea.

Sometimes your system is not perfectly tuned and you need to simply get the data in.

Michigan Technical University was having time-outs when trying to load large ICS files into their Zimbra calendars. By large we mean 130,000 to 150,000 lines of iCalendar data.

For a quick fix they produced a Perl script to break the ICS files into smaller chunks. Thanks to Cynthia at MTU we can publish this here for those of you who may need it.

The "splitics.pl" script:

• Splits large calendar ICS files into smaller parts.


• Adds a tag (mm) to all meetings.
  

To run this:
Step 1 - collect the files
Make the ICS files and the zcs script. Some of the ICS files are large.
Put the ics files into a directory, perhaps called "calendar-migration"

Step 2 - Use the splitics.pl script
cd calendar-migration
for i in *.ics; do ../splitics.pl $i; done

Step 3 - create the zcs file to use later
cat *.zcs > /tmp/whole.zcs

Step 4 -
move the "whole.zcs" file to the Zimbra server
suppress resource responses with this command:
zmprov mcr zimbraCalResAutoAcceptDecline FALSE
zmprov -f whole.zcs

Step 5 - undo the suppress responses when ready

NOTE: This script also inserts the tag "mm" to all migrated appointments and meetings.

#!/usr/bin/perl # for i in *.ics; do ../splitics.pl $i; done # cat *.zcs > TODO $base_name=$ARGV[0]; $base_name =~ s/.ics$//; print "$base_name.zcs\n"; open(ZCS,">$base_name.zcs") die "Couldn't open $base_name.zcs : $!\n"; print ZCS "SelectMailbox $base_name\n"; print ZCS "emptyFolder Calendar\n"; open(ICS,"<$ARGV[0]") die "Couldn't open $ARGV[0] : $!\n"; while () { last if (/BEGIN:VEVENT/); $headers .= $_; } $event_count=0; $file_number=1; $event=$_; while() { if (/BEGIN:VEVENT/) { # It's a new event! Yay! if ($event_count > 50) { # See if we should dump the output open(OUTPUT,">$ARGV[0].$file_number") die "Couldn't open $ARGV[0].$file_number for output : $!\n"; print OUTPUT $headers; print OUTPUT $event; print OUTPUT "END:VCALENDAR\n"; close(OUTPUT); print ZCS "importURLIntoFolder Calendar http://yourserver/ZIMBRA/USERID/$ARGV[0].$file_number/n"; # print ZCS "postRestURL Calendar $ARGV[0].$file_number\n"; $file_number++; $event_count=0; $event="BEGIN:VEVENT\n"; } else { # Just keep adding it on $event .= $_; $event_count++; } } else { # It's either the end, or it's another line if (/^SUMMARY:/) { # Tag so we know it's a mm meeting $_ =~ s/$/ \(MM\)/; } $event .= $_; } } # We're at the end of our run... dump what is left over open(OUTPUT,">$ARGV[0].$file_number") die "Couldn't open $ARGV[0].$file_number for output : $!\n"; print OUTPUT $headers; print OUTPUT $event; close(OUTPUT); #print ZCS "postRestURL Calendar $ARGV[0].$file_number\n"; print ZCS "importURLIntoFolder Calendar http://yourserver/ZIMBRA/USERID/$ARGV[0].$file_number/n"; print ZCS "\n"; close(ZCS);

Migrate calendar data from Exchange 2003 to iCalendar via the M: drive

We did a little bit on this a while ago and never saw the need to take it much further, but we figured we'd sketch it here in case anybody else wanted to take this as a launch point.

Suppose you want to migrate your calendar data to Zimbra (or some other place that takes in iCalendar format) from Exchange 2000/2003 under your direct control? Note: For Exchange 2007 you are out of luck with this method (no access to the M: drive).

So instead, expose the M: drive. One of our previous posts tells you how.

You'll be able to drill down to the Calendar folder and see all your appointments and meetings as .EML files. Believe it or not embedded in these EMLs is all the ICS data you need to insert into Zimbra (I'm not promising you're not going to have to modify it, but it's there).

So let's look at one of them.

Received: by trout.sumatra.local id <01C894A4.D2E612E0@trout.sumatra.local>; Wed, 2 Apr 2008 05:35:01 -0400 Content-class: urn:content-classes:appointment Subject: private noon Date: Wed, 2 Apr 2008 05:35:01 -0400 Message-ID: <F47225046904E34AAA50D7CFB96277FC03AED5@trout.sumatra.local> X-MS-Has-Attach: MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C894A4.D2E612E0" X-MS-TNEF-Correlator: Thread-Topic: private noon Thread-Index: AciUpNLoQOwqLSklRJaoHMVqqUnkhQ== Sensitivity: Private x-mimeole: Produced By Microsoft Exchange V6.5 From: "zyg furmaniuk" <zyg@sumatra.local> This is a multi-part message in MIME format. ------_=_NextPart_001_01C894A4.D2E612E0Content-Type: text/html; charset="iso-8859-1"Content-Transfer-Encoding: binary ------_=_NextPart_001_01C894A4.D2E612E0Content-class: urn:content-classes:appointmentContent-Type: text/calendar; method=REQUEST; charset="utf-8"Content-Transfer-Encoding: 8bit BEGIN:VCALENDAR METHOD:REQUEST PRODID:Microsoft CDO for Microsoft Exchange VERSION:2.0 BEGIN:VTIMEZONE TZID:(GMT-05.00) Eastern Time (US & Canada) X-MICROSOFT-CDO-TZID:10 BEGIN:STANDARDDTSTART:16010101T020000TZOFFSETFROM:-0400TZOFFSETTO:-0500RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=11;BYDAY=1SUEND:STANDARDBEGIN:DAYLIGHTDTSTART:16010101T020000TZOFFSETFROM:-0500TZOFFSETTO:-0400RRULE:FREQ=YEARLY;WKST=MO;INTERVAL=1;BYMONTH=3;BYDAY=2SUEND:DAYLIGHTEND:VTIMEZONEBEGIN:VEVENTDTSTAMP:20080414T173637ZDTSTART;TZID="(GMT-05.00) Eastern Time (US & Canada)":20080402T120000 SUMMARY:private noon UID:040000008200E00074C5B7101A82E00800000000E072D44B8394C801000000000000000 0100000002D6C78C6FF3E504D8F29FB17733C0F74 ORGANIZER;CN="zyg furmaniuk":MAILTO:zyg@sumatra.local LOCATION: DTEND;TZID="(GMT-05.00) Eastern Time (US & Canada)":20080402T130000 SEQUENCE:0 PRIORITY:5 CLASS:Private CREATED:20080402T093448Z LAST-MODIFIED:20080402T093501Z STATUS:TENTATIVE TRANSP:OPAQUE X-MICROSOFT-CDO-BUSYSTATUS:BUSY X-MICROSOFT-CDO-INSTTYPE:0 X-MICROSOFT-CDO-INTENDEDSTATUS:BUSY X-MICROSOFT-CDO-ALLDAYEVENT:FALSE X-MICROSOFT-CDO-IMPORTANCE:1 X-MICROSOFT-CDO-OWNERAPPTID:-1 X-MICROSOFT-CDO-APPT-SEQUENCE:0 X-MICROSOFT-CDO-ATTENDEE-CRITICAL-CHANGE:20080402T093501Z BEGIN:VALARM ACTION:DISPLAY DESCRIPTION:REMINDER TRIGGER;RELATED=START:-PT05H00M00S END:VALARM END:VEVENT END:VCALENDAR ------_=_NextPart_001_01C894A4.D2E612E0--

So you remove everything that is not iCalendar (i.e., the email header and that small footer following the END:VCALENDAR) and swap out your email address or domain if necessary. I didn't in the example that follows and it didn't come in properly.

Upload to Zimbra using something like curl, and your before and after shots look like this:

Keep in mind, this will happen one meeting or activity at a time.

Have we rigorously tested every aspect of this? HECK NO! But this gave us enough to go on that we knew if we needed to modify something (like possibly recurrence patterns, exceptions, or maybe the GUID format) that we'd able to figure it out. Again, this is meant to suggest a more general methodology for moving calendar data out of Exchange 2000/2003.

Of course you can do pretty much the same thing with your email as well. We tested that and got attachments to come over. BUT, we learned to add an additional tag X-Zimbra-Received as per this article.

For a masterful discussion of using the M: drive in Exchange for a migration see: Migrating from POP3/SMTP servers using the M Drive

Why migrate server-side rather than client-side?

We spent part of this week looking at how to convert Meeting Maker data into iCalendar data to import into Outlook via Import on the client side. If this is your first time reading this blog please understand that analysis is a little bit like waking up one morning to find that the Kaiser won the First World War.

So why would we look at this? Because sometimes folks come back to us and ask for additional history for some users, and for just a few it's a lot easier to accomplish client-side rather than server-side.

Our usual average server-side migration rate is 750 objects per minute (whether using EWS on E2K7 or CDO on E2K3).

Our method was to use our off-the-shelf technology to convert Oracle Calendar and Meeting Maker into iCalendar for Zimbra insertions. With few modifications we were able to convert to a format Outlook would read directly.

Client-side the average insertion rate is between 200 and 300 objects per minute. Translation: client-side you get one-half to one-third the speed of server-side.

Meeting Maker to iCalendar - no upgrades required

On a phone call this week one of our current migration clients told us that after they were done they were going to upgrade their obsoleted Meeting Maker server. Since this is the first time we've heard this one we asked "why?"

Their answer was a good one: They wanted to be able to export their Meeting Maker data to iCalendar format.

Since we're all about saving you folks money -- we just want to point out that we have an application for exporting your Meeting Maker server to ICS files (i.e., iCalendar), mapping all users and everything.

So if you need it in the future, please just ask.