Inserting Oracle Calendar Server / Meeting Maker into Zimbra

By this point the idea is to make the actual insertion into Zimbra a boring anti-climax.

We have the same goal inserting legacy data into Microsoft Exchange, but because of Exchange architecture this is a little more intricate than Zimbra (So we deal with it separately).

Your primary tool is called zinsert.exe, which you invoke from the Command Prompt (which those of us of a certain age refer to as "DOS").

Simply typing zinsert (-Enter-) will give you a brief summary of functionality.

First off, use ODBC (available under Control Panel) to point to the conversion database. Let's call your ODBC connection "Zimbra."

To create a series of ICS files with appropriate cross-linked GUIDs, use this command:

zinsert -fo -cs DSN=ZIMBRA -tf_utc_to_timezone

In general, converting for the appropriate time zone, use the -tf_utc_to_timezone option unless we determine another one is better (not likely, but we keep them there in case -- you never know what the US Congress is going to do with Daylight Savings Time).

Inserting into Zimbra

That's what the -post command is for, which as of this date we haven't actually had to use (also it will require knowing each user's password which is a pain)

You can use curl, or (thanks to a college in New Jersey), you can use this method.

This solution doesn't require an admin to know the user's password.

It does require a configured web server and assumes there is a separate ICS for each user calendar.

1. The ICS files need to be placed on a web server that the Zimbra server will be able to access via http (or https). I'll assume a base of http://YOURDOMAIN/migrate/.

2. Create a file (e.g. /tmp/importcals.zcs) that will be fed to zmprov with contents like this:

selectMailbox USERID1

emptyFolder Calendar <--- optional command

importURLIntoFolder Calendar http://YOURDOMAIN/migrate/USERID1.ics

selectMailbox USERID2

emptyFolder Calendar <--- optional command

importURLIntoFolder Calendar http://YOURDOMAIN/migrate/USERID2.ics

3. Run that through zmprov.

zmprov -f /tmp/importcals.zcs

So we revised the Oracle Calendar Converter User Interface

We think it's easier to follow. If you're working with us it's in your FTP account.

Mapping Oracle Calendar/ Meeting Maker users for a Zimbra insertion

NB: This is for migration into Zimbra. Migration into Exchange is a whole different issue which we deal with separately.


You have your Oracle Calendar Server data, you've converted it into an Access database -- now you just need to map your users for Zimbra.

Why? Usually there's at least a handful of users who have changed their names through marriage, divorce, or the witness protection program.

Also since you're migrating anyway many sites use the opportunity to standardize their naming conventions.

Step 1: Open the database and go to the Users Table:

There's a lot going on here, but don't let it scare you.

In the Email column just put your new Zimbra email

Note that even though the login for "Adam Ant" is "aant" we can substitute "adam_ant" or anything else we need so long as it matches the Zimbra address.

This simple Query will duplicate your login IDs to the Email field, appending the appropriate domain name (which you'll need or else guests will not be notified of meeting updates). Of course, this is generally modifiable into anything that makes sense for your environment.

Next step: inserting your data!

Converting your OCS exports to migration format

So you've gotten your calendar data out of Oracle Calendar Server and now you wonder "What ELSE do I have to do with it?"

Well, you have to convert it into a (new) relational database so that we can reassemble any attributes not exported from OCS.

Like what you ask? Guest lists, guest responses, recurrence patterns, all the sorts of things you want.

Do you need to go through this? Well, if you want to do this all server-side and keep all the information that makes schedulers valuable, yes.

The tool that takes your exports runs in Windows and looks like this: How do I use this?

While the design reminds you of an early farm tractor (which we are very proud of) - it's designed with the same single-minded focus: get the job done as quickly as possible and the heck with how it looks (you're tossing it out after this is done anyway).

The idea here is to take in the files you got from UNICPOUTU, and match them with users and resources from USERS.TXT and RESOURCES.TXT.

Specifically this will:

• Match export data to users and resources (the users and resources will be mapped to your new calendar system)
• Add guests to your meeting
• Group one-time instances into “recurring meetings” (makes the insertion run faster)
• Remove old items you do not intend to insert (makes the insertion run faster)
• Reformat OCS-specific items to calendar-neutral terms (e.g. map Daily Notes and Daily Events to All Day Events)

Note that if you're planning on multiple test insertions you only need to read the users and resources into the database once. It is possible to parallelize conversions, e.g., convert A-L on one machine, M-Z on another and join them together to get the job done in half the time (but please talk with us before you do that).

This also includes a blank template database (MS Access format), which we recommend you rename a copy of to something descriptive, like "December 1 migration test." Please preserve the database version in the file name; it really helps when debugging.

Let's go into some details on the specifics.

Time Zone: Select the Time Zone of your OCS node. This is really crucial otherwise everything will be off on your insertion by some multiple of hours. If you have users outside of the default time zone you can modify the USERS.TXT file for them (see earlier post).

Drop Items Prior to this Date: You can selectively export items from certain date ranges -- but in case you have a data set with everything and want to pare it down for a test insertion, you have that option here.

Threshold for recurrence patterns: When we get the events from OCS they are all individual events rather than a series. We reconstruct the series, but (as a little thinking will show you) with exceptions to recurrences this can get a little dicey. Our software analyzes the sequence and comes up with its best guess of the recurrence pattern, taking account the threshold you set.

A couple of quick things to note:

• Historical events (i.e., any events before the date you convert them) all go in as individual events (it's faster this way) with meeting guest list in the Agenda field (except for BCC guest whom only the meeting proposer sees)
• There needs to be at least five recurring events before we attempt to re-create the sequence. (Quick math -- at an 80% threshold, one exception to a five event pattern puts you at the threshold)
• If we can't match a pattern all events will go in as individual instances. So worst case scenario is that a recurring meeting or event comes in as a series of individual instances. (But as opposed to an ics RDATEs list these will display properly in Outlook clients).
• If you migrate only a subset of users, meetings to and from users outside that subset will not show up. Think about this one: if the meeting owner or guest is not in the system you're migrating to -- how can they be on a meeting in your new system? This is the main reason we're always advocating "Big Bang" migrations for meeting-centric corporate cultures.

How does this work?

In two phases: the first phase reads the OCS exports and turns them into an intermediate file. The second phase reads the intermediate file into the database template.

Why not just put everything into the database as we're going along? It's faster this way.

Everything in a calendar server migration is a trade-off between time and convenience.

It's certainly easier for the average admin to work with an Access file. But the overhead of constant transactions at the volume we need is huge. Saving them for the end makes the system much faster.

How to Extract Data from Oracle Calendar Server for a migration into Exchange or Zimbra

This is Part I of a three-part series. Part II will be about converting this output and Part III is about inserting it into your target system.

So you've seen your future and it no longer involves Oracle Calendar Server (OCS).

We're there with you.

The problem we deal with now is: How to get the Data OUT of OCS.

As usual, we like to rely on the native tool set of the legacy system (less code on the legacy system implies easier to maintain the migration process as a whole).

Let’s say you run a single Oracle Calendar Server node (number 1) with administrative password “jimmorrison” and have users John Lennon, Jerry Garcia, Jimmy Page, Puffy Amiumi, and Walter Liberace. As well as the Mozart Conference Room and Shea Stadium. The following command lines will pull all data Sumatra requires:


uniuser -ls -format "%s%:%g%:%uid%:%id%:%node-id%:" -n 1 -p jimmorrison >users.txt
uniuser -resource -ls "S=*" -n 1 -p jimmorrison >resources.txt
unicpoutu -u "S=Liberace/G=Walter" -f exp-liberace.txt -n 1 -p jimmorrison
unicpoutu -u "S=Garcia/G=Jerry" -f exp-garcia.txt -n 1 -p jimmorrison
unicpoutu -u "S=Lennon/G=John" -f exp-lennon.txt -n 1 -p jimmorrison
unicpoutu -u "S=Amiumi/G=Puffy" -f exp-amiumi.txt -n 1 -p jimmorrison
unicpoutu -u "S=Page/G=Jimmy" -f exp-page.txt -n 1 -p jimmorrison

Execute this batch file to extract these data files, and place them in a convenient directory.

How this is organized, and what we do with it.

The first two lines extract the names of all users and all resources in a standard format.

Sumatra Development LLC reads these two files in first to establish the list of users and resources being migrated. These become really important to make sure the connections among users is maintained when we insert into either Exchange or Zimbra (Notes 8 coming if we get sufficient interest). The programmers among you will recognize this is crucial to maintain GUID integrity.

THEN, all user calendar data in the exp-(name).txt files are read in, and matched against these users and resources.

The users.txt file looks something like this:

Garcia:Jerry:Jerry.Garcia:256:1:
Lennon:John:John.Lennon:257:1:
Amiumi:Puffy:Puffy.Amiumi:258:1:
Liberace:Walter:Walter.Liberace:260:1:
Page:Jimmy:Jimmy.Page:262:1:

NB: It is certainly possible to script these so that the output from the initial users export can be the input for a script that automatically exports all user data/

For users with a different default Time Zone from the node time zone, manually add the Oracle Time Zone code following them in the user.txt file.

Let’s say Puffy Amiumi is based in Japan and therefore uses JST.

The resulting users.txt file would be the following:

Garcia:Jerry:Jerry.Garcia:256:1:
Lennon:John:John.Lennon:257:1:
Amiumi:Puffy:Puffy.Amiumi:258:1:JST
Liberace:Walter:Walter.Liberace:260:1:
Page:Jimmy:Jimmy.Page:262:1:


The resources.txt file looks like this:

. R=Shea Stadium/N=ResNum0000/CA=200,000/S=Steinbrenner/G=George/
+ LOC=Da Bronx\r\nNew York/PHONE=617-555-1212/FAX=617-555-1213/ID=259/
+ EMAIL=shea@sumatra.local/UID=Shea Stadium/ALLOW-CONFLICT=NO/ENABLE=TRUE/
+ LANG=en/NODE-ID=1

. R=CR Mozart/N=ResNum0002/CA=10/S=Mozart/G=Wolfgang/
+ LOC=3 Colonial Terrace Strasse\r\nWien, Ostereich/PHONE=617-555-1212/
+ FAX=617-555-1213/ID=261/EMAIL=crmozart@sumatra.com/UID=CR Mozart/
+ ALLOW-CONFLICT=NO/ENABLE=TRUE/LANG=en/NODE-ID=1/PUBLISHEDTYPE=PUBLISHED

More on OCS Calendar Data
Our recommended method for extracting Oracle data is to use the off-the-shelf Oracle utility:

UNICPOUTU

This is typically found in the directory:

C:\ocsinfra\ocal\bin on Windows servers
ORACLE_HOME$/ocal/bin on UNIX/Linux servers

The command:

UNICPOUTU –u “S=Garcia/G=Jerry” –f output.txt –n 1

will export Jerry Garcia’s entire calendar from OCS Node 1 to the output file output.txt.

You will be asked for the calendar administrator password (or alternately can pass it with the –p parameter).

This is also configurable for different start and end dates, depending on how much data you wish to transfer.



So if you only wanted to take data since January 1, 2006, you would use:
UNICPOUTU –u “S=Garcia/G=Jerry” –f output.txt -start 1 jan 2006 –n 1



NB: Dates can only be in "day month year" format.


Resources
If resources are only guests and do not have their own non-meeting events, you do not need to export them. If however you have reason to, we include the information here.

The command:

UNICPOUTR –u “R=CR Mozart” –f output.txt –n 1
Will export the Conference Room “CR Mozart” on OCS Node 1 to output.txt

Users for a node
The following command will export all of the users on Oracle Calendar Server node 1:

UNIUSER –user –ls “S=*” –n 1

The following command will export all of the resources on Oracle Calendar Server node 1:

UNIUSER –resource –ls “S=*” –n 1

Designate Access Rights
What in Outlook / Exchange are called the Delegate rights are in Oracle called Designate rights. To extract these use:

UNIACCESSRIGHTS –ls
More detail on this can be found in your Oracle Calendar Server Reference manual.
Sumatra does not currently insert Designate Access Rights into Exchange or Zimbra from OCS – but if this is a requirement for you please consult with us (the issue is more about trade-offs on the Exchange 2003 side than the Oracle side or the Zimbra).

Why not UNIICAL?
If you're asking this question you probably already know the answers:

• It does not export ATTENDEES
• It does not export ATTENDEE responses
• It exports RDATEs rather than RRULEs (which will cause you trouble down the line, not least of which because it really does a number on Outlook if you use that as a client for any target systems)

Still with me? Wait a few days for Part II: Converting and Mapping your OCS data.

So Yahoo acquired Zimbra?

So Yahoo acquired Zimbra?

Good.

I was never impressed with Yahoo's calendaring.

Here's hoping it improves.

Migrating Data from Resource Scheduler to Exchange 2007 Resources

A typical week here at Sumatra involved at least one request to take data from one system we've never seen before and put it into one we know (or vice-versa).

A few weeks ago it involved a request to migrate data from Resource Scheduler for Exchange from PeopleCube.


The idea was that they wanted to start using the native room and resource scheduling capability in Exchange 2007 instead additional software.


This presents some interesting problems.

Before we coded anything, we thought we'd ask our reading public if there was anyone else out there who wanted to migrate Resource Scheduler data, because it's only worth doing this if there's more than one site.

Oracle Calendar Server -- Multiple User Names and Migration

We were astounded to discover in customer data that it was possible to have non-unique user names in Oracle Calendar Server.

However, as the screenshots show, it was very easy for us in our test lab to create two "Jerry Garcia" users:

Despite the fact that I entered a valid password for only one of them I get the following screen on log-in asking me to select the one I mean (as though there's a way to tell them apart?)

This is bad enough -- but we then took the next step and invited them both to the same meeting.

Using the standard UNICPOUTU output this is the resulting data:

S 8780430

D 30

T Meeting with the two Jerrys

I 0

R N2

M Lennon John

W Lennon John

A TRUE 3 10

C [ .+] Garcia Jerry

C [ .-] Garcia Jerry

O

... making it impossible to tell which Jerry is which.

SO we've just added the capability to detect this in the USERS.TXT file before we get too far into a migration. If this situation arises, an error will pop up:

Yet another significant Entourage / Outlook discrepancy

If some of your users have Entourage clients and create recurring meeting requests (and if they're not creating recurring meeting requests what are they doing with an Exchange account anyway?), then please check out this recent KB article (number 933093):

You receive a message that states that your meeting request was declined when you use Entourage for Mac to send a recurring meeting request to an Exchange 2003 resource mailbox

Oracle Calendar Server to Zimbra and Exchange

Yep, we've got Oracle Calendar Server to Zimbra migration working.

John's calendar in OCS 9.0.4 looks like this, note the Declined meeting (in red) and the Tentative meeting (in blue):




In Zimbra it converts to this (with the Tentative and Declined acting as they should in Zimbra):

As we've told some of you, the tool that we used to take the OCS exports and re-create recurrence patterns (ICS as RRULES) is being re-worked for field use (as opposed to the way we used to do it in Sumatra labs). It looks like this:

with our trademark "as few buttons as possible but make all of them useful" design philosophy.

In Outlook / Exchange 2007, the same calendar looks like this (Declined and Tentative are dealt with in the Outlook INBOX rather than directly on the calendar):

Note also that the OCS Day Event becomes an All Day Event in both Outlook/Exchange and Zimbra.