Exchange 2010 Permissions summary and debugging

Depending on the situation there are four permissions your service account must have to function successfully in either a migration or using Sumatra calendar tools:

1.       Impersonation

2.       Full access

3.       Read access to the GC

4.       Other details: Allow Log On Locally

In which situations are these permissions used?

This summarizes the types of permissions you must use when using Sumatra calendar technology.

Situation	Use Permission	Notes
Migrating Calendar Data into Exchange	Impersonation	Resource mailboxes are Disabled accounts by default, so in a full-state calendar migration they are ENABLED temporarily so that the Sumatra process can populate data correctly.
“Faster simpler” ICS calendar migration to Exchange	Impersonation
Using the SuHoliday cmdlet or the Sumatra Pump on users	Impersonation	Putting holidays into user calendars requires only impersonation
Using the SuHoliday cmdlet or Sumatra Pump on resources	Full access	Why Full access in this case?  Impersonation will not work unless you enable the accounts.  In a migration there are many reasons for doing this, but for holidays that is a wasteful extra step.  Use Full access.
Terminating an Existing User	Impersonation	It’s basically a migration in reverse, so you use the same permissions as a migration
Removing broken meetings from resource or user calendars	Full access	Don’t mess around in this case.  You’re trying to scrub out bad data, don’t let low permissions get in the way of a fast job.

Impersonation

Impersonation grants the service account permission to ‘send-as’, and ‘receive-as’ the user account.   Note, however, that impersonation works only when the account is enabled.  For disabled accounts you will need full access.

To impersonate in Exchange 2010, create a new ManagementRoleAssignment (called “_suImp8”) for your service account (called “exsu”.)

new-ManagementRoleAssignment    -Name:_suImp8    -Role:ApplicationImpersonation    -User:exsu@cod.sumatra.local

Full Access, Send-as, Receive-as

Full Access grants the service account permission to access the user account.   Full access allows you to read from and write to folders in both enabled and disabled accounts.   If you are just cancelling meetings from the conference room, full access is sufficient.  If you want to send mail on behalf of a disabled user/room, you will also have to grant send-as receive-as (see the next section)

To grant your service account (called “exsu”,)  full access for a room (“crar210q”), use the add-mailboxpermission cmdlet.

Add-MailboxPermission     -Identity: crar210q     -User: exsu@cod.sumatra.local     -AccessRights: FullAccess     -InheritanceType: All

See our blog: (http://calendarservermigration.blogspot.com/2009/10/fullaccess-fails-with-error-specified.html)

Note that group policies sometimes prevent permissions from being inherited.  Please use Active Directory Users and Computers (ADUC) to ensure the permissions were set!  Find the account (crar210q) and right-hand click to obtain properties.  Select the security tab, then advanced. (If the security tab is missing, select Advanced Features under View.)   You can check the permissions, or the effective permissions.  You should not see deny checked!

Add Send-as, Receive-as

If you have to add send-as receive-as, here is the commandlet

Add-ADPermission

    “CR 101B”

    -user: exsu

    -AccessRights:  genericall

    -ExtendedRights: "receive as","send as",

                        "ms-exch-epi-may-impersonate","ms-exch-epi-impersonation"

    -InheritanceType: All      

Read access to the Global Catalog

Many enterprises grant access to the global catalog if the user is a member of the domain.  If login is failing, anonymous access is probably disabled (since Windows 2000 DCs).  Make sure you are an authenticated user.

Other Details: Allow log on locally                       

Make sure your service account is allowed to log on locally (as in the Local security policy, or if you have multiple machines, set via Group Management Policy, screen shot below.) Otherwise you will generate a 401 error.

Note that in the example above we have both a specific service account and a Group of Service accounts.  Using groups in this way is an effective means of managing several accounts if you need to segment them for Exchange data insertion.

Debugging Exchange 2010 Permissions Problems

Setting permissions correctly is one of the largest stumbling blocks in the process.    Here is a list of the HTTP errors, and ways to debug (and fix) permissions.

HTTP Response	Most Likely Issue	Solution
401	Service account not allow to “log on locally”	Grant permission to “log on locally” via group or local security policy
	The CAS and Mailbox servers are not members of Windows Authorization Access Group.	Add all computers as members to “Windows Authorization Access Group” in ADU&C.
	BASIC authentication is not enabled for the EWS virtual directory in IIS	Set Basic authentication in IIS; remember to restart IIS
	The "SERVICE ACCOUNT" is not authorized to submit requests to the CAS Server	Create a new-ManagementRoleAssignment, and grant ApplicationImpersonation rights  to the service account.  Also remember to check the service account creds to ensure they password is correct. Paste the "ews url" into a browser. Enter the service account creds, when prompted. Do you see a EWS WSDL page? (Note: this could show up as a 500 error in some instances.)
500	The "test user" does not exist in Exchange or is not mailbox enabled	Verify account exists in the domain, it is enabled, a mailbox user (try to access the account in OWA using the service account credentials).  If the account is disabled, did you grant “fullAccess” to the service account?
	The "SERVICE ACCOUNT" cannot impersonate the "test user"	Verity there is a management_role assignment "ApplicationImpersonation" (Ex10) or ExtendedRights:"ms-Exch-EPI-Impersonation","ms-Exch-EPI-May-Impersonate" (ex07) for the SERVICE ACCOUNT that is applied to the server or the user you are attempting to test.

Start with IIS Basic authentication on the EWSvirtual directory. It’s the easiest to see / fix.

Basic debugging protocol – 401 error

Open a browser window, and try to open you EWS url.  If you typically point to the load balancer, point to one CAS server instead.  Try to open the ews url e.g., http://ex10/ews/exchange.asmx.   You should be prompted for credentials.  Enter the service account credentials.  If the credentials are rejected, your service account may not be allowed to log on locally.  If you can login, try to insert a “test” appointment using suExchange.  If you see a 401, it will be due to basic authentication not set OR the CAS/MBX server(s) are not members of windows authorization access group.

Issue: Service account not allowed to log on locally.

Here’s an easy way to confirm you cannot log on locally.  Go to the cas server you pointed to in the EWS url, and open up the Security event log.  Search for event ID 4625, keyword Audit Failure.  You’ll know you have to grant log on locally if you see your service account, with failure information “the user has not been grated the requested logon type at this machine”.   If so, allow the service account to log on locally via a group policy or local security policy.

Issue: Basic Authentication not set

Look in the IIS logs.  If you see a 401 error, check IIS.  If basic authentication is disabled, enable it. Remember to cycle IIS:  “iisreset /noforce.”

Issue: Computers are not members of Windows Authorization Access Group

If you are still getting a 401 error, ensure that ALL exchange computers and domain controllers are members of windows authorization access group.

Basic debugging protocol – 500 error

Issue: Service account does not have impersonation permissions or full access

If you are still getting a 401 error, try logging into an active end user’s mailbox via OWA (like your own!) using the service account credentials.  If you see an error in OWA:

Check the Application event logs on the CAS server for Event ID 17. If you do, then create a “New-ManagementRoleAssignment” to grant the service account ApplicationImpersonation permissions (see “Impersonation,” above.)

Throttling in Hosted Exchange

Microsoft does not close a window without slamming a door and bricking you in.

One of our least favorite topics again comes to the fore: Throttling in hosted Exchange.

Redmond claims these changes are customer-driven, which may or may not be so, but from the standpoint of customers who want to migrate a lot of calendar data into Hosted Exchange, these modifications are like getting a grenade tossed in your lap.

In an on-premises migration it's straightforward to turn these limits off.  Going into Hosted you're at the mercy of Microsoft (good luck with that) or the cleverness of your developer (yes, we can handle this).


If like me you find an hour long video of Exchange Web Services Affinity and Throttling as and act of stultification on the order of Hypnotoad, you can read most of the gist of this at More throttling changes for Exchange Online.


Our recommendations going forward for Hosted Migrations:   


During validation, point to different CAS servers to reduce CAS-server throttling (our code is going to need to change to do this)


During an insertion, use MULTIPLE service accounts which means using parallel insertion processes and point these to different CAS servers.  We're set up for this already, but we now recommend it in smaller migrations than we used to.


During migration, set the batch input to at least 50 calendar objects.


Props to Andrew at UC Irvine who called this to our attention by stepping on the landmine in test last week.

2

Microsoft Access 2010 x64 database access

As more Sumatra customers stand-up x64 systems, they hit the "Unrecognized database format" error.  The issue:  There isn't an x64 MS Access driver.  But there is a solution:   

Download the 2007 Office System Driver: Data Connectivity Components

Choose Microsoft Office 12.0 Access Database Engine OLE DB Provider!!

If you are editing the suExchange _config.xml file, you'll find those settings in the DatabaseProvider tag.  The choices are:

x64: Provider=Microsoft.ACE.OLEDB.12.0;Data Source=
or x32: PROVIDER=Microsoft.Jet.OLEDB.4.0;Data Source=

Test insertion: 401 or 500 error resolution protocol

Suppose you're testing your permissions and get either a ‘500’ or a ‘401’ error.

Use the following table to sort those issues out.

HTTP Response	Most Likely Issue	Solution
500	The "test user" does not exist in Exchange  or  is not mailbox enabled	Verify account exists in the domain, it is enabled, a mailbox user  (try to access the account in OWA using the service account credentials)
	The "SERVICE ACCOUNT" cannot impersonate the "test user"	Verity  there is a management_role assignment "ApplicationImpersonation" (Ex10) or ExtendedRights:"ms-Exch-EPI-Impersonation","ms-Exch-EPI-May-Impersonate" (ex07) for the SERVICE ACCOUNT that is applied to the server or the user you are attempting to test.
401	BASIC authentication is not enabled for the EWS virtual directory in IIS	Set Basic authentication in IIS;  remember to restart IIS
	The "SERVICE ACCOUNT" is not authorized to submit requests to the CAS Server	Is this the account you granted impersonate rights to? Are the creds correct?  Paste the "ews url" into a browser.  Enter the service account creds, when prompted.  Do you see an EWS WSDL page?

Start with IIS Basic authentication on the EWSvirtual directory.  It’s the easiest to see / fix.

Hotmail to Windows Live @ Edu in Japan

We don't migrate email because so many other folks do and it's not that technically challenging or difficult.

Except in some instances.

You should be aware of the following  case out of Japan with a Hotmail to Live@edu migration.

How Microsoft *Almost* Screwed Up a Japanese University.

My bet is that a little bit of the kind of planning, preparation, and testing that we're rabid advocates of could have avoided this.  

Shared Mailboxes -- Warning Will Robinson!

One of the new features in Exchange 2010 Sp1 contained a client side feature that enabled auto mapping of shared mailboxes to user’s Outlook 2010 profiles.)  This should be a major win for the shared mailboxes (e.g, IT Vacation Calendar, Helpdesk Coverage, etc.....)  AutoMapping allows the "delegate" to simply open Outlook and *poof*  there is the shared calendar.  No more navigating the GAL!  No more support headaches (at least in this area.) 

In fact, Steve Goodman has written a step-by-step approach in his blog post Auto-Mapping shared mailboxes in Exchange 2010 Sp1 with Outlook 2010 and Outlook 2007. 

There is a caveat, though.  What happens if you have users who do not want the mailboxes they "share"  to show up in their Outlook? I hit this issue, and have blogged about how to remove automapping.

If you automatically populating shared mailboxes in end user's calendrs is something you want, consider the implications from of a shared calendar showing up in an end user's calendar -- and leaving the end user no facility to remove those mailboxes. 

Remove an "AutoMapped" Mailbox from Outlook 2010

My Outlook takes forever to open.  I've finally figured out why:  I have added too many shared mailboxes that Outlook has to open.  Only I don't remember adding them.  Time to to remove them.  I right-hand click on the shared calendar and "remove calendar."  Outlook throws this error message: ("The group of folders is associated with an e-mail account.  To remove the account, click the File Tab, and then the Info tab, click Account Settings.  Select the e-mail account and then click Remove.)

If the solution were only that easy.  This is a shared mailbox that I have FULLACCESS to;  there is NO EMAIL account in my profile. 

The contributor: (or "feature") is "automapping!! (Here is how to set automapping.)  The solution, for EXCHANGE 2010 Sp2, is to DISABLE AUTOMAPPING!  Here are the powershell commands taken from a Micosoft Technet forum:

Disable automapping between "shared mailbox" and "delegate":
Add-MailboxPermission "Shared Mailbox" -User -AccessRights FullAccess -AutoMapping:$false
  
If you want to set it on all the mailboxes:
$mailbox = Get-Mailbox
$test = $mailbox |
     Foreach {Add-mailboxPermission $_.Name –User ‘administrator’ –AccessRight FullAccess –Automapping $false}

That's it!
Russ

When your ICS export has TWO SMTP addresses for the same user

So from the annals of "what we see in the real world is always more interesting than what we make up ourselves...."


We migrated a client from Zimbra to Exchange and found a user with TWO separate SMTP email addresses.  Call them Krueger@company.com and Freddy.Krueger@company.com.  How did this happen?  The company changed their email address policy six months ago; the new policy makes all primary SMTP addresses to be first.last.

The Sumatra tool uses ICS export file name to identify the user who owns this calendar information.  Thus, the user Kruger (KRUEGER.CALENDAR.ICS) would miss all meetings or appointments he owns under FREDDY.KRUEGER@company.com."

You with me so far?

The solution IS fairly simple; Customers in field report it works beautifully.

1. Insert KRUEGER.CALENDAR.ICS 
2. Rename the primary SMTP on Exchange to FREDDY.KRUEGER@company.com
3. Copy KRUEGER.CALENDAR.ICS to FREDDY.KRUEGER.CALENDAR.ICS
4. Insert FREDDY.KRUEGER.CALENDAR.ICS
5. Rename his primary SMTP if you wish.

The other approach, for the brave, is to replace all of the mailto:krueger@company.com with  mailto:freddy.krueger@company.com.  Be mindful you don't find and replace the wrong SMTP address.

Is there still enough GroupWise out there to make a calendar migration tool worthwhile?

We've been having such success with how easily the "faster, simpler" method works with Zimbra and Oracle that we're wondering if there's any demand for any other calendar migrations that have been under-served.  The one that comes to our minds is GroupWise.

Anybody want an easy way to migrate their GroupWise calendars to Exchange server-side?

Let us know.

Zimbra to Exchange New Functionality

Pretty soon we're going to have migrated more people from Exchange to Zimbra (in a few months) than we migrated from Meeting Maker to Zimbra (in a few years).  Cool!!!!


We just added some functionality at the request of a site.


The subject is reading the contacts in the "Suggested Contacts" folder in Zimbra and migrating those to Exchange.


Now it's possible.


The newest zCalReader looks for files named:

USER.contacts.vcf 

and 
USER.econtacts.vcf 


and loads them both into “Contacts” in Exchange.

Why cannot we load econtacts.vcf into the Outlook/Exchange "Suggested Contacts" folder? Because Exchange Web Services does not currently give us access to that folder. 


We actually found in a migration this weekend a 24 Mb exported Suggested Contacts list containing over three hundred thousand contacts.  That caused zCalReader to bomb (we just weren't scoped for that many contacts).

Symantec Endpoint Protection running on Exchange Servers: Fool me twice....shame on me

Note to self:  Disable all instances of Symantec Endpoint Protection (SEP) on ALL Exchange servers before and during a calendar migration.

Why?

* Insertion throughput dropped 99%: from 750 transactions/minute to four (4) transactions/minute.
* SEP stripped all of the Sumatra strings we associate with calendar items to process messages.

This product bit us two years ago.  It just bit us again.

It's your choice to deploy SEP in your Exchange environment (or not.)   But do not deploy SEP during a Sumatra calendar migration.

Exchange Transaction logs - capacity planning

Sumatra's enterprise customers ask me how much space they should allocate for transaction logs.  I came across Ross Smith's Technet post.  Ross suggests a formula to compute the transaction logfile growth:

For ever 100 messages, Exchange generates 20 transation logs.
According to another Technet article, "Understanding Mailbox Database and Log Capacity,"  each transaction log is 2MB.

All calendaring transactions are messages.  Thus, to estimate the logfile size (in MB), sum the total number of transactions (generated through a macro in the Sumatra DB, M_ShowCounts) and multiply by (2/100).

Changing Google Calendar Notifications

The New York Times has a good post today about changing your Google Calendar Notifications.

Migrating Group Calendar from Oracle Calendar to Exchange

An interesting case came up the past few days as a site was migrating from Oracle Calendar Server to Live@ Edu. 

They maintained a calendar in Oracle for users to post when they were In or Out and wanted to take this to hosted Exchange as a Shared Calendar.

Since the Designate model in OCS is very different from the Delegate model in Exchange, when you're converting the OCS export files, choose this option for the raw data from those calendars and then run an insertion.  You'll get the results you desire.

Why this option? When events were added to the calendar by Designates in OCS they were OWNED by the Designates, and not the actual calendar.  The above option normalizes that for an Exchange environment by making the events owned by the shared calendar.

Add Holidays to Resources in Exchange 2010

Okay, okay.  I know we wrote that the SuHoliday cmdlet would not add holidays to resources.

BUT -- if you use FULL ACCESS instead of Impersonate, the current download-able version works like a champ for this.

To try this out:

·        Provision a service account (say, "exsu") that is NOT an Enterprise Admin

o   Set impersonate RBAC for that account

§  new-ManagementRoleAssignment -Name:_suImp8Exsu -Role:ApplicationImpersonation -User:'exsu@cod.sumatra.local'

o   Try to do an insert for a user AND

o  For resources "cr101b" or "room 222"

Results should not be good, but NOW....

·         Add fullaccess to that service account

o   Get-Mailbox -filter {isResource -eq $true}  -resultsize unlimited | add-mailboxpermission -user exsu -accessrights fullaccess   -InheritanceType: All

o   Add holidays for that room

Holiday Insertions in Live @ Edu

Kudos to Rachel in Georgia for her holiday insertion into Live @ Edu.

She's used SuHoliday on over 8000 users.  Thus far we think this is a record but we usually only hear from people when there are problems.

Also based on her experience and feedback we've added capabilities to our most recent version of the cmdlet, including:

• Not setting reminders for holidays
• Better logging
• Default time zone handling
• Memory handling improvements
• Improved resilience when Exchange throttles your insertion

We're discussing when and how to roll this out to the main cmdlet download.

Zimbra to Exchange Migrations Field-Proven

Just got the word from Portland, Oregon that our faster, simpler method for Zimbra to Exchange calendar migrations successfully migrated 800 Zimbra users' calendar, tasks, and contacts into Office 365 this past weekend.
We're considering this a success.
Yippee!

Sumatra Undo in Action

Ever since high school film strips we've been convinced that visual aids are the most effective.

And ever since college we've been convinced that designing a software system without a back out strategy is just darned irresponsible.

So we combined both of these lessons in our video of Sumatra's UNDO capability in action.

Zimbra to Exchange Calendar Migration - The Video

We've had a good response from one of our test sites on the faster, simpler Zimbra to Exchange calendar migration method so we did a video of the process to show you in real time how it all works.

Enjoy.

And if you want to see the full screen version, you can watch that off our website.

Faster, easier Zimbra ICS to Exchange migration

UPDATE: June 2015.  We've modified the faster-simpler ICS method so that it's faster-simpler AND full state (works like aces for Oracle Calendar Server! and in this case Zimbra gives us the recurrences already so we're one step ahead.)  We can do it for Zimbra if there is sufficient enterprise-level interest.

One of the developing trends we've seen is sites moving out of Zimbra and into Exchange.

Imapsync is excellent for moving the email, but the calendars, tasks, and contacts are another issue.

While we've have a full-state migration solution for over a year, a lot of the folks who want to move want something simple, but still server-side, and oh, could we PLEASE keep meetings live?

It's a tall order, but as usual we deliver.

This is the basic screen:

In short what you do is export your Zimbra calendars, tasks, and contacts server-side (using either zmmailbox, cURL, or whatever you're comfortable with) in ICS and VCF format, then run this application against the resulting files.

With the proper credentials on your Exchange server (and yes, we have tested it against Office 365 and Live @ Edu), these will upload.  Meetings will be re-proposed (so your users will see invitations in their inboxes come the Monday morning post-migration).  The alternative is the full-blown Sumatra process which is overkill for many of you moving out of Zimbra.

Rather than our three step process (export, map user IDs in a database, and insert into Exchange), this is a TWO step process (export, import) with no intermediate mapping.  So if you're changing a lot of user IDs going from Zimbra to Exchange, this is probably NOT the recommended process for you.

We're in test right now with this at a couple of sites and will let you know when it's ready for prime time.  We'll give you the full documentation set and a video then too.

Get Time Zones for your users via PowerShell

We recently released a cmdlet that bulk-inserts holidays in Exchange 2010 (see the Sumatra website or the Sumatra Blog).

A holiday is a simple all day calendaring event in Outlook. Yet simple calendaring events can be tricky! Consider: when your end users work in different time zones! If you force an all day event into one time zones, all users who are not in that time zone will see their holidays span multiple days. Not a happy scenario. What's the solution?

We wrote a script that uses Exchange 2010 "get-mailboxRegionalConfiguration" cmdlet to find the timezones. If used in conjunction with get-mailbox, you can output a file that has the user information plus the timezone. Problem solved!

This script produces a file that outputs PrimarySMTPAddress + TimeZone:

#Define your 'default' timezone (if none is set)
$myDefaultTimezone="Eastern Standard Time"

#Define the output file
$myOutputFile="userlist.txt"

#Define the list of User Accounts to process
$myMailboxList = get-mailbox -Filter {RecipientTypeDetails -eq "UserMailbox"}  select-object Identity,PrimarySMTPAddress

#If file exists, delete the file
$fileExists=test-Path $myOutputFile
if ($fileExists -eq "True"){del $myOutputFile}


#Loop through list and get
foreach ($t in $myMailboxList) {
    $priSMTP=$t.PrimarySMTPAddress
    $xi=get-mailboxRegionalConfiguration -Identity $t.Identity
    if ($xi.TimeZone -eq $Null) {$tt=$myDefaultTimezone} Else {$tt=$xi.TimeZone}
    write-output "$priSMTP $tt" >> $myOutputFile
}

write-output "Done!  see the file $myOutputFile"

You can also download getUserTimezones.zip

If you have another way, please share!

-Russ

Holiday cmdlet for Exchange 2010

We've been inserting holidays server-side on Exchange for a while and after some feedback last year we started re-thinking how to accomplish this.

These are the files in the downloadable ZIP

We've written a PowerShell cmdlet we call suHoliday that inserts holidays server-side.  We've run it through our internal testing against on-premises Exchange 2010, Live @ Edu, and Office 365, so now we think it's time to let it out into the wild and see what you folks can do with it.

The sample CSV file for US holidays

 Your inserted holidays will look like this in a user's calendar:

If you want to download it and run it through its paces in your test lab, you may do so here.  We just ask you to answer a few questions so we can keep track of where it's going.  If it works for you consider making a donation of US$20 per instance.  It'll make it easier for us to consider updates and new features for next year.

What's it do now?

Insert server-specific or user-specific holidays through 2012 with NO user intervention.

Customize for different state or national holidays.

Define Free/Busy status.

Script adding holidays at user provisioning time (e.g., by piping in from get-mailbox).

Support for multiple time zones.

Define All-Day Events or appointments at specific times.

Support international holidays / date formats. For example: 2012 UK Bank Holidays

Try it out and tell your friends.


Limitations (or, what do you want for free / ultra low cost?)

• This only inserts holidays for the year 2012 (well, we give you a few weeks into 2013).
• All inserted events have "Inserted courtesy of the Exchange Calendaring experts: Sumatra Development" in the agenda. (yep, even if you license it)
• We support via electronic means, so keep an eye on our blog.

Advance Notice: Sumatra in SF Sept 30, Oct 1

Sumatra (well, Zyg) is going to be in San Francisco on September 30 and October 1,  Mainly he's there to go to the Opera, but since that always happens at night and his engaging wife will be working at Wells-Fargo, if any of our calendar-oriented acquaintances in the Bay Area are looking for some guidance, drop a line.

Propagating Changed SIP URIs to Existing Meetings

One of our favorite sites contacted us about a problem they're anticipating. Knee-deep in solving said problem we at Sumatra wonder if anyone else has the same issue.

They're changing a number of SMTP addresses and want to change the associated Lync SIPs to match.  Here's Microsoft's guidance on how to do that for Office 365.  And a different take on scripting a solution.  See Impact of Changing a User's SIP Address for a full discussion.

Changing the SIPs is not the problem, but the number of existing meetings with the old SIPs that are then left in your users' calendars IS a problem that requires updating.

So in this example of an existing calendar item, let's say riuliano became russ_iuliano, to keep end users from going bug-house you'd want to modify all the LiveMeeting URLs in existing calendar objects server-side and update them.

And this is the cue for Sumatra's ability to manipulate Exchange calendar data.

Does anyone else out there have the same problem?

More Weirdness in an Over-Loaded Google Calendar

While experimenting to see if I could delete calendar items from a calendar and thereby finally clear an over-loaded test account I got this message:

 "Oops"?  Cute.  Too bad I do not know any serious corporations that ask for cute.

The goal was to see if deleting items would get me below a threshold, or if the threshold was irreversible.

The weird thing is that once I got that message, previously-deleted objects began re-populating the calendar.  Calendar zombies had risen from the grave!

Clearly there's a cache of the deleted items.  I have no information on when, how often, or if it gets cleared in a single session.

There is also some interesting behavior with old items.  To see this, load 10-15 years worth of calendar data and then travel back to some month in the year 2000.  The following unobtrusive message will display while the data renders (and it seems to take a while):

So there's either some background mechanism shunting historical data into slower, longer-term storage, or the system is not really optimized for rendering arbitrary stretches of time.

Added on September 6, 2011:
A variation of the above: cannot load your data -- come back when it's more convenient for Google....

Unable to Delete All Data in a Google Calendar

So let's say you're testing your calendar insertion (a prudent step which we not only endorse but require).  You either have a large calendar or run several insertions to put in over 40,000 items.  I suspect the problem starts with 32K objects, just because we are so suspiciously close to a magic binary number.
You go into your settings with the hope of deleting all data from your test account (and please make sure it is a test account):

You select Delete giving you this dialog box:

Where you promise you REALLY REALLY DO want to Delete all events, and click the Delete all events button.

Less than 60 seconds later you see the following dialog box and all your data is still in the calendar.

We do not seem to be alone.  We posted this on Google Calendar's Help Forum (whose only value has been confirmation from another user with the same problem). None of our current clients are going to hit this limit unless they do multiple insertions without practicing calendar hygiene.  

So practice calendar hygiene.

But we do want to get this out as a warning to everyone.

Addition on March 3, 2015: This post is now hugely popular.  So if there are folk out there who want to take an entire Exchange server of calendar data (we're talking an enterprise here) into Google calendar, and keep all the meetings live, let us know.  We've done Exchange to Exchange that way but will only add the Exchange to Google capability if we have a real customer.

Shout out to the CalConnect Folks!

A shout out of thanks to the CalConnect folks for putting us on their Blogroll.

Thanks!

Preliminary Google Calendar Upload Speed Tests

We've made no secret about the molasses-in-Antarctica upload speed of EWS in Office365 and Live@ Edu. 

So for an interesting comparison -- how long does it take to upload calendar data into Google Calendar?

In a test derived from real world data set (legacy Meeting Maker being the organ donor in this case)  our current technology uploaded 13,651 appointments to Google Calendar in 9:12 (i.e., 9 minutes and 12 seconds).  Just in case I happened to hit a low time for network traffic I ran the test again with additional loads on my PC and got all data inserted in 15:54 (call it 16 minutes).

Using the more conservative figure, this represents an average upload rate of about 853 calendar objects per minute into Google Calendar!  The faster result gives 1480 objects per minute, but I doubt that is sustainable over the duration of a migration.

For those of you who do not understand why we get excited by these numbers: this is similar to the peak throughput we see inserting into an on-premises Exchange installation (you always hear us use the figure 850 calendar objects per minute for estimation purposes).  Our timings on hosted Exchange come in at about 120 calendar objects per minute.

So Google Calendar is about ten times faster than an upload into Hosted Exchange!

Let me repeat that -- our early testing indicates that calendar uploads into Google Calendar execute an order of magnitude faster than an upload into hosted Exchange.

Now, let me point our a few things to beware of: these numbers may vary as our code evolves, but are in accord with the field experience of one of our test sites (which motivated this timing).

I have no idea what Google does right that Microsoft does not, given that both companies are in total control of their data centers, server code, and APIs.  I do know that for purposes of migration speed Google Calendar kicks Exchange calendar's buttocks all the way to the curb and then slam dances its corpse into pavement pizza.

Since regular readers will know that your author does not believe in letting ANY of the guilty off without some sentence, I want to point out that as mediocre as Microsoft EWS documentation and support has been, it is light years ahead of Google's documentation of their calendar APIs which our team has taken to completely ignoring because it's led us down too many bad paths already. 

And after inserting 40,000+ objects into Google Calendar, the Delete under Settings does not seem to work:

iCal Sharing in Exchange 2010 Sp1

Sumatra is about to release a solution to migrate legacy calendar data to Google. A customer asked how his end users could read shared calendars from folks outside the organization (and who use Exchange for calendaring.)


We passed along this article, in which Steve Goodman wrote a superb post describing how Exchange 2010 Sp1 allows users to share calendars with non-Exchange users (e.g., Google, Zimbra) using public or encoded URLs. (And users can be alloweed to do this via OWA!)

And remember, this is calendar SHARING, not cross-server calendar synchronization.

Update Rollup 4 for Exchange 2010 - re-issued

I was working offiste for the last two weeked and missed the story: Microsoft discovered a problem with Outlook 2010's interaction with Exchange 2010 server, and had to retract Update Rollup 4 for Exchange 2010 they released on June 22,2011. KB251545 described the problem: moving or copying public folders didn't work "as expected."

The release was announced on Technet. Here is the new link to download the rollup

BTW, Update Rollup 5 for Exchange Server 2010 Service Pack 1 release remains "on-schedule" for release in August 2011.

First Field Migration into Zimbra 7 Accomplished

We ran our first field migration into Zimbra 7 this weekend (out of a Meeting Maker legacy system).

We had early reports of problems with recurring meetings in Z7 migrations, but tested well in advance to make sure that did not happen.  All is well.

Prevent ghost delegates: find (and remove) delegates before terminating user

In a previous post, I outlined a process to find and remove Ghost Delegates from Exchange. (To recap: you turn a delegate a ghost delegate by deleting the account from AD without removing the delegate permissions in Outlook.)

In this post, I'll outline a process that avoids the ghost delegate problem (by finding and removing delegates BEFORE deleting the terminated user's account.) There are two steps: search AD, and then remove the delegates.

The hard part was finding all users that granted delegate rights to the "soon-to-be-deleted-account." I dredged up two "oldies but goodies:" LDIFDE and CSVDE to do a reverse-lookup for a terminated user using the "public delegate" fields:

• PublicDelegates "What mailbox(es) did I give delegate rights to"
• PublicDelegatesBL: "What mailbox(es) am I a delegate of"

Here are both commands:

LDIFDE.EXE -F delegateLDIFDE.TXT -D "OU=TestUsers,DC=myDC,DC=mydomain,DC=com" -L "name,mail,PublicDelegatesBL" -R "(&mail=termuser@mydomain.com)(PublicDelegatesBL=*))"


csvde -f delegateCSVDE.csv -s myDC -l "name,mail,PublicDelegatesBL" -r "(&mail=termuser@mydomain.com)(PublicDelegatesBL=*))"

Note: the LDIFDE command limits the scope of its work to the "OU=TestUsers;" csvde seaches the entire enterprise directory.

Here is an output from LDIFDE, in which we found all of PublicDelegatesBL of "Andre Admin" (it's just Big Boss):

Next, use Glen Scale's powershell code to confirm the terminated user is a delegate, and then remove that user from all accounts he was granted delegate rights. (The following commands were described in a prior post.) Step 7 is where the terminated user is removed as a delegate. I have shown how to display delegates and forwarding rules, just as an FYI.

1. set-ExecutionPolicy RemoteSigned -force
2. import-Module ./Messageops-Exchange.psd1
3. $myCred=Get-Credential -Credential myservice@mydomain.com
4. $newprofile=new-messageops.ewsprofile -identity:myservice@mydomain.com -exchangeversion:exchange2010_sp1 -casURL:https://mycas.mydomain.com -Credential:$myCred
5. Get-MessageOps.MailboxDelegateReport-p:$newprofile -id:bigboss@mydomain.com
6. Get-MessageOps.MailboxDelegateForwardingRules -p:$newprofile -id:bigboss@mydomain.com
7. Remove-MessageOps.MailboxDelegate -p:$newprofile -id:bigboss@mydomain.com -DelegateAddress:termuser@mydomain.com
   

Notes: the "terminated" user is termuser@mydomain.com; the account that granted delegate rights was bigboss@mydomain.com.

I've pulled the URLs and commands into a text file that you can download to simplify typing....

--Russ



p.s.: I tried Exchange Management Shell, using a new Exchange 2010 commandlet: "Get-MailboxFolderPermission." But it was difficult to do a reverse-lookup (i.e., find all users who grated the "terminated user" permissions: I had to type that user's display name correctly or got no hits.... Plus, this commandlet isn't available for Exchange 2007 customers.


Here is how to get Big Boss' permissions:
Get-MailboxFolderPermission -Identity bigboss:\Calendar

FYI, you can set accessrights, particularly editor rights, between "big boss" and "her admin," that you can't do if your end users access their Exchange calendars via OWA (only):

Set-MailboxFolderPermission -Id:bigboss:\Calendar -User:herAdmin -AccessRights:Editor

The Ghost Delegate Exorcist

A Sumatra client called for help - a flood of NDRs were driving their Exchange server to its knees. The problem: a boss granted his admin delegate rights his calendar. Along with that, all calendar-messages were forwarded to to the admin. After the admin left the company, the admin's mailbox was deleted.

All was fine until someone invited the boss to a meeting. The boss received an NDR after the delegate forwarding rule sent that calendar message to the [now deleted] delegate. That NDR calendar message was sent back to the boss which got forwarded to the [still deleted] delegate, which generated another NDR......)

No big deal. They went into the boss' Outlook delegate and tried to remove the delegate, but couldn't -- Outlook reported the delegate was "(not found):"

The delegates are there but you can't delete them. Ghost delegates.....
The root cause: they didn't remove the delegate permissions and forwarding rules before they deleted the admin assistant's mailbox.

In this post, I'll talk about how to rid Exchange of those ghost delegates. In the next post, I'll talk about how to find and remove delegate permissions BEFORE you delete the terminated user's account.

For the curious, Glen Scales blogged about "How to deal with invalid delegates," and "Displaying delegate forward rules." Even better, Glen wrote some powershell code to exorcise our ghost delegate!

Here is what we did to remove the ghost delegates

1. Download and install the Exchange Web Services Managed API
2. Create a sub directory, and copy the Microsoft.Exchange.WebServices dll into the directory; Run Exchange Powershell (as administrator), and change to that sub directory.
3. Set the execution policy, import the module, and create a credential variable (using your service account):
set-ExecutionPolicy RemoteSigned -force import-Module ./Messageops-Exchange.psd1 $myCred=Get-Credential
4. Next, define an exchange profile (as above, we provide our service account with impersonation or full access permissions. See our blog post on impersonation in Exchange 2007 or Exchange 2010), and point to our CAS server directly
   
   
   $newprofile = new-messageops.ewsprofile -identity:mySVCacct@mydomain.com-exchangeversion:exchange2010_sp1 -casURL:https://mycas.mydomain.com -Credential:$myCred
   
   
5. See the list of valid and invalid (ghost) delegates and rules:
   
   
   Get-MessageOps.MailboxDelegateReport -p:$newprofile -id:bigboss@mydomain.com
   
   Get-MessageOps.MailboxDelegatesInvalid -p:$newprofile -id:bigboss@mydomain.com
   
   Get-MessageOps.MailboxDelegateInvalidForwardingRules -p:$newprofile -id:bigboss@mydomain.com
6. Finally, remove invalid forwarding rules, and THEN remove the delegates (in that order!)
   
   
   Remove-MessageOps.MailboxDelegateInvalidForwardingRules.unsupported -p:$newprofile -id:bigboss@mydomain.com
   
   Remove-MessageOps.MailboxDelegatesInvalid.UnSupported -p:$newprofile -id:bigboss@mydomain.com

I've pulled the URLs and commands into a text file that you can download to simplify typing....

-Russ

Migrating Zimbra Calendar to Hosted Microsoft Exchange

We've been getting a lot of inquiries lately from people wanting to migrate calendars out of Zimbra and into Exchange.

A few quick guidelines:

• If you have more than 250 users it is cost-effective to use our full-scale migration technology.  It's completely server-to-server, leaves end-users out of the loop, and has results as though you've been using Exchange calendaring all along.
• For a few hundred users or under, try our Decaf option.  Application here.  Documentation here.  It's a server-side ICS import to hosted Exchange, and right now it's mainly geared towards Oracle Calendar migrations, but we'll open it up to Zimbra users if we see enough demand.

Exchange 2010 to Google Calendar Server-Side Calendar Migrations

You read that right.

Now that we finally have OAuth for Google working (one of our engineers described Google's documentation as having "duplications, contradictions and gaps in between."  I describe it as a "mess") we're inserting calendar data server-side to Google with full-fidelity and no end-user intervention.

Yippee!

We've read the handwriting on the wall and see that there are sites out there looking to bring their calendar data from Microsoft Exchange over to Google and are not thrilled about having to use PSTs to do it (we don't blame you).

If you are contemplating such -- drop us a line.  We want some feedback on how you'd like it implemented.

Our Office 365 Experience Thus Far...

So we've been pretty good letting our readers know that migration performance into both Live @ Edu and Office 365 is about 14% the performance of on-premises Exchange.  And then there's the problems with quotas.

That none of this has changed or been addressed through the beta period of O365 should not be surprising to anybody not under the influence of Prozac or electroshock. 

In fact, despite bringing this to Redmond's attention, we've gotten zero feedback, seen zero progress, and expect zero results.

Now that Office 365 is out of beta and in the production, plan on all these nagging "problems" getting re-classified as "features."

Oracle Beehive to Exchange Migration

Zyg was just back from Charlotte, NC and Russ was soon to head out for a while, when we got an inquiry from one of the few third parties we trust to independently run a migration (Kuttig in Germany -- these guys are sharp and efficient, a pleasure to work with).

Turns out that at least one Oracle Beehive site (see: Is Oracle Beehive DBA  (Dead Before Arrival)?)  wants to migrate to Exchange. 

A quick look at the Oracle Beehive Administrator's Reference Guide and our technology convinces me it should be not too difficult to do at the level we do it at: full-state, server-side to server-side.

Summary changes in our usual process:

Use xCalReader's ICS option.

You'll need a USERS.TXT and RESOURCES.TXT, which should come out of the Beehive utilities list_users and list_resources (be careful of format).

And you will need to export the calendars using the export_icalendar function.

When a user has 15,000+ unread items in the inbox....

We *thought* huge mailboxes (as measured by disk space) could wreak havoc during a migration. Perhaps we are wrong. This is a story about huge inboxes and unread email (the harbinger of problems that plague email systems.)

Last week we moved a client into Exchange 2010 (on-premesis). BUT the Sumatra Exchange insertion tool hung while inserting one user. For hours. Restarting the tool and rebooting boxes didn't help. After much digging we found the problem: that end user had 15,000+ unread emails in her inbox. What was the issue? (no, not why would we migrate a user with 15,000 unread emails in their inbox who probably left the company three years ago....)

The client hit an Exchange threshold. The issue is described in a Microsoft Technet article: Understanding the Performance Impact of High Item Counts and Restricted Views.

The article says: ...In Exchange 2007 ... an acceptable user experience can still be maintained with item counts as high as 20,000 items. We're still researching if that limit changed for Exchange 2010. Our client hit that wall in Exchange 2010.

So who are these big-bad-inbox-boys and girls? Here's a powershell command: (note, I've set a conservative threshold of 10,000 items for just user mailboxes.)

Get-Mailbox -filter {RecipientType -eq "UserMailbox"} -ResultSize unlimited Get-MailboxFolderStatistics where {($_.name -eq "Inbox" -or $_.Name -eq "Calendar") -AND $_.itemsinfolder -ge 10000} sort-object ItemsinFolder -Descending ft Identity,ItemsinFolder,FolderSize

-Russ

Follow-on to the Headaches of Cloud Migration

As a follow-on to our earlier postings, you might want to check out

http://ferris.com/2011/06/03/moving-to-hosted-exchange-plan-for-hiccups/

There is never a free ride when you move an entire server.

International Characters and an OCS to Google Calendar Migration

Short story on International characters migrating into Google Calendar from Oracle Calendar: it works.

Oracle Calendar to Google Calendar - Server-Side Full-State Migration

Let's say you're using Oracle Calendar and, not happy with your current mega-corp, you opt for another one that is at least producing better graphics and walks the walk about this cloud thing.

Our old friend Jimi Hendrix's calendar in OCS

can now be migrated with full state information into this in Google Calendar:

Yep.  You read me right:  Full state.
Meetings are meetings with guest lists and status.
This will also work for Zimbra calendaring to Google and Meeting Maker to Google.
We are still working on a few parts of this, but it is now basically completely functional and in two test sites.
One other thing.  If you wanted to take all your calendar data from something called Microsoft Exchange and migrate it full-state into Google Calendar, that can be arranged.

June 12: In response to queries.  This is all SERVER-SIDE to server-side with no end-user / client-side intervention.