Prevent ghost delegates: find (and remove) delegates before terminating user

In a previous post, I outlined a process to find and remove Ghost Delegates from Exchange. (To recap: you turn a delegate a ghost delegate by deleting the account from AD without removing the delegate permissions in Outlook.)

In this post, I'll outline a process that avoids the ghost delegate problem (by finding and removing delegates BEFORE deleting the terminated user's account.) There are two steps: search AD, and then remove the delegates.

The hard part was finding all users that granted delegate rights to the "soon-to-be-deleted-account." I dredged up two "oldies but goodies:" LDIFDE and CSVDE to do a reverse-lookup for a terminated user using the "public delegate" fields:

• PublicDelegates "What mailbox(es) did I give delegate rights to"
• PublicDelegatesBL: "What mailbox(es) am I a delegate of"

Here are both commands:

LDIFDE.EXE -F delegateLDIFDE.TXT -D "OU=TestUsers,DC=myDC,DC=mydomain,DC=com" -L "name,mail,PublicDelegatesBL" -R "(&mail=termuser@mydomain.com)(PublicDelegatesBL=*))"


csvde -f delegateCSVDE.csv -s myDC -l "name,mail,PublicDelegatesBL" -r "(&mail=termuser@mydomain.com)(PublicDelegatesBL=*))"

Note: the LDIFDE command limits the scope of its work to the "OU=TestUsers;" csvde seaches the entire enterprise directory.

Here is an output from LDIFDE, in which we found all of PublicDelegatesBL of "Andre Admin" (it's just Big Boss):

Next, use Glen Scale's powershell code to confirm the terminated user is a delegate, and then remove that user from all accounts he was granted delegate rights. (The following commands were described in a prior post.) Step 7 is where the terminated user is removed as a delegate. I have shown how to display delegates and forwarding rules, just as an FYI.

1. set-ExecutionPolicy RemoteSigned -force
2. import-Module ./Messageops-Exchange.psd1
3. $myCred=Get-Credential -Credential myservice@mydomain.com
4. $newprofile=new-messageops.ewsprofile -identity:myservice@mydomain.com -exchangeversion:exchange2010_sp1 -casURL:https://mycas.mydomain.com -Credential:$myCred
5. Get-MessageOps.MailboxDelegateReport-p:$newprofile -id:bigboss@mydomain.com
6. Get-MessageOps.MailboxDelegateForwardingRules -p:$newprofile -id:bigboss@mydomain.com
7. Remove-MessageOps.MailboxDelegate -p:$newprofile -id:bigboss@mydomain.com -DelegateAddress:termuser@mydomain.com
   

Notes: the "terminated" user is termuser@mydomain.com; the account that granted delegate rights was bigboss@mydomain.com.

I've pulled the URLs and commands into a text file that you can download to simplify typing....

--Russ



p.s.: I tried Exchange Management Shell, using a new Exchange 2010 commandlet: "Get-MailboxFolderPermission." But it was difficult to do a reverse-lookup (i.e., find all users who grated the "terminated user" permissions: I had to type that user's display name correctly or got no hits.... Plus, this commandlet isn't available for Exchange 2007 customers.


Here is how to get Big Boss' permissions:
Get-MailboxFolderPermission -Identity bigboss:\Calendar

FYI, you can set accessrights, particularly editor rights, between "big boss" and "her admin," that you can't do if your end users access their Exchange calendars via OWA (only):

Set-MailboxFolderPermission -Id:bigboss:\Calendar -User:herAdmin -AccessRights:Editor

BlackBerry Permissions and Exchange Server-Side Holiday insertion

So remember a few days ago when we told you about a great simplification in setting up permissions?

Basically, and you should not be too surprised at this, the permission for BES admin are really really close to the permissions necessary for inserting holidays or running a full-state calendar migration.

Check out this post:

http://forums.crackberry.com/f50/exchange-2007-bes-install-guide-942/

which advises setting permissions as follows:

get-mailboxserver add-exchangeadministrator
BESAdmin –role ViewOnlyAdmin

get-mailboxserver add-adpermission –user
BESAdmin –accessrights ExtendedRight –extendedrights Send-As, Receive-As,
ms-Exch-Store-Admin

And then tells you to follow up with:

Open Active Directory and from the View menu select "Advanced Features". Then
go to each user that will be added to the BES and open their properties, go to
the security tab and add the user BESadmin and add the security permission "Send
As". (This will overcome some MS patches that prevent BES sending emails).
Methods are available to add BESadmin "Send As" rights to all users if required
but this method ensures only the required users have permissions added.

Make BESadmin a local Administrator of the server where you will be
installing the BES software. This is done by right mouse clicking my computer
and selecting “Manage”. From Computer Management expand “Local Users &
Groups” and select Groups. From Groups double click “Administrators” and add
BESadmin.

This is pretty much what we tell you to do in setting up a service account for running a Sumatra migration or inserting holidays, checking for broken meetings, or running the "terminated user" option in our code.

If you do not have a specific service account set up yet, this is a good place to start.

Terminated Users, Broken Meetings, and Exchange 2007

Given the current economic situation we expect that many US corporations will soon be filled with a "right-sized" group of users. This will of course leave rotting corpses throughout the organization and some of them are going to be festering away in your Exchange 2007 store.

We've heard it referred to as the "terminated user" problem and it works like this:

Yorick, a diligent young exec just fresh from his MBA with little experience other than managing his BMW and his credit cards, has been busily setting up meetings with people across all departments. He is now gone but his meetings live on. (Note: We realize this is an idealized scenario. In the real world Yorick would be promoted while people with far more experience but higher salaries and more benefits would be nuked wholesale, but we try not to be bitter.)

So recurring meetings managed by Yorick, which might or might not be relevant anymore, are sitting in calendars across the company.

You the Exchange Administrator would like to get rid of them wholesale, but there's no way of doing this except for logging in as Yorick and manually finding and cancelling all meetings, making sure to send updates. Multiply this hassle by the body count in your reorg.

But it does not need to be like this.

Since as part of our calendar migrations we're creating and deleting meetings ALL THE TIME, we've started experimenting with some tools based off our insertion code that will go through a user's calendar, cancel all meetings they proposed, and automatically send updates to guests.

Problem solved.

We've started with a congruent problem: finding orphan conference rooms from cancelled meetings and built it into our main code via the "Test" button (which those of you in the midst of E2K7 migrations have gotten used to)

To check for orphan conference rooms:

Why are we telling you all of this? Because we're looking for folks who really want the problem solved and are willing to work with some early code to help us work out the best way to invoke and use this in the Exchange environment.

You're going to need to set Permissions for a service account to run this app as you would for a migration -- so be advised of that. But we're more than happy to share this with folks who contact us directly looking to try it out.

Microsoft makes its own calendar UNDO!

 

Finally Microsoft does something to fix the terminated user problem of ghost meetings.

Remove-CalendarEvents -Identity user@domain.com -CancelOrganizedMeetings -Confirm:$false -verbose

Arm yourselves with knowledge!

The Ghost Delegate Exorcist

A Sumatra client called for help - a flood of NDRs were driving their Exchange server to its knees. The problem: a boss granted his admin delegate rights his calendar. Along with that, all calendar-messages were forwarded to to the admin. After the admin left the company, the admin's mailbox was deleted.

All was fine until someone invited the boss to a meeting. The boss received an NDR after the delegate forwarding rule sent that calendar message to the [now deleted] delegate. That NDR calendar message was sent back to the boss which got forwarded to the [still deleted] delegate, which generated another NDR......)

No big deal. They went into the boss' Outlook delegate and tried to remove the delegate, but couldn't -- Outlook reported the delegate was "(not found):"

The delegates are there but you can't delete them. Ghost delegates.....
The root cause: they didn't remove the delegate permissions and forwarding rules before they deleted the admin assistant's mailbox.

In this post, I'll talk about how to rid Exchange of those ghost delegates. In the next post, I'll talk about how to find and remove delegate permissions BEFORE you delete the terminated user's account.

For the curious, Glen Scales blogged about "How to deal with invalid delegates," and "Displaying delegate forward rules." Even better, Glen wrote some powershell code to exorcise our ghost delegate!

Here is what we did to remove the ghost delegates

1. Download and install the Exchange Web Services Managed API
2. Create a sub directory, and copy the Microsoft.Exchange.WebServices dll into the directory; Run Exchange Powershell (as administrator), and change to that sub directory.
3. Set the execution policy, import the module, and create a credential variable (using your service account):
set-ExecutionPolicy RemoteSigned -force import-Module ./Messageops-Exchange.psd1 $myCred=Get-Credential
4. Next, define an exchange profile (as above, we provide our service account with impersonation or full access permissions. See our blog post on impersonation in Exchange 2007 or Exchange 2010), and point to our CAS server directly
   
   
   $newprofile = new-messageops.ewsprofile -identity:mySVCacct@mydomain.com-exchangeversion:exchange2010_sp1 -casURL:https://mycas.mydomain.com -Credential:$myCred
   
   
5. See the list of valid and invalid (ghost) delegates and rules:
   
   
   Get-MessageOps.MailboxDelegateReport -p:$newprofile -id:bigboss@mydomain.com
   
   Get-MessageOps.MailboxDelegatesInvalid -p:$newprofile -id:bigboss@mydomain.com
   
   Get-MessageOps.MailboxDelegateInvalidForwardingRules -p:$newprofile -id:bigboss@mydomain.com
6. Finally, remove invalid forwarding rules, and THEN remove the delegates (in that order!)
   
   
   Remove-MessageOps.MailboxDelegateInvalidForwardingRules.unsupported -p:$newprofile -id:bigboss@mydomain.com
   
   Remove-MessageOps.MailboxDelegatesInvalid.UnSupported -p:$newprofile -id:bigboss@mydomain.com

I've pulled the URLs and commands into a text file that you can download to simplify typing....

-Russ

Holiday insertion server-side in Exchange 2007

Remember the Sumatra Utilities for Exchange 2003 and their beloved server-side holiday insertion capability?

And you remember how every year you ask us if we've done it for Exchange 2007?

Well, we (finally) rebuilt it for Exchange Web Services. Check out this example.

Friends of Sumatra can use this at no charge (you all know who you are) so just ask us and we'll send it out.

For everyone else we're actually going to charge for the capability this time.

Which brings me to the purpose of this posting: If you have any preferences on how we should do this -- drop us a line. If you do not know our emails you can use our contact form.

Oh yeah -- this is also going to include the broken meeting check, the terminated user utility, and the interface for managing conference room statistics (the full application for the last one will be a separate follow-on offering).

Insert Holidays Server-Side into Exchange 2007

Done.

Ship it.

The Sumatra Utilities for Exchange 2007 are now available for download.

Keep in mind, they won't just insert holidays for the 2010 calendar year, they'll also let you:

• Check for broken meetings in your conference rooms
• Gracefully remove terminated user meetings
• Extract resource use data you can then analyze in a spreadsheet (not full ResourceWatch but it gives you easy access to data that was hard to get before)
• And if you want to start using them to develop your own applications (we've got one business school that's done that and another evaluating), we can do that as well.

Broken Meeting Data and Exchange 2007

Let's say you're in Exchange 2007 and still using Outlook 2003 (not that we ourselves do this or have any clients with this kind of environment, but we hear tell it's still done). So you create a meeting and invite a managed resource. Like this:

Let's say that later on your users do things like deleting resources from meetings and not sending updates. Sort of like this:

Is this avoided by using Outlook 2007? Yes. But 1.) This happens now and 2.) If you upgraded to 2007 from 2003 before you mandated Outlook 2007 to correct this you probably still have the results of this activity floating around your calendars.

The result is what we've been calling Broken Meetings (you'll hear us sometimes call them "orphan" or "zombie" meetings) -- cruft that's making your resources harder to manage by taking up space they shouldn't be.

This wouldn't be a big problem if you could

1. Identify them


2. Remove them

So glad you asked what we were doing about it. Since we've gotten really used to creating well-formed calendar data in Exchange we started reversing the process to find data that isn't well-formed.

The result is this early version of code based on our existing insertion tools:

Check out the FindBrokenMtgs and DelBrokenMtgs buttons. I also need to mention that anyone who's fallen into the various Permissions black holes in E2K7 will immediately (and correctly!) intuit that setting this utility up to dig out all this data can be challenging.

Keep in mind we created a Broken meeting in Room 222 above, so let's feed that in and see what we find:

Looking for Broken Meetings we find the one that we know is waiting to be found.

Next step of course is to remove it.

The process obviously gets a lot more complicated when you add recurring meetings and recurring meeting exceptions to the mix (and we've already dealt with that).

Also the process is closely related to the "Terminated User" problem of how to clear out meetings from former employees (and you'll see oblique references to this on some of the buttons above).

We'd really like to hear feedback on how useful capability like this would be and the best way to present it to an Exchange Admin.

Terminated with extreme prejudice

You know the scenario -- people in your organization are coming and going all the time. When they come in you have a handle on it. When they go, well, that's a different story.

In particular, Exchange has a nasty habit of letting them leave the building but linger on in broken meetings.

We've been working on the problem and we got asked about it yesterday, so I figured it was a good time for a running commentary.

Let's create a user called Elvis Morrison.

Elvis enters the company sets up a few meetings and, in the way of all flesh, is downsized or leaves for greener pastures. When he leaves this is what his calendar looks like:

We can delete Elvis from Active Directory in a few seconds:

But if we just delete him then his meetings will linger (with no way to cancel them). So in the Conference Room 222 calendar, Elvis is still very much alive!

He's also still in end user calendars.

Now, if we wanted to cancel all of Elvis's meetings BEFORE we deleted him from Active Directory, that would be easy. Just run our tool and click Test2: Report/CancelMtgs

You get options like

Let's say we want to cancel them.

And let everyone know why.

And it tells us what's going on.

BUT let's say users have been being deleted for a while or we deleted Elvis before removing his meetings. We can STILL clean out the resources (we can do the users, too, but let's focus on the rooms and resources).

Let's say we're looking to clear out the cruft from Conference Room 222.

Put in the Room ID and click
It will find all instances of Elvis's recurring meetings in it (as well as any other broken meetings, but I happen to know the only ones in there just now belong to Elvis)

And it creates a separate report

To get rid of them (you can edit the report to remove the ones you want to keep) click


Going into the calendar for Room 222 means it now looks like this, cleaned of the broken meetings Elvis organized.

The more astute among you will recognize there are some additional subtleties to this, but I'll save those for another blog posting.

Editorial addition (Sept 14, 2008)

The "Export Mailbox" cmdlet will archive all Elvis's data to a PST, but it does NOT cancel his meetings. See this discussion.

New Flag for Zinsert Zimbra Migration

Sometimes I feel a little like Ron Popeil (I have seen him speak in person and he was amazing) when I talk about new features.

So I'll just launch into that mode now.

Has this ever happened to you?

You have 1500+ users in Meeting Maker and you want to migrate them to Zimbra. So you export your Meeting Maker database with 1200 user IDs mapped to your new Zimbra email, figuring you're going to delete the other 300 low-volume / terminated / lost at sea users later on.

But Sumatra Development (those meanies who don't understand the pain it is to compress Meeting Maker data with the Admin utility) tell you you have to MAP all those users (or go through some Béla Károlyi-class gymnastics variations to remove the relevant data from the database) before their process will work.

Worry no more!

We added a switch "-skipnoemail" if you want to skip over users with either null or empty email addresses, they'll not create ICS files or appear in guest lists. So it saves time all around.

We're not done yet!

We uploaded it to the relevant FTP sites for anyone currently migrating or testing migration.

Call before midnight tonight!

Oracle Calendar migration clients do not need it since they can selectively set the user calendar data they extract.

Oracle Meeting Scheduling Practices and Microsoft Exchange Best Practices

There are several common OCS meeting scheduling practices we need to document, so you, in turn, can educate your users about the differences between scheduling meetings in OCS and Exchange.

FIRST:  Oracle Calendar Server allows a user to organize a meeting and then remove themselves as an attendee.

THIS BEHAVIOR IS IMPOSSIBLE IN EXCHANGE! In fact if users manage to figure out how to do it’s one of the ways to really damage calendar data in Exchange.  We feel it our duty to recommend our customers adopt Microsoft Exchange best practices.  Thus, we do not recommend this behavior get replicated in Exchange.  To help you find and fix those meetings, we have integrated a pre-processing diagnostic step to both diagnose and remedy the issue. 

What it does is to re-create ICS files for the affected users and their meetings.  These files are then inserted using the standard Sumatra process.  The reports allow you to proactively notify or involve any critical calendaring users that they are going to be added to the meetings they though they removed themselves from.

How this works / what you need to do

1.   Export ALL your ICS files into a single directory.  You need to do this anyway for the migration process.

2.   If you create a separate directory that contains the oCalreader, please configure it, and point to the ENTIRE export of ICS files created in step 1

3.   Press the Organizer not Attendee button. Note: this might change in some versions of the tool.  You will have to check “Show Migration Steps”, and then you can click the Organizer button.

                                                        

4.       The tool outputs things in TWO directories

a.    The “Logfile” directory contains three files: the summary text file, along with the two CSV files for the Organizer not Attendee accounts.  These tell you which meeting organizers are affected by this situation.

b.    A subdirectory of the iCalData path gets created called “SpecialICSUsers”. The tool regenerates the ICS files that contain JUST those problem meetings -- and adds the organizer back to the meeting.  This is where the newly generated ICS files are stored

5.      It is possible there are situations where not all meeting organizers’ ATTENDEE record can be found.  Look in the SpecialICSUsers subdirectory, and see if there are any files that start with “_noattendee_”. You will have to figure out how to handle these users/meetings.  For example, if you might see a file called “_noattendee_sarah.jane.smith.ics”. You will have to either have the organizer to the meeting by hand, ask the organizer to add him/herself to the meeting in OCS and then re-export the ICS files, or choose not to migrate the meeting by deleting the files.

6.      oCalReader also checks for missing or invalid email information.  The resultant file is written to the “logfile path”, and called “AccountMissingOrInvalidEmail.txt”.  Read and act on these in advance of your cut-over into production.  For example:

CN=Clara Oswald:mailto:""

CN=Companions Conf Rm:mailto:""

CN=Companions Conf Rm:mailto:100000518943623636038552@email.invalid

CN=Medieval History Room:mailto:""

CN=Medieval History Room:mailto:182D1D7DF8E9ECA5E050C68489657375@email.invalid

CN=Martha Jones:mailto:""

CN=Martha Jones:mailto:100000250843623636038553@email.invalid

                                               

The logfile shows two users, Clara Oswald and Martha Jones, no longer have email addresses (are they terminated accounts?)  It also shows two rooms with problems.  You will have to add those rooms, “Companions Conf Rm” and “Medieval History Room”, to the resources map file to map those accounts to valid SMTP addresses.

7.       Finally to insert meetings where oCalReader has added the organizer back to the meetings, follow these steps:

a)       In the ocalreader directory include your accounts and resource mapping file(s)

b)       Edit the oCalreader Configuration

i.      Change the ICS data file directory to the “SpecialICSUsers” subfolder

ii.     Ensure there are no “limits” set

8.   Push “Read and Insert”

Note these assumptions:

• We add the organizer back to the entire series to preserve recurrence patterns and the integrity of the meeting -- even if the organizer cancels their presence on some (but not all) of the instances.  Implication: There is the potential for duplicated meetings if some occurrences do have the organizer present.)
• We set the organizer to ACCEPT the meeting (that happens by default).  Because the organizer removed themselves from the meeting we set their Free/BUSY status to FREE UNLESS the organizer has set the series to BUSY (then it becomes busy).
• For the very curious this is a full ICS generated by this process.

SECOND:  Oracle Calendar Server allows resources to be meeting organizers.

In Exchange unmanaged public resources should NOT be organizers, but we relax this in certain cases to migrate data.  Why should unmanaged public resources not be meeting organizers?  Resource accounts are DISABLED by default.  As with many rules there are exceptions.  It is perfectly acceptable to have a room direct booked or managed by a delegate (for example: “CEO’s Private Conference Room”).

If you choose to allow direct booking into conference rooms:

Exchange, by default, strips both the subject and owner in the resource calendar

If you wish to retain this information (which is common in Oracle calendar)

You will need to execute this command for those resources: 

set-calendarprocessing  : -deletesubject: $False -addorganizertosubject: $True

This has privacy repercussions!  Showing subjects and organizers reveal potentially sensitive information, such as (these fabricated examples): “Interview James Bond to replace Provost”, “Implications of the 2019 500% Tuition Hike on staff reduction plans” booked in the “President’s Conference Room.” 

You have been warned.

If you do want to determine which resources organize meetings, the “Organizer as Attendee” component generates a file, ResourcesAsMeetingOrganizers, which lists of those accounts. For example, if the Doctor as a designate to the “Tardis Control Room” proposed a meeting on behalf of the “Tardis Control Room”, called “How to use the New Dimensional Portal Controls”, the ResourcesAsMeetingOrganizers file would contain:

CN=Tardis Control Room:mailto:doctor@drwho.timetraveller.org

THIRD:  Oracle calendar server allows resources emails to be assigned to a user account.

In Exchange resources have their own SMTP accounts.  OCS allows the OCS administrator to assign a designate/delegate email to the room (so messages are sent to him/her. For example, the Tardis Break room has an email address that belongs to Amy Pond

ORGANIZER;X-ORACLE-GUID=269167A2DA4992BCE050C6848965230C;CN=Tardis Break Room:mailto:amy.pond@drwho.timetraveller.org

                                                                      

What are your choices?

1)       REMOVE the email address from the OCS rooms and re-export the data;

2)       Leave it unchanged, the designate will become the organizer of the meeting, something that they will not be happy to see on their calendars.

3)       Figure out how big a problem this is.  The “Organizer as Attendee” button generates a file, ResourcesWithUserEmails , that shows the list. For example, the file would contain:

CN=Tardis Break Room:mailto: amy.pond@drwho.timetraveller.org

Block Mail to Recipients Outside of your Organization

We recently announced that we've started work on a  Kerio Server Migration to Office 365.  One of our clients gave us test data from a few departed/terminated users to test our code.  It's easy to test in our Exchange on-prem sandbox to ensure no "external" email gets sent to their users -- we unplug the Ethernet connection to the Router. It's a little more complicated in Office 365, but not all that difficult.  Here are the steps:

In the Exchange Admin Center, under Mail Flow, Rules, click the "+" sign to create a new rule.

• Name the rule.  We called it "Block Mail sent to External Email"
• Select the option from Apply this rule pulldown: "The Recipient is Located...."
• Select the option "Outside The Organization" from the subsequent pulldown that the recipient is located 
• Select "Reject the message with the explanation" from the pulldown "Do the Following..."
• Enter a message (optional):  We entered "The message was not sent. The Recipient is located outside the company."
• We chose to Enforce the rule, and finally
• Saved it

  Here is a screen shot:

So now let's say a user tries to send email outside your domain.  They will be informed that is an unsanctioned action with this message: