Bravo to Dr Abrasive for cracking the Sega Saturn DRM method.
What on earth does this have to do with calendaring?
Well, since we broke Meeting Maker, Oracle Calendar Server, Zimbra, Oracle Beehive, and a few other encoding schemes, we're not allowed to talk about, we're sort of connoisseurs of reverse engineering and call out kudos where they are warranted.
What's kind of amazing is that none of the engineers came forward in the last 20 freaking years with anything that would help this. Of course, for Meeting Maker the group of hacks finally working on it only started to contact us after we'd done everything significant to read their data and insert it into Exchange.
And don't even get us started about the Oracle people!!!!
Monday, July 11, 2016
Wednesday, June 08, 2016
Troubleshoot Outlook Connectivity in Exchange 2016 on-premises
We hit connectivity issues running Outlook 2016 on Exchange 2016 at least once a week. Troubleshooting is difficult because of many platforms and technologies in play.
A recent blog post from the Microsoft Exchange Team titled "Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises)" promises to help us figure it out. There wasn't a check list, per se. Rather it was an organized collection of troubleshooting tips and techniques. We hope it helps us and helps you. (Note: this is NOT intended for Office 365 connectivity issues!)
First, here are examples of the connectivity problems they cite, and we've hit:
Clients prompting for credentials (intermittently or continuously)
Clients getting disconnected
Clients are unable to establish a connection
Clients freezing or going unresponsive
Here is a summary of their tips and recommendations:
* Ensure that everything is fully patched. We find Office Configuration Analyzer Tool (OffCAT) quite helpful. In fact, Microsoft Office released OffCat Version 2.2 June 2016. Download v2.2 here.
* They recommend cached mode vs. online mode to smooth out the user experience. We agree it helps, although it masks connectivity problems.
* Ensure CAS servers are not turning off NIC cards, use outdated drivers, or are not configured for power saving mode. The same holds true for the load balancer -- make sure keep-alive and idle timeouts are set above the 15 minute threshold.
* Too many cores: it's hard to believe that you can have too many cores, but you can. Don't have any more than 24 cores per server
* Configure Exchange performance monitoring ("perfwiz"). MS points you to two articles: Troubleshoot High CPU Utilization in Exchange 2013, and Exchange Monitoring tool, "Exmon"
* Logs: The article recommends Outlook logging, HTTP logging, IIS logging, Exchange Logging, and RPC logs. They recommend a tool, Log Parser Studio, to help parse the logs.
Writing this blog was the easy part. Now we'll have to try each suggestion until we discover what's causing the client connectivity problems.
A recent blog post from the Microsoft Exchange Team titled "Checklist for troubleshooting Outlook connectivity in Exchange 2013 and 2016 (on-premises)" promises to help us figure it out. There wasn't a check list, per se. Rather it was an organized collection of troubleshooting tips and techniques. We hope it helps us and helps you. (Note: this is NOT intended for Office 365 connectivity issues!)
First, here are examples of the connectivity problems they cite, and we've hit:
Here is a summary of their tips and recommendations:
* Ensure that everything is fully patched. We find Office Configuration Analyzer Tool (OffCAT) quite helpful. In fact, Microsoft Office released OffCat Version 2.2 June 2016. Download v2.2 here.
* They recommend cached mode vs. online mode to smooth out the user experience. We agree it helps, although it masks connectivity problems.
* Ensure CAS servers are not turning off NIC cards, use outdated drivers, or are not configured for power saving mode. The same holds true for the load balancer -- make sure keep-alive and idle timeouts are set above the 15 minute threshold.
* Too many cores: it's hard to believe that you can have too many cores, but you can. Don't have any more than 24 cores per server
* Configure Exchange performance monitoring ("perfwiz"). MS points you to two articles: Troubleshoot High CPU Utilization in Exchange 2013, and Exchange Monitoring tool, "Exmon"
* Logs: The article recommends Outlook logging, HTTP logging, IIS logging, Exchange Logging, and RPC logs. They recommend a tool, Log Parser Studio, to help parse the logs.
Writing this blog was the easy part. Now we'll have to try each suggestion until we discover what's causing the client connectivity problems.
Tuesday, April 12, 2016
Double-Booked Meeting Rooms in Office 365 (and how to avoid them)
Update: November, 2017: If you look at newer postings you'll see details on our new method of finding and fixing double booked meetings. If you want to try it, contact us with the size of your Exchange installation and your test domain (really, please don't start in production!).
Let's talk meeting rooms (aka resources) in Office 365 / Exchange 2016.
Why might you NOT "Allow repeating meetings" when configuring meeting room booking options?

Short answer: Because you have to be really careful. Especially in Office 365 if you are working entirely from the user interface.
Note that for a one-time meeting where I have the Room 222 booked all day, attempting in Office 365 to book that resource does not even allow it to be seen as an option (this is a REALLY good thing):
But setting up a recurring meeting that overlaps with it easily allows me to book the conference room over a conflict, which might cause me as the meeting organizer some worry down the road.
To avoid this unpleasantness entirely, you need to use Set-CalendarProcessing via PowerShell.
To see what the settings on this resource are:
Get-CalendarProcessing -Identity "Conference Room 222" | fl
to set the ones that deal with conflicts/recurring items:
Set-CalendarProcessing -Identity "Conference Room 222" -AllowConflicts $true -ConflictPercentageAllowed 80 -AllowRecurringMeetings $true
Note that here we allow a conflict percentage of 80, which is below the threshold we set when setting up our demonstration. If you want to allow no conflicts at all via auto-booking:
Set-CalendarProcessing -Identity "Conference Room 222" -AllowConflicts $false -AllowRecurringMeetings $true
The Microsoft manual for this goes into some good detail:
https://technet.microsoft.com/en-us/library/dd335046(v=exchg.160).aspx
And as always, be on the lookout for functionality that works in Exchange on-premises but differently in Office 365.
Side note: in our classic Sumatra Utilities for Exchange 2003 we included a double-booking checker, and this would not be hard to do but we hear little demand for it in Office 365/ Exchange 2016.
Addition January 23, 2017:
Folks, this post is insanely popular. And we're wondering why.
Our initial motivation is to make sure people do not run into this during a calendar migration. But the number of folks reading this regularly indicates it's a common problem.
Now -- Exchange does a VERY GOOD JOB of notifying a meeting organizer of exactly WHERE conflicts are at the time of booking. Minimal personal organization skills then allow you to make modifications to individual instances to smooth things over.
Only thing we can figure out is that folks don't do this and they're looking for an upcoming "oops! You have a conflict in that meeting room coming up!" notice.
Is that the case or is it something else?
We never learned anything listening to ourselves talk, so if you have something to share please let us know.
Let's talk meeting rooms (aka resources) in Office 365 / Exchange 2016.
Why might you NOT "Allow repeating meetings" when configuring meeting room booking options?
Short answer: Because you have to be really careful. Especially in Office 365 if you are working entirely from the user interface.
Note that for a one-time meeting where I have the Room 222 booked all day, attempting in Office 365 to book that resource does not even allow it to be seen as an option (this is a REALLY good thing):
But setting up a recurring meeting that overlaps with it easily allows me to book the conference room over a conflict, which might cause me as the meeting organizer some worry down the road.
To avoid this unpleasantness entirely, you need to use Set-CalendarProcessing via PowerShell.
To see what the settings on this resource are:
Get-CalendarProcessing -Identity "Conference Room 222" | fl
to set the ones that deal with conflicts/recurring items:
Set-CalendarProcessing -Identity "Conference Room 222" -AllowConflicts $true -ConflictPercentageAllowed 80 -AllowRecurringMeetings $true
Note that here we allow a conflict percentage of 80, which is below the threshold we set when setting up our demonstration. If you want to allow no conflicts at all via auto-booking:
Set-CalendarProcessing -Identity "Conference Room 222" -AllowConflicts $false -AllowRecurringMeetings $true
https://technet.microsoft.com/en-us/library/dd335046(v=exchg.160).aspx
And as always, be on the lookout for functionality that works in Exchange on-premises but differently in Office 365.
Side note: in our classic Sumatra Utilities for Exchange 2003 we included a double-booking checker, and this would not be hard to do but we hear little demand for it in Office 365/ Exchange 2016.
Addition January 23, 2017:
Folks, this post is insanely popular. And we're wondering why.
Our initial motivation is to make sure people do not run into this during a calendar migration. But the number of folks reading this regularly indicates it's a common problem.
Now -- Exchange does a VERY GOOD JOB of notifying a meeting organizer of exactly WHERE conflicts are at the time of booking. Minimal personal organization skills then allow you to make modifications to individual instances to smooth things over.
Only thing we can figure out is that folks don't do this and they're looking for an upcoming "oops! You have a conflict in that meeting room coming up!" notice.
Is that the case or is it something else?
We never learned anything listening to ourselves talk, so if you have something to share please let us know.
Tuesday, April 05, 2016
Exchange 2016 Permissions Debugging Protocol
We put together a permissions debugging protocol for Exchange 2016 / Office 365 calendar migrations.
You can download it here.
You can download it here.
Tuesday, March 22, 2016
Turning off reminders during a calendar migration to Microsoft Exchange / Office 365
We've been doing a bunch of Oracle Calendar Server to Exchange migrations lately and this has the benefit of giving me lots of detail oriented stuff to write about, and the downside that I'm getting tired of writing about it.
So what happens when you do not want to get reminders for meetings as you migrate into Exchange?
This is an Outlook configuration issue. That is to say, this is normal Exchange functionality. Meeting requests in Exchange create a tentatively accepted calendar item that carries with it the “default reminder time” set in Calendar Options. Note that this applies to meeting invitations, not to appointments (calendar events with no attendees) which will all go in with no reminders set in an OCS to Exchange migration (because Oracle Calendar Server strips them off in the ICS export).
The PowerShell syntax, to TURN OFF for all MBX USERS (before migration)
get-mailbox -resultsize unlimited -filter {isResource -eq $false} | Set-MailboxCalendarConfiguration -RemindersEnabled:$false
-DefaultReminderTime:0 -ReminderSoundEnabled:$false
to TURN ON (after migration)
get-mailbox -resultsize unlimited -filter {isResource -eq $false} | Set-MailboxCalendarConfiguration -RemindersEnabled:$true
This can be set in PowerShell for one user as in this screenshot:
Finally, note that if a user is logged in or connected (OWA/Outlook) setting these preferences will not kick in until the user logs out and logs back in. So, if you are testing it, you will see the settings change BUT you won’t see an reminders turn off until you log out/in.
So what happens when you do not want to get reminders for meetings as you migrate into Exchange?
This is an Outlook configuration issue. That is to say, this is normal Exchange functionality. Meeting requests in Exchange create a tentatively accepted calendar item that carries with it the “default reminder time” set in Calendar Options. Note that this applies to meeting invitations, not to appointments (calendar events with no attendees) which will all go in with no reminders set in an OCS to Exchange migration (because Oracle Calendar Server strips them off in the ICS export).
The PowerShell syntax, to TURN OFF for all MBX USERS (before migration)
get-mailbox -resultsize unlimited -filter {isResource -eq $false} | Set-MailboxCalendarConfiguration -RemindersEnabled:$false
-DefaultReminderTime:0 -ReminderSoundEnabled:$false
to TURN ON (after migration)
get-mailbox -resultsize unlimited -filter {isResource -eq $false} | Set-MailboxCalendarConfiguration -RemindersEnabled:$true
-DefaultReminderTime:00:15:00 -ReminderSoundEnabled:$true
This can be set in PowerShell for one user as in this screenshot:
Finally, note that if a user is logged in or connected (OWA/Outlook) setting these preferences will not kick in until the user logs out and logs back in. So, if you are testing it, you will see the settings change BUT you won’t see an reminders turn off until you log out/in.
Thursday, March 17, 2016
Microsoft SQL going after Oracle? Don't forget the calendars in Oracle Calendar Server or Beehive!
Microsoft waves the 'free licenses' flag to try to grab Oracle database users?
Sure.
I'm not certain that the arrogance of Microsoft is preferable to the arrogance of Oracle (what's the difference between Larry Ellison and god? God doesn't think he's Larry Ellison.). But free is a hard argument.
Just remember we can migrate your enterprise calendars (Key word: Enterprise-- if you have 100 Oracle calendars, just use client-side methods. If you have 500+ users and mission-critical meetings, check us out).
Sure.
I'm not certain that the arrogance of Microsoft is preferable to the arrogance of Oracle (what's the difference between Larry Ellison and god? God doesn't think he's Larry Ellison.). But free is a hard argument.
Just remember we can migrate your enterprise calendars (Key word: Enterprise-- if you have 100 Oracle calendars, just use client-side methods. If you have 500+ users and mission-critical meetings, check us out).
Tuesday, March 15, 2016
Special case: When there is no user GUID in your OCS Exports
The ICS export for Oracle Calendar
Server is inconsistent with exporting GUIDS:
As in this edited real world example:
Why is this relevant to anything that you care about?
Because interspersed with these very
valid users you wish to migrate are also decommissioned users you do not. Our process will send invitations to them
during migration.
For valid (whether mapped or not) users
in Exchange this is not an issue and they will be dealt with properly.
For users who have been decommissioned and
do not have addresses on Exchange, this will result in NDRs (Non-delivery
reports) being send back to the meeting organizer for these decommissioned
users. We have no way of knowing who
they are (since they look like other perfectly valid users) and some sites keep
these users in attendee lists as a matter of historical record.
The consequences to you:
1.
Our clean-up
process REMOVES any NDRs before end users see them post-migration. We find users and admins value this ease of
use and clean migration
2.
The decommissioned
user will still be in the meeting attendees list. So: a.) All attendees will still see that user in the attendee list and b.) In future meeting updates
post-migration the organizer will again get NDRs and it will be up to them to
remove the user at that time, or maintain them in the guest list as they so choose.
Wednesday, March 09, 2016
#Microsoft #SQLServer is coming to Linux?
SQL Server is coming to Linux?
Sounds logical, given that we've already been informed parts of the Office 365 backend are Linux. See our earlier post: Microsoft and Linux: Sign of the End Times?
I'm glad to see this happening, also because it potentially makes it so much easier for one of those migrations where we read legacy calendar server databases directly and migrate them into Exchange -- you know:
Oracle Beehive to Exchange
or Apple iCalendar to Exchange
Sounds logical, given that we've already been informed parts of the Office 365 backend are Linux. See our earlier post: Microsoft and Linux: Sign of the End Times?
I'm glad to see this happening, also because it potentially makes it so much easier for one of those migrations where we read legacy calendar server databases directly and migrate them into Exchange -- you know:
Oracle Beehive to Exchange
or Apple iCalendar to Exchange
Tuesday, March 08, 2016
Easy way of finding Oracle Calendar meetings without organizers post-migration to Exchange
You're migrating Oracle Calendar Server to Microsoft Exchange or Office 365 and you want to deal with the situation of Meetings without Organizers. See our earlier post.
Given that we can run an analysis to identify these meetings before your migration you have your choice of informing your users about specific meetings (to make it as easy as possible), or handling it afterwards. Both is probably good.
Easiest way to find them post-migration is to search on the "Meeting_MissingOrganizer" category in Outlook.
But if you are using Office 365 or on-premises via OWA instead of Outlook there's no easy list display.
You can can however find all of these visually using categories.
So set a specific color looking for the category we added on insertion, as so:
Again, better to get the word out to your users before the migration and get them used to the way things work in Exchange as opposed to Oracle Calendar Server, but we're all about the reality of the situation.
Given that we can run an analysis to identify these meetings before your migration you have your choice of informing your users about specific meetings (to make it as easy as possible), or handling it afterwards. Both is probably good.
Easiest way to find them post-migration is to search on the "Meeting_MissingOrganizer" category in Outlook.
But if you are using Office 365 or on-premises via OWA instead of Outlook there's no easy list display.
You can can however find all of these visually using categories.
So set a specific color looking for the category we added on insertion, as so:
And you'll get an easy-to-locate display like this:
Again, better to get the word out to your users before the migration and get them used to the way things work in Exchange as opposed to Oracle Calendar Server, but we're all about the reality of the situation.
Tuesday, March 01, 2016
Oracle Meeting Scheduling Practices and Microsoft Exchange Best Practices
There are several common OCS meeting
scheduling practices we need to document, so you, in turn, can educate your
users about the differences between scheduling meetings in OCS and Exchange.
FIRST:
Oracle Calendar Server allows a user to organize a meeting and then
remove themselves as an attendee.
THIS BEHAVIOR IS IMPOSSIBLE IN
EXCHANGE! In fact if users manage to figure out how to do it’s one of the ways
to really damage calendar data in Exchange. We feel it our duty to recommend our customers
adopt Microsoft Exchange best practices.
Thus, we do not recommend this behavior get replicated in Exchange. To help you find and fix those meetings, we
have integrated a pre-processing diagnostic step to both diagnose and remedy
the issue.
What it does is to re-create ICS files
for the affected users and their meetings.
These files are then inserted using the standard Sumatra process. The reports allow you to proactively notify
or involve any critical calendaring users that they are going to be added to the
meetings they though they removed themselves from.
How this works / what you need to
do
1. Export ALL your ICS files into a single
directory. You need to do this anyway
for the migration process.
2. If you create a separate directory
that contains the oCalreader, please configure it, and point to the ENTIRE export
of ICS files created in step 1
3. Press the Organizer not Attendee button. Note: this
might change in some versions of the tool.
You will have to check “Show Migration Steps”, and then you can click
the Organizer button.
4.
The tool outputs things in TWO
directories
a. The “Logfile” directory contains three
files: the summary text file, along with the two CSV files for the Organizer
not Attendee accounts. These tell you
which meeting organizers are affected by this situation.
b. A subdirectory of the iCalData path gets
created called “SpecialICSUsers”. The tool regenerates the ICS files that
contain JUST those problem meetings -- and adds the organizer back to the
meeting. This is where the newly
generated ICS files are stored
5. It is
possible there are situations where not all meeting organizers’ ATTENDEE record
can be found. Look in the SpecialICSUsers subdirectory, and see if there
are any files that start with “_noattendee_”. You will have to figure out
how to handle these users/meetings. For example, if you might see a file
called “_noattendee_sarah.jane.smith.ics”. You will have to either have the
organizer to the meeting by hand, ask the organizer to add him/herself to the
meeting in OCS and then re-export the ICS files, or choose not to migrate the
meeting by deleting the files.
6. oCalReader
also checks for missing or invalid email information. The resultant file is written to the “logfile
path”, and called “AccountMissingOrInvalidEmail.txt”. Read and act on these in advance of your
cut-over into production. For example:
CN=Clara Oswald:mailto:""
CN=Companions Conf Rm:mailto:""
CN=Companions Conf
Rm:mailto:100000518943623636038552@email.invalid
CN=Medieval History Room:mailto:""
CN=Medieval History
Room:mailto:182D1D7DF8E9ECA5E050C68489657375@email.invalid
CN=Martha Jones:mailto:""
CN=Martha
Jones:mailto:100000250843623636038553@email.invalid
The logfile shows two users, Clara Oswald
and Martha Jones, no longer have email addresses (are they terminated
accounts?) It also shows two rooms with
problems. You will have to add those
rooms, “Companions Conf Rm” and “Medieval History Room”, to the resources map
file to map those accounts to valid SMTP addresses.
7.
Finally to
insert meetings where oCalReader has added the organizer back to the meetings,
follow these steps:
a)
In the
ocalreader directory include your accounts and resource mapping file(s)
b)
Edit the
oCalreader Configuration
i. Change
the ICS data file directory to the “SpecialICSUsers” subfolder
ii. Ensure there are no “limits” set
8. Push “Read and Insert”
Note these assumptions:
- We add the organizer back
to the entire series to preserve recurrence patterns and the
integrity of the meeting -- even if the organizer cancels their presence
on some (but not all) of the instances. Implication: There is the
potential for duplicated meetings if some occurrences do have the
organizer present.)
- We set the organizer to
ACCEPT the meeting (that happens by default). Because the organizer
removed themselves from the meeting we set their Free/BUSY status to FREE UNLESS
the organizer has set the series to BUSY (then it becomes busy).
- For the very curious this is a full ICS generated by this process.
SECOND:
Oracle Calendar Server allows resources to be meeting organizers.
In Exchange unmanaged public resources should
NOT be organizers, but we relax this in certain cases to migrate data. Why should unmanaged public resources not be
meeting organizers? Resource accounts
are DISABLED by default. As with many
rules there are exceptions. It is perfectly
acceptable to have a room direct booked or managed by a delegate (for example:
“CEO’s Private Conference Room”).
If you choose to allow direct booking
into conference rooms:
Exchange, by default, strips both the subject
and owner in the resource calendar
If you wish to retain this information
(which is common in Oracle calendar)
You will need to execute this command
for those resources:
set-calendarprocessing
|
This has privacy repercussions! Showing subjects and organizers reveal
potentially sensitive information, such as (these fabricated examples):
“Interview James Bond to replace Provost”, “Implications of the 2019 500%
Tuition Hike on staff reduction plans” booked in the “President’s Conference
Room.”
You have been warned.
If you do want to determine which
resources organize meetings, the “Organizer as Attendee” component generates a
file, ResourcesAsMeetingOrganizers, which lists
of those accounts. For example, if the Doctor as a designate to the “Tardis
Control Room” proposed a meeting on behalf of the “Tardis Control Room”, called
“How to use the New Dimensional Portal Controls”, the ResourcesAsMeetingOrganizers
file would contain:
CN=Tardis
Control Room:mailto:doctor@drwho.timetraveller.org
THIRD:
Oracle calendar server allows resources emails to be assigned to a user
account.
In Exchange resources have their own
SMTP accounts. OCS allows the OCS
administrator to assign a designate/delegate email to the room (so messages are
sent to him/her. For example, the Tardis Break room has an email address that
belongs to Amy Pond
ORGANIZER;X-ORACLE-GUID=269167A2DA4992BCE050C6848965230C;CN=Tardis
Break Room:mailto:amy.pond@drwho.timetraveller.org
What are your choices?
1)
REMOVE the email address
from the OCS rooms and re-export the data;
2)
Leave it unchanged,
the designate will become the organizer of the meeting, something that they
will not be happy to see on their calendars.
3)
Figure out how big
a problem this is. The “Organizer as
Attendee” button generates a file, ResourcesWithUserEmails , that shows the
list. For example, the file would contain:
CN=Tardis Break
Room:mailto: amy.pond@drwho.timetraveller.org
Monday, February 22, 2016
How to fix: 'Microsoft.ACE.OLEDB.12.0' provider is not registered on the local machine
Our full-state calendar migration executes on a 32-bit or 64-bit architecture. We default to using the JET database
engine, though this can be changed in _Config_XML:
If
you don't have the x64 version of MS Office 2010 installed, (or NO version of
office installed) download the Microsoft Access DB Engine 2010 Redistributable
(pick the x64 version!)
http://www.microsoft.com/download/en/details.aspx?id=13255
Install the x64 version via the Command Prompt with:
AccessDatabaseEngine_X64.exe /passive
Do
it this way OR you'll see the error: The 'Microsoft.ACE.OLEDB.12.0'
provider is not registered on the local machine.
The error will look like this in the error window of our migration tools:
The error will look like this in the error window of our migration tools:
If you look in _Config_XML you will see the default (top line) as
well as the other options should you need to change them.
Friday, February 19, 2016
On .NET Framework 4.6.1 and Exchange DANGER!
Please read On .NET Framework 4.6.1 and Exchange compatibility and then DO NOT install .NET 4.6.1.
Step away from the installer.....
Because if you allow it, to quote Microsoft "Mailboxes are quarantined and databases fail over unexpectedly in Exchange Server 2013."
Step away from the installer.....
Because if you allow it, to quote Microsoft "Mailboxes are quarantined and databases fail over unexpectedly in Exchange Server 2013."
Follow-on to January 1, 1970 and your iPhone
In the annals of "why there?" starting dates for calendar systems, the iPhone is not alone.
Consider UNIAPI_TIME for Oracle Calendar System (a relic of when it was Steltor and prior to that CSandT).
So if you're using OCS and do a UNICPOUTU export (aka the OLD way of doing a full-state calendar migration with us) you'll see lines in the data files that look like:
That "S" entry is the start time measured as the number of seconds since January 1, 1991 at 00:01:00 AM.
And of course you need to be careful of this since the great North American DST redefinition.
Consider UNIAPI_TIME for Oracle Calendar System (a relic of when it was Steltor and prior to that CSandT).
So if you're using OCS and do a UNICPOUTU export (aka the OLD way of doing a full-state calendar migration with us) you'll see lines in the data files that look like:
K Events:
S 9699600
D 30
T J2J
G CR Mozart
I 0
R N2
M Hendrix Jimi
W Hendrix Jimi
A TRUE 0 10
C [ .+] Hendrix Jimi
C [ .-] Garcia Jerry
O
That "S" entry is the start time measured as the number of seconds since January 1, 1991 at 00:01:00 AM.
And of course you need to be careful of this since the great North American DST redefinition.
Tuesday, February 16, 2016
Email Migration Microsoft Exchange to Office 365 via imapsync
Yet another in the ever-expanding popular series of how to use imapsync (one of the best low-cost tools available) to migrate email to Exchange. This time for Microsoft Exchange to Microsoft Exchange or Office 365.
Yes. Microsoft offers tools for this, and we've documented them.
Evaluate them yourselves. We have. That's why we recommend imapsync. Full disclosure: we have no relationship with imapsync and derive exactly zero revenue from this non-existent relationship.
We see a wide spectrum of Exchange expertise (*cough* admin at university in Pennsylvania stymied by Junk E-mail folder *cough*) so we think it's useful to document this process.
Your comments are welcome and will be added and credited.
First step:
Make sure you are enabled for IMAP on both sides of your Exchange migration.
We've gone over this in our post Quick guide to Enabling IMAP in @Office365 / #MSFTExchange 2013.
Second step:
Set up a Service Account with access to all your Exchange accounts. Confused about Exchange permissions and setting up your service account? Read our post The Cookbook Version of Exchange 2013 Migration Rights.
Third step:
Test this before you do something rash.
Gilles (imapsync's AWESOMELY capable author who deserves HUGE credit for his application) has a sensible method of beginning in his sample batch file (seen in this excerpt):
...
Yes. Microsoft offers tools for this, and we've documented them.
Evaluate them yourselves. We have. That's why we recommend imapsync. Full disclosure: we have no relationship with imapsync and derive exactly zero revenue from this non-existent relationship.
We see a wide spectrum of Exchange expertise (*cough* admin at university in Pennsylvania stymied by Junk E-mail folder *cough*) so we think it's useful to document this process.
Your comments are welcome and will be added and credited.
First step:
Make sure you are enabled for IMAP on both sides of your Exchange migration.
We've gone over this in our post Quick guide to Enabling IMAP in @Office365 / #MSFTExchange 2013.
Second step:
Set up a Service Account with access to all your Exchange accounts. Confused about Exchange permissions and setting up your service account? Read our post The Cookbook Version of Exchange 2013 Migration Rights.
Third step:
Test this before you do something rash.
Gilles (imapsync's AWESOMELY capable author who deserves HUGE credit for his application) has a sensible method of beginning in his sample batch file (seen in this excerpt):
...
@REM Three other options are in this example because they are good to start with @REM @REM --dry makes imapsync doing nothing, just print what would be done without --dry. @REM @REM --justfolders does only things about folders (ignore messages). It is good @REM to verify the folder mapping is good for you. @REM @REM --automap guesses folders mapping, for folders like @REM "Sent", "Junk", "Drafts", "All", "Archive", "Flagged".
@REM
...
Or: do a DRY run, make sure it's handling the folders you want, and let the application guess on your folder mapping (for Microsoft Exchange to Office 365 or Exchange not complex).
Not too controversial or inscrutable, right?
So for ONE USER where you have the password for that user on BOTH systems start with:
imapsync.exe ^
--host1 sumatra.com --user1 jimi.hendrix@sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --user2 ^
jimi.hendrix@sumatra.onmicrosoft.com --password2 "XXXXX" ^
--ssl2 --sep2 / --dry --justfolders --automap
once things are doing what you want, you can turn this into:
imapsync.exe ^
--host1 sumatra.com --user1 jimi.hendrix@sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --user2 ^
jimi.hendrix@sumatra.onmicrosoft.com --password2 "XXXXX" ^
--ssl2 --sep2 / --automap
If you set up a service account with FullAccess on Office 365, you accomplish a migration without knowing any password except your service account by using a command like this (where password2 is the service account password):
imapsync.exe ^
--host1 zimbra.sumatra.com ^
--user1 jimi.hendrix@zimbra.sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --port2 993 --sep2 / ^
--user2 jimi.hendrix@sumatra.onmicrosoft.com ^
--authuser2 SERVICE_ACCT@sumatra.onmicrosoft.com ^
--password2 "XXXXX" --ssl2 --dry --justfolders --automap
remember --dry --justfolders are for debugging.
Note in the above, for an Office 365 target system we need to use the "--sep2 /" command.
Note also that this gives you the direct capability to map your user ID, for instance from "jimi.hendrix" on your legacy system to "jhendrix1967" or "jimi.hendrix1967" on your target system.
Iterating over a user list
In any event you are going to need to generate a user list to migrate email. Can you keep your migrating user list separate from your migration script? Answer: YES. This method assumes your legacy ID is the same as your target ID, but allowing for this to change is not a hard extension.
The imapsync ZIP file contains a script for iterating on a user list:
sync_loop_windows.bat
Which also contains an excellent primer on running imapsync in parallel.
This batch file assumes a text file in the form "User1;Password1;User2;Password2;..." (TIP: If you use a service account you will not need the password)
To get a user list from Exchange get-mailbox is the simplest method :
get-mailbox | emailaddress > myusers.txt
You could also use ldapsearch if you already have scripts for that.
If you want to get a list of all users in an OU to migrate by segments:
get-mailbox (e.g., -organizationalunit users
More Migration Details
Throttling in Exchange/Office 365
You're more likely to be throttled in Office 365 since the controls to that environment are largely out of your hands.
As per our usual mantra: test everything before you go into production.
If you get throttled, there are two imapsync switches you can tweak.
One limits the transfer rate to a specific number of messages:
--maxmessagespersecond
Start at 10 (say) and work up or down from there. In calendar migrations we start with 25, but our gut experience tells us calendar data is smaller per object on average.
The other limits the transfer rate by byte if that works better for your network environment.
--maxbytespersecond
You can also check out our post Throttling in Exchange 2013.
Really helpful options
--buffersize 8192000 imapsync has a default I/O buffer of 4 Kb. Upping this to 8 Mb will probably speed things for you
--syncinternaldates: Not for Exchange, but some email systems misuse email dates and you therefore run the risk of the receipt dates on your target system (what imapsync refers to as host2) becoming the date of insertion. This command avoids that unfortunate event.
--fast: this prevents flags from being synced and therefore makes the process (wait for it....) faster. Not an option to invoke if you want / need to sync flags!
--dry: this is a really useful option for development and debugging, It just goes through the motions of logging onto both source and target system and displays the status -- a dry run. Use this or debugging options -debug and -debugimap
See our section on Common Problems and Solutions in our Sumatra's DIY Guide to an Office 365 Migration.
Or: do a DRY run, make sure it's handling the folders you want, and let the application guess on your folder mapping (for Microsoft Exchange to Office 365 or Exchange not complex).
Not too controversial or inscrutable, right?
So for ONE USER where you have the password for that user on BOTH systems start with:
imapsync.exe ^
--host1 sumatra.com --user1 jimi.hendrix@sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --user2 ^
jimi.hendrix@sumatra.onmicrosoft.com --password2 "XXXXX" ^
--ssl2 --sep2 / --dry --justfolders --automap
once things are doing what you want, you can turn this into:
imapsync.exe ^
--host1 sumatra.com --user1 jimi.hendrix@sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --user2 ^
jimi.hendrix@sumatra.onmicrosoft.com --password2 "XXXXX" ^
--ssl2 --sep2 / --automap
If you set up a service account with FullAccess on Office 365, you accomplish a migration without knowing any password except your service account by using a command like this (where password2 is the service account password):
imapsync.exe ^
--host1 zimbra.sumatra.com ^
--user1 jimi.hendrix@zimbra.sumatra.com --password1 "XXXX" ^
--host2 outlook.office365.com --port2 993 --sep2 / ^
--user2 jimi.hendrix@sumatra.onmicrosoft.com ^
--authuser2 SERVICE_ACCT@sumatra.onmicrosoft.com ^
--password2 "XXXXX" --ssl2 --dry --justfolders --automap
remember --dry --justfolders are for debugging.
Note in the above, for an Office 365 target system we need to use the "--sep2 /" command.
Note also that this gives you the direct capability to map your user ID, for instance from "jimi.hendrix" on your legacy system to "jhendrix1967" or "jimi.hendrix1967" on your target system.
Iterating over a user list
In any event you are going to need to generate a user list to migrate email. Can you keep your migrating user list separate from your migration script? Answer: YES. This method assumes your legacy ID is the same as your target ID, but allowing for this to change is not a hard extension.
The imapsync ZIP file contains a script for iterating on a user list:
sync_loop_windows.bat
Which also contains an excellent primer on running imapsync in parallel.
This batch file assumes a text file in the form "User1;Password1;User2;Password2;..." (TIP: If you use a service account you will not need the password)
To get a user list from Exchange get-mailbox is the simplest method :
get-mailbox | fl name,emailaddress > myusers.txt
to give you full name, email address.
For just email address run:
get-mailbox | emailaddress > myusers.txt
You could also use ldapsearch if you already have scripts for that.
If you want to get a list of all users in an OU to migrate by segments:
get-mailbox (e.g., -organizationalunit users
More Migration Details
Throttling in Exchange/Office 365
You're more likely to be throttled in Office 365 since the controls to that environment are largely out of your hands.
As per our usual mantra: test everything before you go into production.
If you get throttled, there are two imapsync switches you can tweak.
One limits the transfer rate to a specific number of messages:
--maxmessagespersecond
Start at 10 (say) and work up or down from there. In calendar migrations we start with 25, but our gut experience tells us calendar data is smaller per object on average.
The other limits the transfer rate by byte if that works better for your network environment.
--maxbytespersecond
You can also check out our post Throttling in Exchange 2013.
Really helpful options
--buffersize 8192000 imapsync has a default I/O buffer of 4 Kb. Upping this to 8 Mb will probably speed things for you
--syncinternaldates: Not for Exchange, but some email systems misuse email dates and you therefore run the risk of the receipt dates on your target system (what imapsync refers to as host2) becoming the date of insertion. This command avoids that unfortunate event.
--fast: this prevents flags from being synced and therefore makes the process (wait for it....) faster. Not an option to invoke if you want / need to sync flags!
--dry: this is a really useful option for development and debugging, It just goes through the motions of logging onto both source and target system and displays the status -- a dry run. Use this or debugging options -debug and -debugimap
See our section on Common Problems and Solutions in our Sumatra's DIY Guide to an Office 365 Migration.
Monday, February 15, 2016
January 1, 1970 and your iPhone.
Four words: Do not do it.
See:
Why the 1970 Bug Bricks Your iPhone
An interesting variation on this to experiment with would be to go back to the 1960s, insert some calendar entries and watch what happens, but I'll leave that for you in the "willing to sacrifice my smartphone for the sake of science" crowd to try that out and report on.
Brings back a couple of incidents where back in the days of Meeting Maker migrations one of the characteristics of bad data was pre-computer era dates (like the 1940s) in corrupted calendar entries. So figuring that we ignored any dates before 1990 if we got them in a Meeting Maker server.
Then one day we found a site with a user who actually ENTERED DATA BACK TO THE 1950s! Mainly birthdays.
See:
Why the 1970 Bug Bricks Your iPhone
An interesting variation on this to experiment with would be to go back to the 1960s, insert some calendar entries and watch what happens, but I'll leave that for you in the "willing to sacrifice my smartphone for the sake of science" crowd to try that out and report on.
Brings back a couple of incidents where back in the days of Meeting Maker migrations one of the characteristics of bad data was pre-computer era dates (like the 1940s) in corrupted calendar entries. So figuring that we ignored any dates before 1990 if we got them in a Meeting Maker server.
Then one day we found a site with a user who actually ENTERED DATA BACK TO THE 1950s! Mainly birthdays.
Tuesday, February 09, 2016
Slow PowerShell Response to Get-Mailbox Debugging Protocol
Gentle reader of calendar migration intent,
Ms. Calendar received the following (edited slightly for clarity):
When setting up an impersonation account in our active environment the first few steps (editor's note: see The Cookbook Version of Exchange Migration Rights) go fine and then when I run this:
Get-Mailbox -resultsize unlimited | add-mailboxpermission -user OurDomainUser -accessrights: fullaccess -InheritanceType: All
It just sits there and never spits out any output. Yesterday I left it running for hours and eventually Exchange Management Shell closed itself but I still get the you don't have access to this email when trying to log in to any email accounts with the service account.
An interesting case which does occur!
There is not a lot of love across the community for Get-Mailbox performance and even less for practical actions to take. See: Performance issues with Get-Mailbox -Database PowerShell cmdlet and Count Mailboxes Per Database Faster.
Ms. Calendar received the following (edited slightly for clarity):
When setting up an impersonation account in our active environment the first few steps (editor's note: see The Cookbook Version of Exchange Migration Rights) go fine and then when I run this:
Get-Mailbox -resultsize unlimited | add-mailboxpermission -user OurDomainUser -accessrights: fullaccess -InheritanceType: All
It just sits there and never spits out any output. Yesterday I left it running for hours and eventually Exchange Management Shell closed itself but I still get the you don't have access to this email when trying to log in to any email accounts with the service account.
An interesting case which does occur!
There is not a lot of love across the community for Get-Mailbox performance and even less for practical actions to take. See: Performance issues with Get-Mailbox -Database PowerShell cmdlet and Count Mailboxes Per Database Faster.
Hats off and thanks to Russ and his team for this migration-centric debugging protocol:
It is probably a problem with "get-mailbox"
To test:
1) First try without any qualifiers:
get-mailbox
or limit it to 100
get-mailbox -resultsize 100
2) Second add the resultsize qualifier
Get-Mailbox -resultsize unlimited
Once they set a static limit instead of unlimited they were able to set impersonation on all accounts.
We have seen in instances where the performance is deplorable it's usually a problem getting the "join"s to work through AD. Keep in mind we're linking two separate AD actions here, Get-Mailbox and Add-MailboxPermission.
We have seen in instances where the performance is deplorable it's usually a problem getting the "join"s to work through AD. Keep in mind we're linking two separate AD actions here, Get-Mailbox and Add-MailboxPermission.
Where is the AD server located?
on the same subnet?
Are there access issues there?
To test if it's a connectivity issue, use PortQry to check for connectivity issues. You can download it from Microsoft. Test ports 389 (LDAP) and 3268 (Global Catalog). There are other possible ports, but these are the main ones used.
On the related issue when you have trouble with Exchange performance itself, see Troubleshooting performance issues with Exchange when RPC request spike high.
On the related issue when you have trouble with Exchange performance itself, see Troubleshooting performance issues with Exchange when RPC request spike high.
Subscribe to:
Posts (Atom)