A funny thing happened on the way to upgrading the Microsoft Directory Sync tool to Azure AD Connect. I had updated DirSync to the new Azure AD Connect tool following the directions that Microsoft provides. Everything seemed to go swimmingly. When it was all said and done, syncing to Azure AD was working fine, but for whatever reason, the changes that were replicated to Azure AD were not replicating to Exchange Online.
From the Beginning
To provide more context on what happened, I had DirSync installed long ago on a separate server (when it was NOT able to be installed on a domain controller). Since then, plenty of updates and iterations were released, including Azure AD Sync, but we stayed with DirSync for the time being. After Azure AD Connect was announced, we decided it was time to upgrade. Not only that, we could install Azure AD Connect on a domain controller and decommission a server with no other function than to perform directory synchronization with Azure AD. This meant we needed to perform a parallel deployment.
- dirsync – Dir Sync Server – Windows Server 2012
- dc02 – Domain Controller – Windows Server 2012 R2
The deployment went as expected and dc02 (with Azure AD Connect installed and configured) was running as expected. When running through the Azure AD Connect Wizard, I configured it using the same Office 365 Azure AD accounts I made specifically for directory synchronization (Dont even get me started on the problems expired passwords cause with DirSync). After everything was completed, I made a change to my test account on prem and viewed it in Azure AD to confirm it was working. I made the assumption that if it made the change there, it would be working on the other Office 365 services as well.
One week later, we had a new user that needed to be created. Simple task that can be carried out by anyway, so I assigned it to a junior analyst who knew what to do. Unfortunately, that’s where things got weird. The user was created successfully on prem and was migrated to Office 365 with no problem and all the steps were taking for the mailbox. The user mailbox was created, but not all the properties were synced over to Exchange Online (title, city, state, etc.). We only learned about this because there was an Exchange mail rule that created a signature and filled in the attributes based on what was in AD.
When I see these kind of problems, there’s a few things I always check first just to make sure it’s working correctly.
- Is the user configured correctly on prem?
- UPN, PrimarySMTP address, etc.
- Has the Synchronization service run recently?
- Has the password for Azure AD account configured with DirSync expired?
I double and triple checked all these things. I recreated the user to see if it was simply a fluke but the same behavior was still happening. What is happening here?
Being a Microsoft Cloud Partner, I opened a case with them to look at it. They confirmed that for whatever reason, Exchange Online was not able to see or use the synced attributes from Azure AD. I think to myself “this has to be a Microsoft problem then”. They work on it for a while, but still come up short.
Then, Microsoft asks if they can see the configuration on prem. I setup a support session and we dig into it. They review the event logs and there are no errors that would point to this being the problem. They continue to engage with the product team to see if they can figure out what is the issue.
That Doesn’t Look Right
At this point, I’m frustrated, the client is frustrated, and Microsoft has no answers for me. We’ve been on this for days now and it still is not working. I believe this is all on Microsoft and feel helpless to do anything for the customer. A terrible feeling. I decide I’m going to search around and see if I can find even a glimmer of hope.
I go to check out the DirSync account I configured. Maybe I fat fingered something and that’s why it isn’t working. So, I take the following steps:
- Log into DC02 with domain admin credentials
- Navigate to C:\Program Files\Microsoft Azure AD Sync\UIShell
- Open up the miisclient.exe
- Click on the “Connectors” tab
- Right Click on the Windows Azure Active Directory connector and select Properties
I scroll down to the Connectivity tab and find something odd. The account I had used as the Cloud account for the tool was not there, but instead some other random one was.
I’m absolutely positive I did not set this account, nor did I even create it! I get my the Microsoft support engineer on the phone and ask him why the account is configured here.
Turns out that Azure AD Connect no longer uses the Azure AD account you specify when setting it up, but instead creates a brand new one for it. The problem here was that when it created the account, it did not give that account the correct permissions to correctly sync to Azure AD. Somehow, it had configured itself so that it could sync some, but they wouldnt work correctly with the other Office 365 services.
I updated the account information with the one I had configured previously and forced a full sync. To my relief, it worked. The support engineer I was working with was not aware of this, but confirmed later with the product team that the account creation aspect of it was intended, the permissions issue was not.
It seems like so many of the problems I run into are all because of permissions. So, the biggest take away I got from this particular incident is to not make assumptions.
I’m also a bit surprised that the Microsoft Support Engineer 1) did not know that the Azure AD Connect tool makes a new account in Azure AD for synchronization and 2) Didnt have me uninstall and reinstall the tool. I figured the problem was on the Office 365 side of things, so I didnt want to take that action. It was a headache that lasted way longer than it should have. Lessons learned, to assume makes an ASS out of U and ME.