skip to Main Content

VMware vSphere & Microsoft LDAP Channel Binding & Signing (ADV190023)

VMware VSphere & Microsoft LDAP Channel Binding & Signing (ADV190023)

Microsoft has recently released warnings to its customer base that, in the March 2020 updates to Windows, it intends to change the default behavior of the Microsoft LDAP servers that are part of an Active Directory deployment. These changes will make secure LDAP channel binding and LDAP signing a default requirement when accessing Microsoft Active Directory using LDAP or LDAPS. These changes are a response to a security concern documented in CVE-2017-8563, where bad actors can elevate their privileges when Windows falls back to NTLM authentication protocols.

Any system that connects to Active Directory via LDAP without using TLS will be negatively affected by this change. This includes VMware vSphere. If you are using unencrypted LDAP (ldap://, not ldaps://) to connect vSphere to Microsoft Active Directory please read further.

All supported versions of VMware vSphere have been verified by VMware Engineering to work as expected after these changes, where we expect unencrypted LDAP authentication to succeed with the old defaults, fail with the new defaults, and succeed when using TLS/LDAPS.

Important Notes

These issues are the direct result of Microsoft’s changes to Windows. While we at VMware are committed to helping our customers navigate issues like these, we do ask that you please direct feedback about Windows changes & updates to Microsoft themselves. Configuring authentication sources in vSphere is a documented & supported activity that the professionals at VMware Global Support Services (GSS) can assist with, but that comes with the prerequisite that your authentication source is usable and cooperative. Please contact Microsoft Support for assistance with reconfiguring Windows Active Directory.

These issues affect any system using LDAP to access Active Directory and are not limited to VMware vSphere. You need to check the rest of your systems, too. Microsoft provides ways to audit clear-text connections to your Active Directory infrastructure which could be helpful.

All screen shots in this post are of vSphere 6.7 Update 3 and the HTML5 vSphere Client. The experience should be similar for vSphere 6.0 and 6.5.

Who Is Affected?

If you have configured vCenter Server to access Active Directory over LDAP with TLS (LDAPS) you will not be affected by this. You can check this by viewing your Identity Sources in the vSphere Client:

The red & green text has been added by us as an illustration. Sources using LDAP (ldap://, port 389) are likely to be affected. Sources using LDAPS (ldaps://, port 636) are likely fine, though we always encourage testing and verification.

What Will Happen

Without proactive action on your part, once Microsoft releases these updates and they are applied the affected identity sources will stop working. This will appear as login failures in the vSphere Client:

vSphere Login - Invalid Credentials

as well as errors when attempting to add or reconfigure an identity source:

vSphere Add Identity Source Error

If you have enabled auditing for clear-text binds in Active Directory you will also see event logs that correspond:

Microsoft LDAP 2889 Auditing Event Viewer

Notes & Suggestions on How to Test

We recommend testing in order to gain familiarity with these updates. Microsoft has released guidance on how to enable these settings:

The document on enabling LDAP Signing in Windows Server 2008 indicates that you need to change the “Default Domain Policy” but in order for it to be effective for domain controllers you must also edit the “Default Domain Controllers Policy” or whichever policy applies to the domain controllers, if you’ve assigned a new one. The procedure in that document also appears to be applicable to all versions of Windows, not just 2008. Once the Group Policy edits are in place you can wait until the Group Policy refreshes automatically or use an Administrator-level shell to issue the “gpupdate /force” command.

Keep in mind that updating Group Policy affects other domain controllers & members as well. vCenter Server can configure multiple LDAP & LDAPS authentication sources, and can specify particular domain controllers, so we recommend creating a new & isolated Active Directory instance for testing (you can see my farm animal theme for test domains & forests in this post, which is separate from my main authentication source). If you must test in production consider doing so with a Group Policy Object assigned specifically to a particular domain controller and take great care.

To test that the settings have taken effect use the “ldp.exe” utility (Start->Run->ldp) from the domain controller itself. From the Connection menu, choose Connect, and enter “localhost” and port 389:

Connecting with ldp.exe

From there, go back to the Connection menu and choose “Bind.” Enter your domain credentials and select “Simple bind” as shown here:

Binding with ldp.exe

You should receive the error “Error 0x2028 A more secure authentication method is required for this server” as shown below, which indicates that you have correctly disabled clear-text LDAP binds.

A more Secure Authentication Method is Required Error

If you do not get this error with this procedure the changes have not taken effect!

From here you can test vCenter Server connectivity to Active Directory, witnessing the behavior seen at the beginning of this post. As mentioned earlier, VMware vCenter Server can have multiple Active Directory instances configured, so testing with an isolated instance of Active Directory is recommended. Similarly, deploying a test vCenter Server appliance is recommended. Take a snapshot of the environment and you can restore it if everything goes wrong. Nested virtualization environments, in general, are an excellent way to test functional changes such as this. William Lam has extensive resources on his personal blog for nested ESXi.

For even more testing isolation, if you deploy the Windows Active Directory VM first and configure AD DNS you can use it as the DNS server for a test deployment of vCenter Server.

Possible Course of Action #1: Enable TLS on Active Directory (LDAPS)

Being security-minded, the first & best recommendation is to secure your authentication with TLS. As a matter of practice, all communications on a network should be encrypted. This is especially true of authentication traffic. Please refer to Microsoft documentation for configuring Active Directory Certificate Services and issuing certificates to Active Directory domain controller services.

Configuring vCenter Server to use LDAPS is straightforward and well-documented at docs.vmware.com. There is one twist: you will need the certificate for the domain controller. You can export it from Windows but if you have access to OpenSSL, either installed on a Windows PC or built into a Linux/UNIX host, this sample command is often easier:

echo | openssl s_client -showcerts -connect cows-ad-a1.cows.local:636

will display the certificate you need, between the “—–BEGIN CERTIFICATE—–“ and “—–END CERTIFICATE—–” lines. Copy those lines and everything between them into a text file and use that when prompted. Replace my farm animal test FQDNs with the native DNS name of the domain controller you are connecting to, of course!

Possible Course of Action #2: Proactively Configure These Settings to The Original Defaults

Being security-minded, making a decision that could negatively affect security can be tough. However, there’s a lot more to information security than just changing registry settings. Information security professionals use the “CIA triad” — confidentiality, integrity, and availability – to thoughtfully weigh the risk of a configuration, and the short timeframes and nature of these changes could seriously impact availability. Additionally, if you’ve already been running in this configuration you likely have compensating controls in place, such as isolation (firewalls, separate networks), to protect against someone observing authentication traffic.

Microsoft has stated that these upcoming patches will change the defaults. Though they haven’t said it directly, this implies that you can proactively set the those settings to the current defaults, using these same processes (set the LdapEnforceChannelSigning registry key to 0, set the group policies to ‘None’). This assumes that Microsoft will not overwrite them with the patch, which is probably reasonable, but we will all want to verify that once the patches are available.

Should you do this? It might be the easiest way forward for now, but it doesn’t improve security, and is likely worth a conversation with your CISO.

Other Considerations

It isn’t a good idea to delay patching. Patching is the ONLY way to remove a vulnerability from a system, and it’s the #1 way organizations and individuals can secure themselves (the #2 way is great passwords & account hygiene). Microsoft is making this big change for a reason, and we don’t yet know what has changed since the original 2017 vulnerability disclosure. It’s very likely that, in a few months, we will learn more about what is driving this. That won’t be good news, whatever it is. By delaying or omitting patches you delay the inevitable and you increase your risk, both from hackers and from well-intentioned humans accidentally applying cumulative updates.

Whatever course of action you choose, this is a great opportunity to make sure that you know and/or can retrieve the password for your vCenter Server SSO instance’s administrator account (often administrator@vsphere.local).

This issue affects any system using LDAP to access Active Directory, not just vSphere. Enabling auditing for the clear-text binds on your Active Directory is a good way to find other systems that may be in this situation, in order to include them in a larger remediation plan. Be mindful of the additional event logging that this might generate.

Active Directory is wonderful in that it’s a multi-master, replicating, distributed database, with built-in availability and recoverability features. Use this to your advantage and plan a staged rollout for both this issue and as a future patching strategy. Similarly, do not forget the flexibility vCenter Server has in targeting specific domain controllers, as well as the ability to apply Group Policies to specific domain controllers.

Thank You

As always, thank you for being a VMware customer. We appreciate you and hope that, like with our continued guidance on CPU vulnerabilities, this has been helpful in navigating these larger industry-wide changes.

For questions and feedback about Active Directory, Windows Updates, and these Microsoft Updates please contact Microsoft directly. We love feedback but if it’s about Active Directory and Windows the best we will be able to do is commiserate with you. Feedback is most powerful when it’s direct from a user to the responsible vendor.

For operational issues with VMware products please contact VMware Global Support Services, who are available 24×7 to assist. If there are other things we can help with, or you have questions or feedback about VMware products, please reach out to your account team or Technical Account Manager. For issues directly concerning this post please leave a comment below.

Last thing – subscribe to this blog and follow us on Twitter or on Facebook! We write a lot of technically-oriented articles like this, covering news and current topics, and we’d love to have you with us.

Thank you!



** This post was originally published on https://blogs.vmware.com/vsphere/2020/01/microsoft-ldap-vsphere-channel-binding-signing-adv190023.html **

 

×Close search
Search