SteelHead is unable to join/rejoin the domain in Active Directory Integrated Mode (Windows 2008 and later) or unexpected SMB2/3 problems occur after deploying Microsoft patches from January 11, 2022 and later

Categories:
SteelHead (Appliance)
Solution Number:
S35726
Last Modified:
2023-06-21
Issue
Observed problems:
1. SteelHead messages log display the following error while joining or rejoining to the domain:  Failed to join domain using ads: failed to verify domain membership after joining: The object name is not found.
 


2. When SteelHead is joined to the domain in Active Directory Integrated mode (Win2008 and later), it may report  NT_STATUS_OBJECT_NAME_NOT_FOUND after Microsoft Security Updates are applied to the Domain Controller (s), and new SMB 2/3 protocol sessions with UNC hardening are intercepted for signing and mutual authentication during the optimization process.

NTLM Auth Failed for user: *\SH_AUTH_MON. NT status string: NT_STATUS_OBJECT_NAME_NOT_FOUND Code: 3221225524 message: The object name is not found.
 
 
NOTE:  The SteelHead appliance is going to check every 30 seconds with the Domain Controller(s) using a non-existent user SH_AUTH_MON when it cannot communicate with the Domain.


3. Kerberos replication test failed with NT_STATUS_ACCOUNT_DISABLED.
 
[Jan 14 22:30:49 66688 -1 domain_auth/repl_test ERROR] {- -} nt status: NT_STATUS_ACCOUNT_DISABLED, vampire status: 8
Error message - Can't contact the DRSUAPI pipe. NT_STATUS_ACCOUNT_DISABLED

 
4. The SteelHead machine name object is 'Disabled' in Active Directory Users and Computers (ADUC).
 

 
NOTE:  When the SteelHead object is 'Disabled', the UserAccountControl bit set is ACCOUNT_DISABLE (0x1002).


5.  The Domain Controller Event Viewer starts to display: "Event ID 5722 - The system cannot find the file specified" for System NETLOGON service under Administrative Events.


 
 
Solution
Microsoft started enforcing msds-KrbTgtLink validation with the release of January 2022 Security Updates for NTLM authentication when the SteelHead is joined to the domain in Active Directory Integrated Mode (Windows 2008 and later) with the userAccountControl attribute 0x5011000 (RODC).   Microsoft announced these improvements and fixes are going to be part of Security Updates going forward and SteelHead RODC machine accounts might fail to establish a Netlogon secure channel.
 

enlightened It is recommended to evaluate the installation of the following Windows security updates to the DC's used by SteelHeads during the domain join or while performing SMB2/3 optimization.

yes  You can subscribe to this article to receive updates via email regarding any changes to the content.


Microsoft Security Updates affecting SteelHead devices joined or rejoined to the domain are:
 
Domain Controller OS version  January 2022 Updates Out-of-band
Windows Server 2012 KB5009586 (Monthly Rollup) KB5010797
Windows Server 2012 R2 KB5009595 (Security Update)
KB5009624 (Monthly Rollup)		 KB5010794
Windows Server 2016 KB5009546 (Security Update) KB5010790
Windows Server 2019 KB5009557 (Security Update) KB5010791
Windows Server 2022 KB5009555 (Security Update) KB5010796
 
For example, February 2022 Cummulative Updates contains all updates deployed from January 2022 table above. Similar process would continue to occur with security releases on monthly basis.
 
Domain Controller OS version  February 2022 Cummulative Updates
Windows Server 2008 Service Pack 2 KB5010384
This security update includes improvements and fixes that were a part of update KB5009627 (released January 11, 2022) and update KB5010799 (released January 17, 2022).
Windows Server 2012 KB5010392
This security update includes improvements and fixes that were a part of update KB5009586 (released January 11, 2022) and update KB5010797 (released January 17, 2022).
Windows Server 2012 R2 KB5010419
This security update includes improvements and fixes that were a part of updates: KB5009624  (released January 11, 2022) and update KB5010794 (released January 17, 2022).
Windows Server 2016 KB5010359
This security update includes improvements and fixes that were a part of update KB5009546 (released January 11, 2022) and update KB5010790 (released January 17, 2022).
Windows Server 2019 KB5010351
This security update includes improvements and fixes that were a part of update KB5009557 (released January 11, 2022) and update KB5010791 (released January 17, 2022).
Windows Server 2022 KB5010354
This security update includes improvements and fixes that were a part of update KB5009555 (released January 11, 2022) and update KB5010796 (released January 17, 2022).
 

What are the steps to detect if the connected DC used by SteelHead appliance is affected by Microsoft security updates?

1. From the DC used by the SteelHead to join or to communicate, validate Microsoft 2022-01 Security Monthly Security update rollup and/or the Out-of-band patch and/or Cumulative Updates were installed running the following PowerShell commands to extract the OS version and installed updates:

PS C:\> systeminfo | findstr OS
PS C:\> Get-HotFix
 
 

2. Go to Microsoft Update Catalog portal and search for installed KB patch ID.  For example, searching for KB5011503 for Windows Server 2019 will include January 2022 security updates: KB5009557 and KB5010791 under "Package Details" tab.
 
 
enlightenedNOTE:  Microsoft is going to include these security updates as part of their monthly patch Tuesday cumulative updates for Windows Server (s) with a new security patch's ID number(s).
 


SOLUTION:
 
enlightenedIMPORTANT:
• Active Directory Integrated Mode (2008 and later) is deprecated in RiOS 9.12.2a and later. For additional information refer to KB article S35945.
• Any updates released January 11, 2022 and later on your domain controllers, might fail to establish a Netlogon secure channel with SteelHead appliance joined the domain in Active Directory Integrated Mode (Windows 2008 and later).
• Deploying WinSec Controller (WSC) physical or virtual appliances are not affected by Microsoft 2022-01 security updates because this platform doesn't required to be joined the domain and meets Microsoft's tiered security model to optimize SMB protocol using secure negotiation. The WinSec Controller is a completely dedicated, non-network appliance that interacts with the domain controller as a Tier 0 entity. For more information, reach out to your Riverbed representative and visit riverbed.com to learn more.


Riverbed is recommending to perform a non-intrusive domain rejoin with Join Accoun Type: (1) WORKSTATION or (2) BDC modes (aka Active Directory Integrated Mode - Windows 2003).

yes Refer to KB article S35945 with the new joining modes available in RiOS 9.12.2a and later: (1) Kerberos Authentication (Workstation) or (2) NTLM authentication (BDC)

 
1. Change the joining mode to WORKSTATION.
This mode can be used with End to End Kerberos (eeKRB) environments where NTLM authentication is not required. See KB article S14342 containing Riverbed Best Practices for Domain Communication.
 
i. Proceed to modify userAccountControl attribute for the SteelHead machine account from Active Directory Integrated Mode (Windows 2008 and later) - RODC 0x5011000 to WORKSTATION trusted machine attribute 0x11000 from a domain controller within the domain.
 
PS C:\> Set-ADComputer -Identity "<Steelhead_Hostname>" -Replace @{UserAccountControl=0x11000}
 

 
ii. Rejoin the SteelHead appliance to the domain with Join Account Type: Workstation
 
  • From UI Console, go to OPTIMIZATION > Active Directory > Domain Join: Domain Settings

 
  • From CLI Console configuration mode run:
domain rejoin login <domain_admin> password ********* join-type workstation
 
 
enlightenedIMPORTANT:   
• Kerberos service account is recommended to support Mutual Authentication and Integrity (UNC Hardening). In WORKSTATION mode it will join as a regular machine object with the userAccountControl attribute WORKSTATION_TRUST_ACCOUNT (0x11000).  In Workstation mode the appliance is going to proxy Kerberos TGS request (KRB_TGS_REQ) on behalf of a client up to a Domain Controller (DC) to perform mutual authentication and integrity. Since the SteelHead integrated with Active Directory does not advertise itself or provide domain functions, the actual changes are few. The machine account is placed in the Computers organizational unit and NO DNS SRV records are created.

• If NTLM-based traffic exists on their network, below commands need to be configured in the SteelHead to bypass it. These commands are recommended from the server-side SteelHead and it requires a service restart.
 
protocol smb2 signing ntlm-bypass enable                 */Enables the pass through of NTLM connections during SMB2 signing.
protocol mapi encrypted ntlm-bypass enable               */Enables the bypass of NTLM-authenticated encrypted MAPI traffic.
no protocol smb2 signing native-krb downgrade enable     */You must enable SMB2 and join the domain before enabling SMB2 signing. This command disables an SMB2 signing downgrade after an end-to-end Kerberos failure. 

• Any connection using NTLM authentication will go in bypass mode from the application layer and marked with a red-triangle from current connection report as an indication latency optimization was not performed.
• Multiple SteelHead devices can be rejoin in WORKSTATION mode using the SteelCentral Controller for SteelHead. Refer to KB article S35809 for additional information.
 
or
 
This mode is recommended for single domain with the privilege to do NTLM pass-through authentication in transparent mode. NTLM pass-through authentication may not work during cross-domain authentication for all environments. See KB article S14342 containing Riverbed Best Practices for Domain Communication.

i. Proceed to modify userAccountControl attribute for the SteelHead machine account from Active Directory Integrated Mode (Windows 2008 and later) - RODC 0x5011000 to BDC trusted server attribute 0x12000 from a domain controller within the domain.
 
PS C:\> Set-ADComputer -Identity "<Steelhead_Hostname>" -Replace @{UserAccountControl=0x12000}

 
ii. Rejoin the SteelHead appliance to the domain with Join Account Type: Active Directory Integrated Mode (Windows 2003). This mode is also known as win2k3-mode or BDC mode.
 
 
From UI Console, go to OPTIMIZATION > Active Directory > Domain Join: Domain Settings
 
or
 
From CLI Console configuration mode run:
domain rejoin login <domain_admin> password ********* join-type win2k3-mode
 
 
enlightenedIMPORTANT:
• Kerberos service account is recommended to support Mutual Authentication and Integrity (UNC Hardening). When the SteelHead object is created in Active Directory the userAccountControl attribute is SERVER_TRUST_ACCOUNT (0x12000).

• In Active Directory Integrated mode (Windows 2003) the appliance is going to proxy Kerberos TGS request (KRB_TGS_REQ) on behalf of a client up to a Domain Controller (DC) to perform mutual authentication and integrity. Since the SteelHead integrated with Active Directory does not advertise itself or provide domain functions, the actual changes are few. The machine account is placed in the Computers organizational unit and NO DNS SRV records are created.

• Multiple SteelHead devices can be rejoin in BDC mode using the SteelCentral Controller for SteelHead. Refer to KB article S35815 for additional information.

• In BDC mode the SteelHead machine role appears as "Writable Domain Controller" from the search results within the "Find Computers" option under Active Directory Users and Computers (ADUC) but RiOS only uses a subset of BDC attributes without any of the existing Domain Controller (DC) roles from Active Directory Domain Services (AD DS). Refer to KB article S35942 for details.
 


WORKAROUND:
SteelHead latency optimization from the client-side SteelHead can be disabled as a temporary workaround until either of the proposed solutions above are implemented.
Environment
SteelHead - All Models
Active Directory
Attachments
NOTICE: Riverbed® product names have changed. Please refer to the Product List for a complete list of product names.
Can't find an answer? Create a case