Workflow Manager Configuration Error

I recently attempted to configure Workflow Manager 1.0 and Service Bus 1.0 for use by SharePoint 2016, using a certificate issued by our domain CA instead of self-generated certificates. I ran into the following error though.

System.Management.Automation.CmdletInvocationException: Could not successfully send message to scope ‘/WF_Management’ despite multiple retires over a timespan of 00:02:07.8300000.. The exception of the last retry is: A recoverable error occurred while interacting with Service Bus. Recreate the communication objects and retry the operation. For more details, see the inner exception..  —> System.TimeoutException: Could not successfully send message to scope ‘/WF_Management’ despite multiple retries over a timespan of 00:02:07.8300000.. The exception of the last retry is: A recoverable error occurred while interacting with Service Bus. Recreate the communication objects and retry the operation. For more details, see the inner exception..  —> System.OperationCanceledException: A recoverable error occurred while interacting with Service Bus. Recreate the communication objects and retry the operation. For more details, see the inner exception. —> Microsoft.ServiceBus.Messaging.MessagingCommunicationException: Identity check failed for outgoing message. The expected DNS identity of the remote endpoint was ‘WFMServer1.contoso.com’ but the remote endpoint provided DNS claim ‘WFMServer3.contoso.com’. If this is a legitimate remote endpoint, you can fix the problem by explicitly specifying DNS identity ‘WFMServer3.contoso.com’ as the Identity property of EndpointAddress when creating channel proxy.  —> System.ServiceModel.Security.MessageSecurityException: Identity check failed for outgoing message. The expected DNS identity of the remote endpoint was ‘WFMServer1.contoso.com’ but the remote endpoint provided DNS claim ‘WFMServer3.contoso.com’. If this is a legitimate remote endpoint, you can fix the problem by explicitly specifying DNS identity ‘WFMServer3.contoso.com’ as the Identity property of EndpointAddress when creating channel proxy.

To cut a long troubleshooting story short, the problem was with the certificate I had requested from the CA. More specifically the DNS extension. I had the following DNS entries in the certificate.

  1. *.contoso.com
  2. WFMServer1.contoso.com
  3. WFMServer2.contoso.com
  4. WFMServer3.contoso.com

For what ever reason, WFM only seems to look at the final DNS entry when trying to add the host to the WFM farm. To confirm this, I tried the installation from all three hosts and it worked fine on WFMServer3.contoso.com, but not the other 2.

I’m still not entirely sure if it was WFM or SB that was causing this issue, but I fixed it by simply revoking the certificate on our CA and re-installing SB and WFM using a certificate with wfm.contoso.com as the Common Name and DNS entries in the following order:

  1. WFMServer1.contoso.com
  2. WFMServer2.contoso.com
  3. WFMServer3.contoso.com
  4. *.contoso.com

 

Making SharePoint Trust Enterprise CAs

If you have an enterprise PKI, or even just a single CA, that issues certificates to services such as ADFS or Workflow Manager that you intend to use with your SharePoint Farm, it might be worth while importing you Root and Intermediate CA Certificates into SharePoint to make it trust the other services.

Currently we have a PKI setup similar to the following:

-Enterprise Root CA (Offline)
—Enterprise Intermediate CA (Offline)
——Web Services CA (Online)
——Authentication CA (Online)

The Root and the Intermediate CA are offline and secured except for planned maintenance, such as publishing new certificate revocation lists. The Web Services CA is the CA that issues the certificates we use for web services like Workflow Manager, and the Authentication CA issues the ADFS signing Certificate, which also needs to be trusted by SharePoint.

First of all, you need to grab a copy of the certificate of each CA in the Chain. If you have your PKI configured correctly, they should be readily available so I won’t go into that process. If you are exporting them from the CA servers, DO NOT export private keys, SharePoint doesn’t need them and that would be a major security concern. Once you have them, copy them to a directory on your SharePoint server(s).

Then, make sure the certificates are in the trusted root certificate authorities on every server in your farm. the easiest was to do this is to publish the certificates using group policy, but you can do this manually if you like. Simply follow these steps on each server in the farm:

  1. Double click the first certificate.
    cert_overview.jpeg
  2. Click Install.
  3. Select “Local Machine” and click Next.
  4. Click Yes on AUC prompt.
  5. Select “Place all certificates in the following store”.
  6. Click Browse and select “Trusted Root Certificate Authorities” or “Intermediate Certificate Authorities” depending on if the certificate is at the root of the chain.
  7. Click Next.
  8. Click Finish.
  9. Repeat for each certificate in the chain.

On your SharePoint server execute the following powershell commands for each certificate, changing “C:\RootCA.cer” to the path of each certificate, and “Enterprise Root CA” to a friendly name for each CA.


$path = “C:\RootCA.cer”
$root = New-Object System.Security.Cryptography.X509Certificates.X509Certificate2($path)
New-SPTrustedRootAuthority -Name "Enterprise Root CA" -Certificate $cert

If you don’t like powershell or are unable to access powershell for whatever reason, you can also install the certificates from Central Admin.:

  1. Navigate to Central Admin -> Security -> Manage Trust.
  2. Click “New”.sharepoint_ca_add_trust.jpeg
  3. Add a friendly Name for the certificate.
  4. Click Browse and select your certificate.
  5. Click OK.
  6. Repeat for each certificate in your PKI chain.

 

SharePoint 2013 Product Configuration Wizard hangs on step 9 of 10

I was patching a SharePoint 2013 farm recently and all was going great while installing the CU’s on the servers. Once all of the patches were installed, I started to run the SharePoint 2013 Products Configuration Wizard on the first WFE server, which went without an issue. The second WFE server seemed to be running fine too, getting to step 9 out of 10. It was on this step for about an hour when I went home for the evening (this is a test environment). The next morning, it was still at the same stage.

Initially I reboot the server and tried again, but got the same results. To remedy this issue, I performed the following actions.

  1. Open services.msc
  2. Stop the SharePoint Timer Service
  3. Browse to C:ProgramDataMicrosoftSharePointConfig
  4. Open the folder with the newest date modified on it.
  5. Delete all files from the folder except for cache.ini
  6. Open cache.ini and change the number within to any random number ensuring it has the same number of digits as the original. I went with 12345.
  7. Run the following command from an elevated command prompt – Psconfig.exe -cmd upgrade -inplace b2b -wait -force

After that, the server was updated and ready to serve content again.

Fixing Office Web Apps 2013 Server Error

After a disaster at work, involving our SAN shitting itself and falling over, we invoked our DR procedures and got services back online. While testing the functionality of our SharePoint 2013 farm, which had a few little issues that were resolved in an hour, I noticed that Office Web Apps 2013 were not working, Giving an nondescript “Server Error”.

webapps-errorI logged onto the Web Apps Server box and ran the Get-OfficeWebAppsFarm powershell command an got the following.

webapps-powershell-error-1

 

Notice the WARNING: It does not appear that this machine is part of an Office Web Apps Server farm message at the top? The Server was also missing from the “Machines” property of the farm.

Since this was a single Office Web Apps server and not an actual farm, the easiest solution was to recreate the farm with the following command.

$farm = Get-OfficeWebAppsFarm
New-OfficeWebAppsFarm -InternalURL $farm.InternalURL -ExternalURL $farm.ExternalURL -CertificateName $farm.CertificateName -EditingEnabled

This command grabs all necessary properties from the original farm. It also assumes you have a volume license for office and wish to enable editing. If you don’t remove -EditingEnabled from the end or suffer Microsoft’s wrath.

After I executed this command the machine was listed in the farm properties and Web Apps were working once again.

 

webapps-powershell-fixed-1

 

SharePoint 2013 webapps don’t work on some sites

I ran in to an issue today that had my head battered for a good half an hour before turning to Google.

The issue involved opening documents, which happened to be in my SkyDrive Pro library, with Office Web Apps. The apps threw different errors, listed below;

Word Document : “Sorry, there was a problem and we can’t open this document. If this happens again, try opening the document in Microsoft Word.”
Excel Document: “We couldn’t find the file you wanted. It’s possible the file was renamed, moved or deleted”
PowerPoint: “Sorry, we ran into a problem. Please try again”
One Note:”Sorry, you don’t have permission to edit this notebook.”

After an hour of Googling and trawling through false positive results (thanks to Microsoft’s genius idea of keeping the “SkyDrive Pro” brand the same in hosted and on-premises SharePoint farms) I found a blog post (linked below) that shone some light on my issue. It also made me realize that the issue wasn’t local to SkyDrive Pro as well.

I turns out that the issue was down to the fact that I provisioned Web Apps using SSL, and the SkyDrive Pro library wasn’t using SSL. This caused OAuth not to function on the http site. Web Apps, being https, didn’t much like this and shit itself. Oh well.

The resolution is either to enable SSL on all SharePoint sites that Web Apps should be accesible in, or simply allow OAuth over http (detailed below.)

Run the following PowerShell script in SharePoint Management Shell on your SharePoint box(es.)

$config = (Get-SPSecurityTokenServiceConfig)
$config.AllowOAuthOverHttp = $true
$config.Update()

Credit for this solution goes to ms_dynamics.

References: Office Web Apps 2013 and SharePoint Issue relating to SSL