Hey all!

First of all Happy New Year and all the best for 2023! Another year, another start of some blogging action. 2022 was extremely busy, hence less blogposts, but let’s see what 2023 has to offer…

For this first post of the year I quickly want to share how to easily test the client certificate which Jamf Pro is using to connect to IIS on the Jamf ADCS Connector server.

For this we’ll use Postman which you can find here, and awesome tool we can use for a variety of troubleshooting scenario’s. There is an Intel as well as an Apple Chip (M1/M2) version, and you may need to create a free account.

The reason we’d want to use this within the realm of troubleshooting the Jamf ADCS connector is because sometimes we need to rule out network related factors impacting the SSL flow and the authentication with the Client Certificate, as well as problems with the Client Certificate itself.

For more information on configuring the ADCS Connector, I’ll just refer to the official Jamf Documentation here, or one of my older blogpost here: https://travellingtechguy.blog/jamf-adcs-connector/. (There is also a troubleshooting section with some clues to hunt for when things go bad on the official doc)

Before we dive into testing the Client Cert we need to know however, or at least assume, that it is indeed the client cert which is causing the problems, and for that, there is only one way to find out! LOGS!

Which logs? Well, as always we start with the Jamf Pro logs, and one thing I want to clearly call out here is that fact that since a good set of JPRO versions, JPRO uses all Tomcat nodes to manage the certificate requests for the endpoints. This means that not only the primary Tomcat server is responsible for sending out the requests to the ADCS Connector! Especially in JamfCloud hosted instances, but as well as in on-prem clustered environments, the logs of all JPRO Tomcat nodes may need to be checked to find the error messages related to the failing ADCS certificate requests.

Now, depending the issue, the log message may be quite straight forward, and clearly state the nature of the issue:

CR_DISP_DENIED: Request denied => Wrong template or policy blocking the request.

PKIX path building failed => JPRO getting a wrong Server cert presented (not matching the one uploaded to the PKI settings, for instance by a Reverse Proxy in front of IIS)

ERROR_INVALID_PARAMETER => wrong parameter or variable configured in the request/profile

UnknownHostException => DNS issue

Connect timed out => Firewall issue

etc...

However, with Client Certificate related issue the error in the logs can be a bit more cryptic.

For instance because the connection is being reset:

Caused by: javax.net.ssl.SSLException: Connection reset

So, if the JPRO logs are not giving us a clear indication of the root cause, we move our focus to the IIS logs in c:\\inetpub\logs\LogFiles\W3SC1 or W3SC2 on the server hosting the ADCS Connector.

Taking the most recent log files we would then check the requests coming in from JPRO. For a successful request we would typically see a request and retrieve call with a status 200:

#Fields: date time s-ip cs-method cs-uri-stem cs-uri-query s-port cs-username c-ip cs(User-Agent) cs(Referer) sc-status sc-substatus sc-win32-status time-taken
2023-01-04 15:09:47 172.16.20.21 POST /api/v1/certificate/request - 443 AdcsProxyAccessUser 172.16.20.10 Java-SDK - 200 0 0 1281
2023-01-04 15:09:47 172.16.20.21 POST /api/v1/certificate/retrieve - 443 AdcsProxyAccessUser 172.16.20.10 Java-SDK - 200 0 0 31

However, before JPRO can actually connect to the ADCS Connector (and IIS), it must first succeed in doing a GET call to check the version of the ADCS Connector API. And this is typically where things already go south when there is a problem with the Client Certificate:

2021-01-16 22:49:53 10.0.1.3 GET /api/v1/version - 443 - 10.0.2.10 Java-SDK - 403 7 5 0

The main clue / confirmation that we do have a Client Certificate issue here is the 403.7 error, but let’s look at the full message and split it up into the various fields:

  • date: 2021-01-16
  • time: 22:49:53
  • s-ip: 10.0.1.3 (Server IP)
  • cs-method: GET
  • cs-uri-stem: /api/v1/version
  • cs-uri-query: –
  • s-port: 443
  • cs-username: –
  • c-ip: 10.0.2.10 (Client IP)
  • cs(User-Agent): Java-SDK
  • cs(Referer): –
  • sc-status: 403
  • sc-substatus: 7
  • sc-win32-status: 5
  • time-taken: 0

A few things to highlight here. The server and client IP, which allow you confirm and check some network related things. Note that the client IP can be the Firewall or Reverse proxy via which the request from JPRO is passing. But more important are the sc-status and sc-substatus, here 403 and 7.

And there we have it, 403.7, 403 meaning “Forbidden”, and .7 specifying it to “Client Certificate Required”.

Please note that 403 – Forbidden should not be confused with 401 – Access Denied, which has a total different meaning and list of sub-status options (most likely ACL’s, server policies, filters or other logon failures).

Last thing, before we finally look at how to use Postman, is that we also have the sc-win32-status which can be interesting to look at. In the example above, 403 7 5, the 5 stands for a win32 status of “Access is denied”. But wait, error 401 means “Access Denied” and I just said that 403 and 401 are totally different. Well, yes, by root cause they are, but when IIS is sending the final response on a client request, it will typically expect the client to acknowledge with a final ‘ACK’. When that happens the communication is done, and IIS will log a win-32 status accordingly. In the situation of a denied access, wether that is because of a 401 or a 403, this would be win32 status 5.

So an authorisation failure caused by a bad or missing Client Certificate, with a client responding correctly to the final message from IIS would result in 403 7 5. While we can also get 401 7 5 in scenarios where the root cause is not the client cert but any the 401 substatus reasons.

Within the realm of JPRO and the ADCS connector, we often also see 403 7 64 however, with 64 meaning “The specified network name is no longer available.” This could in many situations be pointing to a network issue, but in most cases it is just because the client (JPRO) is resetting the connection instead of gracefully sending the final ‘ACK’ to IIS. IIS is then left in state were it did not hear back from the disappointed client receiving the 403.7… and instead of logging win32 status 5, it logs it as 64.

So, a long story to elaborate a bit why we would be looking at testing the Client Cert, how to find the info we need in the logs, and what they mean. Now that we do know when we potentially have a Client Cert issue, let’s see what we can do to test it and isolate other impacting factors like Firewalls or other network issues.

What do we need?

  • the Client Certificate which was uploaded to JPRO
  • Postman
  • confirmation that IIS is configured to accept this client cert
  • inetpub logs to validate our test

Let’s start with getting the Client Certificate… yes, unless you used an alternative client cert, it’s the one the installer of the Jamf ADCS Connector created… and which you uploaded to JPRO. But you can’t download it from JPRO again, so how do we find it?

Well, let’s start with going to the JPRO settings -> PKI Certificates -> your ADCS config. Here you will find the details of the certs you uploaded to JPRO, and those of the Client Cert we need:

This gives us the Subject, Expiration Date and Serial Number… which should allow us to find the cert in the MMC tool of our Windows Server:

Subject matches… Expiry Date matches… but are we 100% sure this is the cert JPRO is using? NO, we need to check the serial number of the cert as well!

Oh… but this looks different compared to what we have in JPRO…

JPRO is stating the serial number is: “29230111967739416702170305393745566456“ while Windows is stating “15fd83c962a4469c44d65e6f14c19af8“, so this is not our cert?

Yes it is! JPRO is using the decimal syntax, while Windows is using hexadecimal, so with a simple hex to decimal conversion we can confirm this is indeed the cert we need, and all is good here!

Cool, so now we know which cert JPRO is using as client cert, and we can export it from MMC:

Yes, we export it with the private key and save it as a pfx file:

Now the part with Postman is really simple. The only thing we need to do is to configure Postman to use a Client Cert authentication for every connection to the base FQDN of our ADCS Connector and do a GET call to the /api/v1/version endpoint of our connector, for instance: “https://adcs.travellingtechguy.dev/api/v1/version”.

To configure the Client Cert Auth we go to the settings -> Certificates where we can add our client cert:

For the host we need to put the base FQDN of our ADCS Connector, so for me adcs.travellingtechguy.dev (without the /api/v1/version), upload the PFX we exported from MMC and type the export password you have set on the pfx file:

The the only thing you need to do next is a GET call to the API endpoint “https://youradcs.domain.com/api/v1/version” to see if it accepts the Client Cert Auth:

If all works fine, depending the version of your ADCS Connector, you should get a return of “1” or “1”,”2″ !

This means that you can successfully authenticate with the client cert to your ADCS connector!

And to be 100% sure we can then check the inetpub logs and trace our test, which should have a log entry like below, showing our Postman request with a 200 status (OK):

2023-01-04 23:26:06 172.16.20.21 GET /api/v1/version - 443 AdcsProxyAccessUser 192.168.20.45 PostmanRuntime/7.29.2 - 200 0 0 468

Now, in case the Client Cert auth does not work, Postman will tell you:

Which the inetpub log will also tell us:

2023-01-04 23:31:25 172.16.20.21 GET /api/v1/version - 443 - 192.168.20.45 PostmanRuntime/7.29.2 - 403 7 5 15

That’s it! Really simple!

Now what are the possible outcomes, and what can we learn from this test? Well, let’s look at the following possible combinations (assuming you correctly configured Postman, and you checked the Client Cert as per above).

NOTE: Important to mention is that we do the Postman test from WITHIN the internal network, behind the Firewall, with simple routing path to our ADCS Connector. Remember that the goal of this test is to rule out network related factors like Firewalls, ReverseProxies... etc.
JAMF PROPOSTMANConclusion
403.7403.7Client cert is NOT OK, or there is an IIS configuration problem
403.7200Client cert and IIS are OK, but something is breaking the Client Cert auth BETWEEN JPRO and the ADCS Connector

If you also get a 403.7 on IIS via Postman, and you matched the Client Cert between JPRO and MMC as per above, you may want to have a look at the authentication details for the AdcsProxyAccessUser account via the Configuration Editor. This to see if the base64 of the Client Cert is matching the one of the Client Cert you are using (the one you exported from MMC in pfx, you can export it again without private key and open it in a text editor to match the base64 with what you have for the AdcsProxyAccessUser). Just like you would do when using an alternative Client Certificate.

If you end up concluding that something in between JPRO and the ADCS connector seems to be breaking the Client Cert authentication, then it’s time to scramble your network and windows team together for some advanced checks on your network and server configuration.

A very very common root cause is that the Firewall is intercepting the SSL connection, offloading or bridging it without resending the Client Cert to the ADCS Connector again, compared to real untouched passthrough of the communication.

Note: if you do end up with successfully authenticating with the client cert via Postman, all network related factors have been ruled out, and JPRO still triggers 403.7, it could be that the Tomcat servers need a restart. This because of PI103922 - Cached ADCS settings cannot be cleared in clustered environments. This happens when you make changes to the PKI settings in JPRO, like uploading a new client cert, and the cert settings do not sync correctly over the different nodes in a clustered instance like JamfCloud.

One very last additional thing, to really confirm that you really isolated the issue to a Client Cert problem, you can also quickly test with changing the SSL settings in IIS to either ‘Ignore’ or ‘Accept’, instead of ‘Require’ for the Client Cert. Wether you leave that as a temporary workaround, when in urgent need of deploying certificates for your enduser, or not, is something I leave as a decision for your secure team. The fact that your Firewall is locked down to the IP’s of your JPRO server may or may not satisfy them to allow this temporary workaround in case of emergency situations. I disables the hard requirement for a client cert validation, but do NOT disable SSL entirely:

I hope this helps isolating one of the main root causes for the ADCS Connector integration not working as expected. I may elaborate some other ADCS Connector related troubleshooting in future post, but this may also be the first of more “using Postman to…” topics and troubleshooting scenarios.

That’s it! As always, if you liked the post, hit the like button, tell your friends about it and leave a comment down below!

Brgds,
TTG