We have an ASP.NET application running on IIS on Windows Server 2016, hosted on a D48s Azure virtual machine (48 cores).
Most of the time, the web app is processing requests from regular users at a pace of 200-300 requests per seconds.
But at times, the server receives high incoming traffic as webhooks from external sites – up to 1000-5000 requests per second.
When this happens, the CPU usage get really high, and we noticed that the Local Security Authority Process (lsass.exe) is consuming most of the CPU at that time: it can take up to 40-50% of the total CPU usage (the orange graph is lsass.exe CPU usage):
Needless to say, the server becomes really busy, and other requests start to slow down.
By the way, each webhook request is very lightweight and adds a record to a table in SQL Server, for later processing.
We made a load test on our server, and found out that lsass becomes so active only when requests are made using https. However, if the very same requests are made using http, lsass.exe is not active at all.
The second discovery was that when using http the server was able to process 4-5 times more requests under the same load, compared to https!
Here is a screenshot from Performance Monitor: the green line show Requests per seconds, and the brown line shows lsass.exe CPU usage. On the left is what happens when using http, and on the right – when using https:
So the bottom line is that:
https makes the requests 4-5 times slower.
When using https, lsass.exe starts to eat a lot of CPU resources.
Questions are:
- Why is lsass.exe so active during https requests?
I found an article on the web saying that lsass.exe was used by IIS 6.0 to cipher / decipher https traffic, but that starting from IIS 7.0 it is no longer used. However, our experiments indicate the contrary.
I don't understand how 4000-5000 requests per seconds having a body of 3-5 kilobytes can make the the CPU of a 48-core server so busy.
Maybe there are some hidden SSL settings in IIS that can make https more efficient?
We found info on SSL offloading: can this be done on Azure?
Update
We created a new Azure VM from scratch (a 32-core D32s), installed IIS and created a simple ASP.NET Web Forms app that has only 1 "Hello World" aspx page that does nothing (no SQL Server requests etc.)
With JMeter we created a load test to this page, and the same pattern appeared here:
1) http: the server was processing 20 000 requests per second, and lsass.exe was not active.
2) https: the server was processing only 1000-1500 requests per seconds, and lsass.exe was consuming 10% of total CPU.
Here is the Performance Minitor graph (http on the left, https on the right):
By the way, JMeter and the ASP.NET web app were run from the same VM, so network round-trips were minimal.
How can https be 15-20 times slower than http in this simple situation?
And what is the role of lsass.exe in this situation?