I have migrated an application from NetCore 3.1 and Ubuntu 18.04 to Net5 and Ubuntu 20.04.
The application has begun failing when a HttpWebRequest object calls the GetResponse method.
I have found a notice from Microsoft warning about major changes in the behaviour, see
And a very interesting post (generic for Linux, not related with Net5) in https://medium.com/siberians-pro/how-to-use-tlsv1-on-ubuntu-20-04-133c2898ad7
But both are not working for me.
What I have tried until now:
- I have created a specific local openssl.cnf file, with this content
openssl_conf = openssl_init
[openssl_init]
ssl_conf = ssl_sect
[ssl_sect]
system_default = system_default_sect
[system_default_sect]
MinProtocol = TLSv1
CipherString = DEFAULT@SECLEVEL=1
- I have exported the variable
OPENSSL_CONFpointing this file
export OPENSSL_CONF=/var/www/spider/openssl.cnf
- I launch my code normally
dotnet Spider.dll
- And I get the exception
fail: Spider.Program[0]
[19/12/2020 08:50:52.724]: Error getting response from https://www.boe.es/diario_boe/xml.php?id=BOE-S-20201216
System.Net.WebException: The SSL connection could not be established, see inner exception.
---> System.Net.Http.HttpRequestException: The SSL connection could not be established, see inner exception.
---> System.Security.Authentication.AuthenticationException: Authentication failed, see inner exception.
---> Interop+OpenSsl+SslException: SSL Handshake failed with OpenSSL error - SSL_ERROR_SSL.
---> Interop+Crypto+OpenSslCryptographicException: error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure
--- End of inner exception stack trace ---
at Interop.OpenSsl.DoSslHandshake(SafeSslHandle context, ReadOnlySpan`1 input, Byte[]& sendBuf, Int32& sendCount)
at System.Net.Security.SslStreamPal.HandshakeInternal(SafeFreeCredentials credential, SafeDeleteSslContext& context, ReadOnlySpan`1 inputBuffer, Byte[]& outputBuffer, SslAuthenticationOptions sslAuthenticationOptions)
--- End of inner exception stack trace ---
at System.Net.Security.SslStream.ForceAuthenticationAsync[TIOAdapter](TIOAdapter adapter, Boolean receiveFirst, Byte[] reAuthenticationData, Boolean isApm)
at System.Net.Security.SslStream.ProcessAuthentication(Boolean isAsync, Boolean isApm, CancellationToken cancellationToken)
at System.Net.Security.SslStream.AuthenticateAsClient(SslClientAuthenticationOptions sslClientAuthenticationOptions)
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.EstablishSslConnectionAsyncCore(Boolean async, Stream stream, SslClientAuthenticationOptions sslOptions, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageHandlerStage.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.SocketsHttpHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClientHandler.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpMessageInvoker.Send(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.SendAsyncCore(HttpRequestMessage request, HttpCompletionOption completionOption, Boolean async, Boolean emitTelemetryStartStop, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.Send(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationToken cancellationToken)
at System.Net.HttpWebRequest.SendRequest(Boolean async)
at System.Net.HttpWebRequest.GetResponse()
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
at Common.Services.WebContent.GetXmlContent() in xxxxxxxxx/ContentService.cs:line 671
NOTICE: I have also tried
- Change the global
openssl.cnfin/etc/ssl/openssl.cnf - Test the same URL with
curl -vvv. It works WITH NO change in openssl.cnf, and shows a correct TLSv1.2 negotiation (in my URL, to AES128-SHA256) - Test the same URL with
openssl s_client -connect host:port. Again, run with no problems.
UPDATE: I have created a simple test code. This code runs without problems in Net5 in Mac, and in a docker container but fails with the previous error in Ubuntu 20.04
using System;
using System.Collections.Generic;
using System.IO;
using System.Net;
using System.Net.Http;
using System.Net.Security;
using System.Threading.Tasks;
namespace BugSSL
{
class Program
{
static void Main(string[] args)
{
Curl("https://www.boe.es/diario_boe/xml.php?id=BOE-S-20201216");
}
static void Curl(string url)
{
HttpWebRequest request;
HttpWebResponse response;
ServicePointManager.SecurityProtocol = SecurityProtocolType.SystemDefault;
try
{
Console.WriteLine("START-------------------------------------------------");
Console.WriteLine($"Getting URL ${url}");
request = WebRequest.CreateHttp(url);
request.Method = "GET";
request.AllowAutoRedirect = true;
request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
request.Headers.Add(HttpRequestHeader.UserAgent, "SSLBugTest/0.0.0");
using (response = (HttpWebResponse)request.GetResponse())
{
Console.WriteLine($"Response status: {response.StatusCode} {response.StatusDescription}");
Console.WriteLine("Response headers");
foreach(string header in response.Headers)
{
Console.WriteLine($" {header}: {response.GetResponseHeader(header)}");
}
Console.WriteLine($"Content-Type: {response.ContentType}");
Console.WriteLine($"Content-Length: {response.ContentLength}");
}
Console.WriteLine("END---------------------------------------------------");
}
catch (Exception e)
{
TextWriter stderr = Console.Error;
stderr.WriteLine($"Error processing {url}. Error: {e.Message}");
stderr.WriteLine(e.StackTrace);
while(e.InnerException!=null)
{
e = e.InnerException;
Console.WriteLine($"Inner exception: {e.Message}");
stderr.WriteLine(e.StackTrace);
}
}
}
}
}
UPDATE1: I have found this issue in github, but it has been closed without diagnostic or solution https://github.com/dotnet/runtime/issues/44191
UPDATE2: I suspected the problem was that Net5 was not using the configured CipherStrings, and I created a test to use all internally configured. THIS DID NOT WORK, but, maybe, the code can help for further diagnose
static async Task Curl2(string url)
{
List<TlsCipherSuite> cipherSuites;
SslClientAuthenticationOptions sslOptions;
SocketsHttpHandler socketHttpHandler;
HttpResponseMessage response;
Console.WriteLine("START-------------------------------------------------");
Console.WriteLine("Using HttpClient and custom socket (custom SslOptions)");
cipherSuites = new List<TlsCipherSuite>();
foreach (TlsCipherSuite cipherSuite in (TlsCipherSuite[]) Enum.GetValues(typeof(TlsCipherSuite)))
{
cipherSuites.Add(cipherSuite);
}
sslOptions = new SslClientAuthenticationOptions();
//sslOptions = new SslClientAuthenticationOptions
//{
// CipherSuitesPolicy = new CipherSuitesPolicy(cipherSuites)
//};
try
{
sslOptions = new SslClientAuthenticationOptions();
socketHttpHandler = new SocketsHttpHandler
{
SslOptions = sslOptions
};
var httpClient = new HttpClient(socketHttpHandler, true);
using (response = await httpClient.GetAsync(url))
{
Console.WriteLine($"Response status: {response.StatusCode}");
Console.WriteLine("Response headers");
foreach (KeyValuePair<string, IEnumerable<string>> header in response.Headers)
{
Console.Write($" {header.Key}: ");
foreach (string value in header.Value)
{
Console.Write($"{value} ");
}
Console.WriteLine("");
}
}
}
catch (Exception e)
{
TextWriter stderr = Console.Error;
stderr.WriteLine($"Error processing {url}. Error: {e.Message}");
stderr.WriteLine(e.StackTrace);
while (e.InnerException != null)
{
e = e.InnerException;
Console.WriteLine($"Inner exception: {e.Message}");
stderr.WriteLine(e.StackTrace);
}
}
Console.WriteLine("END---------------------------------------------------");
}