Background:
I use Kinesis for Android via aws-sdk-android v2.6.17. Our app uses an analytics service that sends data to the Kinesis. The Kinesis data is later pushed into InsightOps for log tracking. Everything was working properly until this week, when there were massive log entries to the InsightOps database. The log volume was so massive that the InsightOps was filled up with gigabytes of storage in a short time. Here's the log that was printing:
E/propClient(13213): PropClient failed to load
E/CONSCRYPT( 4314): ------------------Untrusted chain: ----------------------
E/CONSCRYPT( 4314): == Chain0 ==
E/CONSCRYPT( 4314): Version: 3
E/CONSCRYPT( 4314): SubjectKeyIdentifier: 41604146332935206e0187513fc382e93836f3ae742470e
E/CONSCRYPT( 4314): Serial Number: 40af5c04b698cd8eb60200000000805592
E/CONSCRYPT( 4314): SubjectDN: CN=*.google.com, O=Google LLC, L=Mountain View, ST=California, C=US
E/CONSCRYPT( 4314): IssuerDN: CN=*****
E/CONSCRYPT( 4314): Get not before: Mon Nov 02 23:33:42 PST 2020
E/CONSCRYPT( 4314): Get not after: Mon Jan 25 23:33:42 PST 2021
E/CONSCRYPT( 4314): Sig ALG name: SHA256withRSA
E/CONSCRYPT( 4314): Signature: 19dc37eb48d8434831c3d4eacae164435735775599621816c2441542a984acb5c22ea9fbfc32900ff26401d8f3151930321ecbd542893095090f3bb774ad383bf2557f9f86d9c325c0d6df00c4d60e9721a1c449509d9aafc651965e1a544f7924e62e2c2ca33275f542586588fd8462ca930a24fce5532438d0320a8153f279e2a57edb36466ea273c10ad68f09de5075425abf98958e715aced73721e8e9e1c9c3ce5644b69ef40281e5681ed3e1bad68d2fde0bccb56c1140eba914b478f6612f538494648713cf4478341a00655990a813f09a747463855b569e74b96d902177aa0fc6186dcef2cb8b2007007f16dda0511cccd5b87d05d463c64104971f
E/CONSCRYPT( 4314): Public key:
E/CONSCRYPT( 4314):
E/CONSCRYPT( 4314): 30 59 30 13 06 07 2a 86 48 ce 3d 02 01 06 08 2a 86 48 ce 3d
E/CONSCRYPT( 4314): 03 01 07 03 42 00 04 09 6a 8d 78 d8 ec ef fc 8a a4 fe 55 ad
E/CONSCRYPT( 4314): 43 86 61 09 12 3e ca ec 73 6e 4a 9c 47 35 64 53 6d 49 8a 15
E/CONSCRYPT( 4314): 27 e6 1b 04 82 e9 b3 72 dd 8c cc 1c 55 32 c4 3d 78 87 bd cf
E/CONSCRYPT( 4314): 93 5a 31 c5 05 48 77 0d 2a 58 9e
The investigation:
Fortunately, we got a OutOfMemoryError crash from Firebase Crashlytics. The log trace shows the Kinesis client was continuously getting OpenSSLX509Certificate error. Since the client was printing Conscrypt error, it ended up with a large number of the stack trace logs and caused the app to run out of memory.
Fatal Exception: java.lang.OutOfMemoryError: Failed to allocate a 112 byte allocation with 96 free bytes and 96B until OOM, max allowed footprint 268435456, growth limit 268435456
at java.util.Calendar.createCalendar(Calendar.java:1684)
at java.util.Calendar.getInstance(Calendar.java:1639)
at com.android.org.conscrypt.OpenSSLX509Certificate.toDate(OpenSSLX509Certificate.java:85)
at com.android.org.conscrypt.OpenSSLX509Certificate.<init>(OpenSSLX509Certificate.java:74)
at com.android.org.conscrypt.OpenSSLX509Certificate.fromX509DerInputStream(OpenSSLX509Certificate.java:101)
at com.android.org.conscrypt.OpenSSLX509CertificateFactory$1.fromX509DerInputStream(OpenSSLX509CertificateFactory.java:232)
at com.android.org.conscrypt.OpenSSLX509CertificateFactory$1.fromX509DerInputStream(OpenSSLX509CertificateFactory.java:222)
at com.android.org.conscrypt.OpenSSLX509CertificateFactory$Parser.generateItem(OpenSSLX509CertificateFactory.java:112)
at com.android.org.conscrypt.OpenSSLX509CertificateFactory.engineGenerateCertificate(OpenSSLX509CertificateFactory.java:278)
at java.security.cert.CertificateFactory.generateCertificate(CertificateFactory.java:366)
at com.android.org.conscrypt.SSLUtils.decodeX509Certificate(SSLUtils.java:214)
at com.android.org.conscrypt.SSLUtils.decodeX509CertificateChain(SSLUtils.java:198)
at com.android.org.conscrypt.ConscryptFileDescriptorSocket.verifyCertificateChain(ConscryptFileDescriptorSocket.java:414)
at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(NativeCrypto.java)
at com.android.org.conscrypt.NativeSsl.doHandshake(NativeSsl.java:383)
at com.android.org.conscrypt.ConscryptFileDescriptorSocket.startHandshake(ConscryptFileDescriptorSocket.java:231)
at com.android.okhttp.internal.io.RealConnection.connectTls(RealConnection.java:1480)
at com.android.okhttp.internal.io.RealConnection.connectSocket(RealConnection.java:1424)
at com.android.okhttp.internal.io.RealConnection.connect(RealConnection.java:1368)
at com.android.okhttp.internal.http.StreamAllocation.findConnection(StreamAllocation.java:219)
at com.android.okhttp.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:142)
at com.android.okhttp.internal.http.StreamAllocation.newStream(StreamAllocation.java:104)
at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:392)
at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:325)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:470)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.connect(HttpURLConnectionImpl.java:127)
at com.android.okhttp.internal.huc.HttpURLConnectionImpl.getOutputStream(HttpURLConnectionImpl.java:258)
at com.android.okhttp.internal.huc.DelegatingHttpsURLConnection.getOutputStream(DelegatingHttpsURLConnection.java:218)
at com.android.okhttp.internal.huc.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:26)
at com.amazonaws.http.UrlHttpClient.writeContentToConnection(UrlHttpClient.java:162)
at com.amazonaws.http.UrlHttpClient.execute(UrlHttpClient.java:75)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:371)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:212)
at com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2776)
at com.amazonaws.services.kinesis.AmazonKinesisClient.putRecords(AmazonKinesisClient.java:1452)
at com.amazonaws.mobileconnectors.kinesis.kinesisrecorder.KinesisStreamRecordSender.sendBatch(KinesisStreamRecordSender.java:86)
at com.amazonaws.mobileconnectors.kinesis.kinesisrecorder.AbstractKinesisRecorder.submitAllRecords(AbstractKinesisRecorder.java:136)
at com.*****.KinesisAnalyticsSource.sendAnalytics(KinesisAnalyticsSource.java:144)
I have no idea why the exception is occurring. Could it be because of the Outage Amazon had recently? I'm looking for the possible causes of the issue.