cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})

Question

Currently facing a problem where flask app which is running under uWSGI gives above error after some time.

Exception message which occur is:

cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})

Here is the code of the app for wsgi.py

import sys
import logging

sys.stdout = sys.stderr

from cqlengine.connection import (
cluster as cql_cluster, session as cql_session)
from url.settings import CLUSTER

log = logging.getLogger('uwsgi')

try:
    from uwsgidecorators import postfork
except ImportError:
    # We're not in a uWSGI context, no need to hook Cassandra session
    # initialization to the postfork event.
    pass
else:

@postfork
def cassandra_init():
    """ Initialize a new Cassandra session in the context.

        Ensures that a new session is returned for every new request.
    """
    if cql_cluster is not None:
        cql_cluster.shutdown()
    if cql_session is not None:
        cql_session.shutdown()
    from url.settings import connect_cassandra
    connect_cassandra()
    log.info('Connection with cassandra completed')
from url.app import app as application

Here is the code the connect_cassandra method.

CLUSTER = ['XX.XX.XX.XX', 'XX.XX.XX.XX']


def connect_cassandra():
    # next, setup the connection to your cassandra server(s)...
    # see http://datastax.github.io/python-
      driver/api/cassandra/cluster.html for options
    # the list of hosts will be passed to create a Cluster() instance
    from cassandra.cqlengine import connection
    from cassandra.cqlengine.management import sync_table
    from url.models import Links, LinksAnalytics

    connection.setup(CLUSTER, "contentstudio", protocol_version=3)
    sync_table(Links)
    sync_table(LinksAnalytics)

After restarting app works fine for 20-30 minutes and after that it stops giving this error and sometimes it works which is really frustrating.

Also, in between getting this exception message as well:

cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {: ConnectionException('Host has been marked down or removed',)})

UPDATE 1 with nodetool tpstats

node 1

Pool Name                         Active   Pending      Completed   Blocked  All time blocked
ReadStage                              0         0        1225692         0                 0
MiscStage                              0         0              0         0                 0
CompactionExecutor                     0         0         853120         0                 0
MutationStage                          0         0          62573         0                 0
MemtableReclaimMemory                  0         0           1133         0                 0
PendingRangeCalculator                 0         0              2         0                 0
GossipStage                            0         0        4175516         0                 0
SecondaryIndexManagement               0         0              0         0                 0
HintsDispatcher                        0         0              0         0                 0
RequestResponseStage                   0         0          64064         0                 0
Native-Transport-Requests              0         0       12887762         0             16587
ReadRepairStage                        0         0           6887         0                 0
CounterMutationStage                   0         0              0         0                 0
MigrationStage                         0         0             34         0                 0
MemtablePostFlush                      0         0           1268         0                 0
PerDiskMemtableFlushWriter_0           0         0           1123         0                 0
ValidationExecutor                     0         0              0         0                 0
Sampler                                0         0              0         0                 0
MemtableFlushWriter                    0         0           1125         0                 0
InternalResponseStage                  0         0             45         0                 0
ViewMutationStage                      0         0              0         0                 0
AntiEntropyStage                       0         0              0         0                 0
CacheCleanupExecutor                   0         0              0         0                 0

Message type           Dropped
READ                         0
RANGE_SLICE                  0
_TRACE                       0
HINT                         0
MUTATION                     0
COUNTER_MUTATION             0
BATCH_STORE                  0
BATCH_REMOVE                 0
REQUEST_RESPONSE             0
PAGED_RANGE                  0
READ_REPAIR                  0

node 2

Pool Name                         Active   Pending      Completed   Blocked  All time blocked
ReadStage                              0         0          29325         0                 0
MiscStage                              0         0              0         0                 0
CompactionExecutor                     0         0         407325         0                 0
MutationStage                          0         0          62573         0                 0
MemtableReclaimMemory                  0         0           1133         0                 0
PendingRangeCalculator                 0         0              4         0                 0
GossipStage                            0         0        4174442         0                 0
SecondaryIndexManagement               0         0              0         0                 0
HintsDispatcher                        0         0              0         0                 0
RequestResponseStage                   0         0           6845         0                 0
Native-Transport-Requests              0         0         989812         0                 0
ReadRepairStage                        0         0            102         0                 0
CounterMutationStage                   0         0              0         0                 0
MigrationStage                         0         0             26         0                 0
MemtablePostFlush                      0         0           1268         0                 0
PerDiskMemtableFlushWriter_0           0         0           1123         0                 0
ValidationExecutor                     0         0              0         0                 0
Sampler                                0         0              0         0                 0
MemtableFlushWriter                    0         0           1125         0                 0
InternalResponseStage                  0         0              0         0                 0
ViewMutationStage                      0         0              0         0                 0
AntiEntropyStage                       0         0              0         0                 0
CacheCleanupExecutor                   0         0              0         0                 0

Message type           Dropped
READ                         0
RANGE_SLICE                  0
_TRACE                       0
HINT                         0
MUTATION                     0
COUNTER_MUTATION             0
BATCH_STORE                  0
BATCH_REMOVE                 0
REQUEST_RESPONSE             0
PAGED_RANGE                  0
READ_REPAIR                  0

user_2179574 · Answer

I had the same problem and in my case the problem was that uwsgi doesn't enable threads by default, and python-driver has an internal thread pool.

Try enabling threads: https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html#a-note-on-python-threads

user_6659678 · Answer

Cassandra is a NoSQL distributed DBMS which is written in Java. Java uses a special data structure called HEAP. Java objects reside in the heap area. The heap is created when the JVM (Java Virtual Machine) starts up and may increase or decrease in size while the application runs. Cassandra uses memory in 4 ways:

Java heap
Offheap memory
OS page cache
OS TCP/IP stack I/O cache

The error you get, is due to the insuffiecient OS system memory (OS RAM). Probably, in your case, the heap size is not in accord with the memory.

There are some guidelines and recommendations:

HEAP_SIZE is usually between 1/4 and 1/2 of system memory but not larger than 32 GB. NEW_HEAP_SIZE determines the amount of heap memory allocated to newer objects. The database calculates the default value for this property in MB as the lesser of:

100 times the number of cores
1/4 of the MAX_HEAP_SIZE

The cassandra-env.sh automatically configures the min and max size to the same value using the following formula: Max(Min(1/2 RAM, 1024 Megabytes), Min(1/4 RAM, 32765 Megabytes))

To adjust the JVM heap size, uncomment and set the following parameters in the jvm-server.options file:

Minimum (-Xms)

Maximum (-Xmx)

New generation (-Xmn)

If you use docker for cassandra you can set these parameters in the environment in the cassandra service in the docker-compose.yml file.

For additional information: Cassandra AWS System Memory Guidelines and Chaning heap size parameters

cassandra.cluster.NoHostAvailable: ('Unable to complete the operation against any hosts', {})

2 Answers