My problem is that I have to find in a list, these which are not empty. When I use the filter function is not null, than I get also every row.
My program code looks like this:
...
val csc = new CassandraSQLContext(sc)
val df = csc.sql("SELECT * FROM test").toDF()
val wrapped = df.select("fahrspur_liste")
wrapped.printSchema
The column fahrspur_liste contains the wrapped arrays and this column I have to analyze. When I run the code, than I get this structure for my wrapped array and these entries:
root
|-- fahrspur_liste: array (nullable = true)
| |-- element: long (containsNull = true)
+--------------+
|fahrspur_liste|
+--------------+
| []|
| []|
| [56]|
| []|
| [36]|
| []|
| []|
| [34]|
| []|
| []|
| []|
| []|
| []|
| []|
| []|
| [103]|
| []|
| [136]|
| []|
| [77]|
+--------------+
only showing top 20 rows
Now I want to filter these rows, so that I have only the entries [56],[36],[34],[103], ...
How can I write a filter function, that I get only these rows, which contains a number?