ElasticSearch: When using field collapsing is it possible to return all documents that do not have a value for the collapse key?

I am trying to collapse documents on a given key, ordered by date. The key is not always populated. In cases where the key is null, elastic seems to collapse all documents that do not have this key populated and return the newest one. I would like to only collapse documents were the field is populated and return the document if the key isn't present. Is this possible?

Here is the collapse query I am currently using:

"collapse": {
"field": "COLLAPSE_KEY",
"inner_hits": {
  "name": "collapse_inner_hit",
  "ignore_unmapped": false,
  "from": 0,
  "size": 1,
  "version": false,
  "seq_no_primary_term": false,
  "explain": false,
  "track_scores": false,
  "sort": [
    {
      "DATE_FIELD": {
        "order": "desc"
      }
    }
  ]
}

and in java:

 CollapseBuilder collapseBuilder = new CollapseBuilder(COLLAPSE_KEY);

 InnerHitBuilder collapseHitBuilder = new InnerHitBuilder("collapse_inner_hit");
 collapseHitBuilder.setSize(1);
 collapseHitBuilder.addSort(new FieldSortBuilder(DATE_FIELD).order(SortOrder.DESC));
 collapseBuilder.setInnerHits(collapseHitBuilder);

 searchSourceBuilder.collapse(collapseBuilder);