How do I perform an "OR" filter on an aggregate?

Viewed 230

I am trying to grab the first 10 documents grouped by domain. These 10 documents need to have a value for "crawl_date" that haven't been crawled for a while or haven't been crawled at all (eg a blank value). I have:

curl -XPOST 'http://localhost:9200/tester/test/_search' -d '
{
  "size": 10,
  "aggs": {
    "group_by_domain": {
            "filter": {
                "or":[
                        "term": {"crawl_date": ""},
                        "term": {"crawl_date": ""}  // how do I put a range here? e.g. <= '2014-12-31'
                    ]
            },
      "terms": {
        "field": "domain"
      }
    }
  } 
}'

I am new to ES and using version 2.2. Since the documentation isn't fully updated I am struggling.

EDIT: To clarify, I need 10 urls that haven't been crawled or haven't been crawled for a while. Each of those 10 urls has to come from a unique domain so that when I crawl them I don't overload someone's server.

Another Edit: So, I need something like this (1 link for each of 10 unique domains):

1. www.domain1.com/page
2. www.domain2.com/url
etc...

Instead, I am getting just the domain and the number of pages:

"buckets": [
          {
            "key": "http://www.dailymail.co.uk",
            "doc_count": 212
          },
          {
            "key": "https://sedo.com",
            "doc_count": 196
          },
          {
            "key": "http://www.foxnews.com",
            "doc_count": 118
          },
          {
            "key": "http://data.worldbank.org",
            "doc_count": 117
          },
          {
            "key": "http://detail.1688.com",
            "doc_count": 117
          },
          {
            "key": "https://twitter.com",
            "doc_count": 112
          },
          {
            "key": "http://search.rakuten.co.jp",
            "doc_count": 104
          },
          {
            "key": "https://in.1688.com",
            "doc_count": 92
          },
          {
            "key": "http://www.abc.net.au",
            "doc_count": 87
          },
          {
            "key": "http://sport.lemonde.fr",
            "doc_count": 85
          }
        ]

The "hits" returns multiple pages for just 1 domain:

"hits": [
      {
        "_index": "tester",
        "_type": "test",
        "_id": "http://www.barnesandnoble.com/w/at-the-edge-of-the-orchard-tracy-chevalier/1121908441?ean=9780525953005",
        "_score": 1,
        "_source": {
          "domain": "http://www.barnesandnoble.com",
          "crawl_date": "0001-01-01T00:00:00Z"
        }
      },
      {
        "_index": "tester",
        "_type": "test",
        "_id": "http://www.barnesandnoble.com/b/bargain-books/_/N-8qb",
        "_score": 1,
        "_source": {
          "domain": "http://www.barnesandnoble.com",
          "crawl_date": "0001-01-01T00:00:00Z"
        }
      },
      etc....

Barnes and Noble will quickly ban my UA if I try to crawl that many domains at the same time.

I need something like this:

1. "http://www.dailymail.co.uk/page/text.html",
2. "https://sedo.com/another/page"
3. "http://www.barnesandnoble.com/b/bargain-books/_/N-8qb"
4. "http://www.starbucks.com/homepage/"
etc.
3 Answers
Related