I am crawling the site https://www.recode.net/ and during the fetch job the log showing :
fetching https://www.recode.net/ (queue crawl delay=5000ms)
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=1
Using queue mode : byHost
robots.txt whitelist not configured.
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=1
Using queue mode : byHost
Using queue mode : byHost
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=1
Using queue mode : byHost
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=1
Using queue mode : byHost
..........................
..........................
Fetcher: throughput threshold: -1
Fetcher: throughput threshold retries: 5
fetcher.maxNum.threads can't be < than 50 : using 50 instead
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=1
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=3
Thread FetcherThread has no more work available
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=2
-finishing thread FetcherThread, activeThreads=1
-activeThreads=1, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=1
fetch of https://www.recode.net/ failed with: Http code=403, url=https://www.recode.net/
Thread FetcherThread has no more work available
-finishing thread FetcherThread, activeThreads=0
-activeThreads=0, spinWaiting=0, fetchQueues.totalSize=0, fetchQueues.getQueueCount=0
-activeThreads=0
Fetcher: finished at 2017-09-21 15:39:07, elapsed: 00:00:03
And I didn't get any data to index.
I checked with the site's robots.txt and it's fine and not disallowing my user agent. Also, I am not using any regex to filter url.