This page uses JavaScript and Cookies and this makes problems. It also checks other headers, not only User-Agent.
First: you have to use requests.Session() to keep cookies. Second: you have to load some page (ie. main page) to get these cookies. When you will have cookies then it will accept other URLs. Third: it check also other headers to send cookies.
I run page in browser and use DevTools in Chrome/Firefox to copy all headers used by real browser and I start testing request with different headers. Finally I found it needs
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
'Accept-Language': 'en-US;q=0.7,en;q=0.3',
'Cache-Control': 'no-cache',
}
Other problem is that page uses JavaScript to load images when you scroll page ("lazy-loading") and some url are not in scr but in data-lazy and then src has 'blank.gif'
import requests
from bs4 import BeautifulSoup
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
#"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
#"Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US;q=0.7,en;q=0.3",
"Cache-Control": "no-cache",
#"Connection": "keep-alive",
#"Pragma": "no-cache",
}
s = requests.Session()
s.headers.update(headers) # it will use there hearders in all requests
# --- get cookies ---
url = 'https://pixabay.com/'
r = s.get(url)
print(r.status_code) # 403 but it is not problem
# only for test
#r = s.get(url)
#print(r.status_code) # 200 because it already have cookies
# --- get images ---
url = 'https://pixabay.com/images/search/office/'
r = s.get(url)
print(r.status_code)
#print(r.text)
results = []
soup = BeautifulSoup(r.text, "lxml")
for item in soup.select(".search_results a > img[src]"):
src = item.get("src")
if src is not None and 'blank.gif' not in src:
print('src:', src)
results.append(src)
else:
src = item.get("data-lazy")
print('data-lazy:', src)
results.append(src)
print('len:', len(results))
