I'm trying to cralwing some webpages.
So, i made docker image like this.
FROM public.ecr.aws/bitnami/node:16.13.1
RUN apt-get update && apt-get install -y gnupg
RUN wget -q -O - https://dl-ssl.google.com/linux/linux_signing_key.pub | apt-key add - \
&& echo "deb http://dl.google.com/linux/chrome/deb/ stable main" >> /etc/apt/sources.list.d/google.list
RUN apt-get update && apt-get install -y google-chrome-stable
# It's a good idea to use dumb-init to help prevent zombie chrome processes.
COPY dumb-init_1.2.0_amd64 /usr/local/bin/dumb-init
RUN chmod +x /usr/local/bin/dumb-init
ENTRYPOINT ["dumb-init", "--"]
RUN rm dumb-init_1.2.0_amd64
ENV PUPPETEER_SKIP_CHROMIUM_DOWNLOAD=true
COPY . .
RUN npm install && npm run build
EXPOSE 3000
CMD ["npm", "start"]
puppeteer launch chrome browser
browser = await puppeteer.launch({
executablePath: '/usr/bin/google-chrome-stable',
headless: true,
dumpio: true,
args: [
'--no-zygote',
'--no-sandbox',
'--single-process',
'--disable-setuid-sandbox',
'--disable-gpu',
'--disable-dev-shm-usage'
]
});
Sometimes, puppeteer's dumpio show errors and can't load page, so i can't crawling webpages.
[0913/094932.227208:ERROR:ssl_client_socket_impl.cc(983)] handshake failed; returned -1, SSL error code 1, net_error -101
[0913/094933.523641:WARNING:spdy_session.cc(3502)] Received HEADERS for invalid stream 1
[0913/094933.529341:INFO:render_frame_host_impl.cc(11654)] RenderFrameHostImpl::MaybeGenerateCrashReport url = {targetUrl}, status = 2, exit_code = 9
But i can't find any error info of above messages.
if ERROR:ssl_client_socket_impl is cause, should i set args with --ignore-certificate-errors, --ignore-ssl-errors?