AWS EBS - Rails5 / nginx - robots.txt not found error (404)

Looks like a very similar question have been asked 4 years ago on the official AWS forum: https://forums.aws.amazon.com/thread.jspa?threadID=150904

Only 4 years later a brave guy from AWS stepped in with a reply! Here below the quoted reply:

Hello hello! I'm Chris, the new Ruby platforms person at Elastic Beanstalk. Visiting this thread today, it looks like there's been a lot of pain (and also confusion!) from Beanstalk's Ruby+Puma's handling of static files.

Quick summary: When this thread was created (in 2014), Beanstalk was essentially using the default Nginx that comes with Amazon Linux, with only some logging modifications to support the health monitoring. That spawned this thread, as static files are generally expected to be served the the web server when one is present.

So, the folks here went and fixed the /assets folder. Great! Unfortunately, there was a misunderstanding with the request to fix serving the /public folder - Beanstalk's Puma platform instead serves things in '/public' from '/pubilc', not from '/'. This is definitely an issue, so here's some workarounds:

Workaround 1: Turning on serve static assets. Yes, this wastes some application threads here or there, but if your use case is only robots.txt and favicon.ico, you're only robbing a couple of appserver threads. I'd pick this one unless I was running my application servers hot.

Workaround 2: Write an .ebextension to modify the Nginx configuration to serve /public at /. I'm in the process of writing one, so I'll tack it as a reply to this when I've given it the thought it deserves. Some of the current ones may serve your app's code, so double check the configuration if you've already done this workaround.

I've created a tracking issue for the team with this level of detail, so we'll work to get this corrected. Thank you all for your feedback - we'd love to serve you and your apps better.

Since then, no further replies; if anybody knows the "aws-approved-way" to edit nginx config with .ebextensions let's post it here please! :)

In AWS EB with PUMA, static files under the public folder are served under the /public/ url. Webcrawlers expect the file available at /robots.txt

I've struggled to try and implement routing to these files and settled instead on a more 'Rails' way of implementing this.

1) config/routes.rb

get "/robots.txt", to: "robots#show"

2) app/controllers/robots_controller.rb

class RobotsController < ApplicationController
  def show
    render "show", layout: false, content_type: "text/plain"
  end 
end

3) app/views/robots_txts/show.erb

User-agent: *
Disallow: /

The above link to AWS forums is erroring with a 400 right now, so here's how I fixed this issue. Ruby 2.7 running on AWS2 platform:

Static Files in sub-directory of /public:

Create a file under the .ebextensions folder called static-files.conf. Content should look similar to:

option_settings:
  aws:elasticbeanstalk:environment:proxy:staticfiles:
    /w3c: public/w3c
    /images: public/images

This will ensure that all requests to domain.com/images and domain.com/w3c are served from the appropriate /public sub-directory.

Static Files in top level of /public directory: For top-level files like robots.txt or sitemap.xml add appropriate entry to routes.rb to serve the static content directly:

  get '/robots.txt', to: proc {|env| [200, {}, [File.open(Rails.root.join('public', 'robots.txt')).read]] }
  get '/sitemap.xml', to: proc {|env| [200, {}, [File.open(Rails.root.join('public', 'sitemap.xml')).read]] }

Ensure production.rb has static files config set properly:

config.serve_static_files = false

This last part is most-important.