Nginx web app redirect fails on form submission (redirects to bogus url with %2C in it)

user_10877042 0
Viewed 312

First up, I am new to most of this. I am running a Flask app on a remote (Linode) Ubuntu server. The app is served with Gunicorn and Nginx.

After a recent update, redirects began behaving oddly when I submitted a form.

Specifically, the url I am redirected to looks like this: mywebsite.com%2Cmywebsite.com/my/expected/redirect

(to be clear, mywebsite.com is the base url. On form submission, I am expecting to be redirected to mywebsite.com/my/expected/redirect, however I am redirected to mywebsite.com%2Cmywebsite.com/my/expected/redirect)

This is only happening on the server. On my localhost, this issue is not present.

This is only occurring when I submit a form (through wtf forms). After submitting a form and getting the bogus url, if I manually enter the expected url, I get the expected page - i.e. a confirmation flash message that the form submission was successful, and the database contains the fresh data submitted via the form. So I guess the db.commit() is working.

Not all links are broken, I can navigate from page to page successfully. e.g. I can navigate home from my navbar via this link in the site-wide html header file.

href="{{ url_for('home') }}"

and I can navigate to other pages via these buttons (I know I should be using url_for instead of hard links)

<button class="w3-bar-item w3-button w3-black" onclick="window.location.href = '/recipes';">Recipes</button>
<button class="w3-bar-item w3-button w3-yellow" onclick="window.location.href = '/newrecipe';">Add New Recipe</button>

There are three separate forms which all produce this error. below you'll see them as NewRecipe, NewBoilAddition, StartBoilTimer

I have tried adding _external=True to the url_for redirect, however this fails with Internal Server Error

From research, I understand that %2C is an unescaped comma, therefore I am suspicious that a list of urls is being passed to the redirect, or that there is some other encoding issue that I'm missing. I am especially confused as this bug has only just appeared, when all this code was working prior. I have attempted a rollback to a commit that I am certain this bug was not present in, and the bug was reproducible in that commit. This makes me suspicious that it is gunicorn/nginx responsible, as the app code itself was good.

I am super confused. Any ideas appreciated!

init.py

from flask import Flask, render_template, request, g, redirect, url_for, flash
from flask_sqlalchemy import SQLAlchemy
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.hybrid import hybrid_property

from forms import NewRecipe, NewBoilAddition, StartBoilTimer

# import sqlite3
import pandas as pd
import json
import sys
import os
import config

from datetime import datetime, timedelta

from flask_gtts import gtts

app = Flask(__name__)

app.config['SECRET_KEY'] = 'xxx'
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///site.db'

db = SQLAlchemy(app)

class User(db.Model):

    __tablename__ = 'users'

    id          = db.Column(db.Integer, primary_key=True)
    username    = db.Column(db.String(20), unique=True, nullable=False)
    password    = db.Column(db.String(60), nullable=False)
    bevvys      = db.relationship('Bevvy_list', backref='brewer', lazy=True)

    def __repr__(self):
        return f"User('{self.id}', '{self.username}')"

class Bevvy_list(db.Model):

    __tablename__ = 'bevvy_list'

    id              = db.Column(db.Integer, primary_key=True)
    name            = db.Column(db.String(100), unique=True, nullable=False) #Optimise length of this string
    style           = db.Column(db.String(60), nullable=False) #TODO add in validated field, linking to style guide
    abbreviation    = db.Column(db.String(60), nullable=False)
    iteration       = db.Column(db.Integer, nullable=False)
    iteration_of    = db.Column(db.Integer, nullable=False) #id of parent beer, if iteration = 1 then this equals self.id
    batch_size      = db.Column(db.Integer, nullable=False)
    brewday_date    = db.Column(db.DateTime, nullable=False)
    url             = db.Column(db.String(20), nullable=True)
    user_id         = db.Column(db.Integer, db.ForeignKey('users.id'), nullable=False)
    boils           = db.relationship('Boil', backref='bevvy', lazy=True)

    def __repr__(self):
        return f"Bevvy_list('{self.id}', '{self.name}')"

class Boil(db.Model):

    __tablename__ = 'boil'

    id              = db.Column(db.Integer, primary_key=True)
    description     = db.Column(db.String(100), unique=False, nullable=False)
    time            = db.Column(db.Integer, unique=False, nullable=False)
    end_datetime    = db.Column(db.DateTime, unique=False, nullable=True)
    brew_id         = db.Column(db.Integer, db.ForeignKey('bevvy_list.id'), nullable=False)

    def __repr__(self):
        return f"Boil('{self.id}', '{self.description}', '{self.time}', '{self.end_datetime}')"

##########################################
#          Build app
##########################################

# home navigation
@app.route("/")
def index():
    return render_template("home.html")

@app.route("/recipes")
def recipes():
    bevs_data = Bevvy_list.query.all()
    return render_template("recipes.html", bevs_data=bevs_data)

@app.route("/edit/<int:recipe_id>", methods=['GET', 'POST'])
def edit(recipe_id):
    form = NewBoilAddition(brew_id=recipe_id)
    if form.validate_on_submit():
        boil_addition = Boil(description=form.description.data, \
                            time=form.time.data, \
                            brew_id=recipe_id,\
                            end_datetime=None)
        db.session.add(boil_addition)
        db.session.commit()
        flash(f'Boil Addition {form.description.data} successfully added ', 'success_boil')
        return redirect(url_for('edit', recipe_id=recipe_id))

    start_timer_form = StartBoilTimer()
    boil_additions = Boil.query.filter(Boil.brew_id == recipe_id).all()
    if start_timer_form.validate_on_submit():
        end_all_timers = datetime.now() + timedelta(minutes=max_value(Boil.query.filter(Boil.brew_id == recipe_id).with_entities(Boil.time).all()))
        for addition in boil_additions:
            addition_end_datetime = end_all_timers - timedelta(minutes=addition.time)
            db.session.query(Boil).filter(Boil.id == addition.id).update({'end_datetime':addition_end_datetime})
            db.session.commit()
        return redirect(url_for('edit', recipe_id=recipe_id))

    recipe = Bevvy_list.query.filter(Bevvy_list.id == recipe_id).all()

    return render_template("edit.html", recipe=recipe[0], form=form, \
        boil_additions=boil_additions, start_timer_form=start_timer_form)

# add a new recipe
@app.route("/newrecipe", methods=['GET', 'POST'])
def new_recipe():
    form = NewRecipe()
    recipe_list = Bevvy_list.query.all()
    list_recipes =[]
    for recipe in recipe_list:
        row = [recipe.id, recipe.name, recipe.brewday_date.strftime("%-d %b %y")]
        list_recipes.append(row)
    if form.validate_on_submit():
        recipe = Bevvy_list(name=form.name.data, style=form.style.data, abbreviation=form.abbreviation.data, iteration=form.iteration.data, \
            iteration_of=form.iteration_of.data, batch_size=form.batch_size.data, brewday_date=form.brewday_date.data, user_id=form.user_id.data)
        db.session.add(recipe)
        href = "/edit/" + str(db.session.query(Bevvy_list).order_by(Bevvy_list.id.desc()).first().id)
        db.session.query(Bevvy_list).order_by(Bevvy_list.id.desc()).first().url = href
        db.session.commit()
        flash(f'Recipe for {form.name.data} successfully added', 'success')
        return redirect(url_for('home'), _external=True)

    return render_template("new_recipe.html", title="New Recipe", form=form, modal=list_recipes)

if __name__ == "__main__":
    app.run()

new_recipe.html

<!DOCTYPE html>

{% extends "layouts.html" %}

{% block content %}

  <!-- Modal -->  
<div class="modal fade" id="exampleModalLong" tabindex="-1" role="dialog" aria-labelledby="exampleModalLongTitle" aria-hidden="true">
  <div class="modal-dialog" role="document">
    <div class="modal-content">
      <div class="modal-header">
        <h5 class="modal-title" id="exampleModalLongTitle">All Recipes</h5>
        <button type="button" class="close" data-dismiss="modal" aria-label="Close">
          <span aria-hidden="true">&times;</span>
        </button>
      </div>
      <div class="modal-body">
        <table class="table table-striped table-hover table-sm">
          <thead class="thead-dark">
            <tr>
              <th scope="col">ID</th>
              <th scope="col">Name</th>
              <th scope="col">Brewday Date</th>
            </tr>
          </thead>
          <tbody>
            {% for list in modal %}
              <tr>
              {% for item in list %}
                <td>{{ item }}</td>
              {% endfor %}
              </tr>
            {% endfor %}
          </tbody>
        </table>
      </div>
      <div class="modal-footer">
        <button type="button" class="btn btn-secondary" data-dismiss="modal">Close</button>
      </div>
    </div>
  </div>
</div>

<div class="content-section">
  <form method="POST" action="">
    {{  form.hidden_tag() }}
    <fieldset class="form-group">
      <legend class="border-bottom mb-4">Add New Recipe</legend>
      <div class="form-group"> 
        {{ form.name.label(class="form-control-label") }}
        {% if form.name.errors %}
          {{ form.name(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.name.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.name(class="form-control form-control-lg") }}
        {% endif %}

      </div>
      <div class="form-group"> 
        {{ form.style.label(class="form-control-label") }}
        {% if form.style.errors %}
          {{ form.style(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.style.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.style(class="form-control form-control-lg") }}
        {% endif %}
      </div>
      <div class="form-group"> 
        {{ form.abbreviation.label(class="form-control-label") }}
        {% if form.abbreviation.errors %}
          {{ form.abbreviation(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.abbreviation.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.abbreviation(class="form-control form-control-lg") }}
        {% endif %}
      </div>
      <div class="form-group"> 
        {{ form.iteration.label(class="form-control-label") }}
        {% if form.iteration.errors %}
          {{ form.iteration(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.iteration.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.iteration(class="form-control form-control-lg") }}
        {% endif %}
      </div>
      <div class="form-group"> 
        {{ form.iteration_of.label(class="form-control-label") }}
        {% if form.iteration_of.errors %}
          {{ form.iteration_of(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.iteration_of.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.iteration_of(class="form-control form-control-lg") }}
        {% endif %}
        <button type="button" class="btn btn-primary" data-toggle="modal" data-target="#exampleModalLong">
          Show recipe list for reference
        </button>
      </div>
      <div class="form-group"> 
        {{ form.batch_size.label(class="form-control-label") }}
        {% if form.batch_size.errors %}
          {{ form.batch_size(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.batch_size.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.batch_size(class="form-control form-control-lg") }}
        {% endif %} 
      </div>
      <div class="form-group"> 
        {{ form.brewday_date.label(class="form-control-label") }}
        {% if form.brewday_date.errors %}
          {{ form.brewday_date(class="form-control form-control-lg is-invalid") }}
          <div class="invalid-feedback">
            {% for error in form.brewday_date.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {{ form.brewday_date(class="form-control form-control-lg") }}
        {% endif %}
      </div>
      <div class="form-check"> 
        {% if form.user_id.errors %}
          {% for subfield in form.user_id %}
          <table>
            <td>
                <tr>{{ subfield(class="form-check-input is-invalid") }}</tr>
                <tr>{{ subfield.label(class="form-check-label") }}</tr><br>
            </td>
          </table>
          {% endfor %}
          <div class="invalid-feedback">
            {% for error in form.user_id.errors%}
              <span>{{ error }}</span>
            {% endfor %}
          </div>
        {% else %}
          {% for subfield in form.user_id %}
            <table>
              <td>
                  <tr>{{ subfield(class="form-check-input") }}</tr>
                  <tr>{{ subfield.label(class="form-check-label") }}</tr><br>
              </td>
            </table>
          {% endfor %}
        {% endif %}
      </div>
    </fieldset>
    <div class="form-group"> 
      {{ form.submit(class="btn btn-outline-info")  }}
    </div>
  </form>
</div>

{% endblock content %}

EDIT:

/etc/systemd/system/MYAPP.service

[Unit]
Description=Gunicorn instance to serve zym_app
After=network.target

[Service]
User=root
Group=www-data
WorkingDirectory=/home/lph/zym_app/zym
Environment="PATH=/home/lph/zym_app/zym/zym_app_env/bin"
ExecStart=/home/lph/zym_app/zym/zym_app_env/bin/gunicorn --workers 3 --bind unix:zym_app.sock -m 007 wsgi:app

[Install]
WantedBy=multi-user.target

and /etc/nginx/sites-available/MYAPP 

server {
    listen 80;
    server_name mywebsite.com www.mywebsite.com;

    location / {
        include proxy_params;
        proxy_pass http://unix:/home/lph/zym_app/zym/zym_app.sock;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }
}

EDIT 2: looks like i missed 2 and went straight to 3..

EDIT 3: Having troubleshooted this issue, I now believe it to be an issue with nginx configuration. I tested the flask app on the server with 

gunicorn --bind 0.0.0.0:5000 wsgi:app

This solved the observed redirect issue, the app worked as expected. Therefore, the flask and gunicorn parts are working fine - leaving only nginx.

I also removed the redirect lines from the .validate_on_submit() conditionals, which also solved the issue when running through nginx. Fresh form data was successfully posted to the database, however I had to manually refresh the page after submission, due to the lack of redirects.

This is my nginx config file. /etc/nginx/sites-available/zym_app

server {
    listen 80;    
    server_name mywebsite.com www.mywebsite.com;

    location / {
        include proxy_params;
        proxy_pass http://unix:/home/ZYM/zym/zym_app.sock;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_headers_hash_max_size 512;
        proxy_headers_hash_bucket_size 128;
    }
}

EDIT 4: After trying many nginx config parameter changes, I commented out the offending redirect lines, and now the issue is present when I click on my navbar link to home. Here's the html in that link.

href="{{ url_for('home') }}"
flask nginx python
0 Answers
Related