My colleague and I have been testing the Requests Per Second (RPS) performance of a simple Twisted Python web server + Nginx load balancing proxy server on three different machines, each with significantly different results. Our specific concern is with Machine 3, an AWS Lightsail instance with roughly the same hardware specifications as Machine 1 and Machine 2. We are running the same code, discussed below, on all three machines. Machine 3 appears to be able to only handle 1/10th the number of requests that Machines 1 and 2 can handle.
(The code below is the same for each machine).
Versions
Machine 1
- Ubuntu 20.04
- Nginx 1.18
- Python 3.7.4
- Ip address (dummy IP for presentation) = M1
Machine 2
- Ubuntu 18.04
- Nginx 1.14
- Python 3.7.4
- Ip address (dummy IP for presentation) = M2
Machine 3 (AWS Lightsail Instance)
- Ubuntu 18.04
- Nginx 1.14
- Python 3.7.4
- Ip address (dummy IP for presentation) = M3
Server Configuration
Server configuration, server.conf in conf.d nginx directory, on each machine is equivalent to the following setup:
upstream backend {
least_conn;
server localhost:8000;
server localhost:8001;
server localhost:8002;
server localhost:8003;
server localhost:8004;
server localhost:8005;
server localhost:8006;
server localhost:8007;
}
server {
listen 80;
location /nginx_status {
stub_status;
}
location / {
proxy_pass http://backend;
}
}
Nginx Configuration
The nginx.conf file looks like this:
user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log warn;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
use epoll;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
sendfile on;
tcp_nopush on;
tcp_nodelay on;
keepalive_timeout 300;
keepalive_requests 300;
proxy_buffering off;
include /etc/nginx/conf.d/*.conf;
}
Simple Twisted POST request Handler
For this post I simply have a Twisted Resource class that handles get and post requests.
from twisted.web import resource
class Server_class(resource.Resource):
isLeaf = True
def render_POST(self,request):
### Does stuff here ###
def render_GET(self,request):
### Does stuff here ###
I then setup 8 threads on the ports 8000-8007:
Create Server Threads
if __name__ == "__main__":
#This is normally done via list comprehension, writing out for clarity of presentation
s0 = server.Site(Server_class())
s1 = server.Site(Server_class())
...
s7 = server.Site(Server_class())
#Attaches Server_class instances to specific port
reactor.listenTCP(8000,s0)
reactor.listenTCP(8001,s1)
...
reactor.listenTCP(8007,s7)
reactor.run()
Test Script
The final file I have is a test script, call it request_test_script.py:
import multiprocessing as mp
import requests
import threading as th
####List to hold all responses received####
manager = mp.Manager()
all_responses_received = manager.list()
def Send_HTTP_Post_Request(session):
global all_responses_received
url = "http://specific.ip.address"
r = session.post(url, headers=...,data = ...)
all_responses_received.append(response_information)
return
def Test_Method():
global all_responses_received
session = requests.Session()
ticker = th.Event()
pooler = mp.Pool(mp.cpu_count())
x = 0
frequency = .001 #This number is changed to test different RPS rates
while not ticker.wait(frequency) and x != 10000:
pooler.apply_async(Send_HTTP_Post_Request,args = (session))
x = x + 1
print(all_responses_received)
return
if __name__ == "__main__":
Test_Method()
The test results are presented below. (Machine 1 to Machine 2 means Machine 2 was running the Server_class threads and Machine 1 was running the test script).
Test Results
- Machine 1 to Machine 1: 2000+ RPS
- Machine 1 to Machine 2: 1000+ RPS
Machine 1 to Machine 3/AWS Lightsail: 150 RPS
Machine 2 to Machine 1: 2000+ RPS
- Machine 2 to Machine 2: 1000+ RPS
Machine 2 to Machine 3/AWS Lightsail: 200 RPS
Machine 3/AWS Lightsail to Machine 3/AWS Lightsail: 280 RPS
Notes: Machine 1 and Machine 2 were on the same local network. Machine 3, the AWS Lightsail instance, is located in the US. Network latency between Machine 1 and 3 and Machine 2 and 3 is roughly 25ms. It takes roughly .2ms for the thread to process the request and send out a response (This is the code in the render_POST method).
Again, each machine has roughly equivalent CPU processing power, memory storage, etc. Machine 1 and Machine 2 are two machines on my local network so they could talk directly to each other, whereas requests to and from the AWS Lightsail instance were remote.
Monitoring Results
When I check http://IP_Address/nginx_status for all three machines, none show evidence for dropped packets. When I send 10000 requests at a rate 1000+ rps from Machine 1 to Machine 2, http://IP_Address/nginx_status shows roughly 10000 requests received. This is the case for tests from Machine 2 to Machine 1, Machine 1 to Machine 1, and Machine 2 to Machine 2 (Sometimes a couple of requests might get missed, not a serious issue).
However, when I send 1000+ rps to Machine 3, nginx_status shows only about 2500-3000 of the 10000 requests actually being received and responded to. It doesn't show dropped packets, it just doesn't seem to receive all the requests. At low rates, e.g. 200 rps or 100 rps, Machine 3 receives and handles all requests just fine.
Conclusion
My colleague and I have tested a range of potential solutions on Machine 3 to improve rps rates, including Nginx configurations, monitoring port activity, and temporarily disabling firewalls. These adjustments have made minimal impacts on the number of requests machine 3 could handle. All monitoring procedures taken show the server handling all the requests it does receive, it just doesn't seem to be receiving all the requests.
We are wondering what could be causing such dramatic difference in test results. Is it network issues or machine specific issues? Is this AWS specific? Is there something with the private network that Machine 1 and 2 are on that could be causing issues? Is there something in the network that could be bottlenecking the number of requests that can go out? If that's the case, how can we test/fix it? We are unsure what more we can test or improve upon.
Thanks!