Storage Success

Our web-based storage system was able to completely saturate the gigabit link to a compute cluster. The serving nodes were two Sun x2100's, each with an older Apple XRAID attached. All access was through the grizzly based webserver. Separate testing was able to show this system could handle between 1200 and 2200 small file requests/s per node.

Being Evil

Someone was using SiteSucker on a demo site and was behaving badly, this site also hosts our redmine install on the same DB and started to slow down my work. To block we had a few options, iptables the IP, block the IP w/in apache (htaccess) or do the evil thing, just block based on their user-agent. I chose the latter.

RewriteEngine on
...
RewriteCond %{HTTP_USER_AGENT} ^SiteSucker2.3.1
RewriteRule ^(.*) /badbot.html

This is evil since the persons crawler will get out badbot.html page. The person will then use firefox/ie/whatever to browse to the page and look to see why it's not working, but since it's sending a differend user-agent, they will be allowed to browse. For anyone who knows how to configure a crawler it isn't an issue changing the supplied agent, but then again that person will likely be able to control their crawler and not kill my web server. Here's a snapshot of the logs showing the crawler pulling badbot (size 174) followed by a browse attempt from safari.

82.232.60.250 - - [09/Mar/2010:19:52:41 -0500] "GET /cgi-bin/isadg/viewitem.pl?item=14379 HTTP/1.1" 200 174 "http://narademo.umiacs.umd.edu/cgi-bin/isadg/viewseries.pl?seriesid=1054" "SiteSucker2.3.1 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBook1%2C1)"
82.232.60.250 - - [09/Mar/2010:19:52:42 -0500] "GET /cgi-bin/isadg/viewitem.pl?item=14377 HTTP/1.1" 200 174 "http://narademo.umiacs.umd.edu/cgi-bin/isadg/viewseries.pl?seriesid=1054" "SiteSucker2.3.1 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBook1%2C1)"
82.232.60.250 - - [10/Mar/2010:11:03:13 -0500] "GET /cgi-bin/isadg/viewrg.pl?rgid=44 HTTP/1.1" 200 2286 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; fr-fr) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"

Test details

Here's the details of the performance testing listed below, hardware etc and the raw IOZone numbers. Client Details:

• Dual-Core AMD Opteron(tm) Processor 1220 (Sun 2100)
• 2.8Ghz, 2GB Memory, 1Gbps network, RHEL4

Server Details:

• Intel Xeon 5120, 1.86Ghz (dell 1950)
• 1GB Memory, 1Gbps network, RHEL5
• Perc5i raid controller setup in 2 7-disk raid 5 volumes
• Disk: Dell MD1000 w/ 750Gb SATA 7200rpm drives

Untuned results (4G test, 4k records) in KB/s

                                                   random  random    bkwd      
     KB  reclen   write rewrite    read    reread    read   write    read 
4194304    4096   93137   68662    28869    29300   23118   57440   23186 

Tuned results:

                                                   random  random    bkwd
     KB  reclen   write rewrite    read    reread    read   write    read
4194304    4096   95360   71281   108200   112603   67761   69528   68227

Notice the huge gains in read performance 28MB/s to 108MB/s for sequential and 23MB/s to 67MB/s for non-sequential. Here's the theoretical max based on running iozone on the server. It looks like nfs is adding overhead on rewrites and possibly random reads and writes with sequential reads/writes bottlenecking at the 1Gbps connection

                                                   random  random    bkwd      
     KB  reclen   write rewrite    read    reread    read   write    read 
2097152    4096  302379  234327   174385   196905  102032  144843   99866

• Raw server performance:
• Untuned performance: xls txt
• Tuned performance: xls txt
• Server to raid xls txt

NFS Performance

We're testing a moderately fast nfs server. Testing on our data set of large files, ~100M we were seeing performance a little over 100MB/s. Now, when we shared the same filesystem out via nfs performance dropped to a mere 44MB/s and no matter what the r/wsize of the client, would not increase. Both machines are connected via a 1Gbps link that has been shown to move 941mb/s. Here's a snapshot what the server workload looked like on the server. The first line shows the local testing, notice the 108MB/s and the second shows the new access. The interesting part is the device utilization during the tests, even though we were pushing half the requests/second and moving half the bandwidth, we were at 90% with a higher wait time 4ms vs 1.7ms.

Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb              30.50     0.00 906.00  0.00 108530.00     0.00   239.58     1.61    1.78   1.08  98.15
sdb             870.00     0.00 500.50  0.00 44626.00     0.00   178.33     2.06    4.09   1.83  91.55

Now, let's tweak the ioscheduling a little by setting slice_idle to 1 as recommended in the dell guide below. 'echo 1 > /sys/block/sdb/queue/iosched/slice_idle' This affects how long the io scheduler will wait in each queue before looking for more work. On local access where you will have lots of low latency requests you probably want to wait for more work, however over a network with lots of clients you will want to switch as soon as possible

sdb            1164.18     0.00 1049.75  0.00 71150.25     0.00   135.56     2.34    2.23   0.87  91.79

let's go a little further and set it to 0

sdb             144.00     0.00 2428.50  0.00 92034.00     0.00    75.79     3.95    1.63   0.33  79.60

Now we're up to 92MB/s which is above 90% of what our network can do. Good enough!

• Dell 10Gbit tuning
• RHEL5 io tuning guide
• description of iosched params