Sending header and body at once
September 19, 2012
This article explains one technique to improve the performace of the simple-sendfile package on which Warp relies. Readers are supposed to read "Improving the performance of Warp".
Before Andreas suggested me that the open()/close() system calls are a performance bottleneck, I found another bottleneck in the simple-sendfile package. Before talking how to avoid open()/close(), I would like to start with how to fix the latter bottleneck. I believe that this is a good introduction to understanding the architecture of Warp.
Warp is an HTTP engine for WAI (Web Application Interface). It runs WAI applications over HTTP. The type of WAI applications is as follows:
type Application = Request -> ResourceT IO Response
That is, an application takes
Request and returns
Response. As you can guess, Warp first receives an HTTP request
from a client and parses it to
Request. Then, Warp gives the
to an application and takes a
Response from it. Finally, Warp builds
an HTTP response based on
Response and sends it back to the
As I pointed out in "Mighttpd – a High Performance Web Server in Haskell", "system calls are evil for network programming in Haskell". So, Warp is defined to use as few system calls as possible.
In the HTTP request parser, Warp uses
Network.Socket.ByteString.recv which issues
the recv() system call. If a client uses HTTP pipelining,
multiple HTTP requests can be received with one recv().
In the HTTP response builder, the situation is complicated because
Response has three constructors:
ResponseBuilder Status ResponseHeaders Builder ResponseSource Status ResponseHeaders (Source (ResourceT IO) (Flush Builder)) ResponseFile Status ResponseHeaders FilePath (Maybe FilePart)
Network.Socket.ByteString.sendAll- based on the send() system call-
is used to send both an HTTP response header and body with a fixed buffer.
ResponseFile, the old Warp used
Network.Socket.ByteString.sendMany to send an HTTP response header
Network.Sendfile.sendfile to send a file as an HTTP response body.
They are based on the writev() and sendfile() system call, respectively.
The combination of writev() and sendfile() had a problem.
When I measured the performance of Warp, I always did it with high concurrency. That is, I always make multiple connections at the same time. It gave me a good result. However, when I set the number of concurrency to 1, I found Warp is really really slow on Linux and FreeBSD.
I realized that this is because Warp uses
the combination of writev() and sendfile().
In this case, an HTTP header and body are sent in separate TCP packets.
To send them in a single TCP packet (when possible),
I implemented a new function called
which is available in simple-sendfile v0.2.4 or later.
On Linux, it uses the send() system call with the
MSG_MORE flag to store
a header and the sendfile() system call to send both the stored header and
a file. On FreeBSD, it uses the sendfile() system call with the header
arguments to send both a header and a file. This trick ensures that
both a header and a body is sent in a single TCP packet (if the file is small enough).
This makes Warp at least 100 times faster when measured with
I will explain how to avoid the open()/close() system calls
Network.Sendfile.sendfileWithHeader in the next article.