Step one… Sysctl parameters. The most fucking evil parameter is net.inet.tcp.inflight.enable. Disable it! Actually I tuned more parameters relevant to TCP/IP stack and kernel internals:
kern.ipc.maxsockbuf=16777216 kern.ipc.nmbclusters=32768 kern.ipc.somaxconn=32768 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.maxvnodes=800000 net.inet.tcp.inflight.enable=0 net.inet.udp.maxdgram=57344 net.inet.udp.recvspace=65536 net.local.stream.recvspace=65536 net.inet.tcp.sendbuf_max=16777216
OK, transfers over FTP were then running at 100Mbps. Fine. But HTTP file transfers were still fucking slow. Locally it was two times slower than FTP and much more slower over the internet (HTTP: 4-10 Mbps vs. FTP: 100Mbps).
Step two... Turning off sendfile(2) in Apache HTTPd.
This is also incredibly fucking option and there is no documentation pointing at it... Even Google does not know what to do when the Apache fucks up on FreeBSD 8.1. Well, I added one single line to /usr/local/etc/apache22/httpd.conf:
And that's it. Both FTP and HTTP are running at full speed.