Experimental IRC log happs-2008-03-21

Available formats: content-negotiated html turtle (see SIOC for the vocabulary)

Back to channel and daily index: content-negotiated html turtle

These logs are provided as an experiment in indexing discussions using IRCHub.py, Irc2RDF.hs, and SIOC.

00:07:32<mae>does happs scale well with serving files as well?
00:07:38<mae>i mean, it doesn't block, does it?
00:14:58<MarcWeber>mae: What do you mean by blocking? It starts a new thread for each request.. So in this sense it's not blocking.
00:15:09<mae>i see
00:15:16<mae>the most expensive part seems to be initiating the tcp connection
00:15:29<MarcWeber>But I'm not sure what happens if you start serving 300MB files. You have to look at the response data type.
00:15:32<MarcWeber>One sec
00:16:12<MarcWeber>mae: rsBody :: L.ByteString
00:16:18<MarcWeber>is used in data Respons = ...
00:18:19<MarcWeber>I'm not too familiar with lazy bytestrings.. So I don't know what happens. I've read that it allocates chunks and reads those chunks in sequence (kind of string builder ?).. But I'm not sure wether it will block whle reading say a 400MB movie to mem and then start serving it? Hopefully it's lazy enough to start serving before having read the whole movie. But can't tell you yet. Maybe you have to try it?
00:21:28<MarcWeber>mae: Have a look at getFile ..
00:21:46<MarcWeber>HAppS/Server/HTTP/FileServe.hs
00:22:04<mae>ok
00:22:23<mae>does happs support keep alive or does it close the connection each time?
00:22:37<MarcWeber>mae: Try firefox and use firebug.. :)
00:23:35<MarcWeber>I think I've seen some options. Can't tell you.
00:24:19<MarcWeber>In HTTPHandler there is keepAliveC = "Keep-Alive" HAve a look at all occurences ..
00:28:29<MarcWeber>mae: HAve you seen my question at #haskell?
00:28:53<mae>yeah looks like it does support it
01:06:08<Lemmih>mae: You can also use HAppS in conjunction with lighttpd/apache if the built-in server isn't strong enough.
01:06:34<mae>yeah i am just doing various httperf tests
01:06:44<mae>and when i scale it up it drags, the cpu isnt up and neither is the memory use
01:06:48<mae>i just don't understand
01:07:19<mae>ok nm, i think its httperf that sucks
01:07:22<mae>lol
01:07:25<mae>thats quite amusing
01:07:32<mae>httperf goes to 100 perc usage
01:07:35<mae>happs is at 0.5
01:18:43<MarcWeber>mae: There is also ab(2) from apache
01:18:58<mae>heh
01:19:06<mae>i am trying something else
01:19:24<mae>i think it might be my isp or my vps provider doing throttling to prevent DoS
01:19:32<mae>since its from the same ip
01:19:56<MarcWeber>mae: HAppS also contains a HTTP client library function simpleHTTP.. So maybe write your own tool?
01:20:02<mae>heh
01:48:01<mae>well doing httperf seems to show that it is much better on the localhost
01:48:12<mae>i am finding differences between high concurrency and low concurrency but it seems that in general happs handles lots of connections and a few connections with the same efficiency
01:48:29<mae>i think that httperf confounds the tests because it probably uses some slow threads or something
01:48:45<mae>i haven't been able to max out the cpu of happs with it
01:48:47<mae>only like 30 perc
01:48:57<mae>while its taking 30
01:49:09<MarcWeber>mae: Than start it twice :)
01:49:15<mae>heh
01:49:18<mae>well you see
01:49:19<MarcWeber>httpref & httperf ..
01:49:30<mae>it seems to block until it connects all your concurrent conections
01:49:36<mae>so by that logic i accomplished the same thing
01:49:37<MarcWeber>If this does'nt make your cpu get hot try a fork bomb ..
01:49:40<mae>but what happens is
01:49:59<mae>my requests per second is about 477ish with 10 concurrent connections and 150 concurrent connections
01:50:11<mae>and thats doing 100 calls per request
01:50:33<mae>so it doesn't slow down
01:50:47<mae>but the time it takes to receive a response is slowed
01:51:00<mae>(but i think this is httperfs timing methods confounding the results)
01:51:32<MarcWeber>mae: Mmmh. I think you can also change the rts system. And you can compile differntly using real native threads or ghc rts managed threads afaik
01:51:45<mae>right
01:51:51<mae>but the managed threads shouldn't be any slower here
01:52:14<mae>if i push the concurrency up to 200 or so i just get an error message
01:52:18<MarcWeber>I don't have any experience here.. But keep on posting. Then I'll learn a lot.
01:52:19<mae>i think its hitting some sort of limit
01:52:23<mae>httperf that is
01:52:27<mae>hehe
01:52:51<mae>Request rate: 596.7 req/s (1.7 ms/req)
01:52:51<mae>with 10000 sequential connectios
01:53:04<mae>1.7 ms per response.. lol
01:53:11<mae>that is so insane if i compare that to apache
01:53:54<MarcWeber>mae: My tests on my local machine has showed that HAppS is about as fast as apache. (But I haven't tuned any of them).. as long as you only server some characters.
01:54:07<mae>well
01:54:08<MarcWeber>(I've used PHP to make this comparison)
01:54:12<mae>right
01:54:24<mae>i'm more referring to how it behaves under concurrency
01:54:27<mae>apache collapses
01:54:27<MarcWeber>And there are many ways to configure apache..
01:54:32<mae>i've tested it
01:54:42<mae>in fact it will eat up the system threads and won't start unless you reboot
01:54:44<mae>i've had that happe
01:54:58<mae>mpm-worker
01:55:03<MarcWeber>mae: Depends. Three is prefork and another compile time option at least.
01:55:19<mae>they have a new event based core which is similar to lighttpd which doesn't use system threads so then the only upper limit is file descriptors
01:55:32<MarcWeber>Ah. Didn't know that
01:55:40<mae>its experimental i guess
01:55:56<mae>but anyways, my point is, i have tested apaches "best" configuration mpm worker
01:55:59<mae>tweaked the settings
01:56:07<mae>and it hits a concurrency wall
01:56:29<MarcWeber>I've seen this in some charts that lighthttpd does scale much better.
01:56:35<mae>happs seems to be like lighttpd in the sense that, it will continue to act consistently with no or a whole lot of concurrency, the only wall is file descriptors
01:56:49<mae>and also like yaws
01:56:53<MarcWeber>mae: Can you give me your adress ? then I can make a internet benchmark. I've a shared server as well
01:57:07<mae>which address?
01:57:20<MarcWeber>Your adress youve passed to httperf
01:57:24<mae>ah
01:57:44<mae>yeah again, i think my isp is doing something weird if one source creates too many connections
01:57:50<mae>mattelder.org
01:58:02<mae>(my vps provider i mean)
01:58:21<mae>because i was benching from another datacenter i have a box in
01:58:25<mae>and i know bandwidth is NO issue there
01:58:27<mae>; )
01:59:22<mae>i'm running httperf from 2 locations right now ; )
01:59:31<MarcWeber>It's damn slow..
01:59:44<mae>yeah
01:59:47<MarcWeber>I haven't got 10 pages till now
01:59:48<mae>i can't even do it over the net
01:59:55<mae>i think they throttled me
02:00:01<mae>at first it was hauling ass
02:00:10<mae>i'm at 1and1 so they are pretty big
02:00:24<mae>i'm guessing they have bridged traffic shapers and whatnot to curb the DoS attacks
02:00:25<MarcWeber>23s for 10 pages ..
02:00:50<MarcWeber>no way
02:01:32<mae>i can access it fine in my web browser
02:02:04<mae>let me show u
02:03:12<mae>http://hpaste.org/6533
02:03:19<mae>227 req/sec
02:03:40<mae>but i think in this case is entirely dependant on the suckiness of httperf
02:03:47<mae>so now watch me kick up the concurrency:
02:04:39<MarcWeber>mae: Where are you located?
02:04:44<mae>one thing i do notice is that, concurrency ups the cpu alot
02:04:44<MarcWeber>I'm from Germany..
02:04:48<mae>in the states
02:04:53<mae>marc i'm benching this on the server itself
02:05:02<mae>like i said, they are throttling my connectino or something
02:05:14<mae>because over the net i can't bench it anymore
02:05:19<mae>it gets really slow
02:05:37<MarcWeber>mae: Then I no longer wonder that I noly got 10 pages within 10 secs ..
02:06:04<mae>so here is what i am talking about, the same total number of requests is there
02:06:15<mae>but the reqs/minute doubled in this case
02:06:17<mae>http://hpaste.org/6534
02:06:37<mae>so its safe to at least make the conclusion that happs behaves just as good with high concurrency or low concurrency
02:06:43<mae>(until you hit your file descriptor limit)
02:07:25<mae>the response was slightly slower
02:07:32<mae>but just by 100ms
02:07:45<mae>not a bad regression for 100 concurrent people slamming your server
02:09:16<MarcWeber>I'd like to continue working on intergating ghc and vim..
02:09:29<mae>heh
02:09:49<mae>ok marc, sorry to take so much of your time, i hope you enjoyed my conclusions ; )
02:10:07<mae>by the way, that sounds fantastic, vi is definitely my editor of choice
02:10:44<MarcWeber>mae Don't worry. I yell before it's too late :)
02:11:15<mae>ok well i'm not talking to you anyways, i'm talking to #happs in general
02:11:20<mae>so don't take it personally ; )
05:51:06<mae>is Old.Data and New.Data a holdover from
05:51:09<mae>the old way
05:51:20<mae>because this is a different system then Versioned 0 Nothing
05:51:20<mae>etc

Back to channel and daily index: content-negotiated html turtle