--- Log opened Mon May 18 00:00:45 2009
03:42 < mae_> hola, just uploaded hopefully a much more robust version of build auto
03:43 < mae_> if you setup a new guestbook app, try: happstack build auto "cabal build" dist/build/guestbook-server/guestbook-server
10:01 < mae_> good morning!
15:31 < gcollins> is anyone around?
15:37 < abuiles> Hi
15:38 < abuiles> gcollins: How is it going ?
15:38 < gcollins> not bad! i'm getting married in a couple of weeks so i'm pretty busy
15:39 < gcollins> i was wondering if anyone had anything else to say about re-doing the http stack for 0.4
15:39 < gcollins> there are a couple of things everyone's been asking for (sendfile support, simpler monad stack, etc)
15:39 < abuiles> have you checked build auto ?
15:40 < gcollins> no, not yet
15:40 < abuiles> I'm dunno about the formers.
15:40 < gcollins> i'm personally not super-interested in that feature
15:41 < abuiles> I see
15:42 < abuiles> Did you know something about lockfile  ?
15:42 < abuiles> I mean, I don't know what is that about ....
15:43 < gcollins> I know a little about lockfile stuff, just from programming on unix for years
15:45 < abuiles> any reference ?
15:47 < abuiles> never mind,, I just went to wikipedia
15:48 < mae_work> gcollins: you don't find "build auto" useful? :)
15:49 < mae_work> i was thinking it quite handy
15:49 < gcollins> I haven't tried it yet, actually -- right now I'm mostly treating happstack like a convenient http library
15:49 < gcollins> i don't use the state stuff either :(
15:49 < mae_work> heh
15:50 < mae_work> well basically it can be used for anything that you build with ghc
15:50 < mae_work> it abuses ghc to find out if there are changes (because ghc only links a new binary if there is)
15:50 < gcollins> i just skimmed the code -- it runs that periodically in a loop, right?
15:50 < mae_work> so anything that you are creating with either a build script or is using something like cabal, can use it
15:50 < mae_work> yeah
15:50 < mae_work> builds then waits 5 secs
15:50 < gcollins> a nice-to-have there would be a binding to inotify
15:51 < mae_work> not a mac user :)
15:51 < mae_work> "i'm not super interested in that feature"
15:51 < gcollins> :)
15:51 < mae_work> you mean for growl right
15:51 < gcollins> no -- inotify is a filesystem-watching api. so you can watch a directory and get a notification if the contents change
15:52 < mae_work> but yeah basically the way i use it in the guestbook app (it has a cabal file)
15:52 < gcollins> on osx i think you'd use epoll
15:52 < mae_work> happstack build auto "cabal build" dist/build/path/to/my/output/bin
15:52 < gcollins> and i'm writing you from my macbook :)
15:52 < mae_work> and then tail -f build.err.log
15:52 < gcollins> what's the use case for this thing, btw?
15:52 < mae_work> (stdout and stderr from the build command go to build.out.log and build.err.log)
15:52 < gcollins> it's supposed to make development more convenient?
15:52 < mae_work> um use case is, i don't wanna have to type in cabal build every time
15:53 < mae_work> yeah basically
15:53 < mae_work> at first i mucked around with ghci
15:53 < mae_work> but the dependency check is a completely different implementation from cabal
15:53 < mae_work> and it still has weird issues with template haskell
15:53 < gcollins> i usually do most of my haskell development from inside emacs
15:53 < mae_work> emacs does this?
15:53 < mae_work> i am so used to vi i could never get into emacs
15:53 < gcollins> C-c C-l punts the current buffer over to ghci
15:54 < mae_work> there are disadvantages with ghci as well
15:54 < mae_work> ie performance will seriously drag depending on how fat your stuff is
15:54 < mae_work> you fix that by building some of the modules
15:54 < mae_work> so they aren't interpreted
15:54 < gcollins> also emacs has a neat integration with the interpreter so if you put the cursor over a symbol it'll go look up the type in the minibuffer
15:54 < mae_work> in bytecode
15:54 < mae_work> yeah that kicks ass
15:54 < gcollins> yeah i wouldn't run a production system from ghci
15:54 < mae_work> i guess it really depends
15:55 < mae_work> if your dev cycle is code/build/check browser
15:55 < mae_work> i think that is definitely the use case
15:55 < mae_work> but it is not limited to that
15:55 < gcollins> yeah it doesn't work so well with my workflow
15:55 < mae_work> what is your workflow? :)
15:56 < gcollins> which is write a function, punt to interpreter, clean up errors/warnings, THEN build a binary
15:56 < gcollins> emacs makes the repl-loop style really convenient
15:56 < gcollins> actually i've been meaning to talk to you about the http stuff
15:57 < gcollins> i have a feeling that something is pretty broken with http pipelining inside happstack
15:58 < gcollins> at work i'm writing an analytics-gathering engine using happstack -- when i connected my test program to the server using pipelining (on linux) performance went through the floor
15:58 < gcollins> strace said that it was spending 99% of its time in "futex"
15:58 < gcollins> waiting on some lock, either in happstack (not so sure about that) or in the runtime system (more likely)
15:59 < gcollins> honestly i'm a little bit leery of the http stuff in happstack -- the lazy i/o worries me
16:02 < abuiles> mae_work: Hey! ,, yeah I'm sure I rebuild again.. however, I'm doing it again.
16:03 < gcollins> brb
16:08 < gcollins> back
16:30 < mae_work> gcollins: so you think the ghc runtime might be to blame?
16:31 < mae_work> or rather the lazy-io impl
16:31 < mae_work> i do see strange things happening with handles all the time
16:31 < gcollins> it's possible -- i found a temporary workaround and didn't investigate further
16:32 < gcollins> i tried to build it with GHC head (because people suggested that they'd fixed a lot of parallelism bugs) but the program segfaulted
16:32 < mae_work> i would really like to look into using a tighter allocation loop, something like the enumerator in hyena
16:32 < mae_work> not relying on lazy io
16:32 < mae_work> too unpredictable
16:32 < gcollins> i think that oleg guy made a good argument for that
16:33 < gcollins> the question is whether to go with an enumerator-style or imperative-style implementation
16:33 < mae_work> well really, i think that the actual request loop is very small
16:33 < mae_work> i wonder how hard it would be to swap it out
16:33 < gcollins> not hard at all i think
16:33 < gcollins> i've been thinking about doing that
16:33 < mae_work> what are the pros and cons of each?
16:34 < mae_work> i know imperative is more error-prone, but probably more idiomatic to most people
16:34 < gcollins> honestly i don't know enough about that left-fold enumerator stuff to be able to comment
16:34 < mae_work> i know little about enumerators except that I guess it would use linear memory in theory right? using tail recursion?
16:34 < gcollins> you know what we should do: use the server functions from the HTTP library
16:35 < gcollins> yeah, i think it allows you to process things in chunks
16:35 < mae_work> http://github.com/tibbe/hyena/blob/c981126f941ced0827dcef7eedff45842cd5bd91/Hyena/Http.hs
16:35 < gcollins> the downside of that approach is that it's a bit of a mind-bender at first
16:35 < mae_work> one thing i have considered is to simply not support http 1.0 (1.1 + only) and then we can treat everything as chunked encoding
16:36 < mae_work> alleviating a number of problems
16:36 < mae_work> this sort of assumes the app server pattern though
16:36 < gcollins> yes
16:36 < mae_work> some people want to use it as a front facing server
16:36 < gcollins> i think http/1.0 support is a must
16:36 < tibbe> mae_work: you called? ;)
16:36 < gcollins> personally i'm proxying it through lighttpd right now
16:36 < mae_work> tibbe: hehe how is hyena coming?
16:36 < tibbe> mae_work: slowly but surely
16:37 < mae_work> gonna release a cabal soon?
16:37 < tibbe> mae_work: I'm reimplementing the internals, it'll support pipelining and all sorts of cool stuff
16:37 < gcollins> that's nice code, kudos
16:37 < gcollins> very clean
16:37 < mae_work> gcollins: i have sort of assumed in the back of my mind that i would use hyena eventually, and maintain what we have for now (unless someone comes up with something better that is)
16:37 < tibbe> mae_work: I'm doing some tricky CPS stuff now, I hope that once I sort that out I can make a release
16:38 < mae_work> CPS?
16:38 < tibbe> mae_work: let me put the code somewhere
16:38 < tibbe> mae_work: sec
16:39 < gcollins> i don't think it matters which http engine to use, as long as it's fast and will let us plumb in sendfile()
16:39 < mae_work> thats true.
16:39 < gcollins> CPS = continuation-passing style
16:39 < mae_work> and that its compatible ?:)
16:39 < mae_work> that is what i like about the engine currently
16:39 < tibbe> mae_work: http://www.hpaste.org/fastcgi/hpaste.fcgi/view?id=5008#a5008
16:39 < mae_work> it has some bugs, but a lot of compatibility stuff has already been tested
16:39 < gcollins> we're working to an interface here, any engine can be made to work
16:40 < tibbe> mae_work: implementing all the parsing combinators in this style seems to be faster
16:40 < mae_work> gcollins: right
16:40 < tibbe> gcollins: to be honest I'm not sure how to expose sendfile in hyena
16:40 < gcollins> i have a feeling that the HTTP library implementation is pretty good
16:40 < abuiles> guys , is any way ? of doing unregister in cascade? I Have happstack-data-0.2.1 happstack-ixset-0.2.1 happstack-server-0.2.1 happstack-state-0.2.1 happstack-0.2.1 happstack-helpers-0.22  , but I have the latests as well. I'm just gonna leave 0.3
16:40 < gcollins> tibbe: allow access to the raw socket
16:40 < gcollins> i mean
16:41 < gcollins> that's a little facile
16:41 < mae_work> tibbe: sendfile needs a raw socket and a filename, the rest is operating-system-level magic
16:41 < tibbe> gcollins: yes that's probably the way to go
16:41 < tibbe> gcollins: network-bytestring might come with a wrapper for it soon
16:41 < gcollins> just advertise it as "unsafe"
16:41 < tibbe> gcollins: so it works on Windows
16:41 < gcollins> nice!
16:41 < gcollins> i mean, the C shim for that wouldn't be too nasty to write
16:41 < mae_work> wrapper for what?
16:41  * tibbe needs a windows hacker for network-bytestring
16:42 < mae_work> windows nt supports sendfile (but its call transmitfile)
16:42 < tibbe> mae_work: for the unix and windows variations of sendfile
16:42 < mae_work> ah ok
16:42 < mae_work> so network bytestring has this licked already?
16:42 < gcollins> tibbe: i finally scrubbed my house of windows machines about six months ago
16:42 < mae_work> or rather will
16:42 < tibbe> network-bytestring now supports scatter/gather I/O
16:42 < tibbe> mae_work: it will
16:42 < mae_work> ok
16:43 < tibbe> mae_work: been short on time lately
16:43 < gcollins> i know the feeling
16:43 < mae_work> gcollins: so then, http library in core sounds reasonable, but does it / will it support sendfile?
16:43 < tibbe> the iteratee stuff is a bit mind boggeling on the implementation side
16:44 < gcollins> well -- i haven't actually tested how robust it is, but i've poked in there
16:44 < tibbe> I've been staring at the implementation of one of the parsing combinators for a while now, I have it working in direct style but converting it to CPS is tricky
16:44 < gcollins> it'll definitely support sendfile
16:44 < mae_work> heh
16:45 < mae_work> tibbe: what are you developing hyena for?
16:45 < gcollins> receiveHTTP :: HStream ty => HandleStream ty -> IO (Result (Request ty))
16:45 < gcollins> respondHTTP :: HStream ty => HandleStream ty -> Response ty -> IO ()
16:45 < mae_work> gcollins: so you see happstack-server as a higher level library that can run on any http engine potentially right? (this is how i see it)
16:45 < gcollins> it's pretty low-level --- instead of respondHTTP you could write a custom handler that used sendfile
16:45 < tibbe> mae_work: I want a web application server "to rule them all" so web framework writers don't have to roll their own every time
16:45 < gcollins> mae_work: pretty much
16:46 < tibbe> mae_work: I plan to standardize some interface (ala Python's WSGI and Ruby's Rack)
16:46 < mae_work> gcollins: still has a way to go before it deserves that place :) like the stuff that was on the ml
16:46 < mae_work> tibbe: great
16:46 < mae_work> tibbe: you must support sendfile!
16:46 < mae_work> then we won't have to do that work :)
16:46 < gcollins> yeah it's pretty alpha still but so is everything else
16:47 < gcollins> i settled on happstack because it looked more complete than the other options
16:47 < mae_work> tibbe: what windows questions do you have (regarding network-bytestring) I wouldn't call myself a "windows hacker" but I actually implemented several kernel32 ffi interfaces w/ c2hs recently
16:47 < gcollins> i think we should deprecate ServerPartT honestly
16:48 < mae_work> right, very few cases where you would want a WebPartT but not a ServerPartT I g uess
16:48 < mae_work> guess *
16:49 < mae_work> and with the typeclasses we added in 0.2, it almost becomes transient now
16:50 < gcollins> the whole thing can be collapsed down to something much less complicated
16:50 < gcollins> and i'd prefer the datatype to be more abstract honestly -- ServerPartT makes you know way too much about its "guts"
16:52 < tibbe> mae_work: I basically need someone to wrap the windows equivalents of the readv/writev system calls
16:53 < tibbe> mae_work: I don't have a machine to test on
16:53 < mae_work> tibbe: i can probably do that
16:53 < mae_work> you have a test suite or something? so i can make sure the patch works before i send it to you?
16:54 < tibbe> mae_work: there are some tests
16:54 < tibbe> mae_work: if you have something that kinda works I could try to expand the test suite
16:55 < tibbe> (testing socket code sucks in comparison to pure code)
16:55 < abuiles> mae_work: I just checked. I rebuild and reinstall, and the process continues runn?ing.  in the other hand, lockfile is working as expected :)
16:55 < mae_work> tibbe: ok, well i mean I just don't know what all your looking for in your test (to make sure it behaves similarly to what you expecT)
16:55 < mae_work> hmm
16:55 < mae_work> thats strange
16:55 < tibbe> mae_work: http://github.com/tibbe/network-bytestring/tree/master
16:55 < mae_work> abuiles: what platform
16:56 < abuiles> linux mint
16:56 < abuiles> felicia
16:56 < mae_work> thats right
16:56 < tibbe> mae_work: the test suite basically creates a server/client pair and sends a string between them
16:56 < tibbe> mae_work: check the tests/ directory
16:56 < abuiles> if you want I can show you my screen,, vnc
16:57 < tibbe> mae_work: you could cut-n-paste the testSendAll and replace sendAll by sendMany
16:58 < mae_work> um
16:58 < mae_work> abuiles: can't right now
16:58 < mae_work> abuiles: what is the exact command you are entering?
16:59 < mae_work> tibbe: so you already have the windows stuff in here no? http://github.com/tibbe/network-bytestring/blob/3f33f16bb846458b28ad6453e40de22aa8ad00ba/Network/Socket/ByteString/Internal.hs
16:59 < abuiles> mae_work:  happstack build auto "cabal configure & cabal install" ./dist/build/guestbook-server/guestbook-server
16:59 < tibbe> mae_work: yes, contributed by someone else
16:59 < mae_work> tibbe: oh
17:00 < mae_work> so err, it is complete, but not tested/
17:00 < mae_work> ?
17:00 < tibbe> mae_work: it's incomplete
17:00 < tibbe> mae_work: check the implementation of Network.Socket.ByteString.sendMany
17:01 < tibbe> mae_work: there are instructions for contributing in the README, I accept plain diffs if you don't want to set up git
17:03 < mae_work> hm ok
17:03 < abuiles> mae_work: In order to kill the app, I have to do 2 times ctrl-c, dunno if that is a strange behaviour .
17:04 < mae_work> abuiles: so you don't get the prompt back until a second ctrl-c?
17:04 < abuiles> mae_work: exactly .
17:06 < abuiles> could be something of the distro  ?
17:08 < mae_work> tibbe: can't find a winsock function that looks similar to readv and writev, http://msdn.microsoft.com/en-us/library/ms741394(VS.85).aspx
17:08 < mae_work> abuiles: nah, what version of ghc?
17:09 < abuiles> 6.10.2
17:13 < tibbe> mae_work: WSARecv
17:13 < mae_work> nm found it
17:13 < mae_work> http://msdn.microsoft.com/en-us/library/aa365469.aspx
17:13 < mae_work> ReadFileScatter Function
17:13 < tibbe> mae_work: or so I've read
17:13 < mae_work> win2k +
17:14 < mae_work> WriteFileGather
17:14 < mae_work> http://msdn.microsoft.com/en-us/library/aa365749(VS.85).aspx
17:14 < tibbe> mae_work: http://msdn.microsoft.com/en-us/library/ms741688(VS.85).aspx
17:14 < tibbe> mae_work: has several buffers
17:14 < mae_work> hmm
17:15 < mae_work> wonder what the difference is
17:15 < mae_work> between WSARecv and ReadFileScatter
17:16 < abuiles> mae_work : which version of ghc are you using ?
17:16 < mae_work> abuiles: 6.10.1 last i checked i think
17:16 < mae_work> it could be that the new ghc handles ctrl-c differently
17:16 < mae_work> because there are two threads which need to die
17:17 < abuiles> mae_work: I'm going to install 6.10.1 to check. whether it works or continues the same.
17:17 < tibbe> mae_work: not sure, I think the WSA one is the one to use as the other WSA functions are socket functions
17:18 < mae_work> ah
17:18 < mae_work> so ReadFileScatter is probably more high level
17:18 < mae_work> ah ok
17:18 < mae_work> ReadFileScatter takes a file handle
17:19 < mae_work> WSA takes a socket
17:19 < mae_work> (these are the same on linux no?
17:21 < mae_work> tibbe: when will you implement sendfile for network bytestring?
17:21 < tibbe> mae_work: patches welcome ;)
17:21 < tibbe> mae_work: otherwise as soon as I have time
17:21 < tibbe> mae_work: I really want to get a first version of hyena out there
17:21 < mae_work> tibbe: if you implement sendfile, I will work on the windows portion of readv/sendv and also sendfile
17:21 < mae_work> tibbe: 10-4
17:22 < tibbe> mae_work: I can't promise you a date but I'll try to get to it
17:22 < tibbe> mae_work: if you "follow" me on github you'll see when it's done
17:22 < mae_work> heh
17:22 < mae_work> ok
17:23 < mae_work> gcollins: ok so stepping up a few semantic levels higher, can you organize your proposals for overhauling the http stack into a wiki page?
17:24 < gcollins> sure. but not for a while, my wedding & honeymoon are coming up soon
17:24 < mae_work> gcollins: ok
17:24 < mae_work> congratulations :)
17:24 < gcollins> i actually will probably have some time to code in budapest
17:24 < mae_work> right
17:24 < gcollins> thanks! you have a new child coming soon too, right -- or has he/she arrived yet?
17:25 < mae_work> very soon
17:25 < mae_work> in a week or so
17:25 < mae_work> trying to release 0.3.1 before that :)
17:25 < mae_work> apparently alex jacobson just added cabal support to sp, somewhat negating the need for my "auto build" tool
17:25 < mae_work> but oh well
17:26 < mae_work> I really see happstack evolving with the http stuff for awhile
17:26 < mae_work> until it feels tight and nice I don't think I wanna delve into the state stuff too deeply
17:27 < abuiles> sorry to ask, but what is sp ?
17:28 < mae_work> searchpath
17:29 < abuiles> than
17:29 < abuiles> *ks
17:31 < mae_work> gcollins: ok well, if you do have time, I would love to see a wiki page, lets try to be as concrete as possible though, so we can move from idea to patches without too much fuss
17:31 < mae_work> I want to clean house
17:31 < gcollins> i like the state stuff in theory but databases are really tricky and it'd take a lot to get me to trust it with anything important
17:31 < gcollins> ok -- i'll spend some time thinking about it for sure
17:32 < mae_work> i wonder if creighton ever finished porting all that crypto stuff so we can move that dep outwards
17:33 < mae_work> gcollins: yeah thats the problem, it will take a lot of work to become "production ready", but a native haskell database could definitely cut the time of development down if done right, it is still in the "experimental" stage
17:33 < mae_work> hell, we are using it for patch-tag though, hasn't had any problems yet
17:34 < gcollins> do you think it'll scale to millions of transactions daily?
17:34 < gcollins> i don't trust ORMs either for what it's worth -- i've seen too much rotten SQL come out of things like hibernate et al
17:36 < mae_work> same here
17:36 < mae_work> i used rails for awhile
17:36 < mae_work> if i went back to sql dev, i'd probably handcode the routines
17:36 < gcollins> oh man --- don't get me started on that stuff
17:36 < mae_work> heh
17:37 < mae_work> the whole idea of RAD (rapid application development) is a pipe dream, because you always have to end up coming full circle anyways, and realizing the tool has given you dogshit
17:38 < gcollins> yeah -- the people involved in that stuff want to make out like they've invented this mindblowing new thing that's soooo revolutionary
17:39 < gcollins> meanwhile it's 200x slower than C
17:39 < mae_work> the compelling thing for me about MACID
17:39 < mae_work> is that you cut out the query bottleneck
17:39 < mae_work> so for large, flexible, accessible, volumes of data, sql is very good
17:39 < gcollins> and those of us who have been around computers for more than a couple of years are like "we've read this script before"
17:40 < mae_work> but for serving up queries with raw speed, you can't beat in-memory
17:40 < gcollins> that's definitely true -- your dataset has to fit in memory though
17:40 < mae_work> hehe
17:40 < mae_work> memory is cheap now
17:40 < mae_work> : )
17:40 < mae_work> + most of your data is thin
17:40 < mae_work> and binary data belongs on a filesystem
17:40 < mae_work> or s3 or something
17:40 < mae_work> (if its lots of it)
17:41 < mae_work> same argument applies there too
17:41 < mae_work> "all your data has to fit on the filesystem"
17:41 < gcollins> that'd work for the majority of apps but there are datasets it'd break for in a couple of days
17:41 < mae_work> you just need to fix your perception :)
17:41  * tibbe is of the believe that relational databases don't scale.
17:42 < gcollins> tibbe: i agree
17:42 < tibbe> but then I work on YouTube
17:42 < gcollins> i'm actually working on something pretty neat in this area
17:42 < tibbe> I want something like Amazon's Dynamo written in Haskell
17:43 < tibbe> Dynamo is essentially a persistent memcache with some extra bells and whistles
17:43 < gcollins> tibbe: i'm working on something similar to that (not so fancy, just key-value-pair) backed with tokyo tyrant
17:43 < tibbe> gcollins: sounds interesting
17:43  * abuiles feel like a kid listening to adults conversations.
17:44 < gcollins> it'll be very similar to "lightcloud", just a thin consistent-hashing frontend
17:45 < gcollins> the consistent-hashing stuff was super-simple, i've finished it already -- the "real work" begins with the server part of it
17:46 < gcollins> ideally it should be transparent to the user that you're talking to a distributed key-value store, i'll probably implement the tokyo-tyrant protocol (with an authenticating frontend)
17:47 < tibbe> ok
17:47 < gcollins> the tricky thing is writing good tools to add/remove nodes from the cluster
17:47 < tibbe> it needs to be reliable and fast
17:47 < gcollins> agreed
17:47 < tibbe> there are some nice OS implementations out there
17:47 < gcollins> tokyo tyrant is pretty much the fastest key-value store i've found
17:47 < gcollins> i'm a really big fan
17:48 < tibbe> gcollins: http://project-voldemort.com/
17:48 < tibbe> gcollins: I'll check it out tomorrow
17:48 < gcollins> yep, i've looked into that one
17:48 < gcollins> the java dependency makes me turn up my nose a bit
17:48 < gcollins> but the design looks solid to me
17:49 < tibbe> I meant the design more than the implementation
17:49 < tibbe> something to use as inspiration
17:49 < tibbe> bed time
17:49 < tibbe> gnight
17:50 < gcollins> night
17:51 < abuiles> night
18:02 < abuiles> gcollins: how do you run your site, with cgi or something like that ?
18:02 < gcollins> reverse-proxied through lighttpd
18:05 < mae_work> yeah
18:05 < mae_work> thats my weapon of choice
18:06 < mae_work> (if i'm doing it end-to-end)
18:06 < mae_work> because of the nice proxy balancer
18:06 < mae_work> but I think AWS now has an ip-level load balancer service
18:06 < mae_work> which is nice
18:07 < gcollins> we use amazon at work, i should bring that up -- we've been using haproxy on our frontend
18:07 < gcollins> all of that stuff is apache/mod_wsgi/python
18:20 < mae_work> what is
18:20 < mae_work> the amazon stuff?
18:23 < gcollins> no, our stuff
18:24 < mae_work> ah
18:24 < gcollins> although we're rolling out our first haskell server soon
18:24 < mae_work> neat.
18:24 < mae_work> where do you work?
18:24 < gcollins> given that we'll be using this in production, i may be able to get some 9-to-5 time into happstack in the next few months
18:24 < gcollins> www.polarmobile.com
18:25 < gcollins> we do content delivery to smartphones
18:26 < mae_work> gcollins: re 9-5 time in happstack, great!
18:26 < mae_work> gcollins: how do you feel about moving towards hyena as the core?
18:27 < abuiles> mae_work: indeed, is ghc-10.0.2 problem..
18:27 < gcollins> i'd want to see benchmarks
18:27 < mae_work> i mean really there are two main directions for Happstack.HTTP, one is to clean house, simplify the monad stack, have nice error handling etc..
18:27 < mae_work> two is to move to a http core that doesn't use lazy io
18:27 < gcollins> right
18:27 < mae_work> would one or two be the 9-5 project
18:27 < gcollins> and those are orthogonal to each other
18:28 < gcollins> i'd probably want to improve the http/network stuff first
18:28 < abuiles> mae_work: with ghc-10,0,1 works perfect .
18:28 < mae_work> um
18:28 < mae_work> meaning the core, right?
18:28 < mae_work> abuiles: ok, i'll have to take that into account
18:28 < mae_work> abuiles: thanks for testing that out
18:29 < mae_work> probably there is a slightly different behavior that I have to cater to
18:29 < mae_work> i.e. i might need to explicitly handle a sigterm, for instance
18:29 < mae_work> (want to avoid that though)
18:29 < gcollins> right, the part that renders a Response out the socket
18:29 < mae_work> gcollins: ok
18:30 < abuiles> mae_wokr : not problem man .... Hope I can continue helping out the project ( although I not have the same experience as you guys,,, just a college kid :) ... but very enthusiastic  :) )
18:30 < gcollins> re: simplifying/reworking ServerPart -- that's an aesthetic vs. a correctness issue
18:30 < mae_work> gcollins: i mean, i think hyena in the long run would be beneficial in the sense that there is less maintenance overhead for the project
18:31 < gcollins> I can see it having multiple backends actually
18:31 < mae_work> a more conservative approach would bring better performance and/or behavior (with things like pipe-lining and large files) -- carefully replace a few key items in the http loop
18:32 < gcollins> that's a no-brainer to start at least
18:32 < gcollins> that http inner loop is not a ton of code
18:32 < mae_work> right
18:32 < mae_work> so i was thinking multiple backend was a good idea
18:32 < mae_work> because it gives more time for people to test
18:32 < mae_work> before we switch the "default"
18:33 < mae_work> and we can bench each one independently
18:33 < gcollins> well, it's necessary anyways --- you need to race them :)
18:33 < mae_work> yep
18:33 < mae_work> so step 1, abstract the core from the rest of the bits, make some sort of shim
18:33 < mae_work> step 2, make sure nothing broke with the "legacy" core
18:33 < abuiles> ok guys,, good night,, will be back to irc by thursday.. this time from sweet home....
18:33 < mae_work> step 3, implement new cores
18:33 < gcollins> well, from simpleHTTP' to runServerPartT is not a ton of code
18:34 < gcollins> abuiles: bye
18:34 < mae_work> true
18:34 < gcollins> i think we can just delineate at that boundary
18:34 < gcollins> i imagine the next version will run other things besides ServerPartTs
18:35 < mae_work> yeah, i forget sometimes I am working an a language that facilitates this types of things much easier
18:35 < gcollins> haskell FTW
18:35 < mae_work> gcollins: what do you mean "other things"
18:35 < gcollins> well, you need a mode that gives you access to the raw socket
18:35 < mae_work> ServerPartT was from when CPS was the one true way (tm)
18:35 < mae_work> but now we generalized the transformers and can use a do-block style
18:36 < gcollins> i wrote a mailing list post about it a while back
18:36 < mae_work> i am still not sure that CPS is easy for most people to understand
18:36 < gcollins> http://groups.google.com/group/HAppS/tree/browse_frm/thread/1913f4fea4fb9c65/3a0924db62f7fc80?rnum=1&_done=%2Fgroup%2FHAppS%2Fbrowse_frm%2Fthread%2F1913f4fea4fb9c65%2F3a0924db62f7fc80%3Ftvc%3D1%26q%3Dgregory%2Bcollins%26#doc_969da8b5abb40545
18:36 < mae_work> gcollins: is that really necessary?
18:36 < gcollins> if you wanna be able to use sendfile
18:36 < mae_work> couldn't we simply pass a fun and have that handed a socket at a lower level
18:36 < mae_work> and likewise for sendfile, pass a file path down
18:38 < gcollins> you mean expose "SendFile f" as a datatype constructor on Response?
18:38 < gcollins> and then have the backend deal with it?
18:38 < mae_work> something like that
18:38 < mae_work> yeah
18:38 < gcollins> that's another idea i hadn't considered
18:38 < gcollins> that might actually be better
18:38 < mae_work> i like that better than exposing the raw socket
18:38 < mae_work> at the app level
18:39 < mae_work> it also allows us to not have to "fake" using the filter etc machinery
18:39 < gcollins> hm
18:39 < mae_work> when we are going to ignore it anyways
18:39 < gcollins> Would the backend be a typeclass then?
18:39 < mae_work> perhaps
18:40 < gcollins> either that or a wrapping function
18:40 < mae_work> let me take a look here
18:41 < gcollins> actually yeah, I like the idea of adding a constructor to Response a lot
18:42 < mae_work> yep
18:43 < gcollins> we can just fix fileServe then
18:44 < mae_work> take a look at this
18:44 < mae_work> http://haskell.pastebin.com/m469ed012
18:45 < mae_work> only diff is
18:45 < mae_work> instead of rsBody
18:45 < mae_work> rsFilePath
18:45 < gcollins> i like it -- except won't that be a name clash?
18:45 < mae_work> ?
18:45 < mae_work> with what
18:46 < mae_work> oh i see
18:46 < mae_work> um
18:46 < mae_work> shouldn't be
18:46 < mae_work> since they are the same type
18:46 < mae_work> good q
18:46 < mae_work> see if you can run it through ghci :)
18:47 < gcollins> yep
18:47 < gcollins> i still don't know all of the rules by heart :)
18:49 < mae_work> gcollins: nope it works fine
18:50 < mae_work> so i guess the only issue here is that
18:50 < mae_work> someone could feasibly set the header ContentLength
18:50 < mae_work> in this instance, we must ignore it
18:50 < mae_work> or rather
18:50 < mae_work> overwrite it
18:50 < gcollins> the backend that handles SendFile will set its own content-length based on fstat()
18:50 < gcollins> right
18:51 < mae_work> hmm
18:51 < mae_work> so we have to open the file handle
18:51 < mae_work> to check the size?
18:51 < mae_work> low overhead?
18:51 < gcollins> no
18:51 < gcollins> very low overhead
18:51 < mae_work> http://www.opengroup.org/onlinepubs/009695399/functions/fstat.html
18:52 < mae_work> k
18:52 < mae_work> so given this idea
18:52 < mae_work> this should actually super simple to implement
18:52 < gcollins> or stat()
18:52 < gcollins> rather
18:52 < mae_work> (ignoring the integration into fileServe for now)
18:52 < gcollins> right
18:52 < mae_work> and aside from the C pushups
18:52 < mae_work> and if we move to something like hyena later
18:52 < mae_work> which has sendfile support
18:52 < mae_work> we make the translation
18:52 < gcollins> well for round one you can just send the file out the socket by hand
18:52 < mae_work> but the interface can stay the same
18:53 < mae_work> right
18:53 < gcollins> optimize it later
18:53 < mae_work> oh i see
18:53 < gcollins> like after it gets integrated into network-bytestring
18:53 < mae_work> well
18:53 < mae_work> right ok
18:54 < gcollins> i mean, that'll be the "canonical" way of doing that within a month or two
18:54 < mae_work> i don't see why sendfile semantically even belongs in network-bytestring
18:54 < mae_work> as you will never ever create a bytestring
18:54 < gcollins> and it'll be a 10-line patch to wire it in
18:54 < gcollins> that's a good point :)
18:54 < gcollins> it should probably go into network proper
18:55 < mae_work> before i talked to tibbe today I was simply gonna create a portable sendfile cabal package
18:55 < mae_work> and throw it on hackage
18:55 < mae_work> network seems like the right place
18:55 < mae_work> but acceptance is a pain in the ass
18:55 < gcollins> i'd do that
18:55 < mae_work> besides, the haskell platform will start using hackage libs anyways
18:55 < gcollins> actually
18:55 < mae_work> which?
18:55 < gcollins> release a hackage lib
18:56 < mae_work> yeah
18:56 < mae_work> it can always  move
18:56 < gcollins> because you'll get bug reports, it'll be easier to convince people to put it into network later
18:56 < mae_work> yeppers
18:56 < gcollins> it's cross-platform?
18:57 < mae_work> yeah
18:57 < mae_work> windows and linux
18:57 < mae_work> (windows 2000 or later)
18:57 < gcollins> osx?
18:57 < mae_work> only in windows its called TransmitFile
18:57 < mae_work> yeah osx should have it
18:57 < mae_work> read here
18:57 < mae_work> no clue
18:57 < mae_work> http://sendfile.darwinports.com/
18:57 < gcollins> int
18:57 < gcollins>      sendfile(int fd, int s, off_t offset, off_t *len, struct sf_hdtr *hdtr,
18:57 < gcollins>          int flags);
18:57 < gcollins> is the interface the same?
18:58 < mae_work> for what
18:58 < mae_work> TransmitFile?
18:58 < mae_work> not sure
18:58 < mae_work> but i can make it match
18:58 < mae_work> with a c shim
18:58 < mae_work> http://msdn.microsoft.com/en-us/library/ms740565.aspx
18:58 < gcollins> no i mean that's the osx sendfile call
18:59 < mae_work> i think it is probably the same as linux
18:59 < mae_work> since it seems to be part of standard c lib
19:00 < gcollins> ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);
19:01 < gcollins> it's different
19:01 < mae_work> that seems to be the std one
19:01 < gcollins> the osx one is different
19:01 < mae_work> dammit man
19:02 < mae_work> someone has to have created a portable sendfile shim already
19:02 < gcollins> it's not a ton of work actually
19:02 < mae_work> i don't even begin to know the keywords that would unhide such a diamond in the rough
19:02 < mae_work> gcollins: i know, but then you have to worry about cross platform testing etc
19:02 < mae_work> its not about difficulty
19:03 < mae_work> its about someone whose already went through this pain and tested it for me
19:03 < mae_work> : )
19:04 < gcollins> yeah
19:04 < gcollins> i don't see anything
19:05 < mae_work> auto tools has something i bet :)
19:05 < mae_work> but i don't wanna mess with that
19:06 < mae_work> so ok
19:06 < gcollins> sweet jesus no
19:06 < mae_work> i can't assume that the *nixes have a same impl
19:06 < gcollins> right
19:06 < mae_work> how do i even begin to target them
19:06 < mae_work> arg
19:06 < gcollins> i would pick linux / osx
19:06 < gcollins> or wait for contributors
19:06 < mae_work> i will pick linux :)
19:06 < gcollins> i'd write an osx version, no problem
19:07 < mae_work> yeah
19:07 < gcollins> that's 15 minutes work
19:07 < mae_work> just need to figure out the build flags to target
19:07 < mae_work> right
19:07 < mae_work> ok so i have your signature in blood
19:07 < mae_work> you will work on osx should i create such a library
19:07 < mae_work> i will work on windows and linux
19:07 < mae_work> : )
19:07 < gcollins> i'd also provide a "default" implementation just in case the others fail
19:08 < gcollins> ok
19:08 < mae_work> ie send by hand
19:08 < mae_work> well
19:08 < gcollins> right
19:08 < gcollins> send by hand
19:08 < mae_work> or expect that the person should know what they are doing
19:08 < mae_work> and just throw an exception on a non-supported platform
19:08 < mae_work> : )
19:09 < mae_work> we can create a higher level fun
19:09 < mae_work> like
19:09 < gcollins> speaking as a consumer of this product i really would prefer a unified, no-hassle solution
19:09 < mae_work> well
19:09 < gcollins> i.e. sendFile :: FilePath -> Handle -> IO ()
19:09 < mae_work> right
19:09 < mae_work> what i mean is
19:09 < mae_work> no one should be constructing SendFile
19:09 < mae_work> unless they really know what they are doing
19:10 < mae_work> the type would probably be
19:10 < mae_work> sendFile :: FilePath -> IO Response
19:10 < mae_work> (Response being the Datatype proper, not the instance)
19:11 < mae_work> or
19:11 < gcollins> for happstack, yes
19:11 < mae_work> sendFile :: FilePath -> MimeType -> IO Response
19:11 < gcollins> i meant for a "Network.SendFile" package on hackage
19:11 < mae_work> gcollins: ahh ok so your saying sendfile itself should have a fallback if the platform is unsupported
19:11 < gcollins> right
19:12 < gcollins> so you can still compile and use it on hp-ux (or whatever)
19:12 < gcollins> VMS
19:12 < gcollins> PDP-11
19:13 < mae_work> oh good god
19:13 < mae_work> pdp-11
19:13 < mae_work> you must have read the history of hackerdom
19:13 < mae_work> : )
19:13 < mae_work> or just be a real big geek
19:13 < gcollins> ehm
19:13 < mae_work> = )
19:14 < gcollins> my fiance and i are watching star trek right now
19:14 < mae_work> ahh
19:14 < mae_work> that was a good flick
19:14 < mae_work> wait your in the theatre?
19:14 < gcollins> no, i mean next generation back episodes
19:14 < mae_work> oh oh
19:14 < gcollins> giant dork, yes
19:14 < gcollins> does solaris have sendfile() yet?
19:14 < gcollins> that's the only other relevant platform
19:15 < mae_work> who knows
19:15 < mae_work> who knew that sendfile would be such a hit
19:15 < mae_work> in the end its all about throughput of flat files
19:15 < gcollins> free/open bsd
19:15 < mae_work> because we can't stick it all in memory
19:15 < mae_work> and *nix filesystem is still the best damn way to handle it
19:15 < gcollins> zero copy from kernel/userland is a really good idea
19:16 < gcollins> that i think is the main motivation
19:16 < gcollins> straight from the buffer cache out to the network interface using DMA
19:17 < gcollins> vs. two kernel-userland memory copies and a couple of context switches
19:17 < mae_work> ok so
19:17 < mae_work> once upon a time
19:17 < mae_work> i was watching lighttpd
19:18 < mae_work> is this epoll / linux-specific / asynchronous fd stuff really that great
19:18 < mae_work> or is it just rice for http server designers
19:19 < gcollins> no, it's that good
19:19 < mae_work> is it portable? at all?
19:19 < gcollins> not really. OSX and Windows have their own proprietary interfaces to this kind of I/O
19:19 < mae_work> graceful degradation..
19:20 < mae_work> gcollins: but windows and osx have their own version?
19:20 < gcollins> there's something called "libevent" which abstracts that stuff
19:20 < mae_work> ah
19:20 < gcollins> osx has something called "kqueue" which i think is similar
19:20 < mae_work> ok so
19:21 < mae_work> what do you think of using libevent
19:21 < mae_work> worthit or not so worthit
19:21 < mae_work> probably needs auto tools..
19:21 < gcollins> i think it might be worth it
19:21 < gcollins> actually
19:22 < gcollins> there's this:
19:22 < gcollins> http://hackage.haskell.org/cgi-bin/hackage-scripts/package/hlibev
19:22 < mae_work> yeah, i just hate the idea of not being able to use "cabal install"
19:22 < gcollins> based on this:
19:22 < gcollins> http://libev.schmorp.de/bench.html
19:22 < mae_work> yeah but libev will require manual compilation etc
19:23 < gcollins> maybe it can be an optional backend
19:24 < mae_work> yeah
19:25 < mae_work> but again, just depends how hard you wanna press the scalability pedal
19:25 < gcollins> in theory that would be pretty fast
19:25 < mae_work> what is beautiful about this though is, the power of haskell in the app, with the raw scalability potential by having a very thin layer between the app and the kernel
19:25 < gcollins> yep
19:26 < mae_work> and since haskell is known to be pretty damn fast with hashmaps etc in memory
19:26 < mae_work> combining these two would be awesome
19:26 < mae_work> usually you just get either a really optimized "plain" http server with which you have to hook up cgi (killing your perf) or an app server which is portable with no rice
19:26 < mae_work> : )
19:26 < gcollins> i mean, we've been doing web in languages like php and python and tcl for ages -- getting within 2-3x of C would be fine for 99% of applications
19:27 < gcollins> here's my opinion on that
19:27 < gcollins> it's not a "performance" issue -- it's really a matter of cash money
19:27 < gcollins> if server X can handle double the load of server Y then that's $60/mo I don't have to spend on application servers
19:28 < mae_work> yep
19:28 < mae_work> if your problem is scalability this is true
19:28 < gcollins> of course satisfying performance guarantees is also really important for other reasons
19:28 < gcollins> that's why facebook killed myspace
19:29 < mae_work> whats facebook running on
19:29 < gcollins> php i think?
19:29 < gcollins> lighthttpd maybe?
19:29 < mae_work> heh
19:29 < gcollins> god i can never type that
19:30 < gcollins> youtube runs lighttpd, i think
19:30 < mae_work> i got past that crutch by mouthing it in my head "light-tpd"
19:30 < mae_work> it does
19:30 < mae_work> for video streaming
19:30 < mae_work> bazillions and bazillions of movies
19:30 < gcollins> they definitely have a scalability problem
19:31 < gcollins> which they've solved really really well actually
19:31 < gcollins> that's so highly impressive
19:31 < mae_work> yeah
19:31 < mae_work> then again, they were a startup
19:31 < mae_work> and bought later
19:31 < mae_work> not a child of herman-miller cush blood-sucking startup cash.
19:31 < gcollins> :)
19:31 < mae_work> i am not sure if that even made any sense
19:32 < mae_work> well they were a startup
19:32 < mae_work> they just weren't a megacorp with lots of money to spend on BS
19:32 < gcollins> efficiency is important for sure
19:32 < gcollins> even for medium-traffic sites
19:32 < mae_work> my company tries to throw hardware at everything
19:32 < gcollins> that never works
19:32 < gcollins> well
19:33 < mae_work> so true
19:33 < mae_work> you know
19:33 < gcollins> it temporarily works sometimes
19:33 < mae_work> we have like 500 consumers of an ERP product
19:33 < mae_work> we have 2 netscalers man
19:33 < mae_work> we have several warehouses
19:33 < mae_work> we have a 3-node oracle cluster with each node containing quad-dual cores and 32gb ram
19:33 < mae_work> mind you this was spec'd out about 3 years ago
19:33 < mae_work> and purchased 3 years ago
19:34 < gcollins> yep -- megabucks
19:34 < mae_work> the database is literally only 32gb
19:34 < mae_work> it could fit in the ram of one of them!
19:34 < gcollins> Load average: 0.04, 0.08, 0.03
19:34 < mae_work> yeah heh
19:34 < mae_work> the netscalers are active/passive
19:34 < mae_work> literally at peak times it is like 1.3 percent cpu
19:34 < mae_work> the rest of the time maybe it cracks 0.5
19:35 < gcollins> man -- i've always worked at startups
19:35 < mae_work> but all their performance problems stem from fundamental issues that can't be fixed by throwing hardware at it
19:35 < gcollins> exactly
19:36 < gcollins> i'm a firm believer in the judicious application of computer science
19:36 < mae_work> right now i'm dealing with some business intelligence stuff
19:36 < mae_work> hehe
19:36 < gcollins> that's an oxymoron if i've ever heard one
19:37 < mae_work> : )