03:50:36 <mm_freak> stepcut: what do you think about not reinventing HTTP serving for happstack 8?  i.e. what do you think about using WAI or something similar?
03:51:01 <mm_freak> after all HTTP serving is pretty much a solved problem
03:52:13 <stepkut> mm_freak: it's actually not a solved problem
03:54:02 <mm_freak> ok, let's not focus on that…  what do you think about using an existing solution?  WAI is sufficiently powerful (it also supports streaming and multiple backends), constantly improved and backed by commercial users
03:54:36 <stepkut> meh
03:54:53 <stepkut> I am not convinced that conduits is superior to pipes
03:54:55 <mm_freak> i know, it hurts to give up originality, given that happstack was the first major web framework =)
03:55:22 <stepkut> well. happstack was aiming to move to wai+hyena before yesod was even around :)
03:55:43 <mm_freak> did WAI exist before yesod?
03:55:48 <stepkut> yes
03:55:58 <mm_freak> oh, ok
03:56:12 <stepkut> as did type-safe urls, and most other features of yesod :)
03:56:16 <mm_freak> well, pipes have a nice theoretical foundation, but i have to say, conduits are more practical
03:56:34 <stepkut> practical in what way?
03:57:01 <stepkut> I've implemented an http backend using pipes and it was both pleasant and fast
03:57:30 <stepkut> my past experiences trying to use things from yesod world have not left me wanting more
03:57:33 <mm_freak> well, it may be that HTTP serving doesn't benefit a lot from conduits, except perhaps that conduits are a bit faster
03:57:38 <mm_freak> to be honest, i still love iteratees
03:58:06 <stepkut> :)
03:58:17 <stepkut> I'm still pretty ok with lazy IO :)
03:58:55 <mm_freak> is that what happstack-server does?
03:58:59 <stepkut> yes
03:59:10 <stepkut> and it has yet to cause me problems in any happstack sites
03:59:20 <stepkut> but.. you do have to be attentive when working on the internals
03:59:45 <mm_freak> well, one problem i noticed with happstack is that it's not trivial to have streaming responses
04:00:21 <stepkut> yes
04:00:31 <stepkut> that is the primary motivation for using something like pipes
04:00:56 <mm_freak> as AJAX gets more and more popular we should definitely solve that…  although as a nice side effect of my Continue monad streaming would be almost trivial to implement =)
04:00:58 <stepkut> streaming things from files isn't bad, or generating a stream from a pure functino
04:01:07 <stepkut> but generating streams that need IO is a bit of a pain
04:01:19 <mm_freak> yeah
04:01:44 <mm_freak> there is plenty of room for exploring the AJAX realm
04:02:36 <mm_freak> i'm still searching for a good theoretical/mental model of AJAX that unifies streaming/push and individual requests
04:02:48 <stepkut> anyway, the reason for creating hyperdrive is (1) i don't really trust that warp is correct (2) it's a small enough code base that being tied to someone else's whims and release cycles does not seem like a big win (3) there are some other significant design changes I want to try for dealing with high loads
04:03:56 <mm_freak> what do you propose for parsing HTTP?  i'm not aware of usable attoparsec parsers
04:04:30 <stepkut> not attoparsec
04:04:39 <stepkut> something similar to happy but a little different
04:04:53 <mm_freak> why?
04:05:17 <stepkut> because the HTTP specs provide EBNF specifications of what a correct parser should do
04:05:28 <mm_freak> ok, good point =)
04:05:56 <stepkut> it's silly to hope that someone got it right when they implemented it in attoparsec, when we can verify that they did (and document where we purposely deviate from the spec)
04:07:00 <mm_freak> well, attoparsec is FAST…  what do you think of generating attoparsec parsers from a QQ DSL?
04:07:27 <stepkut> that is one solution I am thinking of
04:08:15 <mm_freak> ok, my Continue library is almost ready for release
04:08:24 <stepkut> attoparsec is fast for being a combinator library.. but there is certainly no reason why we should expect attoparsec to be faster than a good parser generator library which can do whatever compile optimizations it wans
04:08:27 <stepkut> cool
04:08:35 <mm_freak> i'll write a WAI-based proof of concept framework, where you can see what the code looks like
04:08:42 <stepkut> ACTION still needs to investigate tables
04:08:51 <stepkut> ok
04:08:55 <mm_freak> (i'll use WAI because i want to demonstrate streaming as well)
04:08:59 <stepkut> :)
04:09:13 <stepkut> wai is not a bad choice at the moment
04:09:30 <stepkut> hyperdrive will, of course, have a similar abstraction layer
04:09:56 <stepkut> but.. since it is unusable at the moment, that doesn't help you
04:09:59 <stepkut> it is fast though ;)
04:10:11 <stepkut> at least twice as fast as warp on my system I think
04:10:12 <mm_freak> well, actually attoparsec parsers come close to what a very good parser generator would have generated…  it has speed and continuability ready-made
04:10:25 <stepkut> right
04:10:26 <mm_freak> i wasn't aware of hyperdrive
04:10:29 <mm_freak> let me check
04:10:33 <stepkut> what attoparsec lacks is verification
04:11:02 <mm_freak> i'm loving the dependency list of hyperdrive =)
04:11:03 <stepkut> hyperdrive is entirely proof-of-concept at the moment.. and doesn't even have all the proof or concepts in it yet
04:11:22 <stepkut> right now it just shows that you can right a non-compliant http server using pipes :)
04:11:30 <stepkut> it will be more interesting with the verified parser stuff is being used
04:11:58 <mm_freak> yeah, i'll go with WAI or even snap for now…  depending on how low-level i want to go
04:12:10 <mm_freak> WAI is very low level, so not very convenient
04:12:17 <stepkut> yeah
04:12:32 <mm_freak> for a proof of concept snap may be a better choice after all
04:12:44 <stepkut> \o/
04:12:59 <stepkut> they are, of course, switching away from enumerator IO entirely :p
04:13:18 <mm_freak> i know, but where are they going?  conduits or pipes?
04:13:29 <stepkut> no.. away *entirely*
04:13:43 <stepkut> they are just using plain old strict IO
04:13:58 <stepkut> there is a new library, io-streams
04:14:14 <mm_freak> well, i guess for snap that's fine…  they were not really 'using' enumerators after all =)
04:14:24 <stepkut> :-)
04:14:27 <mm_freak> sending a response is an imperative command in snap
04:15:52 <stepkut> :-/
04:16:24 <stepkut> anyway, I'd like to get a really solid http/https/websocket library that I am happy with
04:17:16 <mm_freak> what bothers me about most low level HTTP libraries is that they are all callback-based
04:17:41 <stepkut> what do you mean by callback-based? They take a function like (Request -> IO Response) ?
04:18:11 <mm_freak> yeah
04:18:22 <stepkut> what are some other options?
04:18:28 <mm_freak> listen/accept
04:18:52 <stepkut> I don't follow
04:18:56 <mm_freak> as a framework developer it would be really nice to be able to write the server loop myself
04:18:59 <stepkut> only one thread can listen, right ?
04:19:06 <mm_freak> yeah
04:19:12 <stepkut> yeah
04:19:41 <mm_freak> it's about the same reason i prefer SDL over GLUT for opengl stuff
04:19:59 <mm_freak> GLUT forces you to use its own main loop and only allows me to register event callbacks
04:20:56 <stepkut> in hyperdrive, we do have a high-level function that takes a (Request -> p Response), but I also try to exposed the build blocks for that so that you could instead write your own event loop
04:21:16 <mm_freak> yeah, that will be great
04:21:28 <mm_freak> and you will definitely use pipes for streaming IO?
04:21:53 <stepkut> one thing I want to do with hyperdrive is try to get more developer feedback into thet event loop so that the developer can, for example, throttle the accept rate if the server load gets to high (if they want)
04:22:18 <stepkut> no, we will *definitely* see if pipes is the *best* choice of IO, streaming or not
04:22:36 <mm_freak> by "streaming IO" i really mean "I/O" =)
04:22:43 <stepkut> sure
04:22:55 <stepkut> we are not committed to any particular io model
04:23:03 <mm_freak> and who is "we"?
04:23:13 <stepkut> mostly just me at the moment :(
04:23:18 <mm_freak> i see =)
04:23:37 <stepkut> though, I have been double checking everything with gabrial
04:23:39 <mm_freak> i'll continue working on continue =)
04:23:45 <stepkut> excellent!
04:23:54 <mm_freak> might actually release the library in the coming two hours =)
04:24:13 <stepkut> so, with warp using conduits and snap using io-streams, it seems worth while to create something using pipes and see how it compares
04:24:37 <stepkut> the thing that is definitely going to happen in hyperdrive is verified http parsing
04:24:56 <mm_freak> verified in what way?  will you actually lift it to the type level?
04:25:40 <stepkut> also, by using the parser generator, it might be possible to generate multiple version that target the different io models with out too much work.. basically factoring out the parser overhead and seeing the affects the io model
04:26:08 <stepkut> verified as in, I am going to copy and paste the EBNF from the HTTP spec into the parser files and the generator is going to verify that the generated parser conform to those specs
04:26:25 <mm_freak> how?  by unit-testing?
04:26:29 <stepkut> no
04:26:39 <stepkut> by proofs of some sort
04:27:05 <stepkut> obviously, you have to believe that the generator itself is correct
04:27:54 <stepkut> HTTP changes VERY SLOWLY.. so I don't need a nice parser that is easy to extend
04:28:06 <stepkut> I need a parser that is very trustworthy and fast
04:28:35 <stepkut> http 1.1 was released 12 years ago?
04:29:30 <stepkut> I love parser combinators.. but the benefits they bring are maybe not the most desirable ones for an HTTP backend
04:30:51 <mm_freak> well, i like the idea of having a quasiquoter that can convert to anything you want
04:31:02 <stepkut> I looked into trying to quickcheck the parser.. but it hard because you need some function that tells you if the test was successful or not
04:31:11 <stepkut> right
04:31:30 <stepkut> making quasiquators out of things is pretty easy
04:32:03 <stepkut> and I intend to design with QQ in mind
04:32:05 <mm_freak> so perhaps the first step is to write an EBNF library with an AST, then start with a simple parsec/attoparsec backend, finally implement an EBNF quasiquoter for it
04:32:10 <stepkut> I already have a QQ that can parse EBNF :)
04:32:15 <stepkut> and an Arbitrary instance
04:32:26 <stepkut> so I can actually generate random, valid HTTP
04:33:07 <stepkut> even more fun.. is I can randomly generate EBNF grammars
04:33:13 <stepkut> and.. boy are they weird
04:33:17 <mm_freak> =)
04:34:07 <mm_freak> the main reason why i don't like parser generators is that they all require external tools, thus they make the compilation process nontrivial
04:34:41 <mm_freak> that's actually true for preprocessors as well…  SHE seems great, but i've never used it for that reason
04:35:26 <stepkut> right
04:35:56 <stepkut> that is one reason why the new HSX supports QQ.. except we need to squash one stupid bug first
04:36:04 <stepkut> need to change a :: String to :: Text
04:36:44 <mm_freak> oh yeah…  that's also the reason i use blaze-html =)
04:47:34 <stepkut> hsx is not properly split into two pieces
04:48:06 <stepkut> and generic XML to Haskell transformer, and a plain haskell library that generates XML/HTML. And neither depends on the other
04:49:48 <mm_freak> i'd love to have an abstraction that gets along without TH/QQ
04:50:13 <mm_freak> templates look so nice in plain blaze-html…
04:51:49 <hpaste> stepcut pasted “randomly generated ABNF grammar” at http://hpaste.org/80959
04:52:00 <stepkut> and randomly generated grammar :1
04:52:28 <mm_freak> wow, that one looks ugly =)
04:52:40 <mm_freak> would be funny to write a random compilable C code generator =)
04:52:51 <stepkut> :)
04:53:28 <mm_freak> if the seed would in whatever sense actually represent the functionality, then you could "grow" programs with evolutionary algorithms =)
04:54:18 <mm_freak> funny idea to use that as an optimization technique
04:56:28 <stepkut> hmm, there seems to be some bug in my ABNF code
04:56:59 <stepkut> in the Arbitrary instance probably
04:57:50 <stepkut> need to redo that right
04:58:03 <stepkut> but the ABNF spec includes a specification of itself
04:58:04 <mm_freak> don't you have an AST?
04:58:18 <stepkut> I do have an AST
04:59:27 <stepkut> in any case, I can then parse the ABNF spec using the the ABNF parser, and then use the Arbitrary instances to generate new ABNF grammars, then parse thoughs, and generate randoms sentences from the random grammars
05:02:04 <stepkut> I can also paste in bits of the HTTP spec and verify that some data is or isn't valid HTTP
05:02:45 <stepkut> I could even generate a 'parser' from that.. but the problem is that, by itself, ABNF doesn't really tell you have to structure the data into a meaningful type
05:03:29 <stepkut> that is the missing piece.. how do you get from which is written in ABNF to a nice Request type (for example)
05:03:34 <mm_freak> it would be interesting to generate both a parser and a type using QQ
05:03:44 <mm_freak> yeah
05:04:00 <stepkut> perhaps, but the types would not be very useful either
05:04:02 <mm_freak> one weakness of QQ is that there is no easy way to see the result
05:04:14 <stepkut> yeah
05:04:25 <mm_freak> that opaqueness was the main reason for me to move away from yesod
05:04:28 <stepkut> that is one thing that I don't like about the yesod qq route stuff
05:05:04 <mm_freak> hehe yeah
05:05:13 <stepkut> in web-routes, you actually write your types -- though there is a web-routes-quasi as well.. but I don't like it because it is not composable like combinators are
05:05:20 <mm_freak> btw, do yesod and happstack cooperate in any way?
05:05:35 <stepkut> not really
05:06:11 <stepkut> when yesod first came out, I worked with ms a bunch on web-routes and make something shared that we could both use.. and then one day he just stopped using it with out saying anything
05:06:28 <stepkut> and I asked why, and he was like, "oh yeah, I probably should have said something."
05:06:58 <mm_freak> i see
05:07:24 <stepkut> like.. he didn't think to ask if we should change something in web-routes, he just decide to fork his own thing with out even thinking about it
05:07:58 <stepkut> also.. I had to spend a lot of time correcting is misinterpretations of the spec
05:08:24 <mm_freak> michael is bound by commercial pressure…  that may be a reason why i wanted to get there fast…  but i don't see any reason not to use web-routes
05:08:32 <stepkut> we do still use the shakespeare-i18n stuff. I submitted a bunch of patches to improve that.
05:08:41 <mm_freak> s/ i / he /
05:09:08 <stepkut> I also complained about more spec violations but he gave his classic, 'we are using this in a bunch of commercial sites, therefore it must be ok' response
05:09:14 <mm_freak> i like shakespearean templates, but the i18n stuff is one of the things i don't like =)
05:09:30 <stepkut> and happstack-authenticate uses the authenticate library
05:10:09 <stepkut> these experiences have not really left me wanting more
05:10:35 <stepkut> I18N blows no matter what you do :)
05:10:43 <mm_freak> on a related topic, are there any plans to make something like lucius?
05:11:07 <stepkut> I'm not really clear on what lucius vs casius actually do
05:11:10 <mm_freak> i've done some i18n experiments which turned out quite nice
05:11:32 <mm_freak> lucius is a parser for a CSS-like language…  it generates compressed CSS from it
05:11:45 <stepkut> fortunately, happstack is not tied to any one i18n solution. But, we currently only have one supported solution..
05:11:51 <stepkut> what is cassius ?
05:11:56 <mm_freak> the great thing about lucius is that you can nest CSS blocks and it does The Right Thing
05:12:07 <mm_freak> cassius is pretty much CSS using indentation style
05:12:18 <mm_freak> lucius is more like regular CSS, but allows that nice nesting stuff
05:12:42 <stepkut> I don't have any strong plans
05:12:54 <mm_freak> you can actually write:  ul { list-style-type: none; li { display: inline-block; }}
05:13:07 <mm_freak> that makes stylesheets actually readable =)
05:13:08 <stepkut> at a minimum I would like to have a nice CSS library with a parser, quasiquoter, and DSL combinators
05:13:26 <stepkut> where the quasiquoter actually parses the CSS
05:13:31 <stepkut> similar to jmacro
05:13:49 <mm_freak> if you do that, please add the support for nested blocks…  it's so damn handy
05:13:59 <mm_freak> it's one of my favorite features from yesod
05:14:09 <stepkut> you can always use lucius :)
05:15:21 <mm_freak> well, if you use lucius without static URL guarantees, there is no reason you have to parse it at compile time
05:15:27 <mm_freak> it can work similar to heist
05:15:39 <mm_freak> shakespeare-css unfortunately doesn't support that
05:15:56 <stepkut> but if you parse it, you get syntax validation at compile time :)
05:16:22 <mm_freak> can you actually make syntax errors in CSS today? ;)
05:16:29 <stepkut> :p
05:16:33 <mm_freak> would be more interesting to get some semantic guarantees
05:16:44 <mm_freak> you never know when to use normal vs. medium vs. …
05:16:54 <stepkut> I just put my css in static .css files
05:17:02 <stepkut> that's been 'good enough' for me so far
05:17:16 <stepkut> the only issue is linking to background images
05:17:26 <mm_freak> you should probably try lucius at some point
05:17:41 <stepkut> possible
05:17:49 <stepkut> going to try tables first though
05:17:49 <mm_freak> it's amazing how much more maintainble your stylesheets get, when you use nesting
05:17:54 <stepkut> i bet
05:18:04 <stepkut> css is horid
05:18:18 <mm_freak> i'm waiting for the first official release of tables
05:18:22 <mm_freak> it's very promising
05:18:32 <stepkut> everytime I use css I feel like I am in some competition against the w3c.. and if my layout actually works I have defeated the evil enemy!
05:19:03 <mm_freak> that's a good way to put it…  the W3C works hard to make their standards unaccessible for humans
05:22:46 <stepkut> so going back in time
05:22:53 <stepkut> if you you start here,
05:22:54 <stepkut> http://hub.darcs.net/stepcut/hyperdrive/browse/hyperdrive/Serve.hs#40
05:23:20 <stepkut> actually
05:23:27 <stepkut> if you start here
05:23:29 <stepkut> http://hub.darcs.net/stepcut/hyperdrive/browse/hyperdrive/Serve.hs#24
05:23:41 <stepkut> you see the typical callback based web server
05:23:57 <stepkut> where 'Handler IO' is the basically (Request -> IO Response)
05:24:21 <stepkut> but it calls, serveSocket, which is the actual accept loop
05:24:29 <stepkut> and which you could write yourself very easily
05:25:03 <stepkut> in your own request loop you could easily call listen/accept yourself
05:25:31 <stepkut> and also configure which pipes are actually used for reading the request and writting the response
05:25:52 <stepkut> in fact.. you might not even use listen/accept at all.. you might get the Requets from some unit tests or something
05:26:08 <stepkut> so, you get complete control over where the data comes from and where it goes
05:26:34 <stepkut> and control over forking off the requset handler.. or not forking
05:26:58 <stepkut> and then you pass that all to the requestLoop
05:27:10 <stepkut> which just glues them together
05:28:00 <stepkut> though, in all this, we are still passing in a callback function
05:28:16 <stepkut> in httpPipe that callback function is actually used
05:28:40 <stepkut> if you really wanted to do away with having a callback at all.. then you would need to do something similar to what httpPipe does in your own event loop
05:29:21 <stepkut> but.. interestingly, even with the callback there.. you can actually have complete control over the listen/accept/fork part of the event loop very easily
05:30:12 <stepkut> or, if you want https support, you just make a serverSocketTLS function that has reader/writer functions that use TLS
05:30:29 <stepkut> if you have improvements though, I'd like to hear them
05:31:07 <stepkut> I want to make it easy to roll-your-own if the standard stuff isn't to your liking
05:32:57 <mm_freak> that pretty much covers what i'd like to have
05:33:14 <mm_freak> it's probably useless for application developers, but very useful for framework developers
05:33:55 <mm_freak> you can do all this through callbacks as well, but then you need IORefs for many things, which is not just ugly, but also quite slow
05:34:46 <stepkut> actually.. I think it can be very useful for application developers
05:35:29 <mm_freak> i think if you use your own accept loop, you're probably writing a framework =)
05:35:36 <stepkut> exposing that event loop makes it much easier for them to add custom instrumentation, and to also add a feedback loop to slow accepts if the server is getting overloaded.. which is a very application specific bit of knowledge
05:35:50 <mm_freak> yeah
05:36:44 <mm_freak> on the other hand, what should throttling do?  you can always wait for a semaphore at the start of the request handler
05:36:45 <stepkut> all current Haskell web frameworks just accepts connections as fast as they can forking off new IO threads
05:37:10 <mm_freak> ok, you could wait before forking
05:37:44 <mm_freak> btw, are you planning to do something against the insane amounts of type classes a proper happstack monad has to instantiate?
05:37:56 <stepkut> i sure hope so
05:38:28 <stepkut> there are an awful lot of classes and monad transformers
05:38:29 <mm_freak> also…  do you plan to continue using monad-control?  i have no idea how to write a MonadControl instance for Continue
05:38:38 <stepkut> hope not
05:38:42 <mm_freak> great
05:38:48 <stepkut> if we are pipes based, I think I can just use pipes-safe?
05:38:52 <mm_freak> it's probably sufficient to use MonadCatchIO
05:39:04 <mm_freak> i mean for the handlers
05:39:10 <stepkut> i didn't write the patches for monad-control in the first place
05:39:21 <mm_freak> monad-control is awful
05:39:21 <stepkut> the handlers are still pipes.. for better or worse
05:40:02 <stepkut> I remember reading a lot of blog posts about which is less broken in the monad-control, MonadCatchIO, etc stuff.. but I don't think I ever found any answer :)
05:40:03 <mm_freak> on the other hand there might be quite a natural MonadControl instance for Continue
05:40:12 <mm_freak> MonadCatchIO is fine
05:40:25 <mm_freak> it's not powerful, but it's very simple and consistent
05:40:57 <mm_freak> i have no idea why people seem to feel the need to use monad-control just to get generalized exception handling
05:41:02 <stepkut> writing monad-control instances certainly did not make me happy
05:41:49 <mm_freak> this is one of the cases, where i totally prefer ad hoc classes for functionality
05:42:20 <mm_freak> would you be interested to see the current state of Continue?
05:42:34 <mm_freak> if yes, i'll upload the repository
05:44:28 <stepkut> I'm actually going to bed now, but tomorro
05:44:44 <mm_freak> alright…  good night
05:45:20 <stepkut> I got this code to work, so it is sleep time
05:45:54 <mm_freak> =)