Experimental IRC log happs-2008-02-23

Available formats: content-negotiated html turtle (see SIOC for the vocabulary)

Back to channel and daily index: content-negotiated html turtle

These logs are provided as an experiment in indexing discussions using IRCHub.py, Irc2RDF.hs, and SIOC.

00:00:12<fxr>Lemmih: congrats.
00:07:42<Lemmih>perspectivet: Not yet.
00:08:23<Lemmih>porrifolius: Um, kinda.
00:11:58<porrifolius>Is the purpose of Components just to modularise the state, or both the state and external interface to that state?
00:12:27<Lemmih>porrifolius: The latter.
00:14:04<porrifolius>Eg I might have a root component, a http access component and a binary protocol. Each of http comp and bin comp need a bit of extra state (and IO service) but share the 'main' state in a fourth component. Is that a intended structure?
00:18:15<porrifolius>'...and a binary protocol access component.' I mean.
00:23:06<Lemmih>Share the main state?
00:24:09<fxr>what I like is HAppS people gives enough attention to not only web developers, also different kind application developers.
00:24:34<Lemmih>porrifolius: I think that should be fine.
00:27:17<porrifolius>Lemmih: By 'main' state I mean for example your list of Products, Customers and Orders.
00:28:06<porrifolius>I'm wondering if it's possible to use Components to deploy additional access mechanisms to the main state.
00:28:39<Lemmih>porrifolius: Additional access mechanisms?
00:29:09<Lemmih>Having four components in a logical unit is fine.
00:30:30<porrifolius>Lemmih: I might have the wrong idea of what a Component is, but...
00:30:44<fxr>porrifolius: you can write a web application and a gui application and all can access the same state.
00:33:10<fxr>porrifolius: I think components are defined, so that dependencies can be resolved.
00:33:18<mightybyte>porrifolius: If I understand you correctly, your description could be done by setting up a web-service with happs for the state. Then several parts of your application could all use the state through the exposed services.
00:34:36<Lemmih>porrifolius: A component is a piece of state that supports non-composible transactions.
00:35:59<mightybyte>Lemmih: When do you have to specify component dependencies?
00:36:39<porrifolius>I'll try to clarify what I'm thinking about:
00:36:47<Lemmih>When the interface to component X might require component Y.
00:37:09<mightybyte>i.e. be composed of Y?
00:37:41<Lemmih>mightybyte: Imagine a component for generating unique ids in a distributed setting. That components might be used by a blog or bbs component.
00:37:56<Lemmih>*That component might
00:38:02<fxr>well take a quick example, if you want to send mail, you depend mail server.
00:38:21<fxr>a mail sender state depends on a mail server state
00:39:37<Lemmih>mightybyte: 'newBlogEntry' will ask for unique id and use that id to insert a new blog entry.
00:39:43<mightybyte>I was a little puzzled why I didn't need to make my SessionData type a Component and have State depend on it (since it was used down in the tree)
00:40:53<Lemmih>You also used String, Map and a bunch of other data types. Not everything has to be a component.
00:42:12<mightybyte>Ok, when does something have to be a component then? When it's an entry point for some state?
00:45:09<Lemmih>Components give you transactions and persistence. If you want those assurances you can encaptulate stuff with a component.
00:45:37<mightybyte>Ahh, so if you want query and update methods on something, it needs to be a component.
00:45:41<Lemmih>Yes.
00:46:09<porrifolius>Lemmih: So transactions cannot span changes to multiple components?
00:46:28<mightybyte>Ok. In my application, I originally was trying to keep my session data separate from my user data.
00:46:32<mightybyte>I ended up not doing that.
00:46:45<mightybyte>Just made update methods that operated on State.
00:46:47<Lemmih>porrifolius: That's correct. (It's a deliberate design choice.)
00:48:33<Lemmih>porrifolius: HAppS is like BigTable, memcacheDB or whatever they're called these days.
00:49:19<mightybyte>When I did that, I had to rewrite my user data update methods to operate on the State component. If I had made the user data a separate component, I guess I wouldn't have had to do that.
00:54:08<porrifolius>Lemmih: Ok.
00:54:23<porrifolius>To clarify what I was thinking before:
00:54:33<porrifolius>I need to have two (or more) public services that present my underlying data, the 'main' state, to the world. A http/web interface and a legacy binary protocol.
00:54:41<porrifolius>Each of these services needs to access the 'main' state but they each have their own extra state. The services could have their state modified by calls to an administrative service that is part of the root Component.
00:55:03<porrifolius>The two services shouldn't have access to each others state. So I think I have four Components: root `depends` http, root `depends` binprot, http `depends` mainstate and binprot `depends` mainstate.
00:56:00<porrifolius>So I have a DAG of Components.
00:56:00<mightybyte>porrifolius: Yeah, I think you could put the 'main' state in it's own separate application that exposes access as a web service.
00:56:11<Lemmih>porrifolius: Everyone has access to everything. Use the module system to limit access.
00:56:37<mightybyte>porrifolius: That way, your mainstate could be completely isolated.
00:56:42<porrifolius>mightybyte: and then reimplement the binary protocol by using calls to the webservice?
00:56:49<mightybyte>porrifolius: Yeah
00:57:08<mightybyte>porrifolius: I've heard that is how places like Amazon do things.
00:57:11<Lemmih>porrifolius: I'd make root depend on http, binprot and mainstate.
00:57:33<porrifolius>mightybyte: hmmm, sounds a bit of a wasteful indirection.
00:58:16<mightybyte>porrifolius: It might be, but it's a nice modular approach. Limits complexity in each individual "module" or "application".
00:58:35<porrifolius>Lemmih: so a tree of Components, and use the module system to prevent http from messing with binprot state.
00:58:49<Lemmih>porrifolius: Yes.
01:00:02<Lemmih>porrifolius: The component dependencies are just there to make sure everything is loaded when the program starts.
01:00:24<Lemmih>(which is actually very important)
01:00:43<porrifolius>mightybyte: but it means that binprot has to be modified whenever the http interface changes, and the http interface may be presenting a disjoint subset of main state from what binprot needs to.
01:01:19<porrifolius>Lemmih: is there some way to ensure that neither http or binprot try to use mainstate before mainstate is loaded?
01:02:02<Lemmih>porrifolius: Yes, make root depend on http, binprot and mainstate.
01:02:06<mightybyte>porrifolius: Ok, then that might not be the best solution.
01:02:45<Lemmih>porrifolius: Everything is initialized before the associated IO code is executed.
01:03:22<porrifolius>Lemmih: so it would be ok to launch SimpleHTTP in https onLoad and a socket listener in binprots onLoad?
01:03:39<Lemmih>porrifolius: Yes.
01:03:46<porrifolius>even though they use mainstate.
01:03:54<Lemmih>porrifolius: Yep.
01:04:07<porrifolius>Lemmih: Ok. Ta for the info.
01:09:01<porrifolius>Lemmih: Just to clarify: Components define the updates and queries that can happen. The transactional bounds of an update is it's defining Component's state, it cannot span multiple Components, not even sub-Components of it's defining Component.
01:09:53<Lemmih>porrifolius: Correct.
01:10:22<porrifolius>Lemmih: And the same goes for queries? You can only query from one Component's state at a time?
01:13:02<Lemmih>porrifolius: You can update and query multiple components concurrently. You just won't get any transactions guarantees.
01:14:07<Lemmih>porrifolius: Queries are quite fast since they don't have to be saved to disk. That's pretty much the only reason we have them.
01:15:27<porrifolius>Lemmih: so are updates (and queries?) serialised on a per-component basis?
01:16:11<Lemmih>porrifolius: Say again?
01:17:05<Lemmih>We keep a single event log for all components if that's what you're asking.
01:17:59<porrifolius>Lemmih: Yeah, one event log or one per component. So there is one event log.
01:19:32<Lemmih>HAppS is quite performance minded.
01:20:15<porrifolius>So, one http POST might be implemented by a sequence of queries and updates across multiple components...
01:20:26<Lemmih>Yes.
01:20:41<porrifolius>each of the queries/updates is trasactionally safe, but the POST itself isn't transactionally safe.
01:20:56<Lemmih>Correct.
01:21:11<Lemmih>We'll most likely introduce IO events later on.
01:21:33<Lemmih>That is, IO code that is guarateed to run at least once.
01:21:40<mightybyte>Ahh, so to make something transactionally safe, you have to make it's top level piece of data a component.
01:21:56<mightybyte>(I guess that's just a trivial restatement of what's been said.)
01:22:29<porrifolius>mightybyte: by the sounds of it have all data you need to read or write within ONE component
01:22:52<mightybyte>Yeah
01:22:54<Lemmih>You don't want transactions when running an application across multiple machines. They're too expensive.
01:23:34<Lemmih>Local transactions + IO events is managable and very fast.
01:24:24<mightybyte>So, don't dump everything into one big component.
01:24:58<porrifolius>At some point you end up implementing the transaction semantics of you system using idempotent events though.
01:25:29<Lemmih>mightybyte: That's right. Not if you want to run your application in a cluster. Granted, that may not be that often.
01:26:04<Lemmih>porrifolius: Exactly.
01:27:46<porrifolius>It's often not simple hand implementing the semantics with application level events though. A design-space tradeoff I suppose.
01:28:33<Lemmih>porrifolius: Application level?
01:30:02<Lemmih>The actual events can't be very big since they can only access local data.
01:31:16<fxr>excuse me, what should be the second parameter to seeOther? any examples?
01:31:49<Lemmih>The body. 'toResponse ()' will do.
01:32:06<Lemmih>Or well. It depends.
01:32:08<fxr>ok
01:32:12<Lemmih>It's a typing issue, really.
01:32:37<fxr>yup I see, I just cant figure out how to create an empty ServerPart
01:32:55<porrifolius>Lemmih: If you want component-spanning transactional safety you'll need to have transaction semantics encoded in your application/state. If the components are distributed you need to send an event... I thought that's what you meant by IO event.
01:37:10<Lemmih>porrifolius: Events are distributed (semi-)automatically by HAppS. We currently don't have a way of guaranteeing that a piece of IO code runs successfully at least once.
01:37:44<Lemmih>porrifolius: The at-least-once guarantee could be satisfied through a kind of IO event (normal events can't do IO).
01:38:38<porrifolius>Lemmih: Yeah, I guess I meant automatic events carrying application level transaction information to distributed components.
01:39:13<porrifolius>Lemmih: You've still got to implement the trasaction semantics in your application.
01:39:42<Lemmih>porrifolius: Indeed you do. And mistakes can be very dangerous.
01:40:42<Lemmih>porrifolius: However, we hope it'll be easier to use a system designed explicitly for that purpose. Many sites do the same thing with a normal SQL database.
01:40:58<porrifolius>I know paxos was mentioned or considered for distributed transactions. Was any investigation made into it (or other schemes)?
01:42:54<Lemmih>We decided to use Spread (spread.org)
01:43:05<porrifolius>Obviously much more overhead in a full-fledged distributed transaction, but perhaps critical for certain systems or parts of systems.
01:43:55<Lemmih>There's very little vital information that has to be shared.
01:45:39<Lemmih>Nodes behave kinda like independent cells, reacting to what they can sense around them.
01:55:29<porrifolius>Spread is a messaging substrate, right? It doesn't implement a distributed transaction algorithm does it? So, HApps, by design, is only going to enable application level distributed transactions?
01:57:29<Lemmih>Right.
01:57:37<Lemmih>And Right.
01:58:50<porrifolius>Where 'distributed' actually means distributed across Components, not necessarily machines.
01:59:29<Lemmih>Right.
02:00:53<porrifolius>Ok. Thanks Lemmih. Oh, by the way, should SMTP, DNS and Plugins work? I've got the head from a few days ago compiling but not those three.
02:01:06<porrifolius>Should I get the freshly baked 0.9.2?
02:02:03<Lemmih>They shouldn't. We'll remove them from the web site soon(ish).
02:02:54<stepcut>Lemmih: are your updates for 0.9.2 done? I am attempting to make .debs and the version changed from 0.9.1 to 0.9.2 in the middle :)
02:03:10<porrifolius>The SMTP especially sounded like it could be useful. Is it bitrotted away or has it been folded into something else?
02:04:02<Lemmih>stepcut: Yes, done. 0.9.2 is fairly stable, even.
02:04:12<stepcut>Lemmih: sweet!
02:04:33<stepcut>stable as in APIs won't be changing every day? Or stable as in does not crash
02:04:55<Lemmih>stepcut: The former.
02:05:18<stepcut>cool, I think I have spent more time updating my code to work with head than I have writing new code ;)
02:05:19<Lemmih>(Well, it shouldn't crash either)
02:05:41<Lemmih>Yeah, HAppS has been a moving target.
02:06:01<stepcut>I am all for breaking things, if it makes it better
02:06:49<stepcut>but, it will be nice when making it better doesn't involve breaking my code :p
02:08:51<Lemmih>That's the direction we'll go after the release. The event system is stable, the http interface is stable, now it's time to iron out internals like replication and partitioning.
02:09:11<stepcut>cool, I didn't really expect anything to be stable until 1.0
02:25:36<fxr>does anyPath captures "/" also?
02:26:17<Lemmih>fxr: No, I don't think so.
02:27:03<Lemmih>fxr: You can use 'withRequest' and 'rqPaths' to get the full path.
02:31:00<fxr>so an implementation of [anyPath $ anyRequest $ ok $ toResponse "foo"] doesn't handle a request like "http://domain/"
02:31:46<Lemmih>'anyPath' pops a path segment.
02:32:29<Lemmih>Just remove 'anyPath $' and it'll handle all paths.
02:32:43<fxr>thanks for the clarification.
02:35:03<dbpatterson>how is the content type set? I'm sending xhtml back (created with the Text.XHtml combinators), but it is just plain text
02:35:21<dbpatterson>ie, anyRequest $ ok $ toResponse $ renderHtml page
02:35:53<Lemmih>dbpatterson: Try: anyRequest $ ok page
02:37:52<Lemmih>The content type is determined from the type of the output.
02:38:45<dbpatterson>very logical, thanks
02:39:29<dbpatterson>question though - when I did that, the plain responses I was sending back - ie, anyRequest $ ok $ toResponse "Blah" - started erroring, saying it expected Html instead of a String... can you not mix the two?
02:39:42<dbpatterson>(not that it really matters, I just changed the example code, but just curious)
02:39:51<dbpatterson>s/example/filler/
02:42:13<Lemmih>[anyRequest $ ok (toResponse page), anyRequest $ ok (toResponse "blah") ]
02:42:30<Lemmih>Or, [flatten $ anyRequest $ ok page, flatten $ anyRequest $ ok "Blah" ]
02:42:56<Lemmih>Maybe it's 'anyRequest $ flatten'. Can't quite remember.
02:43:11<dbpatterson>okay, thanks
03:14:31<dbpatterson>is there an alternative to path that returns the entire path - or what is the cleanest way to get the entire (remaining) path?
03:17:59<Lemmih>withRequest $ \rq -> ok (show (rqURL rq))
03:18:32<Lemmih>Or: withRequest $ \rq -> ok (show (rqPaths rq, rqQuery rq))
03:18:45<Lemmih>See ya tomorrow.
03:18:49<dbpatterson>thanks
14:54:41<fxr>happs.org down?
14:55:08<Lemmih>Yeah, we're looking into it.
20:37:32<fxr>I'm adding more examples to happs_tutorial2 page.
20:43:24<fxr>oh, happs.org is still down.
20:43:28<mightybyte>fxr: Since the new release is out, let's make that page be targeted at the new release.
20:43:59<fxr>mightybyte: yeah, sure.
20:44:23<fxr>mightybyte: you did great work thanks for the documentation.
20:44:33<mightybyte>No problem
20:44:45<mightybyte>I needed it for myself too. :)
20:45:07<fxr>:)
21:25:52<fxr>do you ever need to use path primitive ?
21:27:59<Lemmih>It can be useful at times.
21:28:23<fxr>could you please gimme an example
21:29:13<Lemmih>Say you want to ask on urls like: yoursite.org/projects/1337/
21:29:35<Lemmih>[ dir "projects" [ path $ \projectId -> ... ]]
21:30:59<Lemmih>dir "projects" [ path $ \projectId -> [ require (findProject projectId) $ \projectData -> ... ]]
21:31:10<fxr>great thank you
21:34:14<Lemmih>(Adding error handling is trivial and in many cases not even desirable)
21:49:28<fxr>is this example clear enough or do you prefere adding require to it? http://hpaste.org/5844
21:49:49<fxr>I'll add this to wiki
21:50:30<Lemmih>Looks good.
21:50:51<Lemmih>You might wanna make it less general.
21:51:15<Lemmih>':: Int -> [ServerPart Response]' is good enough for most cases.
21:51:25<fxr>ok great
22:00:01<Lemmih>You guys should feel free to delete/overwrite/redirect the tutorial for 0.8.8.
22:00:52<fxr>I think we should rename happs_tutorial2 to happs
22:01:09<fxr>but I saw that hpaste uses 0.8.8
22:02:06<Lemmih>hpaste isn't buildable.
22:02:41<fxr>wha?
22:03:35<Lemmih>hpaste was built against an old darcs version which isn't available anymore.
22:03:45<Lemmih>(or something like that.)
22:04:02<fxr>so, nobody uses 0.8 ?
22:04:09<Lemmih>Right.
22:05:52<fxr>I think it's time to create http://www.haskell.org/haskellwiki/HAppS page.
22:07:23<fxr>anyway, I'm moving to add an example for multi
22:43:48<fxr>could you please take a look at this example for multi? http://hpaste.org/5847
22:44:10<fxr>like basicAuth and debugFilter, I tried to prepend serverparts.
22:45:04<fxr>but I don't like this one, comments are welcome :)
22:45:11<Lemmih>Looks fine.
22:46:18<Lemmih>I'm tainted of course. Mightybyte probably has better perspectives.
22:48:43<fxr>ok, let's suspend this and move on method example.
22:49:20<fxr>since MethodMatcher is a very nice usage of type classes, I'll enjoy writing an example for method primitive

Back to channel and daily index: content-negotiated html turtle