Experimental IRC log happs-2008-03-28

Available formats: content-negotiated html turtle (see SIOC for the vocabulary)

Back to channel and daily index: content-negotiated html turtle

These logs are provided as an experiment in indexing discussions using IRCHub.py, Irc2RDF.hs, and SIOC.

01:15:58<solrize>anyone know the biggest dataset it's reasonable to put into happs state?
01:19:40<Saizan>solrize: less than physical memory? considering that garbage collectiong takes additional memory
01:19:59<solrize>hmm , thanks. :9
01:20:00<solrize>:(
01:21:08<solrize>is there code around for some kind of persistent updateable map-like data structure that's stored on disk?
01:21:18<solrize>functional database, more or less
01:23:16<Saizan>mmh what's the functional aspect of it?
01:23:33<solrize>saizan basically every update should append stuff to the file instead of overwriting anything
01:23:55<solrize>but you should be able to find an item in O(log n) operations
01:24:16<Saizan>oh so old references would still see the old data?
01:24:19<solrize>yeah
01:24:46<solrize>that's sort of what data.map is except it's in memory
01:25:25<solrize>on each update you'd write out the list of new nodes with some kind of pointer at the very end, so seeking to the end of the file would give you the current data
01:25:42<solrize>at any update you could remember the file length afterwards, and seek back to it later to recreate that state
01:25:50<solrize>maybe there's some cute way to do it with zippers
01:26:04<Saizan>i'm not aware of a library for that
01:26:10<solrize>oh well, thanks
01:29:29<porrifolius>Hi. Is anyone actively using the IxSet stuff or is it a not-ready-for-prime-time prototype?
01:30:17<Saizan>porrifolius: why do you ask?
01:32:37<porrifolius>Well, I was thinking I might use it... don't want to get burned though! ;) A cursory search doesn't show any uses of it in the rest of the happs code, IxSet.update (which would seem useful) is commented out, can't find any examples of using it (other than the IxSet haddock)... things like that.
01:34:20<porrifolius>Saizan: If there is a substantial example of it's use I'd be mighty pleased. It would give me an idea of how it's expected to fit into the broader application architecture.
01:34:37<Saizan>it's not used in the rest of the code primarily because it's intended for users i guess
01:35:40<Saizan>porrifolius: seen Examples/IxState.hs in the repo?
01:37:36<Saizan>porrifolius: IxState doesn't look particulary tuned for performance, btw
01:42:30<porrifolius>Saizan: No I hadn't... just had a quick look. What is it that leaps out at you and says 'untuned for performance'?
01:58:37<Saizan>porrifolius: getOrd rebuilds everything
01:59:08<Saizan>foldr insert empty $ <selected elements>
02:03:29<porrifolius>Saizan: Ahh, you mean getOrd in IxSet? I see, so anything built using IxSet is going to have less than ideal performance. Do you know if the HAppS creators use IxSet? I.E. is it likely to get improved or is it fending for itself now.
02:04:33<Lemmih>There are theoretical issues limiting the possibilities.
02:04:50<Saizan>yeah, it's not an easy problem
02:05:13<Lemmih>The impact of re-indexing might not be a big problem.
02:09:34<Saizan>maybe we could avoid some of it with more structured queries? e.g. if you filter on two indices you can split the two maps and take the intersection, maybe reporting the result as a list, so no reconstruction
02:10:29<Saizan>i guess profiling first is the way
02:11:02<Saizan>it doesn't help that Map and Set are quite strict structures
02:11:47<Lemmih>It doesn't hurt either, does it?
02:20:44<Saizan>Lemmih: since it seems like the code doesn't force indexes at all until they are needed maybe not
02:21:04<Saizan>but i wonder if this can cause a stack/heap overflow then
02:27:06<Lemmih>Sure, if you have a few thousands conditions (:
02:31:11<Saizan>i was thinking of (foldr insert empty longList), first of all change is strict in the last argument, so we have a deep recursion
02:34:31<porrifolius>Are there any examples of representing complex relational data in HAppS? One way would seem to be having an Indexable instance for each table/datatype with primary and foreign keys for doing joins manually, another option I've thought of is just having a plain cyclic graph of datatypes but doing some sort of knot-tying trickery to ensure that changes are represented throughout the graph. Any advice for how to approach the data modelling?
02:35:03<Lemmih>Complex relational data doesn't scale very well.
02:36:18<Saizan>porrifolius: pure cyclic structures are a problem to serialize, since they just look infinite
02:36:39<porrifolius>Lemmih: how would you recommend representing something that is normal-form in, say, 50 database tables?
02:36:50<porrifolius>Saizan: good point.
02:37:14<Lemmih>porrifolius: With SQL, in a relational database.
02:38:31<Lemmih>porrifolius: We're talking screwdrivers and hammers here. They're not easily interchangeable.
02:40:03<porrifolius>Lemmih: oh, ok. So HAppS isn't intended for relational data... beyond a few tables anyway.
02:40:26<Lemmih>Right, HAppS is like Google's BigTable.
17:26:26<Lemmih>Hi gast.
17:33:24<vegai>I wonder if I would be doing a disservice if I bundled all the seperate happs-* packages into a single one in Arch Linux
17:34:08<vegai>Lemmih: do you foresee that all happs-* releases are done synchronously?
17:35:02<Lemmih>vegai: That'll most likely be the case, yeah.
17:35:21<gast>hi lemmih
17:35:43<vegai>personally, I'd prefer slightly larger packages if it makes things easier
17:35:54<vegai>hd space is so cheap these days..
17:36:35<vegai>then again, ....
17:36:45<gast>i have a question to the developper of happs, if ther are any here.. :)
17:36:59<gast>how much experience do you have in web-development?
17:37:18<Lemmih>This much: |------| (not to scale)
17:37:28<gast>do you use happs for the things you want to solve?
17:37:30<gast>lol
17:37:31<gast>ok
17:37:35<vegai>I have about 2 years professional, but 0 with happs :)
17:37:55<gast>ok lemmih, but do you personally use happs?
17:38:19<vegai>I'm trying to push HAppS or something very similar at work for REST services et al
17:38:49<Lemmih>gast: From time to time. I work for HAppS LLC.
17:39:11<gast>what i find a bit frightening is, that for example from version 0.84 to 0.88 and so on ... there were changes which make the code "useless"..
17:39:32<gast>i tried to compile "hpaste" which is written in 0.84.
17:40:21<Lemmih>gast: Yes, hopefully that won't happen anymore.
17:41:10<Lemmih>HAppS was almost completely rewritten from 0.8x to 0.9x
17:41:30<gast>aaah.. ok.
17:42:16<gast>i saw this video about happs and the guy said some things about lamp.
17:42:31<Lemmih>Right.
17:42:47<vegai>is a new release imminent?
17:43:07<gast>how would one do things which are normally done with databases? or is it done with happs-state?
17:43:38<Lemmih>vegai: We just made a release a few weeks ago.
17:44:02<vegai>ok.
17:44:06<Lemmih>gast: Yeps, happs-state is used for data storage.
17:44:38<vegai>are there tarballs available?
17:44:43<gast>is it stored on hard-disk or kept in "ram"-memory?
17:45:02<Saizan>vegai: on hackage
17:45:18<vegai>ok
17:45:27<Lemmih>gast: It's kept in memory.
17:47:11<gast>ok. so the happs-state would work "best" for small databases, where one user doesn't need so much place.
17:48:44<Lemmih>gast: Well, yeah. It'll perform badly if your state is bigger than the physical memory.
17:50:16<Lemmih>gast: However, it's rare to have that much state. Big binary objects can be stored on disk or on S3.
17:50:52<gast>just one question and i'm gone :) for how many years does happs exist.
17:52:14<Lemmih>It has been around for a while and I don't think it'll go away anytime soon.
17:52:20<vegai>ah, a nasty amount of dependencies here. I'll have to work my ass off
17:52:36<Lemmih>vegai: Or get cabal-install (:
17:52:46<vegai>s/have/have the opportunity/
17:53:09<gast>thanks lemmih.
17:53:29<vegai>oh, you see... happs is my package in Arch Linux, but we're currently stalling at version 0.8.4
17:53:59<vegai>I'll probably have to take up a few other packages too if I want to update this
17:54:39<Lemmih>vegai: Grouping all the packages together shouldn't be a problem.
17:54:52<vegai>yeah, that'll help a bit
17:57:51<kaol>would that require combining the cabal files?
17:59:51<vegai>I suppose I wouldn't group the different cabal packages together, just include them all in a single installable arch package

Back to channel and daily index: content-negotiated html turtle