01:00:30 <stepkut> nice
01:01:27 <edwardk> i've cleaned up things a bit more since then
01:01:42 <edwardk> its now to the point where an end user could hand write an instance and not want to kil themselves
01:01:44 <stepkut> even nicer :)
01:02:01 <edwardk> https://github.com/ekmett/tables/blob/master/examples/Foo.hs#L37
01:02:42 <stepkut> neat
01:03:53 <edwardk> basically you give m the keys you want, a 'Tab', which is a sort of 'pre-table' to store them in, and a few fiddly bits for how to access fields and put stuff into the tab, and i give you tables
01:04:18 <stepkut> cool
01:04:59 <edwardk> fyi - autoincrement now necessarily goes off the max key in the table, which makes it a bit slower
01:05:13 <edwardk> but it was necessary for correctness
01:05:35 <stepkut> makes sense
01:05:42 <stepkut> what else would you use ?
01:06:08 <edwardk> i was tracking a 'next' key field a la sql server
01:06:36 <stepkut> so, if you delete a value with the max key and then insert a new record, the id will be reused?
01:06:42 <edwardk> eah
01:06:44 <edwardk> er yeah
01:07:08 <stepkut> the 'high water mark' approach certainly sounds less scary :) But.. correctness is more better
01:07:14 <edwardk> i have no place to put the actual 'next free key'
01:07:28 <edwardk> also HWM has its own issues
01:07:42 <edwardk> because i get multiple table objects floating around
01:08:01 <edwardk> and its often the case that you can take a table down to raw rows and build it back up
01:08:02 <stepkut> in vaguely related news.. I was thinking about the problem of wanting to have a 'pure' map-like type that stores the values on disk. I realize one potentially viable solution is to make the values stored on the disk immutable and just never delete them :)
01:09:01 <edwardk> oh, i shrank the codebase for tables down, its now under 400 lines.
01:09:10 <edwardk> it'll grow a bit when i add some TH support
01:09:20 <edwardk> but its pretty feature complete modulo joins
01:09:28 <stepkut> nice
01:09:34 <stepkut> take that SQL :p
01:10:10 <edwardk> yeah. all i had to do was give up atomicity, consistency, isolation, durability...
01:10:28 <stepkut> well, with acid-state you get that all back :p
01:12:06 <donri> id reuse sounds bad
01:12:29 <edwardk> there isn't a way for me to disallow it sanely
01:12:39 <edwardk> you can avoid the auto-increment support
01:14:01 <edwardk> but since there is an ability to take a table down to its rows and back you'll always have to deal with it
01:14:18 <edwardk> and there isn't any canonical notion of where the high water mark would start, anyways
11:00:37 <HugoDaniel> hi
11:01:41 <HugoDaniel> if an acid update function is running on my IxSet, the whole set is going to be unreadable while the update operation is running, correct ?
12:38:56 <HugoDaniel> brb
15:56:50 <donri> stepcut: idea: couldn't the problem of recalculating indices in ixset be solved easily without introducing a "runQuery" function (HiggsSet, data-store) by having queries return sets instead of ixsets and then using set operations to combine multiple queries?
15:58:09 <donri> entries @= (Author "john@doe.com") @< (Updated t1)
15:58:09 <donri> ==> entries @= (Author "john@doe.com") `Set.intersection` entries @< (Updated t1)
15:58:17 <donri> am i missing something?
15:58:52 <donri> i guess it's not really better than "runQuery" because you have to repeat "entries"
15:59:13 <donri> on the other hand you can naturally query multiple ixsets
16:20:02 <edwardk> stepcut: >>> [auto "hello",auto "world", auto "world"]^@..table.group (head.extract) <.> group extract.to count
16:20:02 <edwardk> [(('h',"hello"),1),(('w',"world"),2)]
16:20:58 <edwardk> you can now extract counts without funny operators
16:21:34 <edwardk> stepcut: >>> [auto "hello",auto "world", auto "world"]^@..table.group extract.to count  -- just does the right thing and gives you the count of hellos and the count of worlds
16:24:39 <donri> HugoDaniel: updates block updates but not queries i think
16:28:42 <edwardk> interesting. the way i have with structured, i am able to get it to only regenerate the sub-indices used by the sub-queries, so foo ^. with … . with … is only bothering to recompute the index used by the second with clause
17:58:35 <HugoDaniel> i saw these messages about acid-state in hackage saying that its slow
17:59:50 <HugoDaniel> here: http://www.haskell.org/pipermail/cabal-devel/2012-September/009069.html
18:00:20 <HugoDaniel> its from september, so maybe they were using an older acid-state version or something
18:02:08 <donri> that only says that it is slow for parsing thousands of cabal package descriptions on startup, and i think dcoutts made that fast now
18:02:39 <donri> it's not really about acid-state, although a bit about cereal (acid-state will use the faster binary later)
18:04:10 <donri> you can avoid many restarts with the remote backend, which also implies that the claim about single process is false
18:05:40 <donri> see also https://github.com/haskell/cabal/issues/1031 where i commented
18:11:01 <HugoDaniel> you are "Dag Odenhall" ?
18:12:49 <donri> yep
18:14:40 <HugoDaniel> cool
18:15:01 <HugoDaniel> perhaps it could be possible to replace IxSet with a bunch of hashmaps
18:15:15 <HugoDaniel> one to keep the data, and the others to keep indices
18:15:42 <edwardk> that sounds alarmingly like IxSet ;)
18:15:59 <HugoDaniel> :D
18:16:11 <edwardk> even moreso like my tables
18:16:11 <HugoDaniel> well
18:17:28 <HugoDaniel> if the data could be indexed by uuid's in the main hashmap, then the index hashmaps need only to return uuids
18:18:44 <HugoDaniel> meh, i need to stop thinking about this and let you guys do your work :)
18:20:49 <donri> not sure hashmap is appropriate for uuids
18:21:31 <HugoDaniel> if everything is done by uuids, keeping several acid-state contexts should improve the update locks
18:21:34 <HugoDaniel> yeah :/
18:22:08 <donri> update locks aren't that bad because they cluster
18:22:53 <donri> but yea uuid maps might be one way to do sharding
18:23:13 <donri> or part of a way at least :p
18:24:41 <HugoDaniel> they cluster ?
18:25:26 <HugoDaniel> hmm, the usual pattern with acid-state seems to be acidifying something like data MyAppBigState = MyAppBigState { ... }
18:25:28 <donri> although... sharding doesn't really help updates blocking. not sure you can "fix" that and keep acid properties neatly. if you don't need acidity between different sets of data you can open several states
18:26:24 <HugoDaniel> i see
18:27:38 <donri> by clustering i mean, when updates are being written, new updates will queue up and be written together, which is supposedly faster than writing each individually
18:30:41 <HugoDaniel> oh cool
18:30:48 <HugoDaniel> i run the RemoteClient.hs example
18:31:02 <HugoDaniel> the poke was slow :/
18:31:41 <HugoDaniel> it tries to do 100k state modifications of the simple increment data type
18:32:04 <HugoDaniel> it took about 3secs
18:32:10 <donri> yea... i remember that being a bit slow
18:32:13 <HugoDaniel> the same in postgresql took about 100ms
18:32:16 <donri> how are you running it though
18:32:25 <HugoDaniel> compiling with -O2
18:32:37 <HugoDaniel> ghc -O2 --make RemoteServer.hs
18:32:51 <donri> -threaded? +RTS -N?
18:32:55 <HugoDaniel> no
18:32:58 <HugoDaniel> just that
18:33:00 <donri> not sure those help
18:33:03 <donri> but try
18:33:07 <HugoDaniel> ok
18:34:32 <stepkut> I started an acid-state benchmarks a while back
18:34:34 <stepkut> http://patch-tag.com/r/stepcut/acid-state-benchmarks
18:34:47 <donri> HugoDaniel: what about StressTest.hs?
18:34:47 <stepkut> it only has one test so far and only compares it against sqlite
18:35:07 <stepkut> which it totally tromped -- perhaps due to my lack of understanding of sqlite ;)
18:37:00 <donri> also i wonder if it helps to forkIO scheduleUpdate or not
18:37:29 <HugoDaniel> yeah
18:38:14 <donri> how are you doing it with postgre?
18:38:38 <HugoDaniel> im running a single query inside psql
18:38:43 <HugoDaniel> im not connecting to the db
18:38:47 <donri> that's cheating
18:38:53 <HugoDaniel> yeah, but still :D
18:39:00 <HugoDaniel> ok, let me try to do it normally
18:39:06 <HugoDaniel> with postgresql-simple
18:39:12 <donri> acid-state is doing 100k "queries"
18:39:27 <donri> you could write a single transaction for it and it would be instant
18:40:09 <stepkut> ah, the joys of benchmarking radically different systems against each other
18:40:17 <HugoDaniel> :D lol
18:40:18 <stepkut> ... trying to make sure you are measuring the same thing
18:40:24 <donri> :)
18:40:35 <stepkut> it would be nice to add more tests to,http://patch-tag.com/r/stepcut/acid-state-benchmarks/snapshot/current/content/pretty/counter/acid-state
18:40:48 <stepkut> but I don't know enough about SQL to give the SQL solutions a fair shake
23:49:12 <donri> i'm adding ixset to data-store benchmarks
23:49:32 <donri> it's so much slower that i'm wondering if i'm doing something wrong here
23:50:43 <donri> lookup by string key: Map ~ 3.5 ms, Store ~ 5.5 ms, IxSet ~ 59 ms