15:49:00 <HugoDaniel> can i automatically derive Indexable for my data types ?
15:49:57 <stepkut> yes
15:49:59 <stepkut> inferIxSet
15:50:10 <stepkut> hmm
15:50:20 <stepkut> yeah
15:50:31 <stepkut> I think that just creates an Indexable instance and a type alias
15:51:01 <stepkut> creating instances by hand using ixFun will get you better performance in theory since it doesn't rely on generics
15:51:22 <stepkut> though, hopefully we will all be moving to edwardk's tables package soon :)
15:54:25 <edwardk> 3.8 is probably 2-3 weeks out
15:54:31 <edwardk> when tables can probably ship shortly after
15:54:34 <edwardk> er then
15:54:36 <stepkut> nice
15:55:01 <edwardk> i'd appreciate a bit of gentle use before someone standardizes on my in-flux api ;)
15:55:07 <stepkut> yeah
15:55:12 <stepkut> I'll make an attempt soon
15:55:21 <stepkut> I have to finish this stripe plugin first
15:55:23 <HugoDaniel> :)
15:55:27 <HugoDaniel> stripe ?
15:55:36 <stepkut> HugoDaniel: credit card processing
15:55:42 <stepkut> https://stripe.com/
15:55:49 <stepkut> US-only.. but I am in the US :)
15:55:56 <HugoDaniel> :D
15:56:01 <edwardk> i have to rewrite a good chunk of tables since the details it relies on in 3.8 have changed ;)
15:56:08 <HugoDaniel> edwardk tables are going to replace IxSet ?
15:56:23 <edwardk> thats the goal
15:56:39 <stepkut> It's pretty awesome.. the API is very clean and well documented. Takes real credit cards, doesn't require a merchant account, has reasonable transaction fees and not monthly fees
15:56:44 <edwardk> tables gives you a nice lens/traversal based api
15:57:14 <stepkut> and you can integrated it cleanly into your site (instead of shuffling users to a 3rd party site) while at the same time you can avoid ever having the CC details touch your server
15:57:14 <HugoDaniel> so, if i use your tables instead of ixset they should work ? :)
15:57:27 <edwardk> HugoDaniel: eventually
15:57:32 <HugoDaniel> :D
15:57:34 <HugoDaniel> ahah, cool
15:57:45 <HugoDaniel> edwardk: whats the package name on hackage ?
15:57:46 <edwardk> itll be a 3-4 weeks before i get around to getting tables shaped up
15:57:49 <edwardk> its not there yet
15:57:52 <HugoDaniel> oh okok
15:57:56 <edwardk> it relies on internal details in HEAD of lens
15:58:09 <edwardk> and i haven't finished the TH code generator for it
15:58:18 <edwardk> github.com/ekmett/tables will give you a peek at the work in progress
15:58:41 <stepkut> ACTION looks forward to adding a section to the happstack book on tables
15:59:34 <HugoDaniel> :)
15:59:35 <edwardk> test^..with fooId (<) 12 . with fooBar (==) "hello" . group fooBaz <.> group fooQuuz
15:59:39 <HugoDaniel> this is great edward!
15:59:56 <stepkut> yeah
16:00:12 <HugoDaniel> wow
16:00:19 <edwardk> that would give you a bidirectionally updatable query, limited to fooId's < 12, where the fooBar field is "hell" and it'd group by those two other fields composing the keys
16:00:29 <edwardk> ou can then go through and update those records
16:00:40 <HugoDaniel> that seems easier than a finely crafted dsl :)
16:01:06 <edwardk> test & with fooId (<) 12 . with fooBar (==) "hello" . group fooBaz <.> group fooQuuz . fooQuaffle +~ 100
16:01:25 <edwardk> would add 100 to the fooQuaffle field in everything in the query
16:01:52 <edwardk> tables are foldable, etc. so you can just read the rows out sorted by primary key
16:02:02 <edwardk> and they support auto-increment, etc.
16:02:12 <edwardk> and non-unique supplemental keys
16:02:27 <edwardk> and you don't need to newtype wrap all your fields and no Data instances are necessary
16:02:34 <edwardk> so you can store functions in tables, etc.
16:02:38 <HugoDaniel> this is top notch
16:03:06 <edwardk> it should be a small improvement from ixset ;)
16:03:09 <HugoDaniel> ixset is already amazing
16:03:43 <HugoDaniel> i can't wait to use it :)
16:04:19 <edwardk> i need to update it to use the new profunctor lens scheme in lens head and add TH to generate the Tabular instance for a type
16:04:27 <stepkut> edwardk: I assume you can still create a SafeCopy/Serialize instance for tables as long as you aren't actually putting functions in your tables?
16:04:47 <edwardk> stepkut: you can of course just use toList and fromList to get a list out to serialize
16:04:49 <stepkut> right
16:04:53 <stepkut> that is how we usually do it
16:05:46 <edwardk> i'm pretty pleased with how the API has turned out so far
16:05:50 <edwardk> i want to do joins soon
16:06:01 <edwardk> inner joins, outer joins, etc. between tables would be nice to have
16:06:12 <edwardk> also hash based indices for some fields would also be nice
16:06:16 <stepkut> fun fun fun
16:06:18 <edwardk> and IntMap based indices
16:06:24 <edwardk> all of them fit the general scheme
16:06:36 <stepkut> this is one of the top two missing features for acid-state
16:06:53 <stepkut> the other feature is a bit trickier
16:07:33 <edwardk> i have all the structure to do it. in theory you can even write back to the join if you join primary or secondary indices on each side
16:07:40 <stepkut> it would be awesome to be able to have something like tables where all the keys are store in RAM, but the values are stored on disk -- but somehow still retain a pure API
16:07:59 <edwardk> im going to punt on that usecase for now
16:08:08 <stepkut> but that requires some blackmagic / unsafePerformIO :)
16:08:41 <edwardk> not only that but you can have 50 tables in memory sharing most of their structure
16:08:46 <edwardk> which is the one backed on disk?
16:09:11 <stepkut> that's the black magic part :)
16:10:06 <edwardk> woot. we just got our new magma-based zippers to work
16:10:25 <edwardk> fyi- you can use a zipper into a table as a cursor
16:10:27 <HugoDaniel> :D
16:10:56 <stepkut> transparent disk cache and sharding are actually pretty closely related I think.. though both have to deal with the issues of (1) IO (2) not accidentally trying to force everything into RAM at once
16:11:03 <edwardk> so you can start the cursor at the first result of a query, and step it forward and back, etc.
16:11:21 <edwardk> update the targets and then rezip to get a new table
16:11:39 <stepkut> neat
16:11:56 <HugoDaniel> how do they behave if used with say... 8 gigs of data in memory ?
16:11:59 <edwardk> you can also delete rows just by assigning to the result of a query
16:12:22 <edwardk> test & with fooId (<1) 100000 .~ empty
16:12:25 <edwardk> HugoDaniel: no idea
16:12:34 <HugoDaniel> toList should be kind of slow
16:12:38 <edwardk> my main goal has been decent asymptotics and sane API
16:13:01 <stepkut> edwardk: you should think about submitting a paper to the Haskell Workshop for next ICFP/CUFP ;)
16:13:16 <edwardk> if i do it is more likely to be about ermine
16:13:44 <HugoDaniel> ermine ?
16:14:23 <edwardk> ermine is the haskell-like pure functional programming language i maintain here at S&P capital IQ, which i designed a couple of years back. it is haskell + row types so you can give strong types to things like sql queries, etc.
16:14:46 <HugoDaniel> oh cool
16:14:58 <edwardk> we're rewriting the compiler in haskell now, and are likely to open source that version of it early next year
16:15:04 <HugoDaniel> the tables project seems cooler to me though :)
16:15:23 <edwardk> tables makes me queasy because it deliberately violates many lens/traversal laws ;)
16:15:45 <HugoDaniel> :)
16:15:51 <stepkut> ermine seems more appropriate for ICFP, and tables for the Haskell Workshop perhaps
16:18:17 <stepkut> also, someone should try to rebuild acid-state on top of cloud haskell.. or figure out why that is a bad idea
16:18:26 <HugoDaniel> cloud haskell ?
16:18:49 <edwardk> well, lens is probably more appropriate to the haskell workshop
16:18:56 <edwardk> since tables builds on something that most there wouldn't know
16:19:03 <stepkut> HugoDaniel: http://www.haskell.org/haskellwiki/Cloud_Haskell
16:19:24 <edwardk> sadly i don't know how to compress the lens infodump to under 2 hours
16:19:26 <stepkut> HugoDaniel: under the hood, Cloud Haskell works a bit like AcidState does
16:19:26 <HugoDaniel> im trying to create a distributed social network with haskell, but i was using postgresql for the job, and now im changing to acid+ixset, still in the very early stages
16:19:39 <HugoDaniel> im trying to rip from buddycloud many thing
16:20:07 <stepkut> edwardk: yeah.. I've had the some problem with trying to teach Happstack.. it is easily a 5 day course :)
16:20:24 <edwardk> woot. the new zippers are ~4x faster in some cases and massively faster in others due to a fixed space leak
16:20:28 <stepkut> edwardk: \o/
16:21:10 <stepkut> HugoDaniel: fun
16:21:25 <edwardk> sadly i had to discard my path compression hack
16:21:44 <HugoDaniel> path compression ? :)
16:24:30 <donri> edwardk: is tables based on relational algebra?
16:24:35 <edwardk> no
16:26:51 <edwardk> tables is based on something more sqlish with primary and secondary keys
16:28:10 <donri> we should add tables to the data-store benchmark suite
16:28:18 <stepkut> yeah
16:30:25 <HugoDaniel> where can i see those benchmarks ?
16:32:28 <donri> http://palmik.net/data-store/benchmark16.html
16:33:04 <donri> http://palmik.net/data-store/benchmark.html
16:33:14 <donri> https://github.com/Palmik/data-store
16:34:11 <edwardk> donri: as soon as i can give you a version that builds again, sure ;)
16:34:22 <donri> \o/
16:35:26 <HugoDaniel> whats the diff. between the 16 version ?
16:36:32 <donri> 16 uses larger collections i think
16:38:27 <HugoDaniel> they are much slower :)
16:38:41 <HugoDaniel> IntMap kicks arse
16:41:47 <donri> bit sad that we need a separate thing for it though
16:48:37 <HugoDaniel> how does data-store differ from ixset ?
16:55:39 <stepkut> HugoDaniel: stronger types, auto-increment (i believe), and better performance if your query requires multiple keys?
16:59:46 <edwardk> i spend a lot of time rebuilding indices for subtables in tables. it may cost me on benchmarks
16:59:51 <edwardk> otoh, i don't pay for TH
17:01:53 <donri> hm we should add ixset too to those benchmarks just to see how awful it is :p
17:02:35 <donri> maybe not *that* awful...
17:03:26 <HugoDaniel> why not just use IntMap for everything ? :D
17:06:36 <donri> because we want calculated multi-indices
17:07:56 <donri> say you want to look up blog posts by tag. with an intmap you either have to iterate every element, or you end up doing what ixset etc do for you and create a map from tags to a set of blog posts
17:08:40 <donri> (which you then have to make sure to update when the intmap is updated)
17:08:50 <HugoDaniel> oh yeah :/
17:09:05 <HugoDaniel> still i like ixset
17:09:17 <HugoDaniel> is it using some special tree to allow for those multiple indexes ?
17:09:29 <stepkut> donri: ixset will be on par with Map for a bunch of operations I expect.. and horrible for a few things
17:09:46 <donri> yea
17:09:51 <HugoDaniel> kd-trees ?
17:10:02 <donri> there was an unreleaxed kdmap project
17:10:30 <donri> i think palmik looked at kd-trees for data-store and decided against them for some supposedly informed reason
17:10:45 <stepkut> kd-trees are hard to balance
17:10:51 <stepkut> but good for geospatial searches
17:10:52 <HugoDaniel> yeah :/
17:10:57 <HugoDaniel> hmm data IxSet a = IxSet [Ix a]
17:11:17 <donri> ixset is kinda amazingly simple
17:11:22 <HugoDaniel> what if vector was used instead of [] ?
17:11:41 <donri> don't think that matters much,  you don't usually have many indices
17:12:01 <HugoDaniel> yeah, true
17:12:11 <donri> note that indices aren't the same as each key
17:12:36 <donri> if you have a million posts indexed on the ID, that's one index not one million
17:13:07 <HugoDaniel> yeah, i like this code, i can read it :D
17:13:30 <HugoDaniel> i see
17:13:32 <HugoDaniel> yeah
17:14:20 <HugoDaniel> im looking at getOrd2
17:15:07 <HugoDaniel> (lt',eq',gt') = Map.splitLookup v'' index
17:15:48 <HugoDaniel> so this is basically a Map of keys to Sets
17:16:14 <donri> look at Ix itself
17:17:25 <HugoDaniel> yes
17:17:29 <HugoDaniel> good, this is very good
17:18:02 <donri> well, there are problems with ixset
17:18:31 <donri> Typeable keys means less static guarantees
17:18:39 <HugoDaniel> so i should be able to optimize memory access if i create for instance a data IsCommon = CommonUser | UncommonUser and index my users with it
17:18:58 <donri> each query directly creates a new ixset which means calculating new indices
17:18:59 <HugoDaniel> donri, still it feels much better than sql
17:19:52 <donri> and you can't easily make legal and useful lenses for it
17:20:03 <donri> so thus, tables and data-store
17:20:45 <HugoDaniel> i bet the code from data-store is the standard fugly optimized haskell code :/
17:20:48 <HugoDaniel> let me check
17:21:29 <stepkut> I once mapped out a way to make the IxSet API more type-safe.. while staying pretty much the same
17:22:30 <HugoDaniel> oh, data-store is not available on hackage ?
17:22:58 <donri> not yet no
17:23:38 <donri> i think palmik is doing it as a thesis work
17:24:39 <HugoDaniel> thats nice
17:24:45 <edwardk> donri: i o the same thing in tables for now
17:25:00 <edwardk> er i do the same thing (w.r.t recalculating indices)
17:25:11 <donri> edwardk: yea got that from your earlier subtables comment :)
17:25:25 <donri> does "for now" suggest it's not strictly necessary for its design?
17:25:29 <edwardk> then i go and reinsert the rows when you come back out
17:25:59 <donri> HugoDaniel: oh there's also HiggsSet which is already on hackage
17:26:06 <edwardk> well, you can always extract them as a list, etc. by folding, but then you cant chain subqueries
17:26:25 <HugoDaniel> donri: checking the code :)
17:27:06 <HugoDaniel> oh, it doesn't build on 7.4
17:28:58 <donri> edwardk: what did you mean about newtype noise in data-store btw?
17:29:10 <edwardk> i meant it about ixset
17:29:18 <edwardk> i looked at data-store only briefly
17:29:26 <edwardk> i wrote tables more as a proof of concept
17:29:32 <edwardk> got bored one morning
17:29:42 <donri> oh you mean because keys must be their own types in ixset
17:29:44 <edwardk> then stepcut got excited ;)
17:29:46 <edwardk> yes
17:29:53 <stepkut> :)
17:30:20 <HugoDaniel> i wish i was l33t enough to build my own multi-index key store data type :)
17:30:56 <donri> who needs to be l33t when we've got all these great minds doing it for us
17:31:11 <HugoDaniel> ahaha :D
17:31:36 <HugoDaniel> i like higgs set approach
17:32:00 <HugoDaniel> https://github.com/lpeterse/HiggsSet/blob/master/src/Data/HiggsSet.hs#L105
17:32:35 <HugoDaniel> "indizes"
17:32:49 <HugoDaniel> german :)
17:33:16 <donri> hehe
17:35:06 <donri> HugoDaniel: https://github.com/Palmik/data-store  https://github.com/ekmett/tables
17:36:08 <donri> stepkut: whatever happened to your relational thing
17:36:13 <donri> is that relevant to this all
17:36:40 <stepkut> http://patch-tag.com/r/stepcut/relational/snapshot/current/content/pretty/Relational.hs
17:36:46 <stepkut> it's just proof of concept
17:39:50 <HugoDaniel> :)
17:50:16 <HugoDaniel> i like higgset the most
17:50:25 <HugoDaniel> too bad it doesn't build
17:51:45 <HugoDaniel> the others i dont understand :(
17:51:58 <HugoDaniel> ixset and higgset are the ones i can read :)
17:54:34 <HugoDaniel> i wish i could understand type families, but they are a bit over my skill
18:58:59 <donri> hehe data-store does a lot of type-level programming
18:59:26 <donri> i haven't really understood tables yet, not that i've tried very hard
19:02:10 <donri> HugoDaniel: type families are easy though. think type classes for the type-level
19:03:15 <donri> type synonyms are like functions on the type-level; families gives you overloading
19:03:16 <edwardk> tables needs to be refactored to use the new profunctor representation in lens
19:05:55 <stepkut> :-|
19:07:04 <stepkut> HugoDaniel: I wrote a tiny bit about type families here, http://happstack.com/docs/crashcourse/WebRoutes.html#web-routes-type-families
19:07:04 <edwardk> it'll be good
19:07:10 <edwardk> it should clean it up a lot
19:07:15 <stepkut> nice
19:07:17 <edwardk> almost all the type family noise should vanish
19:07:56 <stepkut> ooo
19:09:00 <donri> one thing that still confuses me about type families is why you'd want data families as opposed to referring to an outside data type with an associated type
19:10:48 <donri> is it just so you don't have to make up names for the data types? but then you can't define it outside the data family
19:10:49 <edwardk> type families give you inference one way a ~ b   => T a ~ T b .
19:10:53 <edwardk> data families give you inference both ways
19:11:04 <edwardk> a ~ b => T a ~ T b   and T a ~ T b => a ~ b
19:11:14 <donri> ah i think i see what you mean
19:11:20 <edwardk> that latter inference direction is when you need data families.
19:12:01 <donri> a FooId has to be a Column (Foo a), but if Column was merely an associated type you couldn't know from just FooId?
19:12:13 <edwardk> yeah
19:12:24 <donri> makes sense
19:14:17 <donri> stepkut: (not that there is r ShowURL any more)
19:14:24 <donri> s/r/a/
19:22:51 <stepkut> :)
19:23:02 <stepkut> I'll fix that when I get to that section
19:25:48 <edwardk> ok, tables builds again. i didn't get it switched over to a more profunctor-ish style though
20:49:50 <HugoDaniel> :)
22:28:47 <edwardk> ok, i did a cleanup pass over tables
22:28:55 <edwardk> i minimized the api a bit further
22:29:04 <edwardk> this means you don't need so much magic support from lens
22:29:43 <edwardk> So if we start with:
22:29:45 <edwardk> >>> test
22:29:45 <edwardk> fromList [Foo {fooId = 1, fooBar = "One", fooBaz = 1.0},Foo {fooId = 2, fooBar = "Two", fooBaz = 2.0},Foo {fooId = 3, fooBar = "Three", fooBaz = 3.0},Foo {fooId = 4, fooBar = "Four", fooBaz = 4.0},Foo {fooId = 5, fooBar = "Five", fooBaz = 5.0}]
22:30:10 <edwardk> >>> test ^. with FooId (<) 3
22:30:10 <edwardk> fromList [Foo {fooId = 1, fooBar = "One", fooBaz = 1.0},Foo {fooId = 2, fooBar = "Two", fooBaz = 2.0}]
22:30:16 <edwardk> now we use uppercase to match on keys
22:30:38 <edwardk> then we can use any lowercase field accessor (or any other function) to do a non-keyed lookup or filter
22:31:08 <edwardk> >>> test ^. with (length . fooBar) (<=) 3
22:31:08 <edwardk> fromList [Foo {fooId = 1, fooBar = "One", fooBaz = 1.0},Foo {fooId = 2, fooBar = "Two", fooBaz = 2.0}]
22:31:48 <edwardk> you can delete by assigning to that filtered table:
22:31:49 <edwardk> >>> test & with (length . fooBar) (<=) 3 .~ empty
22:31:50 <edwardk> fromList [Foo {fooId = 3, fooBar = "Three", fooBaz = 3.0},Foo {fooId = 4, fooBar = "Four", fooBaz = 4.0},Foo {fooId = 5, fooBar = "Five", fooBaz = 5.0}]
22:32:34 <edwardk> and you can edit the actual type of the fields if the table is configured to allow it:
22:32:35 <edwardk> >>> test & rows.fooBar_ %~ length
22:32:36 <edwardk> fromList [Foo {fooId = 1, fooBar = 3, fooBaz = 1.0},Foo {fooId = 2, fooBar = 3, fooBaz = 2.0},Foo {fooId = 3, fooBar = 5, fooBaz = 3.0},Foo {fooId = 4, fooBar = 4, fooBaz = 4.0},Foo {fooId = 5, fooBar = 4, fooBaz = 5.0}]
22:32:50 <edwardk> here i'm using a lens for the fooBar field
22:34:02 <edwardk> if you edit multiple fields, the edits all take place at the same time. so we can offset a bunch of keys:
22:34:03 <edwardk> >>> test & with FooId (>=) 2.rows.fooId_ +~ 1
22:34:04 <edwardk> fromList [Foo {fooId = 1, fooBar = "One", fooBaz = 1.0},Foo {fooId = 3, fooBar = "Two", fooBaz = 2.0},Foo {fooId = 4, fooBar = "Three", fooBaz = 3.0},Foo {fooId = 5, fooBar = "Four", fooBaz = 4.0},Foo {fooId = 6, fooBar = "Five", fooBaz = 5.0}]
22:35:23 <edwardk> we can do grouping by arbitrary functions or fields similarly
22:35:25 <edwardk> >>> test ^@.. group (length.fooBar)
22:35:25 <edwardk> [(3,fromList [Foo {fooId = 1, fooBar = "One", fooBaz = 1.0},Foo {fooId = 2, fooBar = "Two", fooBaz = 2.0}]),(4,fromList [Foo {fooId = 4, fooBar = "Four", fooBaz = 4.0},Foo {fooId = 5, fooBar = "Five", fooBaz = 5.0}]),(5,fromList [Foo {fooId = 3, fooBar = "Three", fooBaz = 3.0}])]