14:24:06 <donri> dsfox: module-management needs haskell-src-exts >=1.14
19:39:54 <stepkut> back!
19:42:49 <donri> where've you been, stepkut
19:47:00 <stepkut> burning man
19:47:06 <stepkut> talking with alex jacobson :)
19:53:36 <donri> oh! what's he up to these days
19:54:03 <stepkut> business
19:54:15 <stepkut> but was excited to find out that happs lives on
19:54:34 <stepkut> he suggested that for doing client side stuff it could be interesting to do something based around datalog
19:54:56 <stepkut> he was shocked that people have not demanded an S3 backend for acid-state yet
19:55:08 <donri> ISTR edwardk is doing some datalog-like in haskell, hm
19:55:12 <stepkut> someone get on that !
19:55:35 <donri> https://github.com/analytics/analytics
20:00:15 <stepkut> neat
20:00:26 <stepkut> I 'watched' it
20:41:04 <mightybyte> S3 backend for acid-state doesn't seem to gain you much since state size is memory-limited anyway.
20:44:16 <stepkut> mightybyte: durability
20:48:09 <donri> that sounds like it would either mean waiting for uploads to S3 every Update, or, it's not actually ACIDic and we just need to make creatArchive >> backupToS3 convenient?
20:48:58 <mightybyte> Yeah, but with such a small amount of data, durability doesn't seem like such a huge problem.
20:53:05 <donri> i suppose a properly ACIDic S3 backend could be useful for low-update-but-ultra-precious data
20:53:09 <stepkut> not sure why the amount of data is related to its importance..
20:53:32 <stepkut> I mean.. the nuclear launch codes aren't much data.. but they are pretty important ;)
20:55:25 <stepkut> what is the latency on S3 writes?
20:55:32 <stepkut> and how does that compare to EBS?
20:56:53 <mightybyte> We log to S3 pretty heavily.
20:57:03 <mightybyte> But that's all append-only.
20:57:31 <donri> wouldn't an S3 backend for acid-state also be mostly append-only?
20:57:42 <mightybyte> yeah
20:57:43 <donri> if it's just replication for durability
20:57:56 <mightybyte> You could just put the transaction log there.
20:58:11 <mightybyte> ...and skip the checkpoints.
21:02:11 <stepkut> that is the idea
21:02:29 <stepkut> except also have checkpoints
21:03:04 <donri> what for?
21:04:34 <stepkut> if you want to change the arguments to your update events or remove update events, you need a checkpoint
21:04:46 <stepkut> also it is faster to startup from a checkpoint than to replay all the events from the dawn of time
21:05:25 <donri> well i thought the point was replication, not really reading those logs except on catastrophic failure
21:05:38 <stepkut> no, replication is different
21:05:51 <stepkut> the S3 backend is instead of writing the logs to local disk
21:06:21 <stepkut> there are separate ideas on how to use S3 to support replication, but that is different
21:06:28 <donri> hm ok
21:07:01 <stepkut> writing the log files to disk means your acid-state is only as durable as your hard drive.. which is not really that durable
21:07:19 <stepkut> even if you back it up once an hour.. that can be a fatal amount of data loss
21:07:52 <donri> well i was thinking you would replicate to S3 (acidicly) but still log locally *as well*
21:08:15 <donri> isn't that what replication is?
21:08:47 <stepkut> replication is when you have many different servers you can connect to which all have the same data..
21:08:56 <donri> ie. acidic; you block until the update is fully propagated and durable on all bacnends
21:09:29 <stepkut> having a single server that logs to multiple locations isn't really replication
21:09:45 <donri> ah i guess proper replication would also mean you can distribute the actual computation for transactions etc
21:09:51 <stepkut> exactly
21:10:19 <stepkut> the idea here is to simply put the transaction logs on S3 instead of local disk
21:11:55 <donri> ok but i'm thinking if your data fits in RAM it fits on your disk as well, not many reasons *not* to also write to disk :) which means you can read it offline etc
21:12:05 <donri> and you don't need to download from S3 just to start the app
21:12:42 <stepkut> you could have a backend that writes to both..
21:13:17 <stepkut> but ... a man with one watch always knows the time, and man with two is never sure ;)
21:13:40 <donri> i guess no reason *not* to implement a full backend on S3 (including checkpoints and everything). was just suggesting the stupid-simple version could still be useful :)
21:14:15 <donri> stepkut: well i don't know how acid-state storage works, but if it was anything like git objects you could be pretty darn sure...
21:14:38 <stepkut> I wonder if you could make a git backend for acid-state :-/
21:15:04 <donri> i actually tried that for the python version of acid-state once haha
21:15:07 <donri> (zodb)
21:17:54 <mightybyte> Funny that you mention that.
21:18:05 <mightybyte> I keep all of my acid-state data in a git repo
21:18:28 <mightybyte> That's how I do durability.
21:18:31 <mightybyte> :P
21:19:19 <stepkut> yeah, but how often does that git repo get committed?
21:19:49 <mightybyte> However often I want. :)
21:26:12 <stepkut> how about before each update returns?
21:29:10 <mightybyte> fair enough