04:10:23 <stepcut> dons: exactly!
04:10:55 <stepcut> dons: most of the dependencies now should be in HP or fairly trivial, or very relevant (like BlazeHtml)
19:31:33 <Gracenotes> whoooaa, how did my happstack-state database bloat up to 900MB :|
19:33:43 <Gracenotes> I'm not storing that much data... I think...
19:34:34 <Gracenotes> yeah, the database is ~20MB before I start the mirroring processes. hmm. *looks*
19:35:54 <burp> my irc bot with 1.5MB database is blowing up, up to ~150M in memory after some usage :>
19:36:35 <Gracenotes> well, memory usage besides, this is actually the size of events and whatnot..
19:36:44 <Gracenotes> on the disk
19:37:27 <Gracenotes> oooh... maybe I'm passing huge arguments to update functions
19:37:54 <Gracenotes> now why would I be doing a silly thing like that
19:42:42 <mightybyte> burp: I've seen behavior like that is well.
19:43:02 <mightybyte> burp: My database was like 10 megs on disk, and 700 megs in RAM
19:43:18 <burp> :/ that's why I switched to sqlite now for the irc bot
19:43:27 <mightybyte> But a few months ago that improved substantially.
19:43:38 <burp> hm ok, haven't tried newest
19:43:54 <mightybyte> Now it's only 350 megs.
19:44:05 <burp> oh nice ;)
19:44:14 <mightybyte> Yeah.  Much more reasonable.
19:44:21 <mightybyte> Although still bigger than I'd like.
19:46:14 <stepcut> happstack 0.7 will really focus on figuring out memory usage stuff
19:46:48 <Gracenotes> I'm mainly concerned about the database being big on disk, as in-memory is something I'm used to
19:47:00 <Gracenotes> I wonder why a few operations would bring it from 20MB to 900Mb
19:47:24 <Gracenotes> as it's also caused memory to grow by about 500MB
19:47:43 <stepcut> your checkpoint file went from 20MB to 900MB ?
19:48:53 <Gracenotes> 25 Aug 13 02:53 current-0000000000  /  29M Aug 13 03:02 events-0000000000  /  94k Aug 13 11:00 events-0000000001   /   200M Aug 13 12:01 events-0000000002    /  672M Aug 13 20:56 events-0000000003
19:49:22 <Gracenotes> hm, 30MB in this case I think
19:51:24 <stepcut> looks like you definitely have some big events there. Would be interesting to see the size of a checkpoint file after those events
19:52:13 <Gracenotes> I didn't add much data, really, so I'm guessing I passed a big arg to an update function somewhere
19:52:17 <Gracenotes> many times
19:52:38 <stepcut> yeah
19:53:28 <Gracenotes> unless there actually is that much new data
19:55:58 <Gracenotes> erg
19:57:08 <Gracenotes> don't see anything..
20:07:30 <Gracenotes> stepcut: is there any way to see which events have lots and lots of data attached to them?
20:08:03 <Gracenotes> maybe some kind of strings+grep.. except the serialized data also has plenty of strings.
20:14:04 <Gracenotes> hm. the serialized data overwhelmingly has huge lists of all packages with all package versions.
20:14:06 <stepcut> Gracenotes: not yet
20:14:11 <Gracenotes> but not from the main index.</detective>
20:14:25 <Gracenotes> maybe it's the revdeps index.. *checks*
20:14:35 <stepcut> Gracenotes: 0.7 will have better tools for examining the contents of the event and checkpoint files
20:21:13 <Gracenotes> woooooo Happstack 0.7
20:21:25 <Gracenotes> I mean wooooooooooo Happstack 0.6
20:24:06 <siracusa> This is a bad hack to quit a Happstack server in GHCi, any ideas why it just doesn't quit if cancelListen is set to true? http://hpaste.org/fastcgi/hpaste.fcgi/view?id=29018#a29018
20:24:50 <siracusa> It's in module Happstack.Server.HTTP.Listen
20:28:25 <Gracenotes> ooh I see, the problem is reverse dependencies. There are three options in this regard: pass the update functions the main package index; query the main package index outside the update functions and replace the entire revdeps index; keep a copy of the main package index inside and update it incrementally. it seems only the 3rd would prevent 600MB event files. ah well.