03:23:20 <donri> luite: you're right, hxt's parser is slow. http://hackage.haskell.org/package/hxt-expat makes it fast. 16:40:07 <kstt> hello 16:40:55 <kstt> I have copied acid-state database from a machine to an other, and tried to load it with the application at the same version. 16:41:26 <kstt> It refused, complaining that some expected acidic function is missing. 16:41:46 <stepkut> hmm 16:41:52 <kstt> This is surprising since I have checkpointed and stoped the origin application before copying 16:42:07 <stepkut> yes 16:42:15 <kstt> therefore, I'm wondering how acid-state reloads a state 16:42:29 <kstt> is it based on files properties ? (timestamp) 16:42:37 <stepkut> no 16:42:39 <kstt> or on files names ? 16:42:46 <kstt> or only on files content 16:42:52 <stepkut> filenames + the version number in the filename 16:43:02 <kstt> right 16:43:23 <stepkut> does the latest checkpoint file look reasonably sized? 16:44:52 <kstt> if I remove all events files, it loads well, but some content is missing. 16:45:09 <kstt> So I'm wondering if the app checkpointed correctly or not on shutdown 16:45:23 <stepkut> how did you create the checkpoint? 16:45:31 <kstt> breacketing the whole app 16:45:45 <stepkut> your best option is to use createCheckpointAndClose so that no events get written after the checkpoint 16:46:17 <stepkut> are there any event files with timestamps after the checkpoint file? 16:46:23 <kstt> that's precisely it 16:46:37 <kstt> (createCheckpointAndClose) 16:46:41 <stepkut> hmm 16:47:41 <kstt> oh oh oh, wait 16:48:04 <kstt> I might have an idea :) 16:48:10 <stepkut> sweet 16:49:02 <kstt> yeah, found it 16:49:10 <kstt> it does not stop 16:49:19 <kstt> it pretends to, but 'ps' shows it 16:49:28 <kstt> ... investingating 16:49:47 <kstt> However, that does not say whether events file are supposed to be portable or not. 16:49:54 <kstt> And under what circumstance 16:50:05 <stepkut> ah 16:50:16 <kstt> I'm a bit in the case of trying to restore an application from the latest backup of the data directory 16:50:28 <kstt> and failing 16:50:29 <stepkut> event files should be portable. The size and endianess of serialized types is architecture independent 16:50:31 <kstt> and stressing :) 16:51:38 <stepkut> unfortunately, the way that acid-state opens the checkpoint/event files, they usually can not be copied while the server is running (the ones that are still open). Best options is to create a checkpoint and then createArchive, and backup the archive directory 16:52:24 <hpaste> kstt pasted âacid state checkpointsâ at http://hpaste.org/70953 16:53:16 <kstt> so in fact, backuping acid-state REQUIRES checkpointing 16:53:39 <stepkut> at this point in time, yes 16:53:43 <kstt> that's something I overlooked so far 16:54:10 <kstt> lots of users told me they only checkpointed on shutdown 16:54:44 <kstt> thank you very much, I'm glad you taught me that on time 16:55:35 <stepkut> it's is a lot messier than I would like.. but it is not easy to do better 16:55:50 <kstt> the content of the hpaste above is the data directory, application stopped 16:56:17 <kstt> so, IIUC your previous question, yes there are events past latest checkpoint 16:56:31 <stepkut> the thing that makes backing up acid-state hard is the fact that acid-state holds certain files open that you kind of want to backup 16:57:07 <kstt> indeed, I understand 16:57:09 <stepkut> also, in theory, you can backup just the event files and replay those, never creating a checkpoint.. but that means you can never modify your events either 16:58:01 <stepkut> so, creating a checkpoint makes life easier, because then you just need valid Migration instances and you can modify your event methods 16:59:31 <kstt> indeed 16:59:59 <stepkut> when multimaster replication is working, we will be able to use that as a better means of doing remote backups/logging 17:00:14 <kstt> does the hpaste above look ok to you ? Latest checkpoint number is 12, latest event number is 409, application is not running 17:00:55 <stepkut> kstt: that doesn't really mean anything useful.. ls -ld would be more informative 17:01:22 <kstt> oh, I thought the filename sequence what what mattered 17:02:11 <stepkut> re backups, with multimaster you could run a remote node that just listens and logs all the events. Basically.. a continuous backup. When an update event returns successfully, you would know you have it logged on two different machines. 17:02:36 <stepkut> the filename sequence is what matters. As does some information in the files 17:02:57 <kstt> I'm not familiar with -ld. It does not print anything but the data dir root. 17:03:35 <stepkut> I believe the checkpoint file stores the highest event file number. So to restore we find the most recent checkpoint file (according to the filenames), restore the data from that. We also get the event file number that was recorded in that checkpoint and replay any events with higher numbers 17:03:41 <stepkut> ls -ld * 17:03:55 <stepkut> so, acid-state itself does not look at timestamps 17:04:16 <stepkut> but.. as investigators, we can look and see if there are event files with timestamps that appear after the latest checkpoint file 17:04:31 <hpaste> kstt annotated âacid state checkpointsâ with âacid state checkpoints (annotation)â at http://hpaste.org/70953#a70954 17:04:46 <kstt> ok 17:05:07 <stepkut> hmm. looks like the files all have the timestamp from when you copied them 17:05:19 <kstt> so what I just pasted won't interest you since it is from the backup and the backup did not preserve timestamps 17:05:24 <stepkut> yeah 17:05:26 <kstt> yeah 17:05:28 <kstt> :) 17:06:16 <stepkut> so.. one of the tools we need to write is the acid-state tool, which you could then point at the checkpoint file and it would tell you things like what the last event file it saw was 17:06:32 <stepkut> and that would tell us for sure if some of those events happened after the checkpoint 17:06:52 <hpaste> kstt annotated âacid state checkpointsâ with âacid state checkpoints (annotation) (annotation)â at http://hpaste.org/70953#a70955 17:07:14 <kstt> here we go 17:07:38 <kstt> indeed, some events were written *after* the latest checkpoint 17:07:56 <stepkut> yeah 17:08:04 <stepkut> like, 5 hours later? 17:08:31 <stepkut> still.. that should not be a problem in itself 17:08:31 <kstt> yes 17:08:55 <stepkut> unless those new event files used new methods that did not exist yet in the executable you are running now 17:10:24 <stepkut> the error says what acid-state event is missing? 17:12:23 <kstt> website: This method is required but not available: "Sormiou.Acid.AddMissingPagesLoc". Did you perhaps remove it before creating a checkpoint? 17:12:23 <kstt> 17:14:18 <kstt> the binary are supposed to be based on the same source 17:14:42 <kstt> only, one is 32bit and the other 64bit 17:16:10 <stepkut> if you run, strings on the executable, does that constructor name show up? 17:16:28 <stepkut> also, is that constructor still in the latest source? 17:18:27 <kstt> it appears as Sormiou.Acid.AddMissingPagesLoc and AddMissingPagesLoc 17:19:07 <stepkut> hmm, mysterious 17:19:52 <kstt> yeah, well, don't worry too much about it 17:19:56 <stepkut> k 17:20:08 <kstt> I'd just like to understand why it does not checkpoint 17:20:31 <kstt> it looks like : bracket (openLocalStateFrom (dp </> "db" </> "acid-state") defaultSormiouState) (createCheckpointAndClose) runServer 17:20:31 <kstt> 17:20:59 <kstt> I suppose createCheckpointAndClose should be run on sigkill 17:21:47 <stepkut> yeah.. you might need to install an signal handler to catch that 17:21:59 <kstt> ah, crap, latest checkpoint size is 0 17:22:09 <stepkut> that is ok 17:22:09 <kstt> I did not notice 17:22:31 <stepkut> acid-state opens the checkpoint file before it actually needs it 17:22:34 <stepkut> to ensure that it can 17:22:39 <kstt> ok 17:22:44 <stepkut> you wouldn't want to call createCheckpoint and only then find out that you don't have prems 17:22:45 <stepkut> perms 17:22:47 <stepkut> gotta run, bbiab. 17:22:50 <kstt> :) 17:23:01 <kstt> one minute please :) what signal handler should be installed ? any link please ? 19:58:29 <stepkut> alpounet: I uploaded a new web-routes-boomerang that is based around [Text] instead of [String] and includes the functions needed to turn a boomerang route into a PathInfo 20:01:23 <tazjin> Sounds great! 20:02:02 <stepkut> it will be even greater if it actually works :) 20:02:06 <stepkut> I have not tested yet 20:02:32 <tazjin> That's my Happstack team! Shoot first, then ask questions ;P 20:03:02 <stepkut> it compiled! surely it works! 20:03:21 <stepkut> I also updated the crash course and that worked too 20:03:46 <stepkut> so it is mostly the PathInfo stuff that I wonder about.. but that is new anyway 23:55:30 <donri> oh hey "The first prototype of the Scoutess build bot has been finished on my machine!"