03:23:20 <donri> luite: you're right, hxt's parser is slow. http://hackage.haskell.org/package/hxt-expat makes it fast.
16:40:07 <kstt> hello
16:40:55 <kstt> I have copied acid-state database from a machine to an other, and tried to load it with the application at the same version.
16:41:26 <kstt> It refused, complaining that some expected acidic function is missing.
16:41:46 <stepkut> hmm
16:41:52 <kstt> This is surprising since I have checkpointed and stoped the origin application before copying
16:42:07 <stepkut> yes
16:42:15 <kstt> therefore, I'm wondering how acid-state reloads a state
16:42:29 <kstt> is it based on files properties ? (timestamp)
16:42:37 <stepkut> no
16:42:39 <kstt> or on files names ?
16:42:46 <kstt> or only on files content
16:42:52 <stepkut> filenames + the version number in the filename
16:43:02 <kstt> right
16:43:23 <stepkut> does the latest checkpoint file look reasonably sized?
16:44:52 <kstt> if I remove all events files, it loads well, but some content is missing.
16:45:09 <kstt> So I'm wondering if the app checkpointed correctly or not on shutdown
16:45:23 <stepkut> how did you create the checkpoint?
16:45:31 <kstt> breacketing the whole app
16:45:45 <stepkut> your best option is to use createCheckpointAndClose so that no events get written after the checkpoint
16:46:17 <stepkut> are there any event files with timestamps after the checkpoint file?
16:46:23 <kstt> that's precisely it
16:46:37 <kstt> (createCheckpointAndClose)
16:46:41 <stepkut> hmm
16:47:41 <kstt> oh oh oh, wait
16:48:04 <kstt> I might have an idea :)
16:48:10 <stepkut> sweet
16:49:02 <kstt> yeah, found it
16:49:10 <kstt> it does not stop
16:49:19 <kstt> it pretends to, but 'ps' shows it
16:49:28 <kstt> ... investingating
16:49:47 <kstt> However, that does not say whether events file are supposed to be portable or not.
16:49:54 <kstt> And under what circumstance
16:50:05 <stepkut> ah
16:50:16 <kstt> I'm a bit in the case of trying to restore an application from the latest backup of the data directory
16:50:28 <kstt> and failing
16:50:29 <stepkut> event files should be portable. The size and endianess of serialized types is architecture independent
16:50:31 <kstt> and stressing :)
16:51:38 <stepkut> unfortunately, the way that acid-state opens the checkpoint/event files, they usually can not be copied while the server is running (the ones that are still open). Best options is to create a checkpoint and then createArchive, and backup the archive directory
16:52:24 <hpaste> kstt pasted “acid state checkpoints” at http://hpaste.org/70953
16:53:16 <kstt> so in fact, backuping acid-state REQUIRES checkpointing
16:53:39 <stepkut> at this point in time, yes
16:53:43 <kstt> that's something I overlooked so far
16:54:10 <kstt> lots of users told me they only checkpointed on shutdown
16:54:44 <kstt> thank you very much, I'm glad you taught me that on time
16:55:35 <stepkut> it's is a lot messier than I would like.. but it is not easy to do better
16:55:50 <kstt> the content of the hpaste above is the data directory, application stopped
16:56:17 <kstt> so, IIUC your previous question, yes there are events past latest checkpoint
16:56:31 <stepkut> the thing that makes backing up acid-state hard is the fact that acid-state holds certain files open that you kind of want to backup
16:57:07 <kstt> indeed, I understand
16:57:09 <stepkut> also, in theory, you can backup just the event files and replay those, never creating a checkpoint.. but that means you can never modify your events either
16:58:01 <stepkut> so, creating a checkpoint makes life easier, because then you just need valid Migration instances and you can modify your event methods
16:59:31 <kstt> indeed
16:59:59 <stepkut> when multimaster replication is working, we will be able to use that as a better means of doing remote backups/logging
17:00:14 <kstt> does the hpaste above look ok to you ? Latest checkpoint number is 12, latest event number is 409, application is not running
17:00:55 <stepkut> kstt: that doesn't really mean anything useful.. ls -ld would be more informative
17:01:22 <kstt> oh, I thought the filename sequence what what mattered
17:02:11 <stepkut> re backups, with multimaster you could run a remote node that just listens and logs all the events. Basically.. a continuous backup. When an update event returns successfully, you would know you have it logged on two different machines.
17:02:36 <stepkut> the  filename sequence is what matters. As does some information in the files
17:02:57 <kstt> I'm not familiar with -ld. It does not print anything but the data dir root.
17:03:35 <stepkut> I believe the checkpoint file stores the highest event file number. So to restore we find the most recent checkpoint file (according to the filenames), restore the data from that. We also get the event file number that was recorded in that checkpoint and replay any events with higher numbers
17:03:41 <stepkut> ls -ld *
17:03:55 <stepkut> so, acid-state itself does not look at timestamps
17:04:16 <stepkut> but.. as investigators, we can look and see if there are event files with timestamps that appear after the latest checkpoint file
17:04:31 <hpaste> kstt annotated “acid state checkpoints” with “acid state checkpoints (annotation)” at http://hpaste.org/70953#a70954
17:04:46 <kstt> ok
17:05:07 <stepkut> hmm. looks like the files all have the timestamp from when you copied them
17:05:19 <kstt> so what I just pasted won't interest you since it is from the backup and the backup did not preserve timestamps
17:05:24 <stepkut> yeah
17:05:26 <kstt> yeah
17:05:28 <kstt> :)
17:06:16 <stepkut> so.. one of the tools we need to write is the acid-state tool, which you could then point at the checkpoint file and it would tell you things like what the last event file it saw was
17:06:32 <stepkut> and that would tell us for sure if some of those events happened after the checkpoint
17:06:52 <hpaste> kstt annotated “acid state checkpoints” with “acid state checkpoints (annotation) (annotation)” at http://hpaste.org/70953#a70955
17:07:14 <kstt> here we go
17:07:38 <kstt> indeed, some events were written *after* the latest checkpoint
17:07:56 <stepkut> yeah
17:08:04 <stepkut> like, 5 hours later?
17:08:31 <stepkut> still.. that should not be a problem in itself
17:08:31 <kstt> yes
17:08:55 <stepkut> unless those new event files used new methods that did not exist yet in the executable you are running now
17:10:24 <stepkut> the error says what acid-state event is missing?
17:12:23 <kstt> website: This method is required but not available: "Sormiou.Acid.AddMissingPagesLoc". Did you perhaps remove it before creating a checkpoint?
17:12:23 <kstt>  
17:14:18 <kstt> the binary are supposed to be based on the same source
17:14:42 <kstt> only, one is 32bit and the other 64bit
17:16:10 <stepkut> if you run, strings on the executable, does that constructor name show up?
17:16:28 <stepkut> also, is that constructor still in the latest source?
17:18:27 <kstt> it appears as Sormiou.Acid.AddMissingPagesLoc and AddMissingPagesLoc
17:19:07 <stepkut> hmm, mysterious
17:19:52 <kstt> yeah, well, don't worry too much about it
17:19:56 <stepkut> k
17:20:08 <kstt> I'd just like to understand why it does not checkpoint
17:20:31 <kstt> it looks like :  bracket (openLocalStateFrom (dp </> "db" </> "acid-state") defaultSormiouState) (createCheckpointAndClose) runServer
17:20:31 <kstt>  
17:20:59 <kstt> I suppose createCheckpointAndClose should be run on sigkill
17:21:47 <stepkut> yeah.. you might need to install an signal handler to catch that
17:21:59 <kstt> ah, crap, latest checkpoint size is 0
17:22:09 <stepkut> that is ok
17:22:09 <kstt> I did not notice
17:22:31 <stepkut> acid-state opens the checkpoint file before it actually needs it
17:22:34 <stepkut> to ensure that it can
17:22:39 <kstt> ok
17:22:44 <stepkut> you wouldn't want to call createCheckpoint and only then find out that you don't have prems
17:22:45 <stepkut> perms
17:22:47 <stepkut> gotta run, bbiab.
17:22:50 <kstt> :)
17:23:01 <kstt> one minute please :) what signal handler should be installed ? any link please ?
19:58:29 <stepkut> alpounet: I uploaded a new web-routes-boomerang that is based around [Text] instead of [String] and includes the functions needed to turn a boomerang route into a PathInfo
20:01:23 <tazjin> Sounds great!
20:02:02 <stepkut> it will be even greater if it actually works :)
20:02:06 <stepkut> I have not tested yet
20:02:32 <tazjin> That's my Happstack team! Shoot first, then ask questions ;P
20:03:02 <stepkut> it compiled! surely it works!
20:03:21 <stepkut> I also updated the crash course and that worked too
20:03:46 <stepkut> so it is mostly the PathInfo stuff that I wonder about.. but that is new anyway
23:55:30 <donri> oh hey "The first prototype of the Scoutess build bot has been finished on my machine!"