03:32:42 <stepcut> alpounet: I figured something out.. I think buildSources needs a type more like this: buildSources :: BuildHistory ->  [SourceLocation] -> IO BuildReport
03:33:00 <stepcut> where BuildHistory gives information about what was last checked out and also what was last built
03:33:40 <stepcut> also needs some config information..
08:27:57 <Lemmih> stepcut: Yay. I reinstalled acid-state and safecopy and now it doesn't work. (:
08:37:19 <alpounet> stepcut, sounds better to me indeed
08:37:52 <alpounet> so probably like buildSources :: BuildConfig -> BuildHistory -> [SourcesLocation] -> IO BuildReport
15:22:55 <stepcut> yeah
15:23:32 <stepcut> alpounet: I pushed some patches. More later today hopefully.
15:25:24 <alpounet> nice
15:25:28 <alpounet> i have some incoming patch too
15:25:30 <alpounet> maybe for today
15:34:24 <stepcut> cool
15:34:27 <stepcut> I gotta run, bbl
15:36:46 <alpounet> stepcut, i'll push once it handles (totally) package fetching/unpacking/etc
15:36:55 <stepcut> nice
15:36:59 <alpounet> for hackage, ofc
15:37:02 <alpounet> not all of the protocols :P
15:37:02 <stepcut> yeah
16:08:42 <alpounet> oh my, just rolled on a tiny webapp with happstack in ~1h
16:30:40 <alpounet> aaaalright
17:10:27 <alpounet> stepcut, for the "version" thing in SourceInfo, pkg-ver is fine, right?
17:37:08 <alpounet> stepcut, also, do we care about how the cabal file gets "transformed" (when there are conditionals about the os, compiler, whatever) or should we just ignore these bits, when creating the SourceInfo ?
18:18:29 <stepkut> the version in SourceInfo is most likely some version number that is relevant to the location the source is coming from.. such as the git patch number, etc
18:18:50 <alpounet> and for hackage...?
18:19:43 <stepkut> for hackage the package name and version number should be sufficient
18:19:50 <stepkut> since those are required to be unique
18:21:07 <stepkut> how would the transformations affect the SourceInfo ?
18:23:38 <stepkut> the answer might be, flattenPackageDescription? http://hackage.haskell.org/packages/archive/Cabal/1.14.0/doc/html/Distribution-PackageDescription-Configuration.html#v:flattenPackageDescription
18:26:49 <alpounet> stepkut, yeah, but then, we have to take care of not using that PackageDescription when actually building the stuffs
18:27:09 <alpounet> already saw flattenPackageDescription yes
18:27:46 <stepkut> alpounet: when actually building stuff we will be calling cabal install anyway?
18:28:00 <alpounet> if we call the binary then we're fine
18:28:25 <alpounet> we either use flattenPackageDescription or just access the 'packageDescription' field in GenericPackageDescription
18:29:17 <stepkut> I think we want to use GenericPackageDescription for now.. since we do want to have a win32 builder, OS X, etc ?
18:29:21 <stepkut> so we need to keep that OS information around ?
18:30:06 <alpounet> well, depends. for what?
18:30:19 <alpounet> if it's for something cabal install will handle, we actually... don't!
18:30:24 <stepkut> I dunno.. I am not sure why you asked this question in the first place..
18:30:56 <alpounet> well, will we need those informations? the build-depends, the library/executable sections, etc
18:30:57 <stepkut> well... when we have a list of packages to build, we might want to organize and group them
18:31:31 <stepkut> for example, if we are building 3 independent packages with no shared dependencies, then we could potentially build them in parallel
18:31:35 <stepkut> but that means we have to determine if they depend on each other
18:32:31 <stepkut> or if we have, A depends on B which depends on C, and all three have changes, then we might want to build C, then B, then A
18:32:51 <stepkut> (in the same environment, to avoid pointless rebuilding)
18:33:22 <stepkut> well.. largely pointless I think
18:33:35 <stepkut> we could have a strict and lax mode
18:33:51 <alpounet> yeah ok
18:33:56 <alpounet> i'll just use flatten
18:33:59 <stepkut> ok
18:34:08 <stepkut> we can always change it later when we have a clearer picture
18:35:00 <alpounet> yeah
18:37:39 <alpounet> stepkut, other question now, tougher. how do we handle the case when a version number isn't specified?
18:37:59 <stepkut> isn't specified?
18:38:38 <alpounet> well, isn't given, I meant
18:38:48 <alpounet> there's a 'latest' directory, but contains only the docs: http://hackage.haskell.org/packages/archive/happstack-server/latest/
18:39:06 <alpounet> and then there's that weird distrinfo file, which is my only lead: http://hackage.haskell.org/packages/archive/happstack-server/distroinfo
18:39:51 <alpounet> i guess that's where they get the "packaged for <some distribution>" informations
18:40:29 <alpounet> but sometimes there isn't such a file
18:40:48 <stepkut> when fetching a uri, maybe you should look for a valid .cabal file on the server and use the version number from there ?
18:40:50 <alpounet> so i guess i'll just have to somehow fetch the list of directories, read the versions and order them and pick the higher one
18:40:52 <stepkut> http://hackage.haskell.org/packages/archive/happstack-server/0.2.1/
18:41:10 <stepkut> oh, one moment
18:41:21 <alpounet> highest*
18:41:34 <stepkut> yeah
18:41:39 <stepkut> I am looking at how the autobuilder does it
18:42:50 <stepkut> I guess the autobuilder requires an explicit version
18:45:13 <stepkut> moving locations. bbiab.
19:40:13 <stepcut> alpounet: I think the correct way to get the latest version of a package from hackage is to look in the 00-index.tar and find the latest version..
19:42:46 <alpounet> ok
19:48:12 <alpounet> stepcut, that may be as much annoying as the other way
19:50:46 <dcoutts> stepcut, alpounet: there's code to get and read the index file
19:51:05 <dcoutts> no need to reimplement it (if you're working in Haskell)
19:51:31 <alpounet> and we already depend on Cabal anyway
19:53:12 <dcoutts> alpounet: so it's not nicely packaged as a nice lib, but there's a couple fairly reasonable modules in cabal-install you can copy'n'paste
19:54:00 <dcoutts> a package index type, and the file parsing
19:54:14 <dcoutts> alpounet: working on an autobuilder?
19:54:53 <dcoutts> alpounet: that'd be very useful for the new hackage-server btw, it's designed to work with external build bots
19:55:07 <alpounet> dcoutts, on this, precisely: https://patch-tag.com/r/alpmestan/scoutess/wiki
19:55:11 <dcoutts> there's a REST api for uploading build results including logs
19:55:36 <alpounet> and yeah we started working on this last week or so, with stepkut
19:55:40 <dcoutts> great
19:55:42 <alpounet> and hm
19:55:56 <alpounet> i actually borrowed the package index generation procedure from the hackage trac
19:56:08 <alpounet> i do have a function that generates it, so that we have one locally
19:57:23 <alpounet> but now, i'm dealing with the issue of, when the program is told that it must try to build smth with a given dependency it has to fetch from Hackage, if a version number isn't given, how can the program guess it
19:57:36 <alpounet> i'm not really fond of the solutions i've found so far :p
19:57:43 <dcoutts> alpounet: if you'd like to add uploading build reports to a/the hackage server, we've got one running
19:57:55 <dcoutts> there's a build report data format defined
19:58:03 <dcoutts> and some code in cabal-install to generate it
19:58:37 <alpounet> that's gonna be helpful, definitely, it will keep us from redefining everything
19:58:52 <dcoutts> alpounet: you could follow the cabal-install policy which is to use the latest "preferred" version
19:59:09 <dcoutts> that's how it resolves "foo" to "foo-1.0" or whatever
19:59:52 <alpounet> dcoutts, how exactly is "preferred" defined here?
19:59:59 <dcoutts> (though it does something more sophisticated if you're actually asking to install it, using the constraint solver)
20:00:01 <alpounet> the latest local version that's not hidden or smth?
20:00:26 <dcoutts> alpounet: the latest version on hackage, plus the "preferred-versions" file in the 00-index.tar.gz
20:00:32 <alpounet> oh yeah right
20:00:41 <dcoutts> the code in cabal-install will read the preferred versions too
20:01:14 <dcoutts> alpounet: so e.g. this lets people make new major releases and mark them as not yet stable
20:03:04 <dcoutts> alpounet: are you thinking of running testsuites too?
20:03:28 <alpounet> yeah but i remember looking at hackage's preferred-versions file and seeing only a few packages mentionned, like base and two or three others
20:03:28 <dcoutts> that's something we'd like for the hackage-server, but have not yet defined a interface for
20:03:51 <dcoutts> alpounet: right, it'll be used more with the new server, maintainers will have a better UI to use it
20:04:01 <alpounet> ok
20:04:10 <alpounet> and i have been thinking about testsuites too
20:04:20 <alpounet> 'cause ideally we're aiming at a sort of continuous integration tool
20:04:33 <alpounet> with modules you can just switch on and off, like reporting build reports on IRC etc
20:04:48 <dcoutts> so the existing testsuite interface is just "exitcode-stdio-1.0" very simple
20:05:05 <dcoutts> but not any structured info
20:06:40 <dcoutts> alpounet: you might like to think about how hackage-server integration would work, since the idea with the new server is it's easy to run locally, for a project or a team
20:06:53 <dcoutts> some brave people are using it already
20:09:55 <alpounet> sorry, my pizza just got in :P
20:10:11 <dcoutts> np :-)
20:10:42 <alpounet> dcoutts, that's exactly how we've been thinking about scoutess too
20:11:52 <alpounet> for now well, there isn't that much done
20:12:18 <alpounet> kind of discovering new stuffs, like i'm little by little diving into Cabal
20:13:10 <alpounet> but thinking about how scoutess and hackage-server could interact doesn't seem too much work, exactly because there isn't much done now
20:14:27 <alpounet> so yeah it seems normal for a buildbot to have testsuites, so that's something we can consider at some point, but we're really focusing on what's described on the wiki, first
20:14:58 <alpounet> if you have any tips / advices / insight on that, please share it, and also about what's in hackage-server that could help and/or interact with scoutess
20:27:56 <dcoutts> alpounet: so the general idea with hackage-server is that it should be a passive online db
20:28:11 <dcoutts> with clever clients for doing things like build reports, building docs etc
20:28:20 <dcoutts> presenting test suite results
20:28:38 <dcoutts> the storage and presentation side of the system
20:30:16 <dcoutts> alpounet: so the tricky design issue is that there's a fair amount of overlap with what you're thinking of, but the hackage approach is less well suited for development versions
20:30:23 <dcoutts> the continuous build aspect
20:30:35 <dcoutts> but good for releases or versioned snapshots
20:30:57 <stepcut> yeah
20:31:26 <dcoutts> ideally we could work out something that integrates it better
20:31:42 <dcoutts> since while for a public server it's probably best only to deal with releases
20:32:03 <dcoutts> there are many local uses where you want intermediate dev builds
20:32:13 <alpounet> dcoutts, what would you actually need? on-upload action right?
20:32:24 <dcoutts> e.g. we'd like something like this for testing the haskell-platform
20:32:51 <dcoutts> alpounet: I don't understand the question
20:33:43 <dcoutts> alpounet: need for what?
20:33:51 <stepcut> dcoutts: are you thinking of running hackage-server as a local proxy -- where you can override certain packages locally, but then fallback to hackage.haskell.org for the rest ?
20:34:17 <dcoutts> stepcut: that's one use case, but no I was more thinking of as a local repository for a team working on a set of packages
20:34:26 <dcoutts> stepcut: so the server might only contain the output of the team
20:34:42 <dcoutts> and perhaps copies of dependencies
20:34:50 <alpounet> yeah
20:35:00 <stepcut> dcoutts: so they would need the transitive closure of all the packages they depend on ?
20:35:00 <alpounet> that's what we're actually kind of doing
20:35:24 <dcoutts> stepcut: in principle no
20:35:53 <dcoutts> stepcut: because clients can use multiple servers
20:36:07 <dcoutts> so you could say have a read-only mirror of hackage (or subset)
20:36:08 <stepcut> dcoutts: similar to how apt-get works ?
20:36:26 <dcoutts> and a separate writable instance with packages developed by the team
20:36:50 <dcoutts> stepcut: right, we just Data.Map.union the indexes from all listed servers
20:37:09 <stepcut> dcoutts: ok good. That is how I assumed things worked :)
20:37:36 <stepcut> scoutess currently depends on that behaviour
20:39:56 <dcoutts> so I imagine a team in a company with two servers configured in their .cabal/config, one some subset of hackage "current versions validated for internal use" and a second where devs push their internal releases
20:40:08 <dcoutts> the second would also have build & test reports
20:41:05 <dcoutts> stepcut: then what I wonder is how to manage non-releases/snapshots
20:42:26 <stepcut> dcoutts: yes.. that is a bit troublesome. One hackage I thought of for scoutess was having it automatically appending an extra field to the version number to ensure that the number goes up after every commit/bulid
20:42:45 <dcoutts> right, cabal has something similar: cabal sdist --snapshot
20:42:53 <dcoutts> appends the date into the version number
20:42:55 <dcoutts> similar idea
20:43:39 <dcoutts> stepcut: so an incrementing build number would work, then there's UI issues, hackage would list all builds ever :-)
20:44:08 <stepcut> right
20:44:50 <stepcut> for scoutess we have different requirements, so we have a few options
20:45:11 <dcoutts> perhaps it's better to separate them completely, e.g. just link package pages to a separate buildbot page for the package
20:45:11 <stepcut> scoutess is mostly just testing that the build works.. so we can through away the results after we gather the data
20:45:45 <stepcut> but, we might also want to have scoutess keep the latest versions of everything around so that you can do a 'cabal install' from it as well
20:46:05 <stepcut> but, we could still probably get away with only keeping the latest version of the packages around
20:46:10 <stepcut> since it is for development, not release
20:46:33 <dcoutts> stepcut: so it should be easy for scoutess to take the role of providing a package index/archive
20:46:53 <dcoutts> but it's not clear that's the ideal way to integrate
20:46:58 <stepcut> dcoutts: yes. In fact, that is a requirement.
20:47:55 <stepcut> dcoutts: it will watch for changes to source repos, and when it sees a change it will check out the source, create a .tar.gz using cabal sdist, add it to a local index/archive, and then call 'cabal install' to test that it builds and installs correctly (and then also call cabal haddock to generate docs)
20:48:11 <stepcut> but, for that, we need only provide 00
20:48:28 <dcoutts> stepcut: oh but not providing an archive for other clients?
20:48:33 <stepcut> 00-index.tar and the .tar.gz files.. we do not need to actually have a user browseable hackage server
20:48:36 <dcoutts> just for it's internal builder client?
20:48:45 <alpounet> yes
20:49:03 <dcoutts> it's not actually necessary to make an index for the  internal builder client
20:49:08 <stepcut> well, having internal for the build client is a requirement.. making it accessible to other clients is desirable
20:49:11 <dcoutts> if you know what versions of all deps you want to use
20:49:16 <alpounet> the scope of that index is just the "ongoing build trial"
20:49:50 <stepcut> dcoutts: making the 00-index.tar is pretty easy.. and we want to try to simulate what would happen if you were doing a 'cabal install' from hackage as closely as possible..
20:49:53 <alpounet> dcoutts, the point is to see how the project copes with newer commits/versions/whatever of its dependencies
20:50:06 <dcoutts> alpounet: sure
20:50:07 <alpounet> there are 2 different directions
20:50:42 <alpounet> 1/ simulate a cabal install on a fresh platform install, so here we're going to test against the latest versions of the dependencies from hackage
20:50:48 <dcoutts> stepcut: so you actually want to let cabal install decide what deps to use, using it's solver?
20:51:09 <dcoutts> I'd have thought you'd want to have more direct control over deps used
20:51:10 <alpounet> 2/ see if the project still builds with the HEAD versions of the dependencies, when the repo is provided ofc
20:51:17 <stepcut> dcoutts: right
20:51:46 <alpounet> dcoutts, we can, but is there a need for this?
20:51:59 <stepcut> dcoutts: well.. perhaps we will have to have more control.. but, in theory, we are trying to test what would happen if we actually bumped the version and made an upload to hackage
20:52:05 <alpounet> like, being able to specify a range for the deps, and all versions of that range get tested?
20:52:46 <dcoutts> stepcut: once we make the solver in cabal-install available as a lib, I'd take that route
20:53:40 <stepcut> dcoutts: sure.. right now the answer I want to know is, "Does, 'cabal install happstack-server' work using on a clean system with nothing but haskell platform installed"
20:54:03 <stepcut> and I want to test that against the real hackage server nightly, as well as the development sources when the new commits are made
20:54:17 <dcoutts> the requirements are somewhat at odds
20:54:20 <stepcut> we do need to do some level of dependency solving to answer that
20:54:27 <dcoutts> since you want to know if the user's tools will work
20:54:55 <dcoutts> but also want to run "what if" scenarios where you have control over what versions of things get picked
20:55:06 <stepcut> sure
20:55:38 <stepcut> though, so far, the what-ifs that I am most concerned about are ghc/haskell-platform versions and different OSes
20:56:52 <stepcut> though I could be convinced otherwise..
20:58:42 <dcoutts> stepcut: so a similar feature in hackage-server is the "tentative upload" thing where you can upload a beta and get test results etc and if you're happy hit "publish"
20:59:17 <stepcut> neat
20:59:19 <dcoutts> stepcut: I can't help thinking that there should be some design that'd cover these use cases more fully
20:59:28 <dcoutts> or some way to partition it sensibly
20:59:34 <stepcut> maybe
20:59:47 <alpounet> you're slowly moving some of the work to scoutess, aren't you?
20:59:48 <alpounet> :P
21:00:23 <stepcut> I am in favour of reusing code.. that less optimistic about it than I used to be :)
21:00:30 <stepcut> s/that/but/
21:00:43 <alpounet> well
21:00:57 <alpounet> we do have some overlap with both Cabal/cabal-install, and hackage-server
21:01:02 <stepcut> yes
21:01:07 <alpounet> at some point we'll have to decide what to do about that
21:01:21 <dcoutts> I'm not thinking about the code so much as the user experience for people using both
21:01:25 <alpounet> factor these common things in a library? try to somehow stay in sync? ...
21:01:54 <stepcut> dcoutts: is cabal-install available as a library? Or do we need to shell out to the executable ?
21:01:58 <dcoutts> alpounet: we want to make a hackage-client lib with much of the common stuff, index handling, solver
21:02:07 <dcoutts> stepcut: not yet
21:02:12 <stepcut> dcoutts: k
21:03:11 <stepcut> dcoutts: my feeling is that the basic builder for scoutess should be less than 3000 lines of code. So, the easiest thing to do might be to build it first, and then look at how to factor out things/increase integration later
21:04:13 <dcoutts> stepcut: sure, makes sense
21:04:39 <dcoutts> I'm just trying to think what the goal might be later on
21:04:52 <stepcut> yeah
21:05:45 <stepcut> in theory, the building stuff does not need to be in  scoutess at all
21:06:08 <dcoutts> we're going to have a builder for hackage-server too
21:06:17 <dcoutts> but only for released and beta packages obviously
21:06:20 <stepcut> the idea of scoutess is that it gathers information from a bunch of resources and presents it to you in a organized coherent fashion
21:06:28 <alpounet> that's basically virtualhenv/hsenv/whatever + cabal-install + the proper package index anyway
21:07:24 <stepcut> so, that includes information like, 'does the latest development code build', 'does the code on hackage still build', 'can the packages on hackage use all the latest dependencies', 'are there recentl questions about this project on StackOverflow', etc
21:07:42 <dcoutts> stepcut: right, nice
21:07:45 <stepcut> if we can get that information from hackage.. then there is no reason for us to calculate it ourselves..
21:07:46 <alpounet> uh, SO? :D
21:07:48 <alpounet> nice idea
21:08:01 <stepcut> though for closed source projects.. some of that information could not come from the centralized hackage server itself
21:08:24 <dcoutts> stepcut: our idea with hackage's build results was starting from the other end: gather as many build results as possible (build bots + users) and then figure out what to do with the info
21:08:30 <dcoutts> like "does it build on windows"
21:08:42 <stepcut> yeahÂ… maybe I have not found the secret link yet.. but I would really like to be notified af when people post questions about happstack on stackoverflow :)
21:08:53 <stepcut> dcoutts: right
21:09:31 <alpounet> stepcut, i had written something that analyzes the HTML code itself... was hacky but worked
21:09:43 <stepcut> dcoutts: that question is very important to me.. happstack is officially supported on Linux, OS X, FreeBSD, and Win32.. And I do have all those environments at home, but I only test on Linux regularily.. which is not cool
21:10:05 <dcoutts> stepcut: right, so the "beta" feature should be really useful to you there
21:10:32 <dcoutts> stepcut: you'd upload, wait for people's build bots to report, check the results, publish or upload a revision
21:10:52 <stepcut> dcoutts: yeah
21:10:56 <dcoutts> the nice thing is anyone can set up a build bot, so we should get good platform coverage
21:11:11 <stepcut> scary!
21:11:37 <stepcut> haskell buildbots are scary.. given that fact that between runIO in TH and Setup.hs you can do just about anything you want ;)
21:12:04 <dcoutts> it's client's responsibility to sandbox or to only build packages from people they trust or whatever
21:12:08 <stepcut> though, hackage has gotten along pretty well so far on the 'haskell users are not jerks' ideal :)
21:12:42 <dcoutts> but otoh, we're not actively directing the build clients to try particular configurations / versions
21:13:17 <dcoutts> but it'd be entirely possible to configure a build client to do things like "use the HP versions" or "use the latest versions"
21:14:34 <stepcut> yeah
21:14:36 <dcoutts> stepcut: so it's a nice idea, but does not cover your use case of the dev versions
21:14:45 <stepcut> yeah
21:15:07 <alpounet> and we don't really have a notion of "client"
21:15:18 <alpounet> even if we plan to have build slaves support
21:15:26 <alpounet> that's not really the same concept
22:14:50 <alpounet> dcoutts, so, you suggest me to borrow some of cabal-install's code for getting the latest (and "preferred") version of a package?
22:22:01 <dcoutts> alpounet: seems reasonable
22:22:34 <dcoutts> alpounet: for the moment you can get away with ignoring the preferred versions
22:24:01 <alpounet> dcoutts, yeah i guess i'll just figure out the way to do this when i'll dive into cabal install's code