I have (and others here as well) have experience with CI tools like Jenkins (previously Jeeves). For those wondering what CI is like, here is a rough overview as it applies to OpenMW:
If you would like to read more:*Developer pushes something to git repository
*A post-receive hook pushes that commit on to a stack of commits to check for that tree.
*Build bot pops next testable sha1 from the stack
*Build bot checks out the code and runs a test script (compile, link, run, unit tests)
---
*If the test script succeeds, logs that the SHA1 passed tests. Test results are pushed to a webserver for later viewing.
*If the test script fails, logs the failure and collects artefacts.
---
*Test results get pushed.
*Developers get notified.
*After a failure, build bot uses git bisect to find a new commit between the last known good and the current. This becomes the next testable commit.
http://en.wikipedia.org/wiki/Continuous_integration
The tests could be anything with an expected output, like ordering of esx files or known bad esm/esx/plugins should be rejected and not accepted, Morrowind.ini gets processed correctly or if bad data in Morrowind.ini that the importer doesn't blow up and exit badly. These are the type of things we can test for.
I would love to hook directly into functions or at least be able to play with object data via a command-line option like --debug so that we can automate these type of testing.
Things like 'invisible armour' would be non-trivial to test for this way.
What do you guys think?