TITLE OF PAPER: Literate unit testing/ Unit Testing with Doctest URL OF PRESENTATION: _URL_of_powerpoint_presentation_ PRESENTED BY: Jim Fulton REPRESENTING: _name_of_the_company_they_represent_ CONFERENCE: PyCON 2004 DATE: 20040325 LOCATION: _venue_and_room_in_venue_ -------------------------------------------------------------------------- REAL-TIME NOTES / ANNOTATIONS OF THE PAPER: {If you've contributed, add your name, e-mail & URL at the bottom} Review of unit testing Why: quality, think first, stress reduction, freedom to change, documentation Tests are meta-programs tests are programs about programs (a form of meta-program) like anything meta, reasoning about tests can be hard Python's answer: unittest (aka: PyUnit) inspired by JUnit Complex framework for writing tests: test case, test fixture, test runner.. inheritance based API methods to make test assertions Sample: class myClassTestClass (unittest.TestCase): def testOne(self): self.assert_(1 ==1, '1 not equals 1!!!') def test_suite(): return unittest.makeSuite(myClassTestClass) if __name__ == "__main__": unittest.main() Points out that unittest based code doesn't really look like client code. There's too much scaffolding unittest problems Unittest doesn't seem to scale (you might want to write utility modules/code to find all your tests and run them) (they have something like this for Zope) Too much support for abstraction Abstraction can obscure regular code Makes intent less clear in a test Test code doesn't look like client code Words are harder to write than code Focus is testing In Zope's experience, tests are poorly documented ("hmm.. what IS this test doing???") Result: too hard to fix when the test code breaks doctest Correct examples are priceless But incorrect examples worse than worthless Examples in docs become wrong over time as code changes deliberately as endcase behaviors change accidentally (as python changes) as Python changes! Python tutorial a prime example -always wrong about some current detail (and that sucks for newbies) doctest origin - interactive testing "Code a little, test a little" Flesh out the code, test in the shell, repeat best tests written might get thrown away this way Soln: paste those tests into the docs doctest ensures the examples run exactly as advertised forever after (if they don't, doctest will tell you) improves coverage by verifying normal, endcase, and error cases improves docs by demonstrating examples of each - examples are often clearer than prose "literate testing" or "executable docs" doctest example doctest advantages: looks like client code samples always up to date (cause if they're not they fail as tests) DocTestSuite (integration of doctest and unittest) creates a unittest from module tests typically in seperate file What changed? focus shifted on documentation "no" testing calls what you see is what the client does less noise ("no" testing code) connected to implimentation don't need to learn an API to start writing tests --> focus on documentation gets much more important as things get sophisticated!!! Where do you put them? Implementation, but only if it enhances docus In separate tests file. I like to use functions with just doc strings as test holders. In the future we'll often put tests in separate documentation files Telling stories with tests When documenting something we often want to tell a story Long naratives can be especially useful for teasing out dynamic behavior Goes against the general principle of keeping tests independent Doctest Pitfalls Poor debugging support Output must be the same on every run Avoid output containing addresses Dictionary order Floats Output can't contain blank lines Can be hard to fit in 79 characters set up and tear down code can detract from story/sample Zope has ~5600 tsets (~ 3500 in Zope 3, ~1300 in ZODB, ~800 in Zope 2) lessons learned: debugging failed tests is really hard when intent is unclear Future directions Improved unittest integration Better debugging support doctest pretty printer Support for separate set-up and tear-down hooks Limited support for test reuse Use of examples from documentation files Prefers that doctest doesn't support inheritance: Inheritance => abstraction => abstraction reduces clarity Q: Are you transitioning all tests to doctest A: No, but all new tests will probably be doctests Q: Can you do TDD w/ doctest? A: Yes. -------------------------------------------------------------------------- REFERENCES: {as documents / sites are referenced add them below} PAPER: http://www.python.org/pycon/dc2004/papers/4/PyCon2004DocTestUnit/ -------------------------------------------------------------------------- QUOTES: "Specifications written by people who can't write, implemented by people who can't read" -------------------------------------------------------------------------- CONTRIBUTORS: {add your name, e-mail address and URL below} Ted Leung, twl@osafoundation.org, http://www.sauria.com/blog Ryan Wilcox, ryanwilcox@mac.com, http://radio.weblogs.com/0100544/ -------------------------------------------------------------------------- E-MAIL BOUNCEBACK: {add your e-mail address separated by commas } -------------------------------------------------------------------------- NOTES ON / KEY TO THIS TEMPLATE: A headline (like a field in a database) will be CAPITALISED This differentiates from the text that follows A variable that you can change will be surrounded by _underscores_ Spaces in variables are also replaced with under_scores This allows people to select the whole variable with a simple double-click A tool-tip is lower case and surrounded by {curly brackets / parentheses} These supply helpful contextual information. -------------------------------------------------------------------------- Copyright shared between all the participants unless otherwise stated...