publius_ovidius

As an addendum, the language my test suite would be in if we had one is C++.

That's for my main code repo. The Java project I just inherited came with no testing framework... ::sigh::

And after today's meeting it looks like I have co-conspirators who are as interested in getting a test suite in there as I am. Looks like we'll be using TestNG.

Recalling back to my days of large Perl projects, I was never too impressed with Test::Harness. I mean, it's certainly decent, but it just didn't seem to facilitate the sorts of quick tests that are necessary to get good unit coverage and not get bogged down. I've always thought that testing should be more agile than coding, else the suites fall quickly into disrepair. I wonder how much of my current lovely setup for Haskell is due to the language's purity, vs how much has to do with the tools themselves.

After you've used TestNG for a bit (or if you already have) could you please let me know your thoughts on it? I was reading through it to get some ideas for TAP, but I wasn't terribly impressed with what I saw. I mean the framework is powerful enough, but there's a bit to be desired. For example, from this Web site, an example of output XML:

<test-method status="PASS" signature="test2()" name="test2" duration-ms="0"
             started-at="2007-05-28T12:14:37Z" description="someDescription1"
             finished-at="2007-05-28T12:14:37Z">
</test-method>

Well, great. We've now duplicated the duration by having a start and end time and a duration. What happens if they don't match? I've found nothing about this in the description.

And if we look at that full xml to see context:

<testng-results>
    <suite name="Suite1">
        <groups>
            <group name="group1">
                <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
                <method signature="com.test.TestOne.test1()" name="test1" class="com.test.TestOne"/>
            </group>
            <group name="group2">
                <method signature="com.test.TestOne.test2()" name="test2" class="com.test.TestOne"/>
            </group>
        </groups>
        <test name="test1">
            <class name="com.test.TestOne">
                <test-method status="FAIL" signature="test1()" name="test1" duration-ms="0"
                             started-at="2007-05-28T12:14:37Z" description="someDescription2"
                             finished-at="2007-05-28T12:14:37Z">
                    <exception class="java.lang.AssertionError">
                        <short-stacktrace>java.lang.AssertionError
                            ... Removed 22 stack frames
                        </short-stacktrace>
                    </exception>
                </test-method>
                <test-method status="PASS" signature="test2()" name="test2" duration-ms="0"
                             started-at="2007-05-28T12:14:37Z" description="someDescription1"
                             finished-at="2007-05-28T12:14:37Z">
                </test-method>
                <test-method status="PASS" signature="setUp()" name="setUp" is-config="true" duration-ms="15"
                             started-at="2007-05-28T12:14:37Z" finished-at="2007-05-28T12:14:37Z">
                </test-method>
            </class>
        </test>
    </suite>
</testng-results>

Wow. That's breath-takingly ugly. I compare that to the equivalent in TAP with our optional YAML diagnostic syntax:

1..2
not ok 1 - someDescription2
    ---
    signature: test1()
    name: test1
    start: 2007-05-28T12:14:37Z
    end: 2007-05-28T12:14:37Z
    exception:
        class: java.lang.AssertionError
        stacktrace: |
            java.lang.AssertionError
            ... Removed 22 stack frames
ok 2 - someDescription2
    signature: test2()
    name: test2
    start: 2007-05-28T12:14:37Z
    end: 2007-05-28T12:14:37Z
    xtra:
        is-config: true

The beauty of that is a TAP parser only needs to see this:

1..2
not ok 1 - someDescription2
ok 2 - someDescription2

And everything else is optional gravy, making it an extremely useful test protocol. You can implement the core of a complete parser in just a few lines of code and add more as you go. Regrettably, we're still waiting on Schwern's work on Test::Builder 2 before we can get the YAML diagnostics completely working :/

Does TestNG have any way of marking tests as "skip" or "todo"? (There's a way to exclude sets of tests, but that's not quite the same thing as "skip".

By the way, I've been using TestNG too, but I'm not too advanced a programmer, ahaha, only thing I can say about it - it starts up much slower than jUnit. It seems to run fine in suites, but single tests take about 40 seconds if run separately. Though maybe it is only the specifics of that repository I'm currently working with.

I'm not too familiar with TestNG yet, but it's what the other guy knows and I don't have much invested in JUnit or NUnit either so I'm willing to try it. One of the noted advantages over JUnit is that you can make tests depend on other tests so that if some basic test fails you don't bother with running the dependent tests since you already know they'll fail too.

Once I've worked with it a bit I'll let you know what I think.

Oops. Didn't see the sentence "That being said, if you're using an XUnit framework, "number of tests" means, in this context "number of asserts" (in other words, a test method with three asserts would count as 3 tests, not one)."

Then it'd be "501 to 2,000 tests", I guess.

You can change your poll answers by clicking on the poll number at the top, then on "Fill out poll".

Thanks!

If the answer to the first question is no, we can't really answer any of the others.

Our code is mostly not structured appropriately to run isolated tests, and as a department we're still learning both how to do that and how to design appropriate tests. I think we may end up with bigger tests against a test database, first.

In that case, high-level integration tests targeting the UI or API can nail down basic behavior before you refactor to more testable code. UI and API integration tests are typically harder to debug, but they're great for finding bugs that unit tests can't catch.

In a related note, if anyone knows of a UI test for XUL applications, I'd be thrilled if they would tell me.

...answers based on the last time I was working on a code base of any significance, which is to say at my last job. I am developing a new set of tools now from scratch, for which the answers will be mostly the same except for the number of tests (currently no idea) and the percentage of code covered (well over 50%, hopefully 100%, but in practice as close as 100% as I can get without feeling like I'm wasting time.)

How am I to answer questions 2 through 7, when the answer to 1 is "no"?
There is no "none of the above" or "not applicable" option.

Sorry about that. I should have made it clear that you could skip the rest of the poll if need be.

I don't really have a primary codebase, so I answered for the most recent module I've uploaded to the CPAN and which I think is actually finished - CPU::Emulator::Z80.

We don't have a primary codebase at work either - there's several independant ones.

Until recently I was the only developer on what could be considered our primary code base; when I inherited the code there was no test suite, no version control, and all work was done in production. I've slowly (very slowly) built a test suite over the past year, though it is still greatly lacking. Now that we have a second developer to work on some of the backlog, I've started to move my focus to testing.