Sunday, June 1, 2008

Why to use PEX (Program EXploration for .NET). Sample 1. Reason 1.

Sample under test:

public static class XmlUtility
    {
        public static string Encode(char input)
        {
            switch (input)
            {
                case '\n': return "
";
                case '\r': return "
";
                case '&': return "&";
                case '\'': return "'";
                case '"': return """;
                case '<': return "&lt;";
                
                default: return new string(input, 1);
            }
        }
    }

Non-PEX test sample:

        [TestMethod()]
        public void EncodeTest()
        {
            Assert.AreEqual<string>("&#xA;", XmlUtility.Encode('\n'));
            Assert.AreEqual<string>("&#xD;", XmlUtility.Encode('\r'));
            Assert.AreEqual<string>("&amp;", XmlUtility.Encode('&'));
            Assert.AreEqual<string>("&apos;", XmlUtility.Encode('\''));
            Assert.AreEqual<string>("&quot;", XmlUtility.Encode('"'));
            Assert.AreEqual<string>("&lt;", XmlUtility.Encode('<'));
            
            Assert.AreEqual<string>("a", XmlUtility.Encode('a'));
        }

PEX test sample:

[PexMethod()]
        public void EncodeTest(char input)
        {
            if (input == '\n') { Assert.AreEqual<string>("&#xA;", XmlUtility.Encode(input)); return; }
            if (input == '\r') { Assert.AreEqual<string>("&#xD;", XmlUtility.Encode(input)); return; }
            if (input == '&') { Assert.AreEqual<string>("&amp;", XmlUtility.Encode(input)); return; }
            if (input == '\'') { Assert.AreEqual<string>("&apos;", XmlUtility.Encode(input)); return; }
            if (input == '"') { Assert.AreEqual<string>("&quot;", XmlUtility.Encode(input)); return; }
            if (input == '<') { Assert.AreEqual<string>("&lt;", XmlUtility.Encode(input)); return; }
            
            // Everything else should not be encoded
            Assert.AreEqual<string>(new string(input, 1), XmlUtility.Encode(input));
        }

Note, I had to use ifs instead of switch/case as I had a bit of an issue to make PEX working as expected  in case of switch statement in the test method.

So what the difference does it make?

In both tests we effectively provided parameters to make test coverage 100% (sorry for a bit of approximation). Both tests pass.

Though, there is a problem with algorithm and it doesn't encode the '>' input. TDD approach wise we would add a test that makes our encoding method fail, and fix our encoding to encode '>'. TDD purists are known for eating people alive for not following that "test first" strategy, but even following this TDD approach you may not be able to protect against simply overlooking something.

So lets be non TDD and add a line of code to fix our issue:

case '>': return "&gt;";
 
Now lets rerun the tests:
There is no reason why current non-PEX test would fail, so it really passes.
Lets see how PEX test is doing:

image
 
And now we can see a bit of the PEX magic in action!
Behind the scenes, looking at our code under test, PEX generated an extra test method for input of '>' and this failed our
Assert.AreEqual<string>(new string(input, 1), XmlUtility.Encode(input));

"assert for the rest of cases" statement.

So now, I'm actually explicitly forced to add the test case for '>', which I'm doing:

if (input == '>') { Assert.AreEqual<string>("&gt;", XmlUtility.Encode(input)); return; }

To make the PEX tests pass:

image

Conclusions:
1. I should be better on writing unit tests, but I can never be perfect.
2. PEX can really give me a hand for catching some of my possible mistakes in unit tests.

Ultimate link for PEX (Program Exploration for .NET) resources: http://research.microsoft.com/Pex/
First user experience: http://blog.benhall.me.uk/2008/05/microsoft-pex-05-released.html
TDD purists may be watching you! http://intellij.net/forums/thread.jspa?messageID=5215367 (scroll down a bit until the Gabriel Lozano's  post, I had fun reading ...)

7 comments:

Anonymous said...

Hi Stanislav,

Could you give us more details on the switch issues? I tried to repro on my side and the test using the switch implementation worked as well as the if structure (besides, the switch statement gets translated into switch and if structure).

Cheers, Peli

stan said...

Hi Peli. I'm probably just overlooking something then. I sent a sample code to pex@list.research.microsoft.com. Hope it will reach you alright.

Anonymous said...

Regarding the switch: From your mail to pex@list.research.microsoft.com it looks like Pex got confused from an intermittent security check that .NET performed. We'll try to make Pex more resilient in the next release. Btw: I have written another article that shows how to use Pex.

Andrew said...

Hi Stanislav,

Glad to see you're interested in Pex too! :)

Are you using it in "real life" - or just playing with it?

handyman Orange Park said...

I tried to repro on my side and the test using the switch implementation worked as well as the if structure (besides, the switch statement gets translated into switch and if structure).

stan said...

This is quite an old article, I guess they have fixed it since then, good to have it confirmed though!

Orlando Selenu said...

Thank you very much for your article about PEX. We really appreciate tools like PEX, and an insight like yours is precious.

I hope the PEX developing continues (and remains Visual Studio 2008 compatible).