Testing iPhone location app with Automation

Even if we have now ability to add GPX files to the project and iPhone will even "move" between its points, still what if we want to add speed, altitude etc to our location simulation?

It is all easy and possible! Welcome to Automation tool in Instruments. Let me use my speedometer app as a showcase for it. I'll cram in as much as possible in this single session: find memory leaks while my app is continuously processing (simulated) location changes and gather screenshots for the AppStore (you don't really expect me to do it while I drive?! I may not be here then for the next article :)).

Start up automation from xcode (e.g. do alt-run) with target of your physical iphone:

I'll start with leaks profile:

The app starts on the phone and starts recording automatically. No leaks, even if I've waited for a year. As it does nothing... So lets add some location changes that app have to handle:

Actually, you may stop the recording at any time. We are setting things up right now :). Lets even clean up and delete that idle "run" we produced:

Then, open the library of Instrument tools, and then drag the Automation below the Leaks tool:

Now its time to add an automation script, click on Add and then Create:

Here is my script to simulate a few points:

 var target = UIATarget.localTarget();  
 // speed is in meters/sec  
 var points = [  
                 {location:{latitude:48.8899,longitude:14.2}, options:{speed:8, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:48.8899,longitude:14.9}, options:{speed:11, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:48.8899,longitude:14.6}, options:{speed:12, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:48.8899,longitude:14.7}, options:{speed:13, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:49.2,longitude:14.10}, options:{speed:15, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:49.4,longitude:14.8}, options:{speed:15, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:48.8899,longitude:14.9}, options:{speed:9, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:48.8899,longitude:15.1}, options:{speed:8, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 {location:{latitude:48.8899,longitude:16.1}, options:{speed:3, altitude:200, horizontalAccuracy:10, verticalAccuracy:15}},  
                 ];  
 for (var i = 0; i < points.length; i++)  
 {  
      target.setLocationWithOptions(points[i].location,points[i].options);  
      target.captureScreenWithName(i+"_.png");  
      target.delay(1.0);  
 }  

Note, that in addition to location it includes speed, altitude, and accuracy. Speed and accuracy is what is indispensable to simulate the speedometer behaviour correctly per algorithms implemented.

Keep the other options intact for a moment (script starts with recording) and for leaks investigations we don't need to log the results yet.

Let's start a recording now and see if plain old Stan is going to be ashamed. As script runs enjoy the screen of your app as it reacts on location changes, ooh that power!

Yep, plain old Stan should be ashamed at least a bit. There is a leak! It actually leaks on every location change... I'll let you go through your backtrace etc, Intruments give you enough to find where leak is coming from. In my case I forgot to release the CLLocation retained as a property when I was deallocing the holding object. After the fix, I just redeployed the app to the phone and kicked off recording again. No leaks whatsoever!

So now the bonus part. If you want to automate screenshots gathering for you app just pay attention to this line in the above for loop:

target.captureScreenWithName(i+"_.png");  

For every location/speed point I take a screenshot. To keep them on the disk, setup the logging directory and switch on that logging (below the Add script button). This is enough to get this all safely written to the target directory:

This is it I guess! Make sure you explore everything around as I provided only the essence. See the Editor log, Trace log for automation, it all gives some more insight.

Happy testing and simulating!

More on location testing from me: GPX files in xcode.

Gallio Icarus Tests Runner silently crashing on start

This one did not require any debugging as problem declared itself also with gallio control panel showing exception when trying to read an empty settings file.

So, somehow between gallio test gui restarts and its intermittent failures, it managed to emptify that file I guess.
Just deleting everything from C:\Documents and Settings\[user]\Application Data\Gallio\Icarus cured the problem.

Extending Silverlight Unit Testing framework – a shallow look.

Last week I had to do a bit of a digging in the SL unit test framework guts (the one that is part of an SL4 toolkit). The justification for that was a need to execute a service call in between test (or work items) invocations.

Specifically, I had to restore server databases to the original snapshot (via wcf service) after data has become “dirty” and we needed a clean state to continue tests with.

There are 1000+1 ways how to achieve anything in SL, and after digging and implementing I got actually other ideas of how this can be done. But exercise was good for educational purposes nonetheless. Please note, that the code is only a first draft conceptual level implementation, so it looks and is dirty.

Here is a diagram at the level I only had to dive into:

You enter into tests with creation of a test page by passing instance of UnitTestSettings to CreateTestPage method of UnitTestSystem:

private void Application_Startup(object sender, StartupEventArgs e)

        {

            RootVisual = UnitTestSystem.CreateTestPage(HarnessProvider.CreateDefaultSettings(this.GetType().Assembly));

        }

As diagram should be read from bottom to the top let me apply the same narration style :). We do provide customized settings by means of custom HarnessSettingsProvider class:

    public class HarnessSettingsProvider

    {

        public static UnitTestSettings CreateDefaultSettings(Assembly callingAssembly)

        {

            var settings = new UnitTestSettings();

            if (callingAssembly != null)

            {

                settings.TestAssemblies.Add(callingAssembly);

            }

            settings.TestHarness = new CustomTestHarness();

            settings.TestService = new SilverlightTestService(settings);

            return settings;

        }

    }

The only point of customization was settings.TestHarness = new CustomTestHarness(); where we assign our own test harness:

    public class CustomTestHarness : UnitTestHarness

    {

        public override void RestartRunDispatcher()

        {

            this.RunDispatcher = new CustomFastRunDispatcher(new Func<bool>(this.RunNextStep), this.Dispatcher);

            this.RunDispatcher.Complete += new EventHandler(this.RunDispatcherComplete);

            this.RunDispatcher.Run();

        }

        public bool DBCleanupRequired

        {

            get { return ((CustomFastRunDispatcher)RunDispatcher).DBCleanupRequired; }

            set { ((CustomFastRunDispatcher)RunDispatcher).DBCleanupRequired = value; }

        }

    }

Once again, only tiny change here – assignment of our own CustomFastRunDispatcher and introduction of DBCleanupRequired property.

UnitTestHarness is a guy who runs a dispatcher and that is where the core of problem solution is provided:

public class CustomFastRunDispatcher : FastRunDispatcher

    {

        private readonly Func<bool> _runNextStep;

        private readonly Dispatcher _dispatcher;

        private volatile bool _dbCleanupRequired = true;

        public ITestService TestPreparationService { get; set; }

        public CustomFastRunDispatcher( Func<bool> runNextStep, Dispatcher dispatcher ) : base(runNextStep, dispatcher)

        {

            _runNextStep = runNextStep;

            _dispatcher = dispatcher;

            TestPreparationService = new ChannelFactory<ITestService>(typeof(ITestService).Name).CreateChannel();

        }

        public bool DBCleanupRequired

        {

            get { return _dbCleanupRequired; }

            set { _dbCleanupRequired = value; }

        }

        public override void Run()

        {

            if (DBCleanupRequired)

            {

                TestPreparationService.BeginPrepareDatabases(HandlePreparationCallback, null);

            }

            else

            {

                this._dispatcher.BeginInvoke(() => { RunNext(); }); 

            }

        }

        private void HandlePreparationCallback(IAsyncResult ar)

        {

            this._dispatcher.BeginInvoke(

                () =>

                {

                    DBCleanupRequired = false;

                    TestPreparationService.EndPrepareDatabases(ar); 

                    RunNext();

                });

        }

        private void RunNext()

        {

            if (IsRunning || _runNextStep())

            {

                Run();

            }

            else

            {

                OnComplete();

            }

        }

    }

}

As you can see, in case when database state is marked for clean up we call a custom TestPreparation wcf client and only in its callback we progress with the next step (next work item). Again as this is only a draft concept, no exception handling here and it is still subject to verify what unhandled exception in End invocation will do to SL unit testing framework.

In SL unit testing framework we execute “work items”. We can see that for example EnqueueCallback queues actually a CallbackWorkItem for execution:

public virtual void EnqueueCallback(Action testCallbackDelegate)

{

    this.EnqueueWorkItem(new CallbackWorkItem(testCallbackDelegate));

}

Knowing this, the following test can show usage and certain advantage of the proposed customization:

        [TestMethod]

        [Asynchronous]

        public void Should_RestoreDatabasesOnDirty()

        {

            // First callback work item is going to mark db as dirty

            EnqueueCallback(() =>

            {

                MarkDatabaseAsDirty();

            });

            // Prior to the second call back work item we should have called the service

            // to restore to the original snaphots and we expect that DBCleanupRequired

            // is set to false on service async callback. As a bit of a manual check we 

            // can verify that service really was called between those two work items.

            EnqueueCallback(() =>

            {

                Assert.IsFalse(((CustomTestHarness) UnitTestHarness).DBCleanupRequired);

                EnqueueTestComplete();

            });

        }

Where MarkDatabaseAsDirty is a method of a base test class that sets [Custom]UnitTestHarness DBCleanupRequired property.

This is it for my exploration and first draft concept on the subject of extending Silverlight unit testing framework.

Run unit test on MTA thread. VSTS test runner helper.

How much trouble is required to make you invent your own wheel? For me it was first vstesthost exiting on the unhandled exception in the worker thread and then VS running tests by default on the STA thread finally broke my back and I decided to put a small helper together to rectify all the above troubles.

Why to run on the MTA thread? Sometimes you use functions like WaitHandle.WaitAll that do require to be executed on the MTA thread. The exception you get is "WaitAll for multiple handles on a STA thread is not supported". You might as well have to use MTA because of specific COM needs.

You can setup the local test run configuration to run tests on the MTA thread as described here: http://blogs.msdn.com/ploeh/archive/2007/10/21/RunningMSTestInAnMTA.aspx
Although the limitation of this solution is that it is global. In MbUnit you can setup the apartment as a property of a TextFixture attribute, that again I believe may not be granular enough.

I want to be able to run a separate call as MTA or STA really, because doing it can actually be the thing under scope.

Decided so, surely we need to resolve unhandled exceptions code under test may throw that would lead to the unpleasant death of our test host ("VSTestHost.exe has encountered a problem and needs to close.  We are sorry for the inconvenience."). Whatever MS says about this being by design (http://forums.microsoft.com/MSDN/ShowPost.aspx?PostID=92232&SiteID=1), this is a bad example of a blind application of the same design decision to all possible contexts (although safe default is required, handling failures like this always require flexible and context aware solutions).

That is why I put together a helper class (the most recent version available at http://code.google.com/p/toolsdotnet/source/browse/trunk/Tools.Net/src/Tools.Tests.Helpers/TestRunner.cs):

using System;

using System.Threading;

namespace Tools.Tests.Helpers

{

    public class TestRunner

    {

        private Action action;

        private ApartmentState apartmentState;

        private Exception exception;

        public TestRunner(Action action, ApartmentState apartmentState)

        {

            this.action = action;

            this.apartmentState = apartmentState;

        }

        public void Execute()

        {

            // Setup a worker thread

            Thread workerThread = new Thread(new ThreadStart(ExecuteInternal));

            // Set apartment

            workerThread.SetApartmentState(apartmentState);

            workerThread.Start();

            // Wait until work on the worker thread is done

            workerThread.Join();

            // Probe for unhandled exception

            if (exception != null)

            {

                // If exception is present, rethrow here on the main thread

                throw exception;

            }

        }

        private void ExecuteInternal()

        {

            // wrap our original action in the try/catch

            try

            {

                action();

            }

            catch (Exception ex)

            {

                Console.WriteLine(ex);

                // Don't consider race to happen here, subject to think more

                exception = ex;

                // Don't rethrow here as that would kill the test host

            }

        }

    }

}

The usage would be (sorry for such a big sample, don't have time to make it shorter now :):

            var consumerManager = new ConsumerManager();

            var target = new ConsumerManager_Accessor(new PrivateObject(consumerManager));

            new TestRunner(() =>

            CompositePatternTestHelper.TestForCompositeOperation<ConsumerManager_Accessor, IProcess, TrivialAsyncResultMock>

                (

                target, parent => parent.Stop(), child => { child.BeginStop(null, new AsyncCallback(target.ConsumerStoppedCallback)); return new TrivialAsyncResultMock(); }, (parent, child) =>

                    parent.Consumers.Add(child)

                    ),

                    ApartmentState.MTA).Execute();

            Assert.AreEqual(ProcessExecutionState.Stopped, consumerManager.ExecutionState);

And it runs ok, if I change the above sample code to run under STA (ApartmentState.STA).Execute();) I'm getting:

Which is expected.

Testing the composite pattern - reusable test helpers

I face quite often the need to test a composite pattern variations. And truly was duplicating some similar code around. As tests are now my focus for self-improvement (and always, should I be eaten alive by the tdd purists if I lie), I decided to extract some of the snippets into reusable helper methods.
Here is what I got for the calling of the composite pattern "operation":

The usage:

CompositePatternTestHelper.TestForCompositeOperation<ProcessCoordinator, IProcess>(

                parent => parent.Stop(), // Parent operation

                child => child.Stop(), // Should invoke following on every child

                (parent, child) => parent.Processes.Add(child) // How to add add child to the parent

                );

And for the helper method/class implementation (The most recent version available at http://code.google.com/p/toolsdotnet/source/browse/trunk/Tools.Net/src/Tools.Tests.Helpers/CompositePatternTestHelper.cs):

using System;

using Rhino.Mocks;

namespace Tools.Tests.Helpers

{

    public static class CompositePatternTestHelper

    {

        /// <summary>

        /// Helper method to test composite [parent/child] pattern implementation, where calls

        /// to the parent result into calls onto its children.

        /// </summary>

        /// <remarks>Creates the parent object using its default constructor</remarks>

        public static void TestForCompositeOperation<ParentType, ChildType>(Action<ParentType> parentAction, Action<ChildType> childAction, Action<ParentType, ChildType> addChild)

            where ChildType : class

            where ParentType : new()

        {

            // Requires a default ctor to exists

            var parent = new ParentType();

            TestForCompositeOperation(parent, parentAction, childAction, addChild);

        }

        /// <summary>

        /// Helper method to test composite [parent/child] pattern implementation, where calls

        /// to the parent result into calls onto its children.

        /// </summary>

        /// <remarks>Uses the passed in instance of a parent</remarks>

        public static void TestForCompositeOperation<ParentType, ChildType>(ParentType parent, 

            Action<ParentType> parentAction, Action<ChildType> childAction, Action<ParentType, ChildType> addChild)

            where ChildType : class

        {

            // Use Rhino.Mocks to create stubs

            var child1 = MockRepository.GenerateStub<ChildType>();

            var child2 = MockRepository.GenerateStub<ChildType>();

            // Setup two children, the arbitrary choice, but should not really matter

            child1.Expect(childAction);

            child1.Expect(childAction);

            // Add children to the parent

            addChild(parent, child1);

            addChild(parent, child1);

            // Call parent action

            parentAction(parent);

            // Assert parent action resulted in the calls to children

            child1.AssertWasCalled(childAction);

            child1.AssertWasCalled(childAction);

        }

    }

}

Design for testability - testing asynchronous method with callback. Strategy pattern to the rescue!

Implementations of asynchronous pattern can differ, in this entry I decided to share my observations on design features required to make it more testable. Lets take a sample (please note that synchronization on the param is omitted to simplify the scope and real implementation would have to volatile/synchronize):

public class AsyncBench

    {

        private int param;

        internal int Param { get { return param; } }

        public AsyncBench(int param)

        {

            this.param = param;

        }

        public void BeginMethod()

        {

            Func<int, int> method = Method;

            IAsyncResult ar = method.BeginInvoke(Param, AsyncMethodCallback, new State {Field = 10});

        }

        private int Method(int param)

        {

            return param + 1;

        }

        public void AsyncMethodCallback(IAsyncResult ar)

        {

            var asyncResult = ar as AsyncResult;

            try

            {

                param = (asyncResult.AsyncDelegate as Func<int, int>).EndInvoke(ar);

            }

            catch (Exception)

            {

                throw;

            }

        }

    }

    public class State { public int Field { get; set; } }

While this implementation can be totally suitable for some contexts, it lacks some important points from the testability perspective.
We want to be able to verify that Method is called and that Callback is setting the result right.
We also want to be able to wait for the Method to be executed. The solution I found for this is to adjust the implementation towards the Strategy pattern and follow more the pattern of Begin[Method], End[Method]. The current sample is based onto callback (End[Method] would represent a slight variation).
The amended code looks like:

public class AsyncBench

    {

        private int param;

        internal int Param { get { return param;}}

        internal readonly Func<int, int> method;

        public AsyncBench(int param)

        {

            this.param = param;

            this.method = Method;

        }

        public AsyncBench(int param, Func<int, int> method) : this(param)

        {

            this.method = method;

        }

        public IAsyncResult BeginMethod()

        {

            return method.BeginInvoke(param, AsyncMethodCallback, new State {Field = param});

        }

        private int Method(int n)

        {

            return n + 1;

        }

        public void AsyncMethodCallback(IAsyncResult ar)

        {

            var asyncResult = ar as AsyncResult;

            try

            {

                param = (asyncResult.AsyncDelegate as Func<int, int>).EndInvoke(ar);

            }

            catch (Exception)

            {

                throw;

            }

        }

    }

    public class State { public int Field { get; set; } }

Main differences are that BeginMethod returns IAsyncResult so we can let our unit test to wait until the method completion. Method to call is not hardcoded anymore with default value though pointing still to the former Method.
Our test then consists of two methods. One to verify that method is called:

[TestMethod]

        public void BeginMethodTest()

        {

            bool methodCalled = false;

            // setup the method

            Func<int, int> method = (n) =>

                                        {

                                            methodCalled = true;

                                            return -1;

                                        };

            // setup the test instance

            var asyncSample = new AsyncBench(

                20, method);

            IAsyncResult ar = asyncSample.BeginMethod();

            // wait until method call completes

            ar.AsyncWaitHandle.WaitOne();

            // verify it is completed

            Assert.IsTrue(ar.IsCompleted, "Operation should have completed before reaching this point!");

            // verify our delegate was called

            Assert.IsTrue(methodCalled, "Test method should have been called, but it was not!");

            // The bellow assert would require more synchronization and exceeds the testing contract

            //Assert.AreEqual(-1, asyncSample.Param);

        }

And another one to see that callback is doing its job:

[TestMethod]

        public void AsyncMethodCallbackTest()

        {

            bool methodCalled = false;

            // setup method

            Func<int, int> method = (n) =>

            {

                methodCalled = true;

                return -1;

            };

            // setup test instance

            var asyncSample = new AsyncBench(

                20, method);

            IAsyncResult ar = method.BeginInvoke(20, null, new State {Field = 10});

            // wait for the method to complete

            ar.AsyncWaitHandle.WaitOne();

            // verify it has completed

            Assert.IsTrue(ar.IsCompleted, "Operation should have completed before reaching this point!");

            // and was really called

            Assert.IsTrue(methodCalled, "Test method should have been called, but it was not!");

            // use IAsyncResult from our own BeginInvoke for the callback on the test instance

            asyncSample.AsyncMethodCallback(ar);

            // check that EndInvoke worked as expected

            Assert.AreEqual(-1, asyncSample.Param);

        }

Examples provided are very simplistic and just point into common ways I've been pushed to when testing my async methods.
Also in some places I used public or internal accessibility where private could/should have been used, but I just wanted to avoid usage of private accessors for those short samples.

Why to use PEX (Program EXploration for .NET). Sample 1. Reason 1.

Sample under test:

public static class XmlUtility

    {

        public static string Encode(char input)

        {

            switch (input)

            {

                case '\n': return "
";

                case '\r': return "
";

                case '&': return "&";

                case '\'': return "'";

                case '"': return """;

                case '<': return "&lt;";

                default: return new string(input, 1);

            }

        }

    }

Non-PEX test sample:

        [TestMethod()]

        public void EncodeTest()

        {

            Assert.AreEqual<string>("&#xA;", XmlUtility.Encode('\n'));

            Assert.AreEqual<string>("&#xD;", XmlUtility.Encode('\r'));

            Assert.AreEqual<string>("&amp;", XmlUtility.Encode('&'));

            Assert.AreEqual<string>("&apos;", XmlUtility.Encode('\''));

            Assert.AreEqual<string>("&quot;", XmlUtility.Encode('"'));

            Assert.AreEqual<string>("&lt;", XmlUtility.Encode('<'));

            Assert.AreEqual<string>("a", XmlUtility.Encode('a'));

        }

PEX test sample:

[PexMethod()]

        public void EncodeTest(char input)

        {

            if (input == '\n') { Assert.AreEqual<string>("&#xA;", XmlUtility.Encode(input)); return; }

            if (input == '\r') { Assert.AreEqual<string>("&#xD;", XmlUtility.Encode(input)); return; }

            if (input == '&') { Assert.AreEqual<string>("&amp;", XmlUtility.Encode(input)); return; }

            if (input == '\'') { Assert.AreEqual<string>("&apos;", XmlUtility.Encode(input)); return; }

            if (input == '"') { Assert.AreEqual<string>("&quot;", XmlUtility.Encode(input)); return; }

            if (input == '<') { Assert.AreEqual<string>("&lt;", XmlUtility.Encode(input)); return; }

            // Everything else should not be encoded

            Assert.AreEqual<string>(new string(input, 1), XmlUtility.Encode(input));

        }

Note, I had to use ifs instead of switch/case as I had a bit of an issue to make PEX working as expected  in case of switch statement in the test method.

So what the difference does it make?

In both tests we effectively provided parameters to make test coverage 100% (sorry for a bit of approximation). Both tests pass.

Though, there is a problem with algorithm and it doesn't encode the '>' input. TDD approach wise we would add a test that makes our encoding method fail, and fix our encoding to encode '>'. TDD purists are known for eating people alive for not following that "test first" strategy, but even following this TDD approach you may not be able to protect against simply overlooking something.

So lets be non TDD and add a line of code to fix our issue:

case '>': return ">";

 

Now lets rerun the tests:

There is no reason why current non-PEX test would fail, so it really passes.

Lets see how PEX test is doing:

 

And now we can see a bit of the PEX magic in action!

Behind the scenes, looking at our code under test, PEX generated an extra test method for input of '>' and this failed our

Assert.AreEqual<string>(new string(input, 1), XmlUtility.Encode(input));

"assert for the rest of cases" statement.

So now, I'm actually explicitly forced to add the test case for '>', which I'm doing:

if (input == '>') { Assert.AreEqual<string>("&gt;", XmlUtility.Encode(input)); return; }

To make the PEX tests pass:

Conclusions:
1. I should be better on writing unit tests, but I can never be perfect.
2. PEX can really give me a hand for catching some of my possible mistakes in unit tests.

Ultimate link for PEX (Program Exploration for .NET) resources: http://research.microsoft.com/Pex/
First user experience: http://blog.benhall.me.uk/2008/05/microsoft-pex-05-released.html
TDD purists may be watching you! http://intellij.net/forums/thread.jspa?messageID=5215367 (scroll down a bit until the Gabriel Lozano's  post, I had fun reading ...)