This is Part 2 of my series on Automated Testing. Click the link below to read part 1:
An Introduction to Automated Testing
In this episode, we’ll make a cursory review of the automated testing landscape, specifically for the .NET ecosystem. I’ll list out some common frameworks and tools so that you are aware of them and use this list as a jump-off point to more in depth research you may want to do on any of these. Moreover, I’ll draw attention to the types of tools available in the space so that if you are not in the .NET ecosystem, you’ll still be able to gain something by understand some of the classifications in this area so that you can then look for similar ones for your own specific stack.
Testing Frameworks
These are some free tools/frameworks that you can use to write your automated tests. At its core, testing libraries allow you to instantiate and run the piece of code that you want to test and provide a set of assertions to check whether the actual result of your code execution matches the expected result.
While you can certainly create automated tests without the aid of any such external tools, I highly recommend going with an established tool as it likely has addressed a lot of complexities that you may have not even thought of such as surfacing them to other tools such as a test explorer where you can get a dashboard view of all your tests, determine whether they are passing or failing, calculate code-coverage, easily turn tests on and off, execute them at will, individually, by groupings that you make, bootstrap tests in CI/CD pipelines and so much more. Here are a few that you can check out:
MS Test Framework
MS Test Framework has been around for many years and although it lagged behind in features initially, ever since “MS Test V2” released circa 2016, it has been a formidable player.
More info: Unit testing C# with MSTest and .NET – .NET | Microsoft Docs
xUnit
This library, xUnit, is a personal favorite of mine, simply because I’ve had the most familiarity with it. In future posts where I’ll dig a bit deeper with some sample tests, I’ll use this library in my examples. This library is also a fan-favorite among .NET developers with over 168 million downloads on NuGet, at the time of this writing.
NUnit
NUnit has been around for a while and boasts a wide user following. This project is originally a port of JUnit, a popular testing framework for the Java ecosystem.
Assertion Helpers
A key tenant of writing good tests is the human readability of those tests. As I mentioned in Part 1, a developer should be able approach a codebase, read the tests associated with it and get a good understanding of the functionality of the system with little effort. I should also note here that while being succinct and eliminating duplicate code is a tenant of good programming in general, the emphasis when it comes to writing test code is to achieve better readability, even if it means duplicating some code. When a developer is looking at a test, we want to minimize the friction as much as possible by not tucking away a lot of the setup in various helper classes but trying to include most, if not all of the setup directly in the test method. We want to reduce the cognitive load, not requiring the developer to do a whole lot of hunting around in different files and methods to understand what a test is doing. Along the same lines, we want the code to be fluent, being able to be read as fluently as ones natural human language.
Enter the domain of assertion helpers. These helper libraries allow you to construct your tests in such a way that the intent of the test becomes easily readable to the human reader. It uses common English words like “should” and “have” in the API to make the intention of the test jump out at the reader.
string actual = "ABCDEFGHI";
actual.Should().StartWith("AB").And.EndWith("HI").And.Contain("EF").And.HaveLength(9);
Fluent Assertions
This is the assertion helper that I like to use, personally. It has a wide userbase, with strong support and provides a feature rich API.
Website: Fluent Assertions – Fluent Assertions
Shouldly
Website: Shouldly – The Simple .NET Asssertion library, the way assertions should be…
Mocking / Stubbing / Spying
When you enter the realm of testing, especially Unit Testing, you’ll hear terms such as mocking, stubbing, spying and other related keywords. If inclined, you can read a summary of these in this Stack Overflow post, here. I won’t get into the technicalities but address this at a higher-level, in a generic sense.
When testing, especially in the context of writing unit-tests, you want to write the test for a small unit of code, perhaps a single method in a class. However, methods hardly ever reside in a vacuum, often having hooks into other classes and methods, reaching out to a third-party API or is situated somewhere in an intricate labyrinth of business logic. There may be other concerns that have to be addressed, such as authentication or authorization in order to invoke a particular piece of code. These helper tools allow you to quickly create a “stand-in” for those other elements, freeing you from having to deal with the complexities of those extraneous but related pieces, allowing you to zero in on the specific piece of code that you have set out to test.
Moq
Website: Moq (moqthis.com)
A personal favorite of mine. Like the others in this list, provide a quick, unobtrusive way of setting up dependencies in your code.
NSubstitute
Website: NSubstitute: A friendly substitute for .NET mocking libraries
Takes a very pragmatic approach to setting up dependencies without getting too much into the weeds with the subtleties of this area of testing.
FakeItEasy
Website: FakeItEasy – It’s faking amazing
Another library that tries to minimize the complexities that surround this.
Test Runners
As the name implies, these classification of tools allows for the discover of tests in your codebase, listing them, allows for the execution of them, either individually or as groups, reports on the results of the run tests.
Visual Studio Test Runner
Back in the day, pretty much all .NET development was done using one tool – Visual Studio. And as you’d guess, Visual Studio had its own built-in test runner and it continues to have one today. There are harnesses available for every test framework that I mentioned above, allowing you to write tests in your favorite test framework and have them show up in the Visual Studio test runner.
dotnet test
Today, .NET development is not done in Visual Studio alone. In fact, I’d argue that VS Code is where all the new hotness is as it is a much more lightweight tool, supporting a variety of other languages and frameworks in addition to being just a tool for .NET development. Moreover, .NET Framework is cross-platform, with first class support not just in Windows but also on Linux and on the Mac.
You can execute .NET (Core or the new .NET) tests via the .NET CLI, using the command “dotnet test”. This command-line approach provides a way neat way to plug in a host of utilities, integrations with CI/CD pipelines and much more.
Code Coverage Tools
When you start building a suite of tests against your codebase, especially when you’re writing the tests at a later time after you have written your code (note: We’ll address TDD in a later post), you’ll want some way of figuring out the areas of your codebase that are potentially left vulnerable, without any tests covering those code pieces. Code Coverage Tools help in this regard. They scan your codebase, scan your tests and see which execution paths are covered by those tests and which are left untouched. These tools also usually represent the coverage as a percentage giving you a quick indicator of the state of affairs.
One thing that I’d like to point out right away is to avoid the temptation of reading into this too much. It’s quite possible to get a high number for this metric and still have issues in your codebase. Leave it to a developer and they’ll find very clever ways to show a high code coverage percentage for their code by writing a lot of meaningless tests. In the end, those test doesn’t help anyone but in fact, slow things down with the added burden of now having to support those meaningless lines of code. Often, organizations run into trouble when these numbers are highly publicized in dashboards and rules enforcing a certain percentage because it forces the developer to be creative with their tests instead of writing meaningful ones.
With that disclaimer, here are a few tools that allow you to do this:
GitHub – coverlet-coverage/coverlet: Cross platform code coverage for .NET
dotCover: A Code Coverage Tool for .NET by JetBrains
NCover | .NET Code Coverage for .NET Developers
End to End (E2E) Testing Frameworks for Web Applications
These are not specific for .NET but will assist in setting up automated E2E tests for web applications written in pretty much any framework. These tools interact with your application like an end user would: through a web browser. As such, the tool doesn’t care what technology was used to render the web pages it is testing. As long as that browser is serving up HTML and JavaScript, these tools will be able to interact with it.
Cypress.io
Website: Cypress.io
Compared to Selenium (see below), this is the relative newcomer in this space. However, they’ve really impressed the development community that their popularity has sky-rocketed since inception. This project came into everyone’s purview at a time when Selenium was the only other viable option and developers were experiencing some pains with Selenium, especially in terms of addressing network latency – an issue that must be addressed in end-to-end testing scenarios where the automation is reliant on responses from remote systems. Developers would often have to add arbitrary “waits” in their automation code to account for potential delays in receiving responses to their web requests. A web page may take 2 seconds to respond, 5 seconds another time or a gigantic report can take 10 seconds to come back some results. Since these times could not be predetermined, Developers would add instruct the browser to “wait” for a certain time before continuing with the next step which was not an exact science. It would at times make the tests unnecessarily slow or brittle, causing a successful completion in one run and a failure in another with no changes in the code, at all.
Cypress addresses the issue of latency in a more elegant fashion, removing some of those concerns from the developer. No more arbitrary waits. The framework will address the issue by doing its own retries until a specified timeout period eliminating a lot of ugly retry and timeout logic often associated with Selenium based tests.
Selenium
Website: Selenium
This is the grand-daddy of all browser automation. Selenium is an all-purpose browser automation solution that not only provides capabilities for automated testing but also provides a solution for any other use-case that can benefit from browser-automation such as automatically scaping a website for data or automating a boring, repetitive admin task.
The project offers a set of Web Drivers targeting all the major players in the web-browser space: Microsoft Edge, Google Chrome, Mozilla Firefox, Safari, Opera and even Microsoft Internet Explorer (only v11 supported). Then, they provide a set of clients targeting all the major languages/frameworks out there – C#, Ruby, Java, Python and JavaScript so that you can create automations in any of those languages and environments. Finally, they support all three major Operating Systems – Windows, Mac and Linux.
Parting Thoughts
In this long post, we looked at some of the types of tools that make up the Automate Testing arena. Hopefully, I’ve given you a basic understanding of the pieces involved and even examples on each so that you can do your own research and pick a toolkit for your own use cases. In the next one, we’ll put aside the theoretical aspects and actually get our hands dirty with some simple unit tests. Happy coding.