Coder Social home page Coder Social logo

bungalow64 / dbconfirm Goto Github PK

View Code? Open in Web Editor NEW
5.0 3.0 2.0 3.19 MB

A C# testing framework to write and run tests for logic within SQL Server.

Home Page: https://www.dbconfirm.com

License: Apache License 2.0

C# 80.01% TSQL 19.94% Shell 0.05%
testing-framework database c-sharp tdd mstest nunit

dbconfirm's Introduction

Introduction

A C#-based testing framework to write and run tests for logic within SQL Server

Build Status Repo Size Licence Release Date

What is DBConfirm?

DBConfirm is a unit testing framework for SQL Server databases from within .Net projects. Tests can be written to check that stored procedures and views behave as you'd expect, and can be used to help reduce the number of bugs introduced. DBConfirm also provides patterns and tools to easily set up prerequisite data needed for your tests.

Why?

Developers are pretty good at writing unit tests for their application logic already, but sometimes database logic (stored procedures, views, etc.) can be overlooked. A big reason is that traditionally SQL unit tests are difficult to write, or have a very steep learning curve. DBConfirm aims to solve this by allowing SQL tests to be written in the same way that all other unit tests are written, so that they are easy to write, easy to maintain, and easy to run.

How?

The DBConfirm framework is designed to execute tests against a physical instance of the database under test, and ensures that each test run is accurate and repeatable by making sure all effects of a test are rolled back when the test has finished.

Where's the full documentation?

For the full documentation, see dbconfirm.com.

What versions of SQL Server does DBConfirm work with?

DBConfirm is compatible with SQL Server 2014, 2016, 2017, 2019, 2022 and with Azure SQL Database.

What does a DBConfirm test look like?

A simple test (in MSTest) to call a stored procedure then verify that the data has been added, looks like this:

[TestMethod]
public async Task AddUserProcedure_UserIsAdded()
{
    // Call a stored procedure with some parameters
    await TestRunner.ExecuteStoredProcedureNonQueryAsync("dbo.AddUser", new DataSetRow
    {
        ["FirstName"] = "Sarah",
        ["LastName"] = "Connor",
        ["EmailAddress"] = "[email protected]"
    });

    // Get all the data in a table
    QueryResult data = await TestRunner.ExecuteTableAsync("dbo.Users");

    // Make some assertions on the data
    data
        .AssertRowCount(1) // Asserts that there is only 1 row
        .AssertValue(0, "FirstName", "Sarah"); // Asserts that "FirstName" is "Sarah" in the first row
}

Getting started

The quickest way to get started is to use a DBConfirm project template.

Install the template for MSTest (DBConfirm.Templates.SQLServer.MSTest) by executing this command:

dotnet new -i DBConfirm.Templates.SQLServer.MSTest

Alternatively, you can use the DBConfirm.Templates.SQLServer.NUnit template to use the NUnit test framework

Once installed, create a new DBConfirm project by executing this command:

dotnet new dbconfirm-sqlserver-mstest -n "YourProjectName"

Alternatively, for NUnit, execute dbconfirm-sqlserver-nunit

The project will then be added, with a sample unit test class in the root.

You'll need to update the config file to have a correct connection string in there, pointing to the database you want to test. In the root of your test project, find the appsettings.json file and update the DefaultConnectionString value:

{
  "ConnectionStrings": {
    "DefaultConnectionString": "SERVER=(local);DATABASE=TestDB;Integrated Security=true;Connection Timeout=30;"
  }
}

You're now ready to start testing. Add a new test method, and start testing:

// For MSTest
[TestMethod]
public async Task GetUsersView_ContainsFirstNameColumn()
{
    var results = await TestRunner.ExecuteTableAsync("dbo.GetUsers");

    results
        .AssertColumnExists("FirstName");
}

For NUnit, use [Test] instead of [TestMethod].

The fundamentals of a DBConfirm test

Arrange - set up any prerequisite test data

Data can be inserted into any table in the database using either the DBConfirm API or by creating templates that are used to set up complex scenarios across multiple tests.

Act - run the SQL you want to test

Using the DBConfirm API you can trigger any stored procedure, query any view, or run any arbitrary SQL statement.

Assert - check the returned data, and check the state of data in the database

The DBConfirm API provides a whole bunch of assertion methods that you can run on the returned data, including checking specific columns are present, that values are what you expect them to be, and that at least one row meets your conditions, etc.

You can also use the DBConfirm API to query the data in tables, and run the same assertions, to make sure the data is in the exact state that you expect.

For the full documentation, see DBConfirm.com.

Buliding and running the source of DBConfirm

Feel free to fork/clone this repository. Once you have the source files locally, open DBConfirm.sln, and the solution will load. Each NuGet package is its own project; hopefully you will be able to find your way around.

There are a number of test projects, and these fall under 2 categories:

  • unit tests
  • sample project tests

The unit test projects are all within the 5.Tests solution folder. These are just standard unit tests, nothing special there.

The sample project tests, within the 6.SampleSolutions solution foder, are more like integration tests, that carry out DBConfirm tests against an actual database.

Running sample project tests

These tests require an actual database to test against, so for these tests to run, you'll need to set these databases up.

The easiest/quickest way is to use Docker. Once you have Docker installed and running locally, open up your command prompt, go to the root of the DBConfirm solution, and run these commands:

docker compose build
docker compose up

This will set up the databases for you, and host them in SQL 2017, 2019 and 2022. The tests in the sample solutions are already set up to connect to these databases, so you should be able to just run all those tests, and they'll all (hopefully) pass.

If you want to access these databases directly, they are located here:

  • Server: localhost
  • Port: 1401 (for SQL 2017), 1402 (for SQL 2019), 1403 (for SQL 2022)
  • Databases: SampleDB, Northwind
  • Credentials: For the 'sa' password, check the contents of the docker\sqlserver.env file.

The continuous integration builds we have set up also use Docker to set up the test environment, so we can be assured that everything is working as expected.

Alternatively, if you don't want to use Docker, you can host the databases directly. The setup scripts for the databases are within 6.SampleSolutions\1.Databases. Run these on an instance of SQL somewhere, and update the connection strings in the test projects (such as 6.SampleSolutions\2.ProjectTests\Sample.Core.MSTest.Tests\appsettings.json) to get them up and running.

dbconfirm's People

Contributors

bungalow64 avatar dependabot[bot] avatar fmms avatar jamie-burns avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

Forkers

jashelb90 fmms

dbconfirm's Issues

Running 'docker compose up' on a fresh repository clone into Windows results in an error in the import-data.sh script

  • Clone DBConfirm (on a Windows machine)
  • Run 'docker compose build'
  • Run 'docker compose up'

This error will appear in the logs somewhere, and the sample DBs won't be created:

syntax error near unexpected token `$'\r''

The problem is that the import-data.sh file needs to have Unix (LF) line endings, and git is cloning this file with Windows (CLRF). This can be fixed by manually changing the file to LF (e.g., opening it in Visual Studio Code and changing it at the bottom-right), and that fixes it, but ideally git needs to leave the line endings alone.

Update MSTest packages to work with MSTest 3.X.X

After upgrading to MSTest 3.X.X we get this error:

Method not found: 'Void Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual(System.Object, System.Object, System.String, System.Object[])'.

Workaround is to use MSTest.TestAdapter/MSTest.TestFramework 2.2.10

Upgrade to .NET 6

Currently we're using .NET Core 3.1, which comes to end of life December 2022.

We should upgrade to .NET 6.

Upgrade test projects to .NET 8

The test projects are currently using .NET6, and it would be nice to be able to use some of the language features in .NET 7/8.

Upgrade all non-package projects (keep those as netstandard 2.0) to .NET 8.

Apply project-wide refactoring of the following:

Assertions for numbers are type-specific, which can be too restrictive and slow development down

Using v0.1.3, SQL Server and MSTest.

Say there is a stored procedure that returns a column that is set as smallint (e.g., Quantity in Northwind's dbo.CustOrderDetail). I write a test that verifies that this number is 2.

If I write the assertion like this:

result
  .AssertRowCount(1)
  .ValidateRow(0)
  .AssertValues(new DataSetRow
  {
      { "Quantity", 2 }
  });

Then the test fails, with this error:

Assert.AreEqual failed. Expected:<2 (System.Int32)>. Actual:<2 (System.Int16)>. Column Quantity in row 0 has an unexpected value

I would have to cast the expected value to the correct type for the test to pass:

result
  .AssertRowCount(1)
  .ValidateRow(0)
  .AssertValues(new DataSetRow
  {
      { "Quantity", (short)2 }
  });

This applies to all number types - for example decimals, which requires either casting or postfixing with M).

It would be helpful for some implicit casting, where possible. If I write the number 2, then that should be able to be used in an assertion with any number type that can represent this number fully, without failing on a type check. If a number is used that cannot be cast fully (e.g., a 1.5 used to assert a shortint column), then the original type needs to be used, since it wouldn't make sense to use either 1 or 2 for this assertion.

The only downside for relaxing the rules on type checking is that it's no longer possible (in 0.1.3) to write a test that locks down the type of data returned for a given column. This is currently done by default when the value is asserted. So it might be a good idea to add extra validation methods to explicitly check for the column type (e.g., AssertColumnType(string expectedColumnName, Type expectedType)), so that tests can still check this if required.

TL;DR;

  1. When asserting numeric values, attempt to avoid unnecessary type errors (possibly by exact casting)
  2. Add extra validation methods to test for column types (already possible)

Cannot generates templates into a folder that doesn't exist

When running the GenerateTemplatesSQLServer tool, if a destination is selected that doesn't exist (e.g., create a new project, then try to generate templates into a Templates directory that hasn't been created yet) then we get an error saying that the path can't be found:

PS C:\Git\TestProject\DBTests> GenerateTemplatesSQLServer -d "TestDB" -t "*" -n "DBTests.Templates" --destination Templates
Using connection: SERVER=(local);DATABASE=TestDB;Integrated Security=true;Connection Timeout=30;
Found table (dbo.User) and 9 columns
Class definition generated.  Saving...
There has been an error
Could not find a part of the path 'C:\Git\TestProject\DBTests\Templates\UserTemplate.cs'.
   at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
   at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.StreamWriter.ValidateArgsAndOpenPath(String path, Boolean append, Encoding encoding, Int32 bufferSize)
   at System.IO.StreamWriter..ctor(String path)
   at System.IO.File.WriteAllText(String path, String contents)
   at DBConfirm.TemplateGeneration.Logic.FileHelper.WriteAllText(String path, String contents) in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/FileHelper.cs:line 10
   at DBConfirm.TemplateGeneration.Logic.Generator.GenerateFile(IEnumerable`1 processedColumns, String schemaName, String tableName) in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/Generator.cs:line 112
   at DBConfirm.TemplateGeneration.Logic.Generator.GenerateFileAsync() in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/Generator.cs:line 61
   at DBConfirm.TemplateGeneration.Program.<>c.<<Main>b__0_0>d.MoveNext() in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Program.cs:line 18

The tool should try and create the directory if it doesn't already exist.

Docker-compose not creating the databases on local development machine

When running docker-compose on Windows 11, the servers are set up correctly, but the databases aren't added.

I haven't figured out what's changed, or where there's an error, but this can be fixed by running these commands once the containers have been set up:

docker exec -it dbconfirm-sqlserver2019-1 /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -d master -i northwind_setup.sql
docker exec -it dbconfirm-sqlserver2019-1 /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -d master -i sampledb_setup.sql
docker exec -it dbconfirm-sqlserver2017-1 /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -d master -i northwind_setup.sql
docker exec -it dbconfirm-sqlserver2017-1 /opt/mssql-tools/bin/sqlcmd -S localhost -U sa -d master -i sampledb_setup.sql

The password can be found in the docker\sqlserver.env file.

Add negative tests, asserting that verifications should fail when expected

Most of the tests focusing on the verification methods (which are calling the DB) are only checking that the verifications pass as expected.

There can be more to check they fail as expected, too.

For MSTest we can just wrap the verification in a try...catch and assert the caught exception. For NUnit we need to run the test in an isolated process, and catch the exception.

Add helper methods to make it easy to test for exceptions/errors coming back from executing stored procedures

If I have a stored procedure that should throw an exception, I'll want to be able to test for that.

I can wrap the TestRunner.ExecuteStoredProcedureNonQueryAsync call in a function, and test the result for an exception using something like FluentAssertions:

Func<Task> act = async () => await TestRunner.ExecuteStoredProcedureNonQueryAsync("dbo.DoSomethingProcedure",
    new SqlQueryParameter("Value1", "abc"),
    new SqlQueryParameter("Value2", "def"));

act.Should().Throw<SqlException>();

But it would be nice if this was handled within the DBConfirm framework itself. We'd probably need to have a new, dedicated method to check for this (since some of these existing Execute procedures don't return anything, and we wouldn't want exceptions to be hidden), but probably just one would work for all execution types, since we wouldn't need to test any data that's returned (since it should have errored).

add a function to assert Uniqueness

Hi,

thanks for this very nice project.

I have just played a bit and was expecting a function to confirm uniqueness. i.e.

  [TestMethod]
       public async Task unique()
        {
            QueryResult data1 = await TestRunner.ExecuteCommandAsync("SELECT 1 a, 2 b UNION SELECT 2, 2 ORDER BY a");
            data1
              .AssertColumnCount(2)
              .AssertColumnExists("a")
              .AssertColumnsNotExist("c", "d")
              .AssertValue(0, "a", 1) 
// missing?
              .AssertColumnUniqe("a")
              .AssertColumnsUniqe("a", "b")
            ;
        }

Did I miss something? Could those be added?

regards
Felix

Generating templates for all tables for a schema that doesn't have any tables returns a misleading error message

Try generating the templates for all tables for a schema that doesn't have any tables (or for a schema that doesn't exist itself), and the error returned is:

Cannot find table: dbo2.*

For example, say I have a database with some tables in dbo, but where the dbo2 schema doesn't exist. Running:

GenerateTemplatesSQLServer -d "DatabaseName" -t "*" -s "dbo2" 

This shows the error.

Really, this should say:

Cannot find any tables in the dbo2 schema

Confirm compatibility with SQL Server 2022

Check everything works with SQL Server 2022.

  • update readme
  • update docs
  • add DB setup script for SQL Server 2022
  • add test projects for SQL Server 2022
  • update pipelines to run new tests

Add support for xUnit

We currently support MSTest and NUnit. It would be good to also support xUnit to align with other test projects that also use xUnit.

Generating a template for a table name that contains an apostrophe results in an invalid class

Say I have a table in the database (SQL Server) called:

dbo.User's

When the template is generated for this using the GenerateTemplatesSQLServer tool, the apostrophe is maintained in the name, resulting in a class that doesn't compile.

Ideally this special character (and others) need to be replaced, but we need to make sure that we don't end up with duplicate names (e.g., if there is another table called dbo.User_s, then we can't just replace the apostrophe with an underscore).

Cross platform development

Hi - is this project designed for cross platform development? We are looking for a solution to move our db unit tests to Linux.

Add support for MySQL

We would need new packages for:

  • Databases.MySQL
  • Packages.MySQL.MSTest
  • Packages.MySQL.NUnit
  • Templates.MySQL.MSTest
  • Templates.MySQL.NUnit

Plus the scripts for the test database and the new test projects.

Update NUnit packages to work with NUnit 4.X.X

Currently we get this error:

System.MissingMethodException : Method not found: 'Void NUnit.Framework.Assert.AreEqual(System.Object, System.Object, System.String, System.Object[])'.

The workaround is to use NUnit v3.14.0.

Using AssertRowDoesNotExist when using NUnit results in an incorrect test failure

Say we execute a stored procedure that returns a QueryResult object, and we want to check that a specific row does not exist. We can call this:

data
.AssertRowDoesNotExist(new DataSetRow
{
{ "FirstName", "Jeff" },
{ "LastName", "Burns" }
});

When using MSTest this works as expected, but in NUnit we get a test failure:

Multiple failures or warnings in test:

  1. Column FirstName in row 0 has an unexpected value
    Expected string length 4 but was 5. Strings differ at index 1.
    Expected: "Jeff"
    But was: "Jamie"
    ------------^

  2. Column FirstName in row 1 has an unexpected value
    Expected string length 4 but was 3. Strings differ at index 0.
    Expected: "Jeff"
    But was: "AAA"
    -----------^

For example, the AddUser_ValidData_UserAdded test in MSTest works, but fails when run in NUnit: https://github.com/Bungalow64/DBConfirm/blob/main/tests/Sample.Core.MSTest.Tests/StoredProcedures/AddUserTests.cs

This assertion should only fail if a row is found that matches.

Root cause

The problem is that we're iterating through each row and using the same logic to see if the row matches the data, and catching/swallowing the assert exception for each row that doesn't match. This works fine for MSTest, but NUnit still fails the test even though the exception is caught.

Really, we don't want to be using exceptions for this, so we need to rewrite this logic to do the same checks without using assert logic.

The DBConfirm.TemplateGeneration tool fails to generate template for a table containing a nvarchar(max) column

I have a table containing an Identity and nvarchar(max) column:

CREATE TABLE [dbo].[Titles](
	[Id] [int] IDENTITY(1,1) NOT NULL,
	[Title] [nvarchar](max) NOT NULL,
 CONSTRAINT [PK_Titles] PRIMARY KEY CLUSTERED 
(
	[Id] ASC
)WITH (PAD_INDEX = OFF, STATISTICS_NORECOMPUTE = OFF, IGNORE_DUP_KEY = OFF, ALLOW_ROW_LOCKS = ON, ALLOW_PAGE_LOCKS = ON) ON [PRIMARY]
) ON [PRIMARY] TEXTIMAGE_ON [PRIMARY]
GO

Running DBConfirm.TemplateGeneration on this table results in an error:

PS C:\Git\Database1Tests> GenerateTemplatesSQLServer -d "Database1" -t "Titles" --dry-run                         
Using connection: SERVER=(local);DATABASE=Database1;Integrated Security=true;Connection Timeout=30;
Found table (dbo.Titles) and 2 columns
There has been an error
Length cannot be less than zero. (Parameter 'length')
   at System.String.Substring(Int32 startIndex, Int32 length)
   at DBConfirm.TemplateGeneration.Models.ColumnDefinition.TruncateLongString(String value) in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Models/ColumnDefinition.cs:line 138
   at DBConfirm.TemplateGeneration.Models.ColumnDefinition.GetDefaultText() in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Models/ColumnDefinition.cs:line 119
   at DBConfirm.TemplateGeneration.Models.ColumnDefinition.ToDefaultData() in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Models/ColumnDefinition.cs:line 163
   at DBConfirm.TemplateGeneration.Logic.Generator.<>c.<GenerateClass>b__11_5(ColumnDefinition p) in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/Generator.cs:line 183
   at System.Linq.Enumerable.WhereSelectEnumerableIterator`2.MoveNext()
   at System.String.Join(String separator, IEnumerable`1 values)
   at DBConfirm.TemplateGeneration.Logic.Generator.GenerateClass(IEnumerable`1 processedColumns, String schemaName, String tableName) in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/Generator.cs:line 173
   at DBConfirm.TemplateGeneration.Logic.Generator.GenerateFile(IEnumerable`1 processedColumns, String schemaName, String tableName) in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/Generator.cs:line 93
   at DBConfirm.TemplateGeneration.Logic.Generator.GenerateFileAsync() in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Logic/Generator.cs:line 59
   at DBConfirm.TemplateGeneration.Program.<>c.<<Main>b__0_0>d.MoveNext() in /home/vsts/work/1/s/tools/TemplateGeneration.SQLServer/Program.cs:line 18

The tool should be able to generate a template with both columns included.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.