relativitydev / relativity-test-helpers

Open Source Community: Integration testing is a software testing methodology used to test individual software components or units of code to verify their interaction. These components are tested as a single group or organized in an iterative manner.That said, we have created Relativity Integration Test Helpers to assist you with writing good Integration Tests for your Relativity application. You can use this framework to test event handlers, agents or any workflow that combines agents and Eventhandlers. We will continue adding more helpers but in the mean time you should be able to create workspaces, create dbcontext, proxy and create documents with this framework. This framework is only compatible with Relativity 9.5

relativity-integration-test-helpers

Overview

Open Source Community: Integration testing is a software testing methodology used to test individual software components or units of code to verify their interaction. These components are tested as a single group or organized in an iterative manner. Relativity Integration Test Helpers was created to assist with writing best practice Integration Tests for Relativity applications. The framework can be used to test event handlers, agents or any workflow that combines agents and event handlers.

The framework will currently enable you to proxy and create documents, create workspaces and create dbcontext with more functionality to be added in the future.

Compatibility

TestHelpers v8 release is only compatible with Ninebark (12.0) and above. Any previous version of TestHelpers will only be compatible with Relativity until Prairie Smoke (12.2), which will remove RSAPI.

This is also available as a nuget package.

Version 9.1 Updates

With the upgrade of Relativity.DataExchange.Client.SDK/1.15.30+ and Relativity.Transfer.Client/7.4.7+ these updates may bring about breaking changes. This release was tested against Tiger Lily EAU.

Version 8.1 Updates

With the deprecation of RSAPI, TestHelpers was significantly updated to account for this. These updates, however, did bring about many breaking changes.

Note: Previous packages of TestHelpers will still be available, but previous major releases will not be updated with RSAPI removal

Breaking Changes

Context	v7	v8
All	Any IRSAPIClient usage	Removed Use IServicesMgr in IRSAPIClient's place
All	SharedTestHelpers.ConfigurationHelper.FORCE_DBCONTEXT	Removed
All	SharedTestHelpers.ConfigurationHelper.RSAPI_SERVER_ADDRESS	Removed Use RELATIVITY_INSTANCE_ADDRESS or REST_SERVER_ADDRESS
Application	Application.ApplicationHelpers.ImportApplication	Moved to Kepler.ApplicationInstallHelper.InstallApplicationAsync
Client	ArtifactHelpers.Client.Create_Client	Moved to ArtifactHelpers.ClientHelper.CreateClient
Client	ArtifactHelpers.Client.Delete_Client	Moved to ArtifactHelpers.ClientHelper.DeleteClient
Document	ArtifactHelpers.Document.GetDocumentIdentifierFieldColumnName	Removed
Document	ArtifactHelpers.Document.GetDocumentIdentifierFieldColumnNameWithDbContext	Removed
Document	ArtifactHelpers.Document.GetDocumentIdentifierFieldName	Moved to ArtifactHelpers.DocumentHelper.GetDocumentIdentifierFieldName
Document	ArtifactHelpers.Document.GetDocumentIdentifierFieldNameWithDbContext	Removed
Field	ArtifactHelpers.Fields.CreateField	Removed
Field	ArtifactHelpers.Fields.CreateField_Date	Moved to ArtifactHelpers.FieldHelper.CreateFieldDate
Field	ArtifactHelpers.Fields.CreateField_FixedLengthText	Moved to ArtifactHelpers.FieldHelper.CreateFieldFixedLengthText
Field	ArtifactHelpers.Fields.CreateField_LongText	Moved to ArtifactHelpers.FieldHelper.CreateFieldLongText
Field	ArtifactHelpers.Fields.CreateField_MultipleChoice	Moved to ArtifactHelpers.FieldHelper.CreateFieldMultipleChoice
Field	ArtifactHelpers.Fields.CreateField_SingleChoice	Moved to ArtifactHelpers.FieldHelper.CreateFieldSingleChoice
Field	ArtifactHelpers.Fields.CreateField_User	Moved to ArtifactHelpers.FieldHelper.CreateFieldUser
Field	ArtifactHelpers.Fields.CreateField_WholeNumber	Moved to ArtifactHelpers.FieldHelper.CreateFieldWholeNumber
Field	ArtifactHelpers.Fields.CreateField_YesNO	Moved to ArtifactHelpers.FieldHelper.CreateFieldYesNo
Field	ArtifactHelpers.Fields.GetFieldArtifactID	Moved to ArtifactHelpers.FieldHelper.GetFieldArtifactID
Field	ArtifactHelpers.Fields.GetFieldArtifactIDWithDbContext	Removed
Field	ArtifactHelpers.Fields.GetFieldCount	Moved to ArtifactHelpers.FieldHelper.GetFieldCount
Field	ArtifactHelpers.Fields.GetFieldCountWithDbContext	Removed
Folder	ArtifactHelpers.Folder.CreateFolder	Moved to ArtifactHelpers.FoldersHelper.CreateFolder
Folder	ArtifactHelpers.Folder.GetFolderName	Moved to ArtifactHelpers.FoldersHelper.GetFolderName
Folder	ArtifactHelpers.Folder.GetRootFolderArtifactID	Moved to ArtifactHelpers.FoldersHelper.GetRootFolderArtifactID
Group	GroupHelpers.CreateGroup.Create_Group	Moved to GroupHelpers.GroupHelper.CreateGroup
Group	GroupHelpers.DeleteGroup.Delete_Group	Moved to GroupHelpers.GroupHelper.DeleteGroup
KeplerHelper	Any Usage	Removed
TestHelper	GetDbContext	No longer implemented
User	UserHelpers.CreateUser.CreateNewUser	Moved to UserHelpers.UserHelper.Create
User	UserHelpers.CreateUser.FindChoiceArtifactId	Moved to ArtifactHelpers.ChoiceHelper.GetChoiceId
User	UserHelpers.CreateUser.FindClientArtifactId	Moved to ArtifactHelpers.ClientHelper.GetClientId
User	UserHelpers.CreateUser.FindGroupArtifactId	Moved to GroupHelpers.GroupHelper.GetGroupId
User	UserHelpers.DeleteUser.Delete_User	Moved to UserHelpers.UserHelper.Delete
Workspace	WorkspaceHelpers.CreateWorkspace.Create	Moved to WorkspaceHelpers.WorkspaceHelpers.Create
Workspace	WorkspaceHelpers.DeleteWorkspace.Delete	Moved to WorkspaceHelpers.WorkspaceHelpers.Delete
Workspace	WorkspaceHelpers.DeleteWorkspace.Delete	Moved to WorkspaceHelpers.WorkspaceHelpers.Delete

New Runsettings option within TestHelpers

As of update 7.1.0.X of Test Helpers, you can optionally utilize Runsettings instead of app.config for your testing. Both can still be used, but just not within the same Test Fixture, as the code does a hard swap of the value collection after the TestHelpers constructor.

Runsettings vs App.config

To quickly describe Runsettings, it's just an XML file with data to be used only when a test is being run. Azure and other CI/CD pipelines use it to supply parameters during the test. Normally, this would be done via an app.config file, but cloud pipelines do not allow the use of these config files, so the switch to Runsettings is necessary.

Configuring Runsettings in Visual Studio

While in a testing project, you can simply utilize the Test dropdown menu at the top of Visual studio. You can checkout Microsoft's documentation on how to setup a Runsettings file for more details.

Setting up Test Helpers with Runsettings

The gist of the new setup is the new Runsettings file you will need in the project, and the new overloaded constructor for Test Helpers. There are two overloads, one that takes in your Current TestContext, and one that takes in a Dictionary of values. Both will be used to automatically map values to the static ConfigurationHelper TestHelpers creates. You can view samples below for Runsettings files and Constructors.

Note: For ease of use, TestHelpers expects the same parameter keys in Runsettings as it does with app.config.

Sample Runsettings file

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
	<!-- Parameters used by tests at run time -->
	<TestRunParameters>
		<Parameter name="WorkspaceID" value="-1" />
		<Parameter name="TestWorkspaceName" value="Test Workspace Name" />
		<Parameter name="TestWorkspaceTemplateName" value="New Case Template" />
		<Parameter name="AdminUsername" value="[email protected]" />
		<Parameter name="AdminPassword" value="somepassword" />
		<Parameter name="SQLUserName" value="eddsdbo" />
		<Parameter name="SQLPassword" value="somepassword" />
		<Parameter name="SQLServerAddress" value="172.99.99.99" />
		<Parameter name="TestingServiceRapPath" value="C:\Users\user.name\Downloads" />
		<Parameter name="UploadTestingServiceRap" value="MyApp.rap" />
		<Parameter name="EmailTo" value="[email protected]" />
		<Parameter name="RESTServerAddress" value="172.99.99.99" />
		<Parameter name="ServerBindingType" value="http" />
		<Parameter name="RelativityInstanceAddress" value="172.99.99.99" />
	</TestRunParameters>
</RunSettings>

Sample TestHelper Constructor

[TestFixture]
public class TestHelperRunSettingsIntegrationTests
{
	private IHelper SuT;
	[OneTimeSetUp]
	public void SetUp()
	{
		Dictionary<string, string> configDictionary = new Dictionary<string, string>();
		foreach (string testParameterName in TestContext.Parameters.Names)
		{
			configDictionary.Add(testParameterName, TestContext.Parameters[testParameterName]);
		}
		SuT = new TestHelper(configDictionary);
	}

Utilizing MailHelper for tests

Gmail

With gmail, you will need to allow IMAP on your account. It is recommended to make an alternate fake account just for testing. With that, you will just need to use your gmail and gmail password in the GmailMailHelper constructor.

MailTrap

MailTrap Site

You will need at least a free MailTrap account to utilize these helpers. The free account grants you a username, password, and Api Key. The API Key is used to access your inbox which MailTrapMailHelper takes in from its constructor.

Licensing

MailKit NuGet

relativity-test-helpers's Issues

Remove unneeded DLLs.

There are a ton of files that I don't think should be checked into the project. They are likely created when adding a reference to the Import Nuget package. We probably just need to delete and add these directories to the .gitignore file.

• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/cmmap000.bin (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/oilink.exe (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/oilink.jar (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccca-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccch-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccda-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccex-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccfa-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccfi-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccfmt-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccfnt-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccfut-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccind-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/scclo-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/sccut-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers.Example.NUnit/oi/unmanaged/wvcore-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccca-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccch-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccda-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccex-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccfa-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccfi-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccfmt-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccfnt-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccfut-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccind-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/scclo-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/sccut-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Test.Helpers/oi/unmanaged/wvcore-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/cmmap000.bin (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/oilink.exe (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/oilink.jar (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccca-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccch-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccda-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccex-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccfa-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccfi-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccfmt-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccfnt-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccfut-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccind-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/scclo-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/sccut-R1.dll (0)
• A Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/wvcore-R1.dll (0)

Figure out how to get successful merges into a nuget package and published automatically

This should not be that difficult but we need to figure it out.

Incorrect URI for GetRESTServiceUrl

relativity-test-helpers/Source/Relativity.Test.Helpers/Relativity.Test.Helpers/ServiceFactory/ServicesManager.cs

Line 83 in 294173e

servicesUri = new Uri($"{SharedTestHelpers.ConfigurationHelper.SERVER_BINDING_TYPE}://{SharedTestHelpers.ConfigurationHelper.RSAPI_SERVER_ADDRESS}/relativity.services");

GetRESTServiceUrl is returning the exact same thing as GetServicesURL when it should be returning a uri appended with relativity.rest/api

Have better error messages when creating a workspace

I'm using the create workspace method and it throws a null ref exception if the template workspace does not exist. Can we have better error messages when workspace creation fails?

Implement Retry Across Helpers

One issue faced by many who run integration tests is the fickle nature of RSAPI calls in a test environment. At times this leads to failed tests because a setup method didn't succeed. A typical way to handle this is to implement retry functionality. Here's a proposed implementation:

The Command pattern is a common solution to the problem of retry. We could implement an abstract class like the one below and inherit from it commands for each helper method as we need it.

public abstract HelperCommand
{
int NumberOfRetries = Constant.NumberOfRetries;
int CurrentRetryCount = 0;

public void IncrementRetry() => CurrentRetryCount++;
public bool TryAgain() => CurrentRetryCount >= NumberOfRetries;
}

public DeleteCommand : HelperCommand
{
private Action _actionToExecute;
public DeleteCommand(Action actionToExecute)
{
_actionToExecute = actionToExecute;
}

public void Execute()
{
do
{
try { _actionToExecute.Invoke();  }
catch (Exception e)
{ 
base.IncrementRetry();
// handle exception
 }

} while (base.TryAgain())
}

}

// Usage example
public void Delete()
{
var deleteCommand = new DeleteCommand( () => DeleteUser(userID) );
deleteCommand.Execute();
}

A similar example could be done for a CreateCommand, except it will use Func instead of Action as the executing code.

Supply a TestLogger

Integration tests typically stub out the IAPILog interface, which means that app logs are not captured during integration testing. These logs could be vital for two reasons:

1. Helps to diagnose test failures, esp. when in a CI/CD pipeline.

2. Makes integration test logging possible. That is, we could add helpful logs in Test Helpers to further explain what's happening in the final results

My first thought is to send these logs to console, which may work well if the integration tests are executed via the command line. A flat .log file may be a suitable alternative.

add appveyor support for CI

@ppande1020 or @mrobustelli I may need your help to set this up since I don't have the rights to see the repo with appveyor.

Start unit testing some of these methods.

I would like to openly start discussing adding some tests to this project seeing as many people will be contributing to it, making sure changes don't break for others will be helpful. Starting off I have a few questions.

I know Relativity uses nunit for tests do we want to use nunit for this project as well?
Do we want to add some CI/CD for PRs that pass certain gates?

• I know this can open a bag of worms into managing the project but starting the discussion now should help future people.

Add ability to secure items through permissions api

I have a few tests where I need to ensure my system is returning the correct values based on group permissions

Remove any Nunit packages/dependencies from the main project

I understand that the TestTemplate.cs should be added, but it might be in the projects best interest to not have that direct tie with nunit, as I fear this would introduce direct ties to NUnit. We should probably break out any templates/examples to their own respective projects.

Create/Setup Unit tests

As we use this more and more, I feel like we should create tests for this to make sure any updates are fine before merging into Develop and creating NuGet packages.

Specifically, we need tests on TestHellpers.cs. Recently, connecting to another instance was added as a feature and it needed some unit tests because some minor issues kept cropping up due to not being able to fully run through ti until we got the new package.

Make native path configurable in the ImportAPI Helper

I would like to make a change to the Import API helpers allowing the native name to be configurable. Before I do this I have a few asks:

1. What is the use case Relativity has with this file?
   • I'm specifically asking to allow the "\\FakeFilePath\Natives\SampleTextFile.txt" to be injected as a function parameter and not be hard coded.

Add OAuth2 helper functions

Downgrade version of .net

We are still working on lower version of .net and cannot use the test helpers for this. We should downgrade the version to .net4.6.1 or provide this nuget package that can install either version depending on the project

Document Helpers

Creating new Test helpers for the document object, currently we have a lot of Tests that run against the document object and to be able to create a document would be very helpful

• Create a document with no file location set (we default to txt), but also allows for me to set the file type when creating
• Delete document (please ensure the list is not empty and filter out null and 0 ids)

Replace DbContext with Kepler Service

Do we need all the "unmanaged" files still?

With the new update of the import nuget package, can't we get rid of some of the files in the project now?

I.E.
relativity-test-helpers/Source/Relativity.Test.Helpers/Relativity.Tests.Helpers.Tests.Unit/oi/unmanaged/

Design Test Helpers 2.0

With more use, we've identified some ways to improve Test Helpers to be more intuitive.

• Extract app configuration to allow injection of additional config options.
• Expand IHelper classes to include all: AgentHelper, EHHelper, etc.
• Organize object helpers into a DI factory method and simplify the interfaces

Large discussion around the direction of the project

I've seen a few PRs now that are moving to creating/using internal kepler services. Is this the direction the project is moving? if so why? My concern on my end is that I don't want the bloat in my environments. I don't see why we need to introduce more complexity to this project by adding kepler services. I always saw this project as a light weight dll to help developers get integration testing not an all in one solution. Is this project moving to that? again if so why?

Update Idbcontext method and add new DbContext Helper nuget package

Add new Application Install API functionality

The new Application Install API for Relativity versions 10.3.170.1+ should be added to the Application Install Helpers in this project.

The update to the helper will need to be backwards compatible and also smart enough to know which logic to use. It cannot force users to be on a certain version since Test Helpers is used for helping create tests across multiple versions of Relativity

ApplicationHelper ImportApplication needs better state management

Right now the ImportApplication assumes there are two states when installing the application, either running or completed. There are more states that the system can be in and does not handle these states well. I would like the handler to throw a more meaningful error when the install fails.

remove Hard NUnit Framework dependency

It seems the TestHelper constructor takes in a NUnit.Framework.TestContext testContext as a parameter. We do not use NUnit which is causing some interesting errors related to XUnit and Nunit installed together. Removing this dependency and using a class specific to the domain of the TestHelper will overall make things much beter.

Make Test Helper user aware

I'm creating tests that run on behalf of a user and need the Test Helper class aware of which user I want to login. I will need a method exposed to allow Test Helper the ability to be user aware.

Async method without await

In CreateWorkspace.cs line 32 the method definition is marked async, however, there is no await call in the method body.

This would be a quick fix to add the await keyword, however, the async call on line 77 returns an object, ProcessOperationResult, which returns a compiler error when using the await keyword,

'ProcessOperationResult' does not contain a definition for 'GetAwaiter'

This may indicate a defect, not in this source code, but in the API call not being properly async:

proxy.Repositories.Workspace.CreateAsync(templateArtifactID, workspaceDTO);

User create fails no indication

Was trying to use Create user and noticed I was getting a 0 as a return value. After digging in, it turns out the create user function does not double check the success flag and throws an error if the request failed. Please throw an error if create fails

Earlier version compatibility

The readme notes that this package is only compatible with Rel 9.5 and up. Would there be any interest in accepting contributions providing compatibility with earlier versions? Haven't begun working on it yet, but would potentially like to patch for 9.2 / 9.4 compatibility.

For our internal apps I've handled multi-version targeting using multiple build configurations and some manual amendment of project files, so in the absence of other input would likely go with that approach. Let me know, thanks!

Make Project Goatsbeard Compatible

Working on this in branch: goatsbeard-import-api-updates

Create Field Helper should allow creation on a specific object

The Create Field methods might be more useful if the object the field is being created on could be specified. Alternatively, the methods could be renamed to make it obvious that it creates a field on the Document object.

CreateUser function not working

It doesn't throw an exception and return an ArtifactID of 0.