Coder Social home page Coder Social logo

jessebenson / service-fabric-queryable Goto Github PK

View Code? Open in Web Editor NEW
42.0 11.0 9.0 351 KB

Enable query support for your stateful services in Service Fabric via the OData protocol.

License: MIT License

C# 100.00%
microsoft servicefabric query queryable microservices reliable-stateful-service reliable-collections odata

service-fabric-queryable's Introduction

ServiceFabric.Extensions.Services.Queryable

Getting Started

  1. Create a stateful ASP.NET Core service.

  2. Add the ServiceFabric.Extensions.Services.Queryable nuget package.

  3. Add the ODataQueryable middleware to your Startup.cs. This will intercept calls to the /$query endpoint to expose OData query capabilities over the reliable collections in your service.

using ServiceFabric.Extensions.Services.Queryable;

public class Startup
{
	public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
	{
		...
		app.UseODataQueryable();
		...
	}
}

Add the ODataQueryable middleware to your stateful services (using ASP.NET Core stateful services), ensure Reverse Proxy is enabled, and start querying your reliable collections.

Use rcctl to query your middleware

There is a command line tool you can use to query your middleware endpoints called rcctl.

You can install it and start using it with

pip install rcctl

Use REST calls to query your middleware

You can interact directly with the middleware by constructing REST calls. If your service is named 'fabric:/MyApp/MyService' and your reliable dictionary is named 'my-dictionary', try queries like:

Get OData metadata about a single partition stateful service:

  • GET http://localhost:19081/MyApp/MyService/$query/$metadata

Get OData metadata about a partitioned stateful service:

  • GET http://localhost:19081/MyApp/MyService/$query/$metadata?PartitionKind=Int64Range&PartitionKey=0

Get 10 items from the reliable dictionary.

  • GET http://localhost:19081/MyApp/MyService/$query/my-dictionary?$top=10

Get 10 items with Quantity between 2 and 4, inclusively.

  • GET http://localhost:19081/MyApp/MyService/$query/my-dictionary?$top=10&$filter=Quantity ge 2 and Quantity le 4

Get 10 items, returning only the Price and Quantity properties, sorted by Price in descending order.

  • GET http://localhost:19081/MyApp/MyService/$query/my-dictionary?$top=10&$select=Price,Quantity&$orderby=Price desc

Use ReliableIndexedDictionaries to speed up your queries by taking advantage of secondary indices

Some queries may take a long time or not complete. If you expect that you will be querying against a field often, you should consider adding a secondary index on that field, which will dramatically speed up your queries.

To do so, when you are making your dictionary, instead of using

StateManager.GetOrAddAsync<IReliableDictionary>("name")

you should use

StateManager.GetOrAddIndexedAsync("name",
	FilterableIndex<KeyType, ValueType, Property1Type>.CreateQueryableInstance("Property1Name"),
	FilterableIndex<KeyType, ValueType, Property2Type>.CreateQueryableInstance("Property2Name"),
	etc.)

This will create a dictionary with secondary indices on ValueType.Property1Name and ValueType.Property2Name. To find out more about ReliableIndexedDictionary go to the indexing repository

Note: you have to create your indexes the first time you define your dictionary. If you make them later, they will not be truly consistent with the dictionary and this can lead to failed requests or data corruption.

Now, if we made a secondary index on a property called Age on our ValueType, these queries would be faster because of indexing:

  • $filter=Value.Age eq 20
  • $filter=Value.Age gt 25
  • $filter=Value.Age le 40 and Value.Name eq "John"

However the following would not be faster than without using an indexed dictionary:

  • $filter=Value.Name eq "John"
  • $filter=Value.Age le 40 or Value.Name eq "John"

If we added a secondary index on both Age and Name, then all the above queries would be faster.

Check out UserSvc in the BasicApp sample to see both an unindexed and an indexed dictionary being constructed.

Using LINQ to query ReliableIndexedDictionaries

In addition to external requests through the OData middleware, ReliableIndexedDictionaries can be queried from your application code using LINQ. However, similarly to external queries, not all queries will work effectively with indexing. If your query is not supported, you should use GetAllAsync() to get the entire result set and apply your LINQ query against that.

To create a queryable instance of your IReliableIndexedDictionary, call:

var qd = new QueryableReliableIndexedDictionary<TKey, TValue, TValue>(indexedDictionary, stateManager);

Then you can carry out queries such as:

var query = qd.Where(x => x.Age == 25);
var query = qd.Where(x => x.Age >= 25).Select(x => x.Email);
var query = from Person person in qd
            where person.Age >= 25 && person.Email == "[email protected]"
            select person.Email;
var query = qd.Where(x => x.Name.CompareTo("Name1") >= 0);

Some import notes for querying:

  1. Put all your WHERE logic in a single Where statement and join using && and || operators, as the query may not be efficient if it is spread across multiple WHERE clauses.
  2. If you want to compare an IComparable type that does not have ">", "<=" operators, you must give it in the form: property.CompareTo(constant) '>,<,<=,=>' 0

Samples

service-fabric-queryable's People

Contributors

amenarde avatar ashishnegi avatar jessebenson avatar waynemunro avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

service-fabric-queryable's Issues

Not supporting for Actor data

I am having a Service Fabric application with some Actor Services in it.
Can I use the same mechanism for querying all the Actor States?
Or do we have any mechanism for querying the actors?

Thanks
Prabhuji

.NET Core Support

I've had a stab at upgrading the library to work with .NET Core services, using the latest libs (AspNet.OData / UriParser etc) which mostly works.

Metadata is OK, querying returns data, $top works. But $filter etc I cannot get to work and am at a crossroads.

If a fork/repo were provided, would this be something you could advise on, or any suggestions on what would be next steps to get this working? From the look of the codebase, this appears to be very similar in working to what Cosmos must do behind the scenes.

Issues in Service Fabric Queryable extensions

am working on integrating the Queryable extensions of ReliableIndexDictionary and facing few challenges on querying the Reliable Dictionary. Below are the exception details with respect to the query I am performing.

Reliable Indexed Dictionary:

var dictionary = await this.StateManager.GetOrAddIndexedAsync<Guid, File>(
                       "FileDictionaryName",                     
                      new FilterableIndex<Guid, File, string>(nameof(File.FriendlyFileName), (k, v) => v.fileName),                   
                        new FilterableIndex<Guid, File, long>(nameof(File.FileSizeInBytes), (k, v) => v.FileSizeInBytes.GetValueOrDefault()),
                       new SearchableIndex<Guid, File>(nameof(File.FileName), (k, v) => v.fileName)).ConfigureAwait(false);
Queryable Indexed Dictionary:

var queryableDictionary = new QueryableReliableIndexedDictionary<Guid, File, File >(dictionary, this.StateManager);

NOTE: We are unable to get all the records in the collection.

Can anyone please help out on this?

I have posted a question in stack over flow below is the link and issue details,
https://stackoverflow.com/questions/53482969/service-fabric-queryable-extensions-facing-issues-while-performing-linq-queri

Build warnings : use of .net framework library in .net coreclr project

Hi

I am receiving following warnings when I am building : ServiceFabric.Extensions.Services.Queryable

Build Warnings :

1>------ Build started: Project: ServiceFabric.Extensions.Services.Queryable, Configuration: Debug x64 ------
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1608: Detected package version outside of dependency constraint: Microsoft.ServiceFabric.Services 3.1.283 requires Microsoft.ServiceFabric.Data (= 3.1.283) but version Microsoft.ServiceFabric.Data 3.2.162 was resolved.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1608: Detected package version outside of dependency constraint: Microsoft.ServiceFabric.Diagnostics.Internal 3.1.283 requires Microsoft.ServiceFabric (= 6.2.283) but version Microsoft.ServiceFabric 6.3.162 was resolved.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'Microsoft.AspNet.WebApi.Client 5.2.2' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'Microsoft.AspNet.WebApi.Core 5.2.2' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'Microsoft.AspNet.WebApi.OData 5.7.0' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'System.Linq.Dynamic 1.0.7' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>ServiceFabric.Extensions.Services.Queryable -> E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\bin\x64\Debug\netstandard2.0\ServiceFabric.Extensions.Services.Queryable.dll
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1608: Detected package version outside of dependency constraint: Microsoft.ServiceFabric.Services 3.1.283 requires Microsoft.ServiceFabric.Data (= 3.1.283) but version Microsoft.ServiceFabric.Data 3.2.162 was resolved.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1608: Detected package version outside of dependency constraint: Microsoft.ServiceFabric.Diagnostics.Internal 3.1.283 requires Microsoft.ServiceFabric (= 6.2.283) but version Microsoft.ServiceFabric 6.3.162 was resolved.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'Microsoft.AspNet.WebApi.Client 5.2.2' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'Microsoft.AspNet.WebApi.Core 5.2.2' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'Microsoft.AspNet.WebApi.OData 5.7.0' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>E:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\ServiceFabric.Extensions.Services.Queryable.csproj : warning NU1701: Package 'System.Linq.Dynamic 1.0.7' was restored using '.NETFramework,Version=v4.6.1' instead of the project target framework '.NETStandard,Version=v2.0'. This package may not be fully compatible with your project.
1>Done building project "ServiceFabric.Extensions.Services.Queryable.csproj".
1>Successfully created package 'e:\service-fabric-queryable\src\ServiceFabric.Extensions.Services.Queryable\bin\x64\Debug\ServiceFabric.Extensions.Services.Queryable.3.1.300.nupkg'.
1>Done building project "ServiceFabric.Extensions.Services.Queryable.csproj".
========== Build: 1 succeeded, 0 failed, 1 up-to-date, 0 skipped ==========

Are these build warnings known ?

Access records locked by another transaction in Service Fabric reliable indexed dictionary

In our application, we are using service fabric indexed dictionaries to store data. We are also using Service Fabric Queryable package to add querying capabilities on our collections.

In our case, we are having a transaction in which we do multiple updates and commit all of them at once. We have also created indexes on some properties for faster retrieval of data. Due to this indexing a record is internally created in a index dictionary with the indexed property as the key and the object as the value. This is also an AddOrUpdate operation and is included in the same transaction. And the transaction which is used to perform these update operation acquires an exclusive lock on all the newly created records.

Using the same transaction, when we try to perform a query on the queryable indexed dictionary using the property which has been indexed, we are getting a Timeout Exception.

We have gone through the source code in the Queryable extensions package and found that the query which we run on the queryable dictionary creates a new transaction to fetch the filtered records. But this transaction is unable to access the records as the previous transaction in our code has acquired an Exclusive lock on the record.

Below is some sample code which describes our scenario.

public async Task MyMethod() 
{
    using (var tx = this.StateManager.CreateTransaction())
    {
         foreach(var record in records)
         {
              // Here we perform an update operation 
              // The transaction takes an exclusive lock on the created record to perform this operation and releases only after it is committed
              var result = await indexedDictionary.AddOrUpdateAsync(tx, myKey, myValue, (key, value) => myValue);

              // Creating a Queryable dictionary
              var queryableDictionary = new QueryableReliableIndexedDictionary(indexedDictionary, this.StateManager);

              // Running a query on the queryableDictionary 
              var filteredRecords = queryableDictionary.Where(x => x.indexedProperty== propertyValue).ToList();

              // The code to filter the data exists in the Service Fabric Queryable package where a new transaction is being created.
              // Once we query with where condition on the queryable dictionary, the Execute method gets called in the package where the actual filtering takes place.
              // That code looks similar to the code snippet given below.
         }

         await tx.CommitAsync();

    }        
}

Below is the code snippet from the Service Fabric Queryable package.

internal static async Task<object> Execute<TKey, TValue>(Expression expression)
{


    // This transaction is being created which tries to access the records actual dictionary
    using (var tx = stateManager.CreateTransaction())
    {
         // This statement tries to access the dictionary.. but it can't as the previous transaction has acquired an exclusive lock on the record it is trying to access
         // It waits for 4 seconds and throws a Timeout Exception
         IEnumerable<KeyValuePair<TKey, TValue>> pairs = await indexedDictionary.GetAllAsync(tx, keys, TimeSpan.FromSeconds(4), new CancellationToken()).AsEnumerable();
         values = new KeyValueToValueEnumerable<TKey, TValue>(pairs);
         await tx.CommitAsync();
    }
}

Is there any work around for us to be able to query the queryable dictionary without having to commit the previous transaction ?

Unit testing Service Fabric Queryable

Hi Team,

We are trying to create unit test cases for Queryable Reliable Indexed Dictionary, but we are facing below error message:

_InnerException {System.InvalidProgramException: No query over the data source was specified.
at ServiceFabric.Extensions.Services.Queryable.LINQ.ReliableIndexedDictionaryQueryContext.Execute[TKey,TValue](Expression expression, IReliableStateManager stateManager, IReliableIndexedDictionary`2 dictionary, Boolean IsEnumerable)} System.Exception {System.InvalidProgramException}

This issues is observed only on unit tests, which are querying on indexed columns. We are able to run application and query all endpoints but such unit test cases are failing.

Can you please help us out on this issue?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.