fsprojects / azurestoragetypeprovider

An F# Azure Type Provider which can be used to explore Blob, Table and Queue Azure Storage assets and easily apply CRUD operations on them.

Azure Storage Type Provider

An F# Azure Type Provider which can be used to explore Azure Storage assets quickly and easily.

The goal is to create a provider which allows lightweight access to your Azure storage data within the context of e.g. F# scripts or applications to allow CRUD operations quickly and easily.

Support exists reading and writing from Blobs, Tables and Queues, as well as fallback to the standard .NET Azure SDK.

The provider currently does not support .NET Core for the reasons specified here.

Maintainer(s)

• @isaacabraham

The default maintainer account for projects under "fsprojects" is @fsprojectsgit - F# Community Project Incubation Space (repo management)

azurestoragetypeprovider's Issues

Hide Get methods on empty tables.

We know whether a table is empty or not immediately when we try to infer schema. If the schema is empty we could (should?) hide the query methods.

Allow account details to be loaded via a file

I prefer not to commit account details in to source control but with the provider as it currently stands I have no other choice.

Would be good if I could pass a file location which contain those details.

Empty containers and files

If a container has no files in it, the container gets a red squiggly. We should instead generate a single item in the container called something like "No files in Container" or similar.

Delete a file

In addition to being able to Download a file, we should be able to Delete a file from Blob Storage.

Tidy up scaffold

Incorporate an automatic push to nuget servers with tagging and pushing to GH as well.

clean up table storage code

modules need rearranging to be more consistent with blob implementation. Also see if we can refactor the EDM -> .Net property type mapping code (overly verbose copy-n-paste)

Have a nuget dependency on WindowsAzure.Storage

Having the files shipped inside the package has caused to binding redirect problems with pakets redirect option.

Allow simple querying of entities

Allow users some way to form a WHERE clause which can be mapped into an Azure Table Query. Doesn't need to be an IQueryable implementation to start with.

Delete an entity from a Table

Limit batches to 4mb in size

If you insert data more than 4mb in size in Table operations, the SDK throws an error. We should try to prevent this by batching in 4mb limits.

Parameterisation of type provider

Currently the type provider is hard coded to local storage. Allows the user to provide an Azure connection string instead.

Name property on Container

Add a Name property onto Blob Containers.

Provide the ability to lazily yield rows from Table storage

This can be accomplished using the continuation token in Azure Table Storage.

Async support for Tables

Queries on table storage do not support Async operations currently.

Move to fsprojects?

Hi @isaacabraham

Would you be interested in moving this to http://github.com/fsprojects? I don't mind either way but moving can raise visibility and a few other benefits like that.

thanks!
don

Download text files

Files that end in .txt should have a DownloadAsync method on it that will download the contents as a string.

Delete an entire partition

Generate SAS on demand

There should be a simple method on a file to generate a Shared Access Signature for a given TimeSpan.

Blobs are not properties

Blobs are currently all just static types that happen to have the same shape. They should be (static) properties on the container that implement an interface (or interfaces) - this would allow the user to operate on multiple files with collection operations.

Statically typed access to Import/Export Service

I know it's not easy to do but can be nice to have programmatic access to Import/Export Service.
http://msdn.microsoft.com/en-us/library/dn529096.aspx
Right now dev story is horrible because the service is half-ass WCF implementation meaning WsdlService provider cannot be used.
Some helpful links:
http://www.codeproject.com/Articles/287597/Sql-Azure-Import-Export-Service-bacpac-dac-Extract
http://blogs.msdn.com/b/carlosfigueira/archive/2012/03/26/mixing-add-service-reference-and-wcf-web-http-a-k-a-rest-endpoint-does-not-work.aspx
http://stackoverflow.com/questions/8206641/is-the-sql-azure-dac-import-export-service-wcf-or-rest-or-something-else
http://code.msdn.microsoft.com/windowsazure/Windows-Azure-SQL-Database-5eb17fe2

List blobs in folder

There is only one operation on folders for blob containers to download all files in the folder.
It'd be great to be able to list blobs in a folder to iterate on it.
Like:

account.``path/``.``to/``.``folder/``
|> Seq.map (fun blob -> blob.Name)

Create a basic AzureTableEntity

This is to allow developers to write common code against common properties of all entities i.e. row key, partition key and values.

Remove reliance on Azure .NET SDK

The Azure .NET SDK is large (adding over 1mb to the nuget package) and contains lots of fluff that is not needed.
Currently the TP only uses a relatively small proportion of the SDK so should look to hand-roll a thin wrapper around the raw REST API.

Allow use of whole connection string

It would be nice if the type provider just allowed a single connection string like this, which is what Azure gives you in some instances:

type Account = AzureTypeProvider<"DefaultEndpointsProtocol=https;AccountName=nameOfStorage;AccountKey=AHIUDMNWIKCSMNSCI(*WN"NC(*E==">

(That's not my real connection string :) )

Download content as a stream

Each file in blob storage should have a method on it to download asynchronously as a stream so e.g. the file can be downloaded to the local file system.

basic suggestions for partitions

Look at top n rows e.g. 100 and derive a distinct list of partitions.

Download content as a stream

Blobs should provide the capability to download the content as a stream. This is useful for e.g. reading the first few lines in a large file.

Sort out NuGet

Delete an entire Table

I can't open FSharp.Azure.StorageTypeProvider

I did the following:

1. Create a new F# project in VS2013
2. In Package Manager Console, execute while targetting the new project:
   Install-Package FSharp.Azure.StorageTypeProvider
3. In the file, type "open FSharp.Azure.StorageTypeProvider"

but I get:

"The namespace or module 'FSharp' is not defined"

What am I doing wrong? I am new to Nuget, and this is the first package I've downloaded that hasn't "just work"ed.

Support data binding scenarios

There are some interfaces that we could support that would better support data binding scenarios. Have a look at the other type providers out there for how this could be accomplished.

Add support for Path property.

It would be useful on every folder to have a "Path" property - this can be used in conjunction with ListBlobs to retrieve all child blobs. Either that or have a ListBlobs function on the folder.

Infer option fields during schema generation.

If we find fields that only exist in certain rows (or exist but are null) then we should make this field an Option type.

Storage Queue support

Provide basic support for Queues - this means navigating through queues, plus the ability to pop / push / peek at the storage queue, and perhaps a simple API on top of the queue message in order to accept / reject the message etc.

Insert an Entity for a table

Insert an Entity for a table

Hide generated types

Generated types are shown in intellisense to the user. They should only see the Type Provider to make it as easy to use as possible.

DateTimeOffset fields don't work

Accessing a DateTimeOffset field -> bang!

Error checking

Check on arguments passed etc. Currently we only do limited checking, but we should check whether the connection string is valid or not and somehow gracefully notify the caller.

Individuals for Tables

We should supply Individuals for Tables so that you can quickly get at e.g. the first 10 rows of a table or something like that. The SQL Type Provider is a good place to see how this has been done and we could try to replicate that behaviour.

Parameterise Blob commands?

It would be nice to be able to override the connection when downloading blobs etc. as can be done with table data.

Why does downloadFolder call Async.Start?

I don't think that downloadFolder should be calling Async.Start. I would think that it should just return the Async<unit> so that the caller of Download(path) can await it.

Use the latest version of Microsoft.WindowsAzure.Storage

Hi, Is it possible that the AzureStorageTypeProvider uses the latest version of Microsoft.WindowsAzure.Storage assembly instead of version - 3.0.2.0?

Individuals on Queues

Allow the viewing of messages on a Queue directly within the provider.

Directory structures are not live

The type provider needs to automatically reload directories only hitting "." in intellisense rather than caching the results, so that if you e.g. add a new file in Blob Storage, re-entering that folder will automatically get the latest files.

Download to CSV

Allow the user to download either an entire table or a specific partition to a local CSV file.

Simple parameterisation of table storage materialisation

Allow the user to specify that types should be materialised as non-option types, useful when the caller knows that e.g. the shape of an entity in an Azure Table is stable across all rows.

Page Blobs are not supported

I have a blob container called fsstorage that contains the following blobs:

log
log/app000001
log/app000001/2015-01-21
log/app000001/2015-01-21/activity_1421808013040
log/app000001/2015-01-21/crash_1421808013420
...

The type provider seems to get confused with this structure. I can dot up to:

But the next level down I get:

I expect to see activity_1421808013040 | crash_1421808013420 | ... | Download in the IntelliSense. Is this a bug?

Implement strongly-typed IQueryable for Table queries.

Create proper documentation

Rather than a .readme file, there should be a proper FSharp documentation github website using F# Formatting etc.