Coder Social home page Coder Social logo

taskcluster / fast-azure-storage Goto Github PK

View Code? Open in Web Editor NEW
15.0 9.0 8.0 1.07 MB

Fast Azure Storage Client for Node.js

Home Page: https://taskcluster.github.io/fast-azure-storage/

License: Mozilla Public License 2.0

JavaScript 100.00%
azure-storage-service javascript table-storage

fast-azure-storage's Introduction

Fast Azure Storage Client for Node.js

This library implements a low-level and highly optimized interface to Azure Storage Services. Existing node libraries for Azure suffers of excessive complexity, dependencies, being slow and not managing connection correctly.

At this point this library implement most of the APIs for queue, table and blob storage. Pull request with additional feature additions will generally be accepted, as long as patches don't compromise efficiency.

For full documentation see reference documentation or extensive comments in the sources.

Common Client Options

All three clients, Queue, Table and Blob, take a range of common configuration options.

Authentication Options

The following example illustrates how to create clients using shared key authentication.

// Load fast-azure-storage client
var azure = require('fast-azure-storage');

// Common options using shared key authentication
var options = {
  accountId:          '...',
  accessKey:          '...'
};

// Create queue, table and blob clients
var queue = new azure.Queue(options);
var table = new azure.Table(options);
var blob  = new azure.Blob(options);

It's also possible to configure clients with Shared-Access-Signatures as illustrated in the following example.

// Common options using shared-access-signatures
var options = {
  accountId:          '...',
  sas:                sas   // sas in querystring form: "se=...&sp=...&sig=..."
};

In fact it's possible to provide a function that will be used to refresh the Shared-Access-Signature when it's close to expire:

// Common options using shared-access-signatures
var options = {
  accountId:          '...',
  sas:                function() {
    return new Promise(/* fetch SAS from somewhere */);
  },
  // Time to SAS expiration before refreshing the SAS
  minSASAuthExpiry:   15 * 60 * 1000
};

Custom HTTPS Agent Configuration

The fast-azure-storage library comes with a custom https.Agent implementation, optimized for Azure Storage service to reduce latency and avoid errors. By default,Blob,Table and Queue clients will use a global instance of this custom agent configured to allow 100 connections per host.

You may override this behavior by supplying your own agent as follows.

// Common options for HTTPS agent configuration
var options = {
  agent:      new azure.Agent({...}),
};

Please, read the Built-in Azure HTTPS Agent section for details on why this custom https.Agent is necessary. Notice that while it's strongly recommended to use the HTTPS agent that ships with this library as oppose the default https.Agent implementation, it's perfectly sane to tune the options of the HTTPS agent that ships with this library, and even create multiple instances of it if you feel that is necessary.

Azure Table Storage Client

The Azure Storage Table client aims at interfacing Azure Table Storage without abstracting away the storage format and type information stored with each entity. It assumes that opinionated abstractions will do type conversions as necessary.

Simple example of table and entity creation.

// Load fast-azure-storage client
var azure = require('fast-azure-storage');

var table = new azure.Table({
  accountId:    '...',
  accessKey:    '...'
});

// Create table and insert entity
table.createTable('mytable').then(function() {
  return table.insertEntity('mytable', {
    PartitionKey:         '...',
    RowKey:               '...',
    'count':              42,
    '[email protected]':   'Edm.Int64',
    'buffer':             new Buffer(...).toString('base64'),
    '[email protected]':  'Edm.Binary'
  });
});

Table API Reference

See also reference documentation.

  • Table(options)
  • Table#queryTables(options)
  • Table#createTable(name)
  • Table#deleteTable(name)
  • Table#getEntity(table, partitionKey, rowKey, options)
  • Table#queryEntities(table, options)
  • Table#insertEntity(table, entity)
  • Table#updateEntity(table, entity, options)
  • Table#deleteEntity(table, partitionKey, rowKey, options)
  • Table#sas(table, options)
  • Table.filter(expression)

Azure Queue Storage Client

The Azure Storage Queue client aims at interfacing Azure Queue Storage.

Simple example of queue and message creation.

// Load fast-azure-storage client
var azure = require('fast-azure-storage');

var queue = new azure.Queue({
  accountId:    '...',
  accessKey:    '...'
});

// Create queue and insert message
queue.createQueue('myqueue').then(function() {
  return queue.putMessage('myqueue', 'my-message', {
    visibilityTimeout:  10,     // Visible after 10 seconds
    messageTTL:         60 * 60 // Expires after 1 hour
  });
});

Queue API Reference

See also reference documentation.

  • Queue(options)
  • Queue#listQueues(options)
  • Queue#createQueue(name, metadata)
  • Queue#deleteQueue(name)
  • Queue#getMetadata(queue)
  • Queue#setMetadata(queue, metadata)
  • Queue#putMessage(queue, text, options)
  • Queue#peekMessages(queue, options)
  • Queue#getMessages(queue, options)
  • Queue#deleteMessage(queue, messageId, popReceipt)
  • Queue#clearMessages(queue)
  • Queue#updateMessage(queue, text, messageId, popReceipt, options)
  • Queue#sas(queue, options)

Azure Blob Storage Client

The Azure Blob Storage client aims at interfacing Azure Blob Storage. Using this client, text and binary data can be stored in one of the following types of blob:

  • Block blobs, which are optimized for upload large blobs
  • Append blobs, which are optimized for append operations, making it ideal for eg. logging, auditing

Simple example of a container and blob creation.

// Load fast-azure-storage client
var azure = require('fast-azure-storage');

var blob = new azure.Blob({
  accountId:    '...',
  accessKey:    '...'
});

var blobContent = 'Sample content'; // The content can be a string or a Buffer
// Create container and upload a blob
blob.createContainer('mycontainer').then(function() {
  return blob.putBlob('mycontainer', 'myblob', {
    type:  'BlockBlob',     // Type of the blob 
  }, blobContent);
});

Blob API Reference

See also reference documentation.

  • Blob(options)
  • Blob#setServiceProperties(options)
  • Blob#getServiceProperties()
  • Blob#createContainer(name, options)
  • Blob#setContainerMetadata(name, metadata, options)
  • Blob#getContainerMetadata(name, options)
  • Blob#deleteContainer(name, options)
  • Blob#listContainers(options)
  • Blob#getContainerProperties(name, options)
  • Blob#getContainerACL(name, options)
  • Blob#setContainerACL(name, options)
  • Blob#listBlobs(container, options)
  • Blob#leaseContainer(name, options)
  • Blob#putBlob(container, blob, options, content)
  • Blob#getBlob(container, blob, options)
  • Blob#getBlobProperties(container, blob, options)
  • Blob#setBlobProperties(container, blob, options)
  • Blob#getBlobMetadata(container, blob, options)
  • Blob#setBlobMetadata(container, blob, metadata, options)
  • Blob#deleteBlob(container, blob, options)
  • Blob#putBlock(container, blob, options, content)
  • Blob#putBlockList(container, blob, options)
  • Blob#getBlockList(container, blob, options)
  • Blob#getBlockId(prefix, blockNumber, length)
  • Blob#appendBlock(container, blob, options, content)
  • Blob#sas(container, blob, options)

fast-azure-storage's People

Contributors

burmisov avatar ccooper avatar dependabot[bot] avatar djmitche avatar elenasolomon avatar eliperelman avatar imbstack avatar jhford avatar jonasfj avatar jwhitlock avatar leidegre avatar lotas avatar matt-boris avatar petemoore avatar renovate-bot avatar renovate[bot] avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fast-azure-storage's Issues

Problem with libxmljs dependency

I'm faced with problem to deploy Node.js app to Azure. I use fast-azure-storage for uploading files to azure blob storage. Locally every things work perfect, but when I trying to deploy it to Azure I have got error:

`

[email protected] install D:\local\Temp\8d4ca9654babd57\bundle\programs\server\npm\node_modules\libxmljs
debug: > node-pre-gyp install --fallback-to-build --loglevel http
debug:
debug: 'node-pre-gyp' is not recognized as an internal or external command,
debug: operable program or batch file.
debug: npm ERR! code ELIFECYCLE
debug: npm ERR! errno 1
debug: npm ERR! [email protected] install: node-pre-gyp install --fallback-to-build --loglevel http
debug: npm ERR! Exit status 1
debug: npm ERR!
debug: npm ERR! Failed at the [email protected] install script.
debug: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
debug:
debug: npm ERR! A complete log of this run can be found in:
debug: npm ERR! D:\local\AppData\npm-cache_logs\2017-07-14T08_59_56_308Z-debug.log
`

I read Azure documentation, and found - 'Azure App Service does not support all native modules and might fail at compiling those with very specific prerequisites."
Can You fix that ? Or give tips on how to use your package with Azure deploy.

Thanks!

Bundling with webpack raises error

Bundling this library with webpack raises the following error:

Error: Cannot find module './blob'                                                                                                        
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:582:15)                                                           
    at Function.Module._load (internal/modules/cjs/loader.js:508:25)                                                                      
    at Module.require (internal/modules/cjs/loader.js:637:17)                                                                             
    at require (internal/modules/cjs/helpers.js:22:18)                                                                                    
    at Object.get [as Blob] (/node_modules/fast-azure-storage/lib/index.js:17:1)                                                          
    at cache (/src/lib/blob-cache.js:8:1)                                                                                                 
    at Object.<anonymous> (/src/lib/query-maxmind.js:24:1)                                                                                
    at __webpack_require__ (/webpack:bootstrap:19:1)                                                                                      
    at Object.<anonymous> (/src/index.js:4:1)                                                                                             
    at __webpack_require__ (/webpack:bootstrap:19:1)  

The problem lies in the way index lazily exports the modules. I am wondering, is there any reason for this or we could export the modules in the good ol' modules.exports = {...} way?

New releae

Hi, any chance to get a new release that includes the libxmljs changes?
Thanks!

CODE_OF_CONDUCT.md isn't correct

Your required text does not appear to be correct

As of January 1 2019, Mozilla requires that all GitHub projects include this CODE_OF_CONDUCT.md file in the project root. The file has two parts:

  1. Required Text - All text under the headings Community Participation Guidelines and How to Report, are required, and should not be altered.
  2. Optional Text - The Project Specific Etiquette heading provides a space to speak more specifically about ways people can work effectively and inclusively together. Some examples of those can be found on the Firefox Debugger project, and Common Voice. (The optional part is commented out in the raw template file, and will not be visible until you modify and uncomment that part.)

If you have any questions about this file, or Code of Conduct policies and procedures, please reach out to [email protected].

(Message COC003)

Using with Azure Storage Emulator

I am trying to use this module for development purposes before linking to a production account. For this reason I am using the storage emulator but I cannot seem to connect using fast-azure-storage. Here is what I've tried:

const blob = new azure.Blob({ accountId: "devstoreaccount1", accessKey: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==" });

and

const blob = new azure.Blob({ accountId: "devstoreaccount1", accessKey: "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==", sas: "sv=2018-03-28&si=uploads-164B532CC89&sr=c&sig=nZXR2hVmx81jRMhPpy4ZsKzKi2OLhTkziEj66O5fC7A%3D" });

and even
const blob = new azure.Blob({ accountId: "devstoreaccount1", sas: "sv=2018-03-28&si=uploads-164B532CC89&sr=c&sig=nZXR2hVmx81jRMhPpy4ZsKzKi2OLhTkziEj66O5fC7A%3D" });

I just keep getting the following error

Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature

I was using azure-storage with a connection string before as such:
azureStorage.createBlobService( "DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://127.0.0.1:10000/devstoreaccount1;" );

is there an option to do the same with this module?

Dependency Dashboard

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Other Branches

These updates are pending. To force PRs open, click the checkbox below.

  • chore(deps): update dependency request to v2.88.2

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

Detected dependencies

npm
package.json
  • debug ^4.1.1
  • xml2js ^0.6.0
  • jsdoc 4.0.2
  • mocha 10.2.0
  • taskcluster-client 65.0.1
  • yuidocjs 0.10.2
  • node >=8
  • request ^2.88.0
  • fresh ^0.5.2
  • forwarded ^0.2.0
  • mime ^3.0.0

  • Check this box to trigger a request for Renovate to run again on this repository

plans for supporting react-native?

Hi, I am trying to use this library within react-native to upload images to Azure blob storage. I ran into the following issues (with some suggestions)

  1. crypto is included but never used. This results in transpiling issues as RN does not support native modules. I think this line should be removed as the crypto module is never used. (usage of a simple linter with pre-commit could solve issues like these in development phase)
  2. util and events are node core module which don't get transpiled and throws errors. Not sure what would be the best way to go about this. One idea would be use ES6 classes which would remove the dependency on util but we would still have a dep on events.

Would love to hear your thoughts about this.

P.S: The official azure sdk also doesn't support React Native yet. Azure/azure-sdk-for-js#5771 I did not come across any existing libraries that support azure storage in RN yet.

Action Required: Fix Renovate Configuration

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: undefined. Note: this is a nested preset so please contact the preset author if you are unable to fix it yourself.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.