Coder Social home page Coder Social logo

bbc / sqs-producer Goto Github PK

View Code? Open in Web Editor NEW
198.0 32.0 51.0 698 KB

Simple scaffolding for applications that produce SQS messages

Home Page: https://bbc.github.io/sqs-producer/

License: Other

TypeScript 99.58% JavaScript 0.42%
sqs-producer sqs aws awssqs

sqs-producer's Introduction

sqs-producer

NPM downloads Build Status Maintainability Test Coverage

Enqueues messages onto a given SQS queue.

Installation

To install this package, enter the following command into your terminal (or the variant of whatever package manager you are using):

npm install sqs-producer

Note This library assumes you are using AWS SDK v3. If you are using v2, please install v2.2.0:

npm install [email protected] --save-dev

Node Version

We will only support Node versions that are actively or security supported by the Node team. If you are still using an Node 14, please use a version of this library before the v3.2.1 release, if you are using Node 16, please use a version before the v3.3.0 release.

Documentation

Visit https://bbc.github.io/sqs-producer/ for the full API documentation.

Usage

import { Producer } from 'sqs-producer';
import { SQSClient } from '@aws-sdk/client-sqs';

// create simple producer
const producer = Producer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  region: 'eu-west-1'
});

// send messages to the queue
await producer.send(['msg1', 'msg2']);

// get the current size of the queue
const size = await producer.queueSize();
console.log(`There are ${size} messages on the queue.`);

// send a message to the queue with a specific ID (by default the body is used as the ID)
await producer.send([
  {
    id: 'id1',
    body: 'Hello world'
  }
]);

// send a message to the queue with
// - delaySeconds (must be an number contained within 0 and 900)
// - messageAttributes
await producer.send([
  {
    id: 'id1',
    body: 'Hello world with two string attributes: attr1 and attr2',
    messageAttributes: {
      attr1: { DataType: 'String', StringValue: 'stringValue' },
      attr2: { DataType: 'Binary', BinaryValue: new Buffer('binaryValue') }
    }
  },
  {
    id: 'id2',
    body: 'Hello world delayed by 5 seconds',
    delaySeconds: 5
  }
]);

// send a message to a FIFO queue
//
// note that AWS FIFO queues require two additional params:
// - groupId (string)
// - deduplicationId (string)
//
// deduplicationId can be excluded if content-based deduplication is enabled
//
// https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queue-recommendations.html
await producer.send({
  id: 'testId',
  body: 'Hello world from our FIFO queue!',
  groupId: 'group1234',
  deduplicationId: 'abcdef123456' // typically a hash of the message body
});

Credentials

By default the consumer will look for AWS credentials in the places specified by the AWS SDK. The simplest option is to export your credentials as environment variables:

export AWS_SECRET_ACCESS_KEY=...
export AWS_ACCESS_KEY_ID=...

If you need to specify your credentials manually, you can use a pre-configured instance of the SQS Client client.

import { Producer } from 'sqs-producer';
import { SQSClient } from '@aws-sdk/client-sqs';

// create simple producer
const producer = Producer.create({
  queueUrl: 'https://sqs.eu-west-1.amazonaws.com/account-id/queue-name',
  region: 'eu-west-1',
  sqs: new SQSClient({
    region: 'my-region',
    credentials: {
      accessKeyId: 'yourAccessKey',
      secretAccessKey: 'yourSecret'
    }
  })
});

// send messages to the queue
await producer.send(['msg1', 'msg2']);

Development

Test

npm test

Coverage

For coverage report, run the command:

npm run coverage

Lint

To check for problems using ESLint

npm run lint

Contributing

We welcome and appreciate contributions for anyone who would like to take the time to fix a bug or implement a new feature.

But before you get started, please read the contributing guidelines and code of conduct.

License

SQS Producer is distributed under the Apache License, Version 2.0, see LICENSE for more information.

sqs-producer's People

Contributors

cokia avatar dependabot[bot] avatar diegorbaquero avatar fabulator avatar gomugilad6 avatar greenkeeperio-bot avatar gregory avatar imsnif avatar jeanrauwers avatar jhead avatar knicola avatar koresar avatar maowug avatar mmeinzer avatar mogu4iy avatar nicholasgriffintn avatar niklasr avatar nspragg avatar qvantor avatar rcadelw33 avatar robertyoung avatar robinjmurphy avatar soonoo avatar varunnaik avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sqs-producer's Issues

The `.send()` method improvements

  1. Looking at the code I believe that the return type of the .send() method is wrong.
async send(messages: string | string[]): Promise<string[]> {

The function always return undefined. I believe the return type should be void instead of string[].

Should I submit a 1-line PR to fix this?

  1. There were a number of requests from several people to return AWS raw response(s) form this function. The currently returned undefined is somewhat wasteful. It would be great to take the result variables, concat to an array, and return from the .send() method:
const result = await this.sqs.sendMessageBatch(params).promise();
results.push(result);

...

return results;

This is a non-breaking enhancement people really need.

Should I submit a PR?

The Documentation isn't up to date

Describe the bug
In the docs, it says that we can pass accessKeyId and secretAccessKey to the Producer.create method, but we can only pass an SQS object, the README need an update

screenshots
image
image

cb is not a function

Have it all set up and suddenly got a cb is not a function TypeError. Please see code below

`E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:31
throw err;
^

TypeError: cb is not a function
at Producer._sendBatch (E:\Platform\OBSIDIAN\node_modules\sqs-producer\lib\producer.js:133:5)
at Producer.send (E:\Platform\OBSIDIAN\node_modules\sqs-producer\lib\producer.js:158:8)
at Response.sqs.getQueueUrl (E:\Platform\OBSIDIAN\application\routes\upload.js:75:34)
at Request. (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:364:18)
at Request.callListeners (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\sequential_executor.js:105:20)
at Request.emit (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:683:14)
at Request.transition (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:22:10)
at AcceptorStateMachine.runTo (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\state_machine.js:14:12)
at E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request. (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:38:9)
at Request. (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:685:12)
at Request.callListeners (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\sequential_executor.js:77:10)
at Request.emit (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:683:14)
at Request.transition (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:22:10)
at AcceptorStateMachine.runTo (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\state_machine.js:14:12)
at E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\state_machine.js:26:10
at Request. (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:38:9)
at Request. (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\request.js:685:12)
at Request.callListeners (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\sequential_executor.js:115:18)
at Request.emit (E:\Platform\OBSIDIAN\node_modules\aws-sdk\lib\sequential_executor.js:77:10)`

Can only send string events and at most 80 characters

Describe the bug
As referenced elsewhere the types that the send method supports in typescript projects are string and string[], and that is how my team is attempting to use the library. However, when posting a string, the Id attribute is set as the string itself which means that the library can only post stringified JSON events of at most 80 characters. This is an unrealistic limitation. The object version is valid as a workaround, however we work exclusively in typescript and the compile time errors generated when attempting to utilize this api aren't worth the effort to overcome vs. using the AWS SDK directly.

To Reproduce
Steps to reproduce the behavior:

  1. Import the library in to a typescript project
  2. Attempt to send an event > 80 characters in length or with characters not confirming to /[A-Za-z0-9_\-]/
  3. Receive this error: A batch entry id can only contain alphanumeric characters, hyphens and underscores. It can be at most 80 letters long.

Expected behavior
The types accepted are robust enough to reflect that objects can be sent instead of just strings or string arrays and that those objects have keys which are optional. When sending strings there is no limitation that the string be less than 80 characters in length, or consist only of whitelisted characters.

Recommendations
Enable strict mode in tsconfig.json. These issues would most likely have been caught prior to publication by the typescript compiler.

Upgrade chai to v5

Currently upgrading to this version or above causes issues with the tests, we should see how best to resolve those issues to get upgraded.

Next release?

I have some typing issues on the send method of the producer and I see the bug has been resolved on master 2 months ago. When's the next release planned?

Cheers!

queueUrl should not be possibly undefined in ProducerOptions

The problem
There is code to validate that queueUrl isn't undefined, it adds extra logic, I think it's better if this logic is directly in the ProducerOptions type
Suggested solution
Change ProducerOptions type, set queueUrl as a string instead of an undefined | string

sending Message makes two entry of one message into aws SQS using lambda to send the message

var uuid = require('uuid/v1');
var Producer = require('sqs-producer');
var AWS = require('aws-sdk');

exports.handler = (event, context, callback) => {

var producer = Producer.create({
  queueUrl: QUEUE_URL,
  region: 'eu-west-1'
});

var dataToSqsQueue = JSON.stringify(event);
console.log(dataToSqsQueue);

producer.send({
  id:uuid(),
  body: dataToSqsQueue,
  groupId: uuid(),
  deduplicationId: uuid() // typically a hash of the message body 
}, function(err) {
  if (err) console.log(err);
});

};

Remove AWS SDK V2 note

Remove the note on our readme that points to V2 support from September 8th, 2024

Please join the discussion (#103) if you would like to feedback on this decision.

Support Message.body to be an object

The problem
To pass complex payloads in Message body it requires to JSON.stringify it first

            this.sqsProducer.send(
                {
                    body: JSON.stringify({
                        type: "MESSAGE_TYPE",
                        payload: {},
                    }),
                    id: "MESSAGE_ID",
                },
                (err) => {
                    if (err) { }
                },
            );

Suggested solution
Make it work with plain js objects in message body

            this.sqsProducer.send(
                {
                    body: {
                        type: "MESSAGE_TYPE",
                        payload: {},
                    },
                    id: "MESSAGE_ID",
                },
                (err) => {
                    if (err) { }
                },
            );

Extend Message.body to be of union type

export interface Message {
    id: string;
    body: string | Object;
    groupId?: string;
    deduplicationId?: string;
    delaySeconds?: number;
    messageAttributes?: SQS.MessageBodyAttributeMap;
}

Add JSON.stringify to entry

    const entry: SendMessageBatchRequestEntry = {
        Id: message.id,
        MessageBody: typeof message.body === 'string' ? message.body : JSON.stringify(message.body)
    };

Alternatives considered

Additional context

Wait for message result

I trying to use this package in microservice context.
I need to send a message and wait for the result, for example, send a filter and get the list of objects.
How can I wait to get the message results?

Thanks!

[Bug]: `npm link sqs-producer` does not work in new v3

Describe the bug

Install globally. Link it. The npm link fails.

Your minimal, reproducible example

npm i -g sqs-producer && npm link sqs-producer

Steps to reproduce

> npm i -g sqs-producer
> npm link sqs-producer
npm ERR! code 127
npm ERR! path ~/.nvm/versions/node/v16.18.1/lib/node_modules/sqs-producer
npm ERR! command failed
npm ERR! command sh -c -- npm run build
npm ERR! > [email protected] build
npm ERR! > npm run clean && tsc
npm ERR! 
npm ERR! 
npm ERR! > [email protected] clean
npm ERR! > rm -fr dist/*
npm ERR! sh: tsc: command not found

Expected behavior

No errors. sqs-producer installed globally and working fine.

How often does this bug happen?

Every time

Screenshots or Videos

No response

Platform

All

Package version

3.0, 3.1

AWS SDK version

No response

Additional context

Offtopic: This bug template is a bit annoying. It should not mandate for "minimal reproducible example" as a URL

Requires id for FIFO queues

Hello!

I'm trying to send a message to a FIFO queues, which require de groupId and deduplicationId, but the package is requiring me to enter an ID which is never used.

My code:

const producer = Producer.create({
  queueUrl: 'https://sqs.us-east-1.amazonaws.com/xxxxxxxx/yyyyyyyyy.fifo',
  region: 'us-east-1'
})

const content = JSON.stringify({zzzzzzzz: 1})

const message = {
  body: content,
  id: 'woot',
  groupId: 'group1234',
  deduplicationId: shajs('sha256').update(content).digest('hex')
}

producer.send(message, err => {
  if (err) console.log(err)
})

Error: Error: Object messages must have 'id' and 'body' props
at entryFromObject (/lib/producer.js:47:11)

v1.4.0 not on npm

Could you please publish version 1.4.0 to npm please ?

I need the Producer export from PR 15 ;)

Is ts-node-register production dependency intended?

As per contribution guide rising this important question before submitting a 1-line PR.

I found that ts-node-register is a production dependency of the module. ts-node-register has few unnecessary subdependencies.

I would assume including this was an incident. The ts-node-register is not used anywhere.

Should I submit a PR to remove this line from package.json?

Changes to send types

Question
version 1.6.1 types for .send were

send(messages: string | string[] | ProducerMessage | ProducerMessage[], cb: ProducerCallback<void>): void;

this has been changed to

send(messages: string | string[]): Promise<SendMessageBatchResultEntryList>;

does this mean to old format of

{
  id: string;
  body: string;
  messageAttributes?: { [key: string]: ProducerMessageAttribute };
  delaySeconds?: number;
  groupId?: string;
  deduplicationId?: string;
}

is no longer supported or is it just missing types?

Drop support for Node 16

Node 16 will be at its end of life on 11th Sep 2023.

Due to this, support should be dropped for it from this package and we should move to support node 18 and node 20 only.

Why queue url and not queue name?

I find it a bit weird that the create() require both a region and a complete queue url, and not relying on the getQueueUrl?

Is there a specific reason for that?

Consider using debug

I was recently getting a "Failed to send message: 1" with no info.

Please consider adding debug to add optional debugging.

Adding a quick console log to the result from AWS showed that I was adding an extra param not expected, but sqs-producer would not forward this error and no other info was available.

Refactor prototype

Replace the prototype with a class. The public/private methods and state should be maintained.

mocha.opts will be DEPRECATED

Describe the bug
Remove the warning moving await from mocha.opts.

DeprecationWarning: Configuration via mocha.opts is DEPRECATED and will be removed from a future version of Mocha. Use RC files or package.json instead.

[Feature]: env variable AWS_REGION is not used during producer creation

Describe the bug

Env variable AWS_REGION is not used during producer instantiation

Your minimal, reproducible example

https://codesandbox.io/s/frosty-margulis-vrh93k?file=/src/index.js

Steps to reproduce

// those code doesn't generate the same result
Producer.create({
queueUrl: "myQueueUrl"
})

Producer.create({
queueUrl: "myQueueUrl",
region: process.env.AWS_REGION
})

Expected behavior

process.env.AWS_REGION should be used during creation

How often does this bug happen?

Every time

Screenshots or Videos

No response

Platform

all

Package version

3.1.3

AWS SDK version

No response

Additional context

No response

Unlock aws-sdk dependency

Describe the bug
aws-sdk version is locked to version 2.677.0. This version has several security vulnerabilities. Can you unlock it in order to use a compatible version added to the project where we add this package as dependency?

Retrieve messageId

Hi,

it would be very useful to have the MessageId generated by AWS when the message gets pushed. Is there any way to obtain this information in the callback notification?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.