Coder Social home page Coder Social logo

typedorm / typedorm Goto Github PK

View Code? Open in Web Editor NEW
447.0 5.0 46.0 2.88 MB

Strongly typed ORM for DynamoDB - Built with the single-table-design pattern in mind.

License: MIT License

TypeScript 96.87% JavaScript 1.54% Starlark 1.46% Shell 0.13%
dynamodb orm dynamodb-orm aws-dynamodb dynamodb-table data-mapper strongly-typed aws no-sql typescript

typedorm's Introduction

TypeDORM

Code Style: Google semantic-release

Object Relational mapper for DynamoDB, inspired by typeorm.

TypeDORM is an ORM built from ground up using typescript and latest javascript features to provide an easy gateway when doing complex highly relational data modeling in dynamoDB. TypeDORM is built with single-table-design first in mind, but should work as smoothly with regular entity table <-> design pattern. TypeDORM would have not existed without TypeORM and dynamodb-toolbox, big shout-out to these projects and their awesome contributors.

TypeDORM borrows decorator based syntax from TypeORM and provides fully type safe ORM to with dynamodb. TypeDORM currently only support Data Mapper.

Packages

Package Latest Stable Recent Beta Recent Alpha
@typedorm/common NPM Release NPM Release NPM Release
@typedorm/core NPM Release NPM Release NPM Release
@typedorm/testing NPM Release NPM Release NPM Release
@typedorm/document-client NPM Release NPM Release NPM Release

Branches

Branches Stability
main Stability
develop Stability
alpha Stability

Features

  • AWS SDK V2 and V3 unified support
  • Single-Table design pattern first class support
  • DataMapper development pattern
  • Attribute level per entity transformation, enabled via class-transformer
  • Full type safety
  • Declarative relational schema
  • Entity manager - easy to use findOne, find, count, exists, create, update, and delete operations
  • Transaction manager - easy to use write and read operations
  • Batch manager - powerful write and read operations
  • Scan Manager - powerful find, count, parallelScan and scan operations
  • Safer parallel scan with configurable concurrency control
  • Multiple connections support
  • Code follows all possible best practices when modeling for dynamodb
  • Supports specifying non key attribute as unique, built in pagination support for querying
  • Supports updating Primary key attributes
  • Auto Generated values for attributes
  • Auto update attribute values on UPDATE operations
  • Most type safe, powerful and flexible update expression generator engine
  • Dynamic & Static default values for attributes
  • Complex update, key condition and condition expression all made easy to work with
  • Powerful expression builder to auto generate expressions from input
  • Typescript and javascript support
  • Commonjs and ESM module support out of the box (see more here)

And many more to come.

Getting Started

Installation

  1. Install core and common modules from npm. npm install @typedorm/core @typedorm/common --save

  2. Install AWS SDK for nodejs, TypeDORM uses documentClient to interact with dynamodb. a. When using AWS SDK Version 2 npm install aws-sdk --save

    b. When using AWS SDK Version 3 npm install @aws-sdk/client-dynamodb @aws-sdk/lib-dynamodb --save

  3. Install reflect-metadata shim

    npm install reflect-metadata --save

    and import it as the first thing in node entry file as

    import 'reflect-metadata'

Typescript configuration

If you are using TypeDORM with typescript, make sure you also have below options enabled in tsconfig.json

"emitDecoratorMetadata": true,
"experimentalDecorators": true,

Developing with TypeDORM

Creating Table

First thing to do when working with TypeDORM is to setup dynamodb table config. Currently this needs to be manually setup and have also have it configured in deployed table instance(s).

This guide shows how to setup single-table-design

my-table.ts

import {Table, INDEX_TYPE} from '@typedorm/common';

// create table

const myGlobalTable = new Table({
  name: 'test-table',
  partitionKey: 'PK',
  sortKey: 'SK',
  indexes: {
    GSI1: {
      type: INDEX_TYPE.GSI,
      partitionKey: 'GSI1PK',
      sortKey: 'GSI1SK',
    },
    GSI2: {
      type: INDEX_TYPE.GSI,
      partitionKey: 'GSI2PK',
      sortKey: 'GSI2SK',
    },
    LSI1: {
      type: INDEX_TYPE.LSI,
      sortKey: 'LSI1SK',
    },
  },
});

Note: These indexes must match exactly to what is created in dynamo table instance hosted.

Creating an entity

organisation.entity.ts

import {Attribute, Entity, AutoGenerateAttribute} from '@typedorm/common';
import {AUTO_GENERATE_ATTRIBUTE_STRATEGY} from '@typedorm/common';

@Entity({
  name: 'organisation',
  primaryKey: {
    partitionKey: 'ORG#{{id}}',
    sortKey: 'ORG#{{id}}',
  },
  indexes: {
    // specify GSI1 key - "GSI1" named global secondary index needs to exist in above table declaration
    GSI1: {
      partitionKey: 'ORG#{{id}}#STATUS#{{status}}',
      sortKey: 'ORG#{{id}}#ACTIVE#{{active}}',
      type: INDEX_TYPE.GSI,
    },
    // specify LSI1 key
    LSI1: {
      sortKey: 'TICKETS#UPDATED_AT#{{updatedAt}}',
      type: INDEX_TYPE.LSI,
    },
  },
})
export class Organisation {
  @AutoGenerateAttribute({
    strategy: AUTO_GENERATE_ATTRIBUTE_STRATEGY.UUID4,
  })
  id: string;

  @Attribute()
  name: string;

  @Attribute()
  status: string;

  @Attribute()
  active: boolean;

  @AutoGenerateAttribute({
    strategy: AUTO_GENERATE_ATTRIBUTE_STRATEGY.EPOCH,
    autoUpdate: true, // this will make this attribute and any indexes referencing it auto update for any write operation
  })
  updatedAt: number;
}

Initialize default connection

The connection initialization steps slightly defers when using AWS SDK V2 vs V3.

When using AWS SDK V2
import {createConnection} from '@typedorm/core';
import {DocumentClientV2} from '@typedorm/document-client';
import AWS from 'aws-sdk';

const documentClient = new DocumentClientV2(new AWS.DynamoDB.DocumentClient());

// initialize with specifying list of entities
createConnection({
  table: myGlobalTable,
  entities: [Organisation],
  documentClient, // <-- When documentClient is not provided, TypeDORM defaults to use the DocumentClientV2
});

// or initialize with specifying path match for entities
createConnection({
  table: myGlobalTable,
  entities: 'path-to-entities/*.entity.ts',
  documentClient, // <-- When documentClient is not provided, TypeDORM defaults to use the DocumentClientV2
});
When using AWS SDK V3
import {createConnection} from '@typedorm/core';
import {DocumentClientV3} from '@typedorm/document-client';
import {DynamoDBClient} from '@aws-sdk/client-dynamodb';

const documentClient = new DocumentClientV3(new DynamoDBClient({}));

// initialize with specifying list of entities
createConnection({
  table: myGlobalTable,
  entities: [Organisation],
  documentClient, // <-- When documentClient is not provided, TypeDORM defaults to use the DocumentClientV2
});

// or initialize with specifying path match for entities
createConnection({
  table: myGlobalTable,
  entities: 'path-to-entities/*.entity.ts',
  documentClient, // <-- When documentClient is not provided, TypeDORM defaults to use the DocumentClientV2
});

Working with entity manager

import {getEntityManager} from '@typedorm/core';

const org = new Organisation();
org.name = 'My awesome org';
org.status = 'onboarding';
org.active = true;

const entityManger = getEntityManager();

// create item
const response = await entityManger.create(org);

// get item
const org = await entityManger.findOne(Organisation, {
  id: response.id,
  status: 'onboarding',
  active: true,
});

// delete item
await entityManger.delete(Organisation, {
  id: response.id,
  status: 'onboarding',
  active: true,
});

Table of contents

More

DynamoDB is different, different than most other no-sql databases, and therefore data in dynamodb should be stored the way dynamodb expects to get the most benefits out of it. While doing this development experience suffers and all data can become a huge mess very quickly, this is specially true with single-table-design patten. To resolve this, TypeDORM let's declaratively define schema and later takes control from there to provide best development experience possible.

To find out more about how the data looks like when it is stored in dynamo have a look at this detailed guide.

Current Limitations

  • TypeDORM, at the moment WILL NOT create/update table configuration and must be done separately by the developer.

Sponsors

Edge Tech Studio

Previous Sponsors

Nextfaze

Contributions

Please submit an issue for any bugs or ideas here, or you can reach out to me on twitter @whimzy_live.

typedorm's People

Contributors

aaronctech avatar aecorredor avatar apalchys avatar dependabot[bot] avatar derelk avatar jensbodal avatar kennydizi avatar lausek avatar m-szaf avatar meeech avatar michael-pont avatar mishabruml avatar semantic-release-bot avatar shawnwang-avl avatar slushnys avatar smcroskey avatar whimzylive avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

typedorm's Issues

Add support for entity inheritance

Ability to inherit classes, inherited class should be able to get all attributes of all parent classes

May be something like in typeorm at:
src/metadata-builder/MetadataUtils/getInheritanceTree

Initial batch read items support

Add batch read items support

Read support

  • add ReadBatch instance
  • add batch read input transformer
  • add support for read operations in batch manager
  • add test coverage for batch manger
  • update docs to list usage

circular peer-dependency: core <-> common

Hi there,

great project. Thank you very much.

I noticed that both packages ("core" and "common") list each other as a peerDependency. I am using serverless-bundle (https://github.com/AnomalyInnovations/serverless-bundle) to package my lambda functions and I have to list both "core" and "common" as "externals" to get typedorm to work after packaging (not 100% sure why that is, webpack seems to strip some necessary parts). However the circular dependency causes the bundler to go into an endless loop trying installing core, common, core, common and so on.
I can manually remove the peer dependencies and get it to work, but this is of course not a real solution.
What do you think about this? I can provide more details if needed.

Cheers
Paul

Add debug based logging

Add verbose logging backed by "debug" util to allow enabling/disabling logging based on environment variables

Parallel scan support

Ability to perform parallel scans when using scan manager.
This would require processing all requests in a parallel-like structure over the promise.all API.

Taks

  • scan transformer to support transforming parallel scan items
  • scan manager scan to support additional concurrency options and perform multiple requests in parallel using promise.all
  • scan manager's higher level .find and .count to facilitate running parallel requests
  • add test coverage
  • update docs reflecting support for parallel scans

Scan manager

Add scan manager and ability to run scan on tables

Tasks

  • add support for scan request transformation similarity to entity transformer
  • add support for scan response transformation
  • add scan manager low level scan operation
  • add scan manager find operation
  • add scan manager count operation
  • add test coverage
  • update API docs to include new scan manager operations

Improve generic types on public api

update generic types to be consistent throughout the codebase, and improve overall type-safe access.

Changes

  • fix Inconsistent generic type props
  • fix missing property types for entity manager update's body
  • fix missing attributes types when working with deep options
  • update docs to include path to migration to 1.12

entity manager projection support

Ability to specify projection expression when performing queries

Tasks

  • expression builder to support building projections
  • query item projections support
  • findOne item proections support
  • update docs to listing potential usage

Add Filter expression support to entity manager query

Add support for specifying filter expression when querying data using entity manager.

Tasks

  • add filter class type to allow easy filter expression expressions
  • add expressions parser to parse filter options to filter instance
  • expression builder to support creating filter expressions from filter instances
  • entity manager to support filter options when querying items
  • update api docs

Add extended support for enums in key schema

It is not possible to determine without the help of isEnum attribute to determine if the type of property is enum or object, but when one is not specified, do let users have object as property type but perform validation again when we have value (most likey when building schema), and throw appropriate error, indicating why operation failed.

Transaction manager: add support for read transactions

transaction manager to support read transaction

Tasks

  • Simplify Write transaction model
  • Add new document client transaction transcformer
  • Refactor transaction manger write to work operate similar to batch manager
  • Add support for transaction manager read support
  • Add test coverage
  • Update docs

Sync entities with real dynamodb tables

Each typedorm entity can be it's own table in dynamodb or can have one single table for all entitites.

Cli must be able to read config, and understand which design pattern the project is using and create/update tables and it's indexes based on configs provided.

initial batch write/read items support

Add initial support for batch write operations.

Tasks

Write support

  • add WriteBatch instance
  • add support for write operations in batch manager
  • add test coverage for batch manger
  • update docs to list usage

## Read support
- [ ] add ReadBatch instance
- [ ] add support for read operations in batch manager
- [ ] add test coverage for batch manger
- [ ] update docs to list usage
This is now tracked as a separate issue here #81

Add support for advanced condition and filter options

Currently, We do not support performing addition, subtraction on attributes when specifying a condition or update expression, the goal of this ticket is to add support for additional ADD and SUB operators to the expression builder.

One such use case is,

Update user's balance if it is not already zero.

Using document client's, we can write it like:

updateItem({
    TableName: "MyTable",
    Key: {
        "PK": "user#alexdebrie",
        "SK": "account#0123456789"
    },
    ConditionExpression='#balance - :amount > :zero',
    UpdateExpression='SET #balance = #balance - :amount',
    ExpressionAttributeNames={
        "#balance": "balance"
    },
    ExpressionAttributeValues={
        ":amount": { "N": "<amount of transaction>" },
        ":zero": { "N": "0" },
    }
})

this is currently not supported when using entity manager to perform updates.

Possible support

{
  where: {
     balance:  {  
       SUB: "<amount of transaction>", 
       GE: 0
     }
   }
}

Tasks

  • Add expression support for new arithmetic operators: ADD, SUB, MPY, DIV
  • update filter expression interface to accept new operators
  • update condition expression interface to accept new operators
  • update expression input parser to support parsing new operators
  • add test coverage
  • document usage

Add support for sparse indexes

Proposed

When parsing entity indexes, allow for sparse indexes

I.e
for Index with sortKey: PUB_ID#${id} do not auto parse, when index is marked as sparse index

Implementation

  • Add sparse index option on @entity -> indexes (ideally isSparse: boolean)
  • Allow optional values when parsing sparse marked key
  • If parsing an sparse index and referenced variable is missing value, DO NOT ADD partition or sort key attributes for that index

Missing table config throws non user friendly error

When building metadata for entity with missing table configuration, currently misleading error saying "TypeError: Cannot read property 'partitionKey' of undefined", this should be "Missing required table config" or something

Missing decorator metadata on building with esbuild

Decorator types are missing when trying to bundle using esbuild

Cannot read property 'name' of undefined.
at 
let type = Reflect.getMetadata('design:type', target, propertyKey).name;
on file attribute.decorator.js

Add support for modifying unique attributes

Allow modifying unique attributes

This will likely involve creating new item with updated values, remove old one as primary key can not be updated.

This should also be done as a single transaction

  • add support for updating unique items
  • add support for deleting unique items
  • update docs to cover this new changes

Add support using TypeDORM with explicit aws credentials

Currently TypeDORM can only be used on other services running on AWS that can access dynamodb using IAM roles, EC2s and Lambdas.

  • Add support for specifying aws credentials accessKeyId
    secretAccessKey when creating connection, and use provided credentials to connect to dynamodb table
  • Update docs with new connection options

Wrong level nesting with expressions using Logical operators

Generated expression are getting wrong level of nesting where it should be flat

i.e

AND : {
x: '11',
y: '22',
z: '33
}

should have been transformed into
(x = 11) AND (y=22) AND (z=33)

instead it is currently being transformed as
((x = 11) AND (y = 22)) AND (z=33)

Use modular AWS SDK for JavaScript (v3)

Is your feature request related to a problem? Please describe.

The modular AWS SDK for JavaScript (v3) was made generally available on December 15th, 2020. It also includes many frequently requested features, such as a modular packages, first-class TypeScript support and a new middleware stack.

TypeDORM uses AWS SDK for JavaScript (v2) which is expected to be deprecated.

Describe the solution you'd like

Use modular AWS SDK for JavaScript (v3) in TypeDORM.

Describe alternatives you've considered

Stick with AWS SDK for JavaScript (v2), as it's still fully supported and not deprecation date is announced.

Tasks

  • Add Dynamodb v3 document client
  • Add Document client adapter to be able to connect to v2 and v3 versions of document clients
  • Update Managers to call adapters instead of invoking document client methods directly
  • Update Tests to also use the adapter pattern
  • Verify that the introduction to v3 SDK support is non-breaking
  • Update readme explaining how to use SDK v3 client

Auto remove connection when failed to build metadatas for it

Creating connection with entities that are likely to fail to be build will put connection instance in an undesirable state, where connection exists in connection manger's context but but entities are not built.

If failed to build entity metadata for connection, connection should be removed from connection manger.

ConditionExpression support to `put` and `delete` operations

Similarly to Update operation,
add support for specifying conditions to put and delete operations.

Tasks

  • conditional put using the entity manager
  • conditional delete using the entity manager
  • conditional put using the transaction manager
  • conditional delete using the transaction manager
  • update docs

Add support for specifying default value for attribute decorator

@Attribute decorator to support additional option to provide default value, this can accept any default static value type or a function that returns a staic value.

  • handle incoming default value on decorator, (only scalar type supported atm)
  • when running create operation include all default values if they exist
  • if there is both default value and explicitly assigned value, later one should be used

Add support for managed entity count

Add support for storing and managing entity counts based on the specification provided.

DynamoDB does not have a way to retrieve a count of a given entity, but similar results can be achieved by maintaining a count property as a separate item in the table, there are many strategies to do this, like:

Tasks

  • entity decorator to support specifying managedCount as an option
  • when managedCount is enabled, auto-generate new managed count record
  • when modifying creating/deleting item with managed count enabled, also update managed count record to reflect correct values
  • add test coverage
  • document API and add section demoing the use case of managed count in recipes section

Add support for specifying ConditionExpression when performing `update`

Hi there,

I said it before but I will say it again: great library! Thank you very much.

I currently have a case where I need to pass in a ConditionExpression into an update call. As far as I can tell this is not yet possible, but should be easy to expose since you are already using ConditionExpression internally as documented here.

Any thoughts on this?

Add Step by step guide

  • Add document explaining step by step guide to get started with typeDORM
  • Fix broken links on sub packages readme

Add stats logger

Add stats logger to all manager, basically idea is to capture all operation cost (i.e consumed capacity)

Useful for getting insights on how queries are performing

Tasks

  • add stats logger
  • refactor existing logs to support printing unique request id
  • add stats logging to each responses
  • update docs to reflect new changes

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.