Coder Social home page Coder Social logo

passren / pydynamodb Goto Github PK

View Code? Open in Web Editor NEW
16.0 1.0 3.0 318 KB

PyDynamoDB is a Python DB API 2.0 (PEP 249) client for Amazon DynamoDB. A SQLAlchemy dialect is offered as well. Superset official database driver.

License: Other

Python 100.00%

pydynamodb's Introduction

badge package badge test badge downloads badge formation badge codcov badge sqrelia badge sqale badge sqvuln

PyDynamoDB

PyDynamoDB is a Python DB API 2.0 (PEP 249) client for Amazon DynamoDB. SQLAlchemy dialect supported as well.

Objectives

PyDynamoDB implement the DB API 2.0 interfaces based on PartiQL supported by AWS DynamoDB. Although PartiQL can only support DML operations (INSERT, UPDATE, DELETE, SELECT), PyDynamoDB extended the capabilities to support DDL as well. Now you are able to use MySQL-like statements to CREATE/ALTER/DROP tables. Besides DDL statements, some of utility statements are allowed to execute (Such as, List and Describe Table). PyDynamodb provide parameters and result_set converter to make you easily manipulate operations with Python built-in types. Transaction is also partially supported with DB standard operations, like begin() and commit().

Features

  • Compatible with DB API 2.0 Specification
  • PartiQL for DML operations (INSERT, UPDATE, DELETE, SELECT)
  • Limit supported in SELECT statement
  • Extra type conversion and string functions supported in SELECT statement
  • Column alias supported in SELECT statement
  • MySQL-Like statements for DDL operations (CREATE TABLE, ALTER TABLE, DROP TABLE)
  • MySQL-Like statements for Utility operations (LIST/SHOW TABLES, DESC TABLE)
  • Auto data type conversion for parameters and result set (Including date and datetime)
  • Transaction and Batch operations
  • SQLAlchemy dialect provided
  • Compatible for Superset SQL Lab and graphing

Requirements

  • Python
    • CPython 3.8 3.9 3.10 3.11 3.12

Dependencies

  • Boto3 (Python SDK for AWS Services)
    • boto3 >= 1.21.0
    • botocore >= 1.24.7
  • Tenacity (Retry Utility for API calling)
    • tenacity >= 4.1.0
  • SQLAlchemy (The ORM Toolkit for Python, only required if using PyDynamoDB Dialect)
    • SQLAlchemy >= 1.0.0, < 2.0.0
  • Pyparsing (The approach to creating and executing simple grammars)
    • pyparsing >= 3.0.0

Installation

pip install pydynamodb

Guidances

To get more documentation, please visit: PyDynamoDB WIKI.

Basic usage

from pydynamodb import connect

cursor = connect(aws_access_key_id="aws_access_key_id",
                aws_secret_access_key="aws_secret_access_key",
                region_name="region_name").cursor()
cursor.execute('SELECT * FROM "ddb_table_name"')
print(cursor.fetchall())

Cursor iteration

from pydynamodb import connect

cursor = connect(aws_access_key_id="aws_access_key_id",
                aws_secret_access_key="aws_secret_access_key",
                region_name="region_name").cursor()
cursor.execute('SELECT * FROM "ddb_table_name"')
rows = cursor.fetchall()
for row in rows:
    print(row)

Query with parameters

PyDynamoDB is able to serialize the parameters which passed to DDB and deserialize the response to Python built-in types.

from pydynamodb import connect
from datetime import date, datetime
cursor = connect(aws_access_key_id="aws_access_key_id",
                aws_secret_access_key="aws_secret_access_key",
                region_name="region_name").cursor()
cursor.execute("""INSERT INTO "ddb_table_name" VALUE {
                    'partition_key' = ?, 'sort_key' = ?, 'col_str' = ?,
                    'col_num' = ?, 'col_byte' = ?, 'col_ss' = ?,
                    'col_ns' = ?, 'col_bs' = ?, 'col_list' = ?,
                    'col_map' = ?, 'col_nested' = ?,
                    'col_date' = ?, 'col_datetime' = ?
                }""", ["pkey_value", "skey_value", "str", 100, b"ABC", # String, Number, Bytes
                        {"str", "str"}, {100, 100}, {b"A", b"B"}, # String/Numnber/Bytes Set
                        ["str", 100, b"ABC"],  # List
                        {"key1": "val", "key2": "val"}, # Map
                        ["str", 100, {"key1": "val"}], # Nested Structure
                        date(2022, 10, 18), datetime(2022, 10, 18, 13, 55, 34), # Date and Datetime Type
                    ])

cursor.execute('SELECT * FROM "ddb_table_name" WHERE partition_key = ?', ["key_value"])
print(cursor.fetchall())

License

PyDynamoDB is distributed under the MIT license.

pydynamodb's People

Contributors

passren avatar qqq-33 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

pydynamodb's Issues

Statement wasn't well formed, can't be processed: Unexpected keyword

Hi there,

I'm using pydynamodb with sqlalchemy in order to convert a simple app which previously used sqlite.

I made the change to the connection string, and I see that pynamodb can connect using my credentials to the dynamodb table.
But when inserting a record I get the error below:

sqlalchemy.exc.OperationalError: (pydynamodb.error.OperationalError) An error occurred (ValidationException) when calling the ExecuteStatement operation: Statement wasn't well formed, can't be processed: Unexpected keyword
[SQL: INSERT INTO urls (key, secret_key, target_url, is_active, clicks) VALUES (?, ?, ?, ?, ?)]
[parameters: ('KSCBE', 'YMYXBEAP', 'http://globo.com', None, None)]
(Background on this error at: https://sqlalche.me/e/14/e3q8)

This is my database model:


class URL(Base):
    __tablename__ = "urls"

    id = Column(Integer, primary_key=True)
    key = Column(String)
    secret_key = Column(String)
    target_url = Column(String)
    is_active = Column(Boolean)
    clicks = Column(Integer)

Any advice?
Thanks in advance.

How to SELECT a field in a DynamoDB map item?

This works in the DynamoDB PartiQL editor:

SELECT payload."size"
FROM "table"
WHERE pk='foo'

But when I do:

cursor.execute('SELECT payload."size" FROM "table" WHERE pk='foo')

I get this error:

dynamodb error: An error occurred (ValidationException) when calling the ExecuteStatement operation: Statement wasn't well formed, can't be processed: Invalid path dot component

An example of what my data looks like:

[
  {
    'pk': foo',
    'payload': {'size': 2, 'bla': 'bla'},
  },
  {
    'pk': 'bar',
    'payload': {'size': 2, 'bla': 'bla'},
  }
]

How may I go about selecting size from payload?

Note that I intend to use DynamoDB in Apache Superset. Was getting the same error there in the SQL Lab section. Digging around their repo led me here. Installed the package and ran the same query to confirm it wasn't a Superset thing.

Add support for iam authentication

At present, connecting superset to DynamoDB via pyDynamoDB requires us to specify access_key and secret_key that adds to operational maintenance. It would be great if the driver can take iam credentials from the environment. This can be kept behind simple boolean flag similar to what pyathena or redshift-connector already provide. Thanks.

WHERE clauses are not pushed down to PartiQL?

Thank you for making this library!

If I issue:

SELECT * FROM "cache" WHERE "my-value" != ''

to both boto3 directly (pure PartiQL) and to pydynamodb, pydynamodb never returns (the table is large).

I see that the statement it pushes down is SELECT * FROM "cache".

I am guessing it is eagerly pre-fetching data and locally evaluating the WHERE.

I would expect the entire SELECT statement to be pushed down as it is to PartiQL, which is able to handle this.

Support for SQLAlchemy v2+

Hello! I was planning on using this library in conjunction with SQLModel, and SQLModel requires SQLAlchemy v2. During some initial work when using 0.6.2 of PyDynamoDB and SQLAlchemy 2.0.31, I found some errors that look to be related to implementing v1 (see below). Do you have plans to support SQLAlchemy v2? Or is this just a simple bug?

Error:

Traceback (most recent call last):
  File "/Users/mkkatica/Documents/Workspace/pydantic-apigateway/pydantic_apigateway/api/api.py", line 52, in p
    session.bulk_save_objects([item])
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4574, in bulk_save_objects
    self._bulk_save_mappings(
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4751, in _bulk_save_mappings
    with util.safe_reraise():
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 146, in __exit__
    raise exc_value.with_traceback(exc_tb)
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 4740, in _bulk_save_mappings
    bulk_persistence._bulk_insert(
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/orm/bulk_persistence.py", line 200, in _bulk_insert
    result = persistence._emit_insert_statements(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", line 1048, in _emit_insert_statements
    result = connection.execute(
             ^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1418, in execute
    return meth(
           ^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 515, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1632, in _execute_clauseelement
    compiled_sql, extracted_params, cache_hit = elem._compile_w_cache(
                                                ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 703, in _compile_w_cache
    compiled_sql = self._compiler(
                   ^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 316, in _compiler
    return dialect.statement_compiler(dialect, self, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/compiler.py", line 1429, in __init__
    Compiled.__init__(self, dialect, statement, **kwargs)
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/compiler.py", line 870, in __init__
    self.string = self.process(self.statement, **compile_kwargs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/compiler.py", line 915, in process
    return obj._compiler_dispatch(self, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/sqlalchemy/sql/visitors.py", line 141, in _compiler_dispatch
    return meth(self, **kw)  # type: ignore  # noqa: E501
           ^^^^^^^^^^^^^^^^
  File "/Users/mkkatica/.pyenv/versions/pydantic-apigateway/lib/python3.12/site-packages/pydynamodb/sqlalchemy_dynamodb/pydynamodb.py", line 111, in visit_insert
    crud_params = crud._get_crud_params(self, insert_stmt, compile_state, **kw)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: _get_crud_params() missing 1 required positional argument: 'toplevel'

Query Error in Superset with PyDynamoDB

I've connected Superset to DynamoDB using the PyDynamoDB driver. However, when I try to query DynamoDB through Superset, I find that it lacks several functions such as "split" and "replace," which prevents me from querying the tables as needed.

For example, I have a table with an "id" column that combines the database and table names. I need to split this column to get the database and table names separately.

Eg query: "SELECT substring(id,1,position('.' in id)-1) id1 FROM "my-table";"
I am getting an error: Amazon DynamoDB Error
dynamodb error: Expected CaselessKeyword 'FROM', found '(' (at char 16), (line:1, col:17)

I am following the guidance provided in this document: https://github.com/passren/PyDynamoDB/wiki/5.-Superset

Can someone please help me with this?

contains() or begins_with() don't work.

  1. contains() and begins_with() are not available in PartiQL [SELECT statements].

The syntax below is executable.
SELECT * FROM UserTB WHERE user_name = 'tes0125' Limit 2

contains() or begins_with() don't work. [SELECT statements]
SELECT * FROM UserTB WHERE user_name = 'tes0125' AND begins_with(range_key, 'FRIEND#') Limit 2

  1. Is there any way to handle paging?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.