Comments (5)
For those who have googled this issue:
- You can use
pd.read_sql_query
because it doesn't emit first queryEXISTS ...
but only executes your query. - To keep first rows in place you have to add
FORMAT TabSeparatedWithNamesAndTypes
to your query explicitly.
from clickhouse-sqlalchemy.
Hi!
Can you provide more code including engine configuration and data that been inserted to rate
table?
from clickhouse-sqlalchemy.
I followed the tuturial on the main page...
Create the table
from sqlalchemy import create_engine, Column, MetaData, literal
from clickhouse_sqlalchemy import Table, make_session, get_declarative_base, types, engines
uri = 'clickhouse://default:@localhost/test'
engine = create_engine(uri)
session = make_session(engine)
metadata = MetaData(bind=engine)
Base = get_declarative_base(metadata=metadata)
class Rate(Base):
day = Column(types.Date, primary_key=True)
value = Column(types.Int32)
__table_args__ = (
engines.Memory(),
)
table = Rate.__table__
table.create()
insert some data
from datetime import date, timedelta
from sqlalchemy import func
today = date.today()
rates = [{'day': today - timedelta(i), 'value': 200 - i} for i in range(100)]
# Emits single INSERT statement.
session.execute(table.insert(), rates)
and read it with pandas
import pandas as pd
query = """
select day, value from rate limit 2;
"""
df = pd.read_sql(query, engine)
print(df)
let me know if you need anything else. Thanks!
from clickhouse-sqlalchemy.
Hi, @Marigold!
Sorry for late response.
Short answer: use native interface uri = 'clickhouse+native://default:@localhost/test'
and it'll be fine.
Long answer. Pandas read_sql
method actually emits two queries:
EXISTS TABLE select day, value from rate limit 2;
select day, value from rate limit 2;
There are two interfaces supported by clickhouse-sqlalchemy
:
- HTTP. It is the first one, but it has some limitations. For example we need to add
FORMAT TabSeparatedWithNamesAndTypes
for parsing on the python side. This interface is default now. This behavior will be changed in future releases. - TCP. This is the same interface used by
clickhouse-client
. This interface is more flexible. It appeared few moths after HTTP interface was implemented in this library.
In case of HTTP interface following SQL is rendered: EXISTS TABLE select day, value from rate limit 2; FORMAT TabSeparatedWithNamesAndTypes
. Trailing semicolon doesn't make any sense, query will fail even without it. See ClickHouse logs /var/log/clickhouse-server/clickhouse-server.log
. In read_sql
terms this table doesn't exist if EXISTS
query is failed.
You should use TCP interface. It hasn't problems with FORMAT clause.
from clickhouse-sqlalchemy.
Awesome response, thanks a lot for putting your time into this project!
from clickhouse-sqlalchemy.
Related Issues (20)
- Create a cluster table orm class with engines.Distributed which has a logs attribute, how to use variable to indicate it?
- alembic does not autogenerate engine for clickhouse table HOT 4
- Patreon does not work HOT 1
- Add support for Date32 HOT 2
- Handling Clickhouse Alembic Migrations for Clickhouse cluster HOT 2
- Support/example for creating views HOT 2
- `create_all` for all the `MaterializedView`
- With chdb this clickhouse downsized memory database, can clickhouse-sqlalchemy support it or not?
- Support VariantType
- Table reflection for DateTime64 timezone will be extra quoted
- Alembic_Version Records disappered after clickhouse alembic migration HOT 5
- using http mode, connecting database failed when account password ends with @ HOT 2
- Support nested Map column
- Cannot seem to run ALTER command on replicas of the same shard HOT 2
- Sqlalchemy can't catch asynch's error
- query_id cannot be set with the native TCP connection
- Feature Request: Support clickhouse-connect's NEW AsyncClient wrapper HOT 5
- Nested maps, tuples, enums don't work HOT 2
- Support Lightweight DELETE Statement HOT 2
- Table metadata fails to reflect if is_deleted column is set along with version in ReplacingMergeTree engine
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from clickhouse-sqlalchemy.