DuckDB website and documentation
Home Page: https://duckdb.org
License: MIT License
This repository hosts the source code for the DuckDB website. Please file any questions or issues relating to the website or documentation here.
The DuckDB codebase is hosted in the DuckDB repository.
To build the site with Jekyll (installed locally or run via Docker), check out our site build guide.
Please consult the contributor's guide for instructions on how to contribute to the documentation.
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot]")
In the context of #123 we have rewritten the history of this repository to remove (many) redundant large files. This has reduced the size of this repo from 1.3GB to 75MB. However, if you have an outstanding repository cloned or forked the old history will still be present there. It is recommended that you re-clone or re-fork this repository to prevent that from causing problems.
I had some questions which I don't think are answered by the docs. The "Data Dypes Overview" briefly mentions ROW, MAP, and ARRAY, but not LIST and STRUCT, which are only mentioned in the "Nested Types" page.
ARRAY, INT[], and LIST?INT[3])? Are these limits respected?LIST or ARRAY specified in a DDL statement?LIST expressions ([1, 2, 3]) be used to insert into ARRAYs?STRUCT, ROW, and MAP?STRUCT expressions ({'foo': ...}) be used to insert into MAPs or ROWs?Presumably they work like PG or standard SQL, but the docs could be expanded.
<=, >= and != looks like unicode ≤, ≥ and ≠.
consider turning off ligatures or using a font without them.
Now:
With font-variant-ligatures: none;:
System detected seems to be hardcoded in the links:
On Windows 10 Firefox 86.0a1
Similarly on Edge 87.0.664.66
so that we can find the document of a feature or function easily.
At https://duckdb.org/docs/sql/functions/dateformat, the headings are missing padding.
I want to test nest query by duckdb. First I use create and insert to create a table with a map filed.
Then I want to query map data. But I didn't see some example in the documents, I trid many times, but not worked. So someone can help me ?
import duckdb
import time
if __name__ == "__main__":
con = duckdb.connect()
start = time.time()
con.execute("create table mcule(id INTEGER, map_col MAP(VARCHAR ,VARCHAR ))")
con.execute("insert into mcule VALUES (1,map(['asia'],['asdfa']))")
#con.execute("select * from mcule where map(map_col)(['asia'],['asdfa']"))
# con.execute("select element_at(map_col,['asia'])")
# print(con.fetchall())
# con.execute("copy (select * from mcule) to 'nesttest.csv' (FORMAT 'CSV')")
# con.execute("select people.name from 'nesttest.parquet'")
con.execute("select * from mcule")
print(con.fetchall())
end = time.time()
print("test time: " + str(end - start))
error
Traceback (most recent call last):
File "createNesTTest.py", line 12, in <module>
con.execute("select * from mcule where map(map_col)['asia']='asdfa'")
RuntimeError: We need exactly two lists for a map
Traceback (most recent call last):
File "createNesTTest.py", line 12, in <module>
con.execute("select * from mcule where map(['asia'],['asdfa'])")
RuntimeError: Conversion Error: Unimplemented type for cast (MAP<VARCHAR, VARCHAR> -> BOOLEAN)
I suspect the check for underlining should be stricter.
I tried running "SELECT distinct l_shipinstruct FROM lineitem" and "SELECT distinct l_shipinstruct FROM lineitem order by 1 desc" on the https://duckdb.org/demo/ website and I either get an incorrectly rendered result or HTTP 503 errors.
But the query does work occasionally.
Request URL: https://duckdbdemo.project.cwi.nl//fetch?callback=jQuery3510355319707602344_1629874848617&ref=53knwZQ5mH&_=1629874848637
Request Method: GET
Status Code: 502 Proxy Error
Request URL: https://duckdbdemo.project.cwi.nl//query?callback=jQuery351010110534406317129_1629874584547&q=SELECT%20distinct%20l_shipinstruct%20FROM%20lineitem%20order%20by%201%20desc&_=1629874584573
Request Method: GET
Status Code: 503 Service Temporarily Unavailable
I wanted to build a central list of documentation tasks in order to prioritize them. I've tied in existing issues where applicable. Feedback is welcome! Please let me know what I forgot or if the order of importance should be changed.
Tactical items:
Walkthroughs / tutorials (These could be blog posts maybe?)
Larger items:
https://duckdb.org/docs/sql/functions/nested
Map Functions
| Function | Description | Example | Result | |:—|:—|:—|:—| | map[entry] | Alias for element_at | map([100, 5], ['a', 'b'])[100] | 42 | | element_at(map, key) | Return a list containing the value for a given key or an empty list if the key is not contained in the map. The type of the key provided in the second parameter must match the type of the map’s keys else an error is returned. | SELECT element_at(map([100, 5], [42, 43]),100); | 42 | | cardinality(map) | Return the size of the map (or the number of entries in the map). | cardinality( map([4, 2], ['a', 'b']) ); | 2 | | map() | Returns an empty map. | map() | {} |
We are also missing some of the newer keywords like Qualify and Having. Is there a good way to pull a list of keywords with a DuckDB script where we could build a DuckDB-specific dialect for CodeMirror?
Thanks!
Hi folks! Thank you for taking documentation seriously. Reading through the docs has helped onboard me to duckdb very quickly.
One thing I noticed: INTERVAL functions are documented by INTERVAL isn't. Could be that the function defs are enough to get started but it took me a Github issue discussion & looking at the source to understand the data type.
Hey folks!
Before I write up a how-to-guide, would you mind taking a look at this approach to using Python threads? Is this the best practice? I couldn't get it to work with cursors, so if that is a better method than not checking the same thread, I'm open to changing this!
Thanks!
import duckdb
from threading import Thread, current_thread
import pandas as pd
def insert_from_thread(duckdb_con, results_df_dict):
# Insert a row with the name of the thread
thread_name = str(current_thread().name)
results_df_dict[thread_name] = duckdb_con.execute("""INSERT INTO my_inserts VALUES (?)""", (thread_name,)).df()
duckdb_con = duckdb.connect(check_same_thread=False) # In Memory DuckDB
duckdb_con.execute("""CREATE OR REPLACE TABLE my_inserts (thread_name varchar)""")
thread_count = 10
threads = []
results_df_dict = {}
# Kick off multiple threads (in the same process)
# Pass in the same connection as an argument, and an object to store the results
for i in range(thread_count):
threads.append(Thread(target=insert_from_thread,
args=(duckdb_con, results_df_dict,),
name='my_thread_'+str(i)))
for i in range(thread_count):
threads[i].start()
for i in range(thread_count):
threads[i].join()
print(results_df_dict)
print(duckdb_con.execute("""SELECT * FROM my_inserts""").df())I just wanted to document this here before I forgot! I'm happy to make this change.
I think that the UnNest function should be mentioned in the Nested types section in addition to the SELECT overview.
In general, I think there might be a few other Postgres functions that aren't in the docs just yet, but we can track those in separate issues.
Hi,
When can we expect the documentation notes for the bellow link??
Feedback to #117
The DuckDB CLI has quite a few features (e.g. .tables) which are not yet documented.
I started writing this earlier this year but realized it's a larger task and abandoned it. I got this far:
## Installation
DuckDB can be installed as a binary. Please see the [installation page](/docs/installation?environment=cli) for details.
Other than SQL commands, SQLite-like instructions can be used:
.help
* `.tables`: Print a list of tables
## History
The command history is saved in the home directory in `.duckdb_history`.
Should be like
to_milliseconds(integer) | Construct a millisecond interval | to_milliseconds(5) | INTERVAL 5 MILLISECOND
to_microseconds(integer) | Construct a microsecond interval | to_microseconds(5) | INTERVAL 5 MICROSECOND
The benchmark pages are showing as blank for me. I am getting a 503 (Service Temporarily Unavailable) from the duckdbdemo.project.cwi.nl calls.
just a tip: no real need to build a cli, you can reuse sqlline if required so.
I understand the integration will not be as tight as possible, but it works, and enables one to use the same cli for all kind of databases:
combi of DuckDB JDBC driver and sqlline
see: https://duckdb.org/docs/data/parquet, and sqlline https://github.com/julianhyde/sqlline
sqlline -u "jdbc:duckdb:" -d "org.duckdb.DuckDBDriver" -n '' -p '' -e "select * from 'userdata1.parquet';"
I can not find any description on what a TEMP or TEMPORARY table is in CREATE TEMP TABLE... statements. Would be helpful with a brief description on this page https://duckdb.org/docs/sql/statements/create_table
One use case I have is that I consider it for "caching" aggregates:
create tmp as select user, count(*) v ... and then run queries like this select city, sum(v) v ... from tmp.
Is it materialised in memory? What if it does not fit in memory?
Is it removed automatically? When? How?
What is the intended use case for temporary tables?
imgbot automatically sends PRs to optimize images and compress them significantly. Could be useful for blog posts.
See for example cmudig/cmudig.github.io#160
such as encode, decode and COALESCE, or add a catalog misc for those functions.
I found this guide on writing good documentation. It essentially advocates for splitting documentation into four distinct groups:
Most of the documentation we have is in the form of Reference, which, while useful, is perhaps not sufficient particularly for beginners.
What do you guys think about splitting the documentation into three separate sections -- Tutorials, How-To Guides and Reference. Most of the current material would go under the Reference section, and we could write up a few language-specific tutorials for getting started and how-to guides for accomplishing common tasks in each of these.
Thoughts?
The node module isn't mentioned in the documentation. I totally get that it's not as much a "data friendly" language as Python and R but there is a large class of data apps that could leverage node in the future. At the minimum a link to the GH readme would suffice.
We should add some documentation on how to import data from other database systems.
e.g. sqlite -> duckdb, postgres -> duckdb, etc
Hey Folks!
It looks like the docs are still showing 0.3.3 as the latest release. How can I upgrade that to 0.3.4?
I searched through this repo for a way to increment the docs to the next version, but I couldn't find any past PR's that showed it.
Thanks!
-Alex
Hello Duck DB team,
I really like the them and I would like to use it in order to build a website of my open source project. I tried to find license notes but I couldn't find any in this repository.
Do you think that I can use this project as boilerplate for my website where I will use only the layout but not the content (text and images).
If yes, then I can perhaps add a link to the original them,
Hey folks!
On Discord, @hannes mentioned the storage_info pragma as a way to get the storage footprint of a table:
PRAGMA storage_info('my_table');The columns provide a ton of info:
row_group_id,
column_name,
column_id,
column_path,
segment_id,
segment_type,
start,
count,
compression,
stats,
has_updates,
persistent,
block_id,
block_offset
I'm looking to calculate the storage size of a table, so I assume that means summing count here.
This would be an awesome pragma to document if possible! Very useful for those of us telling the machine to do a lot of big calculations in DuckDB.
Feedback to #117
The DESCRIBE clause is mentioned in the documentation but the command itself is not documented.
Example:
D create table t(x int primary key, y varchar);
D describe t;
┌───────┬─────────┬──────┬─────┬─────────┬───────┐
│ Field │ Type │ Null │ Key │ Default │ Extra │
├───────┼─────────┼──────┼─────┼─────────┼───────┤
│ x │ INTEGER │ NO │ │ │ │
│ y │ VARCHAR │ YES │ │ │ │
└───────┴─────────┴──────┴─────┴─────────┴───────┘
(I believe the output above is not correct: x should be a Key)
A while ago, I contributed the examples for WITH RECURSIVE (#158 #188)
https://duckdb.org/docs/sql/query_syntax/with
I have since found that these queries are quite inefficient, especially the bidirectional search if there is no path between the start and end nodes.
It would also want to add a proper bidirectional search algorithm, where the two BFS frontiers are advanced simultaneously from the start and the end node.
I don't have time to tackle this now, so I'm opening this issue as a reminder and will get back to it later this spring.
Perhaps this is nothing more than my mis-reading of the website documentation, but I have a collection of parquet files that are saved with a '.parq' extension. It was only after significantly banging my head against the wall that I realized that Duck would not identify the extension as valid and was failing to interrogate the files.
con.execute(
"""
select *
from '/abs_path/2018_01_02.parq'
limit 10
""").fetch_df()Returns RuntimeError: Catalog Error: Table with name /abs_path/2018_01_02.parq does not exist! LINE 3: from '/abs_path/2018_01_02.parq'
If I rename a file to .parquet extension, everything works as intended. This failure happened with both specific filenames and the glob syntax.
Not sure how common .parq is in the wild and if you need to support this, but it would be nice if the full rules for identifying queryable file extensions was more clear.
The instructions for R at https://duckdb.org/docs/installation/ display:
install.packages("https://github.com/duckdb/duckdb/releases/download/master-builds/duckdb_r_src.tar.gz", repos = NULL)
which does not work because https://github.com/duckdb/duckdb/releases/download/master-builds/duckdb_r_src.tar.gz returns a 404 error.
See https://github.com/DavidAnson/markdownlint for more info
This could be enabled as a Github action for PRs
List functions such as unnest and string_split_regex are currently undocumented. An example for their use:
D insert into emails values (1, '[email protected];[email protected]'), (2, '[email protected]'), (3, '');
Error: Catalog Error: Table with name emails does not exist!
Did you mean "x"?
D create table emails(id int, addresses varchar);
D insert into emails values (1, '[email protected];[email protected]'), (2, '[email protected]'), (3, '');
D select id, unnest(string_split_regex(addresses, ';')) from emails;
D select id, unnest(string_split_regex(addresses, ';')) as email from emails;
┌────┬─────────┐
│ id │ email │
├────┼─────────┤
│ 1 │ a@b.com │
│ 1 │ c@d.com │
│ 2 │ e@f.com │
│ 3 │ │
└────┴─────────┘@Alex-Monahan Currently, the size of this repository is massive, approx. 1.2 GB. Even when cloning just the last commit, the resulting directory is 81 MB:
$ git clone --depth 1 [email protected]:duckdb/duckdb-web.git
$ du -hd0 duckdb-web
81M duckdb-web
Here's a list of the top 50k largest files committed, generated with this script.
To nuke old commits, the BFG repository cleaner can be used. This will still require a force push and thus break the commit tree but there are not too many forks of this repository yet.
The QUALIFY clause is missing in the documentation
In the Benchmarks page, the stdout logs are empty.
The link in the Profiling page is broken, it points to:
The live demo on https://duckdb.org/demo/ appears to be down, the Firefox browser complains about
Loading failed for the <script> with source “https://duckdbdemo.project.cwi.nl//query?callback=jQuery35104049208308842104_1628770206616&q=SELECT%20*%20FROM%20lineitem&_=1628770206618”.
Chromium has a problem resolving the hostname duckdbdemo.project.cwi.nl... DNS issue?
This would simplify navigation, as no link is visible on the start page, neither README nor elsewhere.
See duckdb/duckdb#2551
I'm "self-assigning" this since I'm working with recursive CTEs quite a lot.
I read in some Github issue that a simple SELECT query is not parallelized but that by adding ORDER BY one can workaround this limitation.
I think it'd be great if this trick was documented to help discoverability. Unfortunately I can't find the Github issue to link to, but this trick definitely works.
The jekyll docs mention a home grown solution https://jekyllrb.com/tutorials/convert-site-to-jekyll/#10-rss-feed, but also link to: https://github.com/jekyll/jekyll-feed
I found the 'efficient SQL on pandas' blog post when someone posted it on Reddit. When I wanted to go back to it I could not find any links on the website that will take me to the blogs. If you want people to read the blog posts then I suggest you put a link to them on the home page.
We support MAKE_DATE, MAKE_TIME and MAKE_TIMESTAMP in the main code.
Hi,
Thanks for an awesome package. Just a small typo that confused me more than it should as I was copy+pasting from examples:
https://duckdb.org/docs/sql/window_functions
SELECT points,
SUM(points) OVER (
ROWS BETWEEN 1 PREDECING
AND 1 FOLLOWING) we
FROM results
"PREDECING" should be "PRECEDING"
Is it possible at the moment to ignore nulls while computing some aggregations like count, first/last value etc. in window functions like Redshift does?
Thanks for the great work btw 🙏
Update minvalue/maxvalue once duckdb/duckdb#1056 has been resolved.
See discussion in #13 (comment)
please add a detailed compile guide for so/dll file and program that calling the functions in so file.
something like this https://sqlite.org/howtocompile.html
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.