crashed when do insertorupdate becasue of assert in a large db about duckdb

[Tishj]

Feel free to send it to me at [email protected], using something like wetransfer.com should work Note that I am making an exception this one time

hi..I’m wondering if you received my email last week

I replied and asked you to send it through another service, Baidu is not something we can use, can you use SendBig? That should be available in china as well

[duckdblabs-bot]

Thanks for opening this issue! Based on our automated check, it seems that your post contains some code but it does not use code blocks to format it.

Please double-check your post and revise it if necessary. To employ syntax highlighting, it's recommended to use code blocks with triple backticks, e.g.:

```sql
SELECT ...
```

If this is a false positive, feel free to disregard this comment.

[xubobbs]

void UncompressedStringStorage::StringScanPartial(ColumnSegment &segment, ColumnScanState &state, idx_t scan_count,
Vector &result, idx_t result_offset) {
// clear any previously locked buffers and get the primary buffer handle
auto &scan_state = state.scan_state->Cast();
auto start = segment.GetRelativeIndex(state.row_index);

auto baseptr = scan_state.handle.Ptr() + segment.GetBlockOffset();
auto dict = GetDictionary(segment, scan_state.handle);
auto base_data = reinterpret_cast<int32_t *>(baseptr + DICTIONARY_HEADER_SIZE);
auto result_data = FlatVector::GetData<string_t>(result);

int32_t previous_offset = start > 0 ? base_data[start - 1] : 0;

for (idx_t i = 0; i < scan_count; i++) {
	// std::abs used since offsets can be negative to indicate big strings
	uint32_t string_length = std::abs(base_data[start + i]) - std::abs(previous_offset);
	result_data[result_offset + i] =
	    FetchStringFromDict(segment, dict, result, baseptr, base_data[start + i], string_length);

-->here I dump the debug:
scan_count
105
string_length
925352919
segment
{...}
duckdb::SegmentBaseduckdb::ColumnSegment (base):
duckdb::SegmentBaseduckdb::ColumnSegment
db:
{...}
type
type_size:
16
segment_type:
duckdb::PERSISTENT
function
stats
block:
std::shared_ptr (count 15, weak 1) 0x808248
block_id:
2452
offset:
180912
segment_size:
262136
segment_state

[xubobbs]

.schema dsmock_group_tree
CREATE TABLE dsmock_group_tree(_id VARCHAR PRIMARY KEY, tree_id VARCHAR NOT NULL DEFAULT(''), org_id BIGINT NOT NULL DEFAULT(0), asset_id BIGINT NOT NULL DEFAULT(0), updated_at BIGINT NOT NULL DEFAULT(0), deleted_at BIGINT NOT NULL DEFAULT(0));
select count(1) from dsmock_group_tree; it work ok and got result:
┌──────────┐
│ count(1) │
│ int64 │
├──────────┤
│ 105 │
└──────────┘
select org_id from dsmock_group_tree ; it ok
select asset_id from dsmock_group_tree ;it ok
but select tree_id from dsmock_group_tree it crashed by the assert

[Tishj]

Please make a reproducable example, I have extracted the following queries from your report, with those I can't reproduce the issue, please adjust where necessary

statement ok
CREATE TABLE dsmock_group_tree(
    _id VARCHAR PRIMARY KEY,
    tree_id VARCHAR NOT NULL DEFAULT(''),
    org_id BIGINT NOT NULL DEFAULT(0),
    asset_id BIGINT NOT NULL DEFAULT(0),
    updated_at BIGINT NOT NULL DEFAULT(0),
    deleted_at BIGINT NOT NULL DEFAULT(0)
)

statement ok
INSERT INTO dsmock_group_tree (
	_id,
	tree_id,
	org_id,
	asset_id,
	updated_at,
	deleted_at
) VALUES (
	'1492bbd9-6a8a-4f62-9751-71ad1afc4828',
	'9fd27437-887b-424b-8f61-9eb8fd9607e0',
	276195621352,
	276195621352,
	1709260345874609,
	0
)
ON CONFLICT (
	_id
) DO UPDATE SET
	tree_id = EXCLUDED.tree_id,
	org_id = EXCLUDED.org_id,
	asset_id = EXCLUDED.asset_id,
	updated_at = EXCLUDED.updated_at,
	deleted_at = EXCLUDED.deleted_at

(note statement ok is syntax for our unittester, you can disregard those)

[Tishj]

You're also welcome to provide the db file if that's the only way you can make this reproducable.

[xubobbs]

I can give you my test db .but it is over 1G size .how can I send it to you ?

[xubobbs]

I test I can drop some table,and it still keep crash when i use the test sql .hope the total file size can get smaller

[xubobbs]

this file is still 800M ... how can I send it to you?

[Tishj]

Feel free to send it to me at [email protected], using something like wetransfer.com should work
Note that I am making an exception this one time

[xubobbs]

got it and send the downlink link mail to you

[xubobbs]

Feel free to send it to me at [email protected], using something like wetransfer.com should work Note that I am making an exception this one time

hi..I’m wondering if you received my email last week

[xubobbs]

has send db to you by sendbig .thanks

[Tishj]

I received the file and can indeed reproduce the issue, but since the error occurs in decompression, there's too little information for us to work with.

Please let us know if you can make a minimal reproduction of steps that can cause this error to occur when performed on a clean database

[xubobbs]

maybe I need do a test demo for reproduce this .because I also don't know what happend before this issue.but in my test env , it happened twice ..

…

----- 原始邮件 ----- 发件人：Thijs ***@***.***> 收件人：duckdb/duckdb ***@***.***> 抄送人：xubobbs ***@***.***>, Author ***@***.***> 主题：Re: [duckdb/duckdb] crashed when do insertorupdate becasue of assert in a large db (Issue #11124) 日期：2024年03月20日 21点36分 I received the file and can indeed reproduce the issue, but since the error occurs in decompression, there's too little information for us to work with. Please let us know if you can make a minimal reproduction of steps that can cause this error to occur when performed on a clean database — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>