siafoundation / renterd Goto Github PK
View Code? Open in Web Editor NEWA renter for Sia
Home Page: https://sia.tech/software/renterd
License: MIT License
A renter for Sia
Home Page: https://sia.tech/software/renterd
License: MIT License
I would like to be able to configure where the logs file is stored independent of the -dir
parameter.
Would like to see stats that show the average upload, download, storage, and contract prices from active contracts/hosts. It should also show the number of active contracts, in real time, ideally as a simple number, as well as number of active and usable hosts.
Should also be able to select based on date/time ranges.
Overall should be a simple analytics dashboard, probably as a dedicated page in the left-hand menu.
I have a couple of slabs that has been failing to migrate for a few days now, due to excessive fiddling on my part. I've decided to reupload those files now, since the hosts that has the slabs doesn't seem to be coming online again. At this point that requires me to hunt in the db for the slab, then the slice with that slab to find the object id, then the filename in objects
. It would be nice to view which files has been failing to migrate, maybe with an alert, for x hours. Also, when looking up the files in the ui now it says they have "excellent health" with a green check mark, which isn't true.
How to reproduce:
Can't download or remove file from webui. Not tested via api.
It only applies to files that are in a directory that contains those characters, files with name like "testśśś.iso" works fine.
Being tired of my renterd
node falling out of sync every time my PC goes to the sleep mode, I came across an idea. renterd
could keep track of the last time it had received blocks from the peers, and if it exceeded a certain limit (say, 1 hour == 6 blocks), it would then consider itself out of sync.
As the block time fluctuates in Sia network, we often have blocks longer than 10 min, but very rarely longer than 60 min.
Right after my PC wakes up, file uploads and downloads fail, because the hosts are considered gouging (far ahead of my node's block height), and so do other operations. The logs also get spammed with errors.
Currently /objects/:path
returns a list of strings, meaning the official UI and other consumers will need to do n+1 queries for common use cases. It would be useful to return the full objects, or at least the size of each.
@redsolver: It would be useful to have an option to include the file/object size when listing renterd objects. sending a request for every single file is pretty inefficient
https://discord.com/channels/809849352516141067/809858207064653894/1097948510801899610
This ticket is a blocker for renterd
rclone support: https://discord.com/channels/809849352516141067/827644062219239450/1098184384915460137
Ran into a panic in the autopilot loop today. After looking into it, I found that it happened on a single host that had these settings:
{AcceptingContracts:true MaxDownloadBatchSize:17825792 MaxDuration:25920 MaxReviseBatchSize:17825792 NetAddress:datastorage.loseyourip.com:9982 RemainingStorage:1399890903040 SectorSize:4194304 TotalStorage:1399890903040 Address:addr:df82659c915dbdcd8c644e37774c5c0374c5c8e4485e9bdf8547b877653ab7d06601250dd09a WindowSize:144 Collateral:4092520372 H MaxCollateral:~8.863 KS BaseRPCPrice:~8.958 pS ContractPrice:150 mS DownloadBandwidthPrice:~1.014 pS SectorAccessPrice:~1.674 nS StoragePrice:2018976717 H UploadBandwidthPrice:~195.3 MS EphemeralAccountExpiry:168h0m0s MaxEphemeralAccountBalance:1 SC RevisionNumber:326958 Version:1.5.9 SiaMuxPort:9983}
Then it caused an overflow panic:
goroutine 104 [running]:
go.sia.tech/core/types.Currency.Mul64(...)
.../go.sia.tech/[email protected]/types/currency.go:145
go.sia.tech/core/rhp/v2.RPCAppendCost({0x1, 0x1100000, 0x6540, 0x1100000, {0xc004d70720, 0x1f}, 0x145f0000000, 0x400000, 0x145f0000000, {0xdf, ...}, ...}, ...)
.../go.sia.tech/[email protected]/rhp/v2/rhp.go:310 +0x438
go.sia.tech/renterd/autopilot.uploadCostForScore({{0x0}, {0x0, 0x150, 0x0}, {{0xc003bece23, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x3f0, ...}}, ...)
.../renterd/autopilot/hostscore.go:295 +0xc5
go.sia.tech/renterd/autopilot.hostPeriodCostForScore({{0x517cf3e, 0xedbba316a, 0x1cd22c0}, {0x0, 0xedb7b1344, 0x0}, {0x6, 0x87, 0xa8, 0x64, ...}, ...}, ...)
.../renterd/autopilot/hostscore.go:341 +0x345
go.sia.tech/renterd/autopilot.hostScore({{0x0}, {0x0, 0x150, 0x0}, {{0xc003bece23, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x3f0, ...}}, ...)
.../renterd/autopilot/hostscore.go:18 +0xc5
go.sia.tech/renterd/autopilot.isUsableHost({{0x0}, {0x0, 0x150, 0x0}, {{0xc003bece23, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x3f0, ...}}, ...)
.../renterd/autopilot/hostfilter.go:207 +0xae5
go.sia.tech/renterd/autopilot.(*contractor).candidateHosts(0xc003c1d680, {0x10a0340?, 0x7f23888811d8?}, {0xe5580?, 0xf620a0?}, {0xc005f46000, 0x2c2, 0x40bb05?}, 0x0?, 0xc00433e120, ...)
.../renterd/autopilot/contractor.go:996 +0x99b
go.sia.tech/renterd/autopilot.(*contractor).managedFindMinAllowedHostScores(0xc003c1d680, {0x158bcd0, 0xc004c301e0}, {0x1591b08, 0xc003bfb4a0}, {0xc005f46000, 0x2c2, 0x2c7}, 0x2?)
.../renterd/autopilot/contractor.go:921 +0x148
go.sia.tech/renterd/autopilot.(*contractor).performContractMaintenance(0xc003c1d680, {0x158bcd0?, 0xc004bae660?}, {0x1591b08, 0xc003bfb4a0})
.../renterd/autopilot/contractor.go:291 +0xcab```
When I visit http://127.0.0.1:9980/ type API password and press button "Set" I'm getting an error:
"Unexpected token '<', "<!DOCTYPE "... is not valid JSON" - in chromium
"JSON.parse: unexpected character at line 1 column 1 of the JSON data" - in firefox
Create folder "abc", upload a file into that folder. LICENCE in my example.
Create folder "Abc", upload a file into that folder. README.md in my example.
From the root the folder size looks correct;
But entering the folders they show a mix of both the folders:
The health checkmark only works for the folder where the file actually exists and trying to download the other file throws "object not found".
Clear case sensitivity.
Are you running a fork of renterd
?
No
Are you on the mainnet or on the testnet?
Relevant for both mainnet and testnet
The contracts page should have a filter, or just by default show only the contracts that are truly active, which IIRC is the equiv to /api/bus/contracts/set/autopilot
When refreshing the page (F5) it is not remembered what page you are on, but it defaults to http://localhost:9880/files
All pages seem to have this.
When refreshing the page, return/remain on the same page.
Running on the master-testnet docker image.
2023-04-06T12:57:55+02:00 ERROR db stores/metadata.go:453 UNIQUE constraint failed: archived_contracts.fcid {"elapsed": "0.390ms", "rows": 0, "sql": "INSERT INTO `archived_contracts` (`created_at`,`fcid`,`renewed_from`,`total_cost`,`proof_height`,`revision_height`,`revision_number`,`start_height`,`window_start`,`window_end`,`upload_spending`,`download_spending`,`fund_account_spending`,`renewed_to`,`host`,`reason`) VALUES (\"2023-04-06 12:57:55.47\",\"<binary>\",\"<binary>\",\"2873949048004055569640000\",0,0,\"0\",411954,412386,412530,\"0\",\"0\",\"0\",\"<binary>\",\"<binary>\",\"removed\") RETURNING `id`"} 2023-04-06T12:57:55+02:00 WARN db stores/metadata.go:452 transaction attempt 1/5 failed, err: UNIQUE constraint failed: archived_contracts.fcid 2023-04-06T12:57:55+02:00 ERROR db stores/metadata.go:453 UNIQUE constraint failed: archived_contracts.fcid {"elapsed": "0.514ms", "rows": 0, "sql": "INSERT INTO `archived_contracts` (`created_at`,`fcid`,`renewed_from`,`total_cost`,`proof_height`,`revision_height`,`revision_number`,`start_height`,`window_start`,`window_end`,`upload_spending`,`download_spending`,`fund_account_spending`,`renewed_to`,`host`,`reason`) VALUES (\"2023-04-06 12:57:55.672\",\"<binary>\",\"<binary>\",\"2873949048004055569640000\",0,0,\"0\",411954,412386,412530,\"0\",\"0\",\"0\",\"<binary>\",\"<binary>\",\"removed\") RETURNING `id`"} 2023-04-06T12:57:55+02:00 WARN db stores/metadata.go:452 transaction attempt 2/5 failed, err: UNIQUE constraint failed: archived_contracts.fcid 2023-04-06T12:57:55+02:00 ERROR db stores/metadata.go:453 UNIQUE constraint failed: archived_contracts.fcid {"elapsed": "0.446ms", "rows": 0, "sql": "INSERT INTO `archived_contracts` (`created_at`,`fcid`,`renewed_from`,`total_cost`,`proof_height`,`revision_height`,`revision_number`,`start_height`,`window_start`,`window_end`,`upload_spending`,`download_spending`,`fund_account_spending`,`renewed_to`,`host`,`reason`) VALUES (\"2023-04-06 12:57:55.873\",\"<binary>\",\"<binary>\",\"2873949048004055569640000\",0,0,\"0\",411954,412386,412530,\"0\",\"0\",\"0\",\"<binary>\",\"<binary>\",\"removed\") RETURNING `id`"} 2023-04-06T12:57:55+02:00 WARN db stores/metadata.go:452 transaction attempt 3/5 failed, err: UNIQUE constraint failed: archived_contracts.fcid 2023-04-06T12:57:56+02:00 ERROR db stores/metadata.go:453 UNIQUE constraint failed: archived_contracts.fcid {"elapsed": "0.433ms", "rows": 0, "sql": "INSERT INTO `archived_contracts` (`created_at`,`fcid`,`renewed_from`,`total_cost`,`proof_height`,`revision_height`,`revision_number`,`start_height`,`window_start`,`window_end`,`upload_spending`,`download_spending`,`fund_account_spending`,`renewed_to`,`host`,`reason`) VALUES (\"2023-04-06 12:57:56.075\",\"<binary>\",\"<binary>\",\"2873949048004055569640000\",0,0,\"0\",411954,412386,412530,\"0\",\"0\",\"0\",\"<binary>\",\"<binary>\",\"removed\") RETURNING `id`"} 2023-04-06T12:57:56+02:00 WARN db stores/metadata.go:452 transaction attempt 4/5 failed, err: UNIQUE constraint failed: archived_contracts.fcid 2023-04-06T12:57:56+02:00 ERROR db stores/metadata.go:453 UNIQUE constraint failed: archived_contracts.fcid {"elapsed": "0.443ms", "rows": 0, "sql": "INSERT INTO `archived_contracts` (`created_at`,`fcid`,`renewed_from`,`total_cost`,`proof_height`,`revision_height`,`revision_number`,`start_height`,`window_start`,`window_end`,`upload_spending`,`download_spending`,`fund_account_spending`,`renewed_to`,`host`,`reason`) VALUES (\"2023-04-06 12:57:56.276\",\"<binary>\",\"<binary>\",\"2873949048004055569640000\",0,0,\"0\",411954,412386,412530,\"0\",\"0\",\"0\",\"<binary>\",\"<binary>\",\"removed\") RETURNING `id`"} 2023-04-06T12:57:56+02:00 WARN db stores/metadata.go:452 transaction attempt 5/5 failed, err: UNIQUE constraint failed: archived_contracts.fcid
Ability to delete entire directories from the UI with the new DELETE localhost:9980/api/bus/objects/?batch=true
endpoint.
This host has an upload price of 800 SC, well under 4KS. The "Max Upload Price" gouging setting can be difficult since the calculation includes storage costs. Since storage is already covered by "Max Storage Cost," this feels redundant. I propose changing "Max Upload Price" only to use the bandwidth pricing to make these settings more intuitive.
Deleting and resyncing the consensus causes an error:
ConsensusSetSubscribe returned an error: consensus subscription has invalid id - files are inconsistent
Need to reset the consensus change ID in the database
If a consensus scan is needed using ConsensusSetSubscribe
, then it should provide a console status indicator of progress and ideally a percentage.
Without this, on boot, the program feels like it's broken or hanging forever.
It takes a really long time to open large directories.
Vup places all files into the same folder (/s5). Currently this directory has roughly 1600 files.
It takes 1900 calls, 50MB and 30 seconds to load this page.
It looks like the health of each file is loaded when the page is loaded, taking 1 call per file.
Perhaps a workaround would be something like an endpoint that returns only unhealthy files, such that all not-returned files can be assumed healthy when loading the page.
Mime type issue possibly from the Content-Type header. Video files downloaded from web UI have
webm
extension
https://discord.com/channels/809849352516141067/946328563320238190/1095584899899084860
It should look like this? renterd:d5b9fcf
2023-05-13T12:08:07Z INFO autopilot.contractor autopilot/contractor.go:98 performing contract maintenance
2023-05-13T12:08:07Z DEBUG autopilot.contractor autopilot/contractor.go:135 contract set 'autopilot' holds 50 contracts
2023-05-13T12:08:07Z DEBUG autopilot.scanner autopilot/scanner.go:241 scanning 1000 hosts in range 0-1000
2023-05-13T12:08:08Z DEBUG autopilot.scanner autopilot/scanner.go:241 scanning 1000 hosts in range 1000-2000
2023-05-13T12:08:19Z WARN db bus/bus.go:487 SLOW SQL >= 500ms {"elapsed": "1216.875ms", "rows": 126, "sql": "SELECT * FROM `hosts` WHERE public_key IN (\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"Ȑ\ufffd`\ufffd\ufffd\ufffd\ufffd:\ufffdZ\ufffd\ufffd|\ufffd0\ufffdט\ufffd\ufffd\ufffd\ufffd\ufffdhV5Ǻ\ufffd\ufffd\ufffd\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"\ufffdw\ufffd|\ufffd\ufffd\ufffd\\\"X\ufffd ~I[Q/\ufffd\ufffds\ufffd\ufffd8\ufffd5\ufffdg\ufffd\ufffdJ\ufffd-\ufffd\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\")"}
2023-05-13T12:08:40Z DEBUG autopilot.scanner autopilot/scanner.go:241 scanning 1000 hosts in range 2000-3000
2023-05-13T12:08:46Z WARN db bus/bus.go:487 SLOW SQL >= 500ms {"elapsed": "1032.658ms", "rows": 121, "sql": "SELECT * FROM `hosts` WHERE public_key IN (\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"g\ufffd.(\ufffd`Y{F\ufffd\ufffd\ufffd\ufffd4fe\ufffdL\ufffd\ufffd\ufffdr\ufffd\ufffd/Qo\ufffd\ufffd?]\ufffd\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"T\ufffd^\ufffd{d)\ufffd\ufffd\ufffd=pG\ufffd\ufffd\ufffd)\ufffd`R\ufffd\ufffd\ufffd\ufffdc\ufffdR\ufffd齳\ufffd\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\")"}
I'm currently running both hostd and renterd on the testnet. In renterd I have blacklisted my own host (by hostname). However, renterd still formed a contract with it, and continues to use it for new data.
The blacklist was configured before the hostd was installed/announced.
I expected my hostd instance to not receive data from my renterd instance.
Image of my renterd displaying the host is blocked
Running latest docker image of renterd testnet
Please provide us the following information:
Autopilot Config
{
"wallet": {
"defragThreshold": 0
},
"hosts": {
"allowRedundantIPs": true,
"maxDowntimeHours": 1440,
"scoreOverrides": null
},
"contracts": {
"set": "autopilot",
"amount": 50,
"allowance": "600000000000000000000000000000",
"period": 6048,
"renewWindow": 2016,
"download": 1099511627776,
"upload": 5000000000000,
"storage": 5000000000000
}
}
Bus Config
{
"minShards": 4,
"totalShards": 12
}
{
"hostBlockHeightLeeway": 6,
"maxContractPrice": "150000000000000000000000000",
"maxDownloadPrice": "3000000000000000000000000000000",
"maxRPCPrice": "10000000000000000000000",
"maxStoragePrice": "231481481481481481481481",
"maxUploadPrice": "3000000000000000000000000000000",
"minAccountExpiry": 86400000000000,
"minMaxCollateral": "1000000000000000000000000",
"minMaxEphemeralAccountBalance": "1000000000000000000000000",
"minPriceTableValidity": 300000000000
}
Contract Set Contracts
17
I have logs starting at the birth of this node. I don't know what to look for, and what data is in there that isn't safe to share.
Message me on Discord and I'll provide the file if it's needed.
CtrlAltDefeat#3152
I would like to see an endpoint like /api/worker/rhp/scan
that can bulk scan all hosts or just ones deemed unusable as a manual trigger.
Ive had to write a bash script to parse out the hosts endpoint and manually trigger host scans to do this.
When the contracts are received from a satellite, it is possible (especially on a fresh node) that not all contracted hosts have been scanned yet. This triggers a panic when doing the contract checks, specifically here:
pt, err := c.priceTable(ctx, w, host.PublicKey, host.Settings.SiamuxAddr())
,
because the host.Settings are still nil.
When configuring the autopilot on a fresh renter
node and saving the changes, an undefined is not an object (evaluating 'i.default')
error pops up.
No error messages should appear.
The config is saved successfully, though.
Are you running a fork of renterd
? Yes, but this part of code is untouched.
Are you on the mainnet or on the testnet? Mainnet.
Please provide us the following information:
Autopilot Config
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/autopilot/config
Bus Config
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/redundancy
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/gouging
{
"minShards": 3,
"totalShards": 9
}
{
"hostBlockHeightLeeway": 6,
"maxContractPrice": "15000000000000000000000000",
"maxDownloadPrice": "3000000000000000000000000000",
"maxRPCPrice": "1000000000000000000000",
"maxStoragePrice": "631593542901",
"maxUploadPrice": "3000000000000000000000000000",
"minAccountExpiry": 86400000000000,
"minMaxCollateral": "10000000000000000000000000",
"minMaxEphemeralAccountBalance": "1000000000000000000000000",
"minPriceTableValidity": 300000000000
}```
**Contract Set Contracts**
```bash
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/contracts/set/autopilot | grep '"id"' | wc -l
[]
If applicable, upload your renterd.log
file or part of the file you think is relevant to the issue.
I have two contracts (94b98a7983f6786474af2b306a39c570fd82bdf2db3d220d00910de808c734cf and 3d302f6545ed46078aecf8191d8a936a804a682d7127134a9ef0a5a3a3313a7a), which I used to upload about 700MB of data, so upload spendings are about 7 and 8mS, respectively (as displayed in the UI). I have also downloaded about 10GB, but both download and fund account spendings are still 0H.
A host is triggering a SQL Scan bug
ERROR db bus/bus.go:493 too many SQL variables {"elapsed": "195.565ms", "rows": 0, "sql": "SELECT * FROM hosts WHERE public_key IN
{"elapsed": "195.565ms", "rows": 0, "sql": "SELECT * FROM hosts WHERE public_key IN (\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",\"<binary>\",
No SQL error
It will do this repeatedly and then fail to commit. Do not know what host is causing this.
Are you running a fork of renterd
? no
Are you on the mainnet or on the testnet? mainnet
If I hit ctrl-c when uploading a file via curl renterd crashes with an error like the following:
goroutine 4336 [running]:
go.sia.tech/renterd/worker.(*upload).uploadSlab(0xc000ac50e0?, {0x15fc0d0?, 0xc00375df80?}, {0x449865?, 0xc000ac50e0?}, {0xc01ab60000, 0x2800000, 0x2800000}, 0x850000, 0x1, ...)
/home/sia/renterd/worker/upload.go:669 +0x3d5
created by go.sia.tech/renterd/worker.(*uploadManager).Upload
/home/sia/renterd/worker/upload.go:322 +0x9b0
I run curl using the following command:
curl -u ":pass" -H "Content-Type: application/octet-stream" -T testfile -X PUT localhost:9780/api/worker/objects/testfile
Maybe getting an error message or similar but not a hard crash.
Are you running a fork of renterd
?
no
Are you on the mainnet or on the testnet?
mainnet
Please provide us the following information:
Autopilot Config
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/autopilot/config
{
"contracts": {
"set": "autopilot",
"amount": 50,
"allowance": "9000000000000000000000000000",
"period": 6048,
"renewWindow": 2016,
"download": 3000000000000,
"upload": 3000000000000,
"storage": 11000000000000
},
"hosts": {
"allowRedundantIPs": false,
"maxDowntimeHours": 336,
"scoreOverrides": null
},
"wallet": {
"defragThreshold": 1000
}
}
Bus Config
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/redundancy
{
"minShards": 10,
"totalShards": 30
}
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/setting/gouging
{
"hostBlockHeightLeeway": 6,
"maxContractPrice": "450000000000000000000000",
"maxDownloadPrice": "2300000000000000000000000000",
"maxRPCPrice": "2000000000000000000000",
"maxStoragePrice": "254629629630",
"maxUploadPrice": "900000000000000000000000000",
"minAccountExpiry": 86400000000000,
"minMaxCollateral": "10000000000000000000000000",
"minMaxEphemeralAccountBalance": "1000000000000000000000000",
"minPriceTableValidity": 300000000000
}
Contract Set Contracts
curl -u ":[YOUR_PASSWORD]" http://localhost:9980/api/bus/contracts/set/autopilot | grep '"id"' | wc -l
50
If applicable, upload your renterd.log
file or part of the file you think is relevant to the issue.
Per discord @n8maninger hostd has a wallet state endpoint that exposes the scan height, but I'm not seeing one for renterd.
This is needed so that the UX can show the wallet sync.
/bus/wallet/transactions should return latest transactions first, or provide parameters to allow selecting eg latest 50. Currently returns the oldest first.
renterd seed
should print to stdout, not stderr.
The gorm
default length for a string is 191 bytes:
`object_id` varchar(191) DEFAULT NULL,
That's not ideal, should be updated to 4096, the unix max path length, to support large paths.
I noticed than when the contract checks are running, the contractor fetches all active contracts, then, if they are more than what is configured in the autopilot, it puts the rest in the toIgnore
list.
Let's assume I have 28 active contracts, and I have configured the autopilot to have 20 contracts in the set. 8 contracts will be added to the toIgnore
list; at the same time, the toArchive
, toRenew
, and toRefresh
lists are calculated.
Let's further assume that 1 contract is up for renewal, so the contractor will try to renew it. If, for some reason, the renewal fails, the contract set will have 19 contracts until the next loop iteration. If the failure persists, the set will have 19 contracts until this contract expires.
Now, if more contracts are up for renewal, and the renewals fail, the resulting set will have even less contracts left.
Perhaps it would make sense to do the renewals/refreshes first, and then reassemble the toIgnore
list.
I have been away for 6 days, during which renterd
has been shut down, and all active contracts have expired.
Upon the start (after updating to the latest master
), the autopilot loop has archived 24 expired contracts and formed 1 new (it tried to form 2 but one formation failed). The second loop iteration successfully formed 20 new contracts (this is my configured amount), although I believe it should have formed the contracts during the first iteration already.
Still, I have 50 active contracts, of which 29 are far beyond the expiration height, both shown in the UI and when using curl
:
Also, both of my two uploaded files show a 100% health while both having 0 shards (obviously, because there are no contracts left):
The migrator also reports that there are no slabs to migrate (which is obviously wrong).
Downloading from active contracts make it so we try and fund accounts with all hosts. This is an issue because we keep try and fund account with hosts that appear offline.
Currently files cannot be renamed or moved in the UI.
Add ability in the UI to rename / move files from one path to another. Use GET / PUT / DELETE bus/objects/*key
to move metadata.
The wallet transaction list does not show sc inflows from returned allowance which is confusing to the user and also means manually computing the balance sparkline is not accurate.
Contract payouts are not technically transactions. hostd handles this by creating fake transactions for the payouts after they mature, could probably do something similar in renterd https://github.com/SiaFoundation/hostd/blob/bdf30348f1cf974bb1e66b2ee9494735399953ef/wallet/wallet.go#L466-L512
When a contract expires and is not renewed in a due time (for whatever reason), it stays in the list of active contracts until the current block height passes the end height of the contract, and then a panic is triggered:
panic: current blockHeight should be lower than the endHeight
goroutine 61 [running]:
go.sia.tech/core/rhp/v2.ContractRenewalCollateral({0x2c000000, {0xd0, 0xda, 0xe, 0xc3, 0x0, 0xd1, 0x2d, 0xba, 0x61, ...}, ...}, ...)
.../go/pkg/mod/go.sia.tech/core@v0.1.11-0.20230324142811-50da7dba9eb7/rhp/v2/contracts.go:87 +0x311
go.sia.tech/renterd/autopilot.isOutOfCollateral({{{0x3d, 0x30, 0x2f, 0x65, 0x45, 0xed, 0x46, 0x7, 0x8a, 0xec, ...}, ...}, ...}, ...)
.../renterd/autopilot/hostfilter.go:244 +0x145
go.sia.tech/renterd/autopilot.isUsableContract({{0x0}, {0x0, 0x150, 0x0}, {{0xc000aa1a30, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x3f0, ...}}, ...)
.../renterd/autopilot/hostfilter.go:198 +0x107
go.sia.tech/renterd/autopilot.(*contractor).runContractChecks(0xc000ad6140, {0x15714b0, 0xc0004b7110}, {0x1577288, 0xc000ac1890}, {0xc0001e8800, 0x2, 0x4}, 0x0?)
.../renterd/autopilot/contractor.go:488 +0xf25
go.sia.tech/renterd/autopilot.(*contractor).performContractMaintenance(0xc000ad6140, {0x15714b0?, 0xc0004b6150?}, {0x1577288, 0xc000ac1890})
.../renterd/autopilot/contractor.go:269 +0xbf6
.................................
I think the contract should be removed from the active contracts if it is not renewed.
Up and downloadingvia /worker/objects/ using a double slash (eg. /worker/objects//path/to/file.jpeg) results in different behaviour.
I've been testing with different redundancies, and ran into issues when using too many hosts on the testnet, so I decided to reupload the same files, meaning the redundancy would update to the latest settings.
However I made a mistake in my url generation, resulting in a double slash in the path.
When I download the file (GET /worker/objects//path), I can download the file without issue.
When I upload the file (PUT /worker/objects//path), the file is reuploaded to a folder named /
, rather than the original path.
An additional thing. It's impossible to open this directory in the webui, even when changing the url to visit the directory directly.
When spinning up a fresh instance of renterd the following behaviour was observed:
$curl --user :$Pass --location '$IP:$PORT/api/bus/hosts'
returns all announced hosts but all are "scanned": false
scanning 1000 hosts in range 2000-3000
$curl --user :$Pass --location '$IP:$PORT/api/bus/hosts'
returns all announced hosts and all are "scanned": true
and their price tables are populated nowSince upon syncing the blockchain all hosts and their current address are known, they should immediately be scanned for their settings/prices and then be displayed on the "Hosts" page even if no autopilot/gouging is set. Fields that need these settings should display a question mark and a message to tell the user to configure autopilot/gouging to calculate them.
This would improve UX since it makes the software more accessible to users by showing available hosts without the need to dive deeper into the meaning of available parameters. This would also allow renterd to be used for cases where the built-in tools of renterd are to be used for other purposes than only renting like quickly checking hosts or calculating network metrics.
UnRAID, Docker, ghcr.io/siafoundation/renterd:master
Autopilot Config
{
"contracts": {
"set": "autopilot",
"amount": 0,
"allowance": "0",
"period": 12096,
"renewWindow": 3024,
"download": 0,
"upload": 0,
"storage": 0
},
"hosts": {
"allowRedundantIPs": false,
"maxDowntimeHours": 720,
"scoreOverrides": null
},
"wallet": {
"defragThreshold": 1000
}
}
Bus Config
{
"minShards": 10,
"totalShards": 30
}
{
"hostBlockHeightLeeway": 6,
"maxContractPrice": "15000000000000000000000000",
"maxDownloadPrice": "3000000000000000000000000000",
"maxRPCPrice": "1000000000000000000000",
"maxStoragePrice": "631593542897",
"maxUploadPrice": "3000000000000000000000000000",
"minAccountExpiry": 86400000000000,
"minMaxCollateral": "10000000000000000000000000",
"minMaxEphemeralAccountBalance": "1000000000000000000000000",
"minPriceTableValidity": 300000000000
}
(Unmodified)
Contract Set Contracts
none
Accounts are being funded in a constant loop. 18 minutes after startup, the logs contain 2400 lines of succesfully funded accounts.
It is impossible to upload new files.
Running the master-testnet docker image.
Please provide us the following information:
Autopilot Config
{
"wallet": {
"defragThreshold": 0
},
"hosts": {
"allowRedundantIPs": true,
"maxDowntimeHours": 1440,
"scoreOverrides": null
},
"contracts": {
"set": "autopilot",
"amount": 50,
"allowance": "600000000000000000000000000000",
"period": 6048,
"renewWindow": 2016,
"download": 1099511627776,
"upload": 5000000000000,
"storage": 5000000000000
}
}
Bus Config
{
"minShards": 6,
"totalShards": 18
}
{
"hostBlockHeightLeeway": 6,
"maxContractPrice": "150000000000000000000000000",
"maxDownloadPrice": "3000000000000000000000000000000",
"maxRPCPrice": "10000000000000000000000",
"maxStoragePrice": "231481481481481481481481",
"maxUploadPrice": "3000000000000000000000000000000",
"minAccountExpiry": 86400000000000,
"minMaxCollateral": "1000000000000000000000000",
"minMaxEphemeralAccountBalance": "1000000000000000000000000",
"minPriceTableValidity": 300000000000
}
Contract Set Contracts
Fresh startup
{"level":"info","date":"2023-06-12T13:20:06+02:00","component":"autopilot","msg":"autopilot iteration starting"}
{"level":"info","date":"2023-06-12T13:20:06+02:00","component":"autopilot","msg":"using worker worker for iteration"}
{"level":"debug","date":"2023-06-12T13:20:06+02:00","component":"autopilot.scanner","msg":"updated timeout is lower than min timeout, 0s<5s"}
{"level":"debug","date":"2023-06-12T13:20:06+02:00","component":"autopilot.scanner","msg":"updated timeout 0s->5s"}
{"level":"info","date":"2023-06-12T13:20:06+02:00","component":"autopilot.scanner","msg":"host scan started"}
{"level":"debug","date":"2023-06-12T13:20:06+02:00","component":"autopilot.contractor","msg":"updated current period, 0->21543"}
{"level":"info","date":"2023-06-12T13:20:06+02:00","component":"autopilot.contractor","msg":"performing wallet maintenance"}
{"level":"debug","date":"2023-06-12T13:20:06+02:00","component":"autopilot.scanner","msg":"scanning 12 hosts in range 0-1000"}
{"level":"debug","date":"2023-06-12T13:20:06+02:00","component":"autopilot.contractor","msg":"no wallet maintenance needed, plenty of outputs available (79>=50)"}
{"level":"info","date":"2023-06-12T13:20:06+02:00","component":"autopilot.contractor","msg":"performing contract maintenance"}
{"level":"debug","date":"2023-06-12T13:20:06+02:00","component":"autopilot.contractor","msg":"contract set 'autopilot' holds 20 contracts"}
{"level":"debug","date":"2023-06-12T13:20:11+02:00","component":"autopilot.scanner","msg":"removing hosts that have been offline for more than 1440 hours"}
{"level":"debug","date":"2023-06-12T13:20:11+02:00","component":"autopilot.scanner","msg":"host scan finished after 5.071504601s"}
{"level":"error","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"\ncc52ebed: dial tcp: lookup cc52ebed.siacentral.host on 127.0.0.11:53: no such host\nfc1aed42: LatestRevision: ReadResponse: unknown subscriber\n26feca36: handshake failed: invalid signature\n362d9459: handshake failed: invalid signature\n600e112d: handshake failed: invalid signature\n0e82ec64: LatestRevision: failed to fetch pricetable, err: PriceTable: ReadResponse: internal error\neb51aa36: LatestRevision: failed to fetch pricetable, err: PriceTable: ReadResponse: internal error\nd140e139: dial tcp 64.226.73.231:9883: i/o timeout\n99325d62: dial tcp 84.51.29.74:9883: i/o timeout\n249b8a29: dial tcp 212.51.148.7:9883: i/o timeout"}
{"level":"debug","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"fetched 30 contracts from the worker, took 31.340544737s"}
{"level":"debug","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"looking for 100 candidate hosts"}
{"level":"debug","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"selected 34 candidate hosts out of 70","excluded":0,"notcompletedscan":36}
{"level":"debug","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"scored 22 candidate hosts out of 34, took 1.086022ms","zeroscore":1,"unusable":12}
{"level":"debug","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"only found 22 candidate host(s) out of the 100 we wanted","offline":9,"gouging":1,"notacceptingcontracts":3}
{"level":"debug","date":"2023-06-12T13:20:37+02:00","component":"autopilot.contractor","msg":"running contract checks"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"contracts checks completed","contracts":30,"notfound":0,"toKeep":20,"toArchive":0,"toRefresh":0,"toRenew":0}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"run contracts refreshes","torefresh":0,"budget":"600 KS"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"contracts refreshes completed","refreshed":0,"budget":"600 KS"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"run contract formations","usedHosts":30,"required":50,"missing":30,"budget":"600 KS"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"looking for 33 candidate hosts"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"selected 4 candidate hosts out of 70","excluded":30,"notcompletedscan":36}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"scored 0 candidate hosts out of 4, took 514.516µs","zeroscore":1,"unusable":4}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"no candidate hosts found","offline":1,"lowscore":1,"gouging":1,"notacceptingcontracts":3}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"contract formations completed","formed":0,"budget":"600 KS"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.contractor","msg":"contracts after maintenance","formed":0,"renewed":0,"contractset":20}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot","msg":"account refills loop launched"}
{"level":"info","date":"2023-06-12T13:20:46+02:00","component":"autopilot","msg":"autopilot iteration ended"}
{"level":"info","date":"2023-06-12T13:20:46+02:00","component":"autopilot.migrator","msg":"performing migrations"}
{"level":"debug","date":"2023-06-12T13:20:46+02:00","component":"autopilot.migrator","msg":"0 slabs to migrate"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:45395cbbbae7a37dcda7f757892611ee734c9d715873db9540a75171bc91dc33","host":"ed25519:96bf3099f428f3affe45dcb30b904e5f4f3f33da368d9316d32b4fe0e4bec223","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:6b979bd693670ecd3eafffbe84d25080bd08f1715f1092f89777ef2d1c23d227","host":"ed25519:a3aef7db34413e0f1d33140c3cab0a0113317d55b99df6c77a38fbc7a11d2ae1","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:c2ed212fb8f7b248bc0332737ccfcc7a2188dee20ff6bd80771b164c47487301","host":"ed25519:84523ed12e1e2232fb340dd7871a7fb52533c570881505807e64080ff12819b1","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:6436ca402f0ddcdfd5ba8b7008101005f57d0b80c609c7245ca88c828ec50e8e","host":"ed25519:6fdce0663adbff9dd0ced4360721c8aea53433cab61c72c2f0082d3a87085400","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:16fea1fa20afe149e89e548b0491334900ad7c68aa18a0c799cffd9cb0f3216e","host":"ed25519:075c746dd0e85eea7aeb05a5c4f37c2af35e4a8c48493fccec285ead4dbe5c49","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:c9c756ca2aa09717b6de1b78e6216ad31481009ee356d22e035269fc239fdc2d","host":"ed25519:d140e13949a082e0ce7dfdb843b9cbdc5e759c31bd22f62923b14f83f447b953","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:4907fc981c95070ac0fd2e637d4f6f57b190add95d1ffe49a4b5700d2d708b06","host":"ed25519:0d6efec29208c1596fd45c3c08c0e145fdfa84cdf5ec1e317da057f0c5a59754","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:2756bc489fe2f2ad473d061c414251ecfd969523a404561469415f8e0cd0bb85","host":"ed25519:84956180b91ee359c1792064d7b671bf6c49db8dc6eea80965dfb31904b0b520","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:fc02085af15b1fb2db2609402a1d1f7d397ea78980dfb7cd7310559a3a7e966b","host":"ed25519:fc1aed42f7e59946f76a2b5df5b05908447b213cf7f163a2a4d145adf2351a13","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:7de8a9c2e83ab307c12a8cf084f61ecf549431388ab2f3a67eca758087b1fd85","host":"ed25519:a67114403165c23973fd34048ce18c4c1cb9155f6dddf41f8cb007fea3a6fc0f","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:0c742f9041cca27765981db233817477705a02dca2f982b5420d7597303a2041","host":"ed25519:a95bfd42307179f0affc1512b7014f0c50ab4d208d02f8e18d2ead8bf9b08edb","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:2fae7f02649be1068d69cd41811c67935bf681a0b1a9ec9557d45dc8e8fa3a54","host":"ed25519:9d0dc5d8a28332254debcf81c83ba429283293cc9647f96cd1071dad34717aab","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:6ed73245f3ec82d01a95b03d7a95a8ce2ab7dd51ba918e0a6aa9dbc38caadcc7","host":"ed25519:26feca369bd0e2d75e7a9ed034ea88d0382586f77586f51903296755f7f9acc7","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:71ee4409e548dbd96982bdb6f2a4db1ddfd99dadc379387218268d8b0a00d07f","host":"ed25519:488c63890b9d559e705bcbc142243caf09d3d216af22c131ab9e019356e8ff6e","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:cab9e1e0fa2a7d8a5b5fc02a07c9d79365ae2ab0d311c13bc5ca95afcecf8fbd","host":"ed25519:707d189dfd08469efebf1a51338b3d73eed4ac1b56213a0ed6521179f60a7b71","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:797bceab943bb1d0eb7b9a6e1f4b892e27390d9e18e9f02b8061ae1f43d05b9a","host":"ed25519:cc52ebedb747ef96e3f0e785cffc277200e2fef1968be6ae8a1c65a3b9987a29","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:9cbb38483bab6c299d571170b494fca036b58e396a1c0a75ece35eb850bcdce6","host":"ed25519:95770d3b9bf02e025bb7f76715cccf943974e73c480d81e1d9b480845bca1989","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:56119756fb55187c931e9d3a5b9d10bd4aea6317a60766a17a49ade32cc6ee8e","host":"ed25519:99325d6287ea827d5b2b2f082a93d378b510d8dc5d8d8aa3c9dc9fb6b2dc1193","balance":"1 SC"}
{"level":"info","date":"2023-06-12T13:20:56+02:00","component":"autopilot.accounts","msg":"Successfully funded account","account":"ed25519:67a30f79de8c83e5d888236913f47a63d5c2bbf551180d161e316c2da180d958","host":"ed25519:368dbcbfa590a6c0ae8364f320b7d57d4b0b7281dbcbbfb9ade05ce4aaa189ae","balance":"1 SC"}
If syncing an account with a host fails, it is retried infinitely with 60-70s intervals, spamming the logs with errors:
2023-05-23T15:41:00+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:d120301e656596b5572707d004d8d2e8d6276257786b14dbb7a604671bd0e229", "host": "ed25519:85510c2f465dabddefe9c9983e87b2ed4f059e689b84e2161e91ad8b62cc308a"}
2023-05-23T15:42:10+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:a8d4b432dfc29dc253cc0a6943cf21628b1a478a8664b994114232ed080bf139", "host": "ed25519:d6da8e55eb65522b32ab5188d709fc80dbba6ee5daa79576f2e14b8d12c15d3c"}
2023-05-23T15:42:10+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:d120301e656596b5572707d004d8d2e8d6276257786b14dbb7a604671bd0e229", "host": "ed25519:85510c2f465dabddefe9c9983e87b2ed4f059e689b84e2161e91ad8b62cc308a"}
2023-05-23T15:43:20+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:a8d4b432dfc29dc253cc0a6943cf21628b1a478a8664b994114232ed080bf139", "host": "ed25519:d6da8e55eb65522b32ab5188d709fc80dbba6ee5daa79576f2e14b8d12c15d3c"}
2023-05-23T15:43:20+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:d120301e656596b5572707d004d8d2e8d6276257786b14dbb7a604671bd0e229", "host": "ed25519:85510c2f465dabddefe9c9983e87b2ed4f059e689b84e2161e91ad8b62cc308a"}
2023-05-23T15:44:30+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:a8d4b432dfc29dc253cc0a6943cf21628b1a478a8664b994114232ed080bf139", "host": "ed25519:d6da8e55eb65522b32ab5188d709fc80dbba6ee5daa79576f2e14b8d12c15d3c"}
2023-05-23T15:44:30+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:d120301e656596b5572707d004d8d2e8d6276257786b14dbb7a604671bd0e229", "host": "ed25519:85510c2f465dabddefe9c9983e87b2ed4f059e689b84e2161e91ad8b62cc308a"}
2023-05-23T15:45:40+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:d120301e656596b5572707d004d8d2e8d6276257786b14dbb7a604671bd0e229", "host": "ed25519:85510c2f465dabddefe9c9983e87b2ed4f059e689b84e2161e91ad8b62cc308a"}
2023-05-23T15:45:40+02:00 ERROR autopilot.accounts autopilot/accounts.go:202 failed to sync account's balance: Post "http://127.0.0.1:9980/api/worker/rhp/sync": context deadline exceeded {"account": "ed25519:a8d4b432dfc29dc253cc0a6943cf21628b1a478a8664b994114232ed080bf139", "host": "ed25519:d6da8e55eb65522b32ab5188d709fc80dbba6ee5daa79576f2e14b8d12c15d3c"}```
Started as a question on discord.
Today i noticed files i uploaded earlier, a week or so ago, downloads as zero bytes files. In the UI i can see the correct file size and the health checkmarks are green but hovering over them says "excellent health (NaN%)". If i copy the metadata via the ui the "slabs" array is empty. Files i uploaded earlier today works as expected.
I noticed my logs has plenty of lines like
{"level":"debug","date":"2023-06-26T12:35:37-01:00","component":"autopilot.migrator","msg":"worker: successfully migrated slab 'key:xxxxxxxxxx' (health: 0.65) 57/1110"}
where the 57/1110 counts up but a lot of those groups ends somewhere early like that and doesn't get to 1110/1110. Im just figuring it should migrate all those slabs if it's started but maybe that's not how it works.
I haven't seen any attempts to migrate slabs since 2023-06-27T16:00:57-01:00, I've gotten no noticeable cpu/disk/network activity since then either and i'm seeing this in the logs:
{"level":"info","date":"2023-06-28T05:48:13-01:00","component":"autopilot.migrator","msg":"performing migrations"}
{"level":"debug","date":"2023-06-28T05:48:13-01:00","component":"autopilot.migrator","msg":"0 potential slabs fetched for migration"}
{"level":"debug","date":"2023-06-28T05:48:13-01:00","component":"autopilot.migrator","msg":"0 slabs to migrate"}
Also i've noticed in the logs i have two contracts that keep replacing eachother over and over, a few times per hour for days now:
{"level":"debug","date":"2023-06-28T05:38:07-01:00","component":"autopilot.contractor","msg":"contract fcid:8c3b4dd46ac71b12f68df33f3f768811f1504af14008de4b0dff56e843d59317 was removed from the contract set, size: 4034920448, reason: truncated"}
{"level":"debug","date":"2023-06-28T05:38:07-01:00","component":"autopilot.contractor","msg":"contract fcid:9466b612165614d4b1fa267df52b029172e994e7ddc41ef8dbd3b7a3170f76c9 was added to the contract set, size: 30396121088"}
{"level":"debug","date":"2023-06-28T05:38:07-01:00","component":"autopilot.contractor","msg":"contractset after maintenance","formed":0,"renewed":0,"refreshed":0,"contracts":50,"added":1,"removed":1}
then
{"level":"debug","date":"2023-06-28T05:48:13-01:00","component":"autopilot.contractor","msg":"contract fcid:9466b612165614d4b1fa267df52b029172e994e7ddc41ef8dbd3b7a3170f76c9 was removed from the contract set, size: 30396121088, reason: host's score is below minimum"}
{"level":"debug","date":"2023-06-28T05:48:13-01:00","component":"autopilot.contractor","msg":"contract fcid:8c3b4dd46ac71b12f68df33f3f768811f1504af14008de4b0dff56e843d59317 was added to the contract set, size: 4034920448"}
{"level":"debug","date":"2023-06-28T05:48:13-01:00","component":"autopilot.contractor","msg":"contractset after maintenance","formed":0,"renewed":0,"refreshed":0,"contracts":50,"added":1,"removed":1}
unsure if that's relevant though.
I've restarted renterd a few times the last few days, to pull in new commits and rebuild, but not when i've known it to be migrating anything and not forcefully.
Files should have slabs and be downloadable.
Are you running a fork of renterd
? No, running version built from 1cff37f now.
Are you on the mainnet or on the testnet? Mainnet
Running with the following flags:
./renterd -http localhost:9780 -bus.gatewayAddr localhost:9781
Please provide us the following information:
Autopilot Config
curl -u ":[YOUR_PASSWORD]" http://localhost:9780/api/autopilot/config
{
"contracts": {
"set": "autopilot",
"amount": 50,
"allowance": "9000000000000000000000000000",
"period": 6048,
"renewWindow": 2016,
"download": 3000000000000,
"upload": 3000000000000,
"storage": 11000000000000
},
"hosts": {
"allowRedundantIPs": false,
"maxDowntimeHours": 1440,
"scoreOverrides": null
},
"wallet": {
"defragThreshold": 1000
}
}
Bus Config
curl -u ":[YOUR_PASSWORD]" http://localhost:9780/api/bus/setting/redundancy
{
"minShards": 10,
"totalShards": 30
}
curl -u ":[YOUR_PASSWORD]" http://localhost:9780/api/bus/setting/gouging
{
"hostBlockHeightLeeway": 6,
"maxContractPrice": "850000000000000000000000",
"maxDownloadPrice": "3000000000000000000000000000",
"maxRPCPrice": "2000000000000000000000",
"maxStoragePrice": "578703703704",
"maxUploadPrice": "950000000000000000000000000",
"minAccountExpiry": 86400000000000,
"minMaxCollateral": "10000000000000000000000000",
"minMaxEphemeralAccountBalance": "1000000000000000000000000",
"minPriceTableValidity": 300000000000
}
Contract Set Contracts
curl -u ":[YOUR_PASSWORD]" http://localhost:9780/api/bus/contracts/set/autopilot | grep '"id"' | wc -l
50
If applicable, upload your renterd.log
file or part of the file you think is relevant to the issue.
Here's the last 2000 lines of my renterd.log:
short.log
ratio := new(big.Rat).SetFrac(cutoff.Big(), expectedCollateral.Big())
panic: division by zero
goroutine 179 [running]:
math/big.(*Rat).SetFrac(0xc001d5fbc0, 0xc0396fa020, 0xc0396fa040)
/home/derrick/.gvm/pkgsets/go1.18.6/global/go1.19/src/math/big/rat.go:307 +0x2c5
go.sia.tech/renterd/autopilot.collateralScore({{0x0}, {0x0, 0x0, 0x0}, {{0xc00338e180, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x17a0, ...}}, ...)
portal/vendor/go.sia.tech/renterd/autopilot/hostscore.go:150 +0x28d
go.sia.tech/renterd/autopilot.hostScore({{0x0}, {0x0, 0x0, 0x0}, {{0xc00338e180, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x17a0, ...}}, ...)
portal/vendor/go.sia.tech/renterd/autopilot/hostscore.go:20 +0x1a5
go.sia.tech/renterd/autopilot.isUsableHost({{0x0}, {0x0, 0x0, 0x0}, {{0xc00338e180, 0x9}, 0x32, {0x9fd0803ce8000000, 0x33b2e3c}, 0x17a0, ...}}, ...)
portal/vendor/go.sia.tech/renterd/autopilot/hostfilter.go:212 +0x845
go.sia.tech/renterd/autopilot.(*contractor).candidateHosts(0xc0003b6d20, {0x236d0a0, 0xc00d548fc0}, {0x2374408, 0xc0003780f0}, {0xc0703d4000, 0x149ca, 0x1680a}, 0xc001d62478, 0xc008e86000, ...)
portal/vendor/go.sia.tech/renterd/autopilot/contractor.go:847 +0x14a7
go.sia.tech/renterd/autopilot.(*contractor).managedFindMinAllowedHostScores(0xc0003b6d20, {0x236d0a0, 0xc00d548fc0}, {0x2374408, 0xc0003780f0}, {0xc0703d4000, 0x149ca, 0x1680a}, 0xc008e86000)
portal/vendor/go.sia.tech/renterd/autopilot/contractor.go:773 +0x1f0
go.sia.tech/renterd/autopilot.(*contractor).performContractMaintenance(0xc0003b6d20, {0x236d0a0, 0xc00d548fc0}, {0x2374408, 0xc0003780f0})
portal/vendor/go.sia.tech/renterd/autopilot/contractor.go:171 +0x1145
go.sia.tech/renterd/autopilot.(*Autopilot).Run.func1({0x2374408, 0xc0003780f0})
portal/vendor/go.sia.tech/renterd/autopilot/autopilot.go:259 +0xcd4
go.sia.tech/renterd/autopilot.(*workerPool).withWorker(0xc0003781b0, 0xc05b59fe90)
portal/vendor/go.sia.tech/renterd/autopilot/autopilot.go:148 +0x102
go.sia.tech/renterd/autopilot.(*Autopilot).Run(0xc00037a000)
portal/vendor/go.sia.tech/renterd/autopilot/autopilot.go:206 +0x734
git.lumeweb.com/LumeWeb/portal/renterd.Main.func2()
portal/renterd/main.go:324 +0x2b
created by git.lumeweb.com/LumeWeb/portal/renterd.Main
portal/renterd/main.go:324 +0x2a8c
t := time.NewTicker(cl.d / 2)
going negative?
panic: non-positive interval for NewTicker
goroutine 10119 [running]:
time.NewTicker(0x0)
/home/derrick/.gvm/pkgsets/go1.18.6/global/go1.19/src/time/tick.go:24 +0x1aa
go.sia.tech/renterd/worker.(*contractLock).keepaliveLoop(0xc135c6d080)
portal/vendor/go.sia.tech/renterd/worker/worker.go:1349 +0x4f
created by go.sia.tech/renterd/worker.(*worker).AcquireContract
portal/vendor/go.sia.tech/renterd/worker/worker.go:1388 +0x6fb
Should the downloader support multipart byte-range requests? From siad
, /renter/stream
uses http.ServeContent
and an io.ReadSeeker
, but /renter/download?httpresp=true
matches this by writing straight to w
Line 570 in de8d46c
Uploading large files leads to "Too Many SQL Variables" errors.
2023-05-02T23:32:53Z ERROR db stores/metadata.go:599 too many SQL variables {"elapsed": "567.426ms", "rows": 1369, "sql": "SELECT * FROM `slabs` WHERE `slabs`.`db_slice_id` IN (2561,2562,2563,2564,2565,2566,2567,2568,2569,2570,2571,2572,2573,2574,2575,2576,2577,2578,2579,2580,2581,2582,2583,2584,2585,2586,2587,2588,2589,2590,2591,2592,2593,2594,2595,2596,2597,2598,2599,2600,2601,2602,2603,2604,2605,2606,2607,2608,2609,2610,2611,2612,2613,2614,2615,2616,2617,2618,2619,2620,2621,2622,2623,2624,2625,2626,2627,2628,2629,2630,2631,2632,2633,2634,2635,2636,2637,2638,2639,2640,2641,2642,2643,2644,2645,2646,2647,2648,2649,2650,2651,2652,2653,2654,2655,2656,2657,2658,2659,2660,2661,2662,2663,2664,2665,2666,2667,2668,2669,2670,2671,2672,2673,2674,2675,2676,2677,2678,2679,2680,2681,2682,2683,2684,2685,2686,2687,2688,2689,2690,2691,2692,2693,2694,2695,2696,2697,2698,2699,2700,2701,2702,2703,2704,2705,2706,2707,2708,2709,2710,2711,2712,2713,2714,2715,2716,2717,2718,2719,2720,2721,2722,2723,2724,2725,2726,2727,2728,2729,2730,2731,2732,2733,2734,2735,2736,2737,2738,2739,2740,2741,2742,2743,2744,2745,2746,2747,2748,2749,2750,2751,2752,2753,2754,2755,2756,2757,2758,2759,2760,2761,2762,2763,2764,2765,2766,2767,2768,2769,2770,2771,2772,2773,2774,2775,2776,2777,2778,2779,2780,2781,2782,2783,2784,2785,2786,2787,2788,2789,2790,2791,2792,2793,2794,2795,2796,2797,2798,2799,2800,2801,2802,2803,2804,2805,2806,2807,2808,2809,2810,2811,2812,2813,2814,2815,2816,2817,2818,2819,2820,2821,2822,2823,2824,2825,2826,2827,2828,2829,2830,2831,2832,2833,2834,2835,2836,2837,2838,2839,2840,2841,2842,2843,2844,2845,2846,2847,2848,2849,2850,2851,2852,2853,2854,2855,2856,2857,2858,2859,2860,2861,2862,2863,2864,2865,2866,2867,2868,2869,2870,2871,2872,2873,2874,2875,2876,2877,2878,2879,2880,2881,2882,2883,2884,2885,2886,2887,2888,2889,2890,2891,2892,2893,2894,2895,2896,2897,2898,2899,2900,2901,2902,2903,2904,2905,2906,2907,2908,2909,2910,2911,2912,2913,2914,2915,2916,2917,2918,2919,2920,2921,2922,2923,2924,2925,2926,2927,2928,2929,2930,2931,2932,2933,2934,2935,2936,2937,2938,2939,2940,2941,2942,2943,2944,2945,2946,2947,2948,2949,2950,2951,2952,2953,2954,2955,2956,2957,2958,2959,2960,2961,2962,2963,2964,2965,2966,2967,2968,2969,2970,2971,2972,2973,2974,2975,2976,2977,2978,2979,2980,2981,2982,2983,2984,2985,2986,2987,2988,2989,2990,2991,2992,2993,2994,2995,2996,2997,2998,2999,3000,3001,3002,3003,3004,3005,3006,3007,3008,3009,3010,3011,3012,3013,3014,3015,3016,3017,3018,3019,3020,3021,3022,3023,3024,3025,3026,3027,3028,3029,3030,3031,3032,3033,3034,3035,3036,3037,3038,3039,3040,3041,3042,3043,3044,3045,3046,3047,3048,3049,3050,3051,3052,3053,3054,3055,3056,3057,3058,3059,3060,3061,3062,3063,3064,3065,3066,3067,3068,3069,3070,3071,3072,3073,3074,3075,3076,3077,3078,3079,3080,3081,3082,3083,3084,3085,3086,3087,3088,3089,3090,3091,3092,3093,3094,3095,3096,3097,3098,3099,3100,3101,3102,3103,3104,3105,3106,3107,3108,3109,3110,3111,3112,3113,3114,3115,3116,3117,3118,3119,3120,3121,3122,3123,3124,3125,3126,3127,3128,3129,3130,3131,3132,3133,3134,3135,3136,3137,3138,3139,3140,3141,3142,3143,3144,3145,3146,3147,3148,3149,3150,3151,3152,3153,3154,3155,3156,3157,3158,3159,3160,3161,3162,3163,3164,3165,3166,3167,3168,3169,3170,3171,3172,3173,3174,3175,3176,3177,3178,3179,3180,3181,3182,3183,3184,3185,3186,3187,3188,3189,3190,3191,3192,3193,3194,3195,3196,3197,3198,3199,3200,3201,3202,3203,3204,3205,3206,3207,3208,3209,3210,3211,3212,3213,3214,3215,3216,3217,3218,3219,3220,3221,3222,3223,3224,3225,3226,3227,3228,3229,3230,3231,3232,3233,3234,3235,3236,3237,3238,3239,3240,3241,3242,3243,3244,3245,3246,3247,3248,3249,3250,3251,3252,3253,3254,3255,3256,3257,3258,3259,3260,3261,3262,3263,3264,3265,3266,3267,3268,3269,3270,3271,3272,3273,3274,3275,3276,3277,3278,3279,3280,3281,3282,3283,3284,3285,3286,3287,3288,3289,3290,3291,3292,3293,3294,3295,3296,3297,3298,3299,3300,3301,3302,3303,3304,3305,3306,3307,3308,3309,3310,3311,3312,3313,3314,3315,3316,3317,3318,3319,3320,3321,3322,3323,3324,3325,3326,3327,3328,3329,3330,3331,3332,3333,3334,3335,3336,3337,3338,3339,3340,3341,3342,3343,3344,3345,3346,3347,3348,3349,3350,3351,3352,3353,3354,3355,3356,3357,3358,3359,3360,3361,3362,3363,3364,3365,3366,3367,3368,3369,3370,3371,3372,3373,3374,3375,3376,3377,3378,3379,3380,3381,3382,3383,3384,3385,3386,3387,3388,3389,3390,3391,3392,3393,3394,3395,3396,3397,3398,3399,3400,3401,3402,3403,3404,3405,3406,3407,3408,3409,3410,3411,3412,3413,3414,3415,3416,3417,3418,3419,3420,3421,3422,3423,3424,3425,3426,3427,3428,3429,3430,3431,3432,3433,3434,3435,3436,3437,3438,3439,3440,3441,3442,3443,3444,3445,3446,3447,3448,3449,3450,3451,3452,3453,3454,3455,3456,3457,3458,3459,3460,3461,3462,3463,3464,3465,3466,3467,3468,3469,3470,3471,3472,3473,3474,3475,3476,3477,3478,3479,3480,3481,3482,3483,3484,3485,3486,3487,3488,3489,3490,3491,3492,3493,3494,3495,3496,3497,3498,3499,3500,3501,3502,3503,3504,3505,3506,3507,3508,3509,3510,3511,3512,3513,3514,3515,3516,3517,3518,3519,3520,3521,3522,3523,3524,3525,3526,3527,3528,3529,3530,3531,3532,3533,3534,3535,3536,3537,3538,3539,3540,3541,3542,3543,3544,3545,3546,3547,3548,3549,3550,3551,3552,3553,3554,3555,3556,3557,3558,3559,3560,3561,3562,3563,3564,3565,3566,3567,3568,3569,3570,3571,3572,3573,3574,3575,3576,3577,3578,3579,3580,3581,3582,3583,3584,3585,3586,3587,3588,3589,3590,3591,3592,3593,3594,3595,3596,3597,3598,3599,3600,3601,3602,3603,3604,3605,3606,3607,3608,3609,3610,3611,3612,3613,3614,3615,3616,3617,3618,3619,3620,3621,3622,3623,3624,3625,3626,3627,3628,3629,3630,3631,3632,3633,3634,3635,3636,3637,3638,3639,3640,3641,3642,3643,3644,3645,3646,3647,3648,3649,3650,3651,3652,3653,3654,3655,3656,3657,3658,3659,3660,3661,3662,3663,3664,3665,3666,3667,3668,3669,3670,3671,3672,3673,3674,3675,3676,3677,3678,3679,3680,3681,3682,3683,3684,3685,3686,3687,3688,3689,3690,3691,3692,3693,3694,3695,3696,3697,3698,3699,3700,3701,3702,3703,3704,3705,3706,3707,3708,3709,3710,3711,3712,3713,3714,3715,3716,3717,3718,3719,3720,3721,3722,3723,3724,3725,3726,3727,3728,3729,3730,3731,3732,3733,3734,3735,3736,3737,3738,3739,3740,3741,3742,3743,3744,3745,3746,3747,3748,3749,3750,3751,3752,3753,3754,3755,3756,3757,3758,3759,3760,3761,3762,3763,3764,3765,3766,3767,3768,3769,3770,3771,3772,3773,3774,3775,3776,3777,3778,3779,3780,3781,3782,3783,3784,3785,3786,3787,3788,3789,3790,3791,3792,3793,3794,3795,3796,3797,3798,3799,3800,3801,3802,3803,3804,3805,3806,3807,3808,3809,3810,3811,3812,3813,3814,3815,3816,3817,3818,3819,3820,3821,3822,3823,3824,3825,3826,3827,3828,3829,3830,3831,3832,3833,3834,3835,3836,3837,3838,3839,3840,3841,3842,3843,3844,3845,3846,3847,3848,3849,3850,3851,3852,3853,3854,3855,3856,3857,3858,3859,3860,3861,3862,3863,3864,3865,3866,3867,3868,3869,3870,3871,3872,3873,3874,3875,3876,3877,3878,3879,3880,3881,3882,3883,3884,3885,3886,3887,3888,3889,3890,3891,3892,3893,3894,3895,3896,3897,3898,3899,3900,3901,3902,3903,3904,3905,3906,3907,3908,3909,3910,3911,3912,3913,3914,3915,3916,3917,3918,3919,3920,3921,3922,3923,3924,3925,3926,3927,3928,3929)"}
reported by artur9010#9328
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.