maded2 / plotng Goto Github PK
View Code? Open in Web Editor NEWPlotNG - plotting utility for Chia.Net
License: Apache License 2.0
PlotNG - plotting utility for Chia.Net
License: Apache License 2.0
This would save a bit of vertical space, and the level of precision doesn't seem needed since it's not a realtime clock. (Also, maybe always format seconds as two digits?) Thank you!
Currently I'm running into an issue where my ethernet traffic is bottlenecked because all the plots are done but is uploading to a hard drive on the network all at once, causing the plots to not be available until all of them are copied over. It would be very useful to be able to limit stage 4 upload to one at a time. Would this be possible?
We have the ability to lower the number of plots by editing the config.
It would be nice to be able to hit a button, say d
in the UI to top the specific plot that's being plotted.
It would be important to have a confirmation window appear before actually canceling.
Perhaps I am missing something, but I cannot find any configuration for a separate "temp2" directory (configured with the -2
switch in the chia command line. Did I just miss it somewhere ? - thanks ! 😄
hi! First of all I think this tool is super cool and has made my plotting life a lot better than just using the stupid broken Chia GUI.
right now we have DiskSpaceCheck. Which checks space in the destination directory for approx 105GB free...
as a possible bug report, when I run plotng on a windows system and my destination drive is a mapped SMB share, I always get an error that the drive (0b free)is full despite having space.
I think it would be good if we could specify our own disk space requirement. Maybe we're plotting bigger work units, or maybe we just need to reserve space for other things.
I also think we should have an option to check the temp drive for space. Because:
a) the temp drive can also run out of space which is bad
b) if the temp drive and destination are the same, then we'd need more space
c) to accomodate for other things going on in that drive. like other chia plotters or regular productivity usage.
Hello,
I would like to suggest a feature.
For the plotng to copy created plots in orderly manner one by one to designated HDD or HDD's.
Right now many people use an SSD as "buffer" to speed up plotting [TargetDirectory], when plot files created they saved to this SSD [saves up to 20 minutes per p lot]
Also when bunch of parallel plots finish their jobs together and start copying to HDD [most people fill one HDD at time] it creates a bottleneck and slows the speed even more, but if you have an SSD, it doesnt really that big of a diffrence if its one file or 3 files copied to it, its still plenty fast.
Up till now I was manually copying the plot file from the "buffer" SSD to the HDD.
Also if its posible to RENAME the plot files while it copies, becuase the Chia Farming has a bug when you copy file it sees it as defective untill you restart the farming gui.
To solve this, i just rename .plot files to .pl before I copy, and when they copied i rename back to .plot.
This chia-blockchain Wiki page shows a table estimate of progress percentages, could this information be used to display percent/progress to the UI?
https://github.com/Chia-Network/chia-blockchain/wiki/Beginners-Guide#create-a-plot
Phase | Step | % Progress |
---|---|---|
1 | Computing table 1 | 1% |
1 | Computing table 2 | 6% |
1 | Computing table 3 | 12% |
1 | Computing table 4 | 20% |
1 | Computing table 5 | 28% |
1 | Computing table 6 | 36% |
1 | Computing table 7 | 42% |
2 | Backpropagating on table 7 | 43% |
2 | Backpropagating on table 6 | 48% |
2 | Backpropagating on table 5 | 51% |
2 | Backpropagating on table 4 | 55% |
2 | Backpropagating on table 3 | 58% |
2 | Backpropagating on table 2 | 61% |
3 | Compressing tables 1 and 2 | 66% |
3 | Compressing tables 2 and 3 | 73% |
3 | Compressing tables 3 and 4 | 79% |
3 | Compressing tables 4 and 5 | 85% |
3 | Compressing tables 5 and 6 | 92% |
3 | Compressing tables 6 and 7 | 98% |
4 | Write checkpoint tables | 100% |
Thanks for this great utility!
--Tony
Hi, thanks for your great work.
Since I use multiple plotting machines, can I start 1 server on every machine and connect the same UI on my Desktop to multiple servers?
This would be amazing.
I've been experimenting with different DelaysBetweenPlot values, and suddenly, even when set to 0 or a low value, plotng wouldn't start a new plot.
I looked at the code, I suspect that when DelaysBetweenPlot was set to a high value, createNewPlot() would set server.targetDelayStartTime to that time in the future, in one of these lines:
Line 76 in c8e4bb4
Line 97 in c8e4bb4
If now a lower value is set, createNewPlot() will still always abort here:
Line 70 in c8e4bb4
... never taking into account a changed DelaysBetweenPlot. maybe server.targetDelayStartTime should be reset on config or value reload?
Hello,
I get this message when starting the plotter on mac os big sur. tested on two installations. Unfortunately the message doesn't provide any infos besides that:
Plotting Exit with Error: exit status 255
Tried with different config.json variations, but no luck to getting it work. Plotting with original chia plotter commands works although.
Thank you!
Phase 1 plots limit wasn't working, but it is working now. The problem is plot timers (delay+stagger) are not working properly. See other comments below. (Updated)
Would be cool to be able to store the plot data in a file so the user can analyze that themselves or run their own scripts against that data.
The pane in the -ui option is a nice feature but sometimes you just need the actual data in a file.
I try it on ubuntu linux, and can't get it to work.
When start in gui mode, there is Log message: Get "http://localhost:8484/": dial tcp 127.0.0.1:8484 connect: connection refused.
With no gui there is only a message "Plotting exit with error: exit status 1"
Option to write a log per plot into a specified directory (like the Chia Windows GUI does).
The last x lines per plot on the GUI are good - i would love to have the option for whole log files for each plot tho!
Might be a problem on my side, but chia generates absolutely no logs for me, even tho i have set logging to "info"
Thus it would be much appreciated if your tool would just make a "> {plot_start_time}_log.txt" for each plot :)
#if Chia usually already does this, then ofc. it won't be neccesary and i just have a problem on all my servers
Besides that - great tool, thanks for your awesome work!
plotng is working great for me. I'm using it to do 3 plots in parallel with a 30m stagger. Over time, the plotting looks like this:
##### ##### ##### #####
##### ##### ##### #####
##### ##### ##### #####
As that diagram shows, there's wasted time/resources at the beginning and end of each cycle where it's only doing 1 or 2 plots in parallel. Ideally, I'd like plotng to do this…
##### ##### ##### ##### ##### #####
##### ##### ##### ##### ##### #####
##### ##### ##### ##### ##### #####
…but I haven't figured out the combination of settings that would keep 3 plots going at all time. Is anyone doing this, and if so how did you configure it?
Hi. I would prefer to not have plotng getting Internet traffic and would much prefer to bind it to localhost only, can you please add support for that? Thanks :)
Hey, great tool! Thanks for sharing it.
Would be nice if -r (number of threads, currently defaulted to r=2) and -b (memory per plot, currently hardcoded to 6000) could be made configurable in the .json file.
Thanks :)
Currently, it seems that plotng round-robins the specified temp directories for each new plot process. Since the temp directories might be on drives with different amounts of free space, this can lead to incorrect numbers of parallel plots per drive. One possible solution is to have a parallel array in the config:
...
"NumberOfParallelPlots": 10,
"TempDirectory": ["/mnt/2tb", "/mnt/1tb"],
"TempDirectoryLimits": [7, 3],
...
Or, re-structure the config similar to how plotman allows specifying tmp_overrides
options. I can help make these changes but want to make sure you like the design/config structure first.
Edit: MaxActivePlotPerTemp seems like a good start, but this still requires all the temp dirs to have the same free space.
Seeing reports from users, saying the active plot table now show '-1/4' phase and -1% progress. Can you double check.
Thanks
I run PlotNG UI on a portrait ultrawide monitor, and it'd be nice to be able to see the full stats in this scenario - right now, post multi-host-ui (which is wonderful BTW, thank you), my screen cuts off around the middle of "end time" in archived plots, preventing me from seeing durations of archived plots.
If the Plot ID, which is mostly irrelevant anyway, was visually truncated, say to 10 characters (which is more than enough to uniquely ID a plot using file search or whatever, in normal usage), I could almost definitely see everything at once on even this narrow screen.
I had some plots end with an Errored
status (not sure why yet) and the temp files caused some ongoing plotting issues. It would be cool if plotng knew to delete the tmp files from the temp drive in that case.
In the process of making columns sortable, and adding a couple of other features I wanted, I've basically re-architected the entire UI.
Personally I think it's a cleaner design, but obviously I'm biased. Would you be open to a set of PRs changing how a lot of the UI is managed? Each panel can be updated in a self contained PR to keep them easier to review.
There's also a few changes around race conditions and security (specifically not listening on 0.0.0.0 by default).
This is my "Helper Plotter" its my main PC and it only plots when I use it to help the main one.
Here are my settings, it set to start all 4 plots with 5 minute delay, but as you can see from the screenshot, the last one was started with more then hour delay.
The I drive where the 3 plots run has 878GB of space, its more then enough for 3 plots that at max use 270gb of temp space
[its two 1TB drives in RAID 0 with big overprovisioning so i reduced the size as much as posible]
"Threads": 2,
"Buffers": 0,
"NumberOfParallelPlots": 4,
"TempDirectory": ["D:/ChiaTemp", "I:/1", "I:/2", "I:/3"],
"TargetDirectory": ["F:/Chia"],
"StaggeringDelay": 0,
"ShowPlotLog": false,
"DiskSpaceCheck": true,
"DelaysBetweenPlot": 5,
"MaxActivePlotPerTarget": 0,
"DisableBitField": false,
"MaxActivePlotPerTemp": 1,
"MaxActivePlotPerPhase1": 0,
"UseTargetForTmp2": true,
"BucketSize": 0
Any ideas?
Thanks
Hi,
Please add a setting to the config file for the Chia queue system.
Reason: Me for example i start 7=8 parallel plots but each has a queue enabled
This way when plotting ends it starts a new one without killing chia.exe
And I can enable CPU affinity for each Chia.exe, i get each plot its 2 dedicated cores and it stays like this untill plotting ends.
Also it will allow to use the software at measured intervals, say you want to plot just 10 plots before you do soemthing with the PC, you can set a queue and it will end when all plots are done.
Thanks you
Hi. I'm using chia under plotng on a desktop machine, and would welcome an option in config file to give a "nice" priority to chia processes, so that they won't be as heavy penalizers and impact interactive GUI sessions as much. Can you please add this option for a "nice" value (i think they go from 19 to -20), 19 the lowest, and -20 the highest. I'd like to set all chia plotting processes to something like 10 so that the normal 0 priority processes don't get bogged by Chia.
Thanks
Currently it seems that DelaysBetweenPlot and StaggeringDelay are both global, that is to say, they apply across temp drives if more than one temp drive is set, not within a single temp drive.
What I'm trying to do, with 2 TempDirectory entries:
TempDirectory 1:
Max 4 parallel
2 max phase 1
delay 90 min between plots on this drive
TempDirectory 2 would be the same.
The result I'm looking for would be that every 90 minutes, PlotNG fires one plot on each drive, until MaxActivePlotPerPhase1 is met across both drives. After that, when 90 minutes has elapsed and a plot on either drive moves out of Phase 1, a new plot should fire on that drive.
What I'm seeing instead, with either StaggeringDelay or DelaysBetweenPlot, is 1 plot on TempDirectory 1, wait 90 min, 1 plot on TempDirectory2, wait 90 min, 1 plot on TempDirectory 1, wait 90 min... and so on.
Am I screwing up the config here, or is there not a way to do this currently?
Hi - first, thank you for this great tool!
Is there a way to move between sections in the UI so we can scroll down beyond what is displayed? For example, I have 39 "Archived Plots" but I can only see the most recent 7.
I'm plotting on Win10 with below settings, which are function normally with pure CLI.
As you can see I set the NumberOfParallelPlots to 12.
Before it running up to 8 parallel jobs it was great, but no more parallel job is created since then.
I even waited until the the 1st job is completed, and the next 1st job is started, still there are only 8 parallel jobs.
Is this some windows only issue?
{
"Fingerprint": "",
"FarmerPublicKey": "",
"PoolPublicKey": "",
"Threads": 4,
"Buffers": 4000,
"NumberOfParallelPlots": 12,
"TempDirectory": ["E:\p", "F:\p"],
"TargetDirectory": ["D:\p"],
"StaggeringDelay": 2,
"ShowPlotLog": true,
"DiskSpaceCheck": true,
"DelaysBetweenPlot": 20,
"MaxActivePlotPerTarget": 0,
"DisableBitField": false,
"MaxActivePlotPerTemp": 6
}
This is probably not the right place to implement such feature but I am going to make the suggestion anyway...
when you move/copy a large file to mechanical drive, the write speed is going to be the bottleneck. however if copy the file to an SSD first, the disk space on the plotting drives can be freed sooner so that you can start the next plot sooner while maximizing the free space on the plotting drives. while this might sounds similar to -2 argument, it isn't exactly the same as it only deals with the finalized plot instead of involving with phase 3 and 4.
I think it will minimize time to start the next plot given the max active plots constraint.
Plotng should pause or skip devices that are running low on capacity. For example, a destination drive can continue to have plots prepared for it despite not enough free space existing on the drive to store the plots.
Perhaps plotng can calculate how many plots can be made for a destination drive when starting the plotting process, and then automatically stop that queue when the drive is full?
It would be supercool:
ps. im a windows user
I must say that I really enjoy this tool and it has made my first experience using the chia CLI tool so much smoother.
Would it be possible to store a lot of the data your displaying as a running CSV? I want to use this tool to help determine optimal staggering behavior. Having all the nice log data you have displayed in these windows would make it so much easier to do a post plot analysis.
Things like: plot id, start time, end time, plot dir, destination dir, total time for each phase, number of threads used
and whatever else may come to mind.
To prevent overwriting the previous server logs, it may be helpful to use the approximate startup time of the server. Maybe even have a command line option to enable or disable this kind of logging for people using small or fragile boot drives.
Is it possible to fix the UI display in Windows, some of the letter has been over lapped and the boarder frame doesn't show up correctly
add rsyng feature to -d diskt to remote machine
https://github.com/ericaltendorf/plotman/wiki/Archiving
05-03 08:37:46 Starting archive: rsync --bwlimit=80000 --remove-source-files -P /mnt/dst1/plot-k32-2021-05-03-01-50-b4271f88a74b36b516c242151e00fdda20e3f31ce1f8624465bf05a195009ecd.plot rsync://[email protected]:12000/chia/plots2
Copy the part after 05-03 08:37:46 Starting archive: . In our example that would be:
rsync --bwlimit=80000 --remove-source-files -P /mnt/plots1/plot-k32-2021-05-03-01-50-b4271f88a74b36b516c242151
First, thanks for this tool. Your client/server model was a great idea.
Issue: Time seems in seconds instead of minutes for Windows.
Expected behavior: "DelaysBetweenPlot": 90 # in 90 minutes it should kick off the next plot
Actual Behavior: Next plot starts in 90 seconds.
So it could either be a documentation change to let windows users know, or see what the fix might be for windows interpreting the time in seconds vs minutes.
Hello,
Can this feature be considered please?
Example 7 Parallel plots on 7 drives staggered 3 times, with Maximum Phase1 plots set to 7
When one of the first 7 plots finished a new staggered one should start on the drive that Finished Phase 1.
Right now it will start on the next drive on the list, which is the first one since list restarted.
Regards
I just started using plotng
, and my config file sets NumberOfParallelPlots
to 4 with a StaggeringDelay
of 60.
A couple minutes after starting, I see two plotting operations happening. I'd expect one plot to start immediately, and the next to start in 60 minutes, etc. Am I misunderstanding how it should work?
This is on macOS in case it matters. Thank you!
I'm plotting on hdds, which can each manage one simultaneous plot.
Currently, I'm running one plotng instance per hdd, because if I add each hdd as both temp folder and target folder, a plot might be split over two hdd, causing slowdown.
Unfortunately this doesn't allow plotng's awesome staggering/delaying to be applied?
Would it be possible to allow plotting to a single drive as both tmp and target, but having multiple of those?
It would be great to have more possibilities regarding the actual plotting.
One thing would be the possibility to disable bitfield if preferred.
Hello
I wondering whether we could have a different config for each drive because if you have a few drives with different speed maybe you want to have one fast drive with 3 parallel plots and a slow one with just 1 plot. I understand that plotng don't care about the drive's speed and you can't specify any config for that.
Have a good farming
(venv) [root@chia ~]# go install plotng/cmd/plotng
go install: version is required when current directory is not in a module
Try 'go install plotng/cmd/plotng@latest' to install the latest version
I'm not sure what I've done wrong, but this issue appears when following your instructions.
I'd like to plot2 configured to the target folder.
In my setup, that would save one copy process.
edit what would also be nice if chia could plot into some tmp folder, then use a combined tmp2 + output folder. then, plotng could move the file in one of the configured target folders.
I'd like to be able to stagger the plots evenly or by phase.
E.g., have only one plot running in phase 1 (which uses multiple threads).
Or, if I have 5 parallel plots, have them start at whenever the previous arrived at 100/5=20%.
Right now my plots are IO bound, so without manually increasing the number of parallel plots, they pretty much synchronize towards the end. At some point, I'm waiting for 5 plots to get copied at the same time over a slow network.
Thanks for this wonderful time saver!
I use different NVME's (2 x 2TB and 1 x 1TB).
With PlotNG is it not possible to set a "MaxActivePlotPerTemp" for each Temp directory. The result is, that the 1TB NVME run out of space because PlotNG don't recognize that there is less space than an the other two NVME's.
Is it possible to fix that?
Thanks and Regards
Simon
For those of us using https://dashboard.chia.foxypool.io/ it would be great for plotng-managed plots to be visible in the dashboard, especially if we're running multiple machines.
As per felixbrucker/chia-dashboard-satellite#11 on the dashboard dev's repo, plot managers could expose a similar plotting API to the main Chia GUI to allow his software to pick up on it.
version 0.6, regardless of the value, it will always start the next plot in exactly 1 min (parallels plotting 2).
Line 82 in 5ec1e63
It would be nice if you supported -f
& -p
as an alternative to -a
This allows you to create plots without having the private keys on the plotter.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.