krunch3r76 / entropythief Goto Github PK

the named pipe feature is not supported by Python in windows; however, it appears to be supported in powershell (reference needed). this is being researched along with alternative IPC tools.

references:
https://hackinparis.com/data/slides/2017/2017_Cohen_Gil_The_forgotten_interface_Windows_named_pipes.pdf
https://docs.microsoft.com/en-us/windows/win32/api/winbase/nf-winbase-callnamedpipea
https://rkeithhill.wordpress.com/2014/11/01/windows-powershell-and-named-pipes/

during the mixing and writing of the random bytes to the pipe writer and the view, a shutdown waits until all of it has been written. short-circuiting the sending of the bytes sent to the view could solve the issue but may require adding a separate asyncio communication channel (i.e. queue) which contains only metadata such as events.

additionally, a shared variable can be set so that the controller can recognize whenever Golem is not mid-task.

this should be available and quickly summarized from numerous sources or by referring the reader to section 2.1 at https://software.intel.com/content/www/us/en/develop/articles/intel-digital-random-number-generator-drng-software-implementation-guide.html

when a reader has been attempting to read from the same file descriptor to which a former process was writing and that former process has exited, there appears to be a need to recreate the pipe for the new process to see it (sometimes), ie assign a new file descriptor. this is being investigated.

there are at least a couple first shots to take at this: one, see if asynchronous code running ncurses helps by switching contexts on chunks. two, determine if scrolling off the screen would occur and only write the hind side.

currently it has been necessary to await each task download, but the asynchronous model is designed to attend to many pending io operations (as from other tasks) at once. the problem may be with the callback, and that is being investigated as of this writing (at tag alpha-v7.4)

the assembly routines called from the third party module is apparently creating a memory leak. i am developing a c extension to be used in its place.

when a reader such as the included pipe reader empties the named pipe completely in one read, it appears entropythief does not recognize this. this should be easily fixed soon.

the logic in pipe_writer is causing bytes to be rewritten. it has been mitigated some by removing calls to poll that does not play nice with asyncio. a redesign is in progress.

entropythief will report this:

The model threw the following exception:
ApiException
(400)
Reason: Bad Request
HTTP response headers: <CIMultiDictProxy('Content-Length': '51', 'Content-Type': 'application/json', 'Date': 'Thu, 05 Aug 2021 04:29:01 GMT')>
HTTP response body: {"message":"Insufficient funds to make allocation"}

not sure how i want to handle this generic exception or how i might be proactive about it.

because heroes are cool

planning this enhancement

• it may also benefit from being refactored into python idioms such as generators ..

example: if all tasks fail they are not replaced until all have timed out...

entropythief defies the traditional definition of randomness of having "statistical independence, uniform distribution, and unpredictability". showing if some or all of these criteria are violated may actually prove entropythief is fulfilling its design purpose.

there is utilization on 1 core that often goes to 100% while the pipe is being actively read. this is a suspected entropythief issue that should be resolved by making PipeWriter sleep requiring an asynchronous partial interface (or hopefully not a complete asynchronous redesign) where time is yielded to the cpu without blocking. currently it is "unblocking"

multiple named pipes brings about problems with undefined system hard limits and such when reading hundreds of megabytes of randomness. i have learned much about named pipes and am reprogramming to use a single named pipe (this time as it is meant to be used :-p -- or at least more the unix way having learned) and buffering inside an object.

if the named pipe is being continually read from, the status line won't reflect this real-time during provisioning of new work.
two targets to consider:
TaskResultWriter might be able to communicate this information to the controller
A task itself might have a callback in its data to refresh and inform the controller.

https://yapapi.readthedocs.io/en/latest/api.html#module-yapapi.payload.vm

it is now possible to specify cpu capabilities for the provider's running the vm on the vm object itself. the marketstrategy code can be safely removed and replaced by this.

it may be useful to pause a session in order to make adjustments to worker count and buflen
enhancement in planning