dginev / cortex-peripherals Goto Github PK

Some jobs are so large that /dev/shm potentially overflows. Other jobs have dirty dies, which leave files residing in /dev/shm indefinitely.

To mend that I intend to:

• Detect OOM conditions
• Before accepting new jobs, check for /dev/shm worker files older than 2*timeout_limit and remove them
• In cases where we have less than 50MB available, don't accept new jobs and sleep a minute, hoping the situation improves (or as the files age, that we can delete them).

A simple proof-of-concept analysis service for arXiv would annotate each entry with its storage statistics:

• file size
• number of files in job
• line count of tex files
• custom .sty and .cls files

each of which can be an annotation. It would be nice to create reports from these metrics for the thesis document, to e.g. motivate the cortex dispatcher design (which has taken file size into account, among others)

I want to deploy a benchmark service, timed centrally by CorTeX as a "run", which:

• Requests the imported corpus format
• Receives the payload, unpacks it in a RAM disk (/dev/shm), then erases the unpacked data
• Returns a generic success response

While there will inevitably be variance in bandwidth and various resource loads, such a timed run over a large corpus would provide a good estimate of the overhead cost of using the CorTeX flow.

Depends on implementing dginev/CorTeX#13