bernatp3rs / idl_cpu_pm Goto Github PK
View Code? Open in Web Editor NEWIDL CPU Process Manager
IDL CPU Process Manager
*********************************** ******* CPU_Process_Manager ******* *********************************** The CPU_Process_Manager library allows you to manage parallel computing under IDL. The basic idea of this library is to allow the control over a set of IDL sessions in a flexible way. From the main session, the user can create the cpu_process_manager that will initialize a set of xidl_idlbridges that will control IDL slave sessions. Every session can run as a separate process and can exchange data with the main session by copying IDL variables. If your variables are defined as shared memory segments, all sessions will be able to "see" them at the same time without copying (duplicating) anything. The execution of every session is independent of the other sessions and they will all run asynchronously, allowing overlapping executions (parallel computing). It is very important to note that there some limitations: The cpu_process_manager cannot be used in the IDL virtual machine. Object references cannot be transferred between processes. Only procedures, functions and batch files are allowed to be used with the library. Object methods cannot be called. From the main session, all data is copied in the child processes, which may affect the performance of your application if the amount of data involved is large. The use of shared memory segments may offer a more efficient way to share data. Warning: Incorrect use of shared memory routines can corrupt or even crash your IDL application. proper use of these low level operating system features requires systems programming experience, and is not recommended for those without such experience. All version of the IDL_IDLBridge before IDL 8.3 cause a memory leak when using asynchronous executions. See the reference IDL-68399 in the following link: https://www.l3harrisgeospatial.com/Support/Self-Help-Tools/Help-Articles/Help-Articles-Detail/ArtMID/10220/ArticleID/15156/IDL-Fixed-Issues Some users are experiencing problems when using the IDL-Python bridge in the slave sessions. The Python executable needs to be added to the PATH environment variable and this information is not passed by default. To solve this issue, execute the following code to make sure that Python is visible after the general setup: cpu_pm->setup cpu_pm->Execute_All, idlStmt, nowait=nowait Where idlStmt should be something like (this is a Linux solution): spawn, 'setenv PATH /usr/local/anaconda3:${PATH}' spawn, 'setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/usr/local/anaconda3/linux_34x/lib:/usr/local/idl/bin/bin.linux.x86_64' The sample code below contains the complete communications skeleton for a dynamically load balanced master/slave application. Following the code is a description of the few functions necessary to write typical parallel applications using the cpu_process_manager library. The following program add_value is the function to be executed n times and to be parallelized: function add_value, a wait, .1 ; Represents some time of computation return, a+1 end The following routine is the commented example: pro exemple_cpu_pm array=indgen(8,8) ; Result array_1=array array_2=array ; Serial computing case tt=systime(/sec) for i=0, n_elements(array)-1 do array_1[i]=add_value(array[i]) tt1=systime(/sec)-tt ; Parallel computing case tt=systime(/sec) n_cpu=8l ; Initialize and get the process manager cpu_tm=get_cpu_process_manager(n_cpu,/verbose,wait_time=0.05) ; Setup by setting to all sessions to the same working directory, IDL path and ; compiling the same routines as the main IDL session cpu_tm->setup ; Send one unit of work to each session for i=0, n_cpu-1 do task_id=cpu_tm->call_function('add_value',array[i]) ; Loop over getting new work requests until there is no more work to be done for i=n_cpu, n_elements(array)-1 do begin ; Wait until one of the processes finish task_id=cpu_tm->waitend() ; Receive result and save it array_2[task_id]=cpu_tm->getresult(task_id) ; Send a new work unit task_id=cpu_tm->call_function('add_value',array[i]) endfor ; There is no more work to be done, so receive all the outstanding ; results from the slaves for i=0, n_cpu-1 do begin task_id=cpu_tm->waitend() array_2[task_id]=cpu_tm->getresult(task_id) endfor tt2=systime(/sec)-tt ; Compare computation time, serial vs parallel print, 'standard time ', tt1 print, 'cpu_pm time ', tt2 ; Compare the result (the difference shoule be zero) print, total(abs(array_1 - array_2)) end Copyright (c) 2014 Bernat Puigdomenech Treserras (bernat.puigdomenech at gmail.com) This program is free software: you can redistribute it and/or modify it under the terms of the gnu general public license as published by the free software foundation, either version 3 of the license, or (at your option) any later version. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the gnu general public license for more details. You should have received a copy of the gnu general public license along with this program. If not, see http://www.gnu.org/licenses/
When compiling idl_cpu_pm\xidl_idlbridge__define.pro, idl 8.7.2 gives the error:
Compiled module: XIDL_IDLBRIDGE::CALL_BATCH.
key_info=self->processkey(n_extra,_ref_extra=extra)
^
% Keyword _REF_EXTRA is not allowed when calling a routine.
At: D:\Idl\Workbench_01\wgallery_dev2\software\libs\external\idl_cpu_pm\xidl_idlbridge__define.pro, Line 162
% 1 Compilation error(s) in module XIDL_IDLBRIDGE::PROCESSROUTINE.
% Compiled module: XIDL_IDLBRIDGE::PROCESSVAR.
Changing the line:
key_info=self->processkey(n_extra,_ref_extra=extra)
to:
key_info=self->processkey(n_extra, _extra=extra)
removes the compile error.
Apparently the original is in error: the keyword '_ref_extra' is not allowed in the call to a routine, only in the called routine's declaration. However, idl 8.5.2 does not flag the error and my programs using it run fine.
If I make the correction, some of my programs in idl 8.7.2 run fine but others fail. So far I have not been able to debug this.
William Gallery
[email protected]
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.