samarunraj / lifeguard Goto Github PK

Hi,

The AddWork and SetState classes both call getClassloader instead of
getClassLoader causing compiler errors.

Patch attached.

I have made my own translation of lifeguard to ruby, but I have followed
the exact same logic.  I am seeing a consistent problem, and I want to know
if others see it in lifeguard.

My pool processors
1) remove a job request from the queue
2) send "busy" status
3) do the work
4) send "idle" status
5) go back to (1)

When there is work backed up in the queue, the idle status is followed very
closely by busy.  
I see SQS *consistently* deliver these requests in the reverse order that
they were sent.  This messes up the computation of the load average.

To overcome this, I modified my processor to send status only when it
really goes idle (no more messages in queue) or busy.  This approach seems
to be required also if my pool consists internally of many threads, each
working independently.  They cannot *each* follow the algorithm above
independently.

But with this scheme (reporting overall idle/busy when it changes), the
load average computation does not work very well.

This led me to think about the way the load average was being done in
general.  I am not sure that the approach here is rally valid.  
If I sit idle for 10 minutes, and then get a constant stream of work, it
takes 10 minutes to get to 50%.  If I sit idle for 30 minutes, it takes 30
minutes to get to 50%.  

I was thinking that the load average should re evaluated once a minute
(bump interval), not on receipt of messages. When a status message is
received, the idle/busy status of an instance is saved, and at the bump
interval, compute:
  if busy new_load = 100 else new_load = 0
  load = old_load * 0.5 + new_load * 0.5 

This way the load average rises gradually and falls gradually.
You could make it rise and fall more quickly by using
  load = old_load * 0.25 + new_load * 0.75

I followed the instructions. And i ran the ingest.sh file

I keep getting the following errors.

[root@domU-12-31-39-00-51-B7 123]# ./ingest.sh Project batch001 H-San
conf/TextToPdf.xml 123.txt
Exception in thread "main" java.lang.NoClassDefFoundError:
com/directthought/lifeguard/ingestor/FileIngestor
Caused by: java.lang.ClassNotFoundException:
com.directthought.lifeguard.ingestor.FileIngestor
        at java.net.URLClassLoader$1.run(Unknown Source)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClass(Unknown Source)
        at java.lang.ClassLoader.loadClassInternal(Unknown Source)

Can someone please let me know what i should do to resolve this issue.

Thanks
hareem.

Could some one please tell me how to setup lifeguard to transcode my videos.

Regards
Hareem.

The current source depends on it but only the class files from the jar are 
currently available.  Is it possible to release the source?

What steps will reproduce the problem?
1. Run PoolManager from r91 of lifeguard (SQS2 upgrade), with RECEIVE_COUNT
> 10 in StatusLogger.java or receiveCount >10 in PoolManager.java or
receiveCount in conf/beans.xml

What is the expected output? What do you see instead?
Expect standard queue processing, instead get:
SEVERE: Error reading message, Retrying.
com.xerox.amazonws.sqs2.SQSException: Client error :
(InvalidParameterValue) Value 20 for parameter MaxNumberOfMessages is
invalid. Reason: Must be between 1 and 10, if provided.
        at
com.xerox.amazonws.sqs2.MessageQueue.makeRequestInt(MessageQueue.java:483)
        at
com.xerox.amazonws.sqs2.MessageQueue.receiveMessages(MessageQueue.java:245)
        at
com.xerox.amazonws.sqs2.MessageQueue.receiveMessages(MessageQueue.java:182)
        at com.directthought.lifeguard.StatusLogger.run(StatusLogger.java:80)
        at java.lang.Thread.run(Thread.java:595)
 -or-
com.xerox.amazonws.sqs2.SQSException: Client error :
(InvalidParameterValue) Value 20 for parameter MaxNumberOfMessages is
invalid. Reason: Must be between 1 and 10, if provided.
        at
com.xerox.amazonws.sqs2.MessageQueue.makeRequestInt(MessageQueue.java:483)
        at
com.xerox.amazonws.sqs2.MessageQueue.receiveMessages(MessageQueue.java:245)
        at
com.xerox.amazonws.sqs2.MessageQueue.receiveMessages(MessageQueue.java:182)
        at com.directthought.lifeguard.PoolManager.run(PoolManager.java:212)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:6
50)
        at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
        at java.lang.Thread.run(Thread.java:595)


What version of the product are you using? On what operating system?
r91 from svn (with some local modifications) on Scientific Linux 4.7

Please provide any additional information below.
Changing:
 RECEIVE_COUNT = 10 in StatusLogger.java
 receiveCount >10 in PoolManager.java
 receiveCount >10 in conf/beans.xml
 can be used as a work around.

The header String does not contain a comma after "InputType" which results 
in the InputType and OutputBucket names being combined, and the rest of 
the fields being off-by-one (when viewing in MS Excel).