C/C++ with Gearman (as worker)

The document is very vague about how to register a function. It turned out that the prototype/signature of the function gearman_worker_fn is very import:

typedef void*( gearman_worker_fn)(gearman_job_st *job, void *context, size_t *result_size, gearman_return_t *ret_ptr)

And since all the online document links are broken, I had to create using Doxygen, under folder /usr/local/include/libgearman-1.0

#include <iostream>
#include <libgearman/gearman.h>
#include <cstring>

using namespace std;

 void* gworker_fn_demon(gearman_job_st *job, void *context, size_t *result_size, gearman_return_t *ret_ptr)
{
   auto jobptr = gearman_job_workload(job);//this takes the data from the client
   if (jobptr) std::cout << "job: " << (char*) jobptr << std::endl;
  *ret_ptr = GEARMAN_SUCCESS ;
  *result_size = 6;

  cout<<"job received"<<endl;
  //char* result = new char[100];//bug here: free/new mismatch reported from valgrind
char* result = (char*)malloc(100 * sizeof(char));
for(int ii = 0; ii < 4; ++ii){
  result[ii] = 'a' + ii;
 }
 result[4] = '\n';
 result[5] = '\0';
 return result;//the memory is freed at client side

 }


int main()
{
 auto status_print = [](gearman_return_t gserver_code){
   cout<<gserver_code<< " --  "; 
   if (gearman_success(gserver_code)) cout<<"success";
   if (gearman_failed(gserver_code)) cout<<"failed";
   if (gearman_continue(gserver_code)) cout<<"continue";
   cout<<endl;
 };

 gearman_worker_st* gworker = gearman_worker_create(NULL);
 

 
 const char* ghost = "127.0.0.1";
 in_port_t gport = 4730;

 gearman_return_t gs_code = gearman_worker_add_server(gworker, ghost, gport);

 status_print(gs_code);


 const char* function_name = "wwcc";
 unsigned timeout = 0;
 void * job_context = NULL;

 gs_code = gearman_worker_add_function(gworker,function_name, timeout, gworker_fn_demon, job_context);

 status_print(gs_code);



 gs_code = gearman_worker_work(gworker);

 status_print(gs_code);

 

 gearman_worker_free(gworker);


 cout<<"done"<<endl;
 return 0;
}

Python with Gearman (as worker)

http://pythonhosted.org/gearman/worker.html

#!/usr/bin/python
import gearman
import time
def check_request_status(job_request):
    if job_request.complete:
        print ("Job finished! ")
        print (job_request.result)
    elif job_request.timed_out:
        print ("Job timed out!")
    elif job_request.state == JOB_UNKNOWN:
        print ("Job connection failed!" )

gm_worker = gearman.GearmanWorker(['localhost:4730'])

def task_listener(gearman_worker, gearman_job):
    return gearman_job.data + ' from listener\n'

gm_worker.set_client_id('whatever iid');
gm_worker.register_task('wwcc', task_listener)

gm_worker.work()

C/C++ with gearman (as client)

gearman library (libgearman) can be directly called in C/C++ programs. (gearmand --version 1.1.5)

http://gearman.info/libgearman/examples.html

And my makefile:

 a.out:callrev.cpp
           clang++ -o $@ $^ -std=c++11 -lgearman

Together with the worker command from last post, here is a complete C++ program as a client:

#include <libgearman/gearman.h>
#include <iostream>
#include <cstring>
using namespace std;

int main(){
 gearman_client_st* gclient = gearman_client_create(NULL);
 gearman_return_t gsc = gearman_client_add_server(gclient, "127.0.0.1",4730);

 auto status_print = [](gearman_return_t gserver_code){
  cout<<gserver_code<< " --  "; 
  if (gearman_success(gserver_code)) cout<<"success";
  if (gearman_failed(gserver_code)) cout<<"failed";
  if (gearman_continue(gserver_code)) cout<<"continue";
  cout<<endl;
 };

 status_print(gsc);

 const char* function_name = "wwcc";

 const char* unique = "whatever unique";
 const char* workload = "aa bb cc";
 size_t workload_size = strlen(workload);
 size_t result_size;
 gearman_return_t return_code;
 void* value = gearman_client_do(gclient, function_name, unique, workload, workload_size, &result_size, &return_code);

 status_print(return_code);
 const char* result = static_cast<char*>(value);
 cout<<result<<endl;


 free(value);
 gearman_client_free(gclient);
 
 return 0;
}

Python with Gearman (as client)

My planned structure: C++ as the worker in the backend, Python as the client at the front end, and Gearman is the agent in between.

Now test the front end.

1. Start up gearmand server and and sample worker. (word counter, from last post)

2. Install python-gearman module:
http://www.geeksww.com/tutorials/web_development/python/installation/how_to_install_python_gearman_client_module.php

3. Write a test python file:
http://www.saltycrane.com/blog/2010/04/notes-using-gearman-with-python/
http://pythonhosted.org/gearman/client.html

#!/usr/bin/python
import gearman
def check_request_status(job_request):
    if job_request.complete:
        print ("Job finished! ")
    elif job_request.timed_out:
        print ("Job timed out!")
    elif job_request.state == JOB_UNKNOWN:
        print ("Job connection failed!" )

gm_client = gearman.GearmanClient(['localhost:4730', 'otherhost:4730'])

# See gearman/job.py to see attributes on the GearmanJobRequest
# Defaults to PRIORITY_NONE, background=False (synchronous task), wait_until_complete=True
completed_job_request = gm_client.submit_job("wwcc", "arbitrary binary data")
check_request_status(completed_job_request)

Here the #! version should be matching the default python used by gearman module. I typed #!/usr/bin/python3.3, which would not work.

Start with Gearman

1. Install gearman, server and client:

$ sudo apt-get install gearman-job-server 

2. Creat gearmand log file:

$ mkdir ~/log/gearmand
$ touch ~/log/gearmand/gearmand.log
$ cd ~/log/gearmand

3. Start gearman server on localhost (127.0.0.1), port 4730

$ gearmand --log-file gearmand.log --listen 127.0.0.1 --port=4730 --verbose=INFO

4. Register gearman worker (with funciton name wwcc):

$ gearman -w -h 127.0.0.1 -p 4730 -f wwcc -- wc -l

5. Send data/request from a gearman client:

$  gearman -h 127.0.0.1 -p 4730 -f wwcc < /etc/passwd
$  gearman -h 127.0.0.1 -p 4730 -f wwcc "hello world"

6. The stdout on terminal shows the result, something like:

 48      78     2409
  0        2         11

For default localhost:4730, step 3 can be simplified to, (in addtion, running in background):

$gearmand -d -l gearmand.log

http://gearman.org/getting_started
http://stackoverflow.com/questions/6995202/problem-with-gearman-gearman-could-not-connect
http://stackoverflow.com/questions/14526086/gearman-issues-php-cli

$man gearman
$man gearmand
$gearmand -h
$man gearadmin

Related error messages:

gearmand: Could not open log file "/usr/local/var/log/gearmand.log", from "/home", switching to stderr. (Permission denied)

This means the access to gearmand.log file is limited (owned by root?). Creating a new log file solves this problem (step 2 above).

gearman: gearman_client_run_tasks : flush(GEARMAN_COULD_NOT_CONNECT) localhost:0 -> libgearman/connection.cc:672

The gearmand server is not running, or binds to a different address:port than requested. (step 3 above)

gearmand: unknown option -v

The -v option is not suppored in new versions. use: --verbose=INFO (step 3 above)

  ERROR 2013-04-13 22:15:55.000000 [  main ] Timeout occured when calling bind() for 0.0.0.0:4730 -> libgearman-server/gearmand.cc:616

The listen to address is bad. Specify localhost by -h 127.0.0.1 (step3,4,5 above)

gearmand: error while loading shared libraries: libboost_program_options.so.1.48.0: cannot open shared object file: No such file or directory

$sudo apt-get install libboost-program-options1.48.0
After some updates, the old boost lib was autoremoved. Just reinstall it and problem solved.