Python with Gearman (as client)

My planned structure: C++ as the worker in the backend, Python as the client at the front end, and Gearman is the agent in between.

Now test the front end.

1. Start up gearmand server and and sample worker. (word counter, from last post)

2. Install python-gearman module:
http://www.geeksww.com/tutorials/web_development/python/installation/how_to_install_python_gearman_client_module.php

3. Write a test python file:
http://www.saltycrane.com/blog/2010/04/notes-using-gearman-with-python/
http://pythonhosted.org/gearman/client.html

#!/usr/bin/python
import gearman
def check_request_status(job_request):
    if job_request.complete:
        print ("Job finished! ")
    elif job_request.timed_out:
        print ("Job timed out!")
    elif job_request.state == JOB_UNKNOWN:
        print ("Job connection failed!" )

gm_client = gearman.GearmanClient(['localhost:4730', 'otherhost:4730'])

# See gearman/job.py to see attributes on the GearmanJobRequest
# Defaults to PRIORITY_NONE, background=False (synchronous task), wait_until_complete=True
completed_job_request = gm_client.submit_job("wwcc", "arbitrary binary data")
check_request_status(completed_job_request)

Here the #! version should be matching the default python used by gearman module. I typed #!/usr/bin/python3.3, which would not work.

Start with Gearman

1. Install gearman, server and client:

$ sudo apt-get install gearman-job-server 

2. Creat gearmand log file:

$ mkdir ~/log/gearmand
$ touch ~/log/gearmand/gearmand.log
$ cd ~/log/gearmand

3. Start gearman server on localhost (127.0.0.1), port 4730

$ gearmand --log-file gearmand.log --listen 127.0.0.1 --port=4730 --verbose=INFO

4. Register gearman worker (with funciton name wwcc):

$ gearman -w -h 127.0.0.1 -p 4730 -f wwcc -- wc -l

5. Send data/request from a gearman client:

$  gearman -h 127.0.0.1 -p 4730 -f wwcc < /etc/passwd
$  gearman -h 127.0.0.1 -p 4730 -f wwcc "hello world"

6. The stdout on terminal shows the result, something like:

 48      78     2409
  0        2         11

For default localhost:4730, step 3 can be simplified to, (in addtion, running in background):

$gearmand -d -l gearmand.log

http://gearman.org/getting_started
http://stackoverflow.com/questions/6995202/problem-with-gearman-gearman-could-not-connect
http://stackoverflow.com/questions/14526086/gearman-issues-php-cli

$man gearman
$man gearmand
$gearmand -h
$man gearadmin

Related error messages:

gearmand: Could not open log file "/usr/local/var/log/gearmand.log", from "/home", switching to stderr. (Permission denied)

This means the access to gearmand.log file is limited (owned by root?). Creating a new log file solves this problem (step 2 above).

gearman: gearman_client_run_tasks : flush(GEARMAN_COULD_NOT_CONNECT) localhost:0 -> libgearman/connection.cc:672

The gearmand server is not running, or binds to a different address:port than requested. (step 3 above)

gearmand: unknown option -v

The -v option is not suppored in new versions. use: --verbose=INFO (step 3 above)

  ERROR 2013-04-13 22:15:55.000000 [  main ] Timeout occured when calling bind() for 0.0.0.0:4730 -> libgearman-server/gearmand.cc:616

The listen to address is bad. Specify localhost by -h 127.0.0.1 (step3,4,5 above)

gearmand: error while loading shared libraries: libboost_program_options.so.1.48.0: cannot open shared object file: No such file or directory

$sudo apt-get install libboost-program-options1.48.0
After some updates, the old boost lib was autoremoved. Just reinstall it and problem solved.

create library in C/C++

http://www.yolinux.com/TUTORIALS/LibraryArchives-StaticAndDynamic.html

Mount USB external drive on Linux

1. Plug in the USB disk.

2. Under
/dev/disk/by-label
find the device info, e.g.
Seagate\x20Backup\x20Plus\x20Drive -> ../../usb_bk

(or, simple $ lsusb)

3. sudo makedir /mnt/usb_bk

4. cd ../..

5. mount sdb1 /mnt/usb_bk

Now, the disk shows up under /mnt/usb_bk

6. (optional)
To make it auto mount, first confirm the mount parameters:
$mount
 it should show the mount type:

/dev/sdb1 on /mnt/usb_bk type fuseblk (rw,nosuid,nodev,allow_other,blksize=4096)

Then edit /etc/fstab

/dev/sdb1       /mnt/usb_bk          fuseblk    defaults        0       0 

http://linuxconfig.org/Howto_mount_USB_drive_in_Linux 

http://myubuntublog.wordpress.com/2009/05/25/auto-mount-usb-hard-drive-at-boot/

ARECA backup tool

1. Install Java
sudo apt-get install openjdk-7-jdk openjdk-7-jre icedtea-7-plugin

2. Download and install areca:
http://www.areca-backup.org/

clang 3.2

compile from source code. Follow the instructions on:
http://clang.llvm.org/get_started.html

Note: the ./configure line is changed to
$ CC=gcc CXX=g++ ../llvm/configure


OR, apt-get install
http://askubuntu.com/questions/261636/how-do-i-backport-install-a-newer-version-of-clang

Recursive global replace

 find ./ -type f -exec \
 sed -i 's/\<map\>/unordered_map/g' {} +

under a folder, do an exact match, and recursively replace to a new word.

The motivation was this: I want to upgrade my source code to C++11. To be more efficient, all the old <map> header and usage, which is log(n), should be replaced by the new hash function in <unordered_map>. But there are other names with prefix or postfix "map", which should not be changed.