Saturday, January 7, 2017

Python-rq example

Introduction:
This post is about Python-RQ and job distribution. Python-RQ is a python library that utilizes redis queues to queue jobs. These jobs can be processed by multiple workers you have on the node. This library is very simple to utilize.

Using this library consists of three parts, which are Redis, job creator/distributors, and workers.
Redis is used to make queues for jobs. Both job creators and workers access Redis.
You have your job creation/distribution script, which puts jobs/tasks on Redis queue.
Your worker nodes take the jobs/tasks from Redis and perform their tasks.

Below you can see what one of the ways to set this up is:

You can have Redis and Job Creator and Worker run on the same machine or separate it all out depending on what your needs are.

Problem:
I needed to analyze bunch of binary files quickly and besides that, I just wanted to have infrastructure in place to let me run my python code distributed.

Installation:
I am using Proxmox to do all of this, in real life, you would probably multiple physical machines. I am using Ubuntu 14.04 containers.

We’ll start by installing redis-server and pip.
Run:
apt-get update
apt-get install python-pip redis-server
pip install rq rq-dashboard
To test installation, run:
redis-server –v
rq info
rq-dashboard --help
If everything installed correctly, you shouldn’t have gotten any errors.

Now you need to change redis configuration to make it bind to 0.0.0.0. (or whatever IP address you want Redis to bind to. FYI, Shodan does scan for Redis...)   Edit /etc/redis/redis.conf. Look for ‘bind 127.0.0.1’ and change it to ‘bind 0.0.0.0’
After changing the settings, run:
service redis-server restart

For your worker node and job creator, you would only install pip and rq, and other dependencies you need for your jobs or tasks. In the above example, I’m putting job distributor and redis on the same machine.

Simple example:
You will need to know IP of your redis server, in my case it’s 10.0.0.32.

On my worker node(s), I have the python script that I want to run.
File name: mod1.py
Code:
def func1(x,y):
   z = (x+y)*x
   return z

On my worker node, in the same directory as my mod1.py, I run the following command to start rqworker:
rqworker -u “redis://10.0.0.32”

On my job distribution container, I can run the following code to create and put jobs on Redis:
#Setup connection
from redis import Redis
from rq import Queue
q = Queue(connection=Redis(‘10.0.0.32’))
#Create a task
results = q.enqueue('mod1.func1',1,2)

Remember that mod1.py is our python file with func1 being the function we want to use from that file. We’re also running rqworker in the same directory as that file.
results.result variable will contain the returned value.


Another thing you installed is rq-dashboard. Rq-dashboard is pretty awesome looking webUI that lets you see your queues, jobs, and workers that are connected. If you have any jobs that fail, you can requeue them or cancel them from the webUI.

You can run it by typing rq-dashboard then visit http://IP_OF_SERVER:9181/ to view the dashboard.
Here are some screenshots:

Next time (assuming I have more time), I’ll try to cover using python-rq for PE file analysis.

Resources:

16 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Thank you for sharing wonderful information who wants for this type information
    python training in Hyderabad the best career

    ReplyDelete
  3. Thanks for sharing information post. Limousine King offers affordable limo hire Melbourne that make your accosion special. Wedding Car HireLimo Hire Melbourne prices

    ReplyDelete
  4. Hello Everyone !

    USA Fresh & Verified SSN Leads along with Driving License/ ID Number, AVAILABLE with 99.9% connectivity
    All Leads have genuine & valid information.

    **DETAILS IN LEADS**
    First Name | Last Name | SSN | Dob | Driving License Number | Address | City | State | Zip | Phone Number | Account Number | Payday | Bank Name | Employee Details | IP Address

    *Price for SSN lead $2
    *You can ask for sample before any deal
    *If anyone buy in bulk, we can negotiate
    *Sampling is just for serious buyers

    ==>ACTIVE, FRESH CC & CVV FULLZ AVAILABLE<==
    ->$5 PER EACH

    ->Hope for the long term Business
    ->Interested buyers will be welcome

    **Contact 24/7**
    Whatsapp > +923172721122
    Email > leads.sellers1212@gmail.com
    Telegram > @leadsupplier
    ICQ > 752822040

    ReplyDelete
  5. Eco-friendly Sustainable Hemp Products
    Eco-Friendly Hemp Clothing, Backpacks, Soaps, Pet Supplies, CBD Tinctures and Wellness Products

    Shop Now

    ReplyDelete
  6. Are you ready to give your kitchen or bathroom the gorgeous look it deserves?
    Without spending a fortune? We install granite countertops at affordable prices.
    We want to be your one stop shop for your fabricating and installation of all natural stones,
    Marble, Limestone, Granite, Onyx, Travertine & Slate, including semi-precious stones.
    We make it a priority to deliver the highest quality stone, installation, and fabrication
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces
    Serranos Surfaces

    ReplyDelete
  7. Ucuz, kaliteli ve organik sosyal medya hizmetleri satın almak için Ravje Medyayı tercih edebilir ve sosyal medya hesaplarını hızla büyütebilirsin. Ravje Medya ile sosyal medya hesaplarını organik ve gerçek kişiler ile geliştirebilir, kişisel ya da ticari hesapların için Ravje Medyayı tercih edebilirsin. Ravje Medya internet sitesine giriş yapmak için hemen tıkla: ravje.com

    İnstagram takipçi satın almak için Ravje Medya hizmetlerini tercih edebilir, güvenilir ve gerçek takipçilere Ravje Medya ile ulaşabilirsin. İnstagram takipçi satın almak artık Ravje Medya ile oldukça güvenilir. Hemen instagram takipçi satın almak için Ravje Medyanın ilgili sayfasını ziyaret et: instagram takipçi satın al

    Tiktok takipçi satın al istiyorsan tercihini Ravje Medya yap! Ravje Medya uzman kadrosu ve profesyonel ekibi ile sizlere Tiktok takipçi satın alma hizmetide sunmaktadır. Tiktok takipçi satın almak için hemen tıkla: tiktok takipçi satın al

    İnstagram beğeni satın almak için Ravje medya instagram beğeni satın al sayfasına giriş yap, hızlı ve kaliteli instagram beğeni satın al: instagram beğeni satın al

    Youtube izlenme satın al sayfası ile hemen youtube izlenme satın al! Ravje medya kalitesi ile hemen youtube izlenme satın almak için tıklayın: youtube izlenme satın al

    Twitter takipçi satın almak istiyorsan Ravje medya twitter takipçi satın al sayfasına tıkla, Ravje medya güvencesi ile organik twitter takipçi satın al: twitter takipçi satın al

    ReplyDelete