Saturday, April 15, 2017

Distributing binary processing using Python-rq

I highly recommend looking at this first:


Introduction:
In this post, I’ll cover doing distributed binary analysis with python-rq, redis, and NFS share. I’ll keep the analysis really really simple. We’ll be getting some other PE info. If you look at Resources/Similar Projects section. You’ll find similar and more stable projects as well.


Problem:
I have bunch of files I need to analyze and I want to distribute analysis amongst different servers.


Solution:
As mentioned in previous blog post linked above, I’ll be using Python-rq. Python-rq works with Redis. I’ll also be utilizing NFS for sharing files across multiple worker nodes and pefile for getting PE information.


Setup:
I have three machines with Ubuntu 10.04 on them. Two machines are Workers and one machine is Redis server and a job creator. Additionally, I have NFS server where I will have my binary files and python scripts.


WORKER1 - 10.0.0.14
WORKER2 - 10.0.0.8
DIST - 10.0.0.9
NFS Server - 10.0.0.11:/mnt/storage/malware (Using Freenas)


Install screen or tmux on everything!


We’ll need to install required software on DIST.
I ran:
apt-get update && apt-get install -y python-pip redis-server && pip install rq rq-dashboard


And to confirm everything was installed:
redis-server -v
rq info


Edit Redis-server configuration and make it bind to 10.0.0.9.
Config file: /etc/redis/redis.conf
Locate bind 127.0.0.1 and change it 10.0.0.9.




Run the following to restart redis-server:
service redis-server restart


We will also start rq-dashboard on Dist.
Run the following command in a screen session:
rq-dashboard -H 10.0.0.9


It should start a webserver on port 9181.


On Worker1 and Worker2 we’ll need python-pip and rq. Tip: If you’re using VM’s or cloud, just configure one worker node then deploy copies.
I ran:
apt-get update && apt-get install -y python-pip && pip install rq


And to confirm rq was installed correctly:
rqworker -u redis://10.0.0.9:6379
Now we’ll get NFS share setup.
On worker nodes and dist, I created /share directory where I’ll mount the NFS share.
Also, I forgot this earlier. We need NFS client.
Run the following on worker nodes and dist:
apt-get install -y nfs-common


To mount the NFS file system run (if you’re going to be setting up something stable, edit your fstab file):
mount -t nfs 10.0.0.11:/mnt/storage/malware /share


Run the following to make sure NFS share permissions are set correctly:
echo test > /share/test
cat /share/test




On my worker nodes, I need pefile library. I ran:
pip install pefile


I created /share/malware folder. I’ll put my samples in that folder. If you’re looking for samples to play with check out: https://www.reddit.com/r/Malware/comments/615el6/malware_samples/ :-D




Now we can start writing code to do the analysis.


Code:
I recommend installing ipython on one of your worker nodes and dist node. It makes debugging and testing much easier.


I will be putting my code in /share. We will be running rqworker in that directory as well. Putting code on NFS server makes updating easier.


We’ll need code for processing part and job distribution part.


For processing, I have a function that takes in full path to the PE file and looks at some properties and writes the results to a text file. Code I used is from pefile example document (linked below).
My file name is process_pe.py and function for processing the files is procpe(FILENAME).


Now we can work on job distribution code. My code is really simple for now. I’ll take full file path as argument, add it to queue, and not wait for results. (procpe returns True and False)




Analysis:
We need to cd into /share and run rqworker command (you can run rqworker multiple times on the same machine):
rqworker -u redis://10.0.0.9:6379


On dist, we can cd into /share as well and run the following:
for file in /share/malware/*; do python dist.py $file; done


We can see our jobs in rq-dashboard.
After everything is done being processed, we can see results:


Conclusion/Improvements:
This was a good way for me to learn to distribute things using Python. I’m aware of some of the Apache projects that do stuff like this as well. I feel more comfortable with Python right now. Also, setup for this is really simple.


As you can tell already. I’m not a professional programmer and I’m lazy. There are several problems with this design too. I think I can add multithreading or multiprocessing to make workers process more data. Code is on github, feel free to fork and improve it. If you do, please leave a comment.

I do plan on running this at my university and adding more features. I'll keep updating the code on Github.


If there are alternative methods or I made some mistakes in the post, leave a comment.
(As you can see from the timestamps in screenshots, I wrote this at night. Expect typos)


Thanks.


Resources/Similar Projects:







77 comments:

  1. This expert filters through this information with the objective of spotting data, bits of knowledge, patterns, arrangements and so forth. data science course in pune

    ReplyDelete
    Replies
    1. Hello Everyone !

      USA Fresh & Verified SSN Leads along with Driving License/ ID Number, AVAILABLE with 99.9% connectivity
      All Leads have genuine & valid information.

      **DETAILS IN LEADS**
      First Name | Last Name | SSN | Dob | Driving License Number | Address | City | State | Zip | Phone Number | Account Number | Payday | Bank Name | Employee Details | IP Address

      *Price for SSN lead $2
      *You can ask for sample before any deal
      *If anyone buy in bulk, we can negotiate
      *Sampling is just for serious buyers

      ==>ACTIVE, FRESH CC & CVV FULLZ AVAILABLE<==
      ->$5 PER EACH

      ->Hope for the long term Business
      ->Interested buyers will be welcome

      **Contact 24/7**
      Whatsapp > +923172721122
      Email > leads.sellers1212@gmail.com
      Telegram > @leadsupplier
      ICQ > 752822040

      Delete
  2. I feel very grateful that I read this. It is very helpful and very informative and I really learned a lot from it.
    business analytics course in delhi

    ReplyDelete
  3. Attend The Machine Learning course in Bangalore From ExcelR. Practical Machine Learning course in Bangalore Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Machine Learning course in Bangalore.
    Machine Learning course in Bangalore

    ReplyDelete
  4. Thank you so much for helping me out to find the Data analytics course in Mumbai Organisations and introducing reputed stalwarts in the industry dealing with data analyzing & assorting it in a structured and precise manner. Keep up the good work. Looking forward to view more from you.

    ReplyDelete
  5. Such a very useful article. I have learn some new information.thanks for sharing.
    data scientist course in mumbai

    ReplyDelete

  6. Excelr is providing emerging & trending technology training, such as for data science, Machine learning, Artificial Intelligence, AWS, Tableau, Digital Marketing. Excelr is standing as a leader in providing quality training on top demanding technologies in 2019. Excelr`s versatile training is making a huge difference all across the globe. Enable ?business analytics? skills in you, and the trainers who were delivering training on these are industry stalwarts. Get certification on "
    data science course fees in hyderabad"
    and get trained with Excelr.

    ReplyDelete
  7. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.
    Data science course in mumbai

    ReplyDelete
  8. Nice Blog...Very interesting to read this article. I have learn some new information.thanks for sharing.
    ExcelR Mumbai

    ReplyDelete
  9. This is also a very good post which I really enjoyed reading. It is not every day that I have the possibility to see something like this..
    ExcelR data science

    ReplyDelete
  10. Very nice blog here and thanks for post it.. Keep blogging...
    ExcelR data science training

    ReplyDelete
  11. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one.
    data analytics courses in hyderabad

    ReplyDelete
  12. I am looking for and I love to post a comment that "The content of your post is awesome" Great work!
    data science course in mumbai

    ReplyDelete
  13. Awesome blog. I enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work!
    data science course in mumbai

    ReplyDelete
  14. This is also a very good post which I really enjoyed reading. It is not every day that I have the possibility to see something like this..
    data analytics course hyderabad

    ReplyDelete
  15. Nice information, valuable and excellent work, as share good stuff with good ideas and concepts, lots of great information and inspiration, both of which I need, thanks to offer such a helpful information here. data science course

    ReplyDelete
  16. Excellent Blog! I would like to thank for the efforts you have made in writing this post. I am hoping the same best work from you in the future as well. I wanted to thank you for this websites! Thanks for sharing. Great websites!
    Please check ExcelR Data Science Course in Pune with Placement

    ReplyDelete
  17. Very nice blog and articles. I am realy very happy to visit your blog. Now I am found which I actually want. I check your blog everyday and try to learn something from your blog. Thank you and waiting for your new post.
    ExcelR Data Analytics Course

    ReplyDelete
  18. It is imperative that we read blog post very carefully. I am already done it and find that this post is really amazing. ExcelR Data Analytics Course In Pune

    ReplyDelete
  19. Wow what a Great Information about World Day its exceptionally pleasant educational post. a debt of gratitude is in order for the post.Please check ExcelR Data Science Courses

    ReplyDelete
  20. I need to to thank you for this very good read!! I definitely loved every little bit of it. I have you bookmarked to check out new things you post… data science course bangalore

    ReplyDelete
  21. keep up the good work. this is an Ossam post. This is to helpful, i have read here all post. i am impressed. thank you. this is our data analytics course in mumbai
    data analytics course in mumbai | https://www.excelr.com/data-analytics-certification-training-course-in-mumbai

    ReplyDelete
  22. I have to search sites with relevant information on given topic and provide them to teacher our opinion and the article.

    machine learning course

    artificial intelligence course in mumbai

    ReplyDelete
  23. This is a wonderful article, Given so much info in it, Thanks for sharing. CodeGnan offers courses in new technologies and makes sure students understand the flow of work from each and every perspective in a Real-Time environmen python training in vijayawada. , data scince training in vijayawada . , java training in vijayawada. ,

    ReplyDelete
  24. Great post i must say and thanks for the information. Education is definitely a sticky subject. However, is still among the leading topics of our time. I appreciate your post and look forward to more.

    Dental care for kids

    ReplyDelete
  25. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!. business analytics certification

    ReplyDelete
  26. Impressive! I finally found a great post here. Nice article on data science . It's really a nice experience to read your post. Thanks for sharing your innovative ideas to our vision.
    Data Science Course
    Data Science Course in Marathahalli
    Data Science Course Training in Bangalore

    ReplyDelete
  27. Enjoyed reading this article throughout.Nice post! IoT is the trendy course right now and is going to be in
    a great demand in near future as jobs for this domain will be sky rocketted.To be on par with the current trend we have to
    gain complete knowledge about the subject. For the complete course online
    360Digitmg Iot Certification Training
    360Digitmg Internet of things courses online

    ReplyDelete
  28. I just got to this amazing site not long ago. I was actually captured with the piece of resources you have got here. Big thumbs up for making such wonderful blog page!. data science courses

    ReplyDelete
  29. This is a wonderful article. I really enjoyed reading this article. Thanks for sharing such detailed information.
    Data Science Course
    Data Science Course in Marathahalli

    ReplyDelete
  30. The information provided on the site is informative. Looking forward for more such blogs. Thanks for sharing .
    Artificial Inteligence course in Delhi
    AI Course in Delhi

    ReplyDelete
  31. Attend The Artificial Intelligence course From ExcelR. Practical Artificial Intelligence course Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Artificial Intelligence course.
    Artificial Intelligence course

    ReplyDelete
  32. A good blog always comes-up with new and exciting information and while reading I have feel that this blog is really have all those quality that qualify a blog to be a one.
    data scientist course in pune with placement

    ReplyDelete
  33. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions

    ReplyDelete
  34. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science traning

    ReplyDelete
  35. It’s really great information for becoming a better Blogger. Keep sharing, Thanks...
    Data Science Training In Hyderabad

    ReplyDelete
  36. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data science Interview Questions
    Data Science Course

    ReplyDelete
  37. This comment has been removed by the author.

    ReplyDelete
  38. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data scientist course in hyderabad with placement

    ReplyDelete
  39. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data sciecne course in hyderabad

    ReplyDelete
  40. Thank you a lot for providing individuals with a very spectacular possibility to read critical reviews from this site.
    Python Course in Hyderabad
    Python Training in Hyderabad
    Python Institute in Hyderabad

    ReplyDelete
  41. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
    data analytics course in Bangalore

    ReplyDelete
  42. Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.
    Course in Data Analytics

    ReplyDelete
  43. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries. keep it up.
    data analytics course in Bangalore

    ReplyDelete
  44. wonderful article. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article resolved my all queries.
    Data Science Course

    ReplyDelete
  45. Superb Information, I really appreciated with it, This is fine to read and valuable pro potential, I really bookmark it, pro broaden read. Appreciation pro sharing. I like it. ExcelR Data science Course In Pune

    ReplyDelete
  46. Attend The Course in Data Analytics From ExcelR. Practical Course in Data Analytics Sessions With Assured Placement Support From Experienced Faculty. ExcelR Offers The Course in Data Analytics.
    Course in Data Analytics

    ReplyDelete

  47. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspried me to read more. keep it up.
    Correlation vs Covariance

    ReplyDelete
  48. This was definitely one of my favorite blogs. Every post published did impress me. ExcelR Data Science Course In Pune

    ReplyDelete
  49. This material makes for great reading. It's full of useful information that's interesting,well-presented and easy to understand. I like articles that are well done.
    Best Data Science training in Mumbai

    Data Science training in Mumbai

    ReplyDelete
  50. Thank you so much for this incredible guide. This has given me so much information
    AI Training in Hyderabad

    ReplyDelete
  51. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science course

    ReplyDelete
  52. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  53. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  54. Very interesting blog. Many blogs I see these days do not really provide anything that attracts others, but believe me the way you interact is literally awesome.You can also check my articles as well.

    Data Science In Banglore With Placements
    Data Science Course In Bangalore
    Data Science Training In Bangalore
    Best Data Science Courses In Bangalore
    Data Science Institute In Bangalore

    Thank you..

    ReplyDelete
  55. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression

    ReplyDelete
  56. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  57. This Was An Amazing ! I Haven't Seen This Type of Blog Ever ! Thankyou For Sharing, data science courses

    ReplyDelete

  58. After reading your article I was amazed. I know that you explain it very well. And I hope that other readers will also experience how I feel after reading your article.
    artificial intelligence course

    ReplyDelete
  59. I am really enjoying reading your well written articles. It looks like you spend a lot of effort and time on your blog. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.

    Simple Linear Regression

    Correlation vs Covariance

    ReplyDelete
  60. I have express a few of the articles on your website now, and I really like your style of blogging. I added it to my favorite’s blog site list and will be checking back soon…
    Data Scientist Courses This is a great inspiring article.I am pretty much pleased with your good work.You put really very helpful information...

    ReplyDelete
  61. This is a wonderful article, Given so much info in it, These type of articles keeps the users interest in the website, and keep on sharing more ... good luck.
    business analytics certification

    ReplyDelete
  62. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!
    artificial intelligence course in bangalore

    ReplyDelete
  63. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  64. I like viewing web sites which comprehend the price of delivering the excellent useful resource free of charge. I truly adored reading your posting. Thank you!

    data science interview questions

    ReplyDelete
  65. Very interesting blog Thank you for sharing such a nice and interesting blog and really very helpful article.
    Data Science Course in Hyderabad

    ReplyDelete
  66. Very interesting to read this article.I would like to thank you for the efforts you had made for writing this awesome article. This article inspired me to read more. keep it up.
    Correlation vs Covariance
    Simple linear regression
    data science interview questions

    ReplyDelete
  67. Hello Everyone !

    USA Fresh & Verified SSN Leads along with Driving License/ ID Number, AVAILABLE with 99.9% connectivity
    All Leads have genuine & valid information.

    **DETAILS IN LEADS**
    First Name | Last Name | SSN | Dob | Driving License Number | Address | City | State | Zip | Phone Number | Account Number | Payday | Bank Name | Employee Details | IP Address

    *Price for SSN lead $2
    *You can ask for sample before any deal
    *If anyone buy in bulk, we can negotiate
    *Sampling is just for serious buyers

    ==>ACTIVE, FRESH CC & CVV FULLZ AVAILABLE<==
    ->$5 PER EACH

    ->Hope for the long term Business
    ->Interested buyers will be welcome

    **Contact 24/7**
    Whatsapp > +923172721122
    Email > leads.sellers1212@gmail.com
    Telegram > @leadsupplier
    ICQ > 752822040

    ReplyDelete