Hadoop, Docker, AWS, Partitioning Automation using Python

Here’s our Article on Automation using a Python script. We have created a menu-driven program that can automate Hadoop, Docker, LVM, some services of AWS Cloud, Simple Linear Regression, etc.

We have done the following Tasks as follows:-

Hadoop

• Run any Linux Command Locally & Remotely
• Configure and start Namenode on Local OS / Remote OS
• Configure and start Datanode on Local OS/Remote OS
• Created Virtual Group
• Created Logical Volume
• Contribute Limited Storage to Hadoop Cluster
• Attach More hard-disks to Virtual group dynamically
• Increase Partition Size Dynamically

AWS Cloud

• Configure AWS
• Create a Key Pair
• Create/Describe & Deleted Security Group
• Adding Ingress and Egress rules to existing Security group
• Launch/Describe/Start/Stop/Terminate EC2 Instances
• Create/Describe/delete an EBS volume
• Attach/Detach EBS volume to EC2 Instances
• Create Snapshots of EBS Volume
• Created an S3 bucket accessible to the public
• Inserted the object inside s3 bucket which is accessible to everyone
• Created Cloudfront distribution providing S3 as the origin
• Delete object inside S3 bucket
• Deleted S3 bucket

Docker

• launch Container based on user input
• Automate Apache Webserver inside Container
• Automate Python3 inside the container

Machine Learning

• Predict Salary of the Employee based on Years of Experience — Simple Linear Regression.

Here are some small Demo Videos Based on each of the above tasks we have done

Docker and Web Server Automation

Simple Linear Regression

AWS EBS Automation

Hadoop, Docker, AWS, Partitioning Automation using Python

AWS S3 and CloudFront Automation

Hadoop Automation using Python Script

LVM partition Automation using Python

Here is the Github repo for the same

Do give a clap, if you have liked this. :)

Team Contributions:

1. Rahul H Bhatia — https://www.linkedin.com/in/rahul-h-bhatia-b87483140/
2. Saurabh Suthar - https://www.linkedin.com/in/saurabh-suthar-036120173