Examination of Different Types of Big Data Processing Software

Hui, Shing Feng (2018) Examination of Different Types of Big Data Processing Software. Final Year Project (Bachelor), Tunku Abdul Rahman University College.

Text Hui Shing Feng_FULL TEXT.pdf Restricted to Registered users only Download (1MB)

Abstract

The main purpose of carrying out the project is to inspect different types of big data software and choose one software to further examine it. The main problem with big data is how various software handles the big data. Different software will have different specifications and each of them will provide different functions. The differences of the big data processing software are not readily available and research needs to be done. Therefore, this project will evaluate the effectiveness of big data processing software and provide an insight of the more effective software, MongoDB. The first software experimented is Apache Hadoop, an open-source software framework that outlines the processing of big data and distributed storage of big data. The second software is MongoDB, an open-source software that provides cross-platform and document-oriented database features. It is a NoSQL database program that stores documents in JSON format. The scope of the project is to examine the software aspects such as the flexibility of data query, analysis capability, summarization capability, scalability, processing speed, resiliency, security and stability of the software. The main functions of this project are to enable users to store, process and retrieve data from the big data processing software efficiently. The tools used are MQTT Mosquitto Broker, MQTT Eclipse Paho, Apache Hadoop and MongoDB. The testing criteria is the machine running the project is at optimal health so that the resources needed will be allocated for the application. Apache Hadoop failed to operate on the machine because the minimum resource requirements is not fulfilled. Therefore, MongoDB is focused since it works fine. The strength of the project is to serve as a baseline for any other projects that wish to adapt MongoDB. The weakness of the project is it is unable to provide the detailed differentiations between Hadoop and MongoDB.