Soon, Kien Yuan creator author. three dimensional object my 2023 2023 monographic eng

electronic

xii, 80 pages : illustrations ; 1 CD-ROM. This project focuses on designing and implementing a Data Lakehouse architecture to facilitate self-service analytics. The objectives include creating a collaborative analytics environment, streamlining the management of multiple ETL processes, adopting a costeffective and non-proprietary architecture, integrating with BI tools, ensuring high query performance for interactive visualization, enabling data warehousing capabilities, and offering a self-service data discovery and metadata platform. The project followed an iterative development methodology that involves requirement gathering and planning, design, implementation, testing, deployment, and maintenance phases. The logical design consists of six layers which are data ingestion, storage, catalog, semantic, processing, and consumption. The physical design utilized Dremio as the core component, along with Apache Iceberg and Arrow Flight Engine for data format and query processing. The project also adopted an Integrated Multi-Zone Analytics Framework to handle different data tasks and workloads. The implementation was performed using Docker, and the testing validates the achievement of the objectives. Deployment was done on an Azure Kubernetes Service (AKS) cluster, although the deployment of the Dremio cluster is hindered due to budget and resource limitations. Maintenance activities included security, backup, node monitoring, cost usage, configuration tuning, metadata management, and training and support. This project concludes that the objectives have been achieved and suggests some future enhancements, such as using Project Nessie for catalog storage, considering Dremio Enterprise Edition for advanced features, and exploring Databricks and MLflow if expecting extensive machine learning workloads. specialized Soon Kien Yuan Center for Mathematical Sciences Bachelor of Applied Science In Data Analytics With Honour -- Universiti Malaysia Pahang – 2023 Includes bibliographical references Center for Mathematical Sciences Dissertations Universities and colleges Dissertations Final Year Report Dissertations THE0010027 (Local) UMPSA 250505 20251125111037.0 eng