02953nrm a2200337 i 4500 0 0 lcc 0 6 10000 10000 2025-05-05 0 CD13562 T000003160 2025-05-05 00:00:00 2025-05-05 PSM 102492 102498 MY-KuUP 20251125111037.0 t||||fr|||| 000 0 ta 250505t20232023my ||||f |||| s0| 0 eng d THE0010027 (Local) Hardback UMPSA eng UMPSA rda PSM .S66 2023 r Bc. Soon, Kien Yuan, author. Data lakehouse architecture for self-service data analytics / Soon Kien Yuan Kuantan Pahang : 2023 © 2023 xii, 80 pages : illustrations ; 1 CD-ROM. rdacontent text rdamedia unmediated rdacarrier volume rda text file PDF Center for Mathematical Sciences Bachelor of Applied Science In Data Analytics With Honour -- Universiti Malaysia Pahang – 2023 Includes bibliographical references This project focuses on designing and implementing a Data Lakehouse architecture to facilitate self-service analytics. The objectives include creating a collaborative analytics environment, streamlining the management of multiple ETL processes, adopting a costeffective and non-proprietary architecture, integrating with BI tools, ensuring high query performance for interactive visualization, enabling data warehousing capabilities, and offering a self-service data discovery and metadata platform. The project followed an iterative development methodology that involves requirement gathering and planning, design, implementation, testing, deployment, and maintenance phases. The logical design consists of six layers which are data ingestion, storage, catalog, semantic, processing, and consumption. The physical design utilized Dremio as the core component, along with Apache Iceberg and Arrow Flight Engine for data format and query processing. The project also adopted an Integrated Multi-Zone Analytics Framework to handle different data tasks and workloads. The implementation was performed using Docker, and the testing validates the achievement of the objectives. Deployment was done on an Azure Kubernetes Service (AKS) cluster, although the deployment of the Dremio cluster is hindered due to budget and resource limitations. Maintenance activities included security, backup, node monitoring, cost usage, configuration tuning, metadata management, and training and support. This project concludes that the objectives have been achieved and suggests some future enhancements, such as using Project Nessie for catalog storage, considering Dremio Enterprise Edition for advanced features, and exploring Databricks and MLflow if expecting extensive machine learning workloads. Center for Mathematical Sciences Dissertations Universities and colleges Dissertations Final Year Report Dissertations lcc PSM