000			02953nrm a2200337 i 4500
999			_c102492 _d102498
003			MY-KuUP
005			20251125111037.0
006			t||||fr|||| 000 0
007			ta
008			250505t20232023my ||||f |||| s0| 0 eng d
020			_aTHE0010027 (Local) _qHardback
040			_aUMPSA _beng _cUMPSA _erda
090			_aPSM .S66 2023 r Bc.
100	1		_aSoon, Kien Yuan, _eauthor.
245	1	0	_aData lakehouse architecture for self-service data analytics / _cSoon Kien Yuan
264		1	_aKuantan Pahang : _c2023
264		4	_c© 2023
300			_axii, 80 pages : _billustrations ; _e1 CD-ROM.
336			_2rdacontent _atext
337			_2rdamedia _aunmediated
338			_2rdacarrier _avolume
347			_2rda _atext file _bPDF
500			_aCenter for Mathematical Sciences
502			_aBachelor of Applied Science In Data Analytics With Honour -- Universiti Malaysia Pahang – 2023
504			_aIncludes bibliographical references
520	3		_aThis project focuses on designing and implementing a Data Lakehouse architecture to facilitate self-service analytics. The objectives include creating a collaborative analytics environment, streamlining the management of multiple ETL processes, adopting a costeffective and non-proprietary architecture, integrating with BI tools, ensuring high query performance for interactive visualization, enabling data warehousing capabilities, and offering a self-service data discovery and metadata platform. The project followed an iterative development methodology that involves requirement gathering and planning, design, implementation, testing, deployment, and maintenance phases. The logical design consists of six layers which are data ingestion, storage, catalog, semantic, processing, and consumption. The physical design utilized Dremio as the core component, along with Apache Iceberg and Arrow Flight Engine for data format and query processing. The project also adopted an Integrated Multi-Zone Analytics Framework to handle different data tasks and workloads. The implementation was performed using Docker, and the testing validates the achievement of the objectives. Deployment was done on an Azure Kubernetes Service (AKS) cluster, although the deployment of the Dremio cluster is hindered due to budget and resource limitations. Maintenance activities included security, backup, node monitoring, cost usage, configuration tuning, metadata management, and training and support. This project concludes that the objectives have been achieved and suggests some future enhancements, such as using Project Nessie for catalog storage, considering Dremio Enterprise Edition for advanced features, and exploring Databricks and MLflow if expecting extensive machine learning workloads.
610	2	0	_aCenter for Mathematical Sciences _xDissertations
650		0	_aUniversities and colleges _xDissertations
650		0	_aFinal Year Report _xDissertations
942			_2lcc _cPSM