Performance Analysis of Big Data Workloads in Hybrid Cloud Environments: The Role of File Formats and Bandwidth
Keywords:
columnar storage, Big Data, data locality, hybrid cloud, traffic shaping, tpc-ds, spark, parquet, orcAbstract
This study examines how file format selection (CSV, Parquet, ORC) impacts query performance in a hybrid cloud environment where data remains on-premise while processing occurs in the cloud. Based on TPC-DS benchmarks, our measurements show that columnar formats reduce execution time by over 88% and data transfer by up to 96% compared to CSV. The results indicate that file format selection significantly impacts not only storage efficiency but also network load, especially in distributed hybrid environments.
References
K. Shvachko, H. Kuang, S. Radia, and R. Chansler, “The hadoop distributed file system,” 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies (MSST), pp. 1–10, 2010. [Online]. Available: https://api.semanticscholar.org/CorpusID:13925042
Apache Parquet, Apache Software Foundation. [Online]. Available: https://parquet.apache.org/ (accessed: 2025-05-18)
D. Vohra, “Apache parquet,” in Practical Hadoop Ecosystem: A Definitive Guide to Hadoop-Related Frameworks and Tools. Springer, 2016, pp. 325–335.
Apache ORC, Apache Software Foundation. [Online]. Available: https://orc.apache.org/ (accessed: 2025-05-18)
R. O. Nambiar and M. Pöss, “The making of tpc-ds,” in Very Large Data Bases Conference, 2006. [Online]. Available: https://api.semanticscholar.org/CorpusID:5590663
Apache Ozone, Apache Software Foundation. [Online]. Available: https://ozone.apache.org/ (accessed: 2025-05-18)
Apache Parquet - Data Page Encodings, Apache Software Foundation. [Online]. Available: https://parquet.apache.org/docs/file-format/data-pages/encodings (accessed: 2025-05-18)
M. Pöss, R. O. Nambiar, and D. Walrath, “Why you should run tpc-ds: A workload analysis,” in Very Large Data Bases Conference, 2007. [Online]. Available: https://api.semanticscholar.org/CorpusID:2199276
TPC Benchmark DS Standard Specification, Version 4.0.0, Transaction Processing Performance Council (TPC). [Online]. Available: https://www.tpc.org/TPC Documents Current Versions/pdf/TPC-DS v4.0.0.pdf (accessed: 2025-05-18)
T. Ivanov and M. Pergolesi, “The impact of columnar file formats on sql-on-hadoop engine performance: A study on orc and parquet,” Concurrency and Computation: Practice and Experience, vol. 32, 2019. [Online]. Available: https://api.semanticscholar.org/CorpusID:203159206
A. Ghazal, T. Rabl, M. Hu, F. Raab, M. Poess, A. Crolotte, and H.-A. Jacobsen, “Bigbench: towards an industry standard benchmark for big data analytics,” in ACM SIGMOD Conference, 2013. [Online]. Available: https://api.semanticscholar.org/CorpusID:207202897
Linux IP Routing - ip-route(8), man7.org. [Online]. Available: https://www.man7.org/linux/man-pages/man8/ip-route.8.html (accessed: 2025-05-18)
M. Bosk, M. Gaji’c, S. Schwarzmann, S. Lange, and T. Zinner, “Htbqueue: A hierarchical token bucket implementation for the omnet++/inet framework,” ArXiv, vol. abs/2109.12879, 2021. [Online]. Available: https://api.semanticscholar.org/CorpusID:237940283
Linux Traffic Control: Hierarchy Token Bucket (HTB), man7.org. [Online]. Available: https://man7.org/linux/man-pages/man8/tc-htb.8.html (accessed: 2025-05-18)
Amazon EC2 Instance Types - General Purpose Instances, Amazon Web Services. [Online]. Available: https://docs.aws.amazon.com/ec2/latest/instancetypes/ (accessed: 2025-05-18)
AWS Direct Connect, Amazon Web Services. [Online]. Available: https://aws.amazon.com/directconnect/ (accessed: 2025-05-18)
TPC-DS Query 52, Cloudera. [Online]. Available: https://github.com/cloudera/impala-tpcds-kit/blob/master/query-templates/query52.tpl (accessed: 2025-05-18)
Apache Iceberg, Apache Software Foundation. [Online]. Available: https://iceberg.apache.org/ (accessed: 2025-05-18)
Apache Hudi, Apache Software Foundation. [Online]. Available: http://hudi.apache.org/ (accessed: 2025-05-18)
M. Armbrust et al., “Delta Lake,” Proceedings of the VLDB Endowment, vol. 13, pp. 3411–3424, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:221351108.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Attila Kantó, Attila Csaba Marosi
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
