The China Earthquake Networks Centre (CENC) has recently formed a partnership with ByteDance's data warehousing platform, ByteHouse, to focus on real-time data warehousing. CENC is a public institution directly under the China Earthquake Administration. It serves as an important business hub, core technology platform, and important window for international exchange of basic earthquake information in China’s earthquake prevention and disaster reduction work. The CENC handles earthquake monitoring, prediction and early warning, emergency response, and automation work, and is the national earthquake observatory that guides the business of provincial and central stations.

Zou Rui, Director of the Geophysical Network Department of the CENC, said, “Previously, the centre had been using an open-source data engine, but as the amount of data continued to expand, there were serious shortcomings in real-time data deduplication, data updates, deletions, and cluster operations, and therefore a new solution was urgently needed.”

How to achieve real-time data writing, deduplication, data updates, and deletions while maintaining high-performance queries? This has always been a challenge. For example, with the CENC, the accuracy of data processing is reduced because of the deduplication delay of its current open-source data engine when dealing with massive data daily. Meanwhile, with the continuous growth of metadata, cluster restart times are too long, affecting the timeliness of business operations.

ByteDance has been using ClickHouse on a large scale since 2017 and has the largest ClickHouse cluster in China. With its in-depth usage, ByteDance has accumulated a wealth of experience and made deep optimisations and self-research improvements to ClickHouse. Finally, in August 2021, ByteHouse was officially released and made available through the Volcano Engine.

Architecturally, ByteHouse uses a self-developed high-availability engine HaEngine, a HaUnique engine for real-time data updates and deletions, and a HaKafka engine for high-availability real-time writing. Simultaneously, cluster operation and multi-table association have also been enhanced.

In the real-time data warehousing scenario mentioned above, ByteHouse uses fully self-developed optimisation and has a higher query performance in complex query scenarios. Its rich table engine can not only help the CENC achieve rapid data writing, deduplication, updating, deletion, and analysis, but also support efficient and convenient operation methods, achieving more flexible high-performance real-time queries.

As a cloud-native data warehouse, ByteHouse also supports private deployment, providing a fast analysis experience for government and enterprise users facing real-time analysis scenarios of massive data, such as the CENC, and assisting enterprise digital transformation.