Monzurul Haque Shimul

By leveraging the fast, approximate calculations of DataSketches, complex analytic queries such as cardinality estimation and retention analysis can be completed in less than one second in Druid. In this post, I’m going to show how to count unique or count distinct in Druid using the Datasketches HLL Sketch algorithm which is available as an aggregator in druid-datasketches extension.

By leveraging the fast, approximate calculations of DataSketches, complex analytic queries such as cardinality estimation and retention analysis can be completed in less than one second in Druid. In this post, I’m going to show how to count unique or count distinct in Druid using the Datasketches Theta Sketch algorithm which is available as an aggregator in druid-datasketches extension.

Apache Druid is a high performance real-time analytics database. On the other hand, HyperLogLog algorithm, shortly known as HLL, is a very efficient way to add elements to a set and then estimate the size of that set. In this article, I will demonstrate how to get unique counts from Druid using the HyperLogLog algorithm. The details of HyperLogLog are beyond the scope of this article.

Lookups are a concept in Druid where dimension values are replaced with new values. In Druid lookup, a “key” refers to a dimension value to match, and a “value” refers to its replacement. So if you wanted to map “us” to “United States” then the key would be “us” and the value would be “United States”.

Druid supports passing very small lookups (a few dozen to a few hundred)  at query time as a “map” lookup. Other lookup types are available as extensions, including lookups from local files, remote URIs, or JDBC through lookups-cached-global. Another supported type is globally cached lookups from a Kafka topic through kafka-extraction-namespace.

In this article, I will demonstrate how to configure Druid to use lookups from Json files using the lookups-cached-global extension. 

Lookups are a concept in Druid where dimension values are replaced with new values. In Druid lookup, a “key” refers to a dimension value to match, and a “value” refers to its replacement. So if you wanted to map “us” to “United States” then the key would be “us” and the value would be “United States”.

Druid supports passing very small lookups (a few dozen to a few hundred)  at query time as a “map” lookup. Other lookup types are available as extensions, including lookups from local files, remote URIs, or JDBC through lookups-cached-global. Another supported type is globally cached lookups from a Kafka topic through kafka-extraction-namespace.

In this article, I will demonstrate how to configure Druid to use lookups from a csv file using lookups-cached-global extension.