A HyperLogLog is a probabilistic data structure used in order to count unique values. Mathematically it is defined as a probabilistic data structure to estimate the cardinality of a data set. Computing the count of distinct elements in a large data set is often necessary but computationally intensive. Say you need to calculate the number of distinct users visiting your website in the past week. Doing this with a traditional SQL query on a large data set would take a long period of time and a large amount of memory. But instead of exact count if an approximation is allowed we can achieve this in no time with a small amount of memory usage by using the HyperLogLog algorithm. In this post I’m going to give a quick introduction to Redis HyperLogLog.
About Monzurul Haque Shimul
Count Unique in Druid Using DataSketches HLL Sketch
By leveraging the fast, approximate calculations of DataSketches, complex analytic queries such as cardinality estimation and retention analysis can be completed in less than one second in Druid. In this post, I’m going to show how to count unique or count distinct in Druid using the Datasketches HLL Sketch algorithm which is available as an aggregator in druid-datasketches extension.
Read More about Count Unique in Druid Using DataSketches HLL Sketch
Count Unique in Druid Using DataSketches Theta Sketch
By leveraging the fast, approximate calculations of DataSketches, complex analytic queries such as cardinality estimation and retention analysis can be completed in less than one second in Druid. In this post, I’m going to show how to count unique or count distinct in Druid using the Datasketches Theta Sketch algorithm which is available as an aggregator in druid-datasketches extension.
Read More about Count Unique in Druid Using DataSketches Theta Sketch
Getting Unique Counts from Druid Using HyperLogLog
Apache Druid is a high performance real-time analytics database. On the other hand, HyperLogLog algorithm, shortly known as HLL, is a very efficient way to add elements to a set and then estimate the size of that set. In this article, I will demonstrate how to get unique counts from Druid using the HyperLogLog algorithm. The details of HyperLogLog are beyond the scope of this article.
Read More about Getting Unique Counts from Druid Using HyperLogLog
Configure Druid to Use Lookups from JSON File
Lookups are a concept in Druid where dimension values are replaced with new values. In Druid lookup, a “key” refers to a dimension value to match, and a “value” refers to its replacement. So if you wanted to map “us” to “United States” then the key would be “us” and the value would be “United States”.
Druid supports passing very small lookups (a few dozen to a few hundred) at query time as a “map” lookup. Other lookup types are available as extensions, including lookups from local files, remote URIs, or JDBC through lookups-cached-global. Another supported type is globally cached lookups from a Kafka topic through kafka-extraction-namespace.
In this article, I will demonstrate how to configure Druid to use lookups from Json files using the lookups-cached-global extension.
Read More about Configure Druid to Use Lookups from JSON File