Bregman divergence is a general class to measure “difference” between two data points. For instance, Euclidean distance and Kullback Leibler (KL) divergence are instances of Bregman divergence. In this post, I derive KL divergence from Bregman divergence formulation (for myself).
Bregman divergence is defined by the equation below:
where means inner product. When , this Bregman divergence is equivalent to KL divergence.
From Line 2 to Line 3, since is a probability distribution.
This blog post explains Bregman divergence in detail and gives useful links.η