Inference of a Dyadic Measure and Its Simplicial Geometry from Binary Feature Data and Application to Data Quality

Abstract

We propose a new method for representing data sets with an ordered set of binary features which summarizes both measure-theoretic and topological properties. The method does not require any assumption of metric space properties for the data. A data set with an ordered set of binary features is viewed as a dyadic set with a dyadic measure. We prove that dyadic sets with dyadic measures have a canonical set of binary features and determine canonical nerve simplicial complexes. The method computes the two related representations: multiscale parameters for the dyadic measure and the Betti numbers of the simplicial complex. The dyadic product formula representation formulated in previous work is exploited. The parameters characterize the relative skewness of the measure at dyadic scales and localities. The more abstract Betti number statistics summarize the simplicial geometry of the support of the measure. We prove that they provide a simple privacy property. Our methods are compared with other results for measures on sets with tree structures, recent multi-resolution theory, and computational topology. We illustrate the method on a data quality data set and propose future research directions.

Appendices

Appendix 1: Mutual Constraint Violations by Source

The data in Tables 7, 8, 9, 10, 11, 12, 13, and 14 in this appendix is sufficient to reproduce the data quality analysis described in the paper. The data consisted of data quality statistics for each source describing Table 7 violations for each data item in each source. The sources were the individual sources 2, 3, 4, and 5 and the composite source 1. The mutual constraint violations view of the data for each of the individual sources is shown in tables. The data for the composite source is a composite of the data for the individual tables, so that data is not shown in a table. The raw input spreadsheet tables were pre-processed into 3 column tables. In each of these tables, there is one row for each unique maximal set of constraints violated by a data element. The rows are listed in decreasing order of violation. The first column lists the number of elements whose maximal violation set is the one listed in the row. The third column lists the numbers of the constraints in the maximal violation set. The second column is the label of the path from the roots of the binary tree to the node corresponding to the maximal set. The mutual constraint violations for source 2 are shown in Table 7. The constraint violations for source 3 are shown in Tables 8, 9, and 10. The constraint violations for source 4 are shown in Tables 11 and 12. The constraint violations for source 5 are shown in Tables 13 and 14. The simplicial complex for each data source is generated by listing the mutual constraint violation sets for the data items in each data source. These lists are shown for sources 2, 3, 4, and 5 in the right hand columns of these long tables.

Table 7 Mutual constraint violations for source 2 in decreasing order of violation

Table 8 Mutual constraint violations for source 3 in decreasing order of violation part 1

Table 9 Mutual constraint violations for source 3 in decreasing order of violation part 2

Table 10 Mutual constraint violations for source 3 in decreasing order of violation part 3

Table 11 Mutual constraint violations for source 4 in decreasing order of violation part 1

Table 12 Mutual constraint violations for source 4 in decreasing order of violation part 2

Table 13 Mutual constraint violations for source 5 in decreasing order of violation part 1

Table 14 Mutual constraint violations for source 5 in decreasing order of violation part 2

Appendix 2: Example: Simplicial Complexes for Source 2

The lexicographically sorted list of mutual constraint violations for source 2 inferred from the right hand columns of the table for source 2 is shown in Table 15. This was the input to chomp-simplicial for computation of the Betti numbers for source 2. Note the input to chomp-simplicial for the computation of the Betti numbers of the other sources could be inferred from the right hand columns of the long tables in Appendix 1 for the other sources.

Table 15 Simplicial complex generators for source 2: maximal mutual constraint violation sets

Appendix 3: Representation Lemma for Measures on Tree-Structured Spaces

A natural question to ask is: Does the representation lemma for dyadic measures discussed in Sect. 3.3 and proven in [2] and [8] generalize to measures on sets with a general tree structures? We provide one answer in the next lemma. Fix notions by defining a tree structure on a set X that consists of a tree $\mathcal {T}$, a mapping $\mathcal {T}: nodes(\mathcal {T}) \rightarrow 2^X$ from nodes to subsets of X, and constraints on the mapping: $\mathcal {T}(root) = X$ and the image of a parent node is the disjoint union of the images of its child nodes, i.e., S(n) = ∪_c:child(n) S(c)

Lemma 2 (Representation Lemma for Measures on Tree-Structured Sets)

Let $(X, \mathcal {T}, \mathcal {T})$ denote a tree-structured set. Let ν and μ denote strictly positive and non-negative measures, respectively, on the sigma algebra $\varSigma (\mathcal {T})$ generated by the sets in the image of $\mathcal {T}$ . For a non-root node $n \in \mathcal {T}$ let r >= p >= n denote the set of nodes p on the path from n to the root node r ordered by the parent relationship. Let a _n be the parameter uniquely defined by

$$\displaystyle \begin{aligned} \mu(S(n)) = (1 + a_n) \frac{\nu(S(n))}{\nu(S(parent(n)))} \mu(S(parent(n))) \end{aligned} $$

(12)

if μ(S(parent(n))) ≠ 0. If μ(S(parent(n))) = 0 define a _n = 0.

• 1. $\nu (S(n)) = \nu (X) \prod _{r > p \geq n} {\frac {\nu (S(p))}{\nu (S(parent(p))}}$

• 2. $\mu (S(n)) = \mu (X) \prod _{r > p \geq n} { (1 + a_p) \frac {\nu (S(p))}{\nu (S(parent(p))}}$

• 3. for a non-leaf node p, the function $f: \{S(c): c \in child(p)\} \rightarrow \mathbb {R} , f(S(c)) = a_c \cdot \nu (S(c))\}$ is orthogonal to the functions which are constant on the {S(c) : c ∈ child(p)}

  $$\displaystyle \begin{aligned} 0 = \sum_{c: child(p)} {a_c \cdot \nu(S(c))}\end{aligned} $$

  (13)

  Thus, relative to a choice of a multi-resolution basis for functions on tree consisting of the parent node n and its child nodes, the function f can be expressed as a unique linear combination of card({S(c) : c ∈ child(p)}) − 1 basis functions. For binary trees, the function h _p which is 1 on the left child node c _L and -1 on the right child node c _R is a basis function, and the linear combination is $a_{c_L} \cdot h_p$.

• 4. S maps the set of nodes at each level i in the tree to disjoint partitions $\mathcal {T}_i$ of X. The partition at level i + 1 refines the partition at level i. Let ν _i and μ _i denote the measures on $\varSigma (\mathcal {T}_i)$ , the sigma algebra generated by the sets in $\mathcal {T}_i$ , determined by the first two path formulas. The weak star limit of ν _i and μ _i exists.

• 5. $-1 \leq a_n < \frac {\nu (S(parent(n))) } {\nu (S(n))} - 1$

Proof

The first statement, the path formula for ν, is trivially true by telescoping cancellation. Informally, it just says that the ν(S(n)) equals ν(X) multiplied by the conditional probabilities determined by a path from the root node to n. None of the denominators in the conditional probabilities is zero because ν is strictly positive. The second statement, the path formula for μ, is proved by induction on the length of the path. It is true for a path of length 1 beginning at the root, by the definition of the parameter in the statement of the lemma. The induction step just substitutes the induction hypothesis for μ(S(parent(n))). The third statement is true if μ(S(p)) = 0 because then the parameters a _c for the child nodes are all 0. If μ(S(p)) ≠ 0 the third statement is proved by noting that for a non-leaf node μ(S(p)) =∑_c:child(p) μ(c) because measures are additive on disjoint sets and the tree structure definition requires that the disjoint union of the set images of child nodes equals the set image of the parent node. Substituting the path formula for μ into the right side, expanding, and factoring out μ(S(p)) give

$$\displaystyle \begin{aligned} \mu(S(p)) = \mu(S(p)) \cdot \left(\sum_{c: child(p)} {\frac{\nu(S(c))}{\nu(S(p))} } + \sum_{c: child(S(p))} {a_p \cdot \frac{\nu(S(c))}{\nu(S(p))} } \right) \end{aligned} $$

(14)

The first sum term in parentheses is 1 because the sum of the ν conditional probabilities of the child nodes equals 1. The second sum term in parentheses equals 0. Multiply by ν(p) to obtain the third statement. The first two sentences in the fourth statement are implied by the definition of tree structure. The fourth statement claiming that the weak star limit exists is proved using the same method as used for Lemma 2.1 in [2] and Lemma 3.20 in [8]. The key point is that ν _i(X) and μ _i(X) are constant for all levels i. The fifth statement is true if μ(S(parent(n))) = 0 since then a _n = 0 by definition. If μ(S(parent(n))) ≠ 0, the definition can be rewritten as

$$\displaystyle \begin{aligned} \frac{\mu(S(n))}{ \mu(S(parent(n)))} \cdot \frac{\nu(S(parent(n))) } {\nu(S(n))} = 1 + a_n \end{aligned} $$

(15)

Since ν is strictly positive and μ is non-negative the left side of the equation is non-negative, so 1 + a _n ≥ 0 implying a _n ≥−1, with a _n = −1 only if μ(S(n)) = 0 and μ(S(parent(n))) ≠ 0. If μ(S(n)) > 0 and μ(S(parent(n))) ≠ 0, 1 + a _n is biggest when μ(S(n)) = μ(S(parent(n)) (making the first factor in the equation equal to 1) so the result follows. If the tree is a regular k-adic tree and ν is the naive measure ν(X) = 1 and $\nu (c) = \frac {1}{|child\; nodes|}$ $\frac {\nu (S(parent(n))) } {\nu (S(n))}$ is constant for all nodes and equals k so a _n = k − 1 and is independent of n. This agrees with the dyadic theory where k = 2.

Statements 3 and 5 in the lemma show that the parameter space for general tree measures is much more complex and depends node by node on the measure ν. Statement 5 explains why the parameter space for dyadic measures on binary trees is simpler and easily related to the Haar-like basis. While it is easy to define a dyadic version of Wasserstein distance between dyadic measures, definition of a canonical Wasserstein-like distance between measures on tree-structured sets appears to be a research issue. Perhaps the tree distance theory developed in [4] could be exploited.