# Vertical Visibility Among Parallel Polygons in Three Dimensions

## Abstract

Let \(\mathcal {C}=\{C_1,\ldots , C_n\}\) denote a collection of translates of a regular convex *k*-gon in the plane with the stacking order. The collection \(\mathcal {C}\) forms a *visibility clique* if for every \(i<j\) the intersection \(C_i\) and \(C_j\) is not covered by the elements that are stacked between them, i.e., \((C_i\cap C_j) \setminus \bigcup _{i<l<j}C_l\not =\emptyset \).

We show that if \(\mathcal {C}\) forms a visibility clique its size is bounded from above by \(O(k^4)\) thereby improving the upper bound of \(2^{2^{k}}\) from the aforementioned paper. We also obtain an upper bound of \(2^{2\left( {\begin{array}{c}k\\ 2\end{array}}\right) +2}\) on the size of a visibility clique for homothetes of a convex (not necessarily regular) *k*-gon.

## 1 Introduction

In a visibility representation of a graph \(G=(V,E)\) we identify the vertices of *V* with sets in the Euclidean space, and the edge set *E* is defined according to some visibility rule. Investigation of visibility graphs, driven mainly by applications to VLSI wire routing and computer graphics, goes back to the 1980s [12, 14]. This also includes a significant interest in three-dimensional visualizations of graphs [3, 4, 8, 10].

Babilon et al. [1] studied the following three-dimensional visibility representations of complete graphs. The vertices are represented by translates of a regular convex polygon lying in distinct planes parallel to the *xy*-plane and two translates are joined by an edge if they can *see* each other, which happens if it is possible to connect them by a line segment orthogonal to the *xy*-plane avoiding all the other translates. They showed that the maximal size *f*(*k*) of a clique represented by regular *k*-gons satisfies \(\left\lfloor \frac{k+1}{2} \right\rfloor + 2 \le f(k) \le 2^{2^k}\) and that \( f(3) \ge 14\). Hence, \(\lim _{k\rightarrow \infty } f(k)= \infty \). Fekete et al. [8] proved that \(f(4)=7\) thereby showing that *f*(*k*) is not monotone in *k*. Nevertheless, it is plausible that \(f(k+2)\ge f(k)\) for every *k*, and surprisingly enough this is stated as an open problem in [1]. Another interesting open problem from the same paper is to decide if the limit \(\lim _{k\rightarrow \infty }\frac{f(k)}{k}\) exists. In the present note we improve the above upper bound on *f*(*k*) to \(O(k^4)\) ^{1} and we extend our investigation to families of homothetes of general convex polygons. The main tool to obtain the result is Dilworth Theorem [6], which was also used by Babilon et al. to obtain the doubly exponential bound in [1]. Roughly speaking, our improvement is achieved by applying Dilworth Theorem only once whereas Babilon et al. used its *k* successive applications.

An analogous question was extensively studied for arbitrary, i.e. not necessarily translates or homothetes of, axis parallel rectangles [3, 8], see also [11]. Bose et al. [3] showed that in this case a clique on 22 vertices can be represented. On the other hand, they showed that a clique of size 57 cannot be represented by rectangles.

For convenience, we restate the problem of Babilon et al. as follows. Let \(\mathcal {C}=\{C_1,\ldots , C_n\}\) denote a collection of sets in the plane with the *stacking order* given by the indices of the elements in the collection. By a standard perturbation argument, we assume that the boundaries of no three sets in \(\mathcal {C}\) pass through a common point. The collection \(\mathcal {C}\) forms a *visibility clique* if for every *i* and *j*, \(i<j\), the intersection \(C_i\) and \(C_j\) is not covered by the elements that are stacked between them, i.e., \((C_i\cap C_j) \setminus \bigcup _{i<k<j}C_k\not =\emptyset \). Note that reversing the stacking order of \(\mathcal {C}\) does not change the property of \(\mathcal {C}\) forming a visibility clique. We are interested in the maximum size of \(\mathcal {C}\), if \(\mathcal {C}\) is a collection of translates and homothetes, resp., of a convex *k*-gon. We prove the following.

### **Theorem 1**

If \(\mathcal {C}\) is a collection of translates of a regular convex *k*-gon forming a visibility clique, the size of \(\mathcal {C}\) is bounded from above by \(O(k^4)\).

### **Theorem 2**

If \(\mathcal {C}\) is a collection of homothetes of a convex *k*-gon forming a visibility clique, the size of \(\mathcal {C}\) is bounded from above by \(2^{2\left( {\begin{array}{c}k\\ 2\end{array}}\right) +2 }\).

The paper is organized as follows. In Sect. 2 we give a proof of Theorem 1. In Sect. 3 we give a proof of Theorem 2. We conclude with open problems in Sect. 4.

## 2 Proof of Theorem 1

We let \(\mathcal {C}=\{C_1,\ldots , C_n\}\) denote a collection of translates of a regular convex *k*-gon *C* in the plane with the stacking order given by the indices of the elements in the collection. Let \(\mathbf{c_i}\) denote the center of gravity of \(C_i\). We assume that \(\mathcal {C}\) forms a visibility clique. We label the vertices of *C* by natural numbers starting in the clockwise fashion from the topmost vertex, which gets label 1. We label in the same way the vertices in the copies of *C*. The proof is carried out by successively selecting a large and in some sense regular subset of \(\mathcal {C}\). Let \(W_i\) be the convex wedge with the apex \(\mathbf{c_1}\) bounded by the rays orthogonal to the sides of \(C_1\) incident to the vertex with label *i*. The set \(\mathcal {C}\) is *homogenous* if for every \(1\le i \le k\) all the vertices of \(C_j\)’s with label *i* are contained in \(W_i\). We remark that already in the proof of the following lemma our proof falls apart if *C* can be arbitrary or only centrally symmetric convex *k*-gon.

### **Lemma 1**

If *C* is a regular *k*-gon then \(\mathcal {C}\) contains a homogenous subset of size at least \(\varOmega \left( \frac{n}{k^2}\right) \).

Let \((C_{i_1},\ldots ,C_{i_n})\) be the order in which the ray bounding \(W_i\) orthogonal to the segment Open image in new window of \(C_1\) intersects the boundaries of \(C_j\)’s. The set \(\mathcal {C}\) forms an *i-staircase* if the order \((C_{i_1},\ldots ,C_{i_n})\) is the stacking order. As a direct consequence of Dilworth Theorem or Erdős–Szekeres Lemma [6, 7] we obtain that if \(\mathcal {C}\) is homogenous, it contains a subset of size at least \(\sqrt{|\mathcal {C}|}\) forming an *i*-staircase.

A graph \(G=(\{1,\ldots , n\}, E)\) is a *permutation graph* if there exists a permutation \(\pi \) such that \(ij\in E\), where \(i<j\), iff \(\pi (i)>\pi (j)\). Let \(G_i=(\mathcal {C}',E)\) denote a graph such that \(\mathcal {C}'\) is a homogenous subset of \(\mathcal {C}\), and two vertices \(C_j'\) and \(C_k'\) of \(G_i\) are joined by an edge if and only if the orders in which the rays bounding \(W_i\) intersect the boundaries of \(C_j'\) and \(C_k'\) are reverse of each other. In other words, the boundaries of \(C_j'\) and \(C_k'\) intersect inside \(W_i\), see Fig. 2(a). Thus, \(G_i\)’s form a family of permutation graphs sharing the vertex set. Note that every pair of boundaries of elements in \(\mathcal {C}'\) cross exactly twice.

*k*a regular

*k*-gon is centrally symmetric the graphs \(G_i\) and \(G_{i+k/2 \mod k}\) are identical. For an odd

*k*, we only have \(G_i\subseteq G_{i+\lceil k/2 \rceil \mod k} \cup G_{i+ \lfloor k/2 \rfloor \mod k}\). The notion of the

*i*-staircase and homogenous set is motivated by the following simple observation illustrated by Fig. 2(b).

### **Observation 1**

If \(\mathcal {C}'\) forms an *i*-staircase then there do not exist two indices *i* and *j*, \(i\not =j\), such that both \(G_i\) and \(G_j\) contain the same clique of size three.

The following lemma lies at the heart of the proof of Theorem 1.

### **Lemma 2**

Suppose that \(\mathcal {C}'\) forms an *i*-staircase, and that there exists a pair of identical induced subgraphs \(G_i'\subseteq G_i\) and \(G_j'\subseteq G_j\), where \(i\not =j\), containing a matching of size two. Then \(\mathcal {C}'\) does not form a visibility clique.

### *Proof*

The lemma can be proved by a simple case analysis as follows. There are basically two cases to consider depending on the stacking order of the elements of \(\mathcal {C}'\) supporting the matching *M* of size two in \(G_i'\). Let \(u_1,v_1\) and \(u_2,v_2\), respectively, denote the vertices (or elements of \(\mathcal {C}'\)) of the first and the second edge in *M*, such that \(u_1\) is the first one in the stacking order. By symmetry and without loss of generality we assume that the ray *R* bounding \(W_i\) orthogonal to the segment \(i[(i-1) \mod k]\) of \(C_1\) intersects the boundary of \(u_1\) before intersecting the boundaries of \(u_2,v_1\) and \(v_2\), and the boundary of \(u_2\) before \(v_2\).

First, we assume that *R* intersects the boundary of \(u_2\) before the boundary of \(v_1\). In the light of Observation 1, \(u_1,v_1\) and \(u_2\) look combinatorially like in the Fig. 3(a). Then all the possibilities for the position of \(v_2\) cause that the first and last element in the stacking order do not see each other. Otherwise, *R* intersects the boundary of \(v_1\) before the boundary of \(u_2\). In the light of Observation 1, \(u_1,v_1\) and \(u_2\) look combinatorially like in the Fig. 3(b), but then \(v_2\) cannot see \(u_1\). \(\blacksquare \)

Finally, we are in a position to prove Theorem 1. We consider two cases depending on whether *k* is even or odd. First, we treat the case when *k* is even which is easier.

Thus, let *C* be a regular convex *k*-gon for an even *k*. By Lemma 1 and Dilworth Theorem we obtain a homogenous subset \(\mathcal {C}'\) of \(\mathcal {C}\) of size at least \(\varOmega (\sqrt{\frac{n}{k^2}})\) forming a 1-staircase. Note that for \(\mathcal {C}'\) the hypothesis of Lemma 2 is satisfied with \(i=1\) and \(j=1+k/2\). Since \(\mathcal {C}'\) forms a visibility clique, the graph \(G_1\) does not contain a matching of size two. Hence, \(G_1=(\mathcal {C}'=\mathcal {C}_1,E)\) contains a dominating set of vertices \(\mathcal {C}_1'\) of size at most two. Let \(\mathcal {C}_2=\mathcal {C}_1\setminus \mathcal {C}_1'\). Note that \(\mathcal {C}_2\) forms a 2-staircase and that the hypothesis of Lemma 2 is satisfied with \(\mathcal {C}'=\mathcal {C}_2,i=2\) and \(j=2+k/2 \mod k\). Thus, \(G_2=(\mathcal {C}_2,E)\) contains a dominating set of vertices \(\mathcal {C}_2'\) of size at most two. Hence, \(\mathcal {C}_3=\mathcal {C}_2\setminus \mathcal {C}_2'\) forms a 3-staircase. In general, \(\mathcal {C}_{i}=\mathcal {C}_{i-1}\setminus \mathcal {C}_{i-1}'\) forms an *i*-staircase and the hypothesis of Lemma 2 is satisfied with \(\mathcal {C}'=\mathcal {C}_i, i=i\) and \(j=i+k/2 \mod k\). Note that \(|\mathcal {C}_{k/2+1}|\le 1\). Thus, \(|\mathcal {C}'|\le k+1\). Consequently, \(n=O(k^4)\).

In the case when *k* is odd we proceed analogously as in the case when *k* was even except that for \(\mathcal {C}'\) as defined above the hypothesis of Lemma 2 might not be satisfied, since we cannot guarantee that \(G_i\) and \(G_{j}\) are identical for some \(i\not = j\). Nevertheless, since the two tangents between a pair of intersecting translates of a convex *k*-gon in the plane are parallel we still have \( G_i\subseteq G_{i+\left\lceil \frac{k}{2} \right\rceil \mod k} \cup G_{i+\left\lfloor \frac{k}{2} \right\rfloor \mod k}\) The previous property will help us to find a pair of identical induced subgraphs in \(G_i\), and \(G_{i+\left\lceil \frac{k}{2} \right\rceil \mod k}\) or \( G_{i+\left\lfloor \frac{k}{2} \right\rfloor \mod k}\) to which Lemma 2 can be applied, if \(G_i\) contains a matching *M* of size *c*, where *c* is a sufficiently big constant determined later. It will follow that \(G_i\) does not contain a matching of size *c*, and thus, the inductive argument as in the case when *k* was even applies. (Details will appear in the full version.)

## 3 Homothetes

The aim of this section is to prove Theorem 2. Let *C* denote a convex polygon in the plane. Let \(\mathcal {C}=\{C_1,C_2,\ldots , C_n\}\) denote a finite set of homothetes of *C* with the stacking order. Unlike as in previous sections, this time we assume that the indices correspond to the order of the centers of gravity of \(C_i\)’s from left to right. Let \(\mathbf{c_i}\) denote the center of gravity of \(C_i\). Let \(x(\mathbf{p})\) and \(y(\mathbf{p})\), resp., denote *x* and *y*-coordinate of \(\mathbf{p}\). Thus, we assume that \(x(\mathbf{c_1})< x(\mathbf{c_2}) <\ldots <x(\mathbf{c_n})\)

Suppose that \(\mathcal {C}\) forms a visibility clique. Similarly as in the previous sections we label the vertices of *C* by natural numbers starting in the clockwise fashion from the topmost vertex, which gets label 1. We label in the same way the vertices in the copies of *C*. Consider the poset \((\mathcal {C},\subset )\) and note that it contains no chain of size five. By Dilworth theorem it contains an anti-chain of size at least \(\frac{1}{4} |\mathcal {C}|\). Since we are interested only in the order of magnitude of the size of the biggest visibility clique, from now on we assume that no pair of elements in \(\mathcal {C}\) is contained one in another.

Every pair of elements in \(\mathcal {C}\) has exactly two common tangents, since every pair intersect and no two elements are contained one in another. We color the edges of the clique \(G=(\mathcal {C},{\left( {\begin{array}{c}\mathcal {C}\\ 2\end{array}}\right) })\) as follows. Each edge \(C_iC_j\), \(i<j\), is colored by an ordered pair, in which the first component is an unordered pair of vertices of *G* supporting the common tangents of \(C_i\) and \(C_j\), and the second pair is an indicator equal to one if \(C_i\) is below \(C_j\) in the stacking order, and zero otherwise.

### **Lemma 3**

The visibility clique *G* does not contain a monochromatic path of length two of the form \(C_iC_jC_k\), \(i<j<k\).

We say that a path \(P=C_1C_2\ldots C_k\) in *G* is monotone if \(x(\mathbf{c_1})<x(\mathbf{c_2})<\ldots < x(\mathbf{c_k})\). It was recently shown [9, Theorem 2.1] that if we color the edges of an ordered complete graph on \(2^c+1\) vertices with *c* colors we obtain a monochromatic monotone path of length two. We remark that this result is tight and generalizes Erdős–Szekeres Lemma [7]. Thus, if *G* contains more than \(2^{2\left( {\begin{array}{c}k\\ 2\end{array}}\right) +2}\) vertices it contains a monochromatic path of length two which is a contradiction by Lemma 3.

## 4 Open Problems

Since we could not improve the lower bound from [1] even in the case of homothetes, we conjecture that the polynomial upper bound in *k* on the size of the visibility clique holds also for any family of homothetes of an arbitrary convex *k*-gon. To prove Theorem 2 we used a Ramsey-type theorem [9, Theorem 2.1] for ordered graphs. We wonder if the recent developments in the Ramsey theory for ordered graphs [2, 5] could shed more light on our problem.

## Footnotes

## Notes

### Acknowledgement

We would like to thank Martin Balko for telling us about [9].

## References

- 1.Babilon, R., Nyklová, H., Pangrác, O., Vondrák, J.: Visibility representations of complete graphs. In: Kratochvíl, J. (ed.) GD 1999. LNCS, vol. 1731, pp. 333–341. Springer, Heidelberg (1999) CrossRefGoogle Scholar
- 2.Balko, M., Cibulka, J., Král, K., Kynčl, J.: Ramsey numbers of ordered graphs. arXiv:1310.7208v3
- 3.Bose, P., Everett, H., Fekete, S.P., Houle, M.E., Lubiw, A., Meijer, H., Romanik, K., Rote, G., Shermer, T.C., Whitesides, S., Zelle, C.: A visibility representation for graphs in three dimensions. J. Graph Algorithm Appl.
**2**(3), 1–16 (1998)MathSciNetCrossRefGoogle Scholar - 4.Cohen, R.F., Eades, P., Lin, T., Ruskey, F.: Three-dimensional graph drawing. Algorithmica
**17**(2), 199–208 (1997)zbMATHMathSciNetCrossRefGoogle Scholar - 5.Conlon, D., Fox, J., Lee, C., Sudakov, B.: Ordered Ramsey numbers. arXiv:1410.5292v1
- 6.Dilworth, R.P.: A decomposition theorem for partially ordered sets. Ann. Math.
**51**, 161–166 (1950)zbMATHMathSciNetCrossRefGoogle Scholar - 7.Erdős, P., Szekeres, G.: A combinatorial problem in geometry. In: Gessel, I., Rota, G.-C. (eds.) Classic Papers in Combinatorics. Modern Birkhäuser Classics, pp. 49–56. Birkhäuser Boston, Boston (1987)Google Scholar
- 8.Fekete, S.P., Houle, M.E., Whitesides, S.: New results on a visibility representation of graphs in 3D. In: Brandenburg, F.J. (ed.) GD 1995. LNCS, vol. 1027, pp. 234–241. Springer, Heidelberg (1996) CrossRefGoogle Scholar
- 9.Milans, K.G., Stolee, D., West, D.B.: Ordered ramsey theory and track representations of graphs. http://www.math.illinois.edu/stolee/Papers/MSW12-OrderedRamsey.pdf
- 10.Robertson, G.G., Mackinlay, J.D., Card, S.K.: Cone trees: animated 3D visualizations of hierarchical information. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI 1991, pp. 189–194. ACM, New York, NY, USA (1991)Google Scholar
- 11.Romanik, K.: Directed VR-representable graphs have unbounded dimension. In: Tamassia, R., Tollis, I.G. (eds.) Graph Drawing. LNCS, vol. 894, pp. 177–181. Springer, Berlin Heidelberg (1995)CrossRefGoogle Scholar
- 12.Tamassia, R., Tollis, I.G.: A unified approach to visibility representations of planar graphs. Discrete Comput. Geom.
**1**(1), 321–341 (1986)zbMATHMathSciNetCrossRefGoogle Scholar - 13.Štola, J.: 3D visibility representations by regular polygons. In: Eppstein, D., Gansner, E.R. (eds.) GD 2009. LNCS, vol. 5849, pp. 323–333. Springer, Heidelberg (2010) CrossRefGoogle Scholar
- 14.Wismath, S.K.: Characterizing bar line-of-sight graphs. In: Proceedings of the First Annual Symposium on Computational Geometry, SCG 1885, pp. 147–152. ACM, New York, NY, USA (1985)Google Scholar