Rationally smooth elements of Coxeter groups and triangle group avoidance

Abstract

We study a family of infinite-type Coxeter groups defined by the avoidance of certain rank 3 parabolic subgroups. For this family, rationally smooth elements can be detected by looking at only a few coefficients of the Poincaré polynomial. We also prove a factorization theorem for the Poincaré polynomial of rationally smooth elements. As an application, we show that a large class of infinite-type Coxeter groups have only finitely many rationally smooth elements. Explicit enumerations and descriptions of these elements are given in special cases.

22 pages, 3 figures, version 3: Section 3.3 is now its own Section 4. New Lemma 2.3 on a property of BP-decompositions. Proof of Corollary 3.7 added. Notation changes made for descents sets (S->D). Several other minor changes