Membership problems in nilpotent groups

Abstract

We study both the Submonoid Membership problem and the Rational Subset Membership problem in finitely generated nilpotent groups. We give two reductions with important applications. First, Submonoid Membership in any nilpotent group can be reduced to Rational Subset Membership in smaller groups. As a corollary, we prove the existence of a group with decidable Submonoid Membership and undecidable Rational Subset Membership, confirming a conjecture of Lohrey and Steinberg. Second, the Rational Subset Membership problem in $H_3(\mathbb Z)$ can be reduced to the Knapsack problem in the same group, and is therefore decidable. Combining both results, we deduce that the filiform $3$-step nilpotent group has decidable Submonoid Membership.