It takes a fair amount of expertise to keep a spark cluster up and running. Maybe more people are using managed offerings like Databricks or Google Dataproc.

Probably exactly that - relatively few big ticket customers.