Musk’s numbers are down, and even some who like him don’t seem to love him being so central.