The party size is determined by the amount of charisma you have(if the followers are players or slaves) and the number of party points(if the followers are mercs or animals).
