The party size is determinted by the ammount of charisma you have(if the followers are players or slaves) and the number of party points(if the followers are mercs or animals).
