I wonder whether the following ideas make sense, and whether something in that direction has been written down somewhere; do you know a reference?
The last definition is supposed to be a reformulation of the notion of a class of morphisms in a category being closed under transfinite composition, to be found for example in Definition A.1.2.2 of Lurie's book "Higher Topos Theory".
Two related questions are How would generalizing simplicial sets affect (∞,1)-functors between (∞,1)-categories? and Ordinal category theory?.
Let $\kappa$ be an (infinite) regular cardinal.
Definition. Let $\Delta_{<\kappa}$ be the category of non-empty ordinals $< \kappa$, with order-preserving maps as morphisms. We have $\Delta = \Delta_{<\aleph_0}$.
Definition. Define a $\kappa$-simplicial set to be an object in $\operatorname{Set}_{\Delta_{<\kappa}}$. We identify each $\lambda \in \Delta_{<\kappa}$ with the $\kappa$-simplicial set that it represents.
Definition. For $\lambda \in \Delta_{<\kappa}$ with $\{0,1\} \subseteq \lambda$ and $i \in \lambda$, define the $i$-th horn $\Lambda^\lambda_i \subseteq \lambda$ of $\lambda$ in the obvious way. We call the horn inner if $i$ is neither minimal nor maximal in $\lambda$.
Definition. We call a $\kappa$-simplicial set $X$ a $\kappa$-category if every inner horn $\Lambda^\lambda_i \to X$ in $X$, with $\lambda \in \Delta_{<\kappa}$, can be extended to a map $\lambda \to X$, and, furthermore, for every limit ordinal $\lambda \in \Delta_{<\kappa}$, every map $\lambda \to X$ can be extended to a map $\lambda^+ \to X$, where $\lambda^+$ is the successor of $\lambda$. The $\aleph_0$-categories are precisely the $\infty$-categories.
Definition. Let $\mathcal{C}$ be an $\infty$-category which admits all colimits indexed by an ordinal $<\kappa$. By identifying $\lambda \in \Delta_{<\kappa}$ with the simplicial set which is the nerve of the category defined by the (linearly) ordered set $\lambda$, we can canonically regard $\mathcal{C}$ as a $\kappa$-simplicial set. Define $\tilde{\mathcal{C}} \subseteq \mathcal{C}$ to be the $\kappa$-simplicial subset which contains an element $\lambda \to \mathcal{C}$ of $\mathcal{C}(\lambda)$ if and only if it preserves all colimits indexed by an ordinal $<\kappa$.
Conjecture. The $\kappa$-simplicial set $\tilde{\mathcal{C}}$ is a $\kappa$-category.
Definition. Let $\mathcal{C}$ be an $\infty$-category which admits all colimits indexed by an ordinal $<\kappa$, and let $S \subseteq \mathcal{C}$ be a simplicial subset. We say that $S \subseteq \mathcal{C}$ is closed under $\kappa$-composition if the inclusion of $\kappa$-simplicial sets $S \cap \tilde{\mathcal{C}} \subseteq \tilde{\mathcal{C}}$ has the right lifting property with respect to all inner horns $\Lambda^\lambda_i \subseteq \lambda \in \Delta_{<\kappa}$ and all $\lambda \subseteq \lambda^+$, with $\lambda$ any limit ordinal in $\Delta_{<\kappa}$.