Given a total space $E = M \times F$ of a fiber bundle where $M$ is a smooth manifold and $F$ is the fiber. The fiber $F_x$ corresponding to the point $x \in M$ is the set of morphisms between objects $\Sigma_x := \{\sigma_x \}$ at the point $x \in M$ (that vary from base manifold point to base manifold point). Any section of the fiber bundle can be characterized as $s(E) = (x,\sigma_x \mapsto \sigma'_x)$ with $\sigma_x,\sigma'_x \in \Sigma_x$.
Due to the inhomogenity of the fibers (i.e. the cardinality and the structure of the set $\Sigma_x$ and therefore morphisms between this set is not independent on $x \in M$) no suitable connection can be defined; the fiber is assumed to be non-differentiable. However, I can define transfer functions $\Lambda(x,y)$ with $\Lambda(x,y)(\sigma_y \mapsto \sigma'_y) = (\sigma_x \mapsto \sigma'_x)$ and $x,y, \in M$ which is able to compare different fibers with others.
Is there a way to handle with fiber bundles like this? What I can do if there cannot exist a smooth connection (e.g. Levi-Civita affine connection) on a fiber bundle?
Let me first say that it is kind of strange to start with the total space of a trivial bundle $E = M \times F$, which implies that all fibers are equinumerous, and then write "Due to the inhomogenity of the fibers (i.e. the cardinality and the structure of the set $Σ_x$ and therefore morphisms between this set is not independent on $x∈M$) [...]", which suggests that the fibers are not necessarily equinumerous.
Considering the generality of your question, we won't be able to mimic what is done for instance in Riemannian geometry in order to define a curvature tensor, since such a tensor would also act on vectors tangent to the fiber (the meaning of which is unclear in our context). However, we might be able to get approximate notions.
Given an ordered finite set $\{x_1, \dots, x_n \} \subset M$, what we might call a discrete path $\gamma$ in $M$, we can define a morphism $\Lambda(\gamma) : F_{x_1} \to F_{x_n}$ as the composite morphism $\Lambda(x_{n-1}, x_n) \circ \dots \circ \Lambda(x_1, x_2)$. We might call $\Lambda(\gamma)$ the parallel transport along the (discrete) path $\gamma$. If $\gamma$ is a loop, that is if $x = x_1 = x_n$, then $\Lambda(\gamma) : F_x \to F_x$ is the holonomy around $\gamma$.
In good situations, given a piecewise smooth path $\Gamma : [0,1] \to M$ joining $x$ to $y$, we might be able to define $\Lambda(\Gamma)$ by looking at the parallel transport of finer and finer discrete partitions of $\Gamma$, getting in the end something independent of the partitioning process. Of course, some notion of convergence in $\mathrm{Mor}(F_x, F_y)$ should exist here. When $\Gamma$ is a loop, that is when $x=y$, $\Lambda(\Gamma)$ is the holonomy around $\Gamma$.
In the context of differential geometry, we know (for instance, by the Ambrose-Singer theorem) that the curvature at $x \in M$ is related to the holonomy around infinitesimal loops based at $x$. So we might want to mimic this idea. More precisely, if $\{ \Gamma_n : I \to M \}_{n \in \mathbb{N}}$ is a sequence of loops based at $x$ converging to the constant loop $c_x : I \to \{x \}$, then we expect to have $\lim_{n \to \infty} \Lambda(\Gamma_n) = \Lambda(c_x) = Id$, which says nothing about curvature. So curvature is more of a measure of how the sequence $\{ \Lambda(\Gamma_n)\}_{n \in \mathbb{N}}$ approaches $Id$. In differential geometry, this is done by taking a differential (since there, $\mathrm{Mor}(F_x, F_x) = \mathrm{Diff}(F_x)$ is a smooth manifold), but as we mentioned earlier, in our context, such a notion of differential might not exist.
Notice that holonomy is a global version of curvature ; In differential geometry, since we know how to integrate, we recover holonomies by integrating the curvature 2-form around loops. However, in a more general context, it might be more natural to only consider the holonomies without trying to have an infinitesimal notion of holonomy.