I have a set of data samples and want to perform a hypothesis test. I used the Anderson-Darling-Test so far, where the test statistic is defined by $A=\int_{-\infty}^{\infty}n\frac{(\tilde{F}(x)-F(x))^2}{F(x)(1-F(x))}dF(x)$ ($\tilde{F}(x)$ is empirical distribution, ${F}(x)$ cumulative distribution, $n$ sample size).
If I now test some ($n=10$) data, say, $\{-1.06424,-0.530066,\dots,0.95493\}$. If I test this data set with a distribution that has a cut-off at $x=\pm1$, e.g. Wigner semicircle distribution for center $0$ and radius $1$, why is the p-value not zero? In fact, using standard software (Mathematica, Matlab, ...), I obtain for this data set the p-value $24.4\%$. Additionally, due to the definition of $A$, it does not matter if the forbidden value is at $x=-1.06$ or even far away at $x=-10$.
Clearly, the definition of $A$ is allowing for outlying data, but is there any way to handle such a case in a formal way? Should one take a different hypothesis test? Why (for what purpose) are forbidden values not causing the p-value to vanish?
A combination of reasons, I think:
First, $n = 10$ is a $very$ small sample for a goodness-of-fit test based on a comparison of the ECDF or your data and the 'null' CDF.
Second, Anderson-Darling, Kolmogorov-Smirnov and other ECDF based GOF tests are not very good at noticing 'forbidden' values that are barely outside the support of the null distribution.
Altogether, these tests detect 'differences' between an ECDF and a CDF by looking at metrics like the maximum discrepancy and the 'area' of the discrepancy. If the difference between the the ECDF of your data and the null CDF is not large, the fact that the ECDF has some values outside the support will hardly be noticed.
Shown below are four examples of an ECDF plot of a sample of size $n = 20$ from $Norm(1/2, 1/6),$ each showing the CDF of $Beta(4, 4).$ The distributions share $\mu = 1/2$ and $\sigma = 1/6.$ Agreement is generally not 'good' even if not significantly 'bad'. However, the fact that one normal sample takes a value outside the support $(0,1)$ of the beta distribution is not an important consideration in the discrepancy between ECDF and CDF.