Assume we have two search engines, A and B. I get a list of scores for 10 different queries. Now, I model this with a t-test in order to test significance.
These are my hypothesis:
$H_0: B-A=0$
$H_1: B-A>0$
I obtain $t=1.0803$. My Critical significance level is $\alpha=0.05$ and the critical value is $cv=2.262157$.
Now, drawing the graph, how can I conclude if my hypothesis should be rejected? My assumption is that since my t value is lower than cv, then the hypothesis is rejected. Is this correct? What about the two sides? Can someone clarify?
From the information provided, this must be a two-sided paired t-test, and you must be testing $H_0: \mu_B - \mu_A = 0$ against the alternative $H_1: \mu_B - \mu_A \ne 0.$ Also, you have $T = \sqrt{10}(\bar B - \bar A)/S_{B-A} = 1.0803.$ (At the end you mention a two-sided test, but you have written the alternative for a one-sided test. I suspect some problem using TeX here.)
In these circumstances, you would reject $H_0$ only if $|T| > 2.262.$ But your $T$-statistic is too near to $0$ for you to reject. (So your supposition about when to reject is wrong.)
The critical value 2.262 is for a two-sided test at the 5% level: 95% of the area under the density curve of Student's t distribution (with $n - 1 = 10 - 1 = 9$ degrees of freedom) lies between -2.262 and +2.262. Thus, the total area remaining in each tail is 2.5% and the total area remaining in both tails together is the significance level 5%.