I noticed that if I take the laplace transform of $x^{\lfloor s^2 \rfloor} y(x)$ that I get:
$(-1)^{\lfloor s^2 \rfloor} \frac {d^{\lfloor s^2 \rfloor}Y}{ds^{\lfloor s^2 \rfloor}}(s)$
This is based upon me noting that this is of the form $x^ny(x)$, where n is a constant integer function. Since "n" is independent of x, and floor returns integral values, I believe this is justified.
This prompts me to ask an obvious question:
What does it mean to take the derivative of order $\lfloor x^2 \rfloor$ with respect to x?
I imagine that such a derivative would just be each of the derivatives over piecewise intervals. I know that it's always integral valued, so there is no need to bring in fractional derivatives. Nothing in the notation seems to say that such a thing cannot occur, so it ultimately begs the question as to what it truly means and how it should be evaluated. I imagine that someone must've done this before now.
Of course, it might be illegal to place s within the function being transformed, in which case this question is pointless.
The reason for the identity $\mathcal{L}\{t^nf(t)\} = F^{(n)}(s)$ is due to the fact that you can take the basic identity
$$F(s) = \int_0^\infty f(t) e^{-st} \,dt$$
and differentiate both sides $n$ times with respect to $s$. You can't do something similar to generate a $t^{\lfloor t^2 \rfloor}$ in front of your function, but what you can do is find the ranges over which $\lfloor t^2 \rfloor$ is constant, then break your function up into a piecewise sum of indicator functions times factors of the form $t^c f(t)$, then use linearity to evaluate the Laplace transform. I could only really see doing this for a particular function, not for a general function, where the Laplace transform would be very complicated. You'd probably be better off using power series methods or something of the like.
EDIT:
Answering the question at the end of your post, it is indeed illegal to place the ss inside the function being transformed. The function being transformed, f(t)f(t), is a function of one variable (in the frequency domain). Writing something like $\int_0^{\infty} e^{-st} t^{\lfloor s \rfloor} f(t) \,dt$ amounts to a new transform.