I tried solving this question by
$1.$ $-1$ and $4$ will not be in domain because denominator can not be zero .
$2.$ Either both denominator and numerator will be positive or negative so that whole term in root becomes positive.
But I am not able to solve the upper part , can it be done by taking $\log$ ?
On
The roots of $x^2-3x-4$ are $-1$ and $4$, and, as you correctly observed, those two values can't be in the domain of $f$.
Now the numerator is less than $0$ iff $x>0$ and the denominator is less than $0$ iff $-1<x<4$.
Conclude.
On
To obtain the domain of $f(x) = \sqrt{\frac{3^x-4^x}{x^2-4x-4}}$, the expression inside the radical must be positive and the denominator of the expression must be nonzero.
The numerator $3^ x - 4^x$ is positive when $x < 0$. The denominator $x^2-4x-4$ is positive when $x > 2(1+ \sqrt{2})$ or when $x < 2 (1 - \sqrt{2})$. Hence, the expression inside the radical sign is positive when $x < 2 (1 - \sqrt{2})$ or when $0 <x < 2(2 + \sqrt{2})$.
The values where the denominator are zero are $2(1 \pm \sqrt{2})$. They are not in the stated intervals above.
Thus the domain (in interval notation) is $(- \infty,2 (1 - \sqrt{2})) \cup (0,2 (1 + \sqrt{2}))$.
On
The zeroes of the denominator are $\frac{4 \pm \sqrt{32}}{2} = 2(1 \pm \sqrt{2})$.
Therefore the denominator is negative iff $2(1 - \sqrt{2}) < x < 2(1 + \sqrt{2})$.
The upper part is positive if $x<0$, zero if $x=0$, and negative is $x>0$
So they ae both positive if $x < 2(1 - \sqrt{2})$. They are both negative if $0 \leq x < 2(1 + \sqrt{2})$.
The answer is therefore $(\leftarrow, 2(1 - \sqrt{2})) \cup [0, 2(1 + \sqrt{2}))$
In my opinion, the best way is look for the domain of $f$ from inside to outside. This means:
In the radical you have a rational function. Thus, for $f$ to be well defined, you need that the numerator does not vanish: $$x^2-4x-4\ne 0 \quad\Rightarrow\quad x\ne 2+2\sqrt{2} \quad\mbox{ and }\quad x\ne 2-2\sqrt{2}$$
The expression inside the radical has to be nonnegative (and, of course, well defined). Thus, first we need to check the sign of both the numerator and the denominator:
For the numerator, you have: $$3^x-4^x = 3^x\left(1-\left(\frac{3}{4}\right)^x\right).$$ Note that for any $a>0$, $a^x>0$ for all $x\in\mathbb{R}$. Moreover, $a^x\ge 1$ if and only $x\ge 0$. Therefore: $$3^x-4^x \ge 0 \Leftrightarrow x\le0,$$ $$3^x-4^x \le0 \Leftrightarrow x\le0.$$
For the denominator, you already have the roots of the polynomial. Now, you need to check which is the sign that the polynomial takes in each part of the real line limited by those (for instance, take any value in that interval and evaluate the polynomial). $$x\in(-\infty,2-2\sqrt2) \Rightarrow x^2-4x-4 > 0$$ $$x\in(2+2\sqrt2,\infty) \Rightarrow x^2-4x-4 > 0$$ $$x\in(2-2\sqrt2,2+2\sqrt2) \Rightarrow x^2-4x-4 < 0$$
For the rational expression to be nonnegative, both the numerator and the denominator must have the same sign (and the denominator, nonzero). This implies that:
For both having positive sign: $$x\in(-\infty,0]\cap\left((-\infty,2-2\sqrt2)\cup(2+2\sqrt2,\infty)\right) \Rightarrow x\in(-\infty,2-2\sqrt2)$$
For both having negative sign: $$x\in[0,\infty)\cap(2-2\sqrt2,2+2\sqrt2) \Rightarrow x\in[0,2+2\sqrt2)$$
In conclusion, the domain is the set of points where the expression inside the radical is nonnegative (ontained in point 3) and well defined (obtained in point 1). That is: $$\mathcal{D} (f) = (-\infty,2-2\sqrt2)\cup[0,2+2\sqrt2)$$