What type of famous equation except diophantine equation such that no algorithm can exist to determine whether there is a solution?
I know that if these equation have a solution, then it could be solving with finite amount of time.
By the way, how long does mathematician tend to give up for solving such equations?
The impossibility of solving diophantine equations in general leads to a number of other impossibility results. For example, Richardson's theorem http://en.wikipedia.org/wiki/Richardson%27s_theorem says that in a certain class of functions there is no algorithm to test whether the function is identically $0$.