I got in a healthy discussion with a Computer Science Professor about: Is Gradient Descent (GD) an heusristic method?
He stated is an heuristic method since can converge to different results.
I said that isn't because GD falls always to same local minima if started in same initial conditions.
I forward this discussion to MATH.SE community: Is GD an heuristic method?
What characterize an heuristic method?
Can we say that there are problems more suited to heuristics methods than non heuristic ones (first we need to define what is an heuristic).