Optimizing a real value function is a popular field where many optimization algorithms have been proposed such as descent of gradient, levenberg-marquardt,...
However, suppose that the function is boolean and define as f:{0, 1}^n:-->R, so in this case we can not used the algorithms cited above. So, what is the approach and the popular algorithms used to optimize such functions? Is it possible to do linear or non linear regression in this case?
Optimizing a boolean function is NP-Hard in general, so I wouldn't say that there is one popular approach that would work every time. There are exact methods and heuristics. The exacts methods might not be quick enough depending on the complexity of the boolean function and there are no guarantees that the heuristics will ever find the optimal solution.
A good starting point would be integer programming. It is possible to reduce integer programming to SAT. The most popular algorithms for finding solutions to NP-Hard problems would have to be SAT solvers, so one could use a SAT solver to optimize f. This is one way of optimizing f with readily available solvers, although it might not be the most time-efficient way as a reduction often implies greater computational complexity.