Let the $M$ be a smooth manifold with nonempty boundary. I need to prove that there exists a smooth function $f:M\rightarrow \mathbb{R}$ such that:
- $\forall x\in\partial M$ we have $f(x)=0$ and $rank(f,x)=0$,
- $\forall x\in int(M)$ we have $f(x)\neq 0$
For this task I used partition of unity. Let $(U_{\alpha} ,\phi_{\alpha} )$ be an open cover of $M$ wit chart sets. For every set we define a smooth mapping $f_\alpha : U_\alpha\rightarrow R$ as follows:
- If $U_\alpha \cap \partial M =\emptyset$ then $f_\alpha (p) = 1$
- If $U_\alpha \cap \partial M \neq\emptyset$ then $f_\alpha (p)$ will be a projection of $\phi_\alpha (p)$ to a n-th coordinate $(\pi (\phi_\alpha (p))=\pi (x_1,x_2,\dots ,x_n)=x_n$.
Now let the $\lbrace h_\alpha\rbrace$ be a partition of unity in $U_\alpha$. We define $f=\sum_\alpha h_\alpha f_\alpha$ as follows: $$h_\alpha f_\alpha (p) = \begin{cases} h_\alpha (p) f_\alpha (p), & \text{if $p\in U_\alpha$} \\\\ 0, & \text{if $p\notin U_\alpha$} \end{cases}$$
- Clearly $f$ is smooth (both $f_\alpha$ and $h_\alpha$ are smooth).
- If $p\in\partial M$ then if $p\in U_\alpha$, we have $f_\alpha (p) = 0$ so $f(p)=0$.
- If $p\in int(M)$ then $f_\alpha (p) = 1$ when $U_\alpha \cap \partial M=\emptyset$ and if $U_\alpha \cap \partial M\neq\emptyset$ we have $f_\alpha(p)=\pi(\phi_\alpha (p))>0$ because $\phi_\alpha (p)\notin\partial \mathbb{H}^n$ so $\sum_\alpha h_\alpha f_\alpha \neq 0$.
What about the $rank(f,p)$ for points in boundary of manifold $M$? I don't know if the function I made satisfy the condition and if it does - how to prove it?
Even if $M$ were the upper half-space, then $f(x_1,\dots,x_n) = x_n$ does not have rank $0$ at the boundary (the rank is $1$ there). The remedy should be to choose a function that is flat at the boundary $x_n = 0$, and positive for $x_n > 0$. You might try $f(x_1,\dots,x_n) = e^{-1/x_n}\mathbf 1_{\{x_n > 0\}}$ for your local definition at the boundary.