I'm doing Bayesian recruitment curve fitting where my curve has two parameters $a$ and $b$. Both $a, b \in \mathbb{R}^+$. I have put a Truncated Normal prior on $a$ and a Half Normal prior on $b$. I'm not modeling the covariance on these two parameters.
I want to enforce a regularization penalty that only discourages small $a$ when $b$ is very small.
Basically, I'm looking for a penalty function $f(a, b)$ for $a, b$ which adds a very high positive penalty when both $a, b$ are very small.
Could you please help find such a function?
For instance, let's say I had another parameter $c \in \mathbb{R}$ with a Gaussian prior, and suppose I wanted to discourage negative values for $c$, I would use a penalty function $f(a) = |a| - a$. (see attached figure)
Since the penalty function that I'm looking for is of two variables, I'm having a hard time coming up with it. Any help will be highly appreciated.