How can I find the number of binary search trees up to a given height $h$, not including BSTs with height greater than $h$ for a given set of unique numbers $\{1, 2, 3, \ldots, n\}$?
For example, if the number of nodes is $4$, then the total number of BSTs is $14$. Out of these $14$ BSTs I have to find out the number with height not more than $3$.
We define the height of a tree to be the length of a longest path from the root to a leaf (i.e. the number of edges (not vertices) that such a path contains).
Let $b(h,n)$ be the number of binary search trees with height at most $h$ containing $n$ distinct positive integers as their keys. If the $i$-th largest number is the root, then the left sub-tree must contain $i-1$ keys (the ones smaller than the $i$-th largest) and the right sub-tree must contain $n-i$ keys and both must be of height at most $h-1$. Therefore: $$ b(h,n) = \sum_{i=1}^{n} b(h-1,i-1) \cdot b(h-1,n-i) $$ Note that we have $b(0,0) = 1,$ $b(0,1) = 1$ and $b(0,k) = 0,$ $b(k,0)=1$ for any integer $k > 1,$ which takes care of boundary conditions.
Your example asks for $b(3,4)$ which is still just $14$, since all BSTs on $4$ vertices have height at most $3$. However what you probably meant was $b(2,4)$ which is $4$.
EDIT: Actually $b(2,4)$ is 6 (as was correctly pointed out by @Aayush Aggarwal). The $b(2,4)$ was the result of a miscalculation on my part. The general forumla given above is still correct.