Other ways to word the question so that it's clear:
In a game where players hold a maximum of thirteen cards and a minimum of zero cards, how many possible positions are there?
How many possible Tien Len (Vietnamese Poker, Thirteen) positions are there?
If a person is holding 52 distinct cards and 52 identical cards, how many ways can this person give thirteen cards to player one, thirteen cards to player two, thirteen cards to player three, and thirteen cards to player four, where players are distinct?
This is not my homework, but I am interested in the combinatorics that would go into solving this problem.
I've figured out how to solve the problem when each player must have exactly thirteen cards. It's just ${52 \choose 13} {39 \choose 13} {26 \choose 13} {13 \choose 13}$. However, allowing players to have less than 13 cards makes it more complicated.
I'd appreciate any help. Thanks.
EDIT: I realized that I wasn't clear in my question. In some places I wrote that each player has a minimum of one card, where in others I wrote that the player has a minimum of zero cards. Honestly, I'm more interested in the setup of the solution than the actual number, so answer with whichever constraint you feel like.
Let $a,b,c,d$ represent the number of cards each of players 1,2,3, and 4 have gotten respectively.
As you correctly note, in the case that $a=b=c=d=13$, the result will specifically be $\binom{52}{13}\binom{39}{13}\binom{26}{13}\binom{13}{13}$. This expression can be simplified using multinomials instead of binomials as $\binom{52}{13,13,13,13}=\frac{52!}{13!13!13!13!}$, which equals what you found earlier.
Imagine the deck as a fifth "player" who gets all of the unused cards. He will wind up getting the remaining $52-a-b-c-d$ cards. Let us simplify by setting $e=52-a-b-c-d$. For specific choice of $a,b,c,d$, the result will be $\binom{52}{a}\binom{52-a}{b}\binom{52-a-b}{c}\binom{52-a-b-c}{d}\binom{52-a-b-c-d}{e}$, or more simply written as $\binom{52}{a,b,c,d,e}$.
As each different selection of $a,b,c,d$ will yield a "different" outcome, we can sum over all possible such choices to get an expression for the total as:
$$\sum\limits_{(a,b,c,d)\in \{1,2,\dots,13\}^4}\binom{52}{a,b,c,d,e}$$
I do not see any particular reason why this sum should simplify any further, but a calculator can handle the expression easily enough.
In case my notation is not clear, here is an alternate (more common but lengthier) way to write the same expression:
$$\sum\limits_{a=1}^{13}\sum\limits_{b=1}^{13}\sum\limits_{c=1}^{13}\sum\limits_{d=1}^{13}\frac{52!}{a!b!c!d!(52-a-b-c-d)!}$$