A friend of mine introduced this game to me, and apparently he also learnt about it from a friend of his. We unfortunately don't know the source, but we are deeply interested in studying it. The game is as follows:
With a regular deck facedown, start revealing cards as you count the number of the cards you revealed, say $r$. When $r$ matches the value of the card (with A representing $1$; J, $11$; Q, $12$; and K, $13$), put the matching card aside, and append the rest of the stack to the tail of the deck, again, facedown. Then, start counting again. If a match does not occur; meaning that you have counted to $13$ (your remaining cards can also be less than $13$; in that case, you have counted all your cards), and every card had a different value than its corresponding $r$, then you stop and note the number of the cards you have extracted. Find an initial order of the deck that extracts $51$ cards.
Example. The following initial order will extract $50$ cards. (Note about the format of the list: the first element is the value of the card at the top of the deck.)
[13 8 9 4 1 6 8 10 4 5 5 7 6 9 8 10 6 9 4 1 2 11 11 13 11 2 10 13 11 3 12 4 2 5 7 12 10 7 3 12 1 9 7 5 3 12 6 1 8 3 2 13]
Note that it's not possible to extract all of the cards. If it would, your last card would be $1$, that implies that the card put aside before it is also $1$, and so on. But there is only four $1$'s in the deck.
We suspects it's an exercise from an algebra or combinatorics book. Is this game well-known/studied?
Solution to the puzzle
Here is a solution that extracts 51 out of 52 cards, in the same format you used:
Here is the order of card values we pick while playing out the solution above:
C++ code and program explanation
I found this solution using a solver I coded. My full C++ code is available here; it's pretty messy, but you could probably at least see how to run the code for other deck sizes if you wanted. In terms of understanding, I think the method will be more interesting than the code itself, so I'll explain it here.
My overall approach was backtracking, which basically means "I don't have a concrete solution, so just try random guesses, and if we run into a contradiction then undo the last guess(es) and try again". The program starts with a blank deck and fixes the cards one at a time by choosing what the next card value will be and then placing that card accordingly. At each step, the solver needs to figure out what guesses to try next and also be able to undo guesses when necessary. To accomplish that, the program keeps track of several pieces of information:
Speeding up the program
By implementing the ideas above, I could make a correct solver. However, it would also take way way way too long to solve the case with the 52 card deck. (There are $\binom{52}{4} \binom{48}{4} \cdots \binom{4}{4} \approx 9.2 \times 10^{49}$ possible deck orderings to consider!) The key to a successful backtracking algorithm is finding good ways to a) choose good paths first most of the time and/or b) realize quickly when we're going down a doomed path so we can back up right away instead of wasting lots more time.
The main efficiency-boosting tricks I used in this program were:
1, 1, 1, 1. After coding a brute-force solver and running it for some smaller deck sizes, I decided it seemed usually possible to find a solution under this constraint. Therefore I told the solver to assume this just to get the 1s out of the way and reduce the effective deck size by 4 cards.With these tricks, the program took around 1 hour to find a solution for the 52 card deck.
Bonus: Other deck sizes
I also had my solver run the same problem with different deck sizes. Let $R$ denote the number of ranks (# values per suit) and let $C$ denote the number of suits (how many copies of each value). The program treats both $R$ and $C as parameters that can be easily changed. For example, I tried running with $R$ ranging from $2$ up to $13$ while keeping $C = 4$ the whole time and I got these example solutions:
I of course still encourage you to investigate further, starting from my code or pencil and paper or however else. Just because you have a solution for the 52-card deck doesn't mean there's no more fun to be had thinking about this puzzle :)
Final note: Citations
I do realize that what you actually asked for here was a citation, not a solution! I haven't heard of this problem before and I did a bit of searching but didn't find anything. My only "input" here is that I think you should check sources beyond just algebra and combinatorics; for example, it seems possible that the best solution involves some kind of programming (maybe more efficient than mine), in which case you'd want to try including "codeforces" or "atcoder" in your search queries.
Of course, if you knew for sure whether it's an example in a number theory vs. discrete math vs. programming textbook then that would go a long way toward hinting what strategy you should use!
Anyway, it's also possible this problem is just not very known/studied/published. Shrug, I'll watch with interest to see if someone else comes up with a citation for you.