Aluffi III.6.6 suggests proving the following:
Let $R$ be a ring, and let $F = R^{\oplus n}$ be a finitely generated free $R$-module. Prove that $\text{Hom}_{R-\text{Mod}} (F, R) \cong F$.
I ended up with the following construction. Let $H = \text{Hom}_{R-\text{Mod}} (F, R)$, $A = \{ 1, \dots, n \}$ and consider set-function $f : A \rightarrow H$, $f : i \mapsto \alpha_i$ where $\alpha_i : (r_1, \dots, r_n) \mapsto r_i$. This set-function gives rise to an unique $\sigma : F \rightarrow H$ (since $F$ is free). Then we just need to build a homomorphism $\psi : H \rightarrow F$ and prove that $\sigma \psi = \text{id}_F$ and we're done (since $\psi \sigma = \text{id}_H$ follows from $F$ being free).
Now, let $\alpha \in H$. It feels natural to map $\psi : \alpha \mapsto (\alpha(1, 0, \dots, 0), \alpha(0, 1, 0, \dots, 0), \dots, \alpha(0, \dots, 0, 1))$. Unless I messed up with the (boring) proof boiling down to substituting the definitions for $\sigma$ and $\psi$, $\sigma \psi = \text{id}_F$ really holds, and indeed we're done.
Does this look correct?
I also tried going straight from the definition of a free module and tried proving that $\text{Hom}_{R-\text{Mod}} (F, R)$ satisfies the corresponding universal property, but I wasn't particularly successful at that. So, is there a better (more algebraic or category-theoretic) proof of the claim, preferably accessible to me at this point in the book?
Note that, due to the universal property of $F$, for any $f\in F$ there is a unique homomorphism $h:F\rightarrow R$ making the corresponding diagram commutative. So, just send $f$ to $h$.
P.S. In fact, proving this really gives a homomorphism, I found out that the action of $R$ is preserved only if $R$ is commutative.