Given is an $n\times n$ board with $n\geq 3$. We place a chip in some cells, so that no four chips form a rectangle with sides parallel to the sides of the board. How many chips can we place, at most?
We can place $2n$ chips in positions $(i,i)$ for $1\leq i\leq n$, $(i,i+1)$ for $1\leq i\leq n-1$, and $(n,1)$. Is it possible to improve?
$$\begin{array}{cccc}X&X&X&-\\ -&X&-&X\\ X&-&-&X\\ -&-&X&X\end{array}$$
The $X$'s are coins. This is a counterexample with $N = 4$.
In general, consider $S_k \subset \{1, \ldots, N\}$ consisting of the columns which are filled in row $k$.
The requirement is precisely $|S_k \cap S_\ell| \leq 1$ for all $k \neq \ell$.
An asymptotic lower bound from finite projective planes:
Collections of subsets of $\{1,\ldots, n\}$ such that any pair intersects in at most one are fairly interesting.
A special case of these are given by finite projective planes, where the set is the set of points and the subsets are the lines. This is a much stricter requirement, of course. There may be a clean solution for the maximum in general in your case.
Finite projective planes give us $n^2+n+1$ points, $n^2+n+1$ lines, and $n+1$ points per line.
This gives an $N\times N$ matrix, with $N = n^2+n+1$, with $(n+1)N \approx N\sqrt{N}$ coins.
So this is an asymptotic lower bound (at least restricted to $N$ of this form where $n$ is a prime power).
An asymptotic upper bound:
Let $s_k = |S_k|$. Then we can't repeat a pair, so
$\sum_k {s_k \choose 2} \leq {N \choose 2}$.
Thus $\sum_k s_k(s_k-1) \leq N(N-1)$.
Let's maximize $\sum_k s_k$ subject to $\sum_k s_k(s_k-1) \leq M$.
Let $\mathbf{s}$ be the vector $(s_1,s_2,\ldots,s_N)$. Then
We have $\mathbf{s} \cdot (\mathbf{s} - \mathbf{1}) \leq M$ and we want to bound $\mathbf{s}\cdot \mathbf{1}$.
We have $\mathbf{1} \cdot \mathbf{s} \leq |\mathbf{s}||\mathbf{1}| = \sqrt{N} \mathbf{s}$
So we get $\mathbf{1} \cdot \mathbf{s} \leq \sqrt{N}\sqrt{M + \mathbf{1}\cdot \mathbf{s}}$.
So we have $A \leq \sqrt{N} \sqrt{M+A}$. Squaring we get $A^2 \leq N (M+A)$. So we have $(A - N)A \leq NM$. Thus $A-N \leq \sqrt{NM}$. Thus $A \leq \sqrt{NM}+N$.
Thus $\sum_k s_k \leq \sqrt{N} \sqrt{N(N-1)} + N \approx N \sqrt{N}$.
Some amusement:
The kids card game Spot It! has 55 cards, each with 8 animals (and 57 different animals in total). All the cards are different, and any pair of cards has exactly one pair of animals in common. This is $\mathbb{PF}_7$ with two missing lines (probably for printing reasons).
There's also Spot It Jr.! with 31 cards and 6 animals per card (and 31 different animals in total). This is $\mathbb{PF}_5$.