Edit: My mistake, the PDF does in fact integrate to 1. Nothing interesting here.
We are given the following joint distribution PDF: $$f_{X,Y}\left(x,y\right)=\begin{cases} xy+\frac{3}{4} & x,y\in(0,1]\\ 0 & \text{Otherwise} \end{cases}$$
And asked to calculate the PDF of the sum $Z=X+Y$
My attempt:
Let Z=X+Y. Obviously $f_{Z}\left(z\right)=0$ if z>2 or $z\leq0$. If $z\in(0,2]$ we divide into two cases:
• $z\in(0,1]$
$$f_{Z}\left(z\right)=\int_{\mathbb{R}}f_{X,Y}\left(x,z-x\right)dx\overset{_{(1)}}{=}\int_{0}^{z}x\left(z-x\right)+\frac{3}{4}dx=\frac{zx^{2}}{2}-\frac{x^{3}}{3}+\frac{3}{4}x\Big|_{x=0}^{x=z}=\frac{z^{3}}{6}+\frac{3}{4}z$$
• $z\in(1,2]$ $$f_{Z}\left(z\right)=\int_{\mathbb{R}}f_{X,Y}\left(x,z-x\right)dx\overset{_{(1)}}{=}\int_{z-1}^{1}x\left(z-x\right)+\frac{3}{4}dx=\frac{zx^{2}}{2}-\frac{x^{3}}{3}+\frac{3}{4}x\Big|_{x=z-1}^{x=1}==\left(\frac{z}{2}-\frac{1}{3}+\frac{3}{4}\right)-\left(\frac{z\left(z-1\right)^{2}}{2}-\frac{\left(z-1\right)^{3}}{3}+\frac{3}{4}\left(z-1\right)\right)=-\frac{z^{3}}{6}+\frac{z}{4}+\frac{5}{6}$$
(1) - Since if $z-x\notin(0,1]$ we have $f_{X,Y}\left(x,z-x\right)=0$
$$f_{Z}\left(z\right)=\begin{cases} \frac{z^{3}}{6}+\frac{3}{4}z & z\in(0,1]\\ -\frac{z^{3}}{6}+\frac{z}{4}+\frac{5}{6} & z\in(1,2]\\ 0 & \text{Otherwise} \end{cases}$$
All the calculations check out. However, $f_{Z}\left(z\right)$ integrates to $1\frac{5}{12}$. I fail to see the fault in my solution, any ideas?
Your density looks fine; I don't see how you got $17/12$:
$$\begin{align*} \int_{z=0}^1 \frac{z^3}{6} + \frac{3z}{4} \, dz + \int_{z=1}^2 -\frac{z^3}{6} + \frac{z}{4} + \frac{5}{6} \, dz &= \left[\frac{z^4}{24} + \frac{3z^2}{8} \right]_{z=0}^1 \!\!\! + \left[-\frac{z^4}{24} + \frac{z^2}{8} + \frac{5z}{6} \right]_{z=1}^2 \\ &= \left(\tfrac{1}{24} + \tfrac{3}{8}\right) + \left(-\tfrac{16}{24} + \tfrac{4}{8} + \tfrac{10}{6}\right) - \left(-\tfrac{1}{24} + \tfrac{1}{8} + \tfrac{5}{6}\right) \\ &= 1. \end{align*}$$