A connected sum of two links $K$ and $L$ involves cutting a segment in each link and joining them up as illustrated in the top diagram, the connected sum of two trefoil knots. Is there any terminology for taking a link and doing the same process but to itself, as illustrated by the bottom diagram, which takes the top diagram and does this to it. What can I know about the primality of such a link? The top diagram is clearly a composite knot, but is the bottom diagram composite?
The connect sum of oriented knots is well defined and the common usage. There is no (as far as I know) well-defined connect sum operation on either un-oriented knots or links with more than one component. This is because one needs to check that we can define the connect sum in a canonical way - that is, the choices involved in taking the connect sum will not matter when we consider the product of the operation up to ambient isotopy.
This clearly can't be done for links because, for instance, there are two distinct ways of taking a connect sum of the disjoint union of a trefoil and an unknot with a single trefoil. The first results in a link isotopic to the disjoint union of two trefoils, and the second results in a link isotopic to the disjoint union of an unknot and a connect sum of two trefoils.
The great thing about the connect sum of knots is that it doesn't matter where on the knots we 'cut and glue' as long as we glue with the corresponding orientations, because there is an ambient isotopy taking any one such connect sum to another. Therefore it is well defined up to ambient isotopy (which is all we care about usually in knot theory).
As to whether we can define some operation on a single (oriented) knot which is something like taking the connect sum of the knot within itself, this seems unlikely because there is no way to seperate a knot in space, as we can do with two distinct dijoint knots when we formally construct the usual connect sum. As such, there's no canonical way of forming the usual 'strip' between removed segments of the knot. There may be a way of instead forming a family of such connect sums (probably one for each twist of the strip) but I think even then, there would be problems with showing that it doesn't matter where you decide to remove segments and glue the strip.
As an exercise, show that there is one 'connect sum within the unknot' which gives you a disjoint union of two unknots, and there is also another which gives a Hopf link (twist the strip).