Is there an established way of decomposing a discrete periodic (complex) signal into a sum of non-sinusoidal periodic waveforms (eg square, triangle, and sawtooth)?
For my use case the input waveform is periodic, but as I understand that's a condition which always holds for DFT even if it's easy to work around it in practical applications.
On
If you are after a random basis and you know some statistical properties of the target family of waveforms, have you looked into using a Karhunen-Loève Expansion (KL-Expansion)? In a KL-Expansion, the set of eigen-vectors of the auto-correlation operator of the signal x(t) is a basis (demonstrates convergence in probability) for x(t).
In the discrete x(n) case, you can calculate a quasi-basis using an auto-correlation matrix of x(n) and Cholesky Decomposition. See Clarkson (1993) Optimal and Adaptive Signal Processing page 131
Have you looked at using B-splines? The 0th-order B-spline is a simple pulse, 1st-order (convolution of 2 0th order B-splines) is a triangle, and they keep going up from there...
For whatever transform you want to use/develop, arguably one of the most basic requirements is that they possess the partition of unity property (see Jawerth and Sweldens (1993))—otherwise, it would be difficult to manage even approximating a constant function well. All B-splines have partition of unity property.
B-splines also have a nice connection to the Daubechies wavelets: each p-order Daubechies scaling function has a p-order B-spline component that show up as p zeroes in the complex z-plane.