The full question is: Can you modify a shallow (1 hidden layer) fully connected feed-forward neural network in some way (either adding or removing some aspect of the neural network) to produce a linear regression equation for the same input variables?
I'm still working on getting information on what it means to modify and what the extent of those modifications can be. I'm also finding out if there are any restrictions on the number of inputs, hidden layer nodes, or outputs. I'll update this post soon when I get that information. Edit So any modifications to the neural net are okay such as weights, biases, number of hidden nodes, etc. as long as it still is a fully connected feedforward network with 1 hidden layer.
I know that the main difference between a neural network (NN) and a linear regression (LR) is that an NN has an activation function on each node which allows for non-linear functions to be applied so that the input data can better fit output data that is not linear (or classification data that can't be divided with a linear boundary). The set of multiple linear regression equations for multiple outputs looks like this: 
The NN would look something like this with multiple inputs and outputs and only 1 hidden layer. 
I'm thinking that a NN can achieve that same set of equations by forcing some of the weights and biases to be equal to 1 or 0 such that value that ultimately passes through to the final output layer is the same as the multiple linear regression equations. The activation function can simply be the linear activation function (which is akin to just not having one). The biases on all nodes in the hidden layer can be 0 while the biases on the final output layer could simply be retained to whatever their true value is.
Any thoughts on how to approach the problem? Thanks.
I think you mean "multivariate linear regression" (multiple outputs) rather than "multiple linear regression" (multiple inputs but one output) [see here].
Multivariate linear regression is equivalent to a network without hidden layers and without activation functions. If you must have a hidden layer, you can always have each node of the hidden layer pass its output directly to a corresponding node of the output layer, as you have suggested.