For illustrative purposes, suppose you had an encoder and decoder network for Syllables Per Second, SPSEnc and SPSDec. SPSEnc would output sps_latent_mu and sps_latent_logvar. Then you would sample an sps_latent point from that. SPSDec would turn sps_latent into a syl_mu and syl_logvar. The VAE loss function would be this:
>-KL(from a standard normal distribution to Normal(sps_latent_mu, e^sps_latent_logvar)) + log probability of sampling ground_truth_sylpersec from Normal(syl_mu, e^syl_logvar)

Some suggestions:
- For the Syllables Per Second loss function, replace the MSE Loss with the log probability of sampling the GT syllables-per-second from a normal distribution with parameters syl_mu and e^syl_var. This will give a more accurate loss function.
- Remove the sampler from the Syllables Per Second network. Feed the ground truth syllables-per-second value directly into the TTS Inputs triangle. This is equivalent to teacher forcing. Use the sampler only when you don't have a ground truth value.
- Remove the Regularizer Loss from the Syllables Per Second network. The Regularizer Loss is only for unsupervised latent variables, similar to how the Classifier Loss is only for supervised variables. This one should be supervised since it corresponds directly to the distribution of observable syllables-per-second.
- Provide the Speaker, Emotion, and optionally Prosody (described below) as input to the Syllables Per Second network.
- Maybe replace EmotionNet with a ProsodyNet (described below). In any case, remove EmotionNet.
- Remove the Regularizer Loss from Auxiliary EmotionNet.
- Maybe set the Auxiliary EmotionNet zs output to be a combination of the TochMoji hidden values plus the emotion classification. This is just because the TorchMoji hidden layers seem to give a really good representation of emotion.
- Add an arrow from the Auxiliar EmotionNet zs to the TTS Input Triangle that accumulates all of these encoded values. Use teacher forcing for this during training. (Use teacher forcing for all values where the GT is known.)

For ProsodyNet:
- The input should be the GT Spectrogram and optionally Syllables Per Second. (I explain how to derive this further below.) The output should be a normal distribution. This should have a Regularizer Loss associated with it. The output (a sample from the normal distribution) should be passed to the TTS Input Triangle.

This is the general idea for figuring out which networks you need:
1. Draw out the conditional dependencies between all variables. Make sure there are no cycles.
- Generated emotion depends on the input text.
- Generated syllables-per-second depends on the input speaker, input text, emotion, and (latent) prosody
- Generated spectrogram depends on the input speaker, input text, input (or inferred) emotion, and input (latent) prosody, and input (maybe inferred) syllables-per-second.
2. For all of the latent variables (just prosody in our case), create a neural network with reverse connections, and use only the observable values for inputs. Use a Regularizer Loss for these networks. In our case, since prosody should be used to calculate syllables-per-second and the spectrogram, and since both syllables-per-second and the spectrogram happen to be observable values, this one reverse network should be:
- Syllables-per-second, spectrogram -> prosody distribution (mean and variance)
3. For all of the connections with observable values, create a regular neural network. Have the network output a distribution if it's not too expensive to do so (see my example above using SPSEnc and SPSDec), and have the Classifier Loss for these networks be the probability of sampling the Ground Truth value from that distribution. In our case, these three networks should ideally be:
- Input text -> emotion
- Input speaker, input text, emotion, prosody sample -> syllables per second
- Input speaker, input text, emotion, prosody sample, syllables per second -> spectrogram

Note: I decided not to link prosody with emotion in step 1 so the resulting controls won't overlap.