A creative new paper by Phillipe Goulet Coulombe at Université du Québec à Montréal, "A Neural Phillips Curve and a Deep Output Gap," introduces "hemisphere" NNs. Just look at his Figure 1 and you'll understand the structure instantly. Of course hemisphere structure is a significant restriction, not without costs, but it may be reasonable in many cases, and then it it delivers significant benefits (not least, easily-interpreted results). Phil's Phillips curve analysis is a fine example.
Maybe one could allow for long memory in recurrent hemisphere NNs, or "long short-term memory," as mentioned in some recent blogs. Interesting that both Phil's paper and the Paranthos paper focus on inflation. Long memory (fractional integration) in inflation was always a classic empirical finding supporting the idea of fractional integration/differencing (e.g., first differencing the log price level to convert to inflation seems not enough, but twice differencing seems too much).