In deep linear networks with orthonormal feedforward weights, we prove that local-SSL and BP have identical gradient updates. If orthonormality is broken by shrinking layer widths, we prove that direct feedback from the last layer makes local-SSL gradients more similar to BP gradients.
