I've had some experimentation with torch and here is what I've found:
( 5,7,3) and ( 5,7,3) -> ( 5,7,3)- equal(5,3,4,1) and (5,3,1,1) -> (5,3,4,1)- one dim is 1(5,3,4,1) and ( 3,4,1) -> (5,3,4,1)- one dim doesn't exist(5,3,4,1) and ( 3,1,1) -> (5,3,4,1)- one dim is 1 and one doesn't exist( 4,3,2) and (5,4,1,1) -> (5,4,3,2)( 4,1) and (5,3,1,1) -> (5,3,4,1)- one dim is 1 and two don't exist( 1) and (5,3,4,2) -> (5,3,4,2)- one dim is 1 and others don't exist( ) and (5,3,4,2) -> (5,3,4,2)- scalar and tensor( 0) and (5,3,2,1) -> (5,3,2,0)- empty tensor and tensor( 4,1) and ( 4) -> ( 4,4)- missing dim is 1( 4,2) and ( 4)-> not broadcastable( 5,2,1) and (5,3,2,1)-> not broadcastable(5,3,2,1) and ( 5,3,2)-> not broadcastable
Pytorch documentation says:
Many PyTorch operations support NumPy Broadcasting Semantics.
Does that mean that pytorch follows numpy's broadcasting rules and they are the same or should I experiment with numpy also?
The documentation for numpy doesn't list specific rules as the pytourch'es one. And even pytorch'es documentation is not completely accurate, e.g. it says that each tensor should have at least one dimension but torch.empty(3,4) + torch.tensor(42) works fine (scalar has no dimensions).
The rules that I've found are:
- Starting at the trailing dimension, dim sizes must either be equal or one of them should be 1 (for missing first dims 1 is assumed).
- For each dim the resulting dim is the max of the two.
- In-place operations do not allow the in-place tensor to change shape.
It seems like the most reasonable approach to broadcasting and I don't see why it would be different for numpy. But maybe there are some features.
If someone can provide proofs or at least vouch that numpy has the same rules as pytorch then I'd be grateful since won't have to study numpy broadcasting behavior.