First thing to bear in mind: This expression will not print the full source code of any file you place it in. Only in this one special case, when the source code contains only this line (followed by a new-line character, if we're being pedantic), will it print its own source code when this file is run.
Now let's pick it apart, starting from the beginning (output from running this should be the same at each stage):
print((lambda str='print(lambda str=%r: (str %% str))()': (str % str))())
First of all, let's separate the lambda expression from the print statement (you can read about lambda expressions here, they're essentially a concise notation for defining "small anonymous functions"):
f = lambda str='print(lambda str=%r: (str %% str))()': (str % str)
print(f())
Now, to be a bit more readable, we can replace this lambda expression with an equivalent function:
def f(str='print(lambda str=%r: (str %% str))()'):
return str % str
print(f())
Now we can see that the print statement is equivalent to this:
str = 'print(lambda str=%r: (str %% str))()'
print(str % str)
This is an example of old-style string formatting, which you can read more about here. Essentially, when a string is followed by the % operator, it does something which is conceptually similar to the string.format method.
In particular, if we have an expression like str1 % obj, any instance of the sub-string "%%" in str1 gets replaced by the sub-string "%", and if the sub-string "%r" is present only once, it gets replaced by the result of calling repr(obj). To give a concrete example, the output from calling print("Test string: %% %r %%" % 23) is Test string: % 23 %.
Now, back to the Quine, what we have is equivalent to this:
r_string = repr('print(lambda str=%r: (str %% str))()')
print('print(lambda str={}: (str {} str))()'.format(r_string, "%"))
These statements should make sense if you have a reasonable understanding of Python (you may want to look up the repr function and the string.format method), and the output is:
print((lambda str='print(lambda str=%r: (str %% str))()': (str % str))())
Which was the original source code.
PS if you're question was not "How does this syntax work" but rather "How did someone come up with this particular example in the first place", then (off the top of my head) I have no idea.
But in general, it's easy for a program to print its own source code. All it has to do is contain the statement print(*open(__file__), sep=""). In this example, the file that this statement is saved in can contain other commands, too.
__file__ is a special variable in Python that contains the filename of the source code as a string, and the * does "iterable unpacking".
EDIT: I think your original code might have been missing some brackets? I think it should have been this (note 2 open brackets after print in the string literal, and an extra closing bracket before the end of the string literal):
print((lambda str='print((lambda str=%r: (str %% str))())': (str % str))())