I don't understand this map tuple key compilation error, in F#

This is due to method call syntax.

TryGetValue is not a function, but a method. A very different thing, and a much worse thing in general. And subject to some special syntactic rules.

This method, you see, actually has two parameters, not one. The first parameter is a key, as you expect. And the second parameter is what's known in C# as out parameter - i.e. kind of a second return value. The way it was originally meant to be called in C# is something like this:

Dictionary<int, string> map = ...
string val;
if (map.TryGetValue(42, out val)) { ... }

The "regular" return value of TryGetValue is a boolean signifying whether the key was even found. And the "extra" return value, denoted here out val, is the value corresponding to the key.

This is, of course, extremely awkward, but it did not stop the early .NET libraries from using this pattern very widely. So F# has special syntactic sugar for this pattern: if you pass just one parameter, then the result becomes a tuple consisting of the "actual" return value and the out parameter. Which is what you're matching against in your code.

But of course, F# cannot prevent you from using the method exactly as designed, so you're free to pass two parameters as well - the first one being the key and the second one being a byref cell (which is F# equivalent of out).

And here is where this clashes with the method call syntax. You see, in .NET all methods are uncurried, meaning their arguments are all effectively tupled. So when you call a method, you're passing a tuple.

And this is what happens in this case: as soon as you add parentheses, the compiler interprets that as an attempt to call a .NET method with tupled arguments:

brain.Positions.TryGetValue (x.Instrument, x.Side)
                             ^             ^
                             first arg     |
                                           second arg

And in this case it expects the first argument to be of type Instrument * Side, but you're clearly passing just an Instrument. Which is exactly what the error message tells you: "expected to have type 'Instrument * Side' but here has type 'Instrument'".

But when you add a second pair of parens, the meaning changes: now the outer parens are interpreted as "method call syntax", and the inner parens are interpreted as "denoting a tuple". So now the compiler interprets the whole thing as just a single argument, and all works as before.

Incidentally, the following will also work:

brain.Positions.TryGetValue <| (x.Instrument, x.Side)

This works because now it's no longer a "method call" syntax, because the parens do not immediately follow the method name.

But a much better solution is, as always, do not use methods, use functions instead!

In this particular example, instead of .TryGetValue, use Map.tryFind. It's the same thing, but in proper function form. Not a method. A function.

brain.Positions |> Map.tryFind (x.Instrument, x.Side)

Q: But why does this confusing method even exist?

Compatibility. As always with awkward and nonsensical things, the answer is: compatibility.

The standard .NET library has this interface System.Collections.Generic.IDictionary, and it's on that interface that the TryGetValue method is defined. And every dictionary-like type, including Map, is generally expected to implement that interface. So here you go.

In future, please consider the Stack Overflow guidelines provided under How to create a Minimal, Reproducible Example. Well, minimal and reproducible the code in your question is, but it shall also be complete...

…Complete – Provide all parts someone else needs to reproduce your problem in the question itself

That being said, when given the following definitions, your code will compile:

type Instrument() = class end
type Side() = class end
type PositionData = { Instrument : Instrument; Side : Side; }
    with member __.UpdateTime = 0
module brain =
    let Positions = dict[(Instrument(), Side()), {Instrument = Instrument(); Side = Side()}]
let positions = []

Now, why is that? Technically, it is because of the mechanism described in the F# 4.1 Language Specification under §14.4 Method Application Resolution, 4. c., 2nd bullet point:

If all formal parameters in the suffix are “out” arguments with byref type, remove the suffix from UnnamedFormalArgs and call it ImplicitlyReturnedFormalArgs.

This is supported by the signature of the method call in question: System.Collections.Generic.IDictionary.TryGetValue(key: Instrument * Side, value: byref<PositionData>)

Here, if the second argument is not provided, the compiler does the implicit conversion to a tuple return type as described in §14.4 5. g.

You are obviously familiar with this behaviour, but maybe not with the fact that if you specify two arguments, the compiler will see the second of them as the explicit byref "out" argument, and complains accordingly with its next error message:

Error 2 This expression was expected to have type PositionData ref but here has type Side

This misunderstanding changes the return type of the method call from bool * PositionData to bool, which consequently elicits a third error:

Error 3 This expression was expected to have type bool but here has type 'a * 'b

In short, your self-discovered workaround with double parentheses is indeed the way to tell the compiler: No, I am giving you only one argument (a tuple), so that you can implicitly convert the byref "out" argument to a tuple return type.