It's a bit complicated, so bear with me.
optional<T> has a constructor that takes an in_place_t and an initializer_list<U> (and optional extra arguments). It will then attempt to construct T as if by this statement:
T t(list, ...);
Where list is the initailizer list and ... are any extra arguments.
T is of course a variant<pair, set>. variant is implicitly convertible from any type V through a template constructor. However, this constructor only exists if V is a type such that, given a value v of this type, the following is true:
W w(std::forward<V>(v));
Where W (I'm running out of letters here) is one of the types in the variant. Now variant can have multiple types, so variant basically considers all possible Ws and their constructor overloads. So long as overload resolution selects exactly one such constructor among all possible Ws in the variant, the conversion will work.
pair has no constructor that takes an initializer_list, so none of its constructors count. set<X> has a constructor that takes an initializer_list<X>.
So, back to the original statement. Because optional had an initializer_list<Y> parameter where Y is deduced, the braced-init-list will deduce Y as int. And int is the same type as the set in the variant. So set<int> can be constructed from an initailizer_list<int>.
And therefore, that's what gets created in the variant.
If you had a vector<int> as part of the variant, you'd get a compile error due to the ambiguity of which to call.
And there's not really a way to fix that through in_place gymnastics. You can't do (std::in_place, std::in_place_type<std::set>, {1, 4}), because the braced-init-list won't get properly deduced. So you'd have to do (std::in_place, std::set{1, 4}) and let move construction move from the temporary into the variant.