What is an object?

The error you are getting is not due to the nature of a function object, but to the fact that the argument of names<- is an "anonymous" object and not one which has been linked to a symbol. Consider this:

names(1:10)<-letters[1:10]
#Error in names(1:10) <- letters[1:10] : 
#  target of assignment expands to non-language object 

Same output. In the error, non-language looks like a very convoluted way to say "not a symbol". However, that's pretty logical that you have an error, since you are calling a replacement function and the symbol whose content you asked to replace isn't there. At its root, it has to do with <- operator which expects the left hand side of an assignment to be a symbol (a language object) and not a "standard" object:

1:10<-setNames(1:10,letters[1:10])
#Error in 1:10 <- setNames(1:10, letters[1:10]) : 
#  target of assignment expands to non-language object

Same error again.

This works, of course:

a<-1:10
names(a)<-letters[1:10]

Getting to your example:

f<-function(x) x*x
names(f)<-"square"
# Error in names(f) <- "square" : names() applied to a non-vector

You get the above error if you call explicitly names<- with the "anonymous" function:

"names<-"(function(x) x*x, "square")
# Error: names() applied to a non-vector

Now the error is due to the fact that names<-, although a generic, does not have a method for functions. This is documented in ?names, Details section:

‘names’ is a generic accessor function, and ‘names<-’ is a generic replacement function. The default methods get and set the ‘"names"’ attribute of a vector (including a list) or pairlist.

I just add a couple of lines regarding the main question: "What is an object". In R, everything is an object. The is.object function has a misleading name and a better name should have been is.internalobjectbitset. The doc is pretty clear. The title say: "Is an Object 'internally classed'?" and not something like "is a variable an object?"; in the argument section, for x it says: x: object to be tested, which is rather strange if you want to test whether x is an object in the first place. It also specifies that is intended mainly for internal use and not as a way to define what is an object (which would be a silly function, since everything is an object in R).

Following the comments, I'd add that, all in all, the doc misses to state that, while you can apply names to any R object, you cannot assign names to some types. In my opinion, it's a very minor issue, since names are not just an ordinary attribute, but are very linked to some properties of the language and specifically subsetting. Only objects that can be subset can have names. A function object is not meant to be subset and so R prevents to assign a names attribute which would be basically of no use.

Looking at R source code (names.c) :

{"is.null",       do_is,        NILSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.logical",    do_is,        LGLSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.integer",    do_is,        INTSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.double",     do_is,        REALSXP,1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.complex",    do_is,        CPLXSXP,1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.character",  do_is,        STRSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.symbol",     do_is,        SYMSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.name",       do_is,        SYMSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.environment",do_is,        ENVSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.list",       do_is,        VECSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.pairlist",   do_is,        LISTSXP,1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.expression", do_is,        EXPRSXP,1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.raw",        do_is,        RAWSXP, 1,  1,  {PP_FUNCALL, PREC_FN,   0}},

{"is.object",     do_is,        50,     1,  1,  {PP_FUNCALL, PREC_FN,   0}},

{"isS4",          do_is,        51,     1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.numeric",    do_is,        100,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.matrix",     do_is,        101,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.array",      do_is,        102,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.atomic",     do_is,        200,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.recursive",  do_is,        201,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.call",       do_is,        300,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
{"is.language",   do_is,        301,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},

{"is.function",   do_is,        302,    1,  1,  {PP_FUNCALL, PREC_FN,   0}},
...

shows that objects(50) are a specific type, different from function(302), or other types (numeric, matrix,...)

This is confirmed in definition of do_is in coerce.c:

    case 50:        /* is.object */
    LOGICAL0(ans)[0] = OBJECT(CAR(args));
    break;

Objects have the following methods:

/* Objects */
{"inherits",    do_inherits,    0,  11, 3,  {PP_FUNCALL, PREC_FN,   0}},
{"UseMethod",   do_usemethod,   0,     200, -1, {PP_FUNCALL, PREC_FN,   0}},
{"NextMethod",  do_nextmethod,  0,     210, -1, {PP_FUNCALL, PREC_FN,   0}},
{"standardGeneric",do_standardGeneric,0, 201,   -1, {PP_FUNCALL, PREC_FN,   0}},

Which points to base R documentation :

R possesses a simple generic function mechanism which can be used for an object-oriented style of programming. Method dispatch takes place based on the class(es) of the first argument to the generic function or of the object supplied as an argument to UseMethod or NextMethod.

This shows that objects of type 50, detected by is.object, are simply S3 classes or S4 classes, and is confirmed by is.object documentation:

is.object - Is an object internally classed? It returns TRUE if the object x has the R internal OBJECT bit set, and FALSE otherwise. The OBJECT bit is set when a "class" attribute is added and removed when that attribute is removed, so this is a very efficient way to check if an object has a class attribute. (S4 objects always should.)

In short : an internal object has the class attribute set.

This is quite restrictive compared to objects in common language. Not sure R documentation always means objects(50) when speaking about objects.

Regarding names(function()) error, see clarification by @nicola.

A syntax error and a poorly worded error message says little about the status of objects in the language. This is a red herring.