Is file descriptor table necessary?

According to what I have read, processes get a fd that indexes into user fd table that in turn returns a pointer into file table that maintains offset, pointer to inode and other properties of the opened file. So fd is basically maintaining a way to access file on disk, its position in the file. So cant commands like open directly return pointer into file table rather than going through the indirection of user fd table? Is it there for security or for some other reason, or have i understood this wrong?