Why isn't the __new__() method of my class returning an instance of the class?

You can use collections.UserList. It is made for subclassing lists. This will allow you to assign to self.data in __init__() using your same code and avoid the issues with __new__():

from collections import UserList
import copy

class ArrayList(UserList):
    def __init__(self, *dim, initial = []):
        self.data = None
        for d in range(len(dim)-1, -1, -1):
            if self.data is None:
                self.data = [initial for _ in range(dim[d])]
            else:
                self.data = [copy.deepcopy(self.data) for _ in range(dim[d])]

    def __str__(self):
        return '\n'.join(str(s) for s in self)

var1 = ArrayList(4,3,2,initial=0)
print(var1)
print(type(var1))   # <class 'list'>
print(isinstance(var1, ArrayList))

Prints:

[[0, 0], [0, 0], [0, 0]]
[[0, 0], [0, 0], [0, 0]]
[[0, 0], [0, 0], [0, 0]]
[[0, 0], [0, 0], [0, 0]]
<class '__main__.ArrayList'>
True

Note that the docs suggest:

Subclasses of UserList are expected to offer a constructor which can be called with either no arguments or one argument...

...If a derived class does not wish to comply with this requirement, all of the special methods supported by this class will need to be overridden

Which you're not doing here (although it may not matter for your purposes).

Also, it will probably be easier to just use Numpy if you haven't already considered that:

import numpy as np
np.zeros([4, 3, 2])

I think you are not using properly __new__: this magic method's job should be to handle the creation of a new instance of a certain class (that's why it should not return an object of another type), while __init__ should handle its initialization.

The proper way to do this would be to completely ignore the __new__ method (and I encourage you to continue doing so until you explicitly find yourself in a situation in which you cannot solve your problem with __init__), and instead use the latter.

In you're example, this would simply mean to move the code from __new__ to __init__, and working on self instead of on a new object your create.

But you might have been confused on how to use them because there are few other errors which might have led to difficulties:

1. Let's start with simple stuff: within your comment block, you call var1 is ArrayList. The is operator does not check the type of an object. You should instead consider it as a stronger == operator (the difference being while the latter allows custom behavior by overloading, the former simply checks if both operands are the same object – same id).
2. It is unclear to me if it is useful for ArrayList to inherit from list: maybe that's what you would do in a more advanced/complex program, but really it doesn't bring you anything (even if what I am going to say is very anti-pythonic, but it's still true): most list-like types don't actually inherit from it, they simply internally store lists (or other more complex collections) and show an interface similar to lists. For instance, check the python doc for __getitem__ and __setitem__, which are most likely what you will need.
3. Finally, and this is a more subtle error, you'll get out-of-nowhere errors because of a very stupid (IMO) python “feature”, which is: the initial variable within your constructor is consider static by python (even if static variables don't exist, the trick is that it is actually created only once, and you don't make a deep copy of it when creating, so whenever you will modify a single item, boom, every other item will take that value).

Just to be practical, I'll give you a few examples of how you could achieve what you want.

First, the “best” way (in the sense, the way you should use when becoming more experienced):

import numpy
var1 = numpy.ndarray(4, 3, 2)
var1[1,0,1] = 5
# equivalent to
var1[1][0][1] = 5
# to initialize the whole array to some value
var1[:,:,:] = 0

The DIY way (only for educational purposes):

class Array:
    def __init__(self, *dims, default=0):
        if len(dims) == 0:
            raise TypeError("Cannot have 0 dimentional arrays")
        elif len(dims) == 1:
            self._array = [default for i in range(dims[0])]
        else:
            self._array = [Array(*dims[1:], default=default) for i in range(dims[0])]
    def __getitem__(self, key):
        return self._array[key]
    def __setitem__(self, key, value):
        self._array[key] = value
    def __repr__(self):
        return '[' + ', '.join(repr(e) for e in self._array) + ']'

You could of course not want to create multiple arrays, but there are many ways to improve this. Just toy with it to understand it well.

EDIT:

Let's unshadow some points:

I/ Static variables: ie 90% of the time, a very subtle bug that goes against the common sense, quite hard to debug

Suppose I want to make a custom list class, but, for the sake of the simplicity, let's say I can only get items and append, and the actual list passed (optionally) when initialized:

class MyList:
  def __init__(self, mylist=[]):
    self.mylist = mylist
  def __getitem__(self, key):
    return self.mylist[key]
  def append(self, value):
    self.mylist.append(value)

Well, this code is wrong! Here is what might occur:

a = MyList()
b = MyList()
a.append(3)
a[0] == 3 # true
b[0] == 3 # true?? wtf?

Actually, a.mylist is exactly the same object as b.mylist! I don't mean it's equal, I mean they really share the same memory section, changing one changes the other. This is because there is only one list that is ever created, and that is when the MyList class was defined, well before a was even created!

The correct code would be something like that:

class MyList:
  def __init__(self, mylist=None):
    if mylist == None: self.mylist = []
    else: self.mylist = mylist
  # the rest of the code

II/ Overloading the __new__ magic method without actually creating the object

It is very common for beginners to not understand the difference between __new__ and __init__, and it is totally normal: this is because python does a lot behind the scenes, and for python there is a huge difference between those two, but for us, they are both called with the same arguments just before an object is created, so it's confusing. Let's dive in!

__new__ really builds a new object from nothing, meaning you have to allocate the memory, and return the newly created object. You don't want to do that, unless you know what you are doing and have a very fine motive for doing so, right?

__init__, on the other hand, takes care of initializing the newly created object, which is why it is called right after __new__, and doesn't have to return anything: the new object is already created. This is where you want to add padding to your new list, or whatever you require. The other magic method has only the purpose of creating the object (meaning, getting the memory, saying python “Hey I exist”, etc.).

So, in the comment you wrote that the newly created object were None objects. This is very likely due to the fact that you overloaded __new__, but returned nothing. My advise is to simply forget about using __new__ for the moment, most of what you will do won't require it.