Difference between pickle and opening a file?

Let me see if I can understand where the point of confusion is and give a useful explanation.

open is how you get a file object, which is the interface between your Python program and an actual file on disk. with is a tool used for ensuring that the file object is closed at the appropriate time.

The file object allows you to read and/or write the file, depending on how it was opened. The built-in way to do this is with the object's own functionality. This lets you write whatever data you want, at the expense that you are responsible for figuring out what that data should be; alternately, it lets you read the data and gives you both the power and responsibility that comes from interpreting that data.

The pickle library builds on top of that functionality, to use the file's contents to represent native Python objects. It does the interpretation (parsing) and data-figuring-out (formatting) work for you, accomplishing something that would be difficult by hand. The trade-off is that it works in a specific way, and is fit for only a specific purpose - you won't, for example, be producing or interpreting plain text files, or images, or JSON data, etc. this way any time soon (which you could by writing the data yourself, or by using a different, special-purpose library - except of course for plain text, where there's no point in doing anything beyond using the built-in functionality).

The difference is in what you put in the file, and who is responsible for the underlying file's format / serialization.

With the builtin open, you receive a raw file handle. You can write whatever you want to it. It doesn't have to be structured, it doesn't have to be consistent, hell it doesn't even have to make sense to an outside observer. You are allowed to write whatever you want to a file.

With a pickle, the underlying module is responsible for what is written. It serializes python objects (as much as possible, there are examples of classes that cannot be pickled) in a consistent, reproducible format that can the be re-loaded. IE - you can save the state of actual python objects in a static file, and then reload them and end up with identical objects the next the the interpreter runs. This has advantages when dealing with stateful programs.

Bonus: The shelve module serves as a user-friendly frontend to pickle that behaves like a dictionary. When you close the shelf, the contents are serialized to disk. When you re-open the shelf, the objects are deserialized from the file and accessible the same way a dictionary would be.

pickle allows you to conveniently write python objects to it, and load those objects. How would you use open() to write a dictionary into a file, and be able to load it into your python file with one simple line?

For open(), it will be like:

dct = {'a': 1,
       'b': 2,
       'c': 3,
       'd': 4,
       'e': 5}

with open('file.txt','w') as f:
    f.write('\n'.join([f"{k}, {v}" for k, v in dct.items()]))

with open('file.txt','r') as f:
    dct = {k: int(v) for k, v in [s.split(', ') for s in f.read().splitlines()]}

While with pickle:

import pickle

dct = {'a': 1,
       'b': 2,
       'c': 3,
       'd': 4,
       'e': 5}

with open('file.txt','wb') as f:
    pickle.dump(dct, f)
    
with open('file.txt','rb') as f:
    dct = pickle.load(f)
    

Note the conversion did in the first method, where we need to convert the string into an integer. With pickle, you won't have to worry about that.

pickle is a library function written using with open() to convert the objects into a byte stream to save developer efforts.

Further info: https://docs.python.org/3/library/pickle.html and here is the code: https://github.com/python/cpython/blob/main/Lib/pickle.py