Cannot change global variables within Spark foreach function body

I need to filter through a very large text file(500GB) and export the results with Spark.

The problem here is that it will be too big to fit everything in memory, so I have to use a global buffered writer, which gets flushed about every 500000 records.

Intuitively, I used RDD's foreach function and passed a function to it, like (in pyspark):

# global definitions
def fn(x);
    global counter, writer
    # some logic to generate data
    if counter == 500000:
        writer.write(data)
        writer.flush_buf()
        counter = 1
    else:
        counter = counter + 1
txt_file.foreach(fn)
# write final data

However, here somehow the variable counter is not being updated.

I initially thought it might have something to do with the python-scala bridge, and rewrote some test code with scala:

var value = 10
def fn(x: Int) = {
    value = 20
}
sc.parallelzie(Seq(1, 2, 3, 4, 5), 5).foreach(fn)
println(value)

To my surprise, I still get value=10. I wonder why I'm getting this strange behavior here and what's the correct way of doing it?

Thanks!