Can I process 100 GB of data using Apache Spark on my local machine?

user_8444792 0
Viewed 7210

I have around 100GB data of users and want to process it using Apache Spark on my laptop.I have installed Hadoop and Spark and for the test I uploaded a file of around 9 GB to HDFS and accessed & queried it using pyspak.

The test file has total 113959238 records/rows, when I queried the data for a particular user i.e

select * from table where userid=????

it took around 6 minutes to retrieve the records of that user and if I run on the entire file then it will take a lot of time. The analysis that I to make on that data is to extract the records a users, run some operations on it and then process the data of second user and so on for all the users in file. The data of the user queried, will not be much so it can be loaded in memory and operations can be preformed faster. But querying the record of a user from that big file takes time and will slow the process.

It is said that Spark is lighting fast so surely I will be missing something which is why it is taking that time. One thing that I noted while performing queries was Spark was not utilizing full RAM but almost 100% of CPU.

My machine specs are:

enter image description here

I also queried the data directly of the text file using Spark instead of HDFS file but there wasn't much difference in time. The python code that I wrote is

 from pyspark import SparkContext, SparkConf
 from pyspark.sql import SparkSession, HiveContext,SQLContext
 import time
 conf=SparkConf()
 conf.set("spark.executor.memory", "8g")
 conf.set("spark.driver.memory", "8g")
 sparkSession = SparkSession.builder.appName("example-pyspark-read-and-write").getOrCreate()
 sc=sparkSession.sparkContext.getOrCreate(conf)
 sqlContext=SQLContext(sc)
 #df_load = sparkSession.read.format("csv").option("header","true").load("hdfs://0.0.0.0:19000/test.txt")
 df_load = sparkSession.read.format("csv").option("header","true").load("C:/Data/test_file/test.txt")
 table=df_load.registerTempTable('test')
 sp_tstart=time.time()
 df=sqlContext.sql("select * from test where user_id='12345'")
 db=df.rdd.collect()
 sp_tend=time.time()
 t_time=sp_tend-sp_tstart
 df.show()
 print(t_time/60)

Given my machine specs, is Spark taking normal time or Do I need to configure something? Do I need to upgrade the specs or is it enough for this data?

pyspark apache-spark hadoop python
4 Answers

One of the things to understand with Spark, Hadoop and other Big Data providers is that they aren't aiming to get the maximum possible throughput from an individual machine. They're aiming to let you split the processing efficiently across multiple machines. They sacrifice a certain amount of individual machine throughput to provide horizontal scalability.

While you can run Spark on just a single machine, the main reasons to do so are to learn Spark or to write test code to then deploy to run against a cluster with more data.

As others have noted, if you just want to process data on a single machine, then there are libraries which are going to be more efficient in that scenario. 100GB is not a huge amount to process on a single machine.

From the sound of things you'd be better off importing that data into a database and adding suitable indexing. One other thing to understand is that a lot of the benefit of Big Data systems is supporting analysing and processing most or all of the data. Traditional database systems like Postgres or SQL Server can work well handling Terabytes of data when you're mainly querying for small amounts of data using indices.

user_2580630

If your objective is to analyze 100GB of data using python and there is no requirement for spark, you can also take a look into dask. https://dask.org/ It should be easier to setup and use with python.

For example dask dataframe: https://docs.dask.org/en/latest/dataframe.html

>>> import dask.dataframe as dd
>>> df = dd.read_csv('2014-*.csv')
>>> df.head()
   x  y
0  1  a
1  2  b
2  3  c
3  4  a
4  5  b
5  6  c

>>> df2 = df[df.y == 'a'].x + 1
user_25774160

You don't need Hadoop to process the file locally.

The advantages of Hadoop only apply when you use more than one machine as your file will be chunked and distributed to many processes at once.

Similarly, 100GB of plaintext isn't really "big data"; it still fits on a single machine and if stored in a better format like ORC or Parquet, would be significantly less in size

Also, to get faster times, don't use collect()

If you simply want to lookup data by ID, use a key value database like Redis or Accumulo, not Hadoop/Spark

user_23086830

The type of job you have described his a highly CPU intensive process, which unfortunately is only going to be sped up significantly by running many parallel queries on partitions of the data set. Compound the problem with not having enough system memory to hold the entire dataset and now you are also limited by significant read/writes on the hard drive.

This is the type of task where Spark really shines. The reason you aren't experiencing any improvement in performance is because with a single system you're missing the benefit of Spark entirely, which is the ability to split the data set into many partitions and distribute it across many machines who can work on many different users IDs at the same time.

Each worker node in your cluster will have a smaller data set to look at, which means on each node the entire data set it is looking at can easily be stored in memory. Each find and replace function (one per user ID) can be sent to a single CPU core, which means if you have 5 workers with 16 cores each, you can process 80 IDs at a time, from memory, on an optimized partition size.

Google CloudProc and Azure Databricks are super platforms for doing this. Simply choose the number of workers you need, and the CPU/Memory of each node, and fire up the cluster. Connect to your data and kick off your PySpark code. It can process this data so quickly that even though you are paying by the minute for the cluster, it will end up being very cheap ($10-$20 perhaps).

user_47183500
Related