Section 1
Preview this deck
5. T/F: Spark is modified version of Hadoop
Front
| 5 | 0 | ||
| 4 | 0 | ||
| 3 | 0 | ||
| 2 | 0 | ||
| 1 | 0 |
Active users
1
All-time users
1
Favorites
0
Last updated
1 year ago
Date created
Mar 1, 2020
Cards (246)
Section 1
(50 cards)
5. T/F: Spark is modified version of Hadoop
False!
Which command to submit your spark app:
/spark-home/bin/spark-submit .......options... (5)
6. T/F: Hadoop is a dependency for Spark
False
True or false: you need Hadoop to install spark.
False!
Scala: To load data from text file to RDD. assume SparkContext is available as sc, SqlContext is available as SqlContext
val sfpd = sc.textFile("path_to_file/file.txt" (4)
What was spark's original purpose?
Something to do with mesos
Advantage of spark over Hadoop and storm
Tons of built in tools etc already available in spark
Spark has about __ built in operators for big data analy
80
Matei Zaharia also created
Mesos (co) Fair scheduler (Hadoop )
You can submit your spark app on 4 modes
Local, Mesos, standalone, YARN (5)
Used ______________ to create the RDDs
SparkContext
Python: create data to hold numbers 1 through 999
Data = xrange (1,999)
Scala context
sc
Worker programs can run on _____ and _____
Cluster and local threads
Main program of your spark app is called?
Driver Just like mapreduce
9. 4 types of workload (not limited to) Spark can handle:
1. Batch 2. Iterative 3. Query 4. Streaming
Client talks to server nodes via
Ssh
In Python , how to spread RDD (name data) to 8 nodes
XrangeRDD = sc.parallelize(data,8)
Python: how to change name of XrangeRDD to newRDD
XrangeRDD.setName(newRDD)
11. When and where was Spark born?
In 2009 in UC Berkeley's AMPLab by Matei Zaharia
A Spark programs first creates _______________ object
SparkContext object
Spark extends Mapreduce model to include iterative programming and _________
Stream processing
Spark was created at?
UC Berkeley
We know, Spark supports Java, Python, Scala. Does it support R?
Yup, recently added!
What is pySpark?
Python programming interface to Spark
Spark was built on top of Hadoop _____
Mapreduce
Your spark app is coordinated by _____ object
SparkContext
RDD:more partitions equals
More parallelization
10. Why is the management overload for Spark is low?
Because it can do many types of workloads, you don't mean various tools for managing each.
RDDs are distributed across _______________
workers
Scala command to get Scala ver
sc.version
to build a standalone app, to create a spark context, what to include in the main method (2 statements)
val conf = new SparkConf() .setAppName("MyAPP") val sc = ................... FIRST: SparkConf OBJECT THEN: SparkContext (3)
3. RDD Question: Where does RDD reside? E. On HDFS F. In the memory of the worker nodes G. In the memory of master node H. On local disk
B
Where was Mesos invented?
Also, UC Berkeley
Sparksql: part of spark core distribution?
Yes
4. Key difference between hadoop and spark in terms of reason for speed?
Spark runs on memory
What is lineage for RDD ?
Each RDD remembers how it came about (transformations)
What is RDD is layman's terms?
This is the holy grail of what an RDD is. RDDs are a 'immutable resilient distributed collection of records' which can be stored in the volatile memory or in a persistent storage (HDFS, HBase etc) and can be converted into another RDD through some of the transformations. Source: http://www.thecloudavenue.com/2014/01/resilient-distributed-datasets-rdd.html
Where was Spark invented?
UC Berkeley
Which variable helps transfer of large data sets very efficiently
broadcast
3 languages at lest that can be used to code on spark
Java Python Scala
12. Spark provides built-in APIs in ______________ (3) languages
Java, Scala, and Python
Created by
Matei Zaharia
Spark programs are two programs:
Driver and Workers
Action on RDD returns what?
a value to the driver code (mapr 1)
RDDs are immutable. True/False
True (2)
7. T/F: Spark has its own cluster management. True/False
True
How do you start pyspark on windows?
just a command, as long as you have path added
What is the name of the python interface to Spark?
pyspark
8. The main feature of Spark is _______________
Ans: its in-memory cluster computing
Section 2
(50 cards)
once you get the scala prompt after kicking off spark-shell, 2 commands you can run for sanity check
sc.version sc.appName (case sensitive)
4 components of spark that sits on top of core spark
1.sparksql 2.Mlib 3. GraphX 4. Spark Streaming
how to start up pyspark? (windows)
pyspark (command)
will :history remember commands from the previous sessions?
YES!
every sparkcontext launches a webUI on port?
4040
Which has richer set of libraries: Python or Scala
Python
When code is sent to executors, what is sent?
JAR or Python code passed to sparkContext
calling cache() does what?
saves the RDD in cache
where will you find start-master script? (even on windows)
sbin (peer to bin where spark-shell and pyspark are)
in customer mode, where does the driver start?
same process as the customer app
It is efficient and faster to use _____ serialization
Kryo
How much faster is spark (memory) than hadoop
100 times
To run an application on the Spark cluster, simply pass _________________________________
To run an application on the Spark cluster, simply pass the spark://IP:PORT URL of the master as to the SparkContext constructor.
____________ variable can use to cache read-only values on each node's RAM
broadcast
Once connected to cluster, spark acquires ____ on nodes to do its work
Executors
Which has higher learning curve: Python or Scala
Scala
what does webUI on port 4040 show?
Every SparkContext launches a web UI, by default on port 4040, that displays useful information about the application
Spark was written in ______
Scala
2 modes for independent batch jobs:
1. customer mode 2. batch mode
What is REPL?
The spark-repl is referred to as the interactive spark shell and can be run from your spark installation directory. ./spark shell The spark-repl ( read evaluate print loop ) is a modified version of the interactive scala repl that can be used with spark
You can start a standalone master server by executing:
You can start a standalone master server by executing: ./sbin/start-master.sh
spark-shell (scala) : to get history
:history
Which has much greater community support: Python or Scala
Python
If you really wanted understand Spark, which pays more. Python or Scala
Scala
in scala prompt (spark-shell) way to get help
:help
Name 3 cluster managers sparkContext
Yarn Mesos Spark's standalone cluster
RDD stands for:
Resilient Distributed Data Set
example of 4 things that port 4040 will show
A list of scheduler stages and tasks A summary of RDD sizes and memory usage Environmental information. Information about the running executors
RDDs have transformations, which return ?
transformations return pointers to new RDDs.
spark does ________ analysis or large-scale data
complex
command to start up spark-shell (windows)
spark-shell
once accumulator has been created with the initial value, new tasks can be added to the variable using _____ command
add
most basic abstraction in spark:
RDD
how to get out of pyspark?
quit()
Spark acquires executors, copied cope, what's next?
Sends tasks to executors
spark-submit is a script. T/F
True
how to search history from scala prompt?
:h? (literally) e.g. :h? foo
REPL stands for?
read evaluate print loop
in batch mode, where does the driver start?
in one of the worker forms
spark-shell (scala) command to read a file from external source
:load _PATH_
command to fire up spark master on widnows
bin\spark-class.cmd org.apache.spark.deploy.master.Master and this ran successfully and I got a web page at http://localhost:8080 this says Spark Master at spark://192.168.1.4:7077
Accumulator variables are added through _______________ operation
associative
Mlib is __ times faster than disk-based mahout
9
default URL for client GUI
localhost:8080
7 options for the master URL
local local[K] local[*] spark://host:port mesos://host:port yarn-client yarn-cluster
in scala shell, how to read a file from the spark source directory?
val textFile = sc.textFile("README.md")
How much faster is spark (disk) than hadoop on disk
10
RDDs have actions, which return ?
RDDs have actions, which return values
Components of spark that has machine learning
Mlib
if multiple SparkContexts are running on the same host, they will bind to which ports?
f multiple SparkContexts are running on the same host, they will bind to successive ports beginning with 4040 (4041, 4042, etc).
Section 3
(50 cards)
accumulators are great for for running in ________________
parallel
T/F: unlike mapreduce, excecutors run on the worknode even when spark application is not running a task
True
through which entity driver code communicates to Cluster Manager
SparkContext
Use _______________ to create RDDs
SparkContext
what you set "master" to point to mesos cluster ?
mesos://host:port
if you specify local as master, what does that mean?
1 thread on local box, no parallelism
Lamba in programming is _________________ functions
anonymous
what is the benefit of executors running when task is not running
sharing information from memory
what is the default port for standalone spark cluster?
7077
RDD transformations are lazy. What does that mean?
they are not computed immediately
Name 2 examples of actions
collect count
Which parameter in SparkContext dictates which cluster and how many threads to use?
master
3 most common transformations on scala collections
map filter flatmap
default port for mesos cluster
5050
To install Spark Standalone mode, you simply place a compiled version of Spark on ________________
To install Spark Standalone mode, you simply place a compiled version of Spark on each node on the cluster.
A lambda is a type of function, defined _____________.
A lambda is a type of function, defined inline.
scala REPL = spark ______
scala REPL IS the spark shell
what percentage of spark programs are written in scala?
95% You tube video posted by new century trainer 10/2015 he mentions support for R added, so video cant be that old.
where do you keep a list of hostnames that run worker nodes
conf/slaves To launch a Spark standalone cluster with the launch scripts, you should create a file called conf/slaves in your Spark directory, which must contain the hostnames of all the machines where you intend to start Spark workers, one per line
where is spark-submit script?
bin
Databricks's product?
Notebooks
sbt is used to build spark applications in spark ______
shell
what are "code shorting" advantages of lambda functions?
except that you don't need to bother with: Declaring the function somewhere else (and having to look for it when you revisit the code later). Naming something that you're only using once. Once you're used to function values, having to do without them seems as silly as being required to name every expression, such as: NORMAL WAY: int tempVar = 2 * a + b ... println(tempVar) LAMBDA WAY: println(2 * a + b) (Stackoverflow)
Lambda comes from ____________ calculus
Lambda
pyspark: how to display the content of collection a
a (just a enter)
when spark runs on hadoop, data node is typically same as __________________
worker node
Cluster Manager and _______________ are typically the same device
Spark Master
Collect action causes ________, filter, and map transformations to be executed
parallelize
spark uses ______________ to parallelize and create pipelines for the sub-processes needed to execute a task.
DAG
if the client does, what happens to currently running apps and new apps that have not been submitted (assume no zookeeper)
currently running, no impact new: no longer possible
Lambda allows you to write quick throw away functions without _____________
allows you to write quick throw away functions without naming them
when add to accumulator's initial vale using the add command, who has access to it?
only the driver program
An anonymous function allows you to create_______ from functions
An anonymous function allows you to create functions from functions
accumulators are great for _________ and ___________ (examples)
sums and counters
Simply put, a lambda is a function without a __________
Simply put, a lambda is a function without a name
when is "transformation" is actually done?
when an action is performed on it
what you set "master" to point to standalone spark cluster ?
spark://host:port
3 ways to construct a RDD
1. transform another RDD 2. from files (HDFS or otherwise) 3. parallelize python collections (lists)
example of defining a collection (list) in python
>>> a = [1 2 3 4 5] (>>> is the pyspark prompt)
3 components of worker node
executor cache task
where are the start/stop scripts stored?
sbin
iPython and programs must use ________________ to create a new sparkContext
Constructor
Which company employs most of Spark committers?
Databricks
True/False: pySpark shell and Databricks cloud automatically create sc variable
True!
sbt is built from ____
scala
which has more data science tools? Python or Scala
Python!
True/False: RDD enables operations on collection of elements in parallel.
True
if you have 2 cores in your local box, what should you set yous "master" to?
local[2]
Name 2 examples of transformations
map filter
Scala example of a lambda function:
args.foreach(arg => println(arg)) where the argument to the foreach method is an expression for an anonymous function. (Stackoverflow)
Section 4
(50 cards)
Lambda function that returns arg - 1
Define minusone(foo): return foo -1
if your scala file is test.scala and object inside is HelloWorld, which files will be created when you compile it?
HelloWorld.class HelloWorld$.class
once driver gets slots from the worker nodes, how much communication happens between driver and worker AND between driver and master
NONE
scala is lot like java, so if you know java, learning curve for scala would be less. T/F
True
define a function fsq that will return square of the argument passed to it
def fsq(x: Double) = x*x (just like on the REPL; D has to capitalize)
which method to call to get number of partitions for a RDD?
Rdd1.getnumpartitions
Which language uses lambdas heavily?
LISP!
Spark SQL provides new abstraction called _____________
schemaRDD
If you see: val sc = new SparkContext(conf) what does mean for conf , in terms of dependency
conf has to be defined with certain parameters which will turn be used in creating SparkContext sc
how to print the string hello?
println("hello") quotes needed. Otherwise it tries to find variable hello
Scala lacks the same amount of Data Science libraries and ______ as Python
tools
Close cousin of lambdas in C?
function pointer
Transformation actually does nothing (like a file pointer)
nothing really happens until an action is called
You can cache to many types of locations
Memory, disk and more
python API vs scala API (performance)
neglizible, because its almost all scala under the hood
How much faster is MLib compared to Mahout (disk-based) before it got spark interface?
9 times
if your file is named test.scala , but object inside is called HelloWorld, will that work?
Yes!
If you do 5 transformations on 5gb RDD and do 1 simple action, data is finally pulled and network traffic ensues. Then you do one more small action, then what happens?
data is pulled over again!!
how to add 5 and 6 and save the result into variable x
val x = 5 + 6
Scala is both object oriented and functional. T/F
True
How do you compile helloworld.scala?
scalac helloworld.scala
you do: (scala REPL) 2+2 is the answer saved anywhere?
yes , it is stored in variable res0
if function fsq is define, how do you pass variable a to it and save the result in variable b?
val b = fsq(a)
Once a user application is bundled, it can be launched using the _____________ script
Once a user application is bundled, it can be launched using the bin/spark-submit script
if you write a class in scala, what extension should you give it?
.scala (e.g. HelloWorld.scala)
If you install scala and cygwin on windows , can you run scala from cygwin shell?
yes just add scala's bin to windows PATH
One difference between lambdas and anonymous inner class in Java
The first one is a formal difference, lambdas usually define closures - which means they can refer to variables visible at the time they were defined.
Close cousin of lambdas in Java?
anonymous inner class http://martinfowler.com/bliki/Lambda.html
Can we easily call R directly from python?
People have ported most of the core parts of R to Python and there's easy ways to call R directly from Python (source: quora)
Rdd1.todebugstring(RDD)
public String toDebugString() A description of this RDD and its recursive dependencies for debugging.
what was the appeal of Spark besides speed?
interactive algorithms and queries were not served enough by the batch frameworks (i.e. hadoop oriented programmatic tools)
Map
function played on each element of the RDD
4 Items on the TOP layer of Spark Stack
1. Spark SQL 2. Spark Streaming 3. Mlib 4. GraphX
Is the name "worker" misleading?
yes. because worker manages all the resources (slots)
3 Items on the bottom layer of spark stack
1. Spark Stand Alone 2. Mesos 3. YARN
Example of defining parameters using conf
val conf = new SparkConf() .setMaster("local[2]") .setAppName("CountingSheep") (apache apark web site)
How do you execute a complied class from command prompt?
scala HelloWorld,class OR scala HelloWorld both works!
If you do 5 transformations on 5gb RDD and do 1 simple action, when is data pulled
only when the first action is called
Executors
JVM that actually runs tasks (on worker nodes)
3 (examples) of what scala lacks
1. visualization 2. local tools 3. good local transformations (source: quora)
in scala: how to read a file into RDD
val foofile = sc.textFile(foofile.txt)
2 ways to exit scala shell?
:q and sys.exit
Way to make sure data is not thrown away
Caching
3 layers of Spark Stack:
TOP: Spark SQL +3 others MIDDLE: Spark CORE BOTTOM: Mesos and 2 others
what does spark-submit script take care of:
This script takes care of: setting up the classpath with Spark and its dependencies
Filter
take out only certain elements on RDD and make a new one
Does Ruby have lambdas?
Yes
how to print variable res0?
println(res0)
T/F: C/C++/Java/C# also have lambdas
NO! http://martinfowler.com/bliki/Lambda.html
example of spark-submit script
./bin/spark-submit \ --class <main-class> --master <master-url> \ --deploy-mode <deploy-mode> \ --conf <key>=<value> \ ... # other options <application-jar> \ [application-arguments]
Section 5
(46 cards)