Section 1
Preview this deck
Which of the following is not a component of Spark Ecosystem?
Front
| 5 | 0 | ||
| 4 | 0 | ||
| 3 | 0 | ||
| 2 | 0 | ||
| 1 | 0 |
Active users
0
All-time users
0
Favorites
0
Last updated
6 years ago
Date created
Mar 1, 2020
Cards (149)
Section 1
(50 cards)
Which of the following is not a component of Spark Ecosystem?
Sqoop
What are the benefits of lazy evaluation?
Increase the manageability of the program. Saves computation overhead and increases the speed of the system. Reduces the time and space complexity. provides the optimization by reducing the number of queries.
What are the parameters defined to specify window operation
Window length, sliding interval
What are Paired RDD?
Are the RDD-containing key-value pair. A key.value pair KYP contains two linked data item
How is fault tolerance achieved in Apache Spark?
If any partition of an RDD is lost due to a worker node failure, then that partition can be re-computed from the original fault-tolerant dataset using the lineage of operations.
What are the cases where Apache Spark surpasses Hadoop
The data processing speed increases in the Apache Spark because apache spark run in memory computation The performance of the system increase by 10x - 1000x times Apache spark uses various languages for distributed application development We can use Streaming, SQL, graph and machine leaning (mLib)
cache()
we can cache RDD using the cache() or persist() method. In cache() method all the RDD are in-memory. The dissimilarity between cache() and persist() is the default storage level. For cache() it is MEMORY_ONLY. While in persist() there are various storage levels like: MEMORY_ONLY, MEMORY_AND_DISK, MEMORY_ONLY_SER MEMORY_AND_DISK_SER DISK_ONLY
Persistence
is an optimization technique which saves the result of RDD evaluation. Using this we save the intermediate result for further use. It reduces the computation overhead. We can make persisted RDD through cache() and persist() methods.
What´s Apache Spark?
It´s a framework´s open source, wid range data procesing engine It allow data worker to execute streaming, maching lerning or SQLworklads Spark dont have storage system
Actions
returns final result of RDD computations. It triggers execution using lineage graph to load the data into original RDD. After application of all the intermediate transformation, it gives the final result to driver program or writes it out to file system. Upon applying Actions on an RDD non-RDD values gets generate.
What is lazy evaluation in Spark?
Significa que SPARK evalua las transformaciones de manera perezosa ya que no ejecuta las transformaciones a los RDDs hasta que se ejecuta una acción en el RDD resultante
DataSet
is an extension of DataFrame API wich provides type safe, object-oiented programming interface
Which of the following is not the feature of Spark?
t is cost efficient
Which of the following algorithm is not present in MLlib?
Tanimoto distance
The mains abstraction of Spark is its RDDs
Also, we can cache RDD using the cache() or persist() method
Spark Limitations
Does ont have it´s file management system In-memory capability can become a bottleneck Memory consumption is very high, and the issues for the same are not handled in a user-friendly manner Mlib lack in some available algorihms, fore example, Tanimoto distance
Exist two types of data that should recover in the event of failure:
Data received and replicated Data received but buffered for replication
Narrow transformation
Is the esult of map, filter and such that the data is from a single partition only
Can we add or setup new string computation after SparkContext starts
no
The default storage level of cache() is?
MEMORY_ONLY
Examples of transformations
Map, Filter, ReduceByKey, etc
Wide transformation
are the result of groupByKey and reduceByKey
Dataset was introduced in which Spark release?
Spark 1.6
En qué versión de spark se implementó SQL
version 1.1
The basic abstraction of Spark Streaming is
Dstream
How is data represented in Spark
RDD DataFrame DataSet.
What is Spark Core
It provides parallel and distributed processing for large data sets. Spark core provides speed through in-memory computation RDD is the basic data structure of Spark Core. RDDs are immutable, a partitioned collection of record that can operate in parallel.
How many types of Transformation are there?
Narrow Transformation Wide transformations
Transformations
are lazy operations on an RDD that create one or many new RDDs
RDD
Resilient Distribuited DataSets RDD is the fundamental data structure of spark It is also a read only oartition collection of records The RDD can only be created through deterministic operation: Data in stable storage Parallelizing already existing collecition in driver program Other RDDs
Which Cluster Manager do Spark Support?
Standalone Cluster Manager, MESOS YARN
Which is the abstraction of Apache Spark?
shared variable and RDD
Apache Spark was made open-source in which year?
2010
Which of the following is not output operation on DStream
ReduceByKeyAndWindow
diference between mapReduce and Spark
Map reduce tabaja en disco y spark RDD trabaha en memoria
which languages support Apache spark
Spark provides API in various languages Pyhto, R, Scala and Java
Explain the operations of Apache SPark RDD
Apache Spark RDD supports two typesof operations: Transformations and actions
What are the features of spark
The processing speed of Apache Spark is very high Apache spark is dynamyx in nature We can reuse code for join stream against historical data recoviery is possible in RDD Spark support many languajes It can run independently and also on other cluster manager like Hadoop YARN
In how many ways RDDs can be created?
Parallelized collection External Datasets (Referencing a dataset) Creating RDD from existing RDD
Where we can run spark
We can run spark by itself or on varius existing cluster manager For example: Standalone Deploy Mode, Apache mesos, Hadoop, Yarn
DataFrame
The data organizes into named columns, like a table in a relational database It is also an inmutable distributed collecion of data Allows developers to impose a structure onto a distributed colleciont of data
Which is not a component on the top of Spark Core?
Spark RDD
In Spark Streaming the data can be from what all sources?
Kafka, Flume, Kinesis
Spark is developed in which language
Scala
What are the components of spark ecosystem
Spark Core Spark Streaming Spark SQL Spark MLlib Spark GraphX SparkR
In addition to stream processing jobs, what all functionality do Spark provides?
Machine learning Graph processing Batch-processing
Dstream internally is
Continuous Stream of RDD
Hadoop and spark
Cost Efficient during replication, a large number of servers, huge amount of storage, and the large data center is required. Apache spark is a cost effective solution for big data environment Performance: The basic idea behind Spark was to improve the performance of data processing. And Spark did this to 10x-100x times. And all the credit of faster processing in Spark goes to in-memory processing of data. In Hadoop, the data processing takes place in disc while in Spark the data processing takes place in memory. It moves to the disc only when needed Ease of development The core in Spark is the distributed execution engine Hadoop also supports some of these workloads but Spark eases the development by combining all into the same application.
Is Spark included in every major distribution of Hadoop?
Yes
Apache Spark has API's in
Java, python, scala
Section 2
(50 cards)
Which of the following is the entry point of Spark Application -
Spark Context
Which of the following is true about DataFrame?
DataFrames provide a more user-friendly API than RDDs.
For Multiclass classification problem which algorithm is not the solution?
Decision Trees
You can connect R program to a Spark cluster from -
RStudio R Shell Rscript
In how many ways RDD can be created?
Three
Fault Tolerance in RDD is achieved using
DAG(Directed Acyclic Graph)
In Structured Streaming from file-based sources do we need to specify the schema?
Yes
Which of the following is true about wide transformation -
The data required to compute resides on multiple partitions.
For Regression problem which algorithm is not the solution?
logistic Regression
How many tasks does Spark run on each partition?
one
Which of the following is the entry point of Spark SQL?
b. SparkSession
Which of the following is true for RDD?
We can operate Spark RDDs in parallel with a low-level API
Which of the following is true for Spark core?
It is the kernel of Spark
Can you combine the libraries of Apache Spark into the same Application, for example, MLlib, GraphX, SQL and DataFrames etc.
Yes
Which of the following is false for Apache Spark?
Spark is an open source framework which is written in Java
When we want to work with the actual dataset, at that point we use Transformation?
Flase
Which of the following is true for RDD?
RDD in Apache Spark is an immutable collection of objects
Which of the following is true for Spark R?
It allows data scientists to analyze large datasets and interactively run jobs
Which of the following is true for Spark MLlib?
It is the scalable machine learning library which delivers efficiencies
Which of the following is not true for Hadoop and Spark?
Both have their own file system
The shortcomings of Hadoop MapReduce was overcome by Spark RDD by
a. Lazy-evaluation b. DAG c. In-memory processing
Which of the following is not a function of Spark Context in Apache Spark?
Entry point to Spark SQL
Can we edit the data of RDD, for example, the case conversion?
No
What is action in Spark RDD?
The ways to send result from executors to the driver
Which of the following is true about narrow transformation -
The data required to compute resides on the single partition.
How many Spark Context can be active per JVM?
Only one
Which of the following provide the Spark Core's fast scheduling capability to perform streaming analytics.
Spark Streaming
Which of the following is not an action?
map
Caching is optimizing the technique
true
Which of the following is true for Spark Shell?
It helps Spark applications to easily run on the command line of the system It runs/tests application code interactively It allows reading from many types of data sources
Which of the following is true for Spark SQL?
It enables users to run SQL / HQL queries on the top of Spark.
How much faster can Apache Spark potentially run batch-processing programs when processed in memory than MapReduce can?
100 times faster
Which of the following is the reason for Spark being Speedy than MapReduce?
DAG execution engine and in-memory computation
What does Spark Engine do?
a. Scheduling b. Distributing data across a cluster c. Monitoring data across a cluster
For binary classification problem which algorithm is not the solution?
Naive Bayes
Which of the following is a tool of Machine Learning Library?
Persistence Utilities like linear algebra, statistics Pipelines
In which mode you can better launch the task?
Both Standalone and coarse-grained mode.
What is a transformation in Spark RDD?
Takes RDD as input and produces one or more RDD as output.
The write operation on RDD is
Coarse-grained
RDD are fault-tolerant and immutable True
True
Which of the following is not a transformation?
reduce
When does Apache Spark evaluate RDD?
Upon action
Can you recover Accumulator and Broadcast variable from checkpointing in Spark Streaming?
YES
What are the features of Spark RDD?
In-memory computation Lazy evaluations Fault Tolerance
Does Spark R make use of MLlib in any aspect?
Yes
The read operation on RDD is
Either fine-grained or coarse-grained
Is MLlib deprecated?
No
Is it possible to mitigate stragglers in RDD?
Yes
In which of the following cases do we keep the data in-memory?
a. Iterative algorithms. b. Interactive data mining tools.
SparkContext guides how to access the Spark cluster.
TRUE
Section 3
(49 cards)