Section 1
Preview this deck
create a contour plot with 30 contours of Z array? create the same with a filled contour?
Front
| 5 | 0 | ||
| 4 | 0 | ||
| 3 | 0 | ||
| 2 | 0 | ||
| 1 | 0 |
Active users
2
All-time users
2
Favorites
0
Last updated
1 year ago
Date created
Mar 1, 2020
Cards (240)
Section 1
(50 cards)
create a contour plot with 30 contours of Z array? create the same with a filled contour?
plt.contour(Z, 30) plt.contourf(Z, 30)
Create a list of neighbors in range 1 to 9 and two empty numpy arrays for training and test accuracies
neihbors = np.arange(1,9) training_accuracy = np.empty(len(neighbors)) test_accuracy = np.empty(len(neighbors))
What does np.meshgrid do?
replicates 1D arrays into a 2D array while keeping axes.
What command will save a png from a plot? save as pic
plt.savefig('pic.png)
What kinds of plots are available to the kind argument for a joint plot?
kind = scatter, reg, resid, kde, hex
How do you create a numpy array from a pandas dataframe (df) index? how would you create a list?
numpy array df.index.values list df.index.values.tolist() chaining .values to anything pandas related gives a numpy array
Create a combined violinplot overlayed with a spreadout strip plot: data = tips, x=day, y=tip with violinplot in light gray without the internals so just showing the KDE section.
sns.violinplot(x='day', y='tip', data=tips, inner=None, color='lightgray') sns.stripplot(x='day', y='tip', data=tips, siz=4, jitter=True) plt.label('tip ($)') plt.show()
Use knn to score X, y test sets
knn.score(X_test, y_test)
How do you add controls over axes extents - set limits to x-axis to 0 to 5 and y-axis from -10 to 10
plt.axis([0, 5, -10, 10]) Or plt.xlim([0, 5]) plt.ylim([-10, 10])
Read in image X.png and show it without axes
img = plt.imread('X.png') plt.imshow(img) plt.axis('off') plt.show()
What are the arguments to subplot() command?
subplot(nrows, ncols, nsubplot)
What does np.linspace do? What does np.linspace(-2, 2, 3) create? How about np.linspace(-1, 1, 5)?
creates a 1D array of uniformly spaced values array from -2 to 2 using 3 numbers [-2, 0, 2] [-1, -0.5, 0, 0.5, 1]
Import module from sklearn to split data into training and test sets
from sklearn.model_selection import train_test_split
Create a variable intensity from an image saved as img from all color channels
intensity = img.sum(axis=2)
Create a subplot system with 2-rows, one column. Create two stipplots using auto data set investigating hp grouped horizontally by cyl. For the second plot spread out the points and make the size of points3
#Constructing Strip Plots #Make a strip plot of 'hp' grouped by 'cyl' plt.subplot(2,1,1) sns.stripplot(x='cyl', y='hp', data=auto) # Make the strip plot again using jitter and a smaller point size plt.subplot(2,1,2) sns.stripplot(x='cyl', y='hp', data=auto, size=3, jitter=True) # Display the plot plt.show()
Create a hexbin graph of mpg(y) and hp(x) with gridsize 15 across and 12 high with a range of mpg: 8-48, hp:40-235
plt.hexbin(hp, mpg, gridsize = (15, 12), extent=(40,235,8,48))
What is machine learning?
Andreas Muller: 'The art and science of giving computers the ability to learn and make decisions from data without being explicitly programmed'
For Univariate data what plot types are appropriate?
Strip Plots, Swarm Plots, Box Plots, Violin Plots
Create a 2D histogram with mpg(y), hp(x) with bins of 20 and 20. limit hp to a range to 40-235 and mpg to 8-48
plt.hist2D(hp, mpg, bins=(20,20), range=((40, 235), (8, 48))
Fit a linear regression model on X, y from sklearn?
from sklearn import linear_model reg = linear_model.LinearRegression() reg.fit(X,y)
How do you add a color bar to a plot?
plt.colorbar()
Using matplotlib plot temperature in red and dewpoint in blue against the target variable (t), label x-axis as 'Date' and give title 'Temperature and Dew Point' using subplot
plt.subplot(2,1,1) plt.plot(t, temperature, 'r') plt.xlabel('Date') plt.title('Temperature') pltsubplot(2,1,2) plt.plot(t, dewpoint, 'b') plt.xlabel('Date') plt.title('Dew Point') ptl.tight_layout plt.show()
Create a seaborn strip plot for tip amount per day, with day on x axis and tip amount on y-axis from dataframe tips, apply argument to spread out data
sns.stripplot(x='day', y='tip', data=tip, jitter=True)
Import knn classifier from sklearn?
from sklearn.neighbors import KNeighborsClassifier
Annotate a graph with the label 'setosa' placed at location (5.0, 3.5), with text at (4.25, 4.0) and an arrow in red
plt.annotate('setosa', xy=(5.0, 3.5), xytext=(4.25, 4.0), arrowprops={'color':'red'}
Where is histogram pixel equalization used?
In astronomical and medical images to get more contrast so that features will stand out. This is because it uses the full range of intensities rathern
Using matplotlib plot temperature in red and dewpoint in blue against the target variable (t), label x-axis as 'Date' and give title 'Temperature and Dew Point'
plt.plot(t, temperature, 'r') plt.plot(t, dewpoint, 'b') plt.xlabel('Date') plt.title('Temperature and Dew Point') # at this point plot is held in memory plt.show()
How do you add a legend to a plot? what are some locations?
plt.legend(loc='upper right') Locations: 'upper center/left/right' 'center left/right' 'lower center/left/right' center best right
How are violin plots made?
A kernel density estimate (KDE) is wrapped around the boxplot to show where distribution is thicker.
How do you calculate the covariance matrix and the correlation matrix from a dataframe, df?
df.cov() df.corr()
What is the goal of supervised learning?
Automate time-consuming or expensive manual tasks or make predictions about future. Need labeled data
plot residuals from auto data frame, hp on x-axis, mpg on y-axis from auto dataframe in green
sns.residplot(x='hp', y='mpg', data=auto, color='green')
With lots of data should you use strip, swarm, box or violin plots?
boxplots or violin plots are suggested
Create a seaborn swarm plot from the tips data set with day on the x-axis and top on the y-axis, group/color by sex and print horizontally
sns.swarmplot(x='day', y='tip', data=tips, hue='sex', orient = 'h')
From the dataframe iris, return the column for 'sepal_length' where the column 'species' is equal to setosa
iris.loc[iris['species] == 'setosa', 'sepal_length']
For Bivariate or Multivariate data what plot types are appropriate?
Joint Plots, Pair Plots, Heat Maps
How do you remove the first 5 characters from a character string for every element in a pandas df column, df.col?
df.col = df.col.str.slice(start=5)
What is a histogram?
Create a seaborn joint plot and output kde, plotting bill and tip from dataframe tips.
sns.jointplot(x='bill', y='tip', data=tips kind='kde')
You are loading stocks.csv and you want the first column 'Date' to be a time series index. how is it loaded?
pd.read_csv('/dates.csv', index_col=0, parse_dates=True)
With a timeseries index create a new dataframe from the dataframe temperature for march and april of 2010
march_apr = temperature['2010-03':'2010-04']
Basically describe k-nearest neighbors
Predict label of a datapoint by looking at k closest labeled data points and then taking a majority vote
Use train_test_split to split data in ton 70/30 stratified split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=13, stratify=y)
import seaborn and plot weight by hp with a regression line from the auto dataframe
import seaborn as sns sns.lmplot(x='weight', y='hp', data=auto)
How to you set/change styles in matplotlib.pyplot? What are some styles?
plt.style.use('ggplot') classic seaborn-ticks seaborn-colorblind dark_background grayscale seaborn-bright seaborn-dark-palette seaborn-notebook seaborn-poster seaborn-paper seaborn seaborn-muted _classic_test seaborn-deep seaborn-whitegrid seaborn-pastel seaborn-dark seaborn-white ggplot seaborn-darkgrid bmh fivethirtyeight seaborn-talk
What is a moving window?
Extracts information on longer time scales and aggregates using a summary statistic: average, median, standard deviation
create a seaborn pairplot from the tips dataframe and group by sex
sns.pairplot(tips, hue='sex')
Find the corresponding value in one list to the index of the maximum value in another list. Return value in Y that is max value in X
y[x.argmax()]
Find the max value in a numpy array, x.
x.max()
What does plt.tight_layout() do?
automatically adjusts subplot parameters so that the subplots fit in to the figure area. It improves spacing
Section 2
(50 cards)
What is the first thing you do with data before trying to build a model?
Split off a test set that is never viewed or used for validation and use this to report final model accuracy. Want to ensure models ability to generalize to unseen data.
What is the motivation for cross validation?
model performance ins dependent on the way the data is split and the pecularities of the test set. It may not be representative of the models ability to generalize
Explain how a ROC curve works, what if you set p=0? p=1? what happens?
What metric is used for multi-class classification models?
Accuracy score and confusion matrix
What argument can be added to xgb.cv to stop training earl?
early_stopping_rounds = # early stopping tests the model after every boosting round against a hold-out dataset and stopping the creation of additional boosting rounds if the hold-out metric does not improve for a given number of rounds.
Explain Lasso Regression?
In dataframe df, convert all instances of '?' to np.nan
df[df == '?'] = np.nan
Explain Standardizing and Normalizing, how are they different?
Standardization: Subtract Mean and divide by Variance Normalization: Subtract minimum and divide by range (min = 0, max=1), can also
What is a better metric for a problem suffering from class imbalance?
ROC Curve built from confusion matrix
Why do we use objective functions or loss functions
Quantifies how far off a prediction is from the actual result Measures difference between estimated and true values for some collection of data Goal: Find the model that yields the minimum value of the loss function
Create an xgboost DMatrix from X, y
import xgboost as xgb xgb.DMatrix(data=X, label=y)
In using lasso to select coefficients, fit a lasso model on X, y and plot the names of variables (names) and the coeffiecients from lasso (lasso_coef),
lasso_coef = lasso.fit(X,y).coef_ plt.plot(range(len(names), lasso_coef) plt.xticks(range(len(names), names, rotation=60) plt.ylabel('Coefficients') plt.show()
In RandomizedSearchCV, what is used as an argument instead of param_grid and what additional argument is necessary that is not included in GridSearchCV?
param_distributions, n_iter = number of iterations to stop at
What to you import from sklearn for scaling?
from sklearn.preprocessing import scale or StandardScaler
import Logistic regression from sklearn
from sklearn.linear_model import LogisticRegression
Import and put together steps to Create a GridSearchCV model using ElasticNet using an Imputer with mean as the strategy, StandarScaler in a pipeline. Training on training data and test on 40%, calculate the r2 on test
from sklearn.linear_model import ElasticNet from sklearn.model_selection import GridSearchCV, train_test_split from sklearn.pipeline import Pipeline from sklearn.preprocessing import Imputer, StandardScaler # Setup the pipeline steps: steps steps = [('imputation', Imputer(missing_values='NaN', strategy='mean', axis=0)), ('scaler', StandardScaler()), ('elasticnet', ElasticNet())] # Create the pipeline: pipeline pipeline = Pipeline(steps) # Specify the hyperparameter space parameters = {'elasticnet__l1_ratio':np.linspace(0,1,30)} # Create train and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4, random_state=42) # Create the GridSearchCV object: gm_cv gm_cv = GridSearchCV(pipeline, parameters, cv=3) # Fit to the training set gm_cv.fit(X_train, y_train) # Compute and print the metrics r2 = gm_cv.score(X_test, y_test) print("Tuned ElasticNet Alpha: {}".format(gm_cv.best_params_)) print("Tuned ElasticNet R squared: {}".format(r2))
What can be imported from sklearn for imputation?
from sklearn.preprocessing import Imputer
Import XGBoost and instantiate a classifier
import xgboost as xgb xg_cl = xgb.XGBClassifier(objective='binary:logistic', n_estimators = 10, seed=123)
What are some common regression metrics?
Root mean squared error (RMSE) - use more often Mean absolute error (MAE) Mean absolute percent error (MAPE)
Just because a dataframe has no values as NaN in each column does that mean the dataframe has no missing values?
No, missing values could be labeled 0, ?, missing, etc.
Why do we use GridSearch?
How do we choose the right parameters for several parameters leading to the lowest loss possible when they interact in non-obvious and non-linear ways
Why do we use regularization?
Large coefficients can lead to overfitting, regularization applies a penalty to parameters to ensure that as they grow they are adding value.
What does a recall of 1 mean?
a low threshold in which you have classified all events at the expense of misclassifying with lots of false positives
What are common loss functions used in XGBoost
Regression: reg:linear Classification: reg:logistic (just decision, not probility) Classificatino: binary:logistic - want probability
import classification report and confusion matrix from sklearn and print output for y_pred, y_test?
from sklearn.metrics import classification_report, confusion matrix classification_report(y_test, y_pred) confusion_matrix(y_test, y_pred)
What is XGBoost?
Optimized gradient boosting machine learning library originally written in C++. Popular because of it's speed and performance and core algorithm is parallelizable consitently outperforms single-algorithm methods
Explain Ridge Regression?
How is the regression line chosen
minimizing the loss function , here it's the vertical distance b/w data point and line - Sum of Squared Errors - Ordinary Least Squares (OLS) - Miminize sum of squared residuals
What is used to create dummy variables in scikit learn and pandas? using the pandas method how do you drop the first dummy variable so that redundant information is not passed to a model?
Scikit learn - OneHotEncoder() pandas - df.get_dummis() pd.get_dummies(df, drop_first=True)
What is R^2?
Quantifies the amount of variance in the target variable accounted for by the feature variables
How do you import mean_squared_error from sklearn?
from sklearn.metrics import mean_squared_error
What are the axes of the ROC curve?
y: True Positive Rate (sensitivity, recall) x: False Positive Rate (1-specficity)
Creating an Imputer instance where we want to replace NaN values with the mean using the column
imp = Imputer(missing_values='NaN', strategy='mean', axis=0) imp.fit_transform(df)
What is boosting?
Meta-algorithm. Ensemble used to convert many weak learners into a strong learner. Where the strong learner can be tuned to achieve high accuracy. By iteratively learning a set of weak models on subsets of the data and weighting each weak prediction according to each weak learner's performance. Then combine the weighted predictions to obtain a single weighted prediction that is much better as any single prediction.
How is RandomizedSearchCV different from GridSearchCV?
With your searching over a large space it can be computationally expensive to use GridSearch. Randomized Search creates a (possible infinite) range of hyperparameter values per hyperparameter that you would like to search over. Sets the number of iterations you would like for the search to occur during each iteration, randomly draw a value in the range of specified values for each hyperparameter searched over and train/evaluate a model. After reaching the max number of iterations, select the best configuration
Import a decision tree classifier and Randomized Search CV from sklearn
from sklearn.tree import DecisionTreeClassifier from sklearn.model_selection import RandomizedSearchCV
Supervised Learning?
labeled data, Numeric or categorical features Numeric features should be scaled (z-score) categorical feautures should be one-hot encoded
Create X, y numpy arrays from dataframe df where target variable is outcome.
X = df.drop('outcome').values y = df.oucome
create a cross validation using xgboost using input as an already created dmatrix with the params dictionary as params, 6 folds with 5 trees
xgg.cv(dtrain=dmatrix, params=params, nfold=5, num_boost_round=5, metrics='auc', as_pandas=True, seed = 13)
How do you import Ridge Regressor and what are the arguments?
from sklearn.linear_model import Ridge ridge = Ridge(alpha0.1, normalize = True) Normalize will ensure all variables are on the same scale
What does predict_proba return?
probabiliites for each class in order, for binary you get [prob_0, prob_1]
import GridSearchCV from sklearn. how would you get the best params and best score?
from sklearn.model_selection import GridSearchCV model.best_params_ model.best_score_
Import roc_auc_score from sklearn? how can you use it in cross_val_score?
from sklearn.metrics import roc_auc_score cross_val_score(X, y, cv=5, scoring='roc_auc')
WHat can be caluculated from a confusion matrix?
accuracy, precision, recall (sensitivity), specificity F1 Score: 2 (precisionrecall)/(precision + recall) High precision: not many real emails predicted as spam high recall: predicted most spam emails correctly
What's most common metric for binary classification?
ROC_AUC
Using pandas how could you drop all rows from dataframe df that contain NaN in any column?
df.dropna()
In a pipeline consisting of several steps, how would you create a parameters dictionary to set n_neighbors to a range from 1 to 50 for knn?
parameters = {knn__n_neighbors = np.arange(1, 50)} need double underscore after knn to connect to knn parameter
Why is standardizing necessary?
With algorithms that use distance to inform them, then features on larger scales can unduly influence the model. We normalize (center and standardize) to address this
Import modules and run a 5-fold cross validation linear regression on X, y and store in cv_results
from sklearn.model_selection import cross_val_score from sklearn.linear_model import LinearRegression reg = LinearRegression() cv_results = cross_val_score(reg, X, y, cv=5)
What is class imbalance?
When one class is much greater than another Example spam email, real mail 99%, spam 1%, just call everything real mail and you would be 99% correct but this is not helpful
Section 3
(50 cards)
For gradient descent, if the slope is positive how do the weights change?
Instead of just subtracting the slope from the current value whihc might take too big a step. The solution is to use a learning rate and update each weight by subtracting learning_rate*slope
What is the neg_mean_squared_error metric in sklearn?
It's sklearn's way of calculating the mean_squared_error in an API compatible way. It's optimizing to reduce loss so it was made negative.
How do you save a keras model and reload it? What file format is necessary so that you do not need to save architecture and weights separately?
from keras.models import load_model model.save('my_model.h5) my_model = load_model('my_model.h5) HDF5 is the file type
How does forward propogation work in neural networks?
input array is used in a dot product with the weight arrary for each node, these are summed or fed into and activation function to provide the output of a node. The output of the first hidden node is input for the second layer weight arrays to provide input to the output node.
For the column df.lot, fill missing values with 0.
df.lot.fillna(0)
What are hierachical clustering?
Clusters that contain one another - Animals --> Mammals + Reptiles; Mammals --> humans + apes + rodents. Hierarchical clustering is divided into aggolomertive and divisive clustering.
What activation function just returns the node value?
Identity function
Model capacity is the ability to learn features, how is model capacity increased? What sequence should be followed on a new project?
By adding layers or nodes, both will add model complexity. start with one layer with a small amount of nodes, then add nodes followed by layers until error stops improving and then stop back.
What is forward propagation and backpropagation?
forward propogation sends input data through the hidden layers to the output layer. Backpropogation takes the error and propogates it back through the hidden layers toward the input layer. It calculates the necessary slopes sequentially for the weights closest to the prediction through the hidden layers eventially back to the weights coming from the inputs. These slopes are then used to update the weights. Will always do forward propogation to get an error that is used in back propogation
What is a typical learning rate? I f the slope of the mean squared loss function is -24 and the weight was 2, what would the updated weight be?
0.01 2 - (0.01*-24) = 2.24
How do you create a dictionary from a dataframe (df) with columns as lists associated with the column name as key?
df.to_dict('records')
fit a KMeans model with an expected 3 clusters on Array and get the labels
model = KMeans(n_clusters=3) model.fit(Array) labels = model.predict(Array)
What is the dying neuron problem?
When a neuron takes a value of zero for all rows in your data. Once a node starts always getting negative inputs it may continue only getting negative inputs and thus contributes nothing to the model. Appropriately settting the learning rate can address this issue. Alternatively using more advanced ReLU methods such as LeakyReLU which have a gradient below zero, LeakyReLU = max(0.1x, x)
What argument is required in the first dense layer of a Sequential model?
input_shape = (number_of_columns, )
What is used to verify a loaded model structure?
model.summary()
How do you import and use Early Stopping in keras? What is a typical patience?
from keras.callbacks import EarlyStopping early_stopping = EarlyStopping(patience=2) patience of 2-3 is typical as this is for epochs and if improvement doesn't occur with the 3rd pass through the data then it probably will not.
What does StandardScaler do?
transforms each feature to have mean 0 and variance 1
You have created a dataframe with cluster labels and the target value. Create a crosstabulation of label and target from the dataframe df.
pd.crosstab(df['label'], df['target'])
After running GridSearchCV and RandomizedGridSearchCV (some_model), how can you print out the best parameters and best score?
some_model.best_score_ some_model.best_params_
What linear algebra process is used during forward propogation?
Dot product between weights and input
How do you import kmeans from sklearn?
from sklearn.cluster import KMeans
What is the vanishing gradient problem?
Vanishing gradients occur when many layers have very small slopes due to being on the flat part of the tanh curve. in deep networks, updates to backprop were close to zero. ReLU and variations address the vanishing gradient problem
What is overfitting and underfitting?
In overfitting your model will fit oddities in training data due to happenstance that are not generalizable. Underfitting, the model fails to find important features in the training data.
where is early_stopping placed in a keras flow? How do you include a validation split?
model.fit(predictors, target, epochs=20, validation_split=0.3, callbacks=[early_stopping])
If you have gone through 4 iterations of calculating slopes (using backward propagation) and then updated weights, how many times must you have done forward propagation?
4
What is unsupervised learning?
finds patterns in data falls into clustering and dimension reduction cluster customers by their purchases compressing data using purchase patterns
import and use standard scaler on the samples array to create samples_scaled
from sklearn.preprocessing import StandardScaler scaler = StandardScaler() scaler.fit(samples) samples_scaled = scaler.transform(samples
Why do we use activation functions? Give example using banking transactions with changes in the number of children you have.
These functions allow us to capture non-linearities. Example with banking transactions. Going from 1 child to 2 children could impact your banking transactions differently than going from 3-4 children.
What is gradient descent?
Start at a random point (randomized weights), and take steps until you are somehwere flat, find the slope and take step downhill. if point on a loss function has a slope (derivative) then take a step in the opposite direction to move downhill, continue the process
What is used to measure cluster quality when no labels are present? How can you get this measuement?
Inertia - measures how spread out clusters are - lower is better. Measures distance from each sample to centroid of it's cluster model.inertia_
What's the difference between StandardScaler and Normalizer?
StandardScaler standardizes features by removing the mean and and scaling to unit variance. Normalizer rescales each sample independently of the other StandardScaler acts Column-wise Normalizer acts Row-wise -useful for sparse datasets
How is unsupervised different from supervised?
supervised learning finds patterns for a prediction task using labels unsupervised learning learning patterns without labels or guidance to a p
From a clustering KMeans clustering model (model), how can you output the centroids?
model.cluster_centers_
What is a loss function?
Aggregates errors in predictions from many data points into a single number. This is a measure of the model's predictive performance. Goal is to minimize the loss function
What are batches and epochs in stochastic gradient descent?
In stochastic gradient descent, slopes are calculated on a subset of the data, a batch. Use a different batch of data to calculate the next update. Start over from the beginning once all data is used. Each time through the training data is called an epoch.
What does adding nodes to a hidden layer allow a neural network to account for?
More Interactions
How are feature variances important to kmeans clustering?
in kmeans: feature variance = feature influence, so larger variance features will dominate
How are NN weights updated after an epoch using previous weights, the learning rate and the calculated slope
updated = weights - learning_rate * slope weights - learning_rate (2 - input_data error)
What is log loss or crossentropy?
How can you use inertia to select the number of clusters?
Plot inertia (y) by number of clusters(x), look for elbow in the plot
What is the difference between sigmoid and softmax activation functions?
Explain ReLU activation.
ReLU - Rectified Linear Activation - composed of two linear pieces RELU(x) = 0 if x < 0, x if x >=0
How does regression work, start with average?
Regression starts with the average and then adds the effect of age, weight, other variables
Why is kfold cross validation rarely used in deep learning?
Deep learning is typically applied to large data sets in which repeated training is too time consuming. Also with large data the validation set can be large and representative so it is trusted.
What is required in the compile layer (2 arguments)?
model.compile(optimizer = opt , loss = loss_function)
Using keras how do you convert a categorical target (result) to a Onehot encoded matrix?
from keras.utils import to_categorical target = to_categorical(result)
Which layers of a NN capture complex or 'higher level' interactions?
The later layers. Earlier layers capture simple interactions and these are built upon
If your predictions were all exactly right, and your errors were all exactly 0, the slope of the loss function with respect to your predictions would also be 0. In that circumstance, which of the following statements would be correct? The updates to all weights in the network would also be 0. The updates to all weights in the network would be dependent on the activation functions. The updates to all weights in the network would be proportional to values from the input data.
The updates to all weights in the network would also be 0.
What is the difference between LabelEncoder and OneHotEncoder? How can these be used in a pipeline?
LabelEncoder converts a column of strings into integers OneHotEncoder: takes column of integers and encodes them as dummy variables These cannot currently be used in a pipeline
What needs to be imported from scipy to perform hierarchical clustering? Then create a cluster object called mergings using samples as input.
from scipy.cluster.hierarchy import linkage, dendrogram mergings = linkage(samples, method='complete')
Section 4
(50 cards)
What is the advantage of dimension reduction?
more efficient storage and computation. Can remove less-informative noise features which cause issues with regression/predictino
What is the autocorrelation function?
ACF - shows autocorrelation function for all lags. Any significant non-zero values implies that you can forecast from the past
How can you get a list of acf values?
from statsmodels.tsa.stattools import acf acf(x)
What is NMF and how are it's models different from PCA?
Non-negative matrix factorization. NMF models are interpretable, unlike PCA, making them easy to interpret and explain to others
Import the necessary library and plot the acf for series x up to 20 lags with a confidence interval of 0.05?
from statsmodels.graphics.tsaplots import plot_acf plot_acf(x, lags=20, alpha=0.05)
What do the axes of a TSNE plot mean?
The axes are not interpretable - t-SNE features are different every time - stochastic NOT deterministic
What is PCA?
Principal Component Analysis Fundamental reduction Technique. First step is decorrelation, second step reduces dimension Rotates sampels to be aligned with axes, shipfts data samples so they have mean of 0.
import t-SNE and apply it to samples with a learning rate of 100
from sklearn.manifold import TSNE model = TSNE(learning_rate=100) transformed = model.fit_transform(samples)
How do you create an array with two rows and three columns from a 1d numpy array?
array.reshape( (2,3) )
What is necessary to import from scipy to extract cluster labels from hierarchical clustering using input mergings and a distance of 15?
from scipy.cluster.hierarchy import fcluster labels = fcluster(mergings, 15, criterion='distance')
What is autocorrelation?
Correlation between a variable and a lagged version of itself, also called serial correlation. Usually mean lag1 correlation.
What is the statistical test for Random Walk with drift?
Pt = Mu + Pt-1 + Et Regression Test: Pt - Pt-1 = alpha + Beta*Pt-1 + Et Test: Ho: Beta = 0 (random walk) H1: B < 0 (not random walk) this is Dickey-Fuller Test
What does MaxAbsScaler do to the data?
transforms the data so that all users have the same influence on the model regardless of how many differetn artists they have listened to. Scale each feature by its maximum absolute value. This estimator scales and translates each feature individually such that the maximal absolute value of each feature in the training set will be 1.0. It does not shift/center the data, and thus does not destroy any sparsity.
What's the difference between single linkage and complete linkage?
complete: distance b/w clusters is the distance between furthest points Single: distance b/w closest points of clusters
In a dendrogram, what defines the distance between clusters?
Distance is defined by linkage method complete: distance b/w clusters is maximum distance between their samples
What is the interpretation of NMF Components applied to documents? What is the interpretation applied to images?
When applied to text, NMF is one method of Topic Modeling When applied to images, NMF components are parts of images
When looking at correlation between two stocks should you evaluate their returns or levels?
returns because analyzing two trending series. Example - DJI and UFO sightings, both trend up over time but there is no relationship between the two series.
Can you forecast a random walk?
No, best guess for tomorrow's price is today's price.
Import PCA from sklearn and fit on samples data set?
from sklearn.decomposition import PCA model = PCA() model.fit(samples)
Calculate the daily change in the df.Price column as a new column called df.Change and calculate it's autocorrelation?
df.Change = df.Price.diff() autocorrelation = df.Change.autocorr()
What is White Noise?
A Time Series with: - constant mean - constant variance - zero autocorrelations at all lags
Calculate the autocorrelation of df.Col1?
df.Col1.autocorr()
How can you calculate the percent change or the difference between rows of a dataframe column?
df['col'].pct_change() df['col'].diff()
What must be true for the input and output of NMF?
the input data must be non-negative only This applies to words and images
After pca.fit(samples), how can you get the number of PCs and the explained variance?
pca.n_components_ pca.explained_variance_
How do you create a 1D array from a numpy array with multipe rows?
array.flatten()
What is the preferred way to save a sparse matrix in python instead of as anumpy array? Because sklearn doesn't support csr_matrix, what is used instead?
scipy.sparse.csr_matrix csr_matrix remembers only the non-zero entries to save space in sklearn use TruncatedSVD - performs same transformation as PCA but will accept a csr_matrix as input.
What does a positive or negative autocorrelation mean?
with returns: positive autocorrelation - trend following - a increase yesterday means an increase today negative autocorrelation - mean reverting - an increase yesterday means a decrease today.
What does t-SNE stand for?
t-distributed stochastic neighbor embedding maps samples from high-diimensional space to 2D or 3D. Map approximately preserves nearness of samples. Great for inspecting datasets
How do you create an array of unique values from a pandas column in the dataframe df?
df.column.unique()
How can you calculate similarity of a vector to a dataframe of vectors?
df.dot(vector)
Convert the column Price currently of type string to float in the df dataframe?
df.Price = df.Price.apply(lambda x: float(x))
How do you import NMF, what type of arrays will it take and what must be specified every time?
from sklearn.decomposition import NMF NMF works with NumPy arrays and csr_matrix you must specify n_components everytime
Calculate the correlation between two columns of dataframe df, col1, col2? q
df.col1.corr(df.col2)
What is the learning rate argument for TSNE?
Learning rate is a parameter to be tuned - the wrong choice will have points bunched together. Try values between 50 and 200
What is the intrinsic dimension of a dataset?
The number of features needed to approximate the dataset. What is the most compact representation - pca can tell us.
Merge stocks and bonds dataframes that both have datetime indexes to generate a dataframe for days in which both markets are open?
stocks.join(bonds, how='inner')
What is the benefit of PCA?
Creates non-correlated, PCA features are not linearly correlated. PCA alinges principal components along directions of maximal variance
How is time series data different from cross-sectional data?
Time series data is ordered in time, where for cross-sectional the data point is taken at one point or where time is meaningless - example - google searches for diet - hits low during the holidays then spikes with new year
How can PCA identify intrinsic dimension?
using signficant variance
Create a simple linear regression from Statsmodels for variables, x and y
import statsmodels.api as sm sm.OLS(x, y)
What is the y-axis (numeric) on a dengrogram mean?
height on dendrogram = distance between merging clusters
In pandas how can you turn an index into a datatime index?
df.index = pd.to_datetime(df.index)
Use pandas to resample a dataframe from daily to weekly
df = df.resample(rule = 'W', how = 'last)
What does an alpha = 0.05 mean as a confidence interval for ACF? what should alpha be set to if you do not want to see confidence bands?
There is a 5% change that if true autocorrelation is zero, it will fall outside blue band. alpha = 1.0 will remove CI bands
Create a dendrogram from mergings, with the list companies for labels, rotate 90, font size=6
dendrogram(mergings, labels=companies leaf_rotation = 90, leaf_font_size = 6)
What can be used to compare documents using NMF values for a recommender system?
cosine similarity
What needs to be imported to execute the augmented dickey-fuller test on X?
from statsmodels.tsa.stattools import adfuller adfuller(X)
What is the relationship between r-squared and correlation?
correlation^2 = rsquared sign(corr) = sign(regression slope)
What is the equation for Random Walk?
Today's price is equal to yesterday's price plus some noise. Pt = Pt-1 + Et
Section 5
(40 cards)