Deep Learning final review

Intro of AI

• AI -> machine learning -> deep learning -> generative AI
• Rule-based system
  • Every rule has an exception
  • conflicting and hard to maintain
• Three categories of machine learning
  • Basic algorithms
• Representation and feature
• Three driving forces of AI

Model evaluation

• H0 and H1
• Type I and Type II errors
• Confusion matrix
• Training vs testing vs validation vs cross-validation

Image data & clustering

• Image data representation
  • Color
  • Grayscale
• Clustering and use cases
• K-Means algorithm
  • Three steps
• Issues with image clustering

Feedforward net

• What does deep/feedforward/neural/network mean
• Fully connected layer, neurons, weights and bias
• Parametric model
  • Parameter and hyperparameter
• Loss function
  • Cross entropy
• Optimization procedure
  • Gradient descent
  • backpropagation

Activation functions and regularization

• Common activation functions
  • Sigmoid/tanh, Softmax
  • ReLU, leaky ReLU and parametric ReLU
• Overfitting, appropriate-fitting,
• underfitting and regularizationNorm: L1, L2 Dataset augmentation
• Dropout
  • Training stage and testing stage
• Learning rate and why adaptive
• Training epoch, mini-batch approach

CNN

• Scalar, vector, matrix, tensor
• Convolution, convolutional layer and three depths
  • Locality
  • Spatial invariance
• Filter/kernel, depth slice, sliding/convolving, stride and zero-padding
• Parameter reduction and hierarchy of features
• Max pooling
• Architecting CNN and transfer learning
  • architecting: conv -> pool
  • transfer leanring: strip down the final fc layer and fit a new fc layer
• Image classification applications
• Discriminative modeling vs generative modeling
• Generative Adversarial Networks (GAN)
  • Discriminative modeling: tell whether a thing is generated or not
  • Generative modeling: generate random shits
  • Training goal: generated shits become harder to tellif it’s generated or not
• Temperature in LLM: how much LLM is smoking

Networked data and graph theory

• Vertices/nodes, edges/links, directed/undirected, weighted/unweighted, neighbors/adjacency
• Path, distance, eccentricity, cycle and DAG
  • eccentricity = a node’s longest geodesic (shortest path btwn 2 nodes)
  • DAG: a graph that has no cycles
• Density and clustering coefficient
  • density = edges / num of all possible edges
  • cc: # of pairs of neighbors that are connected
• Centrality measures: degree, closeness, betweenness, eigenvector
  • degree: in & out edges
  • closeness: how easily a node can reach the rest of the ntwk
  • betweenness: how critical is a node connecting different groups
  • eigenvector: neighbor is more important -> itself is more important
• Degree distribution and power law
  • $P(k)～k^{-\gamma}$
  • A small number of things get the most popularity