Discriminative learning of Bayesian networks (BNs) for classification tasks is often beneficial compared to generative learning. This is particularly true in case of model mismatch, i.e. when the BN cannot represent the true data distribution. In the past, we developed maximum margin parameter learning for Bayesian network classifiers and Gaussian Mixture models. Furthermore, we used the margin objective for approximate and exact structure learning. This research is extended within this proposal. The focus is three-fold: (i) Extension of margin-based parameter learning to a hybrid paradigm merging the advantages of generative and discriminative learning. We aim at extending our learning framework to semi-supervised, missing features, and latent variable scenarios. This requires efficient inference during iterative parameter optimization. Additionally, both the discriminative and hybrid learning approach are introduced to potentially deep sum-product networks (SPNs). They explicitly represent the inference process, i.e. structures (including latent variables) exhibiting computational benefits for inference can be exploited. (ii) Discriminative search-and-score structure learning in BNs is time-consuming. We are interested in approximating the non-decomposable discriminative score by a decomposable surrogate to ease the computational costs for score evaluation in BNs. Furthermore, we aim at developing structure learning algorithms for SPNs introducing a global scoring function with an inference cost penalty. (iii) To consolidate SPNs with respect to empirical performance we will compare all developed models to popular generative and discriminative models from the deep community, i.e. restricted Boltzmann machine, auto-encoders, deep belief networks, multi-layer perceptron. Additionally, one particularly interesting recent deep model generative stochastic networks is considered.