Usual high energy New Physics searches are about comparing experiments to predictions. Where predictions many times come from Monte Carlo simulations, or more recently from supervised Machine Learning algorithms. However, these tests could contain potential biases and/or lack of interpretability due to the assumptions or methods involved. Bayesian Inference, or Probabilistic Machine Learning, is a framework in which one can post a probabilistic model on how the data is generated and then use the data to infer the relevant parameters of the model. One of the key add-ons in this framework is not only that one has full control on how much domain knowledge one plugs into the model and how much is inferred, but also that it gives interpretability to the results. Moreover, the framework and available tools are highly enhanced by their outstanding recent development in the ML industry. In this talk we'll describe the modern Bayesian Inference methods and available tools, and then show explicit examples on how it can be used in relevant LHC observables concerning multilepton excesses and jet classifications, among others.