Symbolic Regression Using Likelihood Based Objective Functions

[IsmamHuda]

Hi,

I've been fascinated with Symbolic Regression forever. I am mostly someone interested in statistical modelling, and often reaching for GLM techniques because I might be working with:

• count data (e.g. insurance claim counts) suited for Poisson regression
• strictly positive data (e.g. insurance claim sizes) suited for Gamma regression or lognormal regression (using Normal and log link)
• binary data suited for Logistic regression (aka Bernoulli regression)
• proportion data between 0 and 1, suited for Beta regression

GLMs have always frustrated me however as it's not built-in to the GLM procedure to discover non-linearity & interaction effects. Whilst there are ways around this, they largely involve making lucky guesses about what non-linearity & interaction effects may exist.

So I wanted to combine GLMs ability to predict on data with a specified distribution, with the power of Symbolic Regression.

What I came up with is this approach of Symbolic Regression, using likelihood-based objective functions.

I've produced a working prototype using Gamma distribution, and it appears to perform rather well. I am keen to see if I can get it working for the other distributions too, espescially logistic, but am being challenged by my lack of programming expertise, but hopefully will get there eventually.

HTML view: https://nbviewer.org/github/IsmamHuda/SymbolicRegressionWithLikelihoodBasedObjectives/blob/main/Demonstration.ipynb

Github:
https://github.com/IsmamHuda/SymbolicRegressionWithLikelihoodBasedObjectives/blob/main/Demonstration.ipynb

Would love to champion this as an alternative GLMs, but am conscious of some possible shortcommings:

• no obvious way as to how to support a link function (often needed to have restrict the mean to physically possible values, e.g. a non-ngative mean)
• potential difficulty having many predictors (although GLMs also struggle with this anyway)

Keen to talk and hear if people are interested in what I'm doing. And always keen to collaborate if anyone is interested.

Regards,
Ismam Huda

[IsmamHuda]

UPDATE: I've got logistic regression working now, and made some big improvements to the objective function and cleaned up the notebook.

• Link functions working, imported from GLM.jl
• Ability to use weights
• Visualisation of Actual vs Fitted

Some results:
To perform logistic regression, I generated 5000 synthetic observations of binary Bernoulli data (0 or 1) with the following mean.


And SymbolicRegression was able to identify the follwing equation, which when visualised over the relevant domain is gosh darn close to the actual mean equation I set:

I've done a similar test with Gamma regression also in the notebook.

The notebook is quite large, 50mb, due to the long output from the fitting.
HTML view is available on NBViewer: https://nbviewer.org/github/IsmamHuda/SymbolicRegressionWithLikelihoodBasedObjectives/blob/main/Demonstration.ipynb

But I notice it's not the most reliable so HTML file also uploaded to my github repo too, and can be downloaded from there.