choosing the best model assignment quizlet

Explore Neptune Scale: tracker for foundation models -> Tour a live project 📈

The Ultimate Guide to Evaluation and Selection of Models in Machine Learning

To properly evaluate your machine learning models and select the best one, you need a good validation strategy and solid evaluation metrics picked for your problem. 

A good validation (evaluation) strategy is basically how you split your data to estimate future test performance. It could be as simple as a train-test split or a complex stratified k-fold strategy. 

Once you know that you can estimate the future model performance, you need to choose a metric that fits your problem . If you understand the classification and regression metrics, then most other complex metrics (in object detection, for example) are relatively easy to grasp.  

When you nail those two, you are good.

In this article, I will talk about:

• Choosing a good evaluation method (resampling, cross-validation, etc)
• Popular (and less known) classification and regression metrics
• And bias / variance trade-offs in machine learning. 

So let’s get to it. 

Note from the product team

It makes model evaluation and selection way easier.

Check the docs

Watch the 2-min product demo

See what comparison functionality Neptune offers

Just to make sure we are on the same page, let’s get the definitions out of the way.

What is model evaluation?

Model evaluation  is a process of assessing the model’s performance on a chosen evaluation setup. It is done by calculating quantitative performance metrics like F1 score or RMSE or assessing the results qualitatively by the subject matter experts. The machine learning evaluation metrics you choose should reflect the business metrics you want to optimize with the machine learning solution.

What is model selection?

Model selection  is the process of choosing the best ml model for a given task. It is done by comparing various model candidates on chosen evaluation metrics calculated on a designed evaluation schema. Choosing the correct evaluation schema , whether a simple train test split or a complex cross-validation strategy, is the crucial first step of building any machine learning solution.

How to evaluate machine learning models and select the best one?

We’ll dive into this deeper, but let me give you a quick step-by-step:

Step 1: Choose a proper validation strategy . Can’t stress this enough, without a reliable way to validate your model performance, no amount of hyperparameter tuning and state-of-the-art models will help you.

Step 2: Choose the right evaluation metric. Figure out the business case behind your model and try to use the machine learning metric that correlates with that. Typically no one metric is ideal for the problem.

So calculate multiple metrics and make your decisions based on that. Sometimes you need to combine classic ML metrics with a subject matter expert evaluation. And that is ok.

Step 3: Keep track of your experiment results . Whether you use a spreadsheet or a dedicated  experiment tracker , make sure to log all the important metrics, learning curves, dataset versions, and configurations. You will thank yourself later.

Step 4: Compare experiments and pick a winner.  Regardless of the metrics and validation strategy you choose, at the end of the day, you want to find the best model. But no model is really best, but some are good enough.

So make sure to understand what is good enough for your problem, and once you hit that, move on to other parts of the project, like model deployment or pipeline orchestration.

Model selection in machine learning (choosing model validation strategy)

Resampling methods.

Resampling methods, as the name suggests, are simple techniques of rearranging data samples to inspect if the model performs well on data samples that it has not been trained on. In other words, resampling helps us understand if the model will generalize well .

Random Split

Random Splits are used to randomly sample a percentage of data into training, testing, and preferably validation sets. The advantage of this method is that there is a good chance that the original population is well represented in all the three sets. In more formal terms, random splitting will prevent a biased sampling of data.

It is very important to note the use of the validation set in model selection. The validation set is the second test set and one might ask, why have two test sets?

In the process of feature selection and model tuning, the test set is used for model evaluation. This means that the model parameters and the feature set are selected such that they give an optimal result on the test set. Thus, the validation set which has completely unseen data points (not been used in the tuning and feature selection modules) is used for the final evaluation.

Time-Based Split

There are some types of data where random splits are not possible . For example, if we have to train a model for weather forecasting, we cannot randomly divide the data into training and testing sets. This will jumble up the seasonal pattern! Such data is often referred to by the term – Time Series.

In such cases, a time-wise split is used. The training set can have data for the last three years and 10 months of the present year. The last two months can be reserved for the testing or validation set.

There is also a concept of window sets – where the model is trained till a particular date and tested on the future dates iteratively such that the training window keeps increasing shifting by one day (consequently, the test set also reduces by a day). The advantage of this method is that it stabilizes the model and prevents overfitting when the test set is very small (say, 3 to 7 days).  

However, the drawback of time-series data is that the events or data points are not mutually independent . One event might affect every data input that follows after. 

For instance, a change in the governing party might considerably change the population statistics for the years to follow. Or the infamous coronavirus pandemic is going to have a massive impact on economic data for the next few years. 

No machine learning model can learn from past data in such a case because the data points before and after the event have major differences.

K-Fold Cross-Validation

The cross-validation technique works by randomly shuffling the dataset and then splitting it into k groups. Thereafter, on iterating over each group, the group needs to be considered as a test set while all other groups are clubbed together into the training set. The model is tested on the test group and the process continues for k groups.

Thus, by the end of the process, one has k different results on k different test groups. The best model can then be selected easily by choosing the one with the highest score.

7 Cross-Validation Mistakes That Can Cost You a Lot [Best Practices in ML]

Stratified k-fold.

The process for stratified K-Fold is similar to that of K-Fold cross-validation with one single point of difference – unlike in k-fold cross-validation, the values of the target variable is taken into consideration in stratified k-fold.

If for instance, the target variable is a categorical variable with 2 classes, then stratified k-fold ensures that each test fold gets an equal ratio of the two classes when compared to the training set.

This makes the model evaluation more accurate and the model training less biased.

Bootstrap is one of the most powerful ways to obtain a stabilized model. It is close to the random splitting technique since it follows the concept of random sampling.

The first step is to select a sample size (which is usually equal to the size of the original dataset). Thereafter, a sample data point must be randomly selected from the original dataset and added to the bootstrap sample. After the addition, the sample needs to be put back into the original sample. This process needs to be repeated for N times, where N is the sample size.

Therefore, it is a resampling technique that creates the bootstrap sample by sampling data points from the original dataset with replacement . This means that the bootstrap sample can contain multiple instances of the same data point.

The model is trained on the bootstrap sample and then evaluated on all those data points that did not make it to the bootstrapped sample. These are called the out-of-bag samples.

Healthcare startup Theta Tech AI uses the neptune.ai experiment tracker for that.

Grouping by validation set is super important to us, and many other people would benefit from using the grouping features with validation. Dr. Robert Toth, Founder of Theta Tech AI

Full case study with Theta Tech AI

Dive into documentation

Get in touch  if you’d like to go through a custom demo with your team

Probabilistic measures

Probabilistic Measures do not just take into account the model performance but also the model complexity . Model complexity is the measure of the model’s ability to capture the variance in the data. 

For example, a highly biased model like the linear regression algorithm is less complex and on the other hand, a neural network is very high on complexity.

Another important point to note here is that the model performance taken into account in probabilistic measures is calculated from the training set only . A hold-out test set is typically not required.

A fair bit of disadvantage however lies in the fact that probabilistic measures do not consider the uncertainty of the models and has a chance of selecting simpler models over complex models.

Akaike Information Criterion (AIC)

It is common knowledge that every model is not completely accurate. There is always some information loss which can be measured using the KL information metric. Kulback-Liebler or KL divergence is the measure of the difference in the probability distribution of two variables.

A statistician, Hirotugu Akaike, took into consideration the relationship between KL Information and Maximum Likelihood (in maximum-likelihood, one wishes to maximize the conditional probability of observing a datapoint X, given the parameters and a specified probability distribution) and developed the concept of Information Criterion (or IC). Therefore, Akaike’s IC or AIC is the measure of information loss. This is how the discrepancy between two different models is captured and the model with the least information loss is suggested as the model of choice.

• K = number of independent variables or predictors
• L = maximum-likelihood of the model 
• N = number of data points in the training set (especially helpful in case of small datasets)

The limitation of AIC is that it is not very good with generalizing models as it tends to select complex models that lose less training information.

Bayesian Information Criterion (BIC)

BIC was derived from the Bayesian probability concept and is suited for models that are trained under the maximum likelihood estimation.

• K = number of independent variables
• L = maximum-likelihood
• N = Number of sampler/data points in the training set

BIC penalizes the model for its complexity and is preferably used when the size of the dataset is not very small (otherwise it tends to settle on very simple models).

Minimum Description Length (MDL)

MDL is derived from the Information theory which deals with quantities such as entropy that measure the average number of bits required to represent an event from a probability distribution or a random variable.

MDL or the minimum description length is the minimum number of such bits required to represent the model.

• D = predictions made by the model
• L(h) = number of bits required to represent the model
• L(D | h) = number of bits required to represent the predictions from the model

Structural Risk Minimization (SRM)

Machine learning models face the inevitable problem of defining a generalized theory from a set of finite data. This leads to cases of overfitting where the model gets biased to the training data which is its primary learning source. SRM tries to balance out the model’s complexity against its success at fitting on the data.

How to evaluate ML models (choosing performance metrics)

Models can be evaluated using multiple metrics. However, the right choice of an evaluation metric is crucial and often depends upon the problem that is being solved. A clear understanding of a wide range of metrics can help the evaluator to chance upon an appropriate match of the problem statement and a metric.

Classification metrics

For every classification model prediction, a matrix called the confusion matrix can be constructed which demonstrates the number of test cases correctly and incorrectly classified. 

It looks something like this (considering 1 -Positive and 0 -Negative are the target classes):

• TN: Number of negative cases correctly classified
• TP: Number of positive cases correctly classified
• FN: Number of positive cases incorrectly classified as negative
• FP: Number of negative cases correctly classified as positive

Accuracy is the simplest metric and can be defined as the number of test cases correctly classified divided by the total number of test cases.

It can be applied to most generic problems but is not very useful when it comes to unbalanced datasets. 

For instance, if we are detecting frauds in bank data, the ratio of fraud to non-fraud cases can be 1:99. In such cases, if accuracy is used, the model will turn out to be 99% accurate by predicting all test cases as non-fraud. The 99% accurate model will be completely useless.

If a model is poorly trained such that it predicts all the 1000 (say) data points as non-frauds, it will be missing out on the 10 fraud data points. If accuracy is measured, it will show that that model correctly predicts 990 data points and thus, it will have an accuracy of (990/1000)*100 = 99%! 

This is why accuracy is a false indicator of the model’s health.

Therefore, for such a case, a metric is required that can focus on the ten fraud data points which were completely missed by the model.

Precision is the metric used to identify the correctness of classification.

Intuitively, this equation is the ratio of correct positive classifications to the total number of predicted positive classifications. The greater the fraction, the higher is the precision, which means better is the ability of the model to correctly classify the positive class.

In the problem of predictive maintenance (where one must predict in advance when a machine needs to be repaired), precision comes into play. The cost of maintenance is usually high and thus, incorrect predictions can lead to a loss for the company. In such cases, the ability of the model to correctly classify the positive class and to lower the number of false positives is paramount!

Recall tells us the number of positive cases correctly identified out of the total number of positive cases. 

Going back to the fraud problem, the recall value will be very useful in fraud cases because a high recall value will indicate that a lot of fraud cases were identified out of the total number of frauds.

F1 score is the harmonic mean of Recall and Precision and therefore, balances out the strengths of each. 

It is useful in cases where both recall and precision can be valuable – like in the identification of plane parts that might require repairing. Here, precision will be required to save on the company’s cost (because plane parts are extremely expensive) and recall will be required to ensure that the machinery is stable and not a threat to human lives.

ROC curve is a plot of true positive rate (recall) against false positive rate (TN / (TN+FP)). AUC-ROC stands for Area Under the Receiver Operating Characteristics and the higher the area, the better is the model performance. 

If the curve is somewhere near the 50% diagonal line, it suggests that the model randomly predicts the output variable.

️ F1 Score vs ROC AUC vs Accuracy vs PR AUC: Which Evaluation Metric Should You Choose?

Log loss is a very effective classification metric and is equivalent to -1* log (likelihood function) where the likelihood function suggests how likely the model thinks the observed set of outcomes was. 

Since the likelihood function provides very small values, a better way to interpret them is by converting the values to log and the negative is added to reverse the order of the metric such that a lower loss score suggests a better model.

Gain and Lift Charts

Gain and lift charts are tools that evaluate model performance just like the confusion matrix but with a subtle, yet significant difference. The confusion matrix determines the performance of the model on the whole population or the entire test set, whereas the gain and lift charts evaluate the model on portions of the whole population. Therefore, we have a score (y-axis) for every % of the population (x-axis). 

Lift charts measure the improvement that a model brings in compared to random predictions. The improvement is referred to as the ‘lift’.

The K-S chart or Kolmogorov-Smirnov chart determines the degree of separation between two distributions – the positive class distribution and the negative class distribution. The higher the difference, the better is the model at separating the positive and negative cases.

Regression metrics

Regression models provide a continuous output variable, unlike classification models that have discrete output variables. Therefore, the metrics for assessing the regression models are accordingly designed.

Mean Squared Error or MSE

MSE is a simple metric that calculates the difference between the actual value and the predicted value (error), squares it and then provides the mean of all the errors.

MSE is very sensitive to outliers and will show a very high error value even if a few outliers are present in the otherwise well-fitted model predictions.

Root Mean Squared Error or RMSE

RMSE is the root of MSE and is beneficial because it helps to bring down the scale of the errors closer to the actual values, making it more interpretable.

Compare all model metrics in neptune.ai

Mean absolute error or mae.

MAE is the mean of the absolute error values (actuals – predictions).

If one wants to ignore the outlier values to a certain degree, MAE is the choice since it reduces the penalty of the outliers significantly with the removal of the square terms.

Root Mean Squared Log Error or RMSLE

In RMSLE, the same equation as that of RMSE is followed except for an added log function along with the actual and predicted values.

x is the actual value and y is the predicted value. This helps to scale down the effect of the outliers by downplaying the higher error rates with the log function. Also, RMSLE helps to capture a relative error (by comparing all the error values) through the use of logs.

R-Square helps to identify the proportion of variance of the target variable that can be captured with the help of the independent variables or predictors. 

R-square, however, has a gigantic problem. Say, a new unrelated feature is added to a model with an assigned weight of w. If the model finds absolutely no correlation between the new predictor and the target variable, w is 0. However, there is almost always a small correlation due to randomness which adds a small positive weight (w>0) and a new loss minimum is achieved due to overfitting.

This is why the R-squared increases with any new feature addition. Thus, its inability to decrease in value when new features are added limits its ability to identify if the model did better with lesser features.

Adjusted R-Squared

Adjusted R-Square solves the problem of R-Square by dismissing its inability to reduce in value with added features. It penalizes the score as more features are added.

The denominator here is the magic element which increases with the increase in the number of features. Therefore, a significant increase in R 2 is required to increase the overall value.

Clustering metrics

Clustering algorithms predict groups of datapoints and hence, distance-based metrics are most effective.

Dunn Index focuses on identifying clusters that have low variance (among all members in the cluster) and are compact. The mean values of the different clusters also need to be far apart.

• δ(Xi, Yj) is the intercluster distance i.e. the distance between Xi and Xj
• ∆(Xk) is the intercluster distance of cluster Xk i.e.distance within the cluster Xk

However, the disadvantage of Dunn index is that with a higher number of clusters and more dimensions, the computation cost increases.

Silhouette Coefficient

Silhouette Coefficient tracks how every point in one cluster is close to every point in the other clusters in the range of -1 to +1.: 

• Higher Silhouette values (closer to +1) indicate that the sample points from two different clusters are far away. 
• 0 indicates that the points are close to the decision boundary 
• and values closer to -1 suggests that the points have been incorrectly assigned to the cluster.

Elbow method

The elbow method is used to determine the number of clusters in a dataset by plotting the number of clusters on the x-axis against the percentage of variance explained on the y-axis. The point in x-axis where the curve suddenly bends (the elbow) is considered to suggest the optimal number of clusters.

• 24 Evaluation Metrics for Binary Classification (And When to Use Them)

Trade-offs in ml model selection

Bias vs variance.

On a high level, Machine Learning is the union of statistics and computation. The crux of machine learning revolves around the concept of algorithms or models, which are in fact, statistical estimations on steroids.

However, any given model has several limitations depending on the data distribution. None of them can be entirely accurate since they are just  estimations (even if on steroids) . These limitations are popularly known by the name of  bias  and  variance . 

A  model with high bias  will oversimplify by not paying much attention to the training points (e.g.: in Linear Regression, irrespective of data distribution, the model will always assume a linear relationship). 

Bias occurs when a model is strictly ruled by assumptions – like the linear regression model assumes that the relationship of the output variable with the independent variables is a straight line. This leads to underfitting when the actual values are non-linearly related to the independent variables.

A  model with high variance  will restrict itself to the training data by not generalizing for test points that it hasn’t seen before (e.g.:  Random Forest  with max_depth = None).

Variance is high when a model focuses on the training set too much and learns the variations very closely, compromising on generalization. This leads to overfitting .

The issue arises when the limitations are subtle, like when we have to choose between a random forest algorithm and a gradient boosting algorithm or between two variations of the same decision tree algorithm. Both will tend to have high variance and low bias.

An optimal model is one that has the lowest bias and variance and since these two attributes are indirectly proportional, the only way to achieve this is through a tradeoff between the two. Therefore, the model selection should be such that the bias and variance intersect like in the image below.

This can be achieved by iteratively tuning the hyperparameters of the model in use (Hyperparameters are the input parameters that are fed to the model functions). After every iteration, the model evaluation must take place with the use of a suitable metric.  

Learning curves

The best way to track the progress of model training or build-up is to use learning curves. These curves help to identify the optimal points in a set of hyperparameter combinations and assists massively in the model selection and model evaluation process.

Typically, a learning curve is a way to track the learning or improvement in the ML model performance on the y-axis and the time or experience on the x-axis.

The two most popular learning curves are:

• Training Learning Curve – It effectively plots the evaluation metric score overtime during a training process and thus, helps to track the learning or progress of the model during training.
• Validation Learning Curve – In this curve, the evaluation metric score is plotted against time on the validation set. 

Sometimes it might so happen that the training curve shows an improvement but the validation curve shows stunted performance. 

This is indicative of the fact that the model is overfitting and needs to be reverted to the previous iterations. In other words, the validation learning curve identifies how well the model is generalizing.

Therefore, there is a tradeoff between the training learning curve and the validation learning curve and the model selection technique must rely upon the point where both the curves intersect and are at their lowest.

Ok, but how do you actually do it?

What is next

Evaluating ML models and selecting the best-performing one is one of the main activities you do in pre-production. 

Hopefully, with this article, you’ve learned how to properly set up a model validation strategy and then how to choose a metric for your problem. 

You are ready to run a bunch of experiments and see what works. 

With that comes another problem of keeping track of experiment parameters, datasets used, configs, and results. 

And figuring out how to visualize and compare all of those models and results. 

For that, you may want to check out:

• “ML Experiment Tracking: What It Is, Why It Matters, and How to Implement It” 
• “15 Best Tools for ML Experiment Tracking and Management”
• “Visualizing Machine Learning Models: Guide and Tools”
• “How to Compare Machine Learning Models and Algorithms”

Other resources

Cross-validation and evaluation strategies from Kaggle competitions:

• Image Classification: Tips and Tricks From 13 Kaggle Competitions (+ Tons of References)
• Binary Classification: Tips and Tricks From 10 Kaggle Competitions
• Tabular Data Binary Classification: All Tips and Tricks from 5 Kaggle Competitions
• Text Classification: All Tips and Tricks from 5 Kaggle Competitions
• Image Segmentation: Tips and Tricks from 39 Kaggle Competitions

Evaluation metrics and visualization:

• Recommender Systems: Machine Learning Metrics and Business Metrics
• How to Track Machine Learning Model Metrics in Your Projects
• The Best Tools to Visualize Metrics and Hyperparameters of Machine Learning Experiments
• How to Do Data Exploration for Image Segmentation and Object Detection (Things I Had to Learn the Hard Way)

Experiment tracking videos and real-world case studies:

• Selecting the best computer vision models at Brainly
• How to Use CI/CD to Automate the RL Evaluation Pipeline
• Comparing CI/CD pipeline runs at Continuum Industries
• How to Compare Images Between Runs
• Scaling ML research at AILS Labs
• Visualizing hyperparameter optimization studies at Theta Tech AI
• How to Monitor Model Training Runs Live

Was the article useful?

More about the ultimate guide to evaluation and selection of models in machine learning, check out our product resources and related articles below:, how to compare machine learning models and algorithms, llm fine-tuning and model selection using neptune and transformers, ml experiment tracking: what it is, why it matters, and how to implement it, how deepsense.ai tracked and analyzed 120k+ models using neptune, explore more content topics:, manage your model metadata in a single place.

Join 50,000+ ML Engineers & Data Scientists using Neptune to easily log, compare, register, and share ML metadata.

• Skip to secondary menu
• Skip to main content
• Skip to primary sidebar

Statistics By Jim

Making statistics intuitive

Model Specification: Choosing the Best Regression Model

By Jim Frost 65 Comments

Model specification is the process of determining which independent variables to include and exclude from a regression equation. How do you choose the best regression model? The world is complicated and trying to explain it with a small sample doesn’t help. In this post, I’ll show you how to decide on the model. I’ll cover statistical methods, difficulties that can arise, and provide practical suggestions for selecting your model. Often, the variable selection process is a mixture of statistics , theory, and practical knowledge.

Specification error is when the independent variables and their functional form (i.e., curvature and interactions) inaccurately portray the real relationship present in the data. Specification error can cause bias, which can exaggerate, understate, or entirely hide the presence of underlying relationships. In short, you can’t trust your results! Consequently, you need to understand model selection in statistics to choose the best regression model.

Model Selection in Statistics

The need to decide on a model often begins when a researcher wants to mathematically define the relationship between independent variables and the dependent variable. Typically, investigators measure many variables but include only some in the model. Analysts try to exclude independent variables that are not related and include only those that have an actual relationship with the dependent variable. During the specification process, the analysts typically try different combinations of variables and various forms of the model. For example, they can try different terms that explain interactions between variables and curvature in the data. During this process, analysts need to avoid a misspecification error.

The analysts need to reach a Goldilocks balance by including the correct number of independent variables in the regression equation.

• Too few : Underspecified models tend to be biased.
• Too many : Overspecified models tend to be less precise.
• Just right : Models with the correct terms are not biased and are the most precise.

To avoid biased results, your regression equation should contain any independent variables that you are specifically testing as part of the study plus other variables that affect the dependent variable.

Related post : When Should I Use Regression?

Model Selection Statistics

You can use various model selection statistics that can help you decide on the best regression model. Various metrics and algorithms can help you determine which independent variables to include in your regression equation. I review some standard approaches to model selection, but please click the links to read my more detailed posts about them.

Adjusted R-squared and Predicted R-squared : Typically, you want to select models that have larger adjusted and predicted R-squared values. These statistics can help you avoid the fundamental problem with regular R-squared —it always increases when you add an independent variable. This property tempts you into specifying a model that is too complex, which can produce misleading results.

• Adjusted R-squared increases only when a new variable improves the model by more than chance. Low-quality variables can cause it to decrease.
• Predicted R-squared is a cross-validation method that can also decrease. Cross-validation partitions your data to determine whether the model is generalizable outside of your dataset.

P-values for the independent variables : In regression, p-values less than the significance level indicate that the term is statistically significant. “Reducing the model” is the process of including all candidate variables in the model, and then repeatedly removing the single term with the highest non-significant p-value until your model contains only significant terms.

Stepwise regression and Best subsets regression : These two automated model selection procedures are algorithms that pick the variables to include in your regression equation. These automated methods can be helpful when you have many independent variables, and you need some help in the investigative stages of the variable selection process. These procedures can provide the Mallows’ Cp statistic, which helps you balance the tradeoff between precision and bias.

Real World Complications in the Model Specification Process

The good news is that there are model selection statistics that can help you choose the best regression model. Unfortunately, there are a variety of complications that can arise. Fear not! I’ll provide some practical advice!

• Your best regression model is only as good as the data you collect. Specification of the correct model depends on you measuring the proper variables. In fact, when you omit important variables from the model, the estimates for the variables that you include can be biased. This condition is known as omitted variable bias . If you can’t include a confounder, consider including a proxy variable to avoid this bias.
• The sample you collect can be unusual, either by luck or methodology. False discoveries and false negatives are inevitable when you work with samples.
• Multicollinearity occurs when independent variables in a regression equation are correlated. When multicollinearity is present, small changes in the equation can produce dramatic changes in coefficients and p-values. It can also reduce statistical significance in variables that are relevant. For these reasons, multicollinearity makes model selection challenging.
• If you fit many models during the model selection process, you will find variables that appear to be statistically significant, but they are correlated only by chance. This problem occurs because all hypothesis tests have a false discovery rate. This type of data mining can make even random data appear to have significant relationships !
• P-values, adjusted R-squared, predicted R-squared, and Mallows’ Cp can point to different regression equations. Sometimes there is not a clear answer.
• Stepwise regression and best subsets regression can help in the early stages of model specification. However, studies show that these tools can get close to the right answer but they usually don’t specify the correct model .

Practical Recommendations for Model Specification

Regression model specification is as much a science as it is an art. Statistical methods can help choose the best regression model, but ultimately you’ll need to place a high weight on theory and other considerations.

The best practice for model selection in statistics is to review the literature to develop a theoretical understanding of the relevant independent variables, their relationships with the dependent variable, and the expected coefficient signs and effect magnitudes before you begin collecting data. Building your knowledge helps you collect the correct data in the first place and it helps you specify the best regression equation without resorting to data mining. For more information about this process, read 5 Steps for Conducting Scientific Studies with Statistical Analyses .

Deciding on the model should not be based only on model selection statistics. In fact, the foundation of your model selection process should depend largely on theoretical concerns. Be sure to determine whether your statistical results match theory and, if necessary, make adjustments. For example, if theory suggests that an independent variable is important, you might include it in the regression equation even when its p-value is not significant. If a coefficient sign is the opposite of theory, investigate and either modify the model or explain the inconsistency.

Analysts often think that complex problems require complicated regression equations. However, studies reveal that simplification usually produces more precise models*. When you have several models with similar predictive power, choose the simplest because it is the most likely to be the best regression model.

Start simple and then add complexity only when it is actually needed. As you make a model more complex, it becomes more likely that you are tailoring it to fit the quirks in your particular dataset rather than actual relationships in the population . This overfitting reduces generalizability and can produce results that you can’t trust.

To avoid overly complex models, don’t chase a high R-squared mindlessly. Confirm that additional complexity aligns with theory and produces narrower prediction intervals . Check other measures, such as predicted R-squared, which can alert you to overfitting.

In statistics, statisticians say that a simple but effective model is parsimonious. Learn more about Parsimonious Models: Benefits and Selection .

Residual Plots

When you’re deciding on your model, check the residual plots . Residuals plots are an easy way to avoid biased models and can help you make adjustments. For instance, residual plots display patterns when an underspecified regression equation is biased, which can indicate the need to model curvature . The simplest model that creates random residuals is a great contender for being reasonably precise and unbiased.

Ultimately, model selection statistics alone can’t tell you which regression model is best. They just don’t understand the fundamentals of the subject-area. Your expertise is always a vital part of the model specification process! For more help with the regression model selection process, read my post: Five Regression Analysis Tips to Avoid Common Mistakes .

Choosing the best regression model is one issue, while choosing the right type of regression analysis for your data is an entirely different matter.

If you’re learning regression, check out my Regression Tutorial !

Zellner, A. (2001), Keep it sophisticatedly simple. In Keuzenkamp, H. & McAleer, M. Eds. Simplicity, Inference, and Modelling: Keeping it Sophisticatedly Simple . Cambridge University Press, Cambridge.

Note: I wrote a different version of this post that appeared elsewhere. I’ve completely rewritten and updated it for my blog site.

Share this:

Reader Interactions

February 13, 2024 at 6:34 am

Just finished your book on Regression and in there you give a rule of thumb that you should have at least 10 -15 observations for every term in the model (IV, interaction and polynomial) to avoid overfitting the model. My question is when you add a catagorical variable that has a number of levels (say 4) is that still adding a single term or will the number of levels impact how many observations you might need to stay within the rule thumb?

February 13, 2024 at 5:04 pm

Categorical variables count as multiple terms because of the way they’re entered into the model as a series of indicator (dummy) variables. Depending on your software, you might not see that but it’s happening behind the scenes. I talk about that in the book in the section about categorical variables. If your categorical variable has 4 levels, then the software adds 3 indicator variables. Hence, following the rule of thumb, you should include 30-45 observations for that one categorical variable.

However, there’s no exact consensus on the number of observations per term. The need to add additional observations can vary depending on the strength of the expected effect and the base number of observations you plan to have. But this approach gives you a good rough idea for the minimum. And if you read the section about the automated selection procedures, you know it’s better to have more than the minimums (even if your selecting the variables yourself).

It is important to note that categorical variables do use more degrees of freedom than continuous variables. And if you include interaction terms, the problem multiplies . . . literally! Plan accordingly!

December 31, 2023 at 12:11 am

Dear Professor Jim Frost, I would like to cite your post in my publication. But I could find the publication date? Could you share with me the data and year of your post? Thank you.

December 31, 2023 at 12:22 am

Definitely feel free to cite this article! 🙂

When citing online resources, you typically use an “Accessed” date rather than a publication date because online content can change over time. For more information, read Purdue University’s Citing Electronic Resources .

October 21, 2022 at 2:40 am

If the dependent variable is categorical data, how to test and deal with heteroscedasticity?

April 25, 2022 at 1:00 pm

Thanks for sharing! What would you highlight as the most important elements in relation to model specification in customer analytics, and would there be anything of perticular importance when working with binomial logistic regression, multinomial logistic regression and conjoint analysis?

Best regards

April 26, 2022 at 9:43 pm

In this post, I highlight those important elements. Those same elements apply to the other types of regression as well. When you’re creating a model to explain the relationships between the IVs and DV, you need to blend theory and statistical measures as I discuss in this post. These include theory, statistical measures, and residual plots. The all work together to help you find the best model. Subject area knowledge should be your guide in this process because the final model has to make sense in terms of the relationships, their directions, and magnitudes.

The exception is when you’re create a purely predictive model. That’s where you want to use the model to predict an outcome and aren’t interested in explaining the relationships. In this case, you don’t worry about theory but just want to find IVs that both are easy to measure and predict the DV.

March 22, 2022 at 1:02 pm

how do you solve this question: The fitted equation from a study on infant head circumference is as follows:

head circumference = 1.76 + 0.86×gestational age – 2.82×toxemia + 0.046×(gestational age×toxemia)

where gestational age is measured in weeks and toxemia is an indicator variable for the mother’s toxemia status during pregnancy (1=had toxemia).

For infants whose mothers did not have toxemia during pregnancy, what is the effect of an extra two weeks of gestation? What about for those whose mothers had toxemia? What other information or calculations would you need to decide whether to include this effect in the final model? What effect does the last term represent? How would you interpret this effect? Specify the regression models and interpret regression results

March 22, 2022 at 11:02 pm

Hi Richard,

I don’t want to do your homework or test questions. But the answers about how to decide which terms to include in the model are in this post. To calculate the effects of two weeks, read my post about how to interpret regression coefficients and p-values .

As for the last term, it is an interaction term . Click the link to learn about them!

February 14, 2022 at 5:08 am

Hi JIm, Thank you for this wonderful, easy to understand blog. It is very helpful to many people including me who is not a statistics graduate though i need statistics right now for my PhD dissertation. In my dissertation,, I am using multi level modeling, i have a binary dependent variable as well as binary level 1 independent variables. You mentioned above in your blog, that i can check the residuals to check my model specification. My question is: is checking for residuals applicable to logistic regression wherein all my independent variables and dependent variable are binary?

Hope you can find time to answer my question.

Thank you very much. God bless always.

Enrico Mendoza

August 8, 2021 at 2:08 pm

Hi Jim, Thanks for your helpful responses. I have had PD for about 12 years now. When I see my neurologist, I wanted to show him my results and wanted make sure I did the statistical analysis correctly. Are there anything else I can do to lower the ViFs?

August 5, 2021 at 6:04 pm

Thanks a bunch for your response. My strategy was to first demonstrate that latitude was correlated with PD and then find other variables that could correlate with latitude (since latitude obviously doesn’t cause PD). When I entered the IV (no backwards regression), the VIFs were 18 and 15 for Latitude and max length of day…the rest were < 4 for all IVs. After reading about multicollinearity, I figured Latitude and maximum day length (r=0.96) are structural and one of them (Latitude) could be justifiably eliminated. So when I entered all my IVs except Latitude, all VIPs < 5… except “max daylength, VIP = 6).” Is this acceptable? Then I did a backwards regression and “max daylength” was significant, but r(partial)=0.32 did not seem so exciting. The p value for the magnetic field element was 0.07, r(partial)=0.24. When combined I get mult corr coefficient of 0.56. After a bit of research, I found out that vit D deficiency is associated with length of day which is associated with latitude which may affect PD. Could there be something to the magnetic field strength?

August 6, 2021 at 11:52 pm

Hi Michael,

It seems like something associated with latitude is potentially playing a role. If all you wanted to do was predict PD, then you could just leave latitude in as a predictor and use that to generate predictions. However, because want an explanatory model, you really need to, at some point, remove latitude and include that real variables that causally explain the changes in PD.

The best statistical analyses blend subject area expertise with the statistics. And, I’m certainly not an expert in PD. I don’t know if there is an association between magnetic fields and PD. I know the Earth’s magnetic field is fairly weak but I don’t know if it is strong enough affect PD. So, I wouldn’t presume to have any idea whether it’s a promising lead or not. I would strongly suggest conducting background research to see what others have found.

I wonder if hours of daylight, or lack thereof in the winter, might be a factor. There seems to be a connection between depression and developing Parkinson’s. There is a connection between the long dark days further near a pole and seasonal depression. Perhaps a connection? The vitamin D angle is interesting too. I really don’t know though. Again, research and see what the experts have already found.

I think you have a reasonable approach in identifying candidates though. Things related to latitude.

VIFs greater than five are problematic. You’re right on the borderline.

August 4, 2021 at 1:17 pm

I would like to know if I used the best regression model to show that Parkinson’s Disease is associated with Latitude.

I found a database of countries with # of PD Deaths per 100,000 (age standardized). I noticed that countries at higher latitudes seemed to have a higher prevalence of PD. So, I decided to see if there was any association of PD deaths with Latitude. Grouped countries according to Latitude 0 (n=6), 15 (n=25), 30 (n=14), 45 (n=11), and 60 (n=5) degrees. 1. Latitude I tried to think of independent variables that can affect health status for each country: 2. Human Development Index (The index incorporates three dimensions of human development, a long and healthy life, knowledge, and decent living standards.) 3. Diet (% Fruits and Vegetables) Then I added other parameters I thought would be associated with Latitude: 4. Average yearly Temperature 5. Maximum daylight hours 6. Earth’s Magnetic Field (Z-component) I also added Longitude for “good measure”. 7. Longitude I entered the data into a statistical program using Backwards Least Squares Multiple Regression function and only “Latitude” was statistically significant. The multiple correlation coefficient was 0.53 p<0.0001, n = 58. F ratio = 21.7 p<0.0001, Accepted Normality. Did I use the right method to show that PD death rates are associated with latitude independent of HDI, Diet, Temperature, maximum daylight hours, Earth’s magnetic Field and Longitude? I repeated the program using the same independent variables, but using another neurological disease as the dependent variable, i.e. Alzheimer’s/Dementia deaths per 100,000. This time “HDI” was statistically significant. The multiple correlation coefficient was 0.27 p<0.04, n = 58. F ratio = 4.4 p<0.04, Accepted Normality.

Would you conclude that PD is associated with latitude?

Thanks, Mike

August 4, 2021 at 5:04 pm

Those are interesting results! I would say that there’s evidence that there is a correlation between latitude and PD. However, you can’t tell if it’s a casual relationship or just correlation. I’d imagine it’s probably just correlation because I don’t see how the latitude itself could cause Parkinson’s. I would suggest there are some confounding factors involved. You did include some of the potential confounders, which is great. Was there any correlation between the other variables and Latitude? Did you check the VIFs for multicollinearity? It’s possible that with a medium sample size, a relatively large number of variables for the sample size, plus the chance of multicollinearity could reduce the power of the hypothesis tests and produced the insignificant results even though there might be an effect. In some cases, when variables are correlated, backwards elimination will remove one because there’s not enough significance to go around, so to speak. So, it has to pick one basically. It’s possible that latitude rolled up several aspects into one variable and, therefore, looked more significant to the model.

Alternatively, there could be other confounders.

So, a good place to start is to check the correlation between the other IVs and latitude. Check the VIFs too. See my post about multicollinearity for details !

My sense would be that you’re on the right track, thinking about the right things, but that there’s more at play here. Latitude itself can’t really be causing PD. Some physical properties associated latitude or something about the countries that happen to be at those latitudes is probably more of an underlying factor. I always suggest doing background research to see what other researchers have found. They might have identified the confounders/other casual variables. You don’t need to reinvent the wheel!

April 11, 2021 at 12:16 pm

Hello sir, I have a question regarding response variables. Suppose I have 3 response variables, and I would like to choose one to perform my regression analysis. Is there any way to specify, which one I should choose, without creating separate models for each of them.

April 11, 2021 at 4:13 pm

I can think of several possible ways of the top of my head.

You might choose one response variable if you were aware of research in the same subject-area that suggests using a particular response variable. Or, a particular response variable is more relevant for theoretical reasons. Another could be that a particular response variable is easier to measure, easier to interpret, or easier to apply to your particular use case.

In other words, look at what you really want to learn from your study, how you want to use the results, what other studies have done, and then make a decision that includes those factors.

February 21, 2021 at 5:33 am

November 10, 2020 at 8:32 am

Dear Sir My question is that I have a dep variable say X and a variable of interest Y with some control variables(Z) Now when I run following regressions 1) X at time t , Y & Z at t-1 2) X at time t , Y at t-1 & Z at t 3) X at time t , Y & Z at t

The sign of my variable of interest changes(significance too). If there are not any theory to guide me with respect lag specification of variable of interest and control variables, which one from the above model should I use? What is the general principle?

November 2, 2020 at 8:02 am

Got it! Thank you!!! =)

November 1, 2020 at 9:38 pm

can I use regression to check if a change affects product specifications

November 1, 2020 at 10:20 pm

You can probably use regression to predict whether a change affects a product’s characteristics. However, product specifications are imposed by outside limitations. Products outside the spec limits are considered defects. Spec limits are usually devised because a product won’t be satisfactory outside those limits. Typically, you don’t use regression analysis to determine the spec limits. However, I suppose if you knew enough about the product’s use and could model the relevant factors, you might be able to show that changes in the product could affect the spec limits. I’m not familiar with that being done, but I suppose it’s possible.

If you really need to know the answer, I’d check with industry experts. My take is that it would theoretically be possible if you could model the usage well enough but it’s probably not typical.

October 9, 2020 at 8:33 am

Thank You, Sir.

October 6, 2020 at 9:58 am

Model without intercept gives high R^2, so, should I select that model as the best.

October 8, 2020 at 12:21 am

That’s a deceptive property of fitting a model without the intercept. When you fit the model with an intercept, R-squared assesses the variability around the dependent variable’s mean that the model accounts for. However, when you don’t fit an intercept, R-squared assesses the variability around zero. Because they measure different things, you can’t compare them. The R-squared without the intercept is almost always much higher than the R-squared with the intercept because of this property.

By the way, to learn why you should almost always include the intercept in the model, read my post about the y intercept .

July 14, 2020 at 10:48 am

Thnk u for this it’s really helpful My research thesis is Population growth and unemployment rate in…… So how i specify my model

July 14, 2020 at 2:40 pm

Specifying your model is a process that requires a lot research. Follow the approaches I discuss in this article. I think the first, best place to start is by researching how others have specified their models in the same area. Do a literature review to get ideas for the variables to include.

June 20, 2020 at 9:20 am

Please let me know your thoughts. If you have two different models that you ran a regression on in excel what methods in sequence do you look at to determine which mode is better?

Please critique me. What i currently do is first use the backward or forward approach, then observe the p-values for significance, then use the t-stat and the range of higher then 2 or less then -2 as guideline for a good coefficient predictor. Lastly, what should be done to choose the best model when lets say model A has a adjusted R2 higher then model B but model A has at least on insignificant variable while model B does not?

Please help.

June 20, 2020 at 9:19 pm

Hi Martize,

Here would be my suggestions. Keep in mind that all the statistical measures you mention, and even others, can help guide the process. However, you shouldn’t go by statistics alone. Chasing a high R-squared, or even adjusted R-squared, can lead you astray. Consider all the statistics, but then also think about theory and what that suggests. I’d read that section in this post again (near the end). For your case when have several candidate models where the statistics point in different directions, let theory help you choose. If possible, consider what other studies have done as well.

Stepwise regression can help you identify candidate variables, but studies have shown that it usually does not pick the correct model. Read my article about stepwise and best subsets regression for more details.

For adjusted R-squared, any variable that has a t-value greater than an absolute value of 1 will cause the adjusted R-squared to increase. However, variables with t-values near 1/-1 won’t be statistically significant. So, fitting a model by increasing the adjusted R-squared can cause you to include variables that are not significant but do increase adjusted R-squared–as you have found.

If you’re debating over whether to include a variable or not, it’s generally considered better to include an unnecessary variable than risk excluding an important variable. There are caveats. Including too many insignificant variables can reduce the precision of your model. You also need to be sure than you’re not straying into overfitting your model by including extra variables.

I know that doesn’t give you a concrete answer to go by! But, regression modeling is like that sometimes. But, do focus more on the theoretical/other studies side the of the coin to consider along with the statistical side. Go for simplicity when possible. The simplest model that produces good residual plots and is consistent with theory is often a good candidate.

May 29, 2020 at 10:23 am

Okay, I need to calculate a regression equasion for a multiple regression with 3 ind variables. my text gives the equasion of y=b1x1 + b2x2 +b3x3 +b0 +e, but what are the values for x1, x2, x3? to pug in. I thought I knew yesterday and now I have no clue and can’t find any examples that actually show the equasion with the data and plugged in numbers to find out. I have to include the equasion in my assignment report so I need to know what values to include.

One other thing – If one of the variables is not statistically significant, should I repeat the regression without using that data set at all? I know it will change/decrease my value for r-sq (which Is already very low at 11%).

Note I am using exel with the data analysis toolpack because it Is the program required by my instructor,

May 29, 2020 at 2:06 pm

The x-values represent the variables in your dataset that you include in the model. You can either plug in the observed values for an observation to see what the model predicts for that observation or enter new values to predict a new observation with the specified characteristics.

And, yes, as I point out in this post, typically, you at least consider removing variables that are not significant. Also as I point out, don’t chase the highest R-squared. The model with the highest R-squared is not necessarily the best.

May 16, 2020 at 1:19 am

THANK VERY MUCH YOU JIM !!!!

May 13, 2020 at 12:54 am

Dear Jim am Hadas , I was reading your comments and your constructive suggestions by lots of individuals about statistics questions . I was analyzing data using both descriptive statistic and logit model. The result form descriptive one founds the selected variables have influences but the result of logit for most variable are not statistically significance at 95 % ,for p=5 % only 4 form 15 variables found statistically significant. likert type qestion was used to measure level of participation ( 5 leveled ). Does statistically insignificance imply the variables didn’t influence the dependent variables ? what are the problems there? THANK YOU JIM

May 13, 2020 at 3:45 pm

The first thing to recognize is that there might not be a problem at all. Maybe there just is no relationship between the insignificant independent variables and the dependent variable? That is one possibility Check the literature and theory to assess that.

If you have reason to believe there should be significant relationships for the variables in questions, there are several possibilities. Perhaps your sample size is too small to be able to detect the effect? Perhaps you’ve left out a confounding variable or otherwise violating an assumption that is biasing the estimate to be not significant?

On the other hand, if you have descriptive statistics display an apparent effect, but the variable is not significant in your model, there are several possibilities for that case. Your descriptive statistics do not account for sampling error. You can have visible effects that might be caused by random error rather than by an effect that exists in the population. Hypothesis testing accounts for that possibility. Additionally, when you look at the descriptive statistics, they do not account for (i.e., control for) other variables. However, when you fit a regression model, the procedure controls for the other variables in the model. After controlling for the effect of other variables in the model, what looked like strong results in the descriptive statistic might not actually exist.

Technically, a variable that is not significant indicates that you have insufficient evidence to conclude there is an effect. It is NOT proof that an effect doesn’t exist. For more information about that, read my post about failing to reject the null hypothesis .

There’s a range of potential questions for you to look into!

May 11, 2020 at 1:36 pm

Thank you for such a helpful article!

In our study, we have 3 independent variables and one dependent variable. For all the variables we are using an already developed scale which has around 5-9 questions each and uses the Likert scale for answers. We just wanted to know if we have followed the right steps and wanted your guidance on the same. First, we took the sum of each participants response on every questionnaire. For example, the questionnaire of work autonomy (which is one of our variable) had 5 questions and a participant answered 2, 3, 2, 3, 4 respectively for all 5 questions. Then, we took the mean as 14 as the mean response of the participant on the questionnaire. This mean was calculated for all the respondents, for the all the questionnaires/variables. Then, we used multiple regression analysis to study the effect of the 3 independent variables on the dependent variable. Could you please let us know if we are on the right track and if we have used the correct analysis? Should we use ordinal regression instead?

Thank you so much!

May 12, 2020 at 3:39 pm

Yes, that sounds like a good approach. When you take the average or sum of a Likert scale variable like you are, you can often treat it as a continuous variable.

One potential problem is that as you change values in Likert scales by going from 2 to 3 to 4, etc., you don’t know for sure whether those represent a fixed increases. It’s like when you compare the times of a first place, second place, and third place in a race, they’re not necessarily increasing at a fixed rate. That’s the nature of ordinal variables. You might need to fit curvature, etc. But, if you can fit a model where the residuals look good and the results make theoretical sense, then I think you’ve got a good model!

Best of luck with your analysis!

April 6, 2020 at 6:04 pm

How would I specify a regression model consisting of both continuous and categorical regressors? And how to interpret the output of that model?

April 6, 2020 at 2:33 am

Hi Jim, Thank you for your excellent and intuitive explanations. I’m a graduated student and recently I’m trying to find Interactive relationships between two genes by add their interaction terms in the regression models. I have some questions about choosing the best regression model. The DVs can be affected by several IVs (B1,B2,…,Bn), and my aim is to find which Bn may be regulated by another IV (A). I have built three models to deal with that, but the results are so different. Model 1: DV=A+Bn+A*Bn I only input one pair IVs(A and Bn) in model each time, and then repeat this model n times. When Bn is B1(DV=A+B1+A*B1), all of the terms are significant. —————————————————————- Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -1.732e+03 3.987e+02 -4.343 5.72e-05 *** A 2.658e+01 8.261e+00 3.217 0.00212 ** B1 6.576e+00 2.140e+00 3.073 0.00323 ** A*B1 -8.390e-02 2.889e-02 -2.904 0.00521 ** — Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 1065 on 58 degrees of freedom Multiple R-squared: 0.2037, Adjusted R-squared: 0.1625 F-statistic: 4.945 on 3 and 58 DF, p-value: 0.003994 ————————————————————— Model 2: DV=A+B1+B2+…+Bn+A*Bn To avoid biased results, as you suggested, I add all the IVs that may affect DV. But only one target interaction term is remained. Then repeat this model n times. When interaction term is A*B1, the interaction effect is insignificant. —————————————————————- Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -2.124e+03 2.815e+02 -7.546 7.49e-10 *** A 1.516e+01 5.994e+00 2.530 0.01454 * B1 2.056e+00 1.810e+00 1.136 0.26145 B2 3.657e+00 2.402e+00 1.523 0.13404 B3 6.188e-01 4.108e-01 1.506 0.13822 B4 4.790e-01 3.337e-01 1.435 0.15734 B5 -4.909e-01 1.355e+00 -0.362 0.71871 B6 1.485e+00 6.239e-01 2.381 0.02104 * B7 1.600e+01 5.756e+00 2.780 0.00759 ** B8 2.062e-02 1.827e-02 1.129 0.26433 A*B1 -3.465e-02 2.225e-02 -1.558 0.12551 — Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 674.5 on 51 degrees of freedom Multiple R-squared: 0.7194, Adjusted R-squared: 0.6643 F-statistic: 13.07 on 10 and 51 DF, p-value: 6.148e-11 —————————————————————–

Model 3: DV=A+B1+A*B1+B2+A*B2…+Bn+A*Bn In this model, I add all the IVs(Bn) and their interaction terms with A simultaneously, thus model runs once. In this situation, no significant terms. —————————————————————— Coefficients: Estimate Std. Error t value Pr(>|t|) (Intercept) -2.314e+03 3.984e+02 -5.809 6.45e-07 *** A 2.410e+01 1.277e+01 1.886 0.0658 . B1 5.936e-01 2.170e+00 0.274 0.7857 B2 5.281e+00 6.525e+00 0.809 0.4226 B3 4.074e-01 1.238e+00 0.329 0.7436 B4 4.417e-01 1.202e+00 0.368 0.7150 B5 -4.153e-01 3.814e+00 -0.109 0.9138 B6 2.775e+00 1.777e+00 1.562 0.1255 B7 9.274e+00 1.136e+01 0.816 0.4187 B8 4.297e-02 4.573e-02 0.940 0.3524 A*B1 -1.749e-02 3.531e-02 -0.495 0.6228 A*B2 -8.492e-02 1.707e-01 -0.498 0.6212 A*B3 6.077e-03 2.901e-02 0.209 0.8350 A*B4 1.723e-03 2.737e-02 0.063 0.9501 A*B5 4.894e-02 1.136e-01 0.431 0.6688 A*B6 -5.186e-02 5.362e-02 -0.967 0.3388 A*B7 3.067e-01 5.010e-01 0.612 0.5436 A*B8 -4.106e-04 8.732e-04 -0.470 0.6405 — Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1 Residual standard error: 686 on 44 degrees of freedom Multiple R-squared: 0.7496, Adjusted R-squared: 0.6528 F-statistic: 7.747 on 17 and 44 DF, p-value: 2.326e-08 ——————————————————————– My question: Is the significant interaction effect between A and B1 in model 1 reliable? Which is the best model to find the Interactive relationship between A and Bn? In addition, the IVs above are not centered, as I get same results for interaction terms and the less significant main effect sometimes after centering.

March 5, 2020 at 6:03 pm

Thank you very much for your help and support

March 5, 2020 at 4:28 am

Hey Jim, thanks for your insightful post. Please, are there any steps or factors that best determine whether a data analyst should build one comprehensive model or simply put should build many models on partitions of the data.

February 8, 2020 at 7:07 am

Thank you for your useful content. Is that mean we should use same control variables from previous literature or we can use the most suitable variables after running some experiments.

February 8, 2020 at 3:23 pm

Theory and the scientific literature should guide you when possible. If other studies find that particular variables are important, you should consider them for your study. Because of omitted variable bias , it can be risk in terms bias to not include variables that other studies have found to be important. That is particularly true if you’re performing an observation study rather than a randomized study. However, you can certainly add your own variables into the mix if you’re testing new theories and/or have access to new types of data.

So, be very careful when removing control variables that have been identified as being important. You should have, and be able to explain, good reasons for removing them. Feel freer when it comes to adding new variables.

January 24, 2020 at 3:13 am

what should we do if the output variable is more skewed.skewness>4

January 24, 2020 at 11:55 am

Hi Sravani,

When the output/dependent variable is skewed, it can be more difficult to satisfy the OLS assumptions . Note that the OLS assumptions don’t state that the dependent variable must be normally distributed itself, but instead state that the residuals should be normally distributed. And, obtaining normally distributed residuals can be more difficult when the DV is skewed.

There are several things you can try.

Sometimes modeling the curvature, if it exists, will help. In my post about using regression to make predictions , I use BMI to predict body fat percentage. Body fat percentage is the DV and it is skewed. However, the relationship between BMI and BF% is curved and by modeling that curvature, the residuals are normally distributed.

As the skew worsens, it becomes harder to get good residuals . You might need to transform you DV. I don’t have a blog post about that but I include a lot of information about data transformations in my regression ebook .

Those are several things that I’d look into first.

November 21, 2019 at 11:52 am

Hi Jim, What does it mean when a regression model has a negative prediction R2 while the R2 and adjusted R2 are positive and reasonable?

November 22, 2019 at 12:56 am

Any time the predicted R-squared is notably less than the adjusted/regular r-squared values it means that the model doesn’t predict new observations as well as it explains observations that the were used in the model fitting process. Often this indicates you’re overfitting the model. Too many predictors given the size of dataset. Usually when it’s so bad as to be negative, it’s because the dataset is pretty small. Read my posts about adjusted and predicted R-squared and overfitting for more information.

While the regular R-squared ranges between 0 – 100%, both predicted and adjusted R-squared can have negative values. A negative value doesn’t have any special interpretation other than just being really bad. Some statistical software will round negative values to zero. I tend to see negative values for predicted R-squared more than adjusted R-squared. As you’ll in the post I recommend, it’s often the more sensitive measure to problems with the model.

Take the negative predicted R-squared seriously. You’re probably overfitting your model. I’d also bet that you have fairly small dataset.

November 3, 2019 at 8:17 am

Currently Im doing a research in my Economics Degree. This has been very helpful. I do have some doubts though. My research topic is “Relationship between Inflation and Economic growth in Maldives and how it affects the Maldivian economy”.

For this topic, I’m using GDP as a dependent variable and inflation, unemployment and gdp per capita as independent variables. I want to know whether it’s right to use all of these variables in one equation for this topic? Once i figure that out, it would be easy to run the regression.

Hope you could help me out in this.

Thanks Maryam

November 2, 2019 at 8:30 am

Very useful write up. Thanks Jim Please where a number of empirical models related similar independent variables to a particular dependent variable, what are the usual justifications for opting for a particular empirical model that one intends to build his research on?

November 4, 2019 at 9:08 am

I’d focus on using theory and the literature to guide you. Statistical measures can also provide information. I describe the process that you should use in this blog post.

September 23, 2019 at 3:37 pm

Am truly grateful for this beautiful blog, it truly is assisting me in my dissertation!

So I needed help with what model to use having a binary DV ( poverty). I run different types of logistic regression on my dataset depending on what type of post estimations tests I was carrying out.

As I was testing for goodness of fit that’s estat gof and linktest, of course after running a logistic regression, my prob>chi was equivalent to 0.0000 rejecting the Ho hypothesis which states that the model fits if prob>chi is > 0.0000.

I tried adding more independent variable but to no avail. I have 3 categorical independent variables that are insignificant, 1 continuous independent variable that was insignificant. The other 6 continuous independent variables are significant.

What do I do about those two tests, I seriously need help.

Thanks in advance.

Regards Twiza.

September 24, 2019 at 9:48 pm

Because you have a binary DV, you need to use binary logistic regression. However, it’s impossible for me to determine why your model isn’t fitting. Some suggestions would be to try to fit interaction terms and use polynomials terms. Just like would for an least squares model. Another possibility is to try different link functions.

July 4, 2019 at 2:47 pm

Hi Jim I read your post thoroughly. I still have some doubts. I’m doing multi regression which includes 9 predictor variables. I’ve used p-values to check which of my variables are important. Also i plotted the graph for each independent variable wrt dependent variable and noted the each variable has a polynomial relation at individual level. So how to do multi variate polynomial regression when? Can you please help me with this? Thanks in advance

July 4, 2019 at 2:59 pm

Hi Jagriti,

It’s great that you graphed the data like that. It’s such an important step, but so many people skip it!

It sounds like you just need to add the polynomial terms to your model. I write more about this my post about fitting curves , which explains that process. After you fit the curvature, be sure to check the residual plots to make sure that you didn’t miss anything!

May 2, 2019 at 10:19 am

Hi Jim thanks for your blog. My problem is much simpler than a multiple regression: I have some data showing a curved trend, and I would like to select the best polynomial model (1st, 2nd, 3rd or 4th order polynomial) fitting these data. The ‘best’ model should have a good fit but should also be more simple as possible (the lowest order polynomial producing a good fitting…) Someone suggetsed me the Akaike Information Criterion, that penalizes the complexity of the model. Which are the possible tests or approaches to this (apparently) simple problem? Thank you in advance! Henry Lee

May 2, 2019 at 10:56 am

I definitely agree with your approach about using the simplest model that fits the data adequately.

I write about using polynomials to fit curvature in my post about curve fitting with regression . In practice, I find that 3rd order and higher polynomials are very rare. I’d suggest starting by graphing your data and counting the bends that you see and use the corresponding polynomial, as I describe in the curve fitting post. You should also apply theory, particularly if you’re using 3rd order or higher. Does theory supporting modeling those extra bends in the data or are they likely the product of a fluky sample or a small data set.

As for statistical tests, p-values are good place to start. If a polynomial term is not significant, consider removing it. I also suggest using adjusted R-squared because you’re comparing models with different numbers of terms. Perhaps even more crucial is using predicted R-squared. That statistic helps prevent you from overfitting your model. As you increase the polynomial order, you might just be playing connect the dots and fitting the noise in your data rather than fitting the real relationships. I’ve written a post about adjusted R-squared and predicted R-squared that you should read. I even include an example where it appears like a 3rd order polynomial provides a good fit but predicted R-squared indicates that you’re overfitting the data.

Finally, be sure to assess the residual plots because they’ll show you if you’re not adequately modeling the curvature.

April 26, 2019 at 6:00 pm

I’m doing multiple regression analysis and there are four independent variables for regression analysis. So, it is not possible to know the shape of a graph that indicates the relationship between DB and IV. In this, how can I know the best regression model for my data? for example, linear, quadratic or exponential.

April 26, 2019 at 9:13 pm

I’ve written a blog post about fitting the curvature in your data . That post will answer your questions! Also, consider graphing your residuals by each IV to see if you need to fit curve for each variable. I talk about these methods in even more detail in my ebook about regression . You might check that out!

Best of luck with fitting your model!

March 20, 2019 at 2:59 am

Hey Sir, My question might not be related, but I”m much confused in some problems.Like, When we study human behavior We used some demographic variable like Age and sex of child.Why we used them and what are the rational behind this.And how to interpret them. Thanks.

March 20, 2019 at 10:35 am

You’d use these demographic variables because you think that they correlate with your DV. For instance, understanding age and gender might help you understand changes in the DV. For example, your DV might increase for males compared to females or increase with age. These variables can provide important information in their own right. Additionally, if you don’t include these variables and they are important, you risk biasing the estimates for your other variables. See omitted variable bias for details!

If you include these demographic variables in your model and they are not statistically significant, you can consider removing them from the model.

You interpret this type of variable in the same manner as any other independent variable. See regression coefficients and p-values for details.

February 21, 2019 at 11:45 pm

Thank you for your help Jim.

February 6, 2019 at 12:33 pm

I’m doing a multiple regression analysis on time series data. Can you recommend me some models that I can use for my analysis?

February 6, 2019 at 2:59 pm

Hi Karishma,

You can use OLS regression to analyze time series data. Generally, you’ll need to include lagged variables and other time related predictors. Importantly, you can include predictors that are important to your study, which allows the analysis to estimate effects for them. You can use the model to make predictions. Be sure to pay particular attention to your residuals by order plot and the Durbin-Watson statistic to be sure that your model fits the data.

You can also use ARIMA, which is a regression-like approach to time series data. It includes multiple times series methods in one model (autoregressive, differencing, and moving average components). You can use relatively sophisticated correlational methods to uncover otherwise hidden patterns. You can use the model to make predictions. However, while models the dependent variable, it does not allow you to add other predictors into the model.

There are simpler time series models available, but they are less like regression, so I won’t detail them here.

Unfortunately, I don’t have much experience using regression analyses with time series data. There are undoubtedly other options available.

I hope this helps!

January 20, 2019 at 11:33 am

Thanks for this really helpful blog! I am wondering whether I can use AIC and BIC to help me see which model fits my data best. Or is AIC and BIC only applicable when comparing the same model with different sets of variables (i.e. it tells me which variable selection is the best?). So could I use AIC and BIC to tell me whether a poisson or a negative binomial regression is best? And could I also compare OLS with count data models?

Any advice is much appreciated!

September 19, 2018 at 9:30 am

So in 2015, a fairly similar article was posted on another website. Care to at least give that one as a source?

September 19, 2018 at 11:52 pm

Yes, I wrote both articles. I’ve been adding notes to that effect in several places and will need to add one to this post.

For some reason, the organization removed most author’s names from the articles. If you use the Internet Archive Wayback Machine and view an older version of that article, you’ll see that I am the author.

Thanks for writing!

Comments and Questions Cancel reply

Exponential and Logarithmic Models

Choose an appropriate model for data.

Now that we have discussed various mathematical models, we need to learn how to choose the appropriate model for the raw data we have. Many factors influence the choice of a mathematical model, among which are experience, scientific laws, and patterns in the data itself. Not all data can be described by elementary functions. Sometimes, a function is chosen that approximates the data over a given interval. For instance, suppose data were gathered on the number of homes bought in the United States from the years 1960 to 2013. After plotting these data in a scatter plot, we notice that the shape of the data from the years 2000 to 2013 follow a logarithmic curve. We could restrict the interval from 2000 to 2010, apply regression analysis using a logarithmic model, and use it to predict the number of home buyers for the year 2015.

Three kinds of functions that are often useful in mathematical models are linear functions, exponential functions, and logarithmic functions. If the data lies on a straight line, or seems to lie approximately along a straight line, a linear model may be best. If the data is non-linear, we often consider an exponential or logarithmic model, though other models, such as quadratic models, may also be considered.

In choosing between an exponential model and a logarithmic model, we look at the way the data curves. This is called the concavity. If we draw a line between two data points, and all (or most) of the data between those two points lies above that line, we say the curve is concave down. We can think of it as a bowl that bends downward and therefore cannot hold water. If all (or most) of the data between those two points lies below the line, we say the curve is concave up. In this case, we can think of a bowl that bends upward and can therefore hold water. An exponential curve, whether rising or falling, whether representing growth or decay, is always concave up away from its horizontal asymptote. A logarithmic curve is always concave away from its vertical asymptote. In the case of positive data, which is the most common case, an exponential curve is always concave up, and a logarithmic curve always concave down.

A logistic curve changes concavity. It starts out concave up and then changes to concave down beyond a certain point, called a point of inflection.

After using the graph to help us choose a type of function to use as a model, we substitute points, and solve to find the parameters. We reduce round-off error by choosing points as far apart as possible.

Example 6: Choosing a Mathematical Model

Does a linear, exponential, logarithmic, or logistic model best fit the values listed below? Find the model, and use a graph to check your choice.

First, plot the data on a graph as in Figure 8. For the purpose of graphing, round the data to two significant digits.

Clearly, the points do not lie on a straight line, so we reject a linear model. If we draw a line between any two of the points, most or all of the points between those two points lie above the line, so the graph is concave down, suggesting a logarithmic model. We can try [latex]y=a\mathrm{ln}\left(bx\right)[/latex]. Plugging in the first point, [latex]\left(\text{1,0}\right)[/latex], gives [latex]0=a\mathrm{ln}b[/latex]. We reject the case that a  = 0 (if it were, all outputs would be 0), so we know [latex]\mathrm{ln}\left(b\right)=0[/latex]. Thus b  = 1 and [latex]y=a\mathrm{ln}\left(\text{x}\right)[/latex]. Next we can use the point [latex]\left(\text{9,4}\text{.394}\right)[/latex] to solve for a :

Because [latex]a=\frac{4.394}{\mathrm{ln}\left(9\right)}\approx 2[/latex], an appropriate model for the data is [latex]y=2\mathrm{ln}\left(x\right)[/latex].

To check the accuracy of the model, we graph the function together with the given points.

Figure 9.  The graph of [latex]y=2\mathrm{ln}x[/latex].

We can conclude that the model is a good fit to the data.

Compare the figure above to the graph of [latex]y=\mathrm{ln}\left({x}^{2}\right)[/latex] shown in Figure 10.

Figure 10.  The graph of [latex]y=\mathrm{ln}\left({x}^{2}\right)[/latex]

The graphs appear to be identical when x  > 0. A quick check confirms this conclusion: [latex]y=\mathrm{ln}\left({x}^{2}\right)=2\mathrm{ln}\left(x\right)[/latex] for x  > 0.

However, if x  < 0, the graph of [latex]y=\mathrm{ln}\left({x}^{2}\right)[/latex] includes a “extra” branch, as shown below. This occurs because, while [latex]y=2\mathrm{ln}\left(x\right)[/latex] cannot have negative values in the domain (as such values would force the argument to be negative), the function [latex]y=\mathrm{ln}\left({x}^{2}\right)[/latex] can have negative domain values.

Does a linear, exponential, or logarithmic model best fit the data in the table below? Find the model.

Expressing an Exponential Model in Base e

While powers and logarithms of any base can be used in modeling, the two most common bases are [latex]10[/latex] and [latex]e[/latex]. In science and mathematics, the base e  is often preferred. We can use laws of exponents and laws of logarithms to change any base to base e .

How To: Given a model with the form [latex]y=a{b}^{x}[/latex], change it to the form [latex]y={A}_{0}{e}^{kx}[/latex].

• Rewrite [latex]y=a{b}^{x}[/latex] as [latex]y=a{e}^{\mathrm{ln}\left({b}^{x}\right)}[/latex].
• Use the power rule of logarithms to rewrite y as [latex]y=a{e}^{x\mathrm{ln}\left(b\right)}=a{e}^{\mathrm{ln}\left(b\right)x}[/latex].
• Note that [latex]a={A}_{0}[/latex] and [latex]k=\mathrm{ln}\left(b\right)[/latex] in the equation [latex]y={A}_{0}{e}^{kx}[/latex].

Example 7: Changing to base e

Change the function [latex]y=2.5{\left(3.1\right)}^{x}[/latex] so that this same function is written in the form [latex]y={A}_{0}{e}^{kx}[/latex].

The formula is derived as follows

Change the function [latex]y=3{\left(0.5\right)}^{x}[/latex] to one having e  as the base.

• Precalculus. Authored by : Jay Abramson, et al.. Provided by : OpenStax. Located at : http://cnx.org/contents/[email protected] . License : CC BY: Attribution . License Terms : Download For Free at : http://cnx.org/contents/[email protected].

An open portfolio of interoperable, industry leading products

The Dotmatics digital science platform provides the first true end-to-end solution for scientific R&D, combining an enterprise data platform with the most widely used applications for data analysis, biologics, flow cytometry, chemicals innovation, and more.

Statistical analysis and graphing software for scientists

Bioinformatics, cloning, and antibody discovery software

Plan, visualize, & document core molecular biology procedures

Electronic Lab Notebook to organize, search and share data

Proteomics software for analysis of mass spec data

Modern cytometry analysis platform

Analysis, statistics, graphing and reporting of flow cytometry data

Software to optimize designs of clinical trials

POPULAR USE CASES

The Ultimate Guide to Linear Regression

Get all your linear regression questions answered here

Welcome! When most people think of statistical models, their first thought is linear regression models. What most people don’t realize is that linear regression is a specific type of regression.

With that in mind, we’ll start with an overview of regression models as a whole. Then after we understand the purpose, we’ll focus on the linear part, including why it’s so popular and how to calculate regression lines-of-best-fit! (Or, if you already understand regression, you can skip straight down to the linear part) .

This guide will help you run and understand the intuition behind linear regression models. It’s intended to be a refresher resource for scientists and researchers, as well as to help new students gain better intuition about this useful modeling tool.

What is regression?

In its simplest form, regression is a type of model that uses one or more variables to estimate the actual values of another. There are plenty of different kinds of regression models, including the most commonly used linear regression, but they all have the basics in common. 

Usually the researcher has a response variable they are interested in predicting, and an idea of one or more predictor variables that could help in making an educated guess. Some simple examples include:

• Predicting the progression of a disease such as diabetes using predictors such as age, cholesterol, etc. (linear regression)
• Predicting survival rates or time-to-failure based on explanatory variables (survival analysis) 
• Predicting political affiliation based on a person’s income level and years of education (logistic regression or some other classifier)
• Predicting drug inhibition concentration at various dosages (nonlinear regression)

There are all sorts of applications, but the point is this: If we have a dataset of observations that links those variables together for each item in the dataset, we can regress the response on the predictors. Furthermore:

Fitting a model to your data can tell you how one variable increases or decreases as the value of another variable changes.

For example, if we have a dataset of houses that includes both their size and selling price, a regression model can help quantify the relationship between the two. ( Not that any model will be perfect for this !)

The most noticeable aspect of a regression model is the equation it produces. This model equation gives a line of best fit, which can be used to produce estimates of a response variable based on any value of the predictors ( within reason ). We call the output of the model a point estimate because it is a point on the continuum of possibilities. Of course, how good that prediction actually depends on everything from the accuracy of the data you’re putting in the model to how hard the question is in the first place.

Compare this to other methods like correlation, which can tell you the strength of the relationship between the variables, but is not helpful in estimating point estimates of the actual values for the response.

What is the difference between the variables in regression?

There are two different kinds of variables in regression: The one which helps predict (predictors), and the one you’re trying to predict (response).

Predictors were historically called independent variables in science textbooks. You may also see them referred to as x-variables, regressors, inputs, or covariates. Depending on the type of regression model you can have multiple predictor variables, which is called multiple regression . Predictors can be either continuous (numerical values such as height and weight) or categorical (levels of categories such as truck/SUV/motorcycle).

The response variable is often explained in layman’s terms as “the thing you actually want to predict or know more about”. It is usually the focus of the study and can be referred to as the dependent variable, y-variable, outcome, or target. In general, the response variable has a single value for each observation (e.g., predicting the temperature based on some other variables), but there can be multiple values (e.g., predicting the location of an object in latitude and longitude). The latter case is called multivariate regression (not to be confused with multiple regression). 

What are the purposes of regression analysis?

Regression Analysis has two main purposes:

• Explanatory - A regression analysis explains the relationship between the response and predictor variables. For example, it can answer questions such as, does kidney function increase the severity of symptoms in some particular disease process? 
• Predictive - A regression model can give a point estimate of the response variable based on the value of the predictors. 

How do I know which model best fits the data?

The most common way of determining the best model is by choosing the one that minimizes the squared difference between the actual values and the model’s estimated values. This is called least squares. Note that “least squares regression” is often used as a moniker for linear regression even though least squares is used for linear as well as nonlinear and other types of regression.

What is linear regression?

The most popular form of regression is linear regression, which is used to predict the value of one numeric (continuous) response variable based on one or more predictor variables (continuous or categorical).

Most people think the name “linear regression” comes from a straight line relationship between the variables. For most cases, that’s a fine way to think of it intuitively: As a predictor variable increases, the response either increases or decreases at the same rate (all other things equal). If this relationship holds the same for any values of the variables, a straight line pattern will form in the data when graphed, as in the example below:

However, the actual reason that it’s called linear regression is technical and has enough subtlety that it often causes confusion. For example, the graph below is linear regression, too, even though the resulting line is curved. The definition is mathematical and has to do with how the predictor variables relate to the response variable. Suffice it to say that linear regression handles most simple relationships, but can’t do complicated mathematical operations such as raising one predictor variable to the power of another predictor variable.

The most common linear regression models use the ordinary least squares algorithm to pick the parameters in the model and form the best line possible to show the relationship (the line-of-best-fit). Though it’s an algorithm shared by many models, linear regression is by far the most common application. If someone is discussing least-squares regression, it is more likely than not that they are talking about linear regression.

What are the major advantages of linear regression analysis?

Linear regression models are known for being easy to interpret thanks to the applications of the model equation, both for understanding the underlying relationship and in applying the model to predictions. The fact that regression analysis is great for explanatory analysis and often good enough for prediction is rare among modeling techniques.

In contrast, most techniques do one or the other. For example, a well-tuned AI-based artificial neural network model may be great at prediction but is a “black box” that offers little to no interpretability. 

There are some other benefits too:

• Linear regression is computationally fast, particularly if you’re using statistical software. Though it’s not always a simple task to do by hand, it’s still much faster than the days it would take to calculate many other models.
• The popularity of regression models is itself an advantage. The fact that it is a tried and tested approach used by so many scientists makes for easy collaboration.

Assumptions of linear regression

Just because scientists' initial reaction is usually to try a linear regression model, that doesn't mean it is always the right choice. In fact, there are some underlying assumptions that, if ignored, could invalidate the model.

• Random sample - The observations in your data need to be independent from one another. There are many ways that dependence occurs, for example, one common way is with multiple response data, where a single subject is measured multiple times. The measurements on the same individual are presumably correlated, and you couldn’t use linear regression in this case.
• Independence between predictors - If you have multiple predictors in your model, in theory, they shouldn’t be correlated with one another. If they are, this can cause instability in your model fit, although this affects the interpretation of your model rather than the predictions. See more about multicollinearity here .
• Homoscedasticity - Meaning ‘equal scatter,’ this says that your residuals (the difference between the model prediction and the observed values) should be just as variable anywhere along the continuum. This is assessed with residual plots.
• Residuals are normally distributed - In addition to having equal scatter, in the standard linear regression model, the residuals are assumed to come from a normal distribution. This is commonly assessed using a QQ-plot.
• Linear relationship between predictors and response - The relationships must be linear as described above , ruling out some more complicated mathematical relationships. You can model some “curves” in your data using, say, variable X and variable X^2 ("X squared") as predictors.
• No uncertainty in predictor measurements - The model assumes that all of the uncertainty is in the response variable. This is the most nuanced assumption: Even if you’re attempting to make inferences about a model with predictors that are themselves estimates, this would not affect you unless you need to attribute the uncertainty to the predictors. This field of study is called “measurement error.”

Other things to keep in mind for valid inference:

• Representative sample - The dataset you are going to use should be a representative (and random!) sample of the population you’re trying to make inferences about. To use an intuitive example, you should not expect all people to act the same as those in your household. Since we often underestimate our own bias, the best bet is to have a random sample when you start.
• Sample size - If your dataset only has 5 observations in it, the model will be less effective at finding a real pattern (or if one exists) than if it has 100. There is no one-size-fits-all number for every study, but generally 30 or more is considered the low end of what regression needs.
• Stay in range - Don’t try to make predictions outside the range of the dataset you used to build the model. For example, let’s say you are predicting home values based on square footage. If your dataset only has homes between 1,000 and 3,000 square feet, the model may not be a good judge of the value of an 800 or 4,000 square-foot house. This is called extrapolating, and is not recommended.

Types of linear regression

The two most common types of regression are simple linear regression and multiple linear regression, which only differ by the number of predictors in the model. Simple linear regression has a single predictor. 

Simple linear regression

It’s called simple for a reason: If you are testing a linear relationship between exactly two continuous variables (one predictor and one response variable), you’re looking for a simple linear regression model, also called a least squares regression line. Are you looking to use more predictors than that? Try a multiple linear regression model. That is the main difference between the two, but there are other considerations and differences involved too.

You can use statistical software such as Prism to calculate simple linear regression coefficients and graph the regression line it produces. For a quick simple linear regression analysis, try our free online linear regression calculator .

Interpreting a simple linear regression model

Remember the y = mx+b formula for a line from grade school? The slope was m , and the y-intercept was b , and both were necessary to draw a line. That’s what you’re basically building here too, but most textbooks and programs will write out the predictive equation for regression this way:

Y is your response variable, and X is your predictor. The two 𝛽 symbols are called “parameters”, the things the model will estimate to create your line of best fit. The first (not connected to X) is the intercept, the other (the coefficient in front of X) is called the slope term.

As an example, we will use a sample Prism dataset with diabetes data to model the relationship between a person’s glucose level (predictor) and their glycosylated hemoglobin level (response). Once we run the analysis we get this output:

Best-fit parameters and the regression equation

The first section in the Prism output for simple linear regression is all about the workings of the model itself. They can be called parameters, estimates, or (as they are above) best-fit values. Keep in mind, parameter estimates could be positive or negative in regression depending on the relationship.

There you see the slope (for glucose) and the y-intercept. The values for those help us build the equation the model uses to estimate and make predictions:

Glycosylated Hemoglobin = 2.24 + (0.0312*Glucose)

Notice: That same equation is given later in the output, near the bottom of the page.

Using this equation, we can plug in any number in the range of our dataset for glucose and estimate that person’s glycosylated hemoglobin level. For instance, a glucose level of 90 corresponds to an estimate of 5.048 for that person’s glycosylated hemoglobin level. But that’s just the start of how these parameters are used.

Interpreting parameter estimates

You can also interpret the parameters of simple linear regression on their own, and because there are only two it is pretty straightforward.

The slope parameter is often the most helpful: It means that for every 1 unit increase in glucose, the estimated glycosylated hemoglobin level will increase by 0.0312 units. As an aside, if it was negative (perhaps -0.04), we would say a 1 unit increase in glucose would actually decrease the estimated response by -0.04.

The intercept parameter is useful for fitting the model, because it shifts the best-fit-line up or down. In this example, the value it shows (2.24) is the predicted glycosylated hemoglobin level for a person with a glucose level of 0. In cases like this, the interpretation of the intercept isn’t very interesting or helpful.

Simply put, if there’s no predictor with a value of 0 in the dataset, you should ignore this part of the interpretation and consider the model as a whole and the slope. However, notice that if you plug in 0 for a person’s glucose, 2.24 is exactly what the full model estimates. 

Confidence intervals and standard error

The next couple sections seem technical, but really get back to the core of how no model is perfect. We can give “point estimates” for the best-fit parameters today, but there’s still some uncertainty involved in trying to find the true and exact relationship between the variables. 

Standard error and confidence intervals work together to give an estimate of that uncertainty. Add and subtract the standard error from the estimate to get a fair range of possible values for that true relationship. With this 95% confidence interval, you can say you believe the true value of that parameter is somewhere between the two endpoints (for the slope of glucose, somewhere between 0.0285 and 0.0340).

This method may seem too cautious at first, but is simply giving a range of real possibilities around the point estimate. After all, wouldn’t you like to know if the point estimate you gave was wildly variable? This gives you that missing piece. 

Goodness of fit

Determining how well your model fits can be done graphically and numerically. If you know what to look for, there’s nothing better than plotting your data to assess the fit and how well your data meet the assumptions of the model. These diagnostic graphics plot the residuals, which are the differences between the estimated model and the observed data points.

A good plot to use is a residual plot versus the predictor (X) variable. Here you want to look for equal scatter, meaning the points all vary roughly the same above and below the dotted line across all x values. The plot on the left looks great, whereas the plot on the right shows a clear parabolic shaped trend, which would need to be addressed.

Another way to assess the goodness of fit is with the R-squared statistic, which is the proportion of the variance in the response that is explained by the model. In this case, the value of 0.561 says that 56% of the variance in glycosylated hemoglobin can be explained by this very simple model equation (effectively, that person’s glucose level).

The name R-squared may remind you of a similar statistic: Pearson’s R, which measures the correlation between any two variables. Fun fact: As long as you’re doing simple linear regression, the square-root of R-squared (which is to say, R), is equivalent to the Pearson’s R correlation between the predictor and response variable.

The reason is that simple linear regression draws on the same mechanisms of least-squares that Pearson’s R does for correlation. Keep in mind, while regression and correlation are similar they are not the same thing . The differences usually come down to the purpose of the analysis, as correlation does not fit a line through the data points.

Significance and F-tests

So we have a model, and we know how to use it for predictions. We know R-squared gives an idea of how well the model fits the data… but how do we know if there is actually a significant relationship between the variables? 

A section at the bottom asks that same question: Is the slope significantly non-zero? This is especially important for this model, where the best-fit value (roughly 0.03) seems very close to 0 to the naked eye. How can we feel confident one way or another?

In this case, the slope  is  significantly non-zero: An F-test gives a p-value of less than 0.0001. F-tests answer this for the model as a whole rather than its individual slopes, but in this case there is only one slope anyway. P-values are always interpreted in comparison to a “significance threshold”: If it’s less than the threshold level, the model is said to show a trend that is significantly different from “no relationship” (or, the null hypothesis). And based on how we set up the regression analysis to use 0.05 as the threshold for significance, it tells us that the model points to a significant relationship. There is evidence that this relationship is real.

If it wasn’t, then we are effectively saying there is no evidence that the model gives any new information beyond random guessing. In other words: The model may output a number for a prediction, but if the slope is not significant, it may not be worth actually considering that prediction.

Graphing linear regression

Since a linear regression model produces an equation for a line, graphing linear regression’s line-of-best-fit in relation to the points themselves is a popular way to see how closely the model fits the eye test. Software like Prism makes the graphing part of regression incredibly easy, because a graph is created automatically alongside the details of the model. Here are some more graphing tips , along with an example from our analysis:

Multiple linear regression

If you understand the basics of simple linear regression, you understand about 80% of multiple linear regression, too. The inner-workings are the same, it is still based on the least-squares regression algorithm, and it is still a model designed to predict a response. But instead of just one predictor variable, multiple linear regression uses multiple predictors.

The model equation is similar to the previous one, the main thing you notice is that it’s longer because of the additional predictors. Let’s say you are using 3 predictor variables, the predictive equation will produce 3 slope estimates (one for each) along with an Intercept term:

Prism makes it easy to create a multiple linear regression model, especially calculating regression slope coefficients and generating graphics to diagnose how well the model fits.

What do I need to know about multicollinearity?

The assumptions for multiple linear regression are discussed here. With multiple predictors, in addition to the interpretation getting more challenging, another added complication is with multicollinearity. 

Multicollinearity occurs when two or more predictor variables “overlap” in what they measure. In other places you will see this referred to as the variables being dependent of one another. Ideally, the predictors are independent and no one predictor influences the values of another. 

There are various ways of measuring multicollinearity , but the main thing to know is that multicollinearity won’t affect how well your model predicts point values. However, it garbles inference about how each individual variable affects the response. 

For example, say that you want to estimate the height of a tree, and you have measured the circumference of the tree at two heights from the ground, one meter and two meter. The circumferences will be highly correlated. If you include both in the model, it’s very possible that you could end up with a negative slope parameter for one of those circumferences. Clearly, a tree doesn't get shorter when the circumference gets larger. Instead, that negative slope coefficient is acting as an adjustment to the other variable.

What is the difference between simple linear regression and multiple linear regression?

Once you’ve decided that your study is a good fit for a linear model, the choice between the two simply comes down to how many predictor variables you include. Just one? Simple linear. More than that? Multiple linear.

Based on that, you may be wondering, “Why would I ever do a simple linear regression when multiple linear regression can account for more variables?” Great question!

The answer is that sometimes less is more.  A common misconception is that the goal of a model is to be 100% accurate. Scientists know that no model is perfect, it is a simplified version of reality. So the goal isn’t perfection: Rather, the goal is to find as simple a model as possible to describe relationships so you understand the system, reach valid scientific conclusions, and design new experiments.

Still not convinced? Let’s say you were able to create a model that was 100% accurate for each point in your dataset. Most of the time if you’ve done this, you’ve done one of two things:

• Come to an obvious conclusion that isn’t practically useful (100% of winning basketball teams score more points than their opponent) OR
• You’ve modeled not only the trend in your data, but also the random “noise” that is too variable to count on. This is called “overfitting”: You tried so hard to account for every aspect of the past that the model ignores the differences that will arise in the future.

Other differences pop up on the technical side. To give some quick examples of that, using multiple linear regression means that:

• In addition to the overall interpretation and significance of the model, each slope now has its own interpretation and question of significance.
• R-squared is not as intuitive as it was for simple linear regression.
• Graphing the equation is not a single line anymore. You could say that multiple linear regression just does not lend itself to graphing as easily.

All in all: simple regression is always more intuitive than multiple linear regression!

Interpreting multiple linear regression

We’ve said that multiple linear regression is harder to interpret than simple linear regression, and that is true. Taking the math and more technical aspects out of the question, overall interpretation is always harder the more factors are involved. But while there are more things to keep track of, the basic components of the thought process remain the same: parameters, confidence intervals and significance. We even use the model equation the same way.

Let’s use the same diabetes dataset to illustrate, but with a new wrinkle: In addition to glucose level, we will also include HDL and the person’s age as predictors of their glycosylated hemoglobin level (response). Here’s the output from Prism:

Analysis of variance and F-tests

While most scientists’ eyes go straight to the section with parameter estimates, the first section of output is valuable and is the best place to start. Analysis of variance tests the model as a whole (and some individual pieces) to tell you how good your model is before you make sense of the rest.

It includes the Sum of Squares table, and the F-test on the far right of that section is of highest interest. The “Regression” as a whole (on the top line of the section) has a p-value of less than 0.0001 and is significant at the 0.05 level we chose to use. Each parameter slope has its own individual F-test too, but it is easier to understand as a t-test.

Parameter estimates and T-tests

Now for the fun part: The model itself has the same structure and information we used for simple linear regression, and we interpret it very similarly. The key is to remember that you are interpreting each parameter in its own right (not something you have to keep in mind with only one parameter!). Prism puts all of the statistics for each parameters in one table, including (for each parameter):

• The parameter’s estimate itself
• Its standard error and confidence interval
• A P-value from a t-test

The estimates themselves are straightforward and are used to make the model equation, just like before. In this case the model’s predictive equation is (when rounding to the nearest thousandth):

Glycosylated Hemoglobin = 1.870 + 0.029*Glucose - 0.005*HDL +0.018*Age

If you remember back to our simple linear regression model, the slope for glucose has changed slightly. That is because we are now accounting for other factors too. This distinction can sometimes change the interpretation of an individual predictor’s effect dramatically.

When interpreting the individual slope estimates for predictor variables, the difference goes back to how Multiple Regression assumes each predictor is independent of the others. For simple regression you can say “a 1 point increase in X usually corresponds to a 5 point increase in Y”. For multiple regression it’s more like “a 1 point increase in X usually corresponds to a 5 point increase in Y, assuming every other factor is equal.” That may not seem like a big jump, but it acknowledges 1) that there are more factors at play and 2) the need for those predictors to not have influence on one another for the model to be helpful.

The standard errors and confidence intervals are also shown for each parameter, giving an idea of the variability for each slope/intercept on its own. Interpreting each one of these is done exactly the same way as we mentioned in the simple linear regression example, but remember that if multicollinearity exists, the standard errors and confidence intervals get inflated (often drastically).

On the end are p-values, which as you might guess, are interpreted just like we did for the first example. The underlying method behind the p-value here is a T-test. These only tell how significant each of the factors are, to evaluate the model as a whole we would need to use the F-test at the top. 

Evaluating each on its own though is still helpful: In this case it shows that while the other predictors are all significant, HDL shows no significance since we have already considered the other factors. That is not to say that it has no significance on its own, only that it adds no value to a model of just glucose and age. In fact, now that we know this, we could choose to re-run our model with only glucose and age and dial in better parameter estimates for that simpler model.

Another difference in interpretation occurs when you have categorical predictor variables such as sex in our example data. When you add categorical variables to a model, you pick a “reference level.” In this case (image below), we selected female as our reference level. The model below says that males have slightly lower predicted response than females (about 0.15 less).

Assessing how well your model fits with multiple linear regression is more difficult than with simple linear regression, although the ideas remain the same, i.e., there are graphical and numerical diagnoses.

At the very least, it’s good to check a residual vs predicted plot to look for trends. In our diabetes model, this plot (included below) looks okay at first, but has some issues. Notice that values tend to miss high on the left and low on the right.

However, on further inspection, notice that there are only a few outlying points causing this unequal scatter. If you see outliers like above in your analysis that disrupt equal scatter, you have a few options .

As for numerical evaluations of goodness of fit, you have a lot more options with multiple linear regression. R-squared is still a go-to if you just want a measure to describe the proportion of variance in the response variable that is explained by your model. However, a common use of the goodness of fit statistics is to perform model selection , which means deciding on what variables to include in the model. If that’s what you’re using the goodness of fit for, then you’re better off using adjusted R-squared or an information criterion such as AICc.

Graphing multiple linear regression

Graphs are extremely useful to test how well a multiple linear regression model fits overall. With multiple predictors, it’s not feasible to plot the predictors against the response variable like it is in simple linear regression. A simple solution is to use the predicted response value on the x-axis and the residuals on the y-axis (as shown above). As a reminder, the residuals are the differences between the predicted and the observed response values. There are also several other plots using residuals that can be used to assess other model assumptions such as normally distributed error terms and serial correlation.

Model selection - choosing which predictor variables to include

How do you know which predictor variables to include in your model? It’s a great question and an active area of research.

For most researchers in the sciences, you’re dealing with a few predictor variables, and you have a pretty good hypothesis about the general structure of your model. If this is the case, then you might just try fitting a few different models, and picking the one that looks best based on how the residuals look and using a goodness of fit metric such as adjusted R-square or AICc .

Why doesn't my model fit well?

There are a lot of reasons that would cause your model to not fit well. One reason is having too much unexplained variance in the response. This could be because there were important predictor variables that you didn’t measure, or the relationship between the predictors and the response is more complicated than a simple linear regression model. In this last case, you can consider using interaction terms or transformations of the predictor variables.

If prediction accuracy is all that matters to you, meaning that you only want a good estimate of  the response and don’t need to understand how the predictors affect it, then there are a lot of clever, computational tools for building and selecting models. We won’t cover them in this guide, but if you want to know more about this topic, look into cross-validation and LASSO regression to get started.

Interactions

Interactions and transformations are useful tools to address situations where your model doesn't fit well by just using the unmodified predictor variables.

Interaction terms are found by multiplying two predictor variables together to create a new “interaction” variable. They greatly increase the complexity of describing how each variable affects the response. The primary use is to allow for more flexibility so that the effect of one predictor variable depends on the value of another predictor variable.

For a specific example using the diabetes data above, perhaps we have reason to believe that the effect of glucose on the response (hemoglobin %) changes depending on the age of the patient. Stats software makes this simple to do, but in effect, we multiply glucose by age, and include that new term in our model. Our new model when rounded is:

Glycosylated Hemoglobin = 0.42 + 0.044*Glucose - 0.004*HDL +0.044*Age - .0003*Glucose*Age

For reference, our model without the interaction term was:

Glycosylated Hemoglobin = 1.865 + 0.029*Glucose - 0.005*HDL +0.018*Age

Adding the interaction term changed the other estimates by a lot! Interpreting what this means is challenging. At the very least, we can say that the effect of glucose depends on age for this model since the coefficients are statistically significant. We might also want to say that high glucose appears to matter less for older patients due to the negative coefficient estimate of the interaction term (-0.0002). However, there is very high multicollinearity in this model (and in nearly every model with interaction terms), so interpreting the coefficients should be done with caution. Even with this example, if we remove a few outliers, this interaction term is no longer statistically significant, so it is unstable and could simply be a byproduct of noisy data.

Transformations

In addition to interactions, another strategy to use when your model doesn't fit your data well are transformations of variables. You can transform your response or any of your predictor variables.

Transformations on the response variable change the interpretation quite a bit. Instead of the model fitting your response variable, y , it fits the transformed y . A common example where this is appropriate is with predicting height for various ages of an animal species. Log transformations on the response, height in this case, are used because the variability in height at birth is very small, but the variability of height with adult animals is much higher. This violates the assumption of equal scatter. 

In the plots below, notice the funnel type shape on the left, where the scatter widens as age increases. On the right hand side, the funnel shape disappears and the variability of the residuals looks consistent.

The linear model using the log transformed y fits much better, however now the interpretation of the model changes. Using the example data above, the predicted model is:

ln(y) = -0.4 + 0.2 * x

This means that a single unit change in x results in a 0.2 increase in the log of y . That doesn't mean much to most people. Instead, you probably want your interpretation to be on the original y scale. To do that, we need to exponentiate both sides of the equation, which (avoiding the mathematical details) means that a 1 unit increase in x results in a 22% increase in y .

All of that is to say that transformations can assist with fitting your model, but they can complicate interpretation. 

When linear regression doesn't work

The ubiquitous nature of linear regression is a positive for collaboration, but sometimes it causes researchers to assume (before doing their due diligence) that a linear regression model is the right model for every situation. Sometimes software even seems to reinforce this attitude and the model that is subsequently chosen, rather than the person remaining in control of their research.

Sure, linear regression is great for its simplicity and familiarity, but there are many situations where there are better alternatives.

Other types of regression

Logistic regression.

Linear vs logistic regression: linear regression is appropriate when your response variable is continuous, but if your response has only two levels (e.g., presence/absence, yes/no, etc.), then look into simple logistic regression or multiple logistic regression .

Poisson regression

If instead, your response variable is a count (e.g., number of earthquakes in an area, number of males a female horseshoe crab has nesting nearby, etc.), then consider Poisson regression .

Nonlinear regression

For more complicated mathematical relationships between the predictors and response variables, such as dose-response curves in pharmacokinetics, check out nonlinear regression .

If you’ve designed and run an experiment with a continuous response variable and your research factors are categorical (e.g., Diet 1/Diet 2, Treatment 1/Treatment 2, etc.), then you need ANOVA models. These are differentiated by the number of treatments ( one-way ANOVA , two-way ANOVA , three-way ANOVA ) or other characteristics such as repeated measures ANOVA .

Principal component regression

Principal component regression is useful when you have as many or more predictor variables than observations in your study. It offers a technique for reducing the “dimension” of your predictors, so that you can still fit a linear regression model.

Cox proportional hazards regression

Cox proportional hazards regression is the go-to technique for survival analysis, when you have data measuring time until an event.

Deming regression

Deming regression is useful when there are two variables ( x and y ), and there is measurement error in both variables. One common situation that this occurs is comparing results from two different methods (e.g., comparing two different machines that measure blood oxygen level or that check for a particular pathogen).

Perform your own Linear Regression

Are you ready to calculate your own Linear Regression?  With a consistently clear, practical, and well-documented interface, learn how Prism can give you the controls you need to fit your data and simplify nonlinear regression .

Start your 30 day free trial of Prism   and get access to:

• A step by step guide on how to perform Linear Regression
• Sample data to save you time
• More tips on how Prism can help your research

With Prism, in a matter of minutes you learn how to go from entering data to performing statistical analyses and generating high-quality graphs.