Click or drag to resize
Accord.NET (logo)

IterativeReweightedLeastSquares Class

Iterative Reweighted Least Squares for Logistic Regression fitting.
Inheritance Hierarchy
SystemObject
  Accord.Statistics.Models.Regression.FittingIterativeReweightedLeastSquaresGeneralizedLinearRegression
    Accord.Statistics.Models.Regression.FittingIterativeReweightedLeastSquares

Namespace:  Accord.Statistics.Models.Regression.Fitting
Assembly:  Accord.Statistics (in Accord.Statistics.dll) Version: 3.7.0
Syntax
public class IterativeReweightedLeastSquares : IterativeReweightedLeastSquares<GeneralizedLinearRegression>
Request Example View Source

The IterativeReweightedLeastSquares type exposes the following members.

Constructors
  NameDescription
Public methodIterativeReweightedLeastSquares(GeneralizedLinearRegression)
Constructs a new Iterative Reweighted Least Squares.
Public methodIterativeReweightedLeastSquares(LogisticRegression)
Constructs a new Iterative Reweighted Least Squares.
Top
Properties
  NameDescription
Public propertyComputeStandardErrors
Gets or sets a value indicating whether standard errors should be computed in the next iteration.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyCurrentIteration
Gets the current iteration number.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyGradient
Gets the Gradient vector computed in the last Newton-Raphson iteration.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyHasConverged
Gets or sets whether the algorithm has converged.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyHessian
Gets the Hessian matrix computed in the last Newton-Raphson iteration.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyIterations Obsolete.
Please use MaxIterations instead.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyMaxIterations
Gets or sets the maximum number of iterations performed by the learning algorithm.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyModel
Gets or sets the regression model being learned.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyParameters
Gets the total number of parameters in the model.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyPrevious
Gets the previous values for the coefficients which were in place before the last learning iteration was performed.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyRegularization
Gets or sets the regularization value to be added in the objective function. Default is 1e-10.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertySolution
Gets the current values for the coefficients.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyToken
Gets or sets a cancellation token that can be used to stop the learning algorithm while it is running.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyTolerance
Gets or sets the tolerance value used to determine whether the algorithm has converged.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public propertyUpdates
Gets the last parameter updates in the last iteration.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Top
Methods
  NameDescription
Public methodComputeError Obsolete.
Computes the sum-of-squared error between the model outputs and the expected outputs.
Public methodEquals
Determines whether the specified object is equal to the current object.
(Inherited from Object.)
Protected methodFinalize
Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection.
(Inherited from Object.)
Public methodGetHashCode
Serves as the default hash function.
(Inherited from Object.)
Public methodGetInformationMatrix (Inherited from IterativeReweightedLeastSquaresTModel.)
Public methodGetType
Gets the Type of the current instance.
(Inherited from Object.)
Protected methodInitialize
Initializes this instance.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public methodLearn(Double, Boolean, Double)
Learns a model that can map the given inputs to the given outputs.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public methodLearn(Double, Double, Double)
Learns a model that can map the given inputs to the given outputs.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Public methodLearn(Double, Int32, Double)
Learns a model that can map the given inputs to the given outputs.
(Inherited from IterativeReweightedLeastSquaresTModel.)
Protected methodMemberwiseClone
Creates a shallow copy of the current Object.
(Inherited from Object.)
Public methodRun(Double, Double) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodRun(Double, Double) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodRun(Double, Int32) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodRun(Double, Int32) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodRun(Double, Double, Double) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodRun(Double, Int32, Double) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodRun(Double, Int32, Double) Obsolete.
Runs one iteration of the Reweighted Least Squares algorithm.
Public methodToString
Returns a string that represents the current object.
(Inherited from Object.)
Top
Extension Methods
  NameDescription
Public Extension MethodHasMethod
Checks whether an object implements a method with the given name.
(Defined by ExtensionMethods.)
Public Extension MethodIsEqual
Compares two objects for equality, performing an elementwise comparison if the elements are vectors or matrices.
(Defined by Matrix.)
Public Extension MethodToT
Converts an object into another type, irrespective of whether the conversion can be done at compile time or not. This can be used to convert generic types to numeric types during runtime.
(Defined by ExtensionMethods.)
Top
Remarks

The Iterative Reweighted Least Squares is an iterative technique based on the Newton-Raphson iterative optimization scheme. The IRLS method uses a local quadratic approximation to the log-likelihood function.

By applying the Newton-Raphson optimization scheme to the cross-entropy error function (defined as the negative logarithm of the likelihood), one arises at a weighted formulation for the Hessian matrix.

The Iterative Reweighted Least Squares algorithm can also be used to learn arbitrary generalized linear models. However, the use of this class to learn such models is currently experimental.

References:

  • Bishop, Christopher M.; Pattern Recognition and Machine Learning. Springer; 1st ed. 2006.
  • Amos Storkey. (2005). Learning from Data: Learning Logistic Regressors. School of Informatics. Available on: http://www.inf.ed.ac.uk/teaching/courses/lfd/lectures/logisticlearn-print.pdf
  • Cosma Shalizi. (2009). Logistic Regression and Newton's Method. Available on: http://www.stat.cmu.edu/~cshalizi/350/lectures/26/lecture-26.pdf
  • Edward F. Conor. Logistic Regression. Website. Available on: http://userwww.sfsu.edu/~efc/classes/biol710/logistic/logisticreg.htm

Examples
// Suppose we have the following data about some patients.
// The first variable is continuous and represent patient
// age. The second variable is dichotomic and give whether
// they smoke or not (This is completely fictional data).

// We also know if they have had lung cancer or not, and 
// we would like to know whether smoking has any connection
// with lung cancer (This is completely fictional data).

double[][] input =
{              // age, smokes?, had cancer?
    new double[] { 55,    0  }, // false - no cancer
    new double[] { 28,    0  }, // false
    new double[] { 65,    1  }, // false
    new double[] { 46,    0  }, // true  - had cancer
    new double[] { 86,    1  }, // true
    new double[] { 56,    1  }, // true
    new double[] { 85,    0  }, // false
    new double[] { 33,    0  }, // false
    new double[] { 21,    1  }, // false
    new double[] { 42,    1  }, // true
};

bool[] output = // Whether each patient had lung cancer or not
{
    false, false, false, true, true, true, false, false, false, true
};


// To verify this hypothesis, we are going to create a logistic
// regression model for those two inputs (age and smoking), learned
// using a method called "Iteratively Reweighted Least Squares":

var learner = new IterativeReweightedLeastSquares<LogisticRegression>()
{
    Tolerance = 1e-4,  // Let's set some convergence parameters
    Iterations = 100,  // maximum number of iterations to perform
    Regularization = 0
};

// Now, we can use the learner to finally estimate our model:
LogisticRegression regression = learner.Learn(input, output);

// At this point, we can compute the odds ratio of our variables.
// In the model, the variable at 0 is always the intercept term, 
// with the other following in the sequence. Index 1 is the age
// and index 2 is whether the patient smokes or not.

// For the age variable, we have that individuals with
//   higher age have 1.021 greater odds of getting lung
//   cancer controlling for cigarette smoking.
double ageOdds = regression.GetOddsRatio(1); // 1.0208597028836701

// For the smoking/non smoking category variable, however, we
//   have that individuals who smoke have 5.858 greater odds
//   of developing lung cancer compared to those who do not 
//   smoke, controlling for age (remember, this is completely
//   fictional and for demonstration purposes only).
double smokeOdds = regression.GetOddsRatio(2); // 5.8584748789881331

// If we would like to use the model to predict a probability for
// each patient regarding whether they are at risk of cancer or not,
// we can use the Probability function:

double[] scores = regression.Probability(input);

// Finally, if we would like to arrive at a conclusion regarding
// each patient, we can use the Decide method, which will transform
// the probabilities (from 0 to 1) into actual true/false values:

bool[] actual = regression.Decide(input);
See Also