Lecture 1 on 9/4/97

• What do we mean by design, optimization, and design optimization?
  • Design = Decision making towards the objective of fulfilling a need.
  • Optimization is a methematical tool that helps us solve design problems if we pose them corrrectly.
  • Optimal design = Improving the design so as to achieve the best way of satisfying the original (what your real system needs, not only your model), within the available means.
• Why is modeling important in optimal design?
• Elements of a model: variables, parameters, constants, and mathematical relationships
• Analysis and synthesis (design) models: How are they different?
• Why is the search of the design space difficult?
  We looked at how different algorithms behave for a simple 2-variable Rosenbrock's banana function. It all depends on the topography of the design space, which in turn depends on how well we posed the problem.
• Terminology
  Feasible and infeasible regins of the design space, active and inactive constraints, local and global analysis, unconstrained and constrained optimization, interior and boundary optima.
• The motivation for studying boundedness checking and monotonicity analysis (the Seltzer Pill example).
• How is variational calculus different from the finite dimensional, discrete variable based optimization?
• Applications of variational calculus in structural optimization

• What do we mean by global and local minima (mathematically)?
  • Global minimum: f(x*) =< f(x) for all x belonging to a feasible region, S.
    Then, x* is a global minimum.
  • Local minimum: f(x*) =< f(x) for all x belonging to a neighborhood, N.
    N = {x| x belongs to S with |x-x*| < delta}
    Then, x* is a local minimum.
  • If "=<" is replaced by "<", then we say that they are strict global/local minima.
• Wierstraus Existence theorem
  Conditions for existence of a minimum
  1. f(x) is continuous in S
  2. S is a compact set.
  "compact" ==> closed and bounded set
  "closed" ==> boundary points of S are included
  "bounded" ==> f(x) is defined everywhere in S and is finite
  
• Necessary and Sufficient conditions for a local minimum
  • All minima must satisfy the necessary condition(s).
    
  • All points that satisfy the necessary condition(s) are not necessarily minima.
  • A point that satisfies the sufficient condition(s) is indeed a minimum.
• I order necessary condition and its proof
• II order sufficient condition and its proof
• II order necessary condition
  Why do we state this separately?
• What happens when the second derivative is zero at the point where the first derivative is zero?
  • f = x^4 and f = x^3
• Shaft design example (analytical solution is possible)
• Shear stress in a two-cylinder-contact problem (analytical solution is not possible)
  This is the first motivation for studying numerical methods in the constrained minimization in one variable.
• The second motivation:
  In a problem involving many variables, {x}. {x} is a vector. If we start at {x_k} and want to go to another point {x_(k+1)} where the objective function is a lower, we usually find a descent direction {d} and move along that. Howmuch we move along that direction is decided by line search.
• List of various methods for line search
  • Direct (root finding) methods
    • (1) Gradient method
    • (2) Newton's method
    • (3) Quasi-Newton methods
    • (4) Secant method
  • Polynomial Interpolation methods
    • (5) Quadratic, cubic, etc.
  • Interval based methods
    • (6) Bisection
    • (7) Interval-halving
    • (8)Golden section method
    • (9) Fibonacci search
  • Inexact line search methods
    • (10) Armijo and Goldstein conditions

• Conditions for Unimodality of a function
  Unimodality assumption is necessary for some of the line search methods mentioned in Lecture 2.
  f(x2) < f(x1) if x1 < x2 < x* and
  f(x1) < f(x2) if x2 > x1 > x*
  
  
• Example: Minimize f = x^4 - 32*x
  This example was solved using all the methods below.
  Please work it out yourself using all the methods.
• Gradient method
  This is based on Ist order Taylor's series expansion.
  x(k+1) = x(k) - f'[x(k)]
  An appropriate scaling factor, alpha may be used to scale the gradient in the previous equation. This scheme is known as the Modified Gradient Method.
  
  
• Modified Gradient Method
  x(k+1) = x(k) - alpha * f'[x(k)]
  There may be many ways to make judicious picks on the scaling factor, alpha. The Newton method demonstrates how values of alpha can be computed directly in each iteration.
  
  
• Newton's method
  This is based on IInd order Taylor's series expansion.
  x(k+1) = x(k) - f'[x(k))/f''[x(k)]
  
  
• A proof for the convergence of Newton's Method has been furnished by Kantarovich.
  The conditions set by his theorem can be paraphrased as:
  • (i) the norm of the reciprocal of the second derivative, || f''(x) || should be bounded by beta
  • (ii) the absolute value of the ratio of the first derivative to the secnd derivative of the function || f'(x)/f" || should be bounded by gamma
  • (iii) the function should be Lipschitz continuous with alpha as the Lipschitz constant, and
    
  • (iv) the product of alpha, beta, and gamma should be less than 0.5.
  
  
• Quasi Newton Methods
  When it is not possible to evaluate the first and the second derivatives of the function analytically, we use numerical finite difference methods to compute them.
  f'(x) = { f(x+delta_x) - f(x-delta_x) } / { 2*delta_x }
  f''(x) = { f(x +delta_x) - 2*f(x) + f(x-delta_x) } / { delta_x }^2
  delta_x is a small number.
  
  
• Secant Method
  The second derivative of the function, f''(x) is approximated using the secant (the chord joining two points on a curve).
  S(k) = {f'(x2) - f'(x1) }/ (x2 -x1)
  where x1 and x2 are two points between which a minimum lies.
  Now,
  x(k+1) = x(k) - f'[ x(k) ]/ S(k)
  
  
• Interpolation Schemes
  Within the interval of interest, the function is locally approximated using either a quadratic or a cubic polynomial. And, then we find the minima of the polynomial which serve as the approximations to the minimum of the original function.
  
  • Quadratic:
    (i) Three points
    (ii) Two points and a derivative at one point
  • Cubic:
    (i) Four points
    (iii) Three points and a derivative at one point

Lecture 5 on 9/23/97

• Constrained optimization of one variable
  • Example: Minimize f = x^2-4x+4
    (i) equality constraint: h = x^3-6x^2+5x+12=0
    Feasible region = {-1,3,4}
    (ii) inequality constraint: g = x^3-6x^2+5x+12 <= 0
    Feasible region: (-infinity,-1] & [3,4]
    
    
  • Condition for a constrained local minimum
    Since any feasible movement from x* must be in the increasing direction of f(x)
    i.e. if x = x* + alpha*d with alpha >= 0 & d is the feasible direction,
    then f(x) >= f(x*)
    Considering the I order Taylor's series expansion,
    f(x) = f(x*)+f'(x*)(x-x*)+o(2) f'(x) (alpha*d) = f(x)-f(x*) >= 0 with alpha >= 0
    f'(x) d >=0 since alpha >=0
    
    This is the necessary and sufficient conditon for a constraint local minimum.
    Obviously, it also works for the multiple variables situation:
    (gradient f(x))(delta x*) >= 0
    
    
• Unconstrained optimization for several variables
  Minimize f(x) where x = (x1,x2,...,xn)
  I order Necessary conditions: (gradient)f(x*) = 0
  II order Sufficient conditions: Hessian(f(x*) is positive definite.
  II order Necessary conditions: Hessian(fx*) if positive semi-positive.
  
  Positive definiteness of a matrix implies :
  • All eigenvalues are positive
  • All determinantof the principal minars are positive
  • Row reduced echelon form, all pivots are positive
  
  

  Nature of the stationary point

  Hessian at x*	Eigenvalues	Nature of x*
  Positive definite	All +ve	Local minimum
  Positive semi-definite	All nonnegative	Probably a local minimum
  Negative definite	All -ve	Local maximum
  Negative Semi-definite	All nonpoositive	Probably a local maximum
  Indefinite	Mixed signs	Saddle point

• Example
  Minimize f = x1^2 + 1/x1 + x2 + 1/x2
  gradient f = [2x1 -1/x1^2]=[0 0]
  then x1* = [1/(2^(1/3)) 1] or x2* = [1/(2^(1/3)) 1]
  H = [2 + 2/x1^3 0 ]
  [0 2/x2^3]
  H(x1*) positive definite : x1* is a minimum.
  H(x2*) indefinite : x2* is a saddle point.
  
  
• ! SURPRISE QUIZ !
  
  
• Constrainted optimizaiton for several varialbles
  Graphical Interpretationintroduction
  Several contour plots were drawn and explained.
  
  • Minimize f(x) with one inequality active constraint g at X* : gradient f(x*) + g'(x*) = 0
    That is, the gradient of the function and the gradient of the constraint are in the same direction.
    
  • Minimizing f(x) with inactive constraints is the same as the unconstrained situation because the minimu lies in the interior of the feasible region.
    
  • Minimize f(x) with two inequality constraint g1, g2 at X* :
    gradient f(x*) + c1*g1'(x*) + c2*g2'(*) = 0
    
  • Minimize f(x) with equality constraint h
    The dimension of the feasible region region decrease by one with each equality constraint. That is, if we had a (x1,x2) space, with an equality constraint, we will be constrained to move only along a curve -- meaning one dimension.

• Solutions to (i) the surprrise quiz, (ii) homework #1 and (iii) homewrk #2 were discussed.
• Graphical interpretatioon of the necessary conditions for the constrained optimization were reiterated.
  • At the optimum, the gradient of the objective at a minimum can be expressed as a linear combination of the gradients of the constraints.
  • The scalars with which we multiply the gradients of the constraints are called "multipliers" (in fact, the so called Lagrange Multipliers as we will see later).
  • The multipliers associated with the inactive constraints are zero. (That is the reason why the inactive constraints do not affect the optimum.)
  • Once an inequality constraint becomes active, it effecively is an EQUALITY constraint, since the solution must lie on its boundary.
• Derivation of the necessary condition for the constrained optimization problem with equality constraints.
  • Each equality constraint effectively reduces the dimension of search space (or the design space) by one.
  • Patitioning of the original N variables (x1, x2, ..., xN) into M solution (dependent) variables and P = N - M decision (independent) variables. M = the number of equality constraints.
  • Direct elimination of M solution variables is sometimes possible if the equality constraints can be solved explicitly and eliminate the M solution variables from the objective function. It then becmes an unconstrained optimization But, in general, this may not be possible. We do this, in the gradient space instead.
  • The concept of the reduced gradient.
  • The definitin of the multipliers.
  • Obtaining the necessary conditions.
• The concept of the Lagrangian and relation to the necessary conditions just derived.
  Lagrangian = L = f + sum_of( mu_i * h_i)
  i = 1 ... M

Lecture 7 on 9/30/97

• A brief re-discussion of the derivation of the necessary conditions for the equality constrained optimization problem.
• Its extension to the active inequality constraints.
• The difference between the Lagrange multipliers associated with the equality and active ineequality constrains.
• Proof for why the Lagrange multipliers associated with the active inequality constraints should be positive where as those of the equality are unrestricted in sign.
• The Karush-Kuhn-Tucker (KKT) conditions
• What do the switching conditions mean in terms of computation?
• The concept of constraint qualification and regular point.
• An example to reinforce the above two concepts.
  
  Minimize f = x1^2 + x2^2 - 4*x1 + 4
  Subject to
  .......... g1 = -x1 =< 0
  .......... g2 = -x2 =< 0
  .......... g3 = x2 - (1-x1)^3 =< 0
  
  
• The derivation of the the sufficient conditions for a constrained optimization problem.
  • the concept of the reduced Hessian and the differential quadratic form with respect to the decision variables' small perturbations of the constrained problem.
  • The equivalence of the above "differential quadratic form" of the reduced Hessian and the "differential quadratic form" of the Hessian of the Lagrangian with respect to the small perturbations of all the variables.
  • The "differential quadratic form" of the Hessian of the Lagrangian needs to be positive for those small perturbations in x that satisfy the feasibility condition viz. gard(h) delta(x) = 0 .
    
    Note that this a weaker condition than requiring the positive definiteness of the Hessian of the Lagrangian.
    
    
  • An example to reinforce the above concepts.
    
    Minimize f = x1^2 + x2^2 - 3*x1*x2
    Subject to
    .......... g1 = x1^2 + x2^2 - 6 =< 0
    
    
    Understand why we said the sufficient condition is "weaker" than the positive definiteness condition by solving this example

Lecture 8 on 10/2/97

• Solution of constrained minimization problem:
  min f=x1^2 + x2^2 - 4x1 + 4
  subject to
  g1= -x1 =< 0
  g2= -x2 =< 0
  g3 = x2 - (1-x1)^3 =< 0
  
  This example shows that the KKT theory does not apply because the minimum is at an irregular point. Apply KKT conditions to see that there will be a a conflict in solving them.
  
  
• KKT sufficiency condition for constrained minimization in N variables.
  (Also refer to the Handout)
  
  
• Example to shoow application of the sufficiency condition
  Solution of the second example problem given in the previous lecture, including application of the KKT sufficiency condition to check the minimum point obtained from the KKT necessary conditions.
  
  
• Introduction of the concept of Post-optimal or Sensitivity Analysis, and the physical meaning of the Lagrange multipliers.
  
  • When the objective function is multiplied by some scalar constant K:
    The modified Lagrange multipliers are K times the values in the original problem. We can easily see this from the KKT necessary conditions.
    
  • When the constraints are multiplied by some scalar constant K:
    The modified Lagrange multipliers are (1/K) times the values of those in the original problem. This again follows from the necessary conditions.
    
  • When the constraint is perturbed:
    g =< epsilon
    i.e., delta_g = epsilon then
    delta_f = -lambda * delta_g
    
  
  
  
• An example to demonstrate the significance of post-optimal analysis.
  
  • The original problem: min f(x1,x2) = x1^2 + x2^2 - 3*x1*x2
    subject to
    g1 = x1^2 + x^2 - 6 <= 0
    
  • The modified problem: min f(x1,x2) = x1^2 + x2^2 - 3*x1*x2
    subject to
    g1new = x1^2 + x2^2 - 6 =< 2
    i.e., delta_g = 2
    
  • The dircet calculation shows that the formula derived above is valid.

Lecture 9 on 10/7/97

Lecture 10 on 10/9/97

Lecture 11 on 10/16/97

• Variable metric methods
  H(k+1)=H(k)+delta(H(k))
  B(k)=inverse(H(k))
  B(k+1)=B(k)+delta(B(k))
  
  • The conceptual basis
    
  • Requirements on updating the approximation of the (inverse) of the Hessian matrix
    
  • Rank 1 and Rank 2 updates
    
  • DFP and BFGS formulas
    
  • Complimentary relationship between DFP and BFGS update schemes
    
    
• Convergence types
  • Linear convergence
    
  • Superlinear convergence
    
  • Quadratic convergence
    
    
• Inexact line search methods:
  
  • Armijo's first condition
    
  • Goldstein's condition
    
  • Armijo's second condition
    
    

Lecture 12 on 10/23/97

• Scaling
  
  • Proper scaling modifies the topography of the design space and makes algorithms converge better.
    
    
  • A Simple example to illustrate how the steepest descent method can be made to converge in fewer steps after scaling.
    
    
  • Scaling matrix, D
    x~ = D x
    
    D can be chosen in many ways. Usually diagonal form is preferred. The diagonal elements can be chosen to be either the square root of the diagonal entries in the Hessian or the reciprocal of the absolute value of the respective variabel itself.
    
    A simple transformation that makes all variables vary between -1 and + 1 can be written as:

    x~_i = { 2*x_i - ( u_i + l_i ) } / { ( u_i - l_i ) }
    where l_i < x_i < u_i
    
    

  • Computing gradients and Hessians after scaling
    grad(f(x~) = inv(D grad(f(x)
    H(f(x~) = [inv(D]' H(f(x) [inv(D]
    
    Using the above the old unscaled routines can be used for the scaled problem.
    
    
  
  
Modeling an Optimal Design Problem

Formulating an optimization problem for a real design problem is the focus of modeling.

Papalambros and Wilde (1988) note that the most common mistake in modeling is to leave something out. This type of mistakes will manifest in the form of under-constrained or unbounded models.

In this section, we will only deal with positive variables (unless mentioned otherwise) as the design variables are usually positive (e.g., length, thickness, etc.).

A few important terms:

• Well-bounded:
  If minimum exists, the problem is said to be well-boounded .
• Lower bound
• Greatest Lower bound or Infimum
• Minimum
• Minimizer
• Upper bound
• Lowest Upper Bound or Supremum
• Maximum
• Maximizer
• Well-constrained:
  If a constrained minimum exists, the problem is said to be well-constrained .
• Constraint activity
  
  • Active
    Removal of the constraint changes the minimum
  • Semi-Active
    Removal of the constraint changes some of the minima
  • Inactive
    Removal of the constraint does not change the minimum
  • Critical
    Removal of the constraint makes the problem unbounded or ill-constrained.
  
  
  Note that "critical" is stronger than "active".
  
  
• Partial minimization
  Fix all but one variable at a time and then minimize with respect to that variable. It is called partial minimization
  
  It tells helps us identify the critical connstraints and then eliminate a variable along with the corresponding critical constraint.
  
  
• Monotonicity
  
  A function is said to increase or be increasing monotonically with respect to the single positive finite variable x, if for every x_2 > x_1, f(x_2) > f(x_1). That is,
  { deltaf / deltax } > 0
  Or if the function is differentiable,
  df/dx > 0
  The above definition talks about "positive monotonocity". Similarly, "negative monotonocity" can also be defined.
  
  Notation:
  f(x+) indicates positive monotonocity of f w.r.t. x
  f(x-) indicates negative monotonocity of f w.r.t. x
  
  
• Some Observations on monotonicity
  
  
  • An unconstrained monotonic function is not bunded for positive finite arguments.
  • If an unconstrained function f(x) has a finite positive minimizer, it is not monotonic.
  • If f(x+) with a single constraint g1(x) =< 0 having a positive finite root A such that g1(A) = 0, then f is well-constrained only if g1(x-).
  • If there are more than one monotonic inequalities, and no equalities, f(x+) is well- constrained only if at least one inequality decreseas with x.
    Similarly, f(x-) needs at least one g(x-).
  • A variable is said to be monotonic if it is monotonic in every constraint and the objective.
  
  
  
• Following the above observations, the first monotonicity principle, MP1, can be stated as follows:
  
  In a well-constrained objective function, every "increasing" (decreasing) variable is bounded from "below" (above) by at least one active constraint.

Lecture 13 on 10/28/97

• Air-tank design example solution using the monotonicity analysis
  Monotonicities and critical constraints were identified.
  Variable and the correspnding critical constraints were eliminated one at a time.
  Since the prooblem turns out to be unbounded, an extra constraint was added.
  With this, the solution is obtained without using KKT theory.
  
  
• Recognizing monotonicities for complex functions
  Various rules for discussed.
  
  
• Another example was solved.
  Minimize f = x1 - x2
  Subject to
  g1: 2 x1 + 3 x2 - 10 =< 0
  g2: -5 x1 - 2 x2 + 2 =< 0
  g3: -2 x1 + 7 x2 - 8 =< 0
  
  
  

Lecture 14 on 10/30/97

• Constraint bifurcation
  A non-monotonic constraint can sometimes be split into two or more monotonic constraints. This concept also relates to the regional monotonicity concept wherein we can identify different regions of the design space where the objective and constraints are monotonic.
  
  
• Conditional criticality
  If more than one constraint is critical for a varible, we can immediately decide which of them will be cirtical (active). So, we call them conditionally critical constraints.
  
  
• Multiple criticality
  If one constraint is critical for more than one variable, then it is said to be multiply critical. If this happens, one of the variables needs to be eliminated using the multiply critical constraint.
  
  
• Dominance
  If a constraint is dominated by other constraints, then we can remove the dominated constraint. Dominance of a constraint occurs if the feasible space of this constraint is a subset of the feasible space of the other constraints. For example, if g_i < g_j for all values of x, then g_i is dominated by g_j.
  
  
• Relaxation of a constraint
  If there is a very complex constraint and we want to find if it is going t be active, we can do the following:
  
  Let the complex constraint be g_i.
  Delete g_i and form a relaxed problem.
  Solve the relaxed problem and find x as the solution.
  If x satisfies g_i, g_i is said to be inactive for the original problem.
  If not, g_i should be active or semi-active for the orginal problem
  
  
• Uncriticality
  In a well-constrained problem, if there are constraints that exhibit the same monotonicity w.r.t. a variable as that of the of the objective, then they are called uncritical .

  An important observation in relationship to uncriticality:
  If there is a constraint g_i that is critical w.r.t. x_k and there is another constraint g_j that is uncritical w.r.t. x_k, and g_j depends on no other variables, then g_j is either inactive or the constraints are incnsistent.
  
  

• Activity of an equality constraint
  Equality constraints can be inactive r semi-active "in the sense that removal f this cnstraint dooes not affect the value of the minimum for the problem."
  
  This can happen if there are variables in the problem that are not in the objective constraint.
  
  
• Relevance of a variable
  
  • A non-objective variable is said to be irrelevant if and only if none of the constraints in which it appears are active
  • A variable occuring in the objective function or an active constraint is relevant.
• The Second Monotonicity Principle, MP2
  Every monotoonic non-objective variable in a well-bounded problem is either
  a) irrelevant and can be deleted from the problem together with all constraints in which it ccurs, or
  b) relevant and bunded by two active cnstraints, one from above and one from below.
  
  
• Understanding part (b) of MP2
  Consider:
  Minimize f(x_i+)
  Subject to
  g(x_i-, x_j-) =< 0,
  then f(x_i*) is indirectly negative montonic w.r.t. x_j.
  
  This was proved in the class.
  
  
• An example problem was solved to illustrate MP2.

  Minimize f = -x1
  Subject to
  g1: exp(x1) - x2 =< 0
  g2: exp(x2) - x3 =< 0
  g3: x3 - 10 =< 0
  
  

• Another example was partially solved to highlight the regional monotonicity and the application of MP1 and MP2

  Minimize f = x1^2 + x2^2 - 2 x1 - 4 x2
  Subject to
  g1: x1 + 4 x2 - 5 =< 0
  g2: 2 x1 + 3 x2 - 6 =< 0
  
  

• Directing equality constraints
  As we discussed the MP1 and MP2 principles, they can be applied to only inequality constraints. So, if we have any equality constraints, we need to convert them to inequality constraints. The sign of the inequality ("less than" or "greater than") is decided by the needed monotonicities of the variables involved.
  

Lecture 15 on 11/6/97

Prior to discussing the monotonicity analysis, we had studied the numerical methods for the unconstrained minimization of many variables. Now, we will look at numerical methods for CONSTRAINED optimization variables. We will begin with linear problems first and then move to nonlinear problems.

• What are LP (linear programming) and NLP (nonlinear programming problems?
• Why do we study LP problems even though most of the design problems are of the NLP type?
• General form of the LP problem
  
  Minimize f = c1*x1 + c2*x2 + ... + cN*xN
  Subject to
  a_1j * x1 + a_2j * x2 + ... + a_Nj * xN =< b_j {j = 1,...,M}
  a_1j * x1 + a_2j * x2 + ... + a_Nj * xN = 0 {j = M+1,...,M+L}
  N = number of variables
  M = number of inequality constraints
  L = number of equality constraints
  
  
  
• Some observations:
  • Feasible region for LP problems is a convex polyhedron in N dimensions. For 2-D, it is convex polygon.
  • The minimu for LP problems lies at one of the vertices of the feasible polyhedron. In such a case, the solution is unique.
  • In some special cases, the minimum can lie anywhere on a certan edge of the feasible polyhedron. In such a case, the solution is not unique.
  • Sometimes the feasible region can be unbounded. In such a case, there may or may not be a solution depending in what way the solution is unbounded.
  • Sometimes there may not be any feasible region, in which case, no solution exists.
  
  
  
• KKT theory can be used to solve LP problems. But, we know the problem with it: we need to know which constraints should be active. That means, we need to enumerate and solve all cases--which is a tedious task to do it by hand and inefficinet process to do it on the computer (for large problems with many constraints). Fortunately, there is an efficient method by name the Simplex method .

  Before presenting the Simplex method, we discuss the basis for it first and see how it works.
  
  

• Principles behind the Simplex Method
  • First, we need to write the general form of the LP into a "standard form". In the standard form, we should only have equality constraints and non-negative variables.
  • Inequality constraints can be converted to equality constraints by using slack and surplus variables.
  • Variables that can take both positive and negative values can be written as the difference of two non-negative variables.
  • What do we mean by a feasible solution to an LP problem?
  • What is a basic feasible solution
    This corresponds to a vertex of the feasible polyhedron.
  • The minimum of an LP problem is attained at one of the basic feasible solutions.
  • Howmany basic feasible solutions can there be?
  • The Simplex algorithm consists of two phases.
  • Phase I involves finding a basic feasible solution via obtaining the canonical form of the LP problem.
    What is the canonical form?
  • How do we get the canonical form?

Lecture 16 on 11/11/97

• A numerical example was solved to explain the simplex method. Its Phase I was illustrated by getting the canonical form. Now, we have a basic feasible solution
• What happens to the objective function when we transform the constraints during Phase I.
  Something interesting happens when we include the objective function as the last row.
• What is a basis?
  What are basic and non-basic variables?
• The Phase II of the simplex method now proceeds to find the next basic feasible solution adjacent to the current one.
  "Moving to another basic feasible solution" imples replacing one variable that is in the basis now with a non-basic variable.
  So, we need to decide which one should leave the basis and which one should enter the basis.
• What is the reasoning for deciding the variable that should enter the basis?
• What is the reasoning for deciding which variable should leave the basis?
• Out of the above reasoning, the simplex algorithm emerges. (read the textbook for "organizing" the Phase I and II of the simplex method).
• Where do Lagrange multipliers pop-up in the simplex method?

Having discussed how LP problems can be solved, we now turn to constrained NLP problems.

• We first discuss the Penalty formulations that convert the constrained problems into unconstrained problems by intriducing a new penalized objective function.
• Penalized pseudo objective function>P> phi(x,rp) = f(x>) + rp * P(x)
  rp = penalty parameter (a scalar)
  P(x) = penalty function that is formed from the constraints.

• What are exterior and interior penalty functions?
  And what is their graphical interpretation?
  What is the role of the penalty parameter in each of those?
  What are their advantages and disadvanatges?
  
• Where do Lagrange multipliers pop up in the interior penalty function formulation?
• Numerical example for interior penalty function approach.

Lecture 17 on 11/13/97

• How is extend interior penalty function better than EPF and IPF?
• Why is scaling of the constraints is especially important in penalty formulations?
• What is augmented Lagrange Multiplier method (ALM)?
• What are its advantages? How is it better for numerical implementation?
• A numerical example for ALM.

Next we move to another indirect, like the penalty formulations, method for constrained problems.

• SLP (Sequential Linear Programming) Here, we linearize the objective and the constraints at the starting design point and will solve an LP problem to find a new (hopefully better) design point.
  And then, we linearize about this new point and go to another design point and so on sequentially. There are commercial optimization programs that are based on SLP.

Next, we move to some direct methods. That is, here we don't solve an unconstrained or linear problems as a short cut to solving constrained problems. Rather, we solve the constrained problems directly. We will discuss two direct methods, viz. the method of feasible directions abd the Genralized Reduced Gradient (GRG) method .

• The method of feasible directions
  
  • Usability and feasibility constraints were recalled. and usable and feasible sector was illustrated graphically.
  • Within the usable and feasible sector, how do we choose the best search direction for performing line search?
  • The question posed above is solved as sub-optimization problem. This is the essence of the feasible directions method. The highlight is that this sub-optimization problem is an LP!
  • The DOT/DOC software from VMI inc. (a company owned by the author of our textbook) uses this method. It is quite robust.

Lecture 18 on 11/18/97

• The Generalized Reduced Gradient (GRG) method

  This is a "direct" numerical method for solving the constrained minimization problems. In some ways, it works like the Simplex algorithm. We convert the general form with inequality and equality constraints to a "standard" form that has only equality constraints by introducing slack variables. The theory behind the reduced gradient was already discussed when the KKT conditions were proved. The essence of this method is that each equality constraint reduces the design space dimension by one. If the constraint is simple and explicit enough, a variable can be easily eliminated, but if it is not we indirectly reduce the search space dimension in the "gradient space".

  A feasible design point is needed to start the GRG method. Once it is found, the algorithm proceeds to the boundary of the feasible region and stays on it (like the Simplex algorithm) unitl the constrained minimum is found.
  
  

  • The reduced gradient concept was reviewed. Particularly, the concept of independent and dependent variables is important for the GRG method.
  • Search direction is simply the negative of the reduced gradient.
  • A line search is performed to move optimally along the search direction. In doing so, the feasibility is maintained. Only independent variables are varied freely, and then the dependent variables are adjusted to maintain feasibility. This is done in an inner loop.
• Sequential Quadratic Programming (SQP) method
  • This method combines good features of several algorithms including augmeted lagrange multiplier method, penalty formulations, the method of feasible directions, and SLP. It can be treated as both a direct method as well as an indircet method.
  • LIke the feasible dircetions method, it also involves a sub- optimization problem to find a suitable search dircetion. This problem is a QP. In this QP, a quadratic approximation of the augmented objective is minimized subject to linearlized constraints. And then, the line search is performed on a pseudo-objective that includes penalty due the constraints. The Hessian of the Lagrangian is upated using BFGS method.
  • It is reported to be the most efficient and robust NLP algorithm.

Lecture 19 on 11/20/97

• The relationship between the LP's Simplex method and the NLP's GRG method was briefly discussed again in the context of the homework problem.
• The theoretical basis for SQP was described along with a handout.
• The essentials of the variational claculus were presented along with a handout.
  • What is a functional?
  • What is variational claculus ?
  • Brachistochrone problem as a motivating example
  • Euler-Lagrange equations (ELE) for stationarity of variational calculus problems
  • Shortest distance between two points in a plane solved using ELE
  • Uniform deflection beam analysis using ELE

Lecture 20 on 12/2/97