GREEDY ALGORITHMS. 1. A common and useful paradigm for combinatorial algorithms. Remember the 1980's movie: Wall Street. Michael Douglas says "Greed is good. Greed is right... Greed works". In this lecture, we will explore how well and when greed can work for solving computational or optimization problems. 2. Defining precisely what a greedy algorithm is hard, if not impossible. In an informal way, an algorithm follows the Greedy Design Principle if it makes a series of choices, and each choice is "locally optimized"; in other words, when viewed in isolation, that step is performed optimally. 3. The tricky question is when and why such myopic strategy (looking at each step individually, and ignoring the global considerations) can still lead to globally optimal solutions. In fact, when a greedy strategy leads to an optimal solution, it says something interesting about the STRUCTURE (nature) of the problem itself! In other cases, even if the greedy does not give optimal, in many cases it leads to "provably good" (not too far from optimal) solution. 4. Let us start with a trivial problem, but it will serve to illustrate the basic idea. Example: Coin Changing. US coin denominations: 25, 10, 5, 1 Given an integer x between 0 and 99, make change for x with least number of coins. Mathematically, write x = 25a + 10b + 5c + 1d, so that a+b+c+d is minimum and a,b,c,d >= 0 are ints. Suggest an algorithm for the coin changing problem. 5. Greedy Coin Changing Choose as many quarters as possible. That is, find largest a so that 25a <= x. Next, choose as many dimes as possible to change x - 25a, and so on. An example. Consider x = 73. Choose 2 quarters, so a=2. Remainder: 73 - 2*25 = 23. Next, choose 2 dimes, so b=2. Remainder: 23 - 2*10 = 3. Choose 0 nickels, so c=0. Remainder: 3. Finally, choose 3 pennies, so d=3. Reaminder: 3-3 =0. Solution is a=2, b=2, c=0, d=3. Develop a proof that this algorithm always produces change with least number of coins. 6. Does Greedy Fails Always Work for Coin Changing? Greedy algorithm's correctness depends on the choice of coins. When coins have denominations 25, 10, 5, 1, the greedy always works for any x. But consider the case when coins are of types 12, 5, 1. The greedy does not always return the optimal solution. For x=15, the greedy will use 4 coins: 1*12 + 0*5 + 3*1. The optimal uses 3 coins: 3*5. Moral: Greed, the quick path to success or to ruin! 7. Related Problems: Frobenius Problem. Let N = \sum_i (x_i a_i) denote the sum of money that can be represented with coins a_1, a_2, ..., a_n. If a_1 = 1, then obviously any quantity of money N can be represented. Suppose coins are {2, 5, 10}, then N=1, 3 cannot be represented. But all other N can be represented. Given coins of denomination a_1, a_2, ..., a_n, so that no two have a common factor, find the LARGEST integer N that cannot be changed using these coins. Frobenius problem is NP-Hard! 8. Activity Selection, or Interval Scheduling. We now come to a simple but interesting optimization problem, for which a greedy strategy works, but the strategy is not obvious. We have a list of N activities, which require the use of some resource; e.g. activity = job; resource = processor, or activity = class, resource = lecture hall. Only one activity can be be scheduled on the resource at a time; also once an activity is started, it must be run to completion; no pre-emption. We denote the list of activities as S = {1, 2, ..., n}. Each activity has a specific start time and a specific finish time; the durations of different activities can be different. Specifically, activity i is given as tuple (s(i), f(i)), where s(i) <= (fi) (the finish time must be after the start time). Thus, each activity is pretty inflexible; if chosen, it must be started at time s(i) and end at time f(i). A subset of activities forms a feasible schedule if no two activities overlap (in the time axis). PROBLEM: Design an algorithm to find a feasible schedule with as many activities as possible. Suggest an algorithm to solve this problem. 9. An example instance of the problem. Activity Start Finish 1 1 4 2 3 5 3 0 6 4 5 7 5 3 8 6 5 9 7 6 10 8 8 11 9 8 12 10 2 13 11 12 14 We can visualize the scheduling problem as choosing non-overlapping intervals along the time-axis. ----------------- ---------------- -------- ------------------ --------- -------------- --------------------- ------------- ----------- ------------------------------------------------- -------------------------- 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 10. Potential Greedy Strategies. A. The obvious one: pick the one that starts first. Remove those activities that overlap with it, and repeat. Not optimal: ---- ---- ---- ---- --------------------------------- In this case, greedy with pick just one (the longest one), while optimal has 4. B. Repeatedly pick the activity that has the smallest duration (and does not overlap with those already chosen). Also not optimal. ------------ ------------- -------- C. For each job, count the number of other jobs that overlap with it, and choose the one with the smallest count. This seems a bit better, and does get optimal for both the earlier two cases. Still, this also does not guarantee optimal always. --------- ---------- --------- ---------- --------- --------- --------- --------- --------- --------- --------- --------- --------- The greedy starts by picking the one in the middle, which rightaway ensures that it cannot have more than 3. The optimal chooses the 4 in the top row. 11. The correct greedy strategy. The previous false attempts should give you pause whether any greedy can ensure that it finds optimal in ALL cases. Furtunately, it turns out that there is such a strategy, though it may not be the one that seems the most natural. The correct strategy is to choose jobs in the *earliest finish time* order! More precisely, we sort the jobs in the increasing order of their finish time. By simple relabeling of jobs, let us just assume that f(j_1) <= f(j_2) <= f(j_3) .... <= f(j_n). (Note that in my example, jobs are already labeled this way.) In pseudo-code, we can write: A = {1}; j = 1; // accept job 1 for i = 2 to n do if s(i) >= f(j) then A = A + {i}; j = i; return A In our example, the greedy algorithm first chooses 1; then skips 2 and 3; next it chooses 4, and skips 5, 6, 7; so on. 12. Analysis: Correctness. It is not obvious that this method will (should) always return the optimal solution. After all, the previous 3 methods failed. First of all, it's clear that the method does find a feasible schedule; no two activities accepted by this method can conflict; this is guaranteed by the "if" statement. In order to show optimality, let's argue this way. Suppose OPT is an optimal schedule. Ideally we would like to show that it is always the case that A = OPT; but this is too much to ask; in many cases there can be multiple different optima, and the best we can hope for is that their cardinalities are the same: that is, |A| = |OPT|; the contain the same number of activities. The proof idea, which is a typpical one for greedy algorithms, is to show that the greedy stays "ahead of" the optimal solution at all times. So, step by step, the greedy is doing at least as well as the optimal, so in the end, we can't lose. 13. Some formalization and notation to express the proof. Suppose i_1, i_2, ..., i_k are the set of jobs accepted by Greedy; suppose j_1, j_2, ..., j_m are the set of jobs accepted by optimal OPT. We would like to argue that k = m. Our intuition for greedy is that it chooses jobs in such a way as to make the resource "free again as soon as possible". Thus, for instance, our choice of greedy ensures that f(i_1) <= f(j_1); i_1 finishes no later than j_1. This is the sense in which greedy "stays ahead". We now turn this intuition into a formal statement. Lemma: For any r <= k, we have that f(i_r) <= f(j_r). (That is, the rth job chosen by greedy finishes no later than the rth job chosen by the optimal). * Note that the jobs have non-overlapping durations in both greedy and optimal, so they are uniquely ordered left to right. * Proof. We already saw that the statement is true for r=1, by the design of greedy. Indictively, let us assume that this is true for all jobs upto r-1, and we will prove it for r. So, the induction hypothesis says that f(i_{r-1}) <= f(j_{r-1}). However, we also know that f(j_{r-1}) <= s(j_r). Therefore, we also have f(i_{r-1}) <= s(j_r). That is, the rth job selected by optimal is also available to the greedy for choosing as its rth job. The greedy may pick some other job instead, but if it does, it must be because f(i_r) < f(s_r). Thus, the induction step is complete. Endproof. 14. With the technical expression of the greedy, it is now a simple matter to wrap up the greedy's proof of optimality. Theorem: The greedy algorithm returns an optimal solution for the activity selection problem. Proof. By contradiction. If A were not optimal, then an optimal solution OPT must have more jobs than A. That is, m > k. Consider what happens when r = k in our earlier lemma. We have that f(i_k) <= f(j_k). So, the greedy's last job has finished by the time OPT's kth job finishes. If m > k, meaning there is at least one other job that optimal accepts, that job is also available to Greedy; it cannot conflict with anything greedy has scheduled. Because the greedy does not stop until it no longer has any acceptable jobs left, this is a contradiction. Endproof. 15. Analysis: Running time. The greedy strategy can be implemented in worst-case time O(n log n). We begin by sorting the jobs in increasing order of their finish times, which takes O(n log n). After that, the algorithm simply makes one scan of the list, spending a constant time per job. So total time complexity is O(n log n) + O(n) = O(n log n). ------------------------------------------------------------------------------