by Dimitri P.Bertsekas
ISBNs: 1-886529-43-4 (Vol. I, 4th Edition), 1-886529-44-2(Vol. II, 4th Edition), 1-886529-08-6 (Two-Volume Set, i.e., Vol. I, 4th ed. and Vol. II, 4th edition)
Vol. I, 4TH EDITION, 2017, 576 pages,hardcover
Vol. II, 4TH EDITION: APPROXIMATE DYNAMIC PROGRAMMING 2012, 712pages, hardcover
Vol. I (ISBN10: 1-886529-43-4 or ISBN13: 978-1-886529-43-4): $89.00,
Vol. II (ISBN10: 1-886529-44-2 or ISBN13: 978-1-886529-44-1): $89.00,
Two-volume set, latest editions (ISBN10: 1-886529-08-6 or ISBN13: 978-1-886529-08-3): $134.50
The TWO-VOLUME SET consists of the LATEST EDITIONS OF VOL. I AND VOL. II, i.e., Vol. I, 4th ed. and Vol. II, 4th ed.
Contents,Preface,Ordering,DP Videos (12-hours) from Youtube,Approximate Finite-Horizon DP Videos (4-hours) from Youtube,Home
The leading and most up-to-date textbook on the far-rangingalgorithmic methododogy of Dynamic Programming, which can be used for optimal control,Markovian decision problems, planning and sequential decision making under uncertainty, anddiscrete/combinatorial optimization. The treatment focuses on basic unifyingthemes, andconceptual foundations. Itillustrates the versatility, power, and generality of the method withmany examples and applicationsfrom engineering, operations research, and other fields. It alsoaddresses extensively the practicalapplication of the methodology, possibly through the use of approximations, andprovides an extensive treatment of the far-reaching methodology ofNeuro-Dynamic Programming/Reinforcement Learning.
The first volume is oriented towards modeling, conceptualization, andfinite-horizon problems, but also includes a substantive introductionto infinite horizon problems that is suitable for classroom use. Thesecond volume is oriented towards mathematical analysis andcomputation, treats infinite horizon problems extensively, and provides an up-to-date account of approximate large-scale dynamic programming and reinforcement learning. Thetext contains many illustrations, worked-out examples, and exercises.
This extensive work, aside from its focus on the mainstream dynamicprogramming and optimal controltopics, relates to our Abstract Dynamic Programming (Athena Scientific, 2013),a synthesis of classical research on the foundations of dynamic programming with modern approximate dynamic programming theory, and the new class of semicontractive models, Stochastic Optimal Control: The Discrete-TimeCase (Athena Scientific, 1996),which deals with the mathematical foundations of the subject, Neuro-Dynamic Programming (Athena Scientific,1996), which develops the fundamental theory for approximation methods in dynamic programming,and Introduction to Probability (2nd Edition, Athena Scientific,2008), which provides the prerequisite probabilistic background.
New features of the 4th edition of Vol. I (see the Preface fordetails):
provides textbook accounts of recent original research onapproximate DP, limited lookahead policies, rollout algorithms, model predictive control, Monte-Carlo tree search and the recent uses of deep neural networks in computer game programs such as Go.
includes a substantial number of new exercises, detailed solutions ofmany of which are posted on theinternet (see below).
New features of the 4th edition of Vol. II (see the Preface fordetails):
Contains a substantial amount of new material, as well as a reorganization of old material. The length has increased by more than 60% from the third edition, andmost of the old material has been restructured and/or revised. Volume II now numbers more than 700 pages and is larger in size than Vol. I. It can arguably be viewed as a new book!
A major expansion of the discussion of approximate DP (neuro-dynamic programming), which allows the practical application of dynamic programming to large and complex problems. Approximate DP has become the central focal point of this volume.
Extensive new material, the outgrowth of research conducted in the six years since the previous edition, has been included.
The first account of the emerging methodology of Monte Carlo linear algebra, which extends the approximate DP methodology to broadly applicable problems involving large-scale regression and systems of linear equations.
Expansion of the theory and use of contraction mappings in infinite state space problems andin neuro-dynamic programming.
"Prof. Bertsekas book is an essential contribution that provides practitioners with a 30,000 feet view in Volume I - the second volume takes a closer look at the specific algorithms, strategies and heuristics used - of the vast literature generated by the diverse communities that pursue the advancement of understanding and solving control problems. This is achieved through the presentation of formal models for special cases of the optimal control problem, along with an outstanding synthesis (or survey, perhaps) that offers a comprehensive and detailed account of major ideas that make up the state of the art in approximate methods. The book ends with a discussion of continuous time models, and is indeed the most challenging for the reader. Still I think most readers will find there too at the very least one or two things to take back home with them.
Each Chapter is peppered with several example problems, which illustrate the computational challenges and also correspond either to benchmarks extensively used in the literature or pose major unanswered research questions. At the end of each Chapter a brief, but substantial, literature review is presented for each of the topics covered.
This is a book that both packs quite a punch and offers plenty of bang for your buck. Graduate students wanting to be challenged and to deepen their understanding will find this book useful. PhD students and post-doctoral researchers will find Prof. Bertsekas' book to be a very useful reference to which they will come back time and again to find an obscure reference to related work, use one of the examples in their own papers, and draw inspiration from the deep connections exposed between major techniques. Undergraduate students should definitely first try the online lectures and decide if they are ready for the ride."
Miguel, at Amazon.com, 2018.
Review of Vol. II, 4th Edition:
" This is an excellent textbook on dynamic programming written by a master expositor. Between this and the first volume, there is an amazing diversity of ideas presented in a unified and accessible manner. This new edition offers an expanded treatment of approximate dynamic programming, synthesizing a substantial and growing research literature on the topic."
Benjamin Van Roy, at Amazon.com, 2017.
Amongits special features, the book:
provides a unifying framework for sequential decision making
treats simultaneously deterministic and stochastic control problems popular in modern control theory and Markovian decision popular in operations research
develops the theory of deterministic optimal control problems including the Pontryagin Minimum Principle
introduces recent suboptimal control and simulation-based approximation techniques (neuro-dynamicprogramming), which allow the practical application of dynamic programming to complex problems that involve the dual curse of large dimension and lack of an accurate mathematical model
provides a comprehensive treatment of infinite horizon problems in the second volume, and an introductory treatment in thefirst volume
Reviews of Pre-2005 Editions:
Review of Vols. I and II, 3rd Edition:
"In conclusion, the new edition represents a major upgrade of this well-established book. The coverage is significantly expanded, refined, and brought up-to-date. This is the only book presenting many of the research developments of the last 10 years in approximate DP/neuro-dynamic programming/reinforcement learning (the monographs by Bertsekas and Tsitsiklis, and by Sutton and Barto, were published in 1996 and 1998, respectively). The book is a rigorous yet highly readable and comprehensive source on all aspects relevant to DP: applications, algorithms, mathematical aspects, approximations, as well as recent research. It should be viewed as the principal DP textbook and reference work at present.With its rich mixture of theory and applications, its many examples and exercises, its unified treatment of the subject, and its polished presentation style, it is eminently suited for classroom use or self-study."
Panos Pardalos, inOptimization Methods & Software Journal, 2007.
Review of Vol. I, 3rd Edition:
"In addition to being very well written and organized, the material has several special featuresthat make the book unique in the class of introductory textbooks on dynamic programming. Forinstance, it presents both deterministic and stochastic control problems, in both discrete- andcontinuous-time, and it also presents the Pontryagin minimum principle for deterministic systemstogether with several extensions. It contains problems with perfect and imperfect information,as well as minimax control methods (also known as worst-case control problems or games againstnature). I also has a full chapter on suboptimal control and many related techniques, such asopen-loop feedback controls, limited lookahead policies, rollout algorithms, and modelpredictive control, to name a few. ... In conclusion the book is highly recommendable for anintroductory course on dynamic programming and its applications."
Onesimo Hernandez Lerma, inMathematic Reviews, Issue 2006g.
"In conclusion, this book is an excellent source of reference ... Themain strengths of the book are the clarity of theexposition, the quality and variety of the examples, and its coverageof the most recent advances."
Thomas W.Archibald, in IMA Jnl. of Mathematics Applied in Business & Industry
"Here is a tour-de-force in the field."
David K. Smith, inJnl. of Operational Research Society
"By its comprehensive coverage, very good materialorganization, readability of the exposition, includedtheoretical results, and its challenging examples andexercises, the reviewed book is highly recommendedfor a graduate course in dynamic programming or forself-study. It is a valuable reference for control theorists,mathematicians, and all those who use systems and control theory in theirwork. Students will for sure find the approach very readable, clear, andconcise. Misprints are extremely few."
Vasile Sima, in SIAM Review
"In this two-volume work Bertsekas caters equally effectively totheoreticians who care for proof of such concepts as theexistence and the nature of optimal policies and topractitioners interested in the modeling and the quantitative andnumerical solution aspects of stochastic dynamic programming."
Michael Caramanis, in Interfaces
"The textbook by Bertsekas is excellent, both as a reference for thecourse and for generalknowledge. It is well written, clear and helpful"
Student evaluation guide for the Dynamic Programming and StochasticControl course at theMassachusetts Institute of Technology
The author is McAfee Professor of Engineering at theMassachusetts Institute of Technology and a member of the prestigious US NationalAcademy of Engineering. He is the recipient of the 2001 A. R. Raggazini ACC education award, the 2009 INFORMS expository writing award, the 2014 Kachiyan Prize, the 2014 AACC Bellman Heritage Award, and the 2015 SIAM/MOS George B. Dantsig Prize.He has been teaching the material included in this bookin introductory graduate courses for more than forty years.
Supplementary Material:
The material listed below can be freely downloaded, reproduced, anddistributed.
- Videos and slides on Reinforcement Learning and Optimal Control. A 13-lecture course, Arizona State University, 2019
- Videos on Approximate Dynamic Programming. A 6-lecture, 12-hour short course, Tsinghua University, Beijing, China, 2014
- Lecture slides for a 6-lecture short course on Approximate Dynamic Programming, Tsinghua University, Beijing, China, 2014
- Approximate Finite-Horizon DP videos and slides(4-hours) A 4-lecture series from the author's web site, 2017
- Videos and Slides on Abstract Dynamic Programming, A 5-lecture series on Semicontractive Dynamic Programming, 2016
- Prof. Bertsekas' Course Lecture Slides, 2004
- Prof. Bertsekas' Course Lecture Slides, 2015
- Theoretical problem solutions, Volume1
- CourseMaterial at Open Courseware at MIT
- Material from 3rd edition of Vol. I that was not included in the 4th edition:Minimum Variance Control (Section 5.3 and Appendix F) and Adaptive Control (Section 6.1)
- Prof. Bertsekas' Research Paperson Dynamic and Neuro-Dynamic Programming
- Prof. Bertsekas' Ph.D. Thesis at MIT, 1971, Control of Uncertain Systems with a Set-Membership Description of the Uncertainty, which contains supplementary material for Vol. 1
- Errata
