The Markov chain approximation methods are widely used for the numerical solution of nonlinear stochastic control problems in continuous time. This book extends the methods to stochastic systems with delays and is the first book on the subject. Featuring numerical algorithms and examples with applications to control and modern communications systems, the book will be of great interest to all those who work with stochastic delay equations and whose main interest is in either the use of the algorithms or in the underlying mathematics. An excellent resource for graduate students, researchers, and practitioners, the work may be used as a graduate-level textbook for a special topics course or seminar on numerical methods in stochastic control.