Mathematical Statistics with Applications
science & ELSEVIER technology books
Companion Web Site:
http://www.elsevi...

This content was uploaded by our users and we assume good faith they have the permission to share this book. If you own the copyright to this book and it is wrongfully on our website, we offer a simple DMCA procedure to remove your content from our site. Start by pressing the button below!

Mathematical Statistics with Applications

science & ELSEVIER technology books

Companion Web Site:

http://www.elsevierdirect.com/companions/9780123748485

Mathematical Statistics with Applications by Kandethody M. Ramachandran and Chris P. Tsokos

Resources for Professors: • • • •

All figures from the book available as PowerPoint slides and as jpegs. Links to Web sites carefully chosen to supplement the content of the textbook. Online Student Solutions Manual is now available through separate purchase. Also available with purchase of Mathematical Statistics with Applications, password protected and activated upon registration, online Instructors’ Solutions Manual.

TOOLS

FOR

TEACHING NEEDS ALL textbooks.elsevier.com YOUR

ACADEMIC PRESS To adopt this book for course use, visit http://textbooks.elsevier.com

Mathematical Statistics with Applications

Kandethody M.Ramachandran Department of Mathematics and Statistics University of South Florida Tampa,FL

Chris P.Tsokos Department of Mathematics and Statistics University of South Florida Tampa,FL

AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO

Academic Press is an imprint of Elsevier

Elsevier Academic Press 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 84 Theobald’s Road, London WC1X 8RR, UK ∞ This book is printed on acid-free paper.

Copyright © 2009, Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail: [email protected] You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting “Customer Support” and then “Obtaining Permissions.” Library of Congress Cataloging-in-Publication Data Ramachandran, K. M. Mathematical statistics with applications / Kandethody M. Ramachandran, Chris P. Tsokos. p. cm. ISBN 978-0-12-374848-5 (hardcover : alk. paper) 1. Mathematical statistics. 2. Mathematical statistics—Data processing. I. Tsokos, Chris P. II. Title. QA276.R328 2009 519.5–dc22 2008044556 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library. ISBN 13: 978-0-12-374848-5 For all information on all Elsevier Academic Press publications visit our Web site at www.elsevierdirect.com

Printed in the United States of America 09 10 9 8 7 6 5 4 3 2 1

Dedicated to our families: Usha, Vikas, Vilas, and Varsha Ramachandran and Debbie, Matthew, Jonathan, and Maria Tsokos

This page intentionally left blank

Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix About the Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiii

CHAPTER 1 Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Types of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Sampling Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Errors in Sample Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Sample Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Graphical Representation of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Numerical Description of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.1 Numerical Measures for Grouped Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.2 Box Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Computers and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 3 3 5 8 11 12 13 26 30 33 39 40 41 41 46 47 51

CHAPTER 2 Basic Concepts from Probability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.1 2.2 2.3 2.4 2.5 2.6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Random Events and Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Counting Techniques and Calculation of Probabilities . . . . . . . . . . . . . . . . . . . . . . . . The Conditional Probability, Independence, and Bayes’ Rule . . . . . . . . . . . . . . . . Random Variables and Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Moments and Moment-Generating Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.1 Skewness and Kurtosis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Computer Examples (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.1 Minitab Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

54 55 63 71 83 92 98 107 108 109 110 110 112

vii

viii Contents

CHAPTER 3 Additional Topics in Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Special Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 The Binomial Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Poisson Probability Distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Uniform Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Normal Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5 Gamma Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Joint Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Covariance and Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Functions of Random Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Method of Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 The pdf of Y = g(X), Where g Is Differentiable and Monotone Increasing or Decreasing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Probability Integral Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Functions of Several Random Variables: Method of Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 Transformation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Limit Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Computer Examples (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

114 114 114 119 122 125 131 141 148 154 154 156 157 158 159 163 173 175 175 177 178 180

CHAPTER 4 Sampling Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Finite Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Sampling Distributions Associated with Normal Populations. . . . . . . . . . . . . . . . . 4.2.1 Chi-Square Distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Student t-Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 F-Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Order Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Large Sample Approximations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 The Normal Approximation to the Binomial Distribution . . . . . . . . . . . 4.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

184 187 191 192 198 202 207 212 213 218 219 219 219 219 221

Contents ix

CHAPTER 5 Point Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 5.1 5.2 5.3 5.4

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Method of Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Method of Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Some Desirable Properties of Point Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Unbiased Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Sufﬁciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Other Desirable Properties of a Point Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Efﬁciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.3 Minimal Sufﬁciency and Minimum-Variance Unbiased Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

226 227 235 246 247 252 266 266 270 277 282 283 285

CHAPTER 6 Interval Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 A Method of Finding the Conﬁdence Interval: Pivotal Method . . . . . . 6.2 Large Sample Conﬁdence Intervals: One Sample Case . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Conﬁdence Interval for Proportion, p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Margin of Error and Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Small Sample Conﬁdence Intervals for μ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 A Conﬁdence Interval for the Population Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conﬁdence Interval Concerning Two Population Parameters . . . . . . . . . . . . . . . . . 6.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

292 293 300 302 303 310 315 321 330 330 330 332 333 334

CHAPTER 7 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Sample Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 The Neyman–Pearson Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Likelihood Ratio Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Hypotheses for a Single Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 The p-Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Hypothesis Testing for a Single Parameter. . . . . . . . . . . . . . . . . . . . . . . . . . . .

338 346 349 355 361 361 363

x Contents

7.5 Testing of Hypotheses for Two Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 Independent Samples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Dependent Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Chi-Square Tests for Count Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Testing the Parameters of Multinomial Distribution: Goodness-of-Fit Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.2 Contingency Table: Test for Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.3 Testing to Identify the Probability Distribution: Goodness-of-Fit Chi-Square Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

372 373 382 388 390 392 395 399 399 400 403 405 408

CHAPTER 8 Linear Regression Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 The Simple Linear Regression Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 The Method of Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Derivation of βˆ 0 and βˆ 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.3 Quality of the Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.4 Properties of the Least-Squares Estimators for the Model Y = β0 + β1 x + ε. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.5 Estimation of Error Variance σ 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Inferences on the Least Squares Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Analysis of Variance (ANOVA) Approach to Regression . . . . . . . . . . . . 8.4 Predicting a Particular Value of Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Correlation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Matrix Notation for Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.1 ANOVA for Multiple Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Regression Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

412 413 415 416 421 422 425 428 434 437 440 445 449 451 454 455 455 457 458 461

CHAPTER 9 Design of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 9.2 Concepts from Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 9.2.1 Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467

Contents xi

9.2.2

Fundamental Principles: Replication, Randomization, and Blocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Some Speciﬁc Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Factorial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 One-Factor-at-a-Time Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Full Factorial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Fractional Factorial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Optimal Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Choice of Optimal Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 The Taguchi Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.2 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

471 474 483 483 485 486 487 487 489 493 494 494 494 497

CHAPTER 10 Analysis of Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Analysis of Variance Method for Two Treatments (Optional) . . . . . . . . . . . . . . . . . 10.3 Analysis of Variance for Completely Randomized Design . . . . . . . . . . . . . . . . . . . . 10.3.1 The p-Value Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Testing the Assumptions for One-Way ANOVA . . . . . . . . . . . . . . . . . . . . . . 10.3.3 Model for One-Way ANOVA (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Two-Way Analysis of Variance, Randomized Complete Block Design. . . . . . . 10.5 Multiple Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

500 501 510 515 517 522 526 536 543 543 543 546 548 554

CHAPTER 11 Bayesian Estimation and Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Bayesian Point Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2.1 Criteria for Finding the Bayesian Estimate . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Bayesian Conﬁdence Interval or Credible Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Bayesian Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Bayesian Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

560 562 569 579 584 588 596 596 596

xii Contents

CHAPTER 12 Nonparametric Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Nonparametric Conﬁdence Interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Nonparametric Hypothesis Tests for One Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.1 The Sign Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.2 Wilcoxon Signed Rank Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.3 Dependent Samples: Paired Comparison Tests . . . . . . . . . . . . . . . . . . . . . . . 12.4 Nonparametric Hypothesis Tests for Two Independent Samples. . . . . . . . . . . . . . 12.4.1 Median Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.2 The Wilcoxon Rank Sum Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5 Nonparametric Hypothesis Tests for k ≥ 2 Samples . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.1 The Kruskal–Wallis Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.2 The Friedman Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 12. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

600 601 606 607 611 617 620 620 625 630 631 634 640 642 642 646 648 652

CHAPTER 13 Empirical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 The Jackknife Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 An Introduction to Bootstrap Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 Bootstrap Conﬁdence Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 The Expectation Maximization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Introduction to Markov Chain Monte Carlo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.1 Metropolis Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.2 The Metropolis–Hastings Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.3 Gibbs Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.4 MCMC Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7.1 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 13. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

658 658 663 667 669 681 685 688 692 695 697 698 699 699

CHAPTER 14 Some Issues in Statistical Applications: An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Graphical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3 Outliers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4 Checking Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.1 Checking the Assumption of Normality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.2 Data Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

702 702 708 713 714 716

Contents xiii

14.4.3 Test for Equality of Variances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.4 Test of Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 Modeling Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.1 A Simple Model for Univariate Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.2 Modeling Bivariate Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6 Parametric versus Nonparametric Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7 Tying It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

719 724 727 727 730 733 735 746

Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747 A.I A.II A.III A.IV

Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Review of Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Common Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Probability Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

747 751 757 759

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 799 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803

This page intentionally left blank

Preface This textbook is of an interdisciplinary nature and is designed for a two- or one-semester course in probability and statistics, with basic calculus as a prerequisite. The book is primarily written to give a sound theoretical introduction to statistics while emphasizing applications. If teaching statistics is the main purpose of a two-semester course in probability and statistics, this textbook covers all the probability concepts necessary for the theoretical development of statistics in two chapters, and goes on to cover all major aspects of statistical theory in two semesters, instead of only a portion of statistical concepts. What is more, using the optional section on computer examples at the end of each chapter, the student can also simultaneously learn to utilize statistical software packages for data analysis. It is our aim, without sacriﬁcing any rigor, to encourage students to apply the theoretical concepts they have learned. There are many examples and exercises concerning diverse application areas that will show the pertinence of statistical methodology to solving real-world problems. The examples with statistical software and projects at the end of the chapters will provide good perspective on the usefulness of statistical methods. To introduce the students to modern and increasingly popular statistical methods, we have introduced separate chapters on Bayesian analysis and empirical methods. One of the main aims of this book is to prepare advanced undergraduates and beginning graduate students in the theory of statistics with emphasis on interdisciplinary applications. The audience for this course is regular full-time students from mathematics, statistics, engineering, physical sciences, business, social sciences, materials science, and so forth. Also, this textbook is suitable for people who work in industry and in education as a reference book on introductory statistics for a good theoretical foundation with clear indication of how to use statistical methods. Traditionally, one of the main prerequisites for this course is a semester of the introduction to probability theory. A working knowledge of elementary (descriptive) statistics is also a must. In schools where there is no statistics major, imposing such a background, in addition to calculus sequence, is very difﬁcult. Most of the present books available on this subject contain full one-semester material for probability and then, based on those results, continue on to the topics in statistics. Also, some of these books include in their subject matter only the theory of statistics, whereas others take the cookbook approach of covering the mechanics. Thus, even with two full semesters of work, many basic and important concepts in statistics are never covered. This book has been written to remedy this problem. We fuse together both concepts in order for students to gain knowledge of the theory and at the same time develop the expertise to use their knowledge in real-world situations. Although statistics is a very applied subject, there is no denying that it is also a very abstract subject. The purpose of this book is to present the subject matter in such a way that anyone with exposure to basic calculus can study statistics without spending two semesters of background preparation. To prepare students, we present an optional review of the elementary (descriptive) statistics in Chapter 1. All the probability material required to learn statistics is covered in two chapters. Students with a probability background can either review or skip the ﬁrst three chapters. It is also our belief that any statistics course is not complete without exposure to computational techniques. At

xv

xvi Preface

the end of each chapter, we give some examples of how to use Minitab, SPSS, and SAS to statistically analyze data. Also, at the end of each chapter, there are projects that will enhance the knowledge and understanding of the materials covered in that chapter. In the chapter on the empirical methods, we present some of the modern computational and simulation techniques, such as bootstrap, jackknife, and Markov chain Monte Carlo methods. The last chapter summarizes some of the steps necessary to apply the material covered in the book to real-world problems. The ﬁrst eight chapters have been class tested as a one-semester course for more than 3 years with ﬁve different professors teaching. The audience was junior- and senior-level undergraduate students from many disciplines who had had two semesters of calculus, most of them with no probability or statistics background. The feedback from the students and instructors was very positive. Recommendations from the instructors and students were very useful in improving the style and content of the book.

AIM AND OBJECTIVE OF THE TEXTBOOK This textbook provides a calculus-based coverage of statistics and introduces students to methods of theoretical statistics and their applications. It assumes no prior knowledge of statistics or probability theory, but does require calculus. Most books at this level are written with elaborate coverage of probability. This requires teaching one semester of probability and then continuing with one or two semesters of statistics. This creates a particular problem for non-statistics majors from various disciplines who want to obtain a sound background in mathematical statistics and applications. It is our aim to introduce basic concepts of statistics with sound theoretical explanations. Because statistics is basically an interdisciplinary applied subject, we offer many applied examples and relevant exercises from different areas. Knowledge of using computers for data analysis is desirable. We present examples of solving statistical problems using Minitab, SPSS, and SAS.

FEATURES ■

■ ■

■

■

During years of teaching, we observed that many students who do well in mathematics courses ﬁnd it difﬁcult to understand the concept of statistics. To remedy this, we present most of the material covered in the textbook with well-deﬁned step-by-step procedures to solve real problems. This clearly helps the students to approach problem solving in statistics more logically. The usefulness of each statistical method introduced is illustrated by several relevant examples. At the end of each section, we provide ample exercises that are a good mix of theory and applications. In each chapter, we give various projects for students to work on. These projects are designed in such a way that students will start thinking about how to apply the results they learned in the chapter as well as other issues they will need to know for practical situations. At the end of the chapters, we include an optional section on computer methods with Minitab, SPSS, and SAS examples with clear and simple commands that the student can use to analyze

Preface xvii

■

■

■

■

■

■ ■

data. This will help students to learn how to utilize the standard methods they have learned in the chapter to study real data. We introduce many of the modern statistical computational and simulation concepts, such as the jackknife and bootstrap methods, the EM algorithms, and the Markov chain Monte Carlo methods such as the Metropolis algorithm, the Metropolis–Hastings algorithm, and the Gibbs sampler. The Metropolis algorithm was mentioned in Computing in Science and Engineering as being among the top 10 algorithms having the “greatest inﬂuence on the development and practice of science and engineering in the 20th century.” We have introduced the increasingly popular concept of Bayesian statistics and decision theory with applications. A separate chapter on design of experiments, including a discussion on the Taguchi approach, is included. The coverage of the book spans most of the important concepts in statistics. Learning the material along with computational examples will prepare students to understand and utilize software procedures to perform statistical analysis. Every chapter contains discussion on how to apply the concepts and what the issues are related to applying the theory. A student’s solution manual, instructor’s manual, and data disk are provided. In the last chapter, we discuss some issues in applications to clearly demonstrate in a uniﬁed way how to check for many assumptions in data analysis and what steps one needs to follow to avoid possible pitfalls in applying the methods explained in the rest of this textbook.

This page intentionally left blank

Acknowledgments We express our sincere appreciation to our late colleague, co-worker, and dear friend, Professor A. N. V. Rao, for his helpful suggestions and ideas for the initial version of the subject textbook. In addition, we thank Bong-jin Choi and Yong Xu for their kind assistance in the preparation of the manuscript. Finally, we acknowledge our students at the University of South Florida for their useful comments and suggestions during the class testing of our book. To all of them, we are very thankful. K. M. Ramachandran Chris P. Tsokos Tampa, Florida

xix

This page intentionally left blank

About the Authors Kandethody M. Ramachandran is Professor of Mathematics and Statistics at the University of South Florida. He received his B.S. and M.S. degrees in Mathematics from the Calicut University, India. Later, he worked as a researcher at the Tata Institute of Fundamental Research, Bangalore center, at its Applied Mathematics Division. Dr. Ramachandran got his Ph.D. in Applied Mathematics from Brown University. His research interests are concentrated in the areas of applied probability and statistics. His research publications span a variety of areas such as control of heavy trafﬁc queues, stochastic delay equations and control problems, stochastic differential games and applications, reinforcement learning methods applied to game theory and other areas, software reliability problems, applications of statistical methods to microarray data analysis, and mathematical ﬁnance. Professor Ramachandran is extensively involved in activities to improve statistics and mathematics education. He is a recipient of the Teaching Incentive Program award at the University of South Florida. He is a member of the MEME Collaborative, which is a partnership among mathematics education, mathematics, and engineering faculty to address issues related to mathematics and mathematics education. He was also involved in the calculus reform efforts at the University of South Florida. Chris P. Tsokos is Distinguished University Professor of Mathematics and Statistics at the University of South Florida. Dr. Tsokos received his B.S. in Engineering Sciences/Mathematics, his M.A. in Mathematics from the University of Rhode Island, and his Ph.D. in Statistics and Probability from the University of Connecticut. Professor Tsokos has also served on the faculties at Virginia Polytechnic Institute and State University and the University of Rhode Island. Dr. Tsokos’s research has extended into a variety of areas, including stochastic systems, statistical models, reliability analysis, ecological systems, operations research, time series, Bayesian analysis, and mathematical and statistical modeling of global warming, among others. He is the author of more than 250 research publications in these areas. Professor Tsokos is the author of several research monographs and books, including Random Integral Equations with Applications to Life Sciences and Engineering, Probability Distribution: An Introduction to Probability Theory with Applications, Mainstreams of Finite Mathematics with Applications, Probability with the Essential Analysis, and Applied Probability Bayesian Statistical Methods with Applications to Reliability, among others. Dr. Tsokos is the recipient of many distinguished awards and honors, including Fellow of the American Statistical Association, USF Distinguished Scholar Award, Sigma Xi Outstanding Research Award, USF Outstanding Undergraduate Teaching Award, USF Professional Excellence Award, URI Alumni Excellence Award in Science and Technology, Pi Mu Epsilon, and election to the International Statistical Institute, among others.

xxi

This page intentionally left blank

Flow Chart This ﬂow chart gives some options on how to use the book in a one-semester or two-semester course. For a two-semester course, we recommend coverage of the complete textbook. However, Chapters 1, 9, and 14 are optional for both one- and two-semester courses and can be given as reading exercises. For a one-semester course, we suggest the following options: A, B, C, D.

One semester

Without probability background

With probability background

Ch. 2 A

B

C

D Ch. 3

Ch. 5

Ch. 5

Ch. 5

Ch. 6

Ch. 6

Ch. 6

Ch. 7

Ch. 7

Ch. 7

Ch. 8

Ch. 8

Ch. 8

Ch. 10

Ch.12

Ch. 11

Ch. 5 Ch. 4 Ch. 6 Ch. 5 Ch. 7 Ch. 6

Ch. 8 Ch. 11

Ch. 7 Ch. 12

Ch. 13

Ch. 13

Optional chapters

Ch. 12

Ch. 8

Ch. 10

Ch. 11

Ch. 12

xxiii

This page intentionally left blank

Chapter

1

Descriptive Statistics Objective: Review the basic concepts of elementary statistics. 1.1 Introduction 2 1.2 Basic Concepts 3 1.3 Sampling Schemes 8 1.4 Graphical Representation of Data 13 1.5 Numerical Description of Data 26 1.6 Computers and Statistics 39 1.7 Chapter Summary 40 1.8 Computer Examples 41 Projects for Chapter 1 51

Sir Ronald Aylmer Fisher (Source: http://www.stetson.edu/∼efriedma/periodictable/jpg/Fisher.jpg)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

1

2 CHAPTER 1 Descriptive Statistics

Sir Ronald Fisher F.R.S. (1890–1962) was one of the leading scientists of the 20th century who laid the foundations for modern statistics. As a statistician working at the Rothamsted Agricultural Experiment Station, the oldest agricultural research institute in the United Kingdom, he also made major contributions to Evolutionary Biology and Genetics. The concept of randomization and the analysis of variance procedures that he introduced are now used throughout the world. In 1922 he gave a new deﬁnition of statistics. Fisher identiﬁed three fundamental problems in statistics: (1) speciﬁcation of the type of population that the data came from; (2) estimation; and (3) distribution. His book Statistical Methods for Research Workers (1925) was used as a handbook for the methods for the design and analysis of experiments. Fisher also published the books titled The Design of Experiments (1935) and Statistical Tables (1947). While at the Agricultural Experiment Station he had conducted breeding experiments with mice, snails, and poultry, and the results he obtained led to theories about gene dominance and ﬁtness that he published in The Genetical Theory of Natural Selection (1930).

1.1 INTRODUCTION In today’s society, decisions are made on the basis of data. Most scientiﬁc or industrial studies and experiments produce data, and the analysis of these data and drawing useful conclusions from them become one of the central issues. The ﬁeld of statistics is concerned with the scientiﬁc study of collecting, organizing, analyzing, and drawing conclusions from data. Statistical methods help us to transform data to knowledge. Statistical concepts enable us to solve problems in a diversity of contexts, add substance to decisions, and reduce guesswork. The discipline of statistics stemmed from the need to place knowledge management on a systematic evidence base. Earlier works on statistics dealt only with the collection, organization, and presentation of data in the form of tables and charts. In order to place statistical knowledge on a systematic evidence base, we require a study of the laws of probability. In mathematical statistics we create a probabilistic model and view the data as a set of random outcomes from that model. Advances in probability theory enable us to draw valid conclusions and to make reasonable decisions on the basis of data. Statistical methods are used in almost every discipline, including agriculture, astronomy, biology, business, communications, economics, education, electronics, geology, health sciences, and many other ﬁelds of science and engineering, and can aid us in several ways. Modern applications of statistical techniques include statistical communication theory and signal processing, information theory, network security and denial of service problems, clinical trials, artiﬁcial and biological intelligence, quality control of manufactured items, software reliability, and survival analysis. The ﬁrst of these is to assist us in designing experiments and surveys. We desire our experiment to yield adequate answers to the questions that prompted the experiment or survey. We would like the answers to have good precision without involving a lot of expenditure. Statistically designed experiments facilitate development of robust products that are insensitive to changes in the environment and internal component variation. Another way that statistics assists us is in organizing, describing, summarizing, and displaying experimental data. This is termed descriptive statistics. A third use of statistics is in drawing inferences and making decisions based on data. For example, scientists may collect experimental data to prove or disprove an intuitive conjecture or hypothesis. Through the proper use of statistics we can conclude whether the hypothesis is valid or not. In the process of solving a real-life problem using statistics, the following three basic steps may be identiﬁed. First, consistent with the objective of the problem,

1.2 Basic Concepts 3

we identify the model—the appropriate statistical method. Then, we justify the applicability of the selected model to fulﬁll the aim of our problem. Last, we properly apply the related model to analyze the data and make the necessary decisions, which results in answering the question of our problem with minimum risk. Starting with Chapter 2, we will study the necessary background material to proceed with the development of statistical methods for solving real-world problems. In the present chapter we brieﬂy review some of the basic concepts of descriptive statistics. Such concepts will give us a visual and descriptive presentation of the problem under investigation. Now, we proceed with some basic deﬁnitions.

1.1.1 Data Collection One of the ﬁrst problems that a statistician faces is obtaining data. The inferences that we make depend critically on the data that we collect and use. Data collection involves the following important steps.

GENERAL PROCEDURE FOR DATA COLLECTION 1. Deﬁne the objectives of the problem and proceed to develop the experiment or survey. 2. Deﬁne the variables or parameters of interest. 3. Deﬁne the procedures of data-collection and measuring techniques. This includes sampling procedures, sample size, and data-measuring devices (questionnaires, telephone interviews, etc.).

Example 1.1.1 We may be interested in estimating the average household income in a certain community. In this case, the parameter of interest is the average income of a typical household in the community. To acquire the data, we may send out a questionnaire or conduct a telephone interview. Once we have the data, we may ﬁrst want to represent the data in graphical or tabular form to better understand its distributional behavior. Then we will use appropriate analytical techniques to estimate the parameter(s) of interest, in this case the average household income.

Very often a statistician is conﬁned to data that have already been collected, possibly even collected for other purposes. This makes it very difﬁcult to determine the quality of data. Planned collection of data, using proper techniques, is much preferred.

1.2 BASIC CONCEPTS Statistics is the science of data. This involves collecting, classifying, summarizing, organizing, analyzing, and interpreting data. It also involves model building. Suppose we wish to study household incomes in a certain neighborhood. We may decide to randomly select, say, 50 families and examine their household incomes. As another example, suppose we wish to determine the diameter of a rod, and we take 10 measurements of the diameter. When we consider these two examples, we note that in the ﬁrst case the population (the household incomes of all families in the neighborhood) really exists, whereas in the second, the population (set of all possible measurements of the diameter) is

4 CHAPTER 1 Descriptive Statistics

only conceptual. In either case we can visualize the totality of the population values, of which our sample data are only a small part. Thus we deﬁne a population to be the set of all measurements or objects that are of interest and a sample to be a subset of that population. The population acts as the sampling frame from which a sample is selected. Now we introduce some basic notions commonly used in statistics. Deﬁnition 1.2.1 A population is the collection or set of all objects or measurements that are of interest to the collector.

Example 1.2.1 Suppose we wish to study the heights of all female students at a certain university. The population will be the set of the measured heights of all female students in the university. The population is not the set of all female students in the university.

In real-world problems it is usually not possible to obtain information on the entire population. The primary objective of statistics is to collect and study a subset of the population, called a sample, to acquire information on some speciﬁc characteristics of the population that are of interest. Deﬁnition 1.2.2 The sample is a subset of data selected from a population. The size of a sample is the number of elements in it.

Example 1.2.2 We wish to estimate the percentage of defective parts produced in a factory during a given week (ﬁve days) by examining 20 parts produced per day. The parts will be examined each day at randomly chosen times. In this case “all parts produced during the week” is the population and the (100) selected parts for ﬁve days constitutes a sample.

Other common examples of sample and population are: Political polls: The population will be all voters, whereas the sample will be the subset of voters we poll. Laboratory experiment: The population will be all the data we could have collected if we were to repeat the experiment a large number of times (inﬁnite number of times) under the same conditions, whereas the sample will be the data actually collected by the one experiment. Quality control: The population will be the entire batch of items produced, say, by a machine or by a plant, whereas the sample will be the subset of items we tested. Clinical studies: The population will be all the patients with the same disease, whereas the sample will be the subset of patients used in the study. Finance: All common stock listed in stock exchanges such as the New York Stock Exchange, the American Stock Exchanges, and over-the-counter is the population. A collection of 20 randomly picked individual stocks from these exchanges will be a sample.

1.2 Basic Concepts 5

The methods consisting mainly of organizing, summarizing, and presenting data in the form of tables, graphs, and charts are called descriptive statistics. The methods of drawing inferences and making decisions about the population using the sample are called inferential statistics. Inferential statistics uses probability theory. Deﬁnition 1.2.3 A statistical inference is an estimate, a prediction, a decision, or a generalization about the population based on information contained in a sample. For example, we may be interested in the average indoor radiation level in homes built on reclaimed phosphate mine lands (many of the homes in west-central Florida are built on such lands). In this case, we can collect indoor radiation levels for a random sample of homes selected from this area, and use the data to infer the average indoor radiation level for the entire region. In the Florida Keys, one of the concerns is that the coral reefs are declining because of the prevailing ecosystems. In order to test this, one can randomly select certain reef sites for study and, based on these data, infer whether there is a net increase or decrease in coral reefs in the region. Here the inferential problem could be ﬁnding an estimate, such as in the radiation problem, or making a decision, such as in the coral reef problem. We will see many other examples as we progress through the book.

1.2.1 Types of Data Data can be classiﬁed in several ways. We will give two different classiﬁcations, one based on whether the data are measured on a numerical scale or not, and the other on whether the data are collected in the same time period or collected at different time periods. Deﬁnition 1.2.4 Quantitative data are observations measured on a numerical scale. Nonnumerical data that can only be classiﬁed into one of the groups of categories are said to be qualitative or categorical data.

Example 1.2.3 Data on response to a particular therapy could be classiﬁed as no improvement, partial improvement, or complete improvement. These are qualitative data. The number of minority-owned businesses in Florida is quantitative data. The marital status of each person in a statistics class as married or not married is qualitative or categorical data. The number of car accidents in different U.S. cities is quantitative data. The blood group of each person in a community as O, A, B, AB is qualitative data.

Categorical data could be further classiﬁed as nominal data and ordinal data. Data characterized as nominal have data groups that do not have a speciﬁc order. An example of this could be state names, or names of the individuals, or courses by name. These do not need to be placed in any order. Data characterized as ordinal have groups that should be listed in a speciﬁc order. The order may be either increasing or decreasing. One example would be income levels. The data could have numeric values such as 1, 2, 3, or values such as high, medium, or low. Deﬁnition 1.2.5 Cross-sectional data are data collected on different elements or variables at the same point in time or for the same period of time.

6 CHAPTER 1 Descriptive Statistics

Example 1.2.4 The data in Table 1.1 represent U.S. federal support for the mathematical sciences in 1996, in millions of dollars (source: AMS Notices). This is an example of cross-sectional data, as the data are collected in one time period, namely in 1996.

Table 1.1 Federal Support for the Mathematical Sciences, 1996 Federal agency

Amount

National Science Foundation

91.70

DMS

85.29

Other MPS

4.00

Department of Defense

77.30

AFOSR

16.70

ARO

15.00

DARPA

22.90

NSA

2.50

ONR

20.20

Department of Energy

16.00

University Support National Laboratories Total, All Agencies

5.50 10.50 185.00

Deﬁnition 1.2.6 Time series data are data collected on the same element or the same variable at different points in time or for different periods of time.

Example 1.2.5 The data in Table 1.2 represent U.S. federal support for the mathematical sciences during the years 1995–1997, in millions of dollars (source: AMS Notices). This is an example of time series data, because they have been collected at different time periods, 1995 through 1997.

For an extensive collection of statistical terms and deﬁnitions, we can refer to many sources such as http://www.stats.gla.ac.uk/steps/glossary/index.html. We will give some other helpful Internet sources that may be useful for various aspects of statistics: http://www.amstat.org/ (American

1.2 Basic Concepts 7

Table 1.2 United States Federal Support for the Mathematical Sciences in Different Years Agency

1995

1996

1997

National Science Foundation

87.69

91.70

98.22

DMS

85.29

87.70

93.22

2.40

4.00

5.00

Department of Defense

77.40

77.30

67.80

AFOSR

17.40

16.70

17.10

ARO

15.00

15.00

13.00

DARPA

21.00

22.90

19.50

NSA

2.50

2.50

2.10

ONR

21.40

20.20

16.10

Department of Energy

15.70

16.00

16.00

University Support

6.20

5.50

5.00

National Laboratories

9.50

10.50

11.00

180.79

185.00

182.02

Other MPS

Total, All Agencies

Statistical Association), http://www.stat.uﬂ.edu (University of Florida statistics department), http://www.stats.gla.ac.uk/cti/ (collection of Web links to other useful statistics sites), http://www. statsoft.com/textbook/stathome.html (covers a wide range of topics, the emphasis is on techniques rather than concepts or mathematics), http://www.york.ac.uk/depts/maths/histstat/welcome.htm (some information about the history of statistics), http://www.isid.ac.in/ (Indian Statistical Institute), http://www.math.uio.no/nsf/web/index.htm (The Norwegian Statistical Society), http://www.rss.org.uk/ (The Royal Statistical Society), http://lib.stat.cmu.edu/ (an index of statistical software and routines). For energy-related statistics, refer to http://www.eia.doe.gov/. There are various other useful sites that you could explore based on your particular need.

EXERCISES 1.2 1.2.1.

Give your own examples for qualitative and quantitative data. Also, give examples for crosssectional and time series data.

1.2.2.

Discuss how you will collect different types of data. What inferences do you want to derive from each of these types of data?

1.2.3.

Refer to the data in Example 1.2.4. State a few questions that you can ask about the data. What inferences can you make by looking at these data?

8 CHAPTER 1 Descriptive Statistics

1.2.4.

Refer to the data in Example 1.2.5. Can you state a few questions that the data suggest? What inferences can you make by looking at these data?

1.3 SAMPLING SCHEMES In any statistical analysis, it is important that we clearly deﬁne the target population. The population should be deﬁned in keeping with the objectives of the study. When the entire population is included in the study, it is called a census study because data are gathered on every member of the population. In general, it is usually not possible to obtain information on the entire population because the population is too large to attempt a survey of all of its members, or it may not be cost effective. A small but carefully chosen sample can be used to represent the population. A sample is obtained by collecting information from only some members of the population. A good sample must reﬂect all the characteristics (of importance) of the population. Samples can reﬂect the important characteristics of the populations from which they are drawn with differing degrees of precision. A sample that accurately reﬂects its population characteristics is called a representative sample. A sample that is not representative of the population characteristics is called a biased sample. The reliability or accuracy of conclusions drawn concerning a population depends on whether or not the sample is properly chosen so as to represent the population sufﬁciently well. There are many sampling methods available. We mention a few commonly used simple sampling schemes. The choice between these sampling methods depends on (1) the nature of the problem or investigation, (2) the availability of good sampling frames (a list of all of the population members), (3) the budget or available ﬁnancial resources, (4) the desired level of accuracy, and (5) the method by which data will be collected, such as questionnaires or interviews. Deﬁnition 1.3.1 A sample selected in such a way that every element of the population has an equal chance of being chosen is called a simple random sample. Equivalently each possible sample of size n has an equal chance of being selected.

Example 1.3.1 For a state lottery, 52 identical Ping-Pong balls with a number from 1 to 52 painted on each ball are put in a clear plastic bin. A machine thoroughly mixes the balls and then six are selected. The six numbers on the chosen balls are the six lottery numbers that have been selected by a simple random sampling procedure.

SOME ADVANTAGES OF SIMPLE RANDOM SAMPLING 1. Selection of sampling observations at random ensures against possible investigator biases. 2. Analytic computations are relatively simple, and probabilistic bounds on errors can be computed in many cases. 3. It is frequently possible to estimate the sample size for a prescribed error level when designing the sampling procedure.

1.3 Sampling Schemes 9

Simple random sampling may not be effective in all situations. For example, in a U.S. presidential election, it may be more appropriate to conduct sampling polls by state, rather than a nationwide random poll. It is quite possible for a candidate to get a majority of the popular vote nationwide and yet lose the election. We now describe a few other sampling methods that may be more appropriate in a given situation. Deﬁnition 1.3.2 A systematic sample is a sample in which every Kth element in the sampling frame is selected after a suitable random start for the ﬁrst element. We list the population elements in some order (say alphabetical) and choose the desired sampling fraction. STEPS FOR SELECTING A SYSTEMATIC SAMPLE 1. Number the elements of the population from 1 to N. 2. Decide on the sample size, say n, that we need. 3. Choose K = N/n. 4. Randomly select an integer between 1 to K . 5. Then take every K th element.

Example 1.3.2 If the population has 1000 elements arranged in some order and we decide to sample 10% (i.e., N = 1000 and n = 100), then K = 1000/100 = 10. Pick a number at random between 1 and K = 10 inclusive, say 3. Then select elements numbered 3, 13, 23, . . . , 993.

Systematic sampling is widely used because it is easy to implement. If the list of population elements is in random order to begin with, then the method is similar to simple random sampling. If, however, there is a correlation or association between successive elements, or if there is some periodic structure, then this sampling method may introduce biases. Systematic sampling is often used to select a speciﬁed number of records from a computer ﬁle. Deﬁnition 1.3.3 A stratiﬁed sample is a modiﬁcation of simple random sampling and systematic sampling and is designed to obtain a more representative sample, but at the cost of a more complicated procedure. Compared to random sampling, stratiﬁed sampling reduces sampling error. A sample obtained by stratifying (dividing into nonoverlapping groups) the sampling frame based on some factor or factors and then selecting some elements from each of the strata is called a stratiﬁed sample. Here, a population with N elements is divided into s subpopulations. A sample is drawn from each subpopulation independently. The size of each subpopulation and sample sizes in each subpopulation may vary. STEPS FOR SELECTING A STRATIFIED SAMPLE 1. Decide on the relevant stratiﬁcation factors (sex, age, income, etc.). 2. Divide the entire population into strata (subpopulations) based on the stratiﬁcation criteria. Sizes of strata may vary.

10 CHAPTER 1 Descriptive Statistics

3. Select the requisite number of units using simple random sampling or systematic sampling from each subpopulation. The requisite number may depend on the subpopulation sizes.

Examples of strata might be males and females, undergraduate students and graduate students, managers and nonmanagers, or populations of clients in different racial groups such as African Americans, Asians, whites, and Hispanics. Stratiﬁed sampling is often used when one or more of the strata in the population have a low incidence relative to the other strata.

Example 1.3.3 In a population of 1000 children from an area school, there are 600 boys and 400 girls. We divide them into strata based on their parents’ income as shown in Table 1.3.

Table 1.3 Classiﬁcation of School Children Boys

Girls

Poor

120

240

Middle Class

150

100

Rich

330

60

This is stratiﬁed data.

Example 1.3.4 Refer to Example 1.3.3. Suppose we decide to sample 100 children from the population of 1000 (that is, 10% of the population). We also choose to sample 10% from each of the categories. For example, we would choose 12 (10% of 120) poor boys; 6 (10% of 60 rich girls) and so forth. This yields Table 1.4. This particular sampling method is called a proportional stratiﬁed sampling.

Table 1.4 Proportional Stratiﬁcation of School Children Boys Girls Poor

12

24

Middle Class

15

10

Rich

33

6

1.3 Sampling Schemes 11

SOME USES OF STRATIFIED SAMPLING 1. In addition to providing information about the whole population, this sampling scheme provides information about the subpopulations, the study of which may be of interest. For example, in a U.S. presidential election, opinion polls by state may be more important in deciding on the electoral college advantage than a national opinion poll. 2. Stratiﬁed sampling can be considerably more precise than a simple random sample, because the population is fairly homogeneous within each stratum but there is a sizable variation between the strata.

Deﬁnition 1.3.4 In cluster sampling, the sampling unit contains groups of elements called clusters instead of individual elements of the population. A cluster is an intact group naturally available in the ﬁeld. Unlike the stratiﬁed sample where the strata are created by the researcher based on stratiﬁcation variables, the clusters naturally exist and are not formed by the researcher for data collection. Cluster sampling is also called area sampling. To obtain a cluster sample, ﬁrst take a simple random sample of groups and then sample all elements within the selected clusters (groups). Cluster sampling is convenient to implement. However, because it is likely that units in a cluster will be relatively homogeneous, this method may be less precise than simple random sampling.

Example 1.3.5 Suppose we wish to select a sample of about 10% from all ﬁfth-grade children of a county. We randomly select 10% of the elementary schools assumed to have approximately the same number of ﬁfth-grade students and select all ﬁfth-grade children from these schools. This is an example of cluster sampling, each cluster being an elementary school that was selected.

Deﬁnition 1.3.5 Multiphase sampling involves collection of some information from the whole sample and additional information either at the same time or later from subsamples of the whole sample. The multiphase or multistage sampling is basically a combination of the techniques presented earlier.

Example 1.3.6 An investigator in a population census may ask basic questions such as sex, age, or marital status for the whole population, but only 10% of the population may be asked about their level of education or about how many years of mathematics and science education they had.

1.3.1 Errors in Sample Data Irrespective of which sampling scheme is used, the sample observations are prone to various sources of error that may seriously affect the inferences about the population. Some sources of error can be controlled. However, others may be unavoidable because they are inherent in the nature of the sampling process. Consequently, it is necessary to understand the different types of errors for a proper

12 CHAPTER 1 Descriptive Statistics

interpretation and analysis of the sample data. The errors can be classiﬁed as sampling errors and nonsampling errors. Nonsampling errors occur in the collection, recording and processing of sample data. For example, such errors could occur as a result of bias in selection of elements of the sample, poorly designed survey questions, measurement and recording errors, incorrect responses, or no responses from individuals selected from the population. Sampling errors occur because the sample is not an exact representative of the population. Sampling error is due to the differences between the characteristics of the population and those of a sample from the population. For example, we are interested in the average test score in a large statistics class of size, say, 80. A sample of size 10 grades from this resulted in an average test score of 75. If the average test for the entire 80 students (the population) is 72, then the sampling error is 75 − 72 = 3.

1.3.2 Sample Size In almost any sampling scheme designed by statisticians, one of the major issues is the determination of the sample size. In principle, this should depend on the variation in the population as well as on the population size, and on the required reliability of the results, that is, the amount of error that can be tolerated. For example, if we are taking a sample of school children from a neighborhood with a relatively homogeneous income level to study the effect of parents’ afﬂuence on the academic performance of the children, it is not necessary to have a large sample size. However, if the income level varies a great deal in the feeding area of the school, then we will need a larger sample size to achieve the same level of reliability. In practice, another inﬂuencing factor is the available resources such as money and time. In later chapters, we present some methods of determining sample size in statistical estimation problems. The literature on sample survey methods is constantly changing with new insights that demand dramatic revisions in the conventional thinking. We know that representative sampling methods are essential to permit conﬁdent generalizations of results to populations. However, there are many practical issues that can arise in real-life sampling methods. For example, in sampling related to social issues, whatever the sampling method we employ, a high response rate must be obtained. It has been observed that most telephone surveys have difﬁculty in achieving response rates higher than 60%, and most face-to-face surveys have difﬁculty in achieving response rates higher than 70%. Even a well-designed survey may stop short of the goal of a perfect response rate. This might induce bias in the conclusions based on the sample we obtained. A low response rate can be devastating to the reliability of a study. We can obtain series of publications on surveys, including guidelines on avoiding pitfalls from the American Statistical Association (www.amstat.org). In this book, we deal mainly with samples obtained using simple random sampling.

EXERCISES 1.3 1.3.1.

Give your own examples for each of the sampling methods described in this section. Discuss the merits and limitations of each of these methods.

1.3.2.

Using the information obtained from the publications of the American Statistical Association (www.amstat.org), write a short report on how to collect survey data, and what the potential sources of error are.

1.4 Graphical Representation of Data 13

1.4 GRAPHICAL REPRESENTATION OF DATA The source of our statistical knowledge lies in the data. Once we obtain the sample data values, one way to become acquainted with them is to display them in tables or graphically. Charts and graphs are very important tools in statistics because they communicate information visually. These visual displays may reveal the patterns of behavior of the variables being studied. In this chapter, we will consider one-variable data. The most common graphical displays are the frequency table, pie chart, bar graph, Pareto chart, and histogram. For example, in the business world, graphical representations of data are used as statistical tools for everyday process management and improvements by decision makers (such as managers, and frontline staff) to understand processes, problems, and solutions. The purpose of this section is to introduce several tabular and graphical procedures commonly used to summarize both qualitative and quantitative data. Tabular and graphical summaries of data can be found in reports, newspaper articles, Web sites, and research studies, among others. Now we shall introduce some ways of graphically representing both qualitative and quantitative data. Bar graphs and Pareto charts are useful displays for qualitative data. Deﬁnition 1.4.1 A graph of bars whose heights represent the frequencies (or relative frequencies) of respective categories is called a bar graph.

Example 1.4.1 The data in Table 1.5 represent the percentages of price increases of some consumer goods and services for the period December 1990 to December 2000 in a certain city. Construct a bar chart for these data.

Table 1.5 Percentages of Price Increases of Some Consumer Goods and Services Medical Care

83.3%

Electricity

22.1%

Residential Rent

43.5%

Food

41.1%

Consumer Price Index

35.8%

Apparel & Upkeep

21.2%

Solution In the bar graph of Figure 1.1, we use the notations MC for medical care, El for electricity, RR for residential rent, Fd for food, CPI for consumer price index, and A & U for apparel and upkeep.

14 CHAPTER 1 Descriptive Statistics

100

Percentage

80 60 40 20 0

MC

EI

RR Fd Category

CPI

A&U

■ FIGURE 1.1 Percentage price increase of consumer goods.

Looking at Figure 1.1, we can identify where the maximum and minimum responses are located, so that we can descriptively discuss the phenomenon whose behavior we want to understand. For a graphical representation of the relative importance of different factors under study, one can use the Pareto chart. It is a bar graph with the height of the bars proportional to the contribution of each factor. The bars are displayed from the most numerous category to the least numerous category, as illustrated by the following example. A Pareto chart helps in separating signiﬁcantly few factors that have larger inﬂuence from the trivial many.

Example 1.4.2 For the data of Example 1.4.1, construct a Pareto chart.

Solution First, rewrite the data in decreasing order. Then create a Pareto chart by displaying the bars from the most numerous category to the least numerous category.

Looking at Figure 1.2, we can identify the relative importance of each category such as the maximum, the minimum, and the general behavior of the subject data. Vilfredo Pareto (1848–1923), an Italian economist and sociologist, studied the distributions of wealth in different countries. He concluded that about 20% of people controlled about 80% of a society’s wealth. This same distribution has been observed in other areas such as quality improvement: 80% of problems usually stem from 20% of the causes. This phenomenon has been termed the Pareto effect or 80/20 rule. Pareto charts are used to display the Pareto principle, arranging data so that the few vital factors that are causing most of the problems reveal themselves. Focusing improvement efforts on these few causes will have a larger impact and be more cost-effective than undirected efforts. Pareto charts are used in business decision making as a problem-solving and statistical tool

1.4 Graphical Representation of Data 15

Percentage increase

100 80 60 40 20 0

MC

RR

Fd CPI Category

EI

A&U

■ FIGURE 1.2 Pareto chart.

that ranks problem areas, or sources of variation, according to their contribution to cost or to total variation. Deﬁnition 1.4.2 A circle divided into sectors that represent the percentages of a population or a sample that belongs to different categories is called a pie chart. Pie charts are especially useful for presenting categorical data. The pie “slices” are drawn such that they have an area proportional to the frequency. The entire pie represents all the data, whereas each slice represents a different class or group within the whole. Thus, we can look at a pie chart and identify the various percentages of interest and how they compare among themselves. Most statistical software can create 3D charts. Such charts are attractive; however, they can make pieces at the front look larger than they really are. In general, a two-dimensional view of the pie is preferable.

Example 1.4.3 The combined percentages of carbon monoxide (CO) and ozone (O3 ) emissions from different sources are listed in Table 1.6.

Table 1.6 Combined Percentages of CO and O3 Emissions Transportation (T) 63%

Industrial process (I)

Fuel combustion (F)

Solid waste (S)

10%

14%

5%

Construct a pie chart.

Solution The pie chart is given in Figure 1.3.

Miscellaneous (M) 8%

16 CHAPTER 1 Descriptive Statistics

T(63.0%)

M(8.0%) S(5.0%) I(10.0%)

F(14.0%)

■ FIGURE 1.3 Pie chart for CO and O3 .

Deﬁnition 1.4.3 A stem-and-leaf plot is a simple way of summarizing quantitative data and is well suited to computer applications. When data sets are relatively small, stem-and-leaf plots are particularly useful. In a stem-and-leaf plot, each data value is split into a “stem” and a “leaf.” The “leaf” is usually the last digit of the number and the other digits to the left of the “leaf” form the “stem.” Usually there is no need to sort the leaves, although computer packages typically do. For more details, we refer the student to elementary statistics books. We illustrate this technique by an example.

Example 1.4.4 Construct a stem-and-leaf plot for the 20 test scores given below. 78 91

74 74

82 82

66 75

94 96

71 78

64 84

88 79

55 71

80 83

Solution At a glance, we see that the scores are distributed from the 50s through the 90s. We use the first digit of the score as the stem and the second digit as the leaf. The plot in Table 1.7 is constructed with stems in the vertical position.

Table 1.7 Stem-and-Leaf Display of 20 Exam Scores Stem

Leaves

5

5

6

6

4

7

8

4

1

4

5

8

8

2

8

0

2

4

3

9

4

1

6

9

1

1.4 Graphical Representation of Data 17

The stem-and-leaf plot condenses the data values into a useful display from which we can identify the shape and distribution of data such as the symmetry, where the maximum and minimum are located with respect to the frequencies, and whether they are bell shaped. This fact that the frequencies are bell shaped will be of paramount importance as we proceed to study inferential statistics. Also, note that the stem-and-leaf plot retains the entire data set and can be used only with quantitative data. Examples 1.8.1 and 1.8.6 explain how to obtain a stem-and-leaf plot using Minitab and SPSS, respectively. Refer to Section 1.8.3 for SAS commands to generate graphical representations of the data. A frequency table is a table that divides a data set into a suitable number of categories (classes). Rather than retaining the entire set of data in a display, a frequency table essentially provides only a count of those observations that are associated with each class. Once the data are summarized in the form of a frequency table, a graphical representation can be given through bar graphs, pie charts, and histograms. Data presented in the form of a frequency table are called grouped data. A frequency table is created by choosing a speciﬁc number of classes in which the data will be placed. Generally the classes will be intervals of equal length. The center of each class is called a class mark. The end points of each class interval are called class boundaries. Usually, there are two ways of choosing class boundaries. One way is to choose nonoverlapping class boundaries so that none of the data points will simultaneously fall in two classes. Another way is that for each class, except the last, the upper boundary is equal to the lower boundary of the subsequent class. When forming a frequency table this way, one or more data values may fall on a class boundary. One way to handle such a problem is to arbitrarily assign it one of the classes or to ﬂip a coin to determine the class into which to place the observation at hand. Deﬁnition 1.4.4 Let fi denote the frequency of the class i and let n be sum of all frequencies. Then the relative frequency for the class i is deﬁned as the ratio fi /n. The cumulative relative frequency for the class i is deﬁned by ik=1 fk /n. The following example illustrates the foregoing discussion.

Example 1.4.5 The following data give the lifetime of 30 incandescent light bulbs (rounded to the nearest hour) of a particular type. 872 1150 868

931 987 996

1146 958 1102

1079 1149 1130

915 1057 1002

879 1082 990

863 1053 1052

1112 1048 1116

979 1118 1119

1120 1088 1028

Construct a frequency, relative frequency, and cumulative relative frequency table.

Solution Note that there are n = 30 observations and that the largest observation is 1150 and the smallest one is 865 with a range of 285. We will choose six classes each with a length of 50.

18 CHAPTER 1 Descriptive Statistics

Class

Frequency

Relative frequency

fi

f i fi

Cumulative relative frequency i f k k=1 n

50−900

4

4/30

4/30

900−950

2

2/30

6/30

950−1000

5

5/30

11/30

1000−1050

3

3/30

14/30

1050−1100

6

6/30

20/30

1100−1150

10

10/30

30/30

When data are quantitative in nature and the number of observations is relatively large, and there are no natural separate categories or classes, we can use a histogram to simplify and organize the data. Deﬁnition 1.4.5 A histogram is a graph in which classes are marked on the horizontal axis and either the frequencies, relative frequencies, or percentages are represented by the heights on the vertical axis. In a histogram, the bars are drawn adjacent to each other without any gaps. Histograms can be used only for quantitative data. A histogram compresses a data set into a compact picture that shows the location of the mean and modes of the data and the variation in the data, especially the range. It identiﬁes patterns in the data. This is a good aggregate graph of one variable. In order to obtain the variability in the data, it is always a good practice to start with a histogram of the data. The following steps can be used as a general guideline to construct a frequency table and produce a histogram. GUIDELINE FOR THE CONSTRUCTION OF A FREQUENCY TABLE AND HISTOGRAM 1. Determine the maximum and minimum values of the observations. The range, R = maximum value − minimum value. 2. Select from ﬁve to 20 classes that in general are nonoverlapping intervals of equal length, so as to cover the entire range of data. The goal is to use enough classes to show the variation in the data, but not so many that there are only a few data points in many of the classes. The class width should be slightly larger than the ratio Largest value − Smallest value . Number of classes 3. The ﬁrst interval should begin a little below the minimum value, and the last interval should end a little above the maximum value. The intervals are called class intervals and the boundaries are called class boundaries. The class limits are the smallest and the largest data values in the class. The class mark is the midpoint of a class.

1.4 Graphical Representation of Data 19

4. None of the data values should fall on the boundaries of the classes. 5. Construct a table (frequency table) that lists the class intervals, a tabulation of the number of measurements in each class (tally), the frequency fi of each class, and, if needed, a column with relative frequency, fi /n, where n is the total number of observations. 6. Draw bars over each interval with heights being the frequencies (or relative frequencies).

Let us illustrate implementing these steps in the development of a histogram for the data given in the following example.

Example 1.4.6 The following data refer to a certain type of chemical impurity measured in parts per million in 25 drinkingwater samples randomly collected from different areas of a county. 11 24 35

19 31 18

24 16 24

30 23 18

12 25 27

20 26

25 32

29 17

15 22

21 26

(a) Make a frequency table displaying class intervals, frequencies, relative frequencies, and percentages. (b) Construct a frequency histogram.

Solution (a) We will use five classes. The maximum and minimum values in the data set are 35 and 11. Hence the class width is (35 − 11)/5 = 4.8 5. Hence, we shall take the class width to be 5. The lower boundary of the first class interval will be chosen to be 10.5. With five classes, each of width 5, the upper boundary of the fifth class becomes 35.5. We can now construct the frequency table for the data. Class

Class interval

fi = frequency

Relative frequency

Percentage

1

10.5 − 15.5

3

3/25 = 0.12

12

2

15.5 − 20.5

6

6/25 = 0.24

24

3

20.5 − 25.5

8

8/25 = 0.32

32

4

25.5 − 30.5

5

5/25 = 0.20

20

5

30.5 − 35.5

3

3/25 = 0.12

12

(b) We can generate a histogram as in Figure 1.4.

From the histogram we should be able to identify the center (i.e., the location) of the data, spread of the data, skewness of the data, presence of outliers, presence of multiple modes in the data, and whether the data can be capped with a bell-shaped curve. These properties provide indications of the

Frequency

20 CHAPTER 1 Descriptive Statistics

9 8 7 6 5 4 3 2 1 0

10.5

15.5

20.5 25.5 Data interval

30.5

35.5

■ FIGURE 1.4 Frequency histogram of impurity data.

proper distributional model for the data. Examples 1.8.2 and 1.8.7 explain how to obtain histograms using Minitab and SPSS, respectively.

EXERCISES 1.4 1.4.1.

According to the recent U.S. Federal Highway Administration Highway Statistics, the percentages of freeways and expressways in various road mileage–related highway pavement conditions are as follows: Poor 10%, Mediocre 32%, Fair 22%, Good 21%, and Very good 15%. (a) Construct a bar graph. (b) Construct a pie chart.

1.4.2.

More than 75% of all species that have been described by biologists are insects. Of the approximately 2 million known species, only about 30,000 are aquatic in any life stage. The data in Table 1.4.1 give proportion of total species by insect order that can survive exposure to salt (source: http://entomology.unl.edu/marine_insects/marineinsects.htm).

Table 1.4.1 Species

Percentage

Species

Percentage

Coleoptera

26%

Odonata

3%

Diptera

35%

Thysanoptera

3%

Hemiptera

15%

Lepidoptera

1%

Orthoptera

6%

Other

6%

Collembola

5%

1.4 Graphical Representation of Data 21

(a) Construct a bar graph. (b) Construct a Pareto chart. (c) Construct a pie chart. 1.4.3.

The data in Table 1.4.2 are presented to illustrate the role of renewable energy consumption in the U.S. energy supply in 2007 (source: http://www.eia.doe.gov/fuelrenewable.html). Renewable energy consists of biomass, geothermal energy, hydroelectric energy, solar energy, and wind energy.

Table 1.4.2 Source

Percentage

Coal

22%

Natural Gas

23%

Nuclear Electric Power

8%

Petroleum

40%

Renewable Energy

7%

(a) Construct a bar graph. (b) Construct a Pareto chart. (c) Construct a pie chart. 1.4.4.

A litter is a group of babies born from the same mother at the same time. Table 1.4.3 gives some examples of different mammals and their average litter size (source: http:// www.saburchill.com/chapters/chap0032.html).

Table 1.4.3 Species

Litter size

Bat

1

Dolphin

1

Chimpanzee

1

Lion

3

Hedgehog

5

Red Fox

6

Rabbit

6

Black Rat

11

22 CHAPTER 1 Descriptive Statistics

(a) Construct a bar graph. (b) Construct a Pareto chart. 1.4.5.

The following data give the letter grades of 20 students enrolled in a statistics course. A C

B D

F B

A A

C B

C A

D F

A B

B C

F A

(a) Construct a bar graph. (b) Construct a pie chart. 1.4.6.

According to the U.S. Bureau of Labor Statistics (BLS), the median weekly earnings of fulltime wage and salary workers by age for the third quarter of 1998 is given in Table 1.4.4.

Table 1.4.4 16 to 19 years

$260

20 to 24 years

$334

25 to 34 years

$498

35 to 44 years

$600

45 to 54 years

$628

55 to 64 years

$605

65 years and over

$393

Construct a pie chart and bar graph for these data and interpret. Also, construct a Pareto chart. 1.4.7.

The data in Table 1.4.5 are a breakdown of 18,930 workers in a town according to the type of work. Construct a pie chart and bar graph for these data and interpret.

1.4.8.

The data in Table 1.4.6 represent the number (in millions) of adults and children living with HIV/AIDS by the end of 2000 according to the region of the world (source: http://w3.whosea.org/hivaids/factsheet.htm). Construct a bar graph for these data. Also, construct a Pareto chart and interpret.

1.4.9.

The data in Table 1.4.7 give the life expectancy at birth, in years, from 1900 through 2000 (source: National Center for Health Statistics). Construct a bar graph for these data.

1.4.10.

Dolphins are usually identiﬁed by the shape and pattern of notches and nicks on their dorsal ﬁn. Individual dolphins are cataloged by classifying the ﬁn based on location of distinguishing marks. When a dolphin is sighted its picture can then be compared to the catalog of

1.4 Graphical Representation of Data 23

Table 1.4.5 Mining

58

Construction

1161

Manufacturing

2188

Transportation and Public Utilities

821

Wholesale Trade

657

Retail Trade

7377

Finance, Insurance, and Real Estate Services Total

890 5778 18,930

Table 1.4.6 Country Sub-Saharan Africa

Adults and children living with HIV/AIDS (in millions) 25.30

North Africa and Middle East

0.40

South and Southeast Asia

5.80

East Asia and Paciﬁc

0.64

Latin America

1.40

Caribbean

0.39

Eastern Europe and Central Asia

0.70

Western Europe

0.54

North America

0.92

Australia and New Zealand

0.15

dolphins in the area, and if a match is found, the dolphin can be recorded as resighted. These methods of mark-resight are for developing databases regarding the life history of individual dolphins. From these databases we can calculate the levels of association between dolphins, population estimates, and general life history parameters such as birth and survival rates.

24 CHAPTER 1 Descriptive Statistics

Table 1.4.7 Year

Life expectancy

1900

47.3

1960

69.7

1980

73.7

1990

75.4

2000

77.0

The data in Table 1.4.8 represent frequently resighted individuals (as of January 2000) at a particular location (source: http://www.eckerd.edu/dolphinproject/biologypr.html).

Table 1.4.8 Hammer (adult female)

59

Mid Button Flag (adult female)

41

Luseal (adult female)

31

84 Lookalike (adult female)

20

Construct a bar graph for these data. 1.4.11.

The data in Table 1.4.9 give death rates (per 100,000 population) for 10 leading causes in 1998 (source: National Center for Health Statistics, U.S. Deptartment of Health and Human Services). (a) Construct a bar graph. (b) Construct a Pareto chart.

1.4.12.

In a ﬁscal year, a city collected $32.3 million in revenues. City spending for that year is expected to be nearly the same, with no tax increase projected. Expenditure: Reserves 0.7%, capital outlay 29.7%, operating expenses 28.9%, debt service 3.2%, transfers 5.1%, personal services 32.4%. Revenues: Property taxes 10.2%, utility and franchise taxes 11.3%, licenses and permits 1%, inter governmental revenue 10.1%, charges for services 28.2%, ﬁnes and forfeits 0.5%, interest and miscellaneous 2.7%, transfers and cash carryovers 36%. (a) Construct bar graphs for expenditure and revenues and interpret. (b) Construct pie charts for expenditure and revenues and interpret.

1.4 Graphical Representation of Data 25

Table 1.4.9 Cause

Death rate

Accidents and Adverse Effects

34.5

Chronic Liver Disease and Cirrhosis

9.7

Chronic Obstructive Lung Diseases and Allied Conditions

42.3

Cancer

199.4

Diabetes Mellitus

23.9

Heart Disease

268.0

Kidney Disease

1.4.13.

Pneumonia and Inﬂuenza

35.1

Stroke

58.5

Suicide

10.8

Construct a histogram for the 24 examination scores given next. 78 91

1.4.14.

9.7

74 74

82 82

66 75

94 96

71 78

64 84

88 79

55 71

80 83

73 78

86 79

The following table gives radon concentration in pCi/liter obtained from 40 houses in a certain area. 2.9 7.9 15.9 6.2

0.6 13.5 17.1 2.8 3.8 16.0 2.1 6.4 17.2 0.5 13.7 11.5 2.9 3.6 6.1 8.8 2.2 9.4 8.8 9.8 11.5 12.3 3.7 8.9 13.0 7.9 11.7 6.9 12.8 13.7 2.7 3.5 8.3 15.9 5.1 6.0

(a) Construct a stem-and-leaf display. (b) Construct a frequency histogram and interpret. (c) Construct a pie chart and interpret. 1.4.15.

The following data give the mean of SAT Mathematics scores by state for 1999 for a randomly selected 20 states (source: The World Almanac and Book of Facts 2000). 558 568

503 553

565 510

572 525

546 595

517 502

542 526

(a) Construct a stem-and-leaf display and interpret. (b) Construct a frequency histogram and interpret. (c) Construct a pie chart and interpret.

605 475

493 506

499 568

26 CHAPTER 1 Descriptive Statistics

1.4.16.

A sample of 25 measurements is given here: 9 31 26

28 23 20

14 16 16

29 26 14

21 22 21

27 17

15 19

23 24

23 21

10 20

(a) Make a frequency table displaying class intervals, frequencies, relative frequencies, and percentages. (b) Construct a frequency histogram and interpret.

1.5 NUMERICAL DESCRIPTION OF DATA In the previous section we looked at some graphical and tabular techniques for describing a data set. We shall now consider some numerical characteristics of a set of measurements. Suppose that we have a sample with values x1 , x2 , . . . , xn . There are many characteristics associated with this data set, for example, the central tendency and variability. A measure of the central tendency is given by the sample mean, median, or mode, and the measure of dispersion or variability is usually given by the sample variance or sample standard deviation or interquartile range. Deﬁnition 1.5.1 Let x1 , x2 , . . . , xn be a set of sample values. Then the sample mean (or empirical mean) x is deﬁned by 1 xi . n n

x=

i=1

The sample variance is deﬁned by s2 =

n 1 (xi − x)2 . (n − 1) i=1

The sample standard deviation is s=

s2 .

The sample variance s2 and the sample standard deviation s both are measures of the variability or “scatteredness” of data values around the sample mean x. Larger the variance, more is the spread. We note that s2 and s are both nonnegative. One question we may ask is “why not just take the sum of the differences (xi − x) as a measure of variation?” The answer lies in the following result which shows that if we add up all deviations about the sample mean, we always get a zero value. Theorem 1.5.1 For a given set of measurements x1 , x2 , . . . , xn , let x be the sample mean. Then n i=1

(xi − x) = 0.

1.5 Numerical Description of Data 27

Proof. Since x = (1/n)

n

i=1 xi ,

n

we have n

i=1 xi

= nx. Now

(xi − x ) =

i=1

n

xi −

i=1

n

x

i=1

= nx − nx = 0.

Thus although there may be a large variation in the data values, ni=1 (xi − x) as a measure of spread would always be zero, implying no variability. So it is not useful as a measure of variability.

Sometimes we can simplify the calculation of the sample variance s2 by using the following computational formula: ⎡

2 ⎤ n n 1 2 ⎣ xi − n xi ⎦ s2 =

i=1

i=1

(n − 1)

.

If the data set has a large variation with some extreme values (called outliers), the mean may not be a very good measure of the center. For example, average salary may not be a good indicator of the ﬁnancial well-being of the employees of a company if there is a huge difference in pay between support personnel and management personnel. In that case, one could use the median as a measure of the center, roughly 50% of data fall below and 50% above. The median is less sensitive to extreme data values. Deﬁnition 1.5.2 For a data set, the median is the middle number of the ordered data set. If the data set has an even number of elements, then the median is the average of the middle two numbers. The lower quartile is the middle number of the half of the data below the median, and the upper quartile is the middle number of the half of the data above the median. We will denote Q1 = lower quartile Q2 = M = middle quartile (median) Q3 = upper quartile

The difference between the quartiles is called interquartile range (IQR). IQR = Q3 − Q1 .

A possible outlier (mild outlier) will be any data point that lies below Q1 − 1.5(IQR) or above Q3 + 1.5(IQR).

Note that the IQR is unaffected by the positions of those observations in the smallest 25% or the largest 25% of the data. Mode is another commonly used measure of central tendency. A mode indicates where the data tend to concentrate most.

28 CHAPTER 1 Descriptive Statistics

Deﬁnition 1.5.3 Mode is the most frequently occurring member of the data set. If all the data values are different, then by deﬁnition, the data set has no mode.

Example 1.5.1 The following data give the time in months from hire to promotion to manager for a random sample of 25 software engineers from all software engineers employed by a large telecommunications ﬁrm. 5 22 47

7 21 67

229 25 69

453 23 192

12 24 125

14 34

18 37

14 34

14 49

483 64

Calculate the mean, median, mode, variance, and standard deviation for this sample.

Solution The sample mean is 1 xi = 83.28 months. n n

x=

i=1

To obtain the median, first arrange the data in ascending order: 5 24 125

7 25 192

12 34 229

14 34 453

14 37 483

14 47

18 49

21 64

22 67

23 69

Now the median is the thirteenth number which is 34 months. Since 14 occurs most often (thrice), the mode is 14 months. The sample variance is s2 =

n 1 (xi − x)2 n−1 i=1

1 (5 − 83.28)2 + · · · + (125 − 83.28)2 = 24 = 16,478.

√ and the sample standard deviation is, s = s2 = 128.36 months. Thus, we have sample mean x = 83.28 months, median = 34 months, and mode = 14 months. Note that the mean is very much different from the other two measures of center because of a few large data values. Also, the sample variance s2 = 16,478 months, and the sample standard deviation s = 128.36 months.

Example 1.5.2 For the data of Example 1.5.1, ﬁnd lower and upper quartiles, median, and interquartile range (IQR). Check for any outliers.

1.5 Numerical Description of Data 29

Solution Arrange the data in an ascending order. 5 24 125

7 25 192

12 34 229

14 34 453

14 37 483

14 47

18 49

21 64

22 67

23 69

Then the median M is the middle (13th) data value, M = Q2 = 34. The lower quartile is the middle number below the median, Q1 = [(14 + 18)/2] = 16. The upper quartile, Q3 = [(67 + 69)/2] = 68. The interquartile range, (IQR) = Q3 − Q1 = 68 − 16 = 52. To test for outliers, compute Q1 − 1.5(IQR) = 16 − 1.5(52) = −62 and Q3 + 1.5(IQR) = 68 + 1.5(52) = 146. Then all the data that fall above 146 are possible outliers. None is below −62. Therefore the outliers are 192, 229, 453, and 483.

We have remarked earlier that the mean as a measure of central location is greatly affected by the extreme values or outliers. A robust measure of central location (a measure that is relatively unaffected by outliers) is the trimmed mean. For 0 ≤ α ≤ 1, a 100α% trimmed mean is found as follows: Order the data, and then discard the lowest 100α% and the highest 100α% of the data values. Find the mean of the rest of the data values. We denote the 100α% trimmed mean by xα . We illustrate the trimmed mean concept in the following example.

Example 1.5.3 For the data set representing the number of children in a random sample of 10 families in a neighborhood, ﬁnd the 10% trimmed mean (α = 0.1). 1

2

2

3

2

3

9

1

6

2

1

2

2

2

2

3

3

6

9

Solution Arrange the data in ascending order. 1

The data set has 10 elements. Discarding the lowest 10% (10% of 10 is 1) and discarding the highest 10% of the data values, we obtain the trimmed data set as 1 2 2 2 2 3 3 6 The 10% trimmed mean is 1+2+2+2+2+3+3+6 = 2.6. x0.1 = 8 Note that the mean for the data in the previous example without removing any observations is 3.1, which is different from the trimmed mean.

30 CHAPTER 1 Descriptive Statistics

Examples 1.8.2 and 1.8.7 explain how to obtain a histogram using Minitab and SPSS, respectively. Example 1.8.9 demonstrates the SAS commands to obtain the descriptive statistics. Although standard deviation is a more popular method, there are other measures of dispersion such as average deviation or interquartile range. We have already seen the deﬁnition of interquartile range. The average deviation for a sample x1 , . . . , xn is deﬁned by n

Average deviation =

|xi − x|

i=1

.

n

Calculation of average deviation is simple and straightforward.

1.5.1 Numerical Measures for Grouped Data When we encounter situations where the data are grouped in the form of a frequency table (see Section 1.4), we no longer have individual data values. Hence, we cannot use the formulas in Deﬁnition 1.5.1. The following formulas will give approximate values for x and s2 . Let the grouped data have l classes, with mi being the midpoint and fi being the frequency of class i, i = 1, 2, . . . , l. Let n = li=1 fi . Deﬁnition 1.5.4 The mean for a sample of size n, 1 fi m i , n l

x=

i=1

where mi is the midpoint of the class i and fi is the frequency of the class i. Similarly the sample variance,

s2 =

n 1 fi (mi − x)2 = n−1 i=1

2 fi mi

2 mi fi − i n n−1

.

The following example illustrates how we calculate the sample mean for a grouped data.

Example 1.5.4 The grouped data in Table 1.8 represent the number of children from birth through the end of the teenage years in a large apartment complex. Find the mean, variance, and standard deviation for these data:

Table 1.8 Number of Children and Their Age Group Class Frequency

0–3

4–7

8–11

12–15

16–19

7

4

19

12

8

1.5 Numerical Description of Data 31

Solution For simplicity of calculation we create Table 1.9.

Table 1.9 Class

fi

mi

mi f i

m2i f i

0−3

7

1.5

10.5

15.75

4−7

4

5.5

22

121

8−11

19

9.5

180.5

1714.75

12−15

12

13.5

162

2187

16−19

8

17.5

140

n = 50

mi fi = 515

2450

m2i fi = 6488.5

The sample mean is x=

1 515 = 10.30. fi m i = n 50 i

The sample variance is

2

fi mi 2 2 i 6488.5 − (515) f − m i n 2 50 i = = 24.16. s = n−1 49 √ √ The sample standard deviation is s = s2 = 24.16 = 4.92.

Using the following calculations, we can also ﬁnd the median for grouped data. We only know that the median occurs in a particular class interval, but we do not know the exact location of the median. We will assume that the measures are spread evenly throughout this interval. Let L = lower class limit of the interval that contains the median n = total frequency Fb = cumulative frequencies for all classes before the median class fm = frequency of the class interval containing the median w = interval width of the interval that contains the median

Then the median for the grouped data is given by M =L+

We proceed to illustrate with an example.

w (0.5n − Fb ). fm

32 CHAPTER 1 Descriptive Statistics

Example 1.5.5 For the data of Example 1.5.4, ﬁnd the median.

Solution First develop Table 1.10.

Table 1.10 fi

Cumulative f i

0−3

7

7

0.14

4−7

4

11

0.22

8−11

19

30

0.6

12−15

12

42

0.84

16−19

8

50

1.00

Class

Cumulative f i /n

The ﬁrst interval for which the cumulative relative frequency exceeds 0.5 is the interval that contains the median. Hence the interval 8 to 11 contains the median. Therefore, L = 8, fm = 19, n = 50, w = 3, and Fb = 11. Then, the median is M =L+

3 w (0.5n − Fb ) = 8 + ((0.5)(50) − 11) = 10.211. fm 19

It is important to note that all the numerical measures we calculate for grouped data are only approximations to the actual values of the ungrouped data if they are available. One of the uses of the sample standard deviation will be clear from the following result, which is based on data following a bell-shaped curve. Such an indication can be obtained from the histogram or stem-and-leaf display. EMPIRICAL RULE When the histogram of a data set is “bell shaped” or “mound shaped,” and symmetric, the empirical rule states: 1. Approximately 68% of the data are in the interval (x − s, x + s). 2. Approximately 95% of the data are in the interval (x − 2s, x + 2s). 3. Approximately 99.7% of the data are in the interval (x − 3s, x + 3s).

The bell-shaped curve is called a normal curve and is discussed later in Chapter 3. A typical symmetric bell-shaped curve is given in Figure 1.5.

1.5 Numerical Description of Data 33

Normal distribution 0.4

1 sd

0.3

0.2 2 sd 0.1 3 sd 0.0 ⫺3

⫺2

⫺1

0 x

1

2

3

■ FIGURE 1.5 Bell-shaped curve.

1.5.2 Box Plots The sample mean or the sample standard deviation focuses on a single aspect of the data set, whereas histograms and stem-and-leaf displays express rather general ideas about data. A pictorial summary called a box plot (also called box-and-whisker plots) can be used to describe several prominent features of a data set such as the center, the spread, the extent and nature of any departure from symmetry, and identiﬁcation of outliers. Box plots are a simple diagrammatic representation of the ﬁve number summary: minimum, lower quartile, median, upper quartile, maximum. Example 1.8.4 illustrates the method of obtaining box plots using Minitab.

PROCEDURE TO CONSTRUCT A BOX PLOT 1. Draw a vertical measurement axis and mark Q1 , Q2 (median), and Q3 on this axis as shown in Figure 1.6. 2. Construct a rectangular box whose bottom edge lies at the lower quartile, Q1 and whose upper edge lies at the upper quartile, Q3 . 3. Draw a horizontal line segment inside the box through the median. 4. Extend the lines from each end of the box out to the farthest observation that is still within 1.5(IQR) of the corresponding edge. These lines are called whiskers. 5. Draw an open circle (or asterisks *) to identify each observation that falls between 1.5(IQR) and 3(IQR) from the edge to which it is closest; these are called mild outliers. 6. Draw a solid circle to identify each observation that falls more than 3(IQR) from the closest edge; these are called extreme outliers.

34 CHAPTER 1 Descriptive Statistics

Extreme outliers

3(IQR ) Mild outliers 1.5(IQR ) Whisker

Q3

Q2

Q1 Whisker 1.5(IQR ) Mild outliers 3(IQR )

Extreme outliers ■ FIGURE 1.6 A typical box-and-whiskers plot.

We illustrate the procedure with the following example.

Example 1.5.6 The following data identify the time in months from hire to promotion to chief pharmacist for a random sample of 25 employees from a certain group of employees in a large corporation of drugstores. 5 22 47

7 21 67

229 25 69

453 23 192

12 24 125

14 34

18 37

14 34

14 49

483 64

Construct a box plot. Do the data appear to be symmetrically distributed along the measurement axis?

Solution Referring to Example 1.5.2, we find that the median, Q2 = 34. The lower quartile is Q1 = 14+18 = 16. 2 = 68. The upper quartile is Q3 = 67+69 2 The interquartile range is IQR = 68 − 16 = 52.

1.5 Numerical Description of Data 35

To find the outliers, compute Q1 − 1.5(IQR) = 16 − 1.5(52) = −62 and Q3 + 1.5(IQR) = 68 + 1.5(52) = 146. Using these numbers, we follow the procedure outlined earlier to construct the box plot in Figure 1.7. The * in the box plot represents an outlier. The first horizontal line is the first quartile, the second is the median, and the third is the third quartile.

500 Months

400 300 200 100 0 ■ FIGURE 1.7 Box plot for months to promotion.

By examining the relative position of the median line (the middle line in Figure 1.7), we can test the symmetry of the data. For example, in Figure 1.7, the median line is closer to the lower quartile than the upper line, which suggests that the distribution is slightly nonsymmetric. Also, a look at this box plot shows the presence of two mild outliers and two extreme outliers.

EXERCISES 1.5 1.5.1.

The prices of 12 randomly chosen homes in dollars (approximated to nearest thousand) in a growing region of Tampa in the summer of 2002 are given below. 176

105

133

140

305

215

207

210

173

150

78

96

Find the mean and standard deviation of the sampled home prices from this area. 1.5.2.

The following is a sample of nine mortgage companies’ interest rates for 30-year home mortgages, assuming 5% down. 7.625

7.500

6.625

7.625

6.625

6.875

7.375

5.375

7.500

(a) Find the mean and standard deviation and interpret. (b) Find lower and upper quartiles, median, and interquartile range. Check for any outliers and interpret. 1.5.3.

For four observations, it is given that mean is 6, median is 4, and mode is 3. Find the standard deviation of this sample.

36 CHAPTER 1 Descriptive Statistics

1.5.4.

The data given below pertain to a random sample of disbursements of state highway funds (in millions of dollars), to different states. 1188 537

1050 519

2882 2523

2802 316

780 1117

1171 1578

685 261

(a) Find the mean, variance, and range for these data and interpret. (b) Find lower and upper quartiles, median and interquartile range. Check for any outliers and interpret. (c) Construct a box plot and interpret. 1.5.5.

Maximal static inspiratory pressure (PImax) is an index of respiratory muscle strength. The following data show the measure of PImax (cm H2 O) for 15 cystic ﬁbrosis patients. 105 135

80 105

115 45

95 115

100 40

85 115

90 95

70

(a) Find the lower and upper quartiles, median, and interquartile range. Check for any outliers and interpret. (b) Construct a box plot and interpret. (c) Are there any outliers? 1.5.6.

Compute the mean, variance, and standard deviation for the data in Table 1.5.1 (assume that the data belong to a sample).

Table 1.5.1 Class Frequency

1.5.7.

0–4

5–9

10–14

15–19

20–24

5

14

15

10

6

(a) For any grouped data with l classes with group frequencies fi , and class midpoints mi , show that l

fi (mi − x) = 0.

i=1

(b) Verify this result for the data given in Exercise 1.5.6. 1.5.8.

(a) Given the sample values x1 , x2 , . . . , xn , show that n i=1

(xi − x)2 =

n i=1

xi2 −

n

2 xi

i=1

n

(b) Verify the result of part (a) for the data of Exercise 1.5.5.

.

1.5 Numerical Description of Data 37

1.5.9.

The following are the closing prices of some securities that a mutual fund holds on a certain day: 10.25 5.31 11.25 13.13 43.25 45.00 40.06 28.56 32.00 25.44 22.50 30.00 53.50 29.87 32.00 28.87 (a) (b) (c) (d) (e)

1.5.10.

32.56 51.50 53.37 37.50

37.06 47.00 51.38 30.44

39.00 53.50 26.00 41.37

Find the mean, variance, and range for these data and interpret. Find lower and upper quartiles, median, and interquartile range. Check for any outliers. Construct a box plot and interpret. Construct a histogram. Locate on your histogram x, x ± s, x ± 2s, and x ± 3s. Count the data points in each of the intervals x ± s, x ± 2s, and x ± 3s and compare this with the empirical rule.

The radon concentration (in pCi/liter) data obtained from 40 houses in a certain area are given below. 2.9 7.9 15.9 6.2 (a) (b) (c) (d) (e)

1.5.11.

18.00 22.75 24.75 42.19

0.6 13.5 17.1 2.8 3.8 16.0 2.1 6.4 17.2 0.5 13.7 11.5 2.9 3.6 6.1 8.8 2.2 9.4 8.8 9.8 11.5 12.3 3.7 8.9 13.0 7.9 11.7 6.9 12.8 13.7 2.7 3.5 8.3 15.9 5.1 6.0

Find the mean, variance, and range for these data. Find lower and upper quartiles, median, and interquartile range. Check for any outliers. Construct a box plot. Construct a histogram and interpret. Locate on your histogram x ± s, x ± 2s, and x ± 3s. Count the data points in each of the intervals x, x ± s, x ± 2s, and x ± 3s. How do these counts compare with the empirical rule?

A random sample of 100 households’ weekly food expenditure represented by x from a particular city gave the following statistics:

xi = 11,000, and

xi 2 = 1,900,000.

(a) Find the mean and standard deviation for these data. (b) Assuming that the food expenditure of the households of an entire city of 400,000 will have a bell-shaped distribution, how many households of this city would you expect to fall in each of the intervals, x ± s, x ± 2s, and x ± 3s? 1.5.12.

The following numbers are the hours put in by 10 employees of company in a randomly selected week: 40

46

40

54

18

45

34

60

39

42

(a) Calculate the values of the three quartiles and the interquartile range. Also, calculate the mean and standard deviation and interpret.

38 CHAPTER 1 Descriptive Statistics (b) Verify for this data set that 10 i=1 (xi − x) = 0. (c) Construct a box plot. (d) Does this data set contain any outliers? 1.5.13.

For the following data: 6.3 7.0 4.5

2.9 2.8 4.5

4.5 4.3 5.7

1.1 5.3 0.5

1.8 2.9 6.2

4.0 8.3 3.7

1.2 4.4 0.9

3.1 2.8 2.4

2.0 3.1 3.0

4.0 5.6 3.5

(a) Find the mean, variance, and standard deviation. (b) Construct a frequency table with ﬁve classes. (c) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation for the frequency table constructed in part (b) and compare it to the results in part (a). 1.5.14.

In order to assess the protective immunizing activity of various whooping cough vaccines, suppose that 30 batches of different vaccines are tested on groups of children. Suppose that the following data give immunity percentage in home exposure values (IPHE values). 85 42 79

51 12 90

41 70 43

90 38 40

91 97 89

40 34 85

39 94 71

69 77 30

45 88 25

47 91 21

(a) Find the mean, variance, and standard deviation and interpret. (b) Construct a frequency table with ﬁve classes. (c) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation for the table in part (b) and compare it to the results in part (a). 1.5.15.

The grouped data in Table 1.5.2 give the number of births by age group of mothers between ages 10 and 39 in a certain state in 2000. Find the median for this grouped data and interpret.

1.5.16.

Table 1.5.3 gives the distribution of the masses (in grams) of 50 salmon from a single young cohort.

Table 1.5.2 Age of mother

Number of births

10–14

895

15–19

55,373

20–24

122,591

25–29

139,615

30–34

127,502

35–39

68,685

1.6 Computers and Statistics 39

Table 1.5.3 Weight

155–164

165–174

175–184

185–194

195–204

8

11

18

9

4

Frequency

(a) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation (b) Find the median for this grouped data. 1.5.17.

After a pollution accident, 180 dead ﬁsh were recovered from a stream. Table 1.5.4 gives their lengths measured to the nearest millimeter.

Table 1.5.4 Length of Fish (mm) Frequency

1–19

20–39

40–59

60–79

80–99

38

31

59

45

7

(a) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation. (b) Find the median for this grouped data and interpret.

1.6 COMPUTERS AND STATISTICS With present-day technology, we can automate most statistical calculations. For small sets of data, many basic calculations such as ﬁnding means and standard deviations and creating simple charts, graphing calculators are sufﬁcient. Students should learn how to perform statistical analysis using their handheld calculators. For deeper analysis and for large data sets, statistical software is necessary. Software also provides easier data entry and editing and much better graphics in comparison to calculators. There are many statistical packages available. Many such analyses can be performed with spreadsheet application programs such as Microsoft Excel, but a more thorough data analysis requires the use of more sophisticated software such as Minitab and SPSS. For students with programming abilities, packages such as MATLAB may be more appealing. For very large data sets and for complicated data analysis, one could use SAS. SAS is one of the most frequently used statistical packages. Many other statistical packages (such as R, Splus, and StatXact) are available; the utilities and advantages of each are based on the speciﬁc application and personal taste. For example, R is free software that is being increasingly used by statisticians and can be downloaded from http://www.r-project.org/, and a statistical tutorial for R can be found at http://www.biometrics.mtu.edu/CRAN/. For a good introduction to doing statistics with R, refer to the book by Peter Dalgaard, Introductory Statistics, with R, Springer, 2002. In this book, we will give some representative Minitab, SPSS, and SAS commands at the end of each chapter just to get students started on the technology. These examples are by no means a tutorial for

40 CHAPTER 1 Descriptive Statistics

the respective software. For a more thorough understanding and use of technology, students should look at the users’ manual that comes with the software or at references given at the end of the book. The computer commands are designed to be illustrative, rather than completely efﬁcient. In dealing with data analysis for real-world problems, we need to know which statistical procedure to use, how to prepare the data sets suitable for use in the particular statistical package, and ﬁnally how to interpret the results obtained. A good knowledge of theory supplemented with a good working knowledge of statistical software will enable students to perform sophisticated statistical analysis, while understanding the underlying assumptions and the limitations of results obtained. This will prevent us from misleading conclusions when using computer-generated statistical outputs.

1.7 CHAPTER SUMMARY In this chapter, we dealt with some basic aspects of descriptive statistics. First we gave basic deﬁnitions of terms such as population and sample. Some sampling techniques were discussed. We learned about some graphical presentations in Section 1.4. In Section 1.5 we dealt with descriptive statistics, in which we learned how to ﬁnd mean, median, and variance and how to identify outliers. A brief discussion of the technology and statistics was given in Section 1.6. All the examples given in this chapter are for a univariate population, in which each measurement consists of a single value. Many populations are multivariate, where measurements consist of more than one value. For example, we may be interested in ﬁnding a relationship between blood sugar level and age, or between body height and weight. These types of problems will be discussed in Chapter 8. In practice, it is always better to run descriptive statistics as a check on one’s data. The graphical and numerical descriptive measures can be used to verify that the measurements are sound and that there are no obvious errors due to collection or coding. We now list some of the key deﬁnitions introduced in this chapter. ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Population Sample Statistical inference Quantitative data Qualitative or categorical data Cross-sectional data Time series data Simple random sample Systematic sample Stratiﬁed sample Proportional stratiﬁed sampling Cluster sampling Multiphase sampling Relative frequency Cumulative relative frequency

1.8 Computer Examples 41

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Bar graph Pie chart Histogram Sample mean Sample variance Sample standard deviation Median Interquartile range Mode Mean Empirical rule Box plots

In this chapter, we have also introduced the following important concepts and procedures: ■ ■ ■ ■

■

■ ■

General procedure for data collection Some advantages of simple random sampling Steps for selecting a stratiﬁed sample Procedures to construct frequency and relative frequency tables and graphical representations such as stem-and-leaf displays, bar graphs, pie charts, histograms, and box plots Procedures to calculate measures of central tendency, such as mean and median, as well as measures of dispersion such as the variance and standard deviation for both ungrouped and grouped data Guidelines for the construction of frequency tables and histograms Procedures to construct a box plot

1.8 COMPUTER EXAMPLES In this section, we give some examples of how to use Minitab, SPSS, and SAS for creating graphical representations of the data as well as methods for the computation of basic statistics. Sometimes, the outputs obtained using a particular software package may not be exactly as explained in the book; they vary from one package to another, and also depend on the particular software version. It is important to obtain the explanation of outputs from the help menu of the particular software package for complete understanding. The “Computer Examples” sections of this book are not designed as manuals for the software, nor are they written in the most efﬁcient way. The idea is only to introduce some basic procedures, so that the students can get started with applying the theoretical material they have seen in each of the chapters.

1.8.1 Minitab Examples A good place to get help on Minitab is http://www.minitab.com/resources/. There are many nice sites available on Minitab procedures; for example, Minitab student tutorials can be obtained from

42 CHAPTER 1 Descriptive Statistics

http://www.minitab.com/resources/tutorials/. Here we illustrate only some of the basic uses of Minitab. In Minitab, we can enter the data in the spreadsheet and use the Windows pull-down menus, or we can directly enter the data and commands. We will mostly give procedures for the pull-down menus only. It is up to the user’s taste to choose among these procedures. It should be noted that with different versions of Minitab, there will be some differences in the pull-down menu options. It is better to consult the Help menu for the actual procedure.

Example 1.8.1 (Stem-and-Leaf): For the following data, construct a stem-and-leaf display using Minitab: 78 91

74 74

82 82

66 75

94 96

71 78

64 84

88 79

55 71

80 83

Solution For the pull-down menu, first enter the data in column 1. Then follow the following sequence. The boldface represents the actions.

Graph > Character Graphs > Stem-and-Leaf In Variables: type C1 and click OK We will get the following output: Stem-and-Leaf of C1 Leaf Unit = 1.0 1 5 5 2 6 4 3 6 6 7 7 1 (4) 7 5 9 8 0 4 8 8 3 9 14 1

N = 20

1 8 2

4 8 2

4 9 3

4

The following are the explanations of each column in the stem-and-leaf display, as given in the Minitab Help menu. The display has three columns: Left: Cumulative count of values from the top of the figure down and from the bottom of the figure up to the middle. Middle number in parentheses (stem): Count of values in the row containing the median. Parentheses around the median row are omitted if the median falls between two lines of the display. Right (leaves): Each value is a single digit to place after the stem digits, representing one data value. The leaf unit tells you where to put the decimal place in each number.

1.8 Computer Examples 43

Note that this display is a little different from the one we explained in Section 1.4. However, if we combine the stems and the corresponding leaves, we will get the representation as in Section 1.4.

Example 1.8.2 (Histogram): For the following data, construct a histogram: 25 38

37 16

20 40

31 32

31 33

21 24

12 39

25 26

36 27

27 19

Solution Enter the data in C1, then use the following sequence Graph > Histogram. . . > in Graph variables: type C1 > OK

We will get the histogram as shown in Figure 1.8.

6

Frequency

5 4 3 2 1 0 10

15

20

25

30

35

40

■ FIGURE 1.8 Histogram for data of Example 1.8.2.

If we want to change the number of intervals, after entering Graph variables, click Options. . . and click Number of intervals and enter the desired number, then OK.

Example 1.8.3 (Descriptive Statistics): In this example, we will describe how to obtain basic statistics such as mean, median, and standard deviation for the following data:

44 CHAPTER 1 Descriptive Statistics

5 22 47

7 21 67

229 25 69

453 23 192

12 24 125

14 34

18 37

14 34

14 49

483 64

Solution Enter the data in C1. Then use Stat > Basic Statistics > Display Descriptive Statistics. . . > in Variables: type C1 > click OK We will get the following output: Variable C1 TrMean 69.3

StDev 128.4 Q1 16.0

N 25 SE Mean 25.7

Mean 83.3 Minimum 5.0 Q3 68.0

Median 34.0 Maximum 483.0

Here, TrMean represents the trimmed mean. A 5% trimmed mean is calculated. Minitab removes the smallest 5% and the largest 5% of the values (rounded to the nearest integer) and then averages the remaining values. Also, SE Mean gives the standard error of the mean. It is calculated as StDev/SQRT (N), where StDev is the standard deviation.

Example 1.8.4 (Sorting and Box Plot): For the following data, ﬁrst sort in the increasing order and then construct a box plot to check for outliers. 870 1150 866

922 977 996

1146 958 1102

1120 1088 1028

1079 1139 1130

905 1055 1002

888 1082 990

865 1053 1052

1112 1048 1116

966 1118 1109

Solution After entering the data in C1, we can sort the data in increasing order as follows: Manip > Sort. . . > in Sort column(s): type C1 > in Store sorted column(s) in: type C2 > in Sorted by column: type C1 > OK In column C2, we will get the following sorted data: C2 865 866 870 888 905 922 958 966 977 990 996 1002 1028 1048 1052 1053 1055 1079 1082 1088 1102 1109 1112 1116 1118 1120 1130 1139 1146 1150 If we want to draw a box plot for the data, do the following:

1.8 Computer Examples 45

Graph > Box plot. . . > in Graph variables: under Y, type C1 > OK We will get the box plot as shown in Figure 1.9.

1150 1100 1050 1000 950 900 850 ■ FIGURE 1.9 Box plot data of Example 1.8.4.

Example 1.8.5 (Test of Randomness): Almost all of the analyses in this book assume that the sample is random. How can we verify whether the sample is really random? Project 12B explains a procedure called run test. Without going into details, this test is simple with Minitab. All we have to do is enter the data in C1. Then click Stat > Nonparametric > Runs Test. . . > in variables: enter C1 > OK For instance, if we have the following data: 24 38

31 49

28 51

43 49

28 62

we will get following output: Run Test C1 K = 44.0500 The observed number of runs = 14 The expected number of runs = 11.0000 10 Observations above K 10 below

56 33

48 41

39 58

52 63

32 56

46 CHAPTER 1 Descriptive Statistics

* N Small -- The following approximation may be invalid The test is significant at 0.1681 Cannot reject at alpha = 0.05 ‘‘Cannot reject’’ in the output means that it is reasonable to assume that the sample is random. For any data, it is always desirable to do a run test to determine the randomness.

1.8.2 SPSS Examples For SPSS, we will give only Windows commands. For all the pull-down menus, the sequence will be separated by the > symbol.

Example 1.8.6 Redo Example 1.8.1 with SPSS.

Solution After entering the data in C1,

Analyze > Descriptive Statistics > Explore. . . > At the Explore window select the variable and move to Dependent List; then click Plots. . ., select Stem-and-Leaf , click Continue, and click OK at the Explore Window We will get the output with a few other things, including box plots along with the stem-and-leaf display, which we will not show here.

Example 1.8.7 Redo Example 1.8.2 with SPSS.

Solution After entering the data:

Graphs > Histogram. . . > At the Histogram window select the variable and move to Variable, and click OK We will get the histogram, which we will not display here.

Example 1.8.8 Redo Example 1.8.3 with SPSS.

1.8 Computer Examples 47

Solution Enter the data. Then: Analyze > Descriptive Statistics > Frequencies. . . > At the Frequencies window select the variable(s); then open the Statistics window and check whichever boxes you desire under Percentile, Dispersion, Central Tendency, and Distribution > continue > OK For example, if you select Mean, Median, Mode, Standard Deviation, and Variance, we will get the following output and more:

N Mean Median Mode Std. Deviation Variance

Statistics VAR00001 Valid Missing

25 0 83.2800 34.0000 14.00 128.36488 16477.54333

1.8.3 SAS Examples We will now give some SAS procedures describing the numerical measures of a single variable. PROC UNIVARIATE will give mean, median, mode, standard deviation, skewness, kurtosis, etc. If we do not need median, mode, and so on, we could just as well use PROC MEANS in lieu of PROC UNIVARIATE. We can use the following general format in writing SAS programs with appropriate problem-speciﬁc modiﬁcations. There are many good online references as well as books available for SAS procedures. To get support on SAS, including many example codes, refer to the SAS support Web site: http://support.sas.com/. Another helpful site can be found at http://www.ats.ucla.edu/stat/sas/. There are many other sites that may suit your particular application. GENERAL FORMAT OF AN SAS PROGRAM DATA give a name to the data set; INPUT here we put variable names and column locations, if there are more than one variable; CARDS; (also we can use DATALINES;) Enter the data here; TITLE ‘here we include the title of our analysis’; PROC PRINT; PROC name of procedure (such as PROC UNIVARIATE) goes here; Options that we may want to include (such as the variables to be used) go here; RUN;

48 CHAPTER 1 Descriptive Statistics

After writing an SAS program, to execute it we can go to the menu bar and select run>submit, or click the “running man” icon. On execution, SAS will output the results to the Output window. All the steps used including time of execution and any error messages will be given in the Log window. In order to make the SAS outputs more manageable, we can use the following SAS command at the beginning of an SAS program: options ls=80 ps=50;

ls stands for line size, and this sets each line to be 80 characters wide. ps stands for page size and allows 50 lines on each page. This reduces the number of unnecessary page breaks. In order to avoid date and number, we can use the option commands: Options nodate nonumber;

Example 1.8.9 For the data of Example 1.8.3, use PROC UNIVARIATE to summarize the data.

Solution In the program editor window, type the following if you are entering the data directly. If you are using the data stored in a file, the comment line (with *) should be used instead of the input and data lines.

Options nodate nonumber; DATA ex9; INPUT ex9 @@; DATALINES; 5 7 229 453 12 14 18 14 14 483 22 21 25 23 24

34 37 34 49 64

47 67 69 192 125; PROC UNIVARIATE; TITLE; RUN; In this case we will get the following output:

N Mean Std Deviation

The UNIVARIATE Procedure Variable: ex9 Moments 25 Sum Weights 25 83.28 Sum Observations 2082 128.364884 Variance 16477.5433

1.8 Computer Examples 49

Skewness 2.45719194 Kurtosis 5.47138396 Uncorrected SS 568850 Corrected SS 395461.04 Coeff Variation 154.136508 Std Error Mean 25.6729767 Basic Statistical Measures Location Variability Mean 83.28000 Std Deviation 128.36488 Median 34.00000 Variance 16478 Mode 14.00000 Range 478.00000 Interquartile Range 49.00000 Tests for Location: Mu0=0 Test -Statistic-p ValueStudent’s t t 3.243878 Pr > |t| 0.0035 Sign M 12.5 Pr >= |M| = |S| 0 otherwise.

(a) For what value of λ is f a pdf? (b) Find F (x).

Solution

∞ (a) First note that f (x) ≥ 0. Now, for f (x) to be a pdf, we need −∞ f (x)dx = 1. Because f (x) = 0 for x ≤ 0, Therefore λ = 1. See Figure 2.6. 1=

∞ −∞

∞ f (x)dx = λxe−x dx 0

⎡ ⎤ ∞ ∞ ∞ = λ xe−x dx = λ⎣ −xe−x 0 + e−x dx⎦(using integration by parts) 0 0 ∞ = λ 0 − e−x 0 = λ. 0.5 0.4

P (a # X # b)

f (x )

0.3 0.2 0.1 0.0 a

b X Data

■ FIGURE 2.6 Probability as an area under a curve.

88 CHAPTER 2 Basic Concepts from Probability Theory

0.4

0.3

0.2

0.1

0.0 0

2

4

6

8

10

12

8

10

12

■ FIGURE 2.7 Graph of f (x) = xe−x .

1.2 1.0 0.8 0.6 0.4 0.2 0.0 0

2

4

6

■ FIGURE 2.8 Graph of F (x), x ≥ 0.

(b) The cumulative distribution function is x f (t)dt =

F (x) = −∞

⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩

0, x

te−t dt = 1 − (x + 1) e−x ,

x 750).

0≤y≤4 elsewhere.

98 CHAPTER 2 Basic Concepts from Probability Theory

Solution (a) ∞ E(Y ) =

yf (y)dy −∞

3 = 64

4

yy2 (4 − y) dy

0

= 2.4 and 4 Var(Y ) =

(y − 2.4)2

3 2 y (4 − y) dy 64

0

= 0.64. (b) Using the fact that Var(aY + b) = a2 Var(Y ), we have Var(X) = (300)2 Var(Y ) = 90,000(0.64) = 57,600. (c) P (X > 750) = P(300Y + 50 > 750)

7 =P Y > 3 3 = 64

4

y2 (4 − y) dy = 0.55339.

7/3

2.6.1 Skewness and Kurtosis Even though the mean μ and the standard deviation σ are signiﬁcant descriptive measures that locate the center and describe the spread or dispersion of probability density function f (x), they do not provide a unique characterization of the distribution. Two distributions may have the same mean and variance and yet could be very different, as in Figure 2.12. To better approximate the probability distribution of a random variable, we may need higher moments.

2.6 Moments and Moment-Generating Functions 99

0.5 0.4 0.3 0.2 0.1 0.0 Mean 5 1 Variance 5 1

Mean 5 1 Variance 5 1 ■ FIGURE 2.12 Same mean and variance.

Deﬁnition 2.6.4 The kth moment about the origin of a random variable X is deﬁned as EXk and denoted by μk , whenever it exists. The kth moment about its mean (also called central kth moment) $ # of a random variable X is deﬁned as E (X − μ)k and denoted by μk , k = 2, 3, 4, . . . , whenever it exists. In particular, we have E(X) = μ1 = μ, and σ 2 = μ2 . We have seen earlier that the second moment about mean (variance, σ 2 ) is used as a measure of dispersion about the mean. Deﬁnition 2.6.5 The standardized third moment about mean α3 =

E(X − μ)3 μ3 = 3/2 σ3 μ2

is called the skewness of the distribution of X. The standardized fourth moment about mean α4 =

E(X − μ)4 σ4

is called the kurtosis of the distribution. Skewness is used as a measure of the asymmetry (lack of symmetry) of a density function about its mean. Recall that a distribution, or data set, is symmetric if it looks the same to the left and right of the center point. If α3 = 0, then the distribution is symmetric about the mean, if α3 > 0, the distribution has a longer right tail, and if α3 < 0, the distribution has a longer left tail. Thus, the skewness of a normal distribution is zero. Kurtosis is a measure of whether the distribution is peaked or ﬂat relative to a normal distribution. Kurtosis is based on the size of a distribution’s tails. Positive kurtosis indicates too few observations in the tails, whereas negative kurtosis indicates too many observations in the tail of the distribution. Distributions with relatively large tails are called leptokurtic, and those with small tails are called platokurtic. A distribution which has the same kurtosis as a normal distribution is known as mesokurtic. It is known that the kurtosis for a standard normal distribution α4 = 3. An important expectation is the moment-generating function for a random variable, in a sense, this packages all the moments for a random variable in one expression.

100 CHAPTER 2 Basic Concepts from Probability Theory

Deﬁnition 2.6.6 For a random variable X, suppose that there is a positive number h such that for −h < t < h the mathematical expectation E etX exists. The moment-generating function (mgf) of the random variable X is deﬁned by ⎧ ⎨ etx p(x), MX (t) = E etX = ⎩ etx f (x)dx,

if discrete . if continuous

An advantage of the moment generating function is its ability to give the moments. Recall that the Maclaurin series of the function etx is etx = 1 + tx +

(tx)3 (tx)n (tx)2 + + ··· + + ···· 2! 3! n!

By using the fact that the expected value of the sum equals the sum of the expected values, the moment-generating function can be written as

%

(tX)3 (tX)n (tX)2 MX (t) = E etX = E 1 + tX + + + ··· + + ··· 2!

= 1 + tE[X] +

3!

&

n!

tn # $ t2 2 t3 3 E X + E X + · · · + E Xn + · · · 2! 3! n!

Taking the derivative of MX (t) with respect to t, we obtain 2 dMX (t) (t) = E[X] + tE[X] + t E X2 = MX dt 2! +

t (n−1) # n $ t3 3 E X + ··· E X + ··· + (n − 1)! 3!

Evaluating this derivative at t = 0, all terms except E[X] become zero. We have (0) = E[X]. MX

Similarly, taking the second derivative of MX (t), we obtain (0) = E X2 . MX (n)

Continuing in this manner, from the nth derivative MX (t) with respect to t, we obtain all the moments to be # $ (n) MX (0) = E Xn ,

n = 1, 2, 3, . . . .

We summarize these calculations in the following theorem.

2.6 Moments and Moment-Generating Functions 101

Theorem 2.6.3 If MX (t) exists, then for any positive integer k, d k MX (t) dt k

= MX (0) = μk . (k)

t=0

The usefulness of the foregoing theorem lies in the fact that, if the mgf can be found, the often difﬁcult process of integration or summation involved in calculating different moments can be replaced by the much easier process of differentiation. The following examples illustrate this fact.

Example 2.6.8 Let X be a random variable with pf p(x) =

n x p (1 − p)n−x , x

x = 0, 1, 2, . . . , n.

(This random variable is called a binomial random variable, and the pf is called a binomial distribution.) # $n Show that MX (t) = (1 − p) + pet , for all real values of t. Also obtain mean and variance of the random variable X.

Solution The moment-generating function of X is

MX (t) = E etX

=

n

n x p (1 − p)n−x etx

x=0

x

n n (pet )x (1 − p)n−x . = x x=0

Using the binomial formula, we have # $n MX (t) = pet + (1 − p) ,

−∞ < t < ∞.

The first two derivatives of MX (t) are (t) = n#(1 − p) + pet $(n−1) pet MX

and (t) = n(n − 1)#(1 − p) + pet $(n−2) pet 2 + n#(1 − p) + pet $(n−1) pet . MX

Thus, (0) = np μ = E(X) = MX

102 CHAPTER 2 Basic Concepts from Probability Theory

and σ 2 = E X2 − μ2 = M (0) − (np)2 = n(n − 1) p2 + np − (np)2 = np(1 − p) .

Example 2.6.9 Let X be a random variable with pmf f (x) = e−λ λx /(x!), x = 0, 1, 2, . . . . (Such a random variable is called a Poisson r.v. and the distribution is called a Poisson distribution with parameter λ.) Find the mgf of X.

Solution By definition MX (t) = EetX =

∞

etx f (x)

x=0

=

∞

t x ∞ eλ e−λ λx tx −λ e e = x!

x=0

= e−λ

∞

x=0

% t eλe

x=0 t = eλ(e −1)

x!

x & t e−(λe ) λet x!

% x & t ∞ e−(λe ) λet x!

x=0

∞ −(λet ) t x x t e (λe ) = 1. Thus We observe that e−(λe ) λet /x! is a Poisson pf with parameter λet . Hence x! x=0

from (1), MX (t) = eλ(e −1) . t

Example 2.6.10 Let X be a random variable with pdf given by

f (x) = Find mgf MX (t).

1 −x/β , βe

0,

x>0 otherwise.

2.6 Moments and Moment-Generating Functions 103

Solution By definition of mgf, −∞

etx f (x)dx

Mx (t) = ∞

=

∞ 1 etx e−x/β dx β 0

1 = β

∞ 1 1 − −t x e β dx, t < β 0

%

= =

1 1 − − e β ((1/β) − t)

1 β −t

& ∞ x

x=0

1 1 β = , β 1 − βt 1 − βt

t

0, otherwise.

Can we obtain the probability density of the variable X with the foregoing information?

2.7 CHAPTER SUMMARY In this chapter, we have introduced the concepts of random events and probability, how to compute the probabilities of events using counting techniques. We have studied the concept of conditional probability, independence, and Bayes’ rule. Random variables and distribution functions, moments, and moment-generating functions of random variables have also been introduced. The following lists some of the key deﬁnitions introduced in this chapter. ■ ■ ■ ■ ■

Sample space Mutually exclusive events Informal deﬁnition of probability Classical deﬁnition of probability Frequency interpretation of probability

108 CHAPTER 2 Basic Concepts from Probability Theory

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Axiomatic deﬁnition of probability Multinomial coefﬁcients Conditional probability Mutually independent events Pairwise independent events Random variable (r.v.) Discrete random variable Discrete probability mass function Cumulative distribution function Continuous random variable Expected value kth moment about the origin kth moment about its mean Skewness and kurtosis Moment-generating function

The following important concepts and procedures have been discussed in this chapter: ■ ■ ■ ■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Method of computing probability by the classical approach Some basic properties of probability Computation of probability using counting techniques Four sampling methods: ❏ Sampling with replacement and the objects are ordered ❏ Sampling without replacement and the objects are ordered ❏ Sampling without replacement and the objects are not ordered ❏ Sampling with replacement and the objects are not ordered Permutation of n objects taken m at a time Combinations of n objects taken m at a time Number of combinations of n objects into m classes Some properties of conditional probability Law of total probability Steps to apply Bayes’ rule Some properties of distribution function Some properties of expected value Expectation of function of a random variable Properties of moment-generating functions

2.8 COMPUTER EXAMPLES (OPTIONAL) The three softwares packages, Minitab, SPSS, and SAS, that we are using in this book are not speciﬁcally designed for probability computations. However, the following examples are given to demonstrate that we will be able to use the software for some basic probability computations. We do not recommend using any of these three software packages for probability calculations; they are basically

2.8 Computer Examples (Optional) 109

designed for statistical computations. There are many other software packages such as Maple or MATLAB, that can be used efﬁciently for probability computations.

2.8.1 Minitab Computations In order to ﬁnd the cdf of a random variable, we can use the following commands in Example 2.8.1. We can use the mathematical expressions to ﬁnd the expected value of a discrete random variable.

Example 2.8.1 A random variable X has the following distribution: x p(x)

1 0.2

4 0.2

5 0.1

8 0.15

11 0.35

Find P(X ≤ 4).

Solution Enter x values in C1 and p(x) values in C2.

Calc > Probability Distributions > Discrete. . . > click Cumulative probability, and in Values in: enter C1, Probabilities in: enter C2, click input column: enter C1, in Optional storage: enter C3 > OK We will get the following output in column C3. 0.20

0.40

0.50

0.65

1.00

Example 2.8.2 For the random variable X in Example 2.8.1, ﬁnd E(X).

Solution Enter x values in column C1 (i.e., 1 4 5 8 11), and enter p(x) values in column C2. Use the following procedure.

Calc > Calculator. . . > Store results in variable: type C3 > in Expression: type (C1)*(C2) > click OK Then to find the sum of values in column C3 > Calc > Column Statistics. . . > click Sum and in Input variable: type C3 > click OK

We will get the output as Column Sum Sum of C3 = 6.5500

110 CHAPTER 2 Basic Concepts from Probability Theory

Note that this Sum gives the E(X). In the previous procedure, if we store the expression (C1)*(C1)*(C2) in column C4 and ﬁnd the sum of terms in C4, we will get E X2 . Using this, we will be able to compute Var(X). Using a similar procedure, we can obtain E(Xn ) for any n ≥ 1.

2.8.2 SPSS Examples Example 2.8.3 For the random variable X in Example 2.8.1, ﬁnd E(X).

Solution In column 1, enter the x values and column 2 enter the p(x) values. Then Transform > compute. . . > in target variable: type a name, say, product. Move var00001 and var00002 to Numeric Expression: field and put ‘‘*’’ in between them as (var00001)*(var00002). Then use the SUM(. , .) command to find the value of E(X)

2.8.3 SAS Examples Example 2.8.4 A random variable X has the following distribution: x P(X)

2 0.1

5 0.2

6 0.3

8 0.1

9 0.3

Using SAS, ﬁnd E(X).

Solution For discrete distributions where the random variable takes finite values, we can adapt the following procedure: data evalue; input x y n; z=x*y*n; cards; 2 .1 5 5 .2 5 6 .3 5 8 .1 5 9 .3 5 ; run;

2.8 Computer Examples (Optional) 111

proc means; run;

xρ(x); hence, multiplying by n, We know that if proc means is used just for x∗ y, that will give us 1n the number of values X takes will give us E(X) = xp(x). We will get the following output: The MEANS Procedure Variable N

Mean

Std Dev

Minimum

Maximum

=================================================== x

5 6.0000000 2.7386128 2.0000000

9.0000000

y

5 0.2000000 0.1000000 0.1000000

0.3000000

n

5 5.0000000

5.0000000

z

5 6.5000000 4.8476799 1.0000000

0

5.0000000

13.5000000

From this, we can see that E(X) = 6.5. A direct way to ﬁnd the expected value is by using “PROC IML.” options nodate nonumber; /* Finding expected value of a random variable */ proc iml; /* deﬁning all the variables */ x={2 5 6 8 9}; /* a row vector */ y={.1 .2 .3 .1 .3}; /* probabilities */ /* calculations */ z=x*y‘; /* print statements */ print “Display the vector x and probability y and the expected value”; print x y, z; quit;

We will get the following output: X 2

5

6

8

9

Y 0.1 0.2 0.3 0.1 0.3 Z 6.5

112 CHAPTER 2 Basic Concepts from Probability Theory

PROJECTS FOR CHAPTER 2 2A. The Birthday Problem The famous birthday problem is to ﬁnd the smallest number of people one must ask to get an even chance that at least two people have the same birthday. To solve this you can use the following steps. Find the probability that in a group of k people no two have the same probability. Let q be this probability. Then p = 1 − q is the probability that at least two people have the same birthday. Ignoring leap years, take the sample space S as all sequences of length k with each element one of the 365 days in the year. Thus there are 365k elements in S. (a) Find the total number of sequences with no common birthdays. (b) Assuming that each sequence is equally likely, show that q=

(365)(364) . . . (365 − k + 1) . 365k

(c) Write a computer program for calculating q for k = 2 to 50, and ﬁnd the ﬁrst k for which p > 0.5. This will give the least number of people we should ask to make it an even chance that at least two people will have the same birthday.

2B. The Hardy--Weinberg Law Hereditary traits in offspring depend on a pair of genes, one each contributed by the father and the mother. A gene is either a dominant allele, denoted by A, or a recessive allele, denoted by a. If the genotype is AA, Aa, or aA, then the hereditary trait is A, and if the genotype is aa, then the hereditary trait is a. Suppose that the probabilities of the mother carrying the genotypes aa, aA (same as Aa), and AA are p, q, and r, respectively. Here p + q + r = 1. The same probabilities are true for the father. (a) Assuming that the genetic contributions of the mother and father are independent and the matings are random, show that the respective probabilities for the ﬁrst-generation offspring are p1 = (p + q/2)2 , q1 = 2 (r + q/2) (p + q/2) , r1 = (r + q/2)2 .

Also ﬁnd P(A) and P(a). (b) The Englishman G. H. Hardy and the German W. Weinberg could show that the foregoing probabilities in a population stay constant for generations if certain conditions are fulﬁlled. This is known as the Hardy–Weinberg law. Under the conditions of part (a), using the induction argument, show that the Hardy–Weinberg law is satisﬁed, i.e., pn = p1 , qn = q1 , and rn = r1 for all n ≥ 1. The consequences of the Hardy–Weinberg law are that (i) no evolutionary change occurs through the process of sexual reproduction itself, and (ii) changes in allele and genotype frequencies can result only from additional forces on the gene pool of a species.

Chapter

3

Additional Topics in Probability Objective: In this chapter we present some special distributions, joint distributions of several random variables, functions of random variables, and some important limit theorems. 3.1 Introduction 114 3.2 Special Distribution Functions 114 3.3 Joint Probability Distributions 141 3.4 Functions of Random Variables 154 3.5 Limit Theorems 163 3.6 Chapter Summary 173 3.7 Computer Examples (Optional) 175 Projects for Chapter 3 180

Johann Carl Friedrich Gauss (Source: http://tobiasamuel.ﬁles.wordpress.com/2008/06/carl_friedrich_gauss.jpg)

German mathematician and physicist Carl Friedrich Gauss (1777–1855) is sometimes called the “prince of mathematics.” He was a child prodigy. At the age of 7, Gauss started elementary school,

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

113

114 CHAPTER 3 Additional Topics in Probability

and his potential was noticed almost immediately. His teachers were amazed when Gauss summed the integers from 1 to 100 instantly. At age 24, Gauss published one of the most brilliant achievements in mathematics, Disquisitiones Arithmeticae (1801). In it, Gauss systematized the study of number theory. Gauss applied many of his mathematical insights in the ﬁeld of astronomy, and by using the method of least squares he successfully predicted the location of the asteroid Ceres in 1801. In 1820 Gauss made important inventions and discoveries in geodesy, the study of the shape and size of the earth. In statistics, he developed the idea of the normal distribution. In the 1830s he developed theories of non-Euclidean geometry and mathematical techniques for studying the physics of ﬂuids. Although Gauss made many contributions to applied science, especially electricity and magnetism, pure mathematics was his ﬁrst love. It was Gauss who ﬁrst called mathematics “the queen of the sciences.”

3.1 INTRODUCTION In the previous chapter, we looked at the basic concepts of probability calculations, random variables, and their distributions. There are many special distributions that have useful applications in statistics. It is worth knowing the type of distribution that we can expect under different circumstances, because a better knowledge of the population will result in better inferential results. In the next section, we discuss some of these distributions with some additional distributions presented in Appendix A3. We also brieﬂy deal with joint distributions of random variables and functions of random variables. Limit theorems play an important role in statistics. We will present two limit theorems: the law of large numbers and the Central Limit Theorem.

3.2 SPECIAL DISTRIBUTION FUNCTIONS Random variables are often classiﬁed according to their probability distribution functions. In any analysis of quantitative data, it is a major step to know the form of the underlying probability distributions. There are certain basic probability distributions that are applicable in many diverse contexts and thus repeatedly arise in practice. A great variety of special distributions have been studied over the years. Also, new ones are frequently being added to the literature. It is impossible to give a comprehensive list of distribution functions in this book. There are many books and Web sites that deal with a range of distribution functions. A good list of distributions can be obtained from http://www.causascientia.org/math_stat/Dists/Compendium.pdf. In this section, we will describe some of the commonly used probability distributions. In Appendix A3, we list some more distributions with their mean, variance, and moment-generating functions. First we discuss some discrete probability distributions.

3.2.1 The Binomial Probability Distribution The simplest distribution is the one with only two possible outcomes. For example, when a coin (not necessarily fair) is tossed, the outcomes are heads or tails, with each outcome occurring with some positive probability. These two possible outcomes may be referred to as “success” if heads occurs and “failure” if tails occurs. Assume that the probability of heads appearing in a single toss is p; then the probability of tails is 1 − p = q. We deﬁne a random variable X associated with this experiment

3.2 Special Distribution Functions 115

as taking value 1 with probability p if heads occurs and value 0 if tails occurs with probability q. Such a random variable X is said to have a Bernoulli probability distribution. That is, X is a Bernoulli random variable if for some p, 0 ≤ p ≤ 1, the probability P(X = 1) = p and P(X = 0) = 1 − p. The probability function of a Bernoulli random variable X can be expressed as p(x) = P(X = x) =

px (1 − p)1−x ,

x = 0, 1

0,

otherwise.

Note that this distribution is characterized by the single parameter p. It can be easily veriﬁed that the mean and variance of X are E[X] = p, var(X) = pq, respectively, and the moment-generating function is MX (t) = pet + (1 − p). Even when the experimental values are not dichotomous, reclassifying the variable as a Bernoulli variable can be helpful. For example, consider blood pressure measurements. Instead of representing the numerical values of blood pressure, if we reclassify the blood pressure as “high blood pressure” and “low blood pressure,” we may be able to avoid dealing with a possible misclassiﬁcation due to diurnal variation, stress, and so forth, and concentrate on the main issue, which would be: Is the average blood pressure unusually high? In a succession of Bernoulli trials, one is more interested in the total number of successes (whenever a 1 occurs in a Bernoulli trial, we term it a “success”). The probability of observing exactly k successes in n independent Bernoulli trials yields the binomial probability distribution. In practice, the binomial probability distribution is used when we are concerned with the occurrence of an event, not its magnitude. For example, in a clinical trial, we may be more interested in the number of survivors after a treatment. Deﬁnition 3.2.1 A binomial experiment is one that has the following properties: (1) The experiment consists of n identical trials. (2) Each trial results in one of the two outcomes, called a success S and failure F. (3) The probability of success on a single trial is equal to p and remains the same from trial to trial. The probability of failure is 1 − p = q. (4) The outcomes of the trials are independent. (5) The random variable X is the number of successes in n trials. Earlier we have seen that the number of ways of obtaining x successes in n trials is given by

n! n = . x!(n − x)! x

Deﬁnition 3.2.2 A random variable X is said to have binomial probability distribution with parameters (n, p) if and only if

n x n−x P(X = x) = p(x) = p q x ⎧ n! ⎨ x!(n−x)! px qn−x , x = 0, 1, 2, . . . , n, 0 ≤ p ≤ 1, and q = 1 − p = ⎩ 0, otherwise.

116 CHAPTER 3 Additional Topics in Probability

To show the dependence on n and p, denote p(x) by b(x, n, p) and the cumulative probabilities by B(x, n, p) =

x

b(i, n, p)

i=0

Binomial probabilities are tabulated in the binomial table. By the binomial theorem, we have (p + q)n =

n n x=0

x

px qn−x .

n x n−x p q = 1n = 1, for all n ≥ 1 x=0 x and 0 ≤ p ≤ 1. Hence, p(x) is indeed a probability function. The binomial probability distribution is characterized by two parameters, the number of independent trials n and the probability of success p.

Because (p + q) = 1, we conclude that

x

i=0 b(i, n, p) =

n

Example 3.2.1 It is known that screws produced by a certain machine will be defective with probability 0.01 independently of each other. If we randomly pick 10 screws produced by this machine, what is the probability that at least two screws will be defective?

Solution Let X be the number of defective screws out of 10. Then X can be considered as a binomial r.v. with parameters (10, 0.01). Hence, using the binomial pf p(x), given in Definition 3.2.2, we obtain P(X ≥ 2) =

10 10 x=2

x

(0.01)x (0.99)10−x

= 1 − [P(X = 0) + P(X = 1)] = 0.004.

In Chapter 2, we saw Mendel’s law. In biology, the result “gene frequencies and genotype ratios in a randomly breeding population remain constant from generation to generation” is known as the Hardy–Weinberg law.

Example 3.2.2 Suppose we know that the frequency of a dominant gene, A, in a population is equal to 0.2. If we randomly select eight members of this population, what is the probability that at least six of them will display the dominant phenotype? Assume that the population is sufﬁciently large that removing eight individuals will not affect the frequency and that the population is in Hardy–Weinberg equilibrium.

3.2 Special Distribution Functions 117

Solution First of all, note that an individual can have the dominant gene, A, if the person has traits AA, aA, or Aa. Hence, if the gene frequency is 0.2, the probability that an individual is of genotype A is P(A) = P(AA ∪ Aa ∪ aA) = P(AA) + 2P(Aa) = (0.2)2 + 2(0.2)(0.8) = 0.36.

Let X denote the number of individuals out of eight that display the dominant phenotype. Then X is binomial with n = 8, and p = 0.36. Thus, the probability that at least six of them will display the dominant phenotype is P(X ≥ 6) = P(X = 6) + P(X = 7) + P(X = 8) =

8 10 i=6

i

(0.36)i (0.64)10−i = 0.029259.

For large n, calculation of binomial probabilities is tedious. Many statistical software packages have binomial probability distribution commands. For the purpose of this book, we will use the binomial table that gives the cumulative probabilities B(x, n, p) for n = 2 through n = 20 and p = 0.05, 0.10, 0.15, . . . , 0.90, 0.95. If we need the probability of a single term, we can use the relation P(X = x) = b(x, n, p) = B(x, n, p) − B(x − 1, n, p).

Example 3.2.3 A manufacturer of inkjet printers claim that only 5% of their printers require repairs within the ﬁrst year. If of a random sample of 18 of the printers, four required repairs within the ﬁrst year, does this tend to refute or support the manufacturer’s claim?

Solution Let us assume that the manufacturer’s claim is correct; that is, the probability that a printer will require repairs within the first year is 0.05. Suppose 18 printers are chosen at random. Let p be the probability that any one of the printers will require repairs within the first year. We now find the probability that at least four of these out of the 18 will require repairs during the first year. Let X represent the number of printers that require repair within the first year. Then X follows the binomial pmf with p = 0.05, n = 18. The probability that four or more of the 18 will require repair within the first year is given by P(X ≥ 4) =

18 18 x=4

x

(0.05)x (0.95)18−x

118 CHAPTER 3 Additional Topics in Probability

or, using the binomial table, 18

b(x, 18, 0.05) = 1 − B(3, 18, 0.05)

x=4

= 1 − 0.9891 = 0.0109. This value (approximately 1.1%) is very small. We have shown that if the manufacturer’s claim is correct, then the chances of observing four or more bad printers out of 18 are very small. But we did observe exactly four bad ones. Therefore we must conclude that the manufacturer’s claim cannot be substantiated.

MEAN, VARIANCE, AND MGF OF A BINOMIAL RANDOM VARIABLE Theorem 3.2.1 If X is a binomial random variable with parameters n and p, then E(X) = μ = np Var(X) = σ 2 = np(1 − p). Also the moment-generating function # $n MX (t) = pet + (1 − p) .

Proof. We derive the mean and the variance. The derivation for mgf is given in Example 2.6.5. Using the binomial pmf, p(x) = (n!/(x!(n − x)!))px qn−x , and the deﬁnition of expectation, we have μ = E(X) =

n

xp(x) =

x=0

=

n x=1

n x=0

x

n! px (1 − p)n−x x!(n − x)!

n! px (1 − p)n−x , (x − 1)!(n − x)!

since the ﬁrst term in the sum is zero, as x = 0. Let i = x − 1. When x varies from 1 through n, i = (x − 1) varies from zero through (n − 1). Hence, μ=

n−1

n! pi+1 (1 − p)n−i−1 i!(n − i − 1)!

i=0 n−1

= np

i=0

= np,

(n − 1)! pi (1 − p)n−1−i i!(n − 1 − i)!

3.2 Special Distribution Functions 119

because the last summand is that of a binomial pmf with parameter (n − 1) and p, hence, equals 1. To ﬁnd the variance, we ﬁrst calculate E [X(X − 1)]. E [X(X − 1)] =

n

x(x − 1)

x=0

=

n x=2

n! px (1 − p)n−x x!(n − x)!

n! px (1 − p)n−x , (x − 2)!(n − x)!

because the ﬁrst two terms are zero. Let i = x − 2. Then, E [X(X − 1)] =

n−2 i=0

n! pi+2 (1 − p)n−i−2 i!(n − i − 2)!

= n(n − 1)p2

n−2 i=0

(n − 2)! pi (1 − p)n i!(n − 2 − i)!

= n(n − 1)p2 ,

because the last summand is that of a binomial pf with parameter (n − 2) and p thus equals 1. Note that E(X(X − 1)) = EX2 − E(X), and so we obtain σ 2 = Var(X) = E(X2 ) − [E(X)]2 = E [X(X − 1)] + E(X) − [E(X)]2 = n(n − 1)p2 + np − (np)2 = −np2 + np = np(1 − p).

3.2.2 Poisson Probability Distribution The Poisson probability distribution was introduced by the French mathematician Siméon-Denis Poisson in his book published in 1837, which was entitled Recherches sur la probabilité des jugements en matières criminelles et matière civile and dealt with the applications of probability theory to lawsuits, criminal trials, and the like. Consider a statistical experiment of which A is an event of interest. A random variable that counts the number of occurrences of A is called a counting random variable. The Poisson random variable is an example of a counting random variable. Here we assume that the numbers of occurrences in disjoint intervals are independent and the mean of the number occurrences is constant.

120 CHAPTER 3 Additional Topics in Probability

Deﬁnition 3.2.3 A discrete random variable X is said to follow the Poisson probability distribution with parameter λ > 0, denoted by Poisson(λ), if P(X = x) = f (x, λ) = f (x) =

e−λ λx , x!

x = 0, 1, 2, . . .

The Poisson probability distribution is characterized by the single parameter, λ, which represents the mean of a Poisson probability distribution. Thus, in order to specify the Poisson distribution, we only need to know the mean number of occurrences. This distribution is of fundamental theoretical and practical importance. Rare events are modeled by the Poisson distribution. For example, the Poisson probability distribution has been used in the study of telephone systems. The number of incoming calls into a telephone exchange during a unit time might be modeled by a Poisson variable assuming that the exchange services a large number of customers who call more or less independently. Some other problems where Poisson representation can be used are the number of misprints in a book, radioactivity counts per unit time, the number of plankton (microscopic plant or animal organisms that ﬂoat in bodies of water) per aliquot of seawater, or count of bacterial colonies per petri plate in a microbiological study. In stem cell research, the Poisson distribution is used to analyze the redundancy of clusters in the stem cell database. A Poisson probability distribution has the unique property that its mean equals its variance.

MEAN, VARIANCE, AND MOMENT–GENERATING FUNCTION OF A POISSON RANDOM VARIABLE Theorem 3.2.2 If X is a Poisson random variable with parameter λ, then E(X) = λ Var(X) = λ. Also the moment-generating function is MX (t) = eλ(e −1) . t

The proof of this result is similar to that we used in Theorem 3.2.1 in this section. One needs to use i the Maclaurin’s expansion, eλ = ∞ i=0 (λ /i!).

Example 3.2.4 Let X be a Poisson random variable with λ = 1/2. Find (a) P(X = 0) (b) P(X ≥ 3)

Solution (a) We have P(X = 0) = p(0) =

e−1/2 (1/2)0 = e−1/2 = 0.60653. 0!

3.2 Special Distribution Functions 121

(b) Here we will use complementary event to compute the required probability. That is, P(X ≥ 3) = 1 − P(X ≤ 2) = 1 − [p(0) + p(1) + p(2)] % & e−1/2 (1/2) e−1/2 (1/2)2 −1/2 =1− e + + 1! 2! = 1 − 0.98561 = 0.01439. When n is large and p small, binomial probabilities are often approximated by Poisson probabilities. In these situations, where performing the factorial and exponential operations required for direct calculation of binomial probabilities is a lengthy and tedious process and tables are not available, the Poisson approximation is more feasible. The following theorem states this result.

POISSON APPROXIMATION TO THE BINOMIAL PROBABILITY DISTRIBUTION Theorem 3.2.3 If X is a binomial r.v. with parameters n and p, then for each value x = 0, 1, 2, . . . and as p → 0, n → ∞ with np = λ constant, lim

n→∞

n x

px (1 − p)n−x =

e−λ λx . x!

The proof of this result is similar to that we used in Theorem 3.2.1. In the present context, the Poisson probability distribution is sometimes referred to as “the distribution of rare events” because of the fact that p is quite small when n is large. Usually, if p ≤ 0.1 and n ≥ 40 we could use the Poisson approximation in practice. In general, another rule of thumb is to use Poisson approximation to binomial in the case of np < 5.

Example 3.2.5 If the probability that an individual suffers an adverse reaction from a particular drug is known to be 0.001, determine the probability that out of 2000 individuals, (a) exactly three and (b) more than two individuals will suffer an adverse reaction.

Solution Let Y be the number of individuals who suffer an adverse reaction. Then Y is binomial with n = 2000 and p = 0.001. Because n is large and p is small, we can use the Poisson approximation with λ = np = 2. (a) The probability that exactly three individuals will suffer an adverse reaction is P(Y = 3) =

23 e−2 = 0.18. 3!

That is, there is approximately an 18% chance that exactly three individuals of 2000 will suffer an adverse reaction.

122 CHAPTER 3 Additional Topics in Probability

(b) The probability that more than two individuals will suffer an adverse reaction is P(Y > 2) = 1 − P(Y = 0) − P(Y = 1) − P(Y = 2) = 1 − 5e−2 = 0.323. Similarly, there is approximately a 32.3% chance that more than two individuals will have an adverse reaction.

Now we will discuss some continuous distributions. As mentioned earlier, if X is a continuous random variable with pdf f (x), then b P(a ≤ X ≤ b) =

f (x)dx. a

3.2.3 Uniform Probability Distribution The uniform probability distribution is used to generate random numbers from other distributions and also is useful as a “ﬁrst guess” if no other information about a random variable X is known, other than that it is between a and b. Also, in real-world problems that have uniform behavior in a given interval, we can characterize the probabilistic behavior of such a phenomenon by the uniform distribution. (See Figure 3.1.) Deﬁnition 3.2.4 A random variable X is said to have a uniform probability distribution on (a, b), denoted by U(a, b), if the density function of X is given by ⎧ ⎨ 1 , f (x) = b − a ⎩ 0,

a ≤ x ≤ b, otherwise.

The cumulative distribution function is given by ⎧ 0, ⎪ ⎪ ⎪ ⎨ 1 x−a dx = F (x) = , ⎪ b−a ⎪b − a ⎪ −∞ ⎩ 1, x

f (x ) ⫽ 1/(b ⫺ a ) f (x ) ⫽ 0

f (x ) ⫽ 0 a

b

■ FIGURE 3.1 Uniform probability density.

x 6) =

4 1 dx = . 10 10

6

(c) 8 P(3 < X < 8) =

1 1 dx = . 10 2

3

MEAN, VARIANCE, AND MOMENT–GENERATING FUNCTION OF A UNIFORM RANDOM VARIABLE Theorem 3.2.4 If X is a uniformly distributed random variable on (a, b), then E(X) =

a+b . 2

and Var(X) =

(b − a)2 . 12

Also, the moment-generating function is ⎧ tb ta ⎪ ⎨e − e , MX (t) = t(b − a) ⎪ ⎩ 1,

t = 0 t = 0.

124 CHAPTER 3 Additional Topics in Probability

Proof. We will obtain the mean and the variance and leave the derivation of the moment-generating function as an exercise. By deﬁnition we have ∞ E(X) =

x −∞

b = a

1 dx b−a

⎛ ⎞ b 1 ⎝ x2 ⎠ 1 dx = x b−a b−a 2 a

a+b . = 2

Also E(X2 ) =

b a

= =

⎛ ⎞ 3 b 1 1 ⎝ x ⎠ dx = x2 b−a b−a 3 a

1 b3 − a3 3 b−a 1 2 (b + ab + a2 ) as b3 − a3 = (b − a)(b2 + ab + a2 ). 3

Thus, Var(X) = E(X2 ) − (E(X))2 1 2 (a + b)2 (b + ab + a2 ) − 3 4 1 (b − a)2 . = 12 =

Example 3.2.7 The melting point, X, of a certain solid may be assumed to be a continuous random variable that is uniformly distributed between the temperatures 100◦ C and 120◦ C. Find the probability that such a solid will melt between 112◦ C and 115◦ C.

Solution The probability density function is given by ⎧ ⎨ 1 , f (x) = 20 ⎩ 0

100 ≤ x ≤ 120 otherwise.

3.2 Special Distribution Functions 125

Hence, 115 P(112 ≤ X ≤ 115) =

3 1 dx = = 0.15. 20 20

112

Thus, there is a 15% chance of this solid melting between 112◦ C and 115◦ C.

3.2.4 Normal Probability Distribution The single most important distribution in probability and statistics is the normal probability distribution. The density function of a normal probability distribution is bell shaped and symmetric about the mean. The normal probability distribution was introduced by the French mathematician Abraham de Moivre in 1733. He used it to approximate probabilities associated with binomial random variables when n is large. This was later extended by Laplace to the so-called Central Limit Theorem, which is one of the most important results in probability. Carl Friedrich Gauss in 1809 used the normal distribution to solve the important statistical problem of combining observations. Because Gauss played such a prominent role in determining the usefulness of the normal probability distribution, the normal probability distribution is often called the Gaussian distribution. Gauss and Laplace noticed that measurement errors tend to follow a bell-shaped curve, a normal probability distribution. Today, the normal probability distribution arises repeatedly in diverse areas of applications. For example, in biology, it has been observed that the normal probability distribution ﬁts data on the heights and weights of human and animal populations, among others. We should also mention here that almost all basic statistical inference is based on the normal probability distribution. The question that often arises is, when do we know that our data follow the normal distribution? To answer this question we have speciﬁc statistical procedures that we study in later chapters, but at this point we can obtain some constructive indications of whether the data follows the normal distribution by using descriptive statistics. That is, if the histogram of our data can be capped with a bell-shaped curve (Figure 3.2), if the stem-and-leaf diagram is fairly symmetrical with respect to its center, and/or by invoking the empirical rule “backwards,” we can obtain a good indication whether our data follow the normal probability distribution. Deﬁnition 3.2.5 A random variable X is said to have a normal probability distribution with parameters μ and σ 2 , if it has a probability density function given by f (x) = √

1 2πσ

2 2 e−(x−μ) /2σ , −∞ < x < ∞, −∞ < μ < ∞, σ > 0.

If μ = 0, and σ = 1, we call it standard normal random variable. For any normal random variable with mean μ and variance σ 2 , we use the notation X ∼ N(μ, σ 2 ). When a random variable X has a standard normal probability distribution, we will write X ∼ N(0, 1) (X is a normal with mean 0 and variance 1). Probabilities for a standard normal probability distribution are given in the normal table.

126 CHAPTER 3 Additional Topics in Probability

0.5

0.4

0.3

0.2

0.1

0.0 0 ■ FIGURE 3.2 Standard normal density function.

MEAN, VARIANCE, AND MGF OF A NORMAL RANDOM VARIABLE Theorem 3.2.5 If X ∼ N(μ, σ 2 ), then E(X) = μ and Var(X) = σ 2 . Also the moment-generating function is 1 2 2 MX (t) = etμ+ 2 t σ .

If X ∼ N(μ, σ 2 ), then the z-transform (or z-score) of X, Z = X−μ σ , is an N(0, 1) random variable. This fact will be used in calculating probabilities for normal random variables.

Example 3.2.8 (a) For X ∼ N(0, 1), calculate P(Z ≥ 1.13). (b) For X ∼ N(5, 4), calculate P(−2.5 < X < 10).

Solution (a) Using the normal table, P(Z ≥ 1.13) = 1 − 0.8708 = 0.1292. The shaded part in the graph represents the P(Z ≥ 1.13). 0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1 0.0

0.0 1.13

22.5 5

10

3.2 Special Distribution Functions 127

(b) Using the z-transform, we have

P(−2.5 < X < 10) = P

−2.5 − 5 10 − 5 z0 ) = 0.25. (b) P(Z < z0 ) = 0.95. (c) P(Z < z0 ) = 0.12. (d) P(Z > z0 ) = 0.68.

Solution (a) From the normal table, and using the fact that the shaded area in the figure is 0.25, we obtain z0 ≈ 0.675. (b) Because P(Z < z0 ) = 1 − P(Z ≥ z0 ) = 0.95 = 0.5 + 0.45. This implies, P(Z > z0 ) = 0.05. From the normal table, z0 = 1.645.

0.5 0.4 0.3 0.2 0.1 0.0 Z0

(c) From the normal table, z0 = −1.175. (d) Using the normal table, we have P(Z > z0 ) = 0.5 + P(0 < Z < z0 ) = 0.68. This implies, P(Z ≤ z0 ) = 0.32. From the normal table, z0 = − 0.465.

Example 3.2.10 The scores of an examination are assumed to be normally distributed with μ = 75 and σ 2 = 64. What is the probability that a score chosen at random will be greater than 85?

128 CHAPTER 3 Additional Topics in Probability

Solution Let X be a randomly chosen score from the exam scores. Then, X ∼ N(75, 64).

X − 75 85 − 75 P(X > 85) = P > = 1.25 8 8 = P(Z > 1.25) = 0.1056. 0.5 0.4 0.3 0.2 0.1 0.0 1.25

Thus, there is about a 10.56% chance that the score will be greater than 85.

In practice, whenever a large number of small effects are present and acting additively, it is reasonable to assume that observations will be normal. When the number of data is small, it is risky to assume a normal distribution without a proper testing. Apart from histogram, box-plot, and stem-and-leafdisplays, one of the most useful tools for assessing normality is a quantile quantile or QQ plot. This is a scatterplot with the quantiles of the scores on the horizontal axis and the expected normal scores on the vertical axis. The expected normal scores are calculated by taking the z-scores of (ri −0.5)/n, where ri is the rank ith observation in increasing order. The steps in constructing a QQ plot are as follows: First, we sort the data in an ascending order. If the plot of these scores against the expected normal scores is a straight line, then the data can be considered normal. Any curvature of the points indicates departures from normality. This procedure obtaining a normal plot (QQ plot is similar to normal plot for a normal distribution) is described in Project 4C. Figure 3.3 shows a normal probability plot generated by Minitab. If plotted points do not ﬁt the line well, but bend away from it in places, the distribution may be nonnormal. The shapes in Figure 3.4 will give some indication of the distribution of the data. 0.999 0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 21.5

21.0

■ FIGURE 3.3 Normal probability plot.

20.5

0.0

0.5

3.2 Special Distribution Functions 129

If the layout of points appears to bend up and to the left of the normal line that indicates a long tail to the right, or right skew.

If the layout of points bends down and to the right of the normal line that indicates a long tail to the left, or left skew.

An S-shaped layout of points indicates shorter than normal tails, thus, a smaller variance is expected.

If the layout of points starts below the normal line, bends to follow it, and ends above it, this will indicate long tails. That is, there is more variance than we would expect in a normal distribution.

■ FIGURE 3.4 Shapes indicating distribution of the data.

Almost all of the statistical software packages include a procedure for obtaining the graph of a normal probability plot that can be used to test the normality of a data. A discussion of how to do this is given in Section 14.4. Errors in the measurements can also act in a multiplicative (rather than additive) manner. In that case, the assumption of normality is not justiﬁed. A closely related distribution to normal distribution is the log-normal distribution. A variable might be modeled as log-normal if it can be thought of as the multiplicative effect of many small independent factors. This distribution arises in physical problems when the domain of the variate, X, is greater than zero and its histogram is markedly skewed. If a random variable Y is normally distributed, then exp(Y ) has a log-normal distribution. Thus, the natural logarithm of a log-normally distributed variable is normally distributed. That is, if X is a random variable with log-normal distribution, then ln(X) is normally distributed. Most biological evidence suggests that the growth processes of living tissue proceed by multiplicative, not additive, increments. Thus, the measures of body size should at most follow a log-normal rather than normal distribution. Also, the sizes of plants and animals is approximately log-normal. The log-normal distribution is also useful in modeling of claim sizes in the insurance industry. The probability density function of a log-normal random variable, X, is given as f (x) =

⎧ ⎨ ⎩

2 2 1 √ e−(ln x−μy ) /2σy , xσy 2π

x > 0, σy > 0,

0,

otherwise.

−∞ < μy < ∞

130 CHAPTER 3 Additional Topics in Probability

where μy and σy are the mean and standard deviation of Y = ln(X). These parameters are related to the parameters of the random variable X as follows: + μy = ln

+

μ4x , μ2x + σx2

σy = ln

μ2x + σx2 . μ2x

We can verify that the expected value X is E(X) = eμy +(σy /2) 2

and the variance is Var(X) = (eσy − 1)e2μy +σy . 2

2

The question of when the log-normal distribution is applicable in a given physical problem after a certain amount of data has been obtained can be answered by creating a normal probability plot of ln(X) and testing for normality. Thus, if the natural logarithms of the data show normality, log-normal distribution may be more appropriate. If X is log-normally distributed with parameters μy and σy , and 0 < a < b, then with Y = ln(X) P(a ≤ X ≤ b) = P(ln a ≤ Y ≤ ln b)

Y − μy ln b − μy ln a − μy ≤ ≤ =P σy σy σy = P(a ≤ Z ≤ b ),

where Z ∼ N(0, 1). This probability can be obtained from the standard normal table.

Example 3.2.11 In an effort to establish a suitable height for the controls of a moving vehicle, information was gathered about X, the amounts by which the heights of the operators vary from 60 inches, which is the minimum height. It was veriﬁed that the data that were collected followed the log-normal distribution by normal probability plot of Y = ln X. Assume that μx = 6 in. and σx = 2 in. (a) What percentage of operators would have a height less than 65.5 in.? (b) If an operator is chosen at random, what is the probability that his or her height will be between 64 and 66 in.?

Solution (a) Here, X = 65.5 − 60 = 5.5. Also, + μy = ln

μ4x 2 μx + σx2

+

= ln

64 62 + 2 2

= 1.74,

3.2 Special Distribution Functions 131 + σy = ln

μ2x + σx2 μ2x

+

= ln

Thus,

62 + 22 = 0.053. 62

(ln 5.5) − 1.74 P(X ≤ 5.5) = P(Y ≤ ln 5.5) = P Z ≤ 0.053

= P(Z ≤ −0.67) = 0.2514. Hence, about 25.14% of the heights of the operators vary from 60 inches. (b) Similar to part (a), we get P(4 ≤ X ≤ 6) = P(ln 4 ≤ Y ≤ ln 6)

(ln 6) − 1.74 (ln 4) − 1.74 ≤Z≤ =P 0.053 0.053 = P(−6.67 ≤ Z ≤ 0.98) = 0.8365.

3.2.5 Gamma Probability Distribution The gamma probability distribution has found applications in various ﬁelds. For example, in engineering, the gamma probability distribution has been employed in the study of system reliability. We describe the gamma function before we introduce the gamma probability distribution. The gamma function, denoted by (a), is deﬁned as (a) =

∞ e−x xa−1 dx, a > 0. 0

It can be shown using the integration by parts that for a > 1, (a) = (a − 1) (a − 1). In particular, if n is a positive integer, (n) = (n − 1)!. Deﬁnition 3.2.6 A random variable X is said to possess a gamma probability distribution with parameters α > 0 and β > 0 if it has the pdf given by f (x) =

⎧ ⎨

1 xα−1 e−x/β , βα (α) ⎩ 0,

if x > 0 otherwise.

The gamma density has two parameters, α and β. We denote this by Gamma(α, β). The parameter α is called a shape parameter, and β is called a scale parameter. Changing α changes the shape of the density, whereas varying β corresponds to changing the units of measurement (such as changing from seconds to minutes). Varying these two parameters will generate different members of the gamma family. If we take α to be a positive integer, we get a special case of gamma probability distribution, known as the Erlang distribution. This is used extensively in queuing theory to model waiting times. Figure 3.5 gives an indication of how α and β inﬂuence the shape and scale of f (x).

132 CHAPTER 3 Additional Topics in Probability

Gamma pdfs for (2, 3), (3, 1), (4, 3), and (2, 4) 0.3 Gam(3, 1)

0.25 0.2 0.15

Gam(2, 3)

0.1

Gam(2, 4) Gam(4, 3)

0.05 0

0

5

10

15

20

25

■ FIGURE 3.5 Gamma pdfs for different degrees of freedom.

MEAN, VARIANCE, AND MGF OF A GAMMA RANDOM VARIABLE Theorem 3.2.6 If X is a gamma random variable with parameters α > 0 and β > 0, then E(X) = αβ

and

Var(X) = αβ2 .

Also, the moment-generating function is MX (t) =

1 , (1 − βt)α

t

0.

Hence, using integration by parts, we obtain 1 P(X < 1) = 2

1 0

x2 e−x dx = 1 −

5 = 0.08025. 2e

3.2 Special Distribution Functions 133

0.30 0.25 0.20 0.15 0.10 0.05 0.00

1

Thus, there is about an 8% chance that on a given day the fuel consumption will be less than 1 million gallons. (b) Because the airport can store only 2 million gallons, the fuel supply will be inadequate if the fuel consumption X is greater than 2. Thus, 1 P(X > 2) = 2

∞ x2 e−x dx = 0.677. 2

0.30 0.25 0.20 0.15 0.10 0.05 0.00

2

We can conclude that there is about a 67.7% chance that the fuel supply of 2 million gallons will be inadequate on a given day. So, if the model is right, the airport needs to store more than 2 million gallons of fuel.

We now describe two special cases of gamma probability distribution. In the pdf of the gamma, we let α = 1, we get the pdf of an exponential random variable. Deﬁnition 3.2.7 A random variable X is said to have an exponential probability distribution with parameter β if the pdf of X is given by ⎧ ⎨ 1 e−x/β , f (x) = β ⎩ 0,

β > 0; 0 ≤ x < ∞ otherwise.

Exponential random variables are often used to model the lifetimes of electronic components such as fuses, for survival analysis, and for reliability analysis, among others. The exponential distribution (Figure 3.6) is also used in developing models of insurance risks.

134 CHAPTER 3 Additional Topics in Probability

Exponential (3) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0

5

10

15

20

■ FIGURE 3.6 Probability density function for exponential r.v.

MEAN, VARIANCE, AND MGF OF AN EXPONENTIAL RANDOM VARIABLE Theorem 3.2.7 If X is an exponential random variable with parameters β > 0, then E(X) = β

and

Var(X) = β2 .

Also the moment-generating function is MX (t) =

1 , (1 − βt)

t

200) = 200 dx = 0.2865. The chance that the generator will last more than 160 e 200 hours is about 28.65%.

Another special case of gamma probability distribution that is useful in statistical inference problems is the chi-square distribution. Deﬁnition 3.2.8 Let n be a positive integer. A random variable, X, is said to have a chi-square (χ2 ) distribution with n degrees of freedom if and only if X is a gamma random variable with parameters α = n/2 and β = 2. We denote this by X ∼ χ2 (n). Hence, the probability density function of a chi-square distribution with n degrees of freedom is given by ⎧ ⎨ 1 x(n/2)−1 e−x/2 , f (x) = 2n 2n/2 ⎩ 0,

0≤x 0) (d) E(X) and Var(X)

3.2.2.

Let X be a Poisson random variable with λ = 1/3. Find (a) P(X = 0) (b) P(X ≥ 4).

3.2.3.

For a standard normal random variable Z, ﬁnd the value of z0 such that (a) P(Z > z0 ) = 0.05 (b) P(Z < z0 ) = 0.88 (c) P(Z < z0 ) = 0.10 (d) P(Z > z0 ) = 0.95.

3.2 Special Distribution Functions 137

3.2.4.

Let X ∼ N(12, 5). Find the value of x0 such that (a) P(X > x0 ) = 0.05 (b) P(X < x0 ) = 0.98 (c) P(X < x0 ) = 0.20 (d) P(X > x0 ) = 0.90.

3.2.5.

Let X ∼ N(10, 25). Compute (a) P(X ≤ 20) (b) P(X > 5) (c) P(12 ≤ X ≤ 15) (d) P(|X − 12| ≤ 15).

3.2.6.

A quarterback on a football team has a pass completion rate of 0.62. If, in a given game, he attempts 16 passes, what is the probability that he will complete (a) 12 passes? (b) More than half of his passes? (c) Interpret your result. (d) Out of the 16 passes, what is the expected number of completions?

3.2.7.

A consulting group believes that 70% of the people in a certain county are satisﬁed with their health coverage. Assuming that this is true, ﬁnd the probability that in a random sample of 15 people from the county: (a) Exactly 10 are satisﬁed with their health coverage, and interpret. (b) Not more than 10 are satisﬁed with their health coverage, and interpret. (c) What is the expected number of people out of 15 that are satisﬁed with their health coverage?

3.2.8.

A man ﬁres at a target six times; the probability of his hitting it each time is independent of other tries and is 0.40. (a) What is the probability that he will hit at least once? (b) How many times must he ﬁre at the target so that the probability of hitting it at least once is greater than 0.77? (c) Interpret your ﬁndings.

3.2.9.

A certain electronics company produces a particular type of vacuum tube. It has been observed that, on the average, three tubes of 100 are defective. The company packs the tubes in boxes of 400. What is the probability that a certain box of 400 tubes will contain (a) r defective tubes? (b) At least k defective tubes? (c) At most one defective tube? (d) Interpret your answers to (a), (b), and (c).

3.2.10.

Suppose that, on average, in every two pages of a book there is one typographical error, and that the number of typographical errors on a single page of the book is a Poisson r.v. with λ = 1/2. What is the probability of at least one error on a certain page of the book? Interpret your result.

138 CHAPTER 3 Additional Topics in Probability

3.2.11.

Show that the probabilities assigned by Poisson probability distribution satisfy the requirements that 0 ≤ p(x) ≤ 1 for all x and x p(x) = 1.

3.2.12.

In determining the range of an acoustic source using the triangulation method, the time at which the spherical wave front arrives at a receiving sensor must be measured accurately. Measurement errors in these times can be modeled as possessing uniform probability distribution from −0.05 to 0.05 microseconds. What is the probability that a particular arrival time measurement will be in error by less than 0.01 microsecond? What does your answer mean?

3.2.13.

The hardness of a piece of ceramic is proportional to the ﬁring time. Assume that a rating system has been devised to rate the hardness of a ceramic piece and that this measure of hardness is a random variable that is distributed uniformly between 0 and 10. If a hardness in [5,9] is desirable for kitchenware, what is the probability that a piece chosen at random will be suitable for kitchen use?

3.2.14.

A receiver receives a string of 0s and 1s transmitted from a certain source. The receiver used a majority rule. That is, if the receiver acquires ﬁve symbols, of which three or more are 1s, it decides that a 1 was transmitted. The receiver is correct only 85% of the time. What is P(W ), the probability of a wrong decision if the probabilities of receiving 0s and 1s are equally likely? What can you conclude from your result?

3.2.15.

The efﬁciency X of a certain electrical component may be assumed to be a random variable that is distributed uniformly between 0 and 100 units. What is the probability that X is: (a) Between 60 and 80 units? (b) Greater than 90 units? (c) Interpret (a) and (b).

3.2.16.

The reliability function of a system or a piece of equipment at time t is deﬁned by R(t) = P(T ≥ t) = 1 − F (t)

where T , the failure time, is a random variable with a known distribution. A certain vacuum tube has been observed to fail uniformly over the interval [t1 , t2 ]. (a) Determine the reliability of such a tube at time t, t1 ≤ t ≤ t2 . (b) If 180 ≤ t ≤ 220, what is the reliability of such a tube at 200 hours? (c) The failure or hazard rate function ρ(t) is deﬁned by ρ(t) =

− dR(t) f (t) f (t) dt . = = 1 − F (t) R(t) R(t)

Calculate the failure rate of this vacuum tube. Interpret your result. 3.2.17.

An electrical component was studied in the laboratory, and it was determined that its failure rate was approximately equal to β1 = 0.05. What is the reliability of such a component at 10 hours?

3.2 Special Distribution Functions 139

3.2.18.

Suppose that the life length of a mechanical component is normally distributed. (a) If σ = 3 and μ = 100, ﬁnd the reliability of such a system at 105 hours. (b) What should be the expected life of the component if it has reliability of 0.90 for 120 hours?

3.2.19.

A geologist deﬁnes granite as a rock containing quartz, feldspar, and small amounts of other minerals, provided that it contains not more than 75% quartz. If all the percentages are equally likely, what proportion of granite samples that the geologist collects during his lifetime will contain from 50% to 65% quartz?

3.2.20.

For a normal random variable with pdf, 2 2 1 e−(x−μ) /2σ , f (x) = √ 2πσ

show that

∞

−∞ f (x)dx

∞<x 1, (a) = (a − 1) (a − 1).

3.2.29.

(a) Find the moment-generating function for a gamma probability distribution with parameter α > 0 and β > 0. [Hint: In the integral representation of E(etX ), change the variable t to u = (1 − βt)x/β, with (1 − βt) > 0.] (b) Using the mgf of a gamma probability distribution, ﬁnd E(X) and Var(X).

3.2.30.

Let X be an exponential random variable. Show that, for numbers a > 0 and b > 0, P(X > a + b |X > a ) = P(X > b).

(This property of the exponential distribution is called the memoryless property of the distribution.) 3.2.31.

A random variable X is said to have a beta distribution with parameters α and β if and only if the density function of X is f (x) =

where B(α, β) =

1 0

⎧ α−1 β−1 ⎨ x (1−x) , ⎩

B(α,β)

α, β > 0; 0 ≤ x ≤ 1

0,

otherwise

xα−1 (1 − x)β−1 dx. (α) (β) (α+β) . α α+β and Var(X)

(a) Show that B(α, β) = (b) Show that E(X) = 3.2.32.

=

αβ . (α+β)2 (α+β+1)

The daily proportion of major automobile accidents across the United States can be treated as a random variable having a beta distribution with α = 6 and β = 4. Find the probability that, on a certain day, the percentage of major accidents is less than 80% but greater than 60%. Interpret your answer.

3.3 Joint Probability Distributions 141

3.2.33.

Suppose that network breakdowns occur randomly and independently of each other on an average rate of three per month. (a) What is the probability that there will be just one network breakdown during December? Interpret. (b) What is the probability that there will be at least four network breakdowns during December? Interpret. (c) What is the probability that there will be at most seven network breakdowns during December? Interpret.

3.2.34.

Let X be a random variable denoting the number of events occurring in the time interval (0, t]. Show that X has a gamma probability distribution with parameters n and λ.

3.2.35.

In order to etch an aluminum tray successfully, the pH of the acid solution used must be between 1 and 4. This acid solution is made by mixing a ﬁxed quantity of etching compound in powder form with a given volume of water. The actual pH of the solution obtained by this method is affected by the potency of the etching compound, by slight variations in the volume of water used, and perhaps by the pH of the water. Thus, the pH of the solution varies. Assume that the random variable that describes the random phenomenon is gamma distributed with α = 2 and β = 1. (a) What is the probability that an acid solution made by the foregoing procedure will satisfactorily etch a tray? (b) What would the answer to part (a) be if α = 1 and β = 2?

3.3 JOINT PROBABILITY DISTRIBUTIONS We have thus far conﬁned ourselves to studying one-dimensional or univariate random variables and their properties. In many practical situations, we are required to deal with several, not necessarily independent random variables. For example, we might be interested in a study involving the weights and heights (W, H) of a certain group of persons. In this situation, we need the two random variables (W, H), and it is likely that these two are related. Then it becomes important to study the joint effect of these random variables, which will lead to ﬁnding the joint probability distributions. In this section, we conﬁne our studies to two random variables and their joint distributions, which are called bivariate distributions. We consider the random variables to be either both discrete or both continuous. We now deﬁne joint distribution of two random variables. Deﬁnition 3.3.1 (a) Let X and Y be random variables. If both X and Y are discrete, then f (x, y) = P(X = x, Y = y)

is called the joint probability function (joint pmf ) of X and Y . (b) If both X and Y are continuous then f (x, y) is called the joint probability density function (joint pdf ) of X and Y if and only if b d P(a ≤ X ≤ b, c ≤ Y ≤ d) =

f (x, y)dxdy. a c

142 CHAPTER 3 Additional Topics in Probability

Example 3.3.1 A probability class contains 10 African American, 8 Hispanic American, and 15 white students. If 12 students are randomly selected from this class, and if X = number of black students, and Y = number of white students, ﬁnd the joint probability function of the bivariate random variable (X, Y ).

Solution There are a total of 33 students. The number of ways in which x African American, and y white students can be picked (which means, the remaining 12 − (x + y) students are Hispanic American) can be obtained using the multiplication principle as

10

15

8

x

x

12 − x − y

The number of ways to pick 12 students from 33 students is

.

33

. Hence, the joint probability function is

12 P(X = x, Y = y) =

10

15

x

y

8

12 − x − y 33

12 where 0 ≤ x ≤ 10, 0 ≤ y ≤ 12, and 4 ≤ x + y ≤ 12. The last constraint is needed because there are only eight Hispanic Americans, so the combined minimum number of whites and African Americans should be at least 4.

We follow the notation: x,y to denote x y . The joint distribution of two random variables has to satisfy the following conditions. Theorem 3.3.1 If X and Y are two random variables with joint probability function f (x, y), then 1. f (x, y) ≥ 0 for all x and y. 2. If X and Y are discrete, then x,y f (x, y) = 1, where the sum is over all values (x, y) that are assigned nonzero probabilities. If X and Y are continuous, then ∞ ∞ f (x, y) = 1. −∞ −∞

Given the joint probability distribution (pdf or pmf ), the probability distribution function of a component random variable can be obtained through the marginals.

3.3 Joint Probability Distributions 143

Deﬁnition 3.3.2 The marginal pmf of X denoted by fX (x) (or f (x), when there is no confusion) is deﬁned by ⎧ ∞ ⎪ ⎪ f (x, y)dy, ⎨

if X and Y are continuous,

fX (x) = −∞ ⎪ ⎪ f (x, y), ⎩

if X and Y are discrete.

all y

Similarly, the marginal pdf of Y is deﬁned by ⎧ ∞ ⎪ ⎪ f (x, y)dx, ⎨

if X and Y are continuous,

fY (y) = −∞ ⎪ ⎪ f (x, y), ⎩

if X and Y are discrete.

all x

Note that

⎧ b ⎪ ⎨ f (x)dx, X P(a ≤ X ≤ b) = a ⎪ ⎩ fX (x),

if X and Y are continuous, if X and Y are discrete,

where summation is over all values of X from a to b.

Example 3.3.2 Find the marginal probability density function of the random variables X and Y , if their joint probability function is given by Table 3.1.

Table 3.1 y x

−2

0

1

4

Sum

−1

0.2

0.1

0.0

0.2

0.5

3 5

0.1 0.1

0.2 0.0

0.1 0.0

0.0 0.0

0.4 0.1

Sum

0.4

0.3

0.1

0.2

1.0

Find the marginal densities of X and Y .

Solution By definition, the marginal pdfs of X are given by the column sums (summands over y for fixed x), and the marginal pdfs of Y are obtained by the row sums. Hence, xi −1 3 5 otherwise fX (xi ) 0.5 0.4 0.1 0

yj −2 0 1 4 otherwise fY (yi ) 0.4 0.3 0.1 0.2 0

144 CHAPTER 3 Additional Topics in Probability

Using the joint probability distribution and the marginals, we can now introduce the conditional probability distribution function. Deﬁnition 3.3.3 The conditional probability distribution of the random variable X given Y is given by f (x |y ) = f (x |Y = y ) ⎧ f (x, y) ⎪ ⎪ , ⎨ fY (y) = ⎪ P(X = x, Y = y) ⎪ ⎩ , fY (y)

if X and Y are continuous, fY (y) = 0, if X and Y are discrete.

We note that both the marginal probability densities of X and Y as well as the conditional pdf must satisfy the two important conditions of a pdf. We know that two events A and B are independent if P(A ∩ B) = P(A)P(B). It is usually more convenient to establish independence through the probability functions. Hence, we deﬁne independence for bivariate probability distribution as follows. Deﬁnition 3.3.4 Let X and Y have a joint pmf or pdf f (x, y). Then X and Y are independent if and only if f (x, y) = fX (x)fY (y),

for all x and y.

That is, for independent random variables, the joint pdf is the product of the marginals.

Example 3.3.3 Let f (x, y) =

3x,

0 ≤ y ≤ x ≤ 1,

0,

otherwise.

(a) Find P X ≤ 12 , 14 < Y < 34 . (b) Find the marginals fX (x) and fY (y). (c) Find the conditional f (x |y )(0 < y < 1). Also compute f x|Y = 12 . (d) Are X and Y independent?

Solution (a) The domain of the function f(x,y) is given in Figure 3.8. The required probability P X ≤ 12 , 14 < Y < 34 is the volume over the area of the shaded region as shown by Figure 3.9. That is,

P X ≤ 12 , 14 < Y < 34

1/2 x =

3xdydx 1/4 1/4

3.3 Joint Probability Distributions 145

1 dx 3x x − 4 1/4 1/2 3x3 3x2 = − 3 8

=

1/2

1/4

5 . = 128 y

1

f (x, y )⫽ 3x in this region x

1

■ FIGURE 3.8 Domain of f (x, y).

y y⫽x

1

The region 0 ⬍ x ⬍ 1/2 and 1/4 ⬍ y ⬍ 3/4

0.0

0.2

0.4

0.6

0.8

1.0

x 1.2

■ FIGURE 3.9 Region of integration.

(b) To find the marginals, we note that for each x, y varies from 0 to x(0 < y < x). Therefore x fX (x) =

3xdy = 3x y|x0 = 3x2 ,

0 < x < 1.

0

Similarly, for each y, x varies from y to 1. 1 fY (y) = y

1 3x2 3 3y2 3xdx = = − 2 2 2 y

3 = (1 − y2 ), 2

0 < y < 1.

146 CHAPTER 3 Additional Topics in Probability

(c) Using the definition of conditional density 2x f (x, y) 3x = = 3 , 2) fY (y) 1 − y2 (1 − y 2

f (x |y ) =

y ≤ x ≤ 1.

From this we have f x |y = 12 =

8 2x 2 = 3 x, 1 − 12

1 ≤ x ≤ 1. 2

(d) To check for independence of X and Y 1 fX (1)fY 12 = (3) 98 = 27 8 = 3 = f 1, 2 . Hence, X and Y are not independent.

Recall that in the case of a univariate random variable X, with probability function f (x), we have ⎧ ⎨ xf (x), EX = x ⎩ xf (x)dx,

|x|f (x) < ∞, for discrete r.v. if x if |x|f (x)dx < ∞, for continuous r.v.

Now we deﬁne similar concepts for bivariate distribution. Deﬁnition 3.3.5 Let f (x, y) be the joint probability function, and let g(x, y) be such that ∞ ∞ x,y |g(x, y)|f (x, y) < ∞ in the discrete case, or −∞ −∞ |g(x, y)|f (x, y)dxdy < ∞, in the continuous case. Then the expected value of g(X, Y ) is given by

Eg(X, Y ) =

⎧ ⎪ ⎪ ⎨

g(x, y)f (x, y),

if X, Y are discrete,

x,y

∞ ∞ ⎪ ⎪ g(x, y)f (x, y)dxdy, ⎩ −∞ −∞

if X, Y are continuous.

In particular

E(X, Y ) =

⎧ ⎪ ⎪ ⎨

xyf (x, y),

if X, Y are discrete,

x,y

∞ ∞ ⎪ ⎪ xyf (x, y)dxdy, ⎩ −∞ −∞

if X, Y are continuous.

The following properties of mathematical expectation are easy to verify.

PROPERTIES OF EXPECTED VALUE 1. E(aX + bY ) = aE(X ) + bE(Y ). 2. If X and Y are independent, then E(XY ) = E(X )E(Y ). However, the converse is not necessarily true.

3.3 Joint Probability Distributions 147

Example 3.3.4 Let f (x, y) = 3x, 0 ≤ y ≤ x ≤ 1. (a) Find E(4X − 3Y ), (b) Find E(XY ).

Solution

(a) E(X) = xfX (x)dx and E(Y ) = yfY (y)dy. Recall that earlier (Example 3.3.3) we have computed fX (x) = 3x2 (0 < x < 1) and fY (y) = 3 (1 − y2 ), 0 ≤ y ≤ 1. Using these results, we have 2 1 E(X) =

x3x2 dx =

3 , 4

0

1 E(Y ) =

3 3 y (1 − y2 )dy = . 2 8

0

Hence, E(4X − 3Y ) = 3 −

15 9 = . 8 8

(b) 1x xy(3x)dydx =

E(XY ) =

3 . 10

0 0

Conditional expectations are deﬁned in the same way as univariate expectations, except that the conditional density is utilized in place of the unconditional density function. Deﬁnition 3.3.6 Let X and Y be jointly distributed with pf or pdf f (x, y). Let g be a function of x. Then the conditional expectation of g(x) given, Y = y is E(g(X) |y ) = E(g(X) |Y = y ) ⎧ g(x)f (x |y ), ⎨ = all x ⎩ g(x)f (x |y )dx,

if X, Y are discrete, if X, Y are continuous.

Note that E(g(X) |y ) is a function of y. If we let Y range over all of its possible values, the conditional expectation E(g(X) |Y ) can be thought of as a function of the random variable Y . We will then be able to ﬁnd the mean and variance of E(g(X) |Y ), as given in the following result, the proof of which is left as an exercise. Theorem 3.3.2 Let X and Y be two random variables. Then (a) E(X) = E[E(X|Y )]. (b) Var(X) = E[Var(X|Y )] + Var[E(X|Y )].

148 CHAPTER 3 Additional Topics in Probability

Example 3.3.5 Let X and Y be two random variables with joint density function given by ⎧ ⎨x2 + xy , 0 ≤ x ≤ 1 and 0 ≤ y ≤ 2 3 f (x, y) = ⎩ 0, otherwise. Find the conditional expectation, E X|Y = 12 .

Solution First we will find the conditional density, f (x |y ). The marginal fY (y) =

1

x2 +

xy 1 1 dx = + y, 3 3 6

0 < y < 2.

0

Therefore, f (x|y) =

x2 + xy f (x, y) 3 , = 1 1 fY (y) y + 6 3

0 ≤ x ≤ 1.

Hence, x2 + 6x 12 2 x x + . = f x|Y = 12 = 1 1 5 6 12 + 3 Thus,

E X|Y = 12

1 xf (x |y ) dx

= 0

1 =

x

11 12 2 x x + dx = = 0.733. 5 6 15

0

3.3.1 Covariance and Correlation We will now deﬁne the covariance and correlation coefﬁcient of two random variables. Deﬁnition 3.3.7 (i) The covariance between two random variables X and Y is deﬁned by σXY = Cov(X, Y ) = E(X − μX )(Y − μY ) = E(XY ) − μX μY ,

where μX = E(X) and μY = E(Y ).

3.3 Joint Probability Distributions 149

(ii) The correlation coefﬁcient, ρ = ρ(x, y) is deﬁned by Cov(X, Y ) ρ= √ . Var(X)Var(Y )

Correlation is the measure of the linear relationship between the random variables X and Y . If Y = aX+b(a = 0), then ρ(x, y) = 1. If dependence on X and Y needs to be speciﬁed, we will use the notation, ρXY . From the deﬁnition of the covariance of X and Y , we note that if small values of X, for which (X − μX ) < 0, tend to be associated with small values of Y , for which (Y − μY ) < 0, and similarly large values of X with large values of Y, then Cov(X, Y ) ≡ E[(X − μX )(Y − μY )] can be expected to be positive. On the other hand, if small values of X tend to be associated with large values of Y and vice versa so that (X − μX ) and (Y − μY ) are of opposite signs, then Cov(X, Y ) < 0. Thus, covariance can be thought of as a signed measure of the variation of Y relative to X. If X and Y are independent, then it follows from the deﬁnition of covariance that Cov(X, Y ) = 0. The correlation coefﬁcient of X and Y , is a dimensionless quantity that measures the linear relationship between the random variables X and Y . PROPERTIES OF COVARIANCE AND CORRELATION COEFFICIENT (a) −1 ≤ ρ ≤ 1. (b) If X and Y are independent, then ρ = 0. The converse is not true. (c) If Y = aX + b, then Cov (X; Y ) =

⎧ ⎨ 1,

if a > 0,

⎩−1,

if a < 0.

Note that Cov (X , X ) = Var (X ). (d) If U = a1 X + b1 and V = a2 Y + b2 , then (i) Cov (U, V ) = a1 a2 Cov (X , Y ), and (ii) ρUV =

⎧ ⎨ ρXY ,

if a1 a2 > 0

⎩−ρ

otherwise.

XY ,

(e) Var (aX + bY ) = a2 Var (X ) + b 2 Var (Y ) + 2abCov (X , Y ).

Example 3.3.6 The joint probability density of the random variables X and Y is given by ⎧ ⎨ 1 e−y/8 , f (x, y) = 64 ⎩ 0,

0≤x≤y 1). 3.3.10.

The joint pdf of X and Y is f (x, y) =

⎧ ⎨ 1 (4x + 2y + 1), 28 ⎩

0 ≤ x ≤ 2, 0 ≤ y ≤ 2

0,

elsewhere.

Find (a) fX (x) and fY (y), and (b) f (y |x ). 3.3.11.

Find the joint mgf of the random variables (X, Y ) deﬁned in Problem 3.3.9.

3.3.12.

The joint density of a random variable (X, Y ) is given by f (x, y) =

⎧ 3 3 ⎨x y , ⎩

16

0 ≤ x ≤ 2, 0 ≤ y ≤ 2

0,

elsewhere.

(a) Find marginals of X and Y , and (b) ﬁnd f (y |x ). 3.3.13.

The joint probability function of a discrete random variable (X, Y ) is given by f (x, y) =

⎧ ⎨ ⎩

2 6xy n(n+1)(2n+1) ,

x, y = 1, 2, . . . , n,

0,

otherwise.

Find (a) f (x |y ), and (b) f (y |x ). [Hint: ni=1 i2 = (n(n + 1)(2n + 1))/6.] 3.3.14.

Consider bivariate random variables with the density n x+α−1 f (x, y) = y (1 − y)n−x+β−1 , x

for x = 0, 1, . . . , n and 0 < y ≤ 1.

3.3 Joint Probability Distributions 153

Verify that n x f (x |y ) ∝ y (1 − y)n−x x

and f (y|x) ∝ yx+α−1 (1 − y)n−x+β−1 .

3.3.15.

The joint density function of the discrete random variable (X, Y ) is given in Table 3.3.2.

Table 3.3.2 y x

1

2

3

1

1 6

1 6

1 6

2

1 6

1 12

1 12

3

1 12

1 12

0

(a) Find E(XY ). (b) Find Cov(X, Y ). (c) Find the correlation coefﬁcient ρX,Y . 3.3.16.

The joint probability function of the continuous random variable (X, Y ) is given by f (x, y) =

⎧ ⎨ 1 (4x + 2y + 1), 28 ⎩

0 ≤ x < 2, 0 ≤ y < 2,

0,

otherwise.

(a) Find E(XY ). (b) Find Cov(X, Y ). (c) Find the correlation coefﬁcient ρXY . 3.3.17.

Let X and Y be random variables and U = aX + b, V = cY + d, where a, b, c, d are constants. , if ac > 0 ρXY , Show that ρUV = −ρXY , otherwise.

3.3.18.

Let X and Y be two independent random variables, and let Y = aX + b, where a and b are constants. Show that (a) ρXY = 1 if a > 0, and (b) ρXY = −1 if a < 0.

3.3.19.

If |ρXY | = 1, then prove that P(Y = aX + b) = 1.

154 CHAPTER 3 Additional Topics in Probability

3.3.20.

Let X and Y be two random variables with joint density function f (x, y) =

⎧ ⎨8xy,

0≤x≤y≤1

⎩ 0,

otherwise.

(a) Find the conditional expectation, E X|Y = 34 . (b) Find Cov(X, Y ). 3.3.21.

Let X and Y be two random variables with joint density function f (x, y) =

⎧ ⎨e−y ,

0≤x≤y

⎩ 0,

otherwise.

(a) Find the conditional expectation, E(X|Y = y). (b) Find Cov(X, Y ). (c) Are X and Y independent? Why? 3.3.22.

Let f (x, y) =

c

(1 + x2 ) 1 − y2

,

−∞ < x < ∞,

−1 < y < 1.

Find the c that makes f (x, y) the probability density function of the random variable (X, Y ). Determine whether X and Y are independent. 3.3.23.

If the random variables X and Y are independent and have equal variances, what is the coefﬁcient of correlation between the random variables X and aX +Y , where a is a constant?

3.4 FUNCTIONS OF RANDOM VARIABLES In this section we discuss the methods of ﬁnding the probability distribution of a function of a random variable X. We are given the distribution of X, and we are required to ﬁnd the distribution of g(X). There are many physical problems that call for the derivation of the distribution of a function of a random variable. The following is one of the classical examples. The velocity V of a gas molecule (Maxwell–Boltzmann law) behaves as a gamma-distributed random variable. We would like to derive the distribution of E = mV 2 , the kinetic energy of the gas molecule. Because the value of the velocity is the outcome of a random experiment, so is the value of E. This is a problem of ﬁnding the distribution of a function of a random variable E = g(V ). We now illustrate various techniques for ﬁnding the distribution of g(X) by means of examples.

3.4.1 Method of Distribution Functions Basically the method of distribution functions is as follows. If X is a random variable with pdf fX (x) and if Y is some function of X, then we can ﬁnd the cdf FY (y) = P(Y ≤ y) directly by integrating fX (x) over the region for which {Y ≤ y}. Now, by differentiating FY (y), we get the probability density function fY (y) of Y . In general, if Y is a function of random variables X1 , . . . , Xn , say g(X1 , . . . , Xn ), then we can summarize the method of distribution function as follows.

3.4 Functions of Random Variables 155

PROCEDURE TO FIND CDF OF A FUNCTION OF R.V. USING THE METHOD OF DISTRIBUTION FUNCTIONS 1. Find the region {Y ≤ y } in the (x1 , x2 , . . . , xn ) space, that is ﬁnd the set of (x1 , x2 , . . . , xn ) for which g(x1 , . . . , xn ) ≤ y . 2. Find FY (y ) = P(Y ≤ y ) by integrating f (x1 , x2 , . . . , xn ) over the region {Y ≤ y }. 3. Find the density function fY (y ) by differentiating FY (y ).

Example 3.4.1 Let X ∼ N(0, 1). Using the cdf of X, ﬁnd the pdf of X2 .

Solution Let Y = X2 . Note that the pdf of X is 2 1 f (x) = √ e−x /2 , 2π

−∞ < x < ∞.

Then the cumulative distribution function of Y for a given y ≥ 0 is F (y) = P(Y ≤ y) = P(X2 ≤ y) √ √ = P(− y ≤ X ≤ y) √

y =

2 1 √ e−x /2 dx 2π √

− y

√

y =2 0

2 1 √ e−x /2 dx, 2π

2 (by the symmetry of e−x /2 ).

Hence, by differentiating F (y), we obtain the probability density function as 2 1 fY (y) = √ e−y/2 √ 2 y 2π

=

⎧ ⎨ √1 y−1/2 e−y/2 ,

0 y, . . . , Xn > y) = P(X1 > y)P(X2 > y) . . . P(Xn > y) (because of independence) = (1 − F (y))n . This implies FY1 (y) = 1 − (1 − F (y))n and fY1 (y) = n(1 − F (y))n−1 f (y). Consider Yn . Its cdf is given by FYn (y) = P(Yn ≤ y) = (F (y))n . This implies that fYn (y) = n(F (y))n−1 f (y).

3.4 Functions of Random Variables 159

3.4.5 Transformation Method A simple generalization of the method of distribution functions to functions of more than one variable is the transformation method. We illustrate the method for bivariate distributions. The method is similar for the multivariate case. Let the joint pdf of (X, Y ) be f (x, y). Let U = g1 (X, Y ); V = g2 (X, Y ). The mapping from (X, Y ) to (U, V ) is assumed to be one-to-one and onto. Hence, there are functions, h1 and h2 such that x = h−1 1 (u, v),

and y = h−1 2 (u, v).

Deﬁne the Jacobian of the transformation J by ∂x ∂u J = ∂y ∂u

∂x ∂v . ∂y ∂u

Then the joint pdf of U and V is given by −1 f (u, v) = f (h−1 1 (u, v), h2 (u, v)) |J| .

Example 3.4.6 Let X and Y be independent random variables with common pdf f (x) = e−x , (x > 0). Find the joint pdf of U = X/(X + Y ), V = X + Y .

Solution We have U = X/(X + Y ) = X/V . Hence, X = UV and Y = V − X = V − UV = V (1 − U). Thus, the Jacobian v u J = . −v 1 − u Then |J| = v(1 − u) + uv = v(> 0). Note that 0 ≤ u ≤ 1, 0 < v < ∞. −1 f (u, v) = f h−1 1 (u, v), h2 (u, v) |J| = e−uv e−v(1−u) v = ve−v ,

0 ≤ u ≤ 1, 0 < v < ∞.

160 CHAPTER 3 Additional Topics in Probability

Suppose we want the marginal fV (v) and fU (v), that is, 1 fV (v) =

ve−v du = ve−v ,

0 0.

162 CHAPTER 3 Additional Topics in Probability

3.4.3.

Let f (x, y) be the probability density function of the continuous random variable (X, Y ). If U = XY , show that the probability density function of U is given by ∞ fU (u) =

f

1 , v dv. v v

u

−∞

3.4.4.

The joint pdf of X and Y is f (x, y) = θe−(x+θy) ,

θ > 0, x > 0.

Find the pdf of XY . 3.4.5.

If the joint pdf of (X, Y ) is

− 21 2 x2 +y2 1 f (x, y) = e 4σ1 σ2 , 2πσ1 σ2

− ∞ < x < ∞, − ∞ < y < ∞; σ1 , σ2 > 0

ﬁnd the pdf of X2 + Y 2 . 3.4.6.

Let X1 , . . . , Xn be independent and identically distributed random variables with pdf f (x) = (1/θ)e−x/θ , x > 0, θ > 0. Find the pdf of ni=1 Xi .

3.4.7.

Let f (x, y) be the pdf of the continuous random variable (X, Y ). If U = X + Y , then show that the probability density function of U is given by ∞ fU (u) =

f (u − v, v)dv. −∞

3.4.8.

Let X be uniformly distributed over (−2, 2) and Y = X2 . Find the Cov(X, Y ). Are X and Y independent?

3.4.9.

Let X ∼ N(μ, σ 2 ). Show that is N(0, 1). (a) Z = (X−μ) σ (b) U =

(X−μ)2 σ2

is χ2 (1).

3.4.10.

Let X ∼ N(μ, σ 2 ). Find the pdf of Y = eX.

3.4.11.

The probability density of the velocity, V , of a gas molecule, according to the Maxwell– Boltzmann law, is given by f (v, β) =

⎧ 2 ⎨cv2 e−βv , ⎩

0,

v > 0, elsewhere

where c is an appropriate constant and β depends on the mass of the molecule and the absolute temperature. Find the density function of the kinetic energy E, which is given by E = g(V ) = 12 mV 2 .

3.5 Limit Theorems 163

3.4.12.

Let X and Y be two independent random variables, each normally distributed, with parameters (μ1 , σ12 ), and (μ2 , σ22 ), respectively. Show that the probability density function of U = X/Y is given by fU (u) =

3.4.13.

σ1 σ2 2 , π σ1 + σ22 u2

−∞ < u < ∞.

Let f (x, y) =

1 −1/2σ 2 x2 +y2 e , 2πσ 2

−∞ < x, y < ∞

be the joint pdf of (X, Y ). Let U=

X2 + Y 2

V = tan−1

and

Y , X

0 ≤ V ≤ 2π.

Find the joint pdf of (U, V ). 3.4.14.

Let the joint pdf of (X, Y ) be given by f (x, y) =

⎧ ⎨β−2 e−{(x+y)/β} , ⎩

0,

x, y > 0, β > 0, elsewhere.

X−Y and V = Y . Find the joint pdf of (U, V ). 2 Let X and Y be independent and identically distributed random variables with pdf

Let U = 3.4.15.

f (x) =

⎧ ⎨ 1 e−x/2 , 2

⎩

0,

x ≥ 0, otherwise.

Find the distribution of (X − Y )/2. 3.4.16.

If X and Y are independent and chi-square distributed random variables with n1 and n2 degrees of freedom, respectively. Obtain the joint distribution of (U, V ), where U = X + Y and V = X/Y .

3.5 LIMIT THEOREMS Limit theorems play a very important role in the study of probability theory and in its applications. In Chapter 2, we saw that the frequency interpretation of probability depends on the long-run proportion of times the outcome (event) would occur in repeated experiments. Also, in Section 3.2, we learned that some binomial probabilities can be computed using either the Poisson probability distribution or the normal probability distribution using the limiting arguments. Many random variables that we encounter in nature have distributions close to the normal probability distribution. These modeling

164 CHAPTER 3 Additional Topics in Probability

simpliﬁcations are possible because of various limit theorems. In this section, we discuss the law of large numbers and the Central Limit Theorem. First we give Chebyshev’s theorem, which is a useful result for proving limit theorems. It gives a lower bound for the area under a curve between two points that are on opposite sides of the mean and are equidistant from the mean. The strength of this result lies in the fact that we need not know the distribution of the underlying population, other than its mean and variance. This result was developed by the Russian mathematician Pafnuty Chebyshev (1821–1894).

CHEBYSHEV’S THEOREM Theorem 3.5.1 Let the random variable X have a mean μ and standard deviation σ. Then for K > 0, a constant, P(|X − μ| < Kσ) ≥ 1 −

1 . K2

Proof. We will work with the continuous case. By deﬁnition of the variance of X, σ 2 = E(X − μ)2 =

∞

(x − μ)2 f (x)dx

−∞ μ−Kσ

(x − μ)2 f (x)dx +

= −∞

μ+Kσ

μ−Kσ

μ−Kσ

(x − μ)2 f (x)dx +

≥

(x − μ)2 f (x)dx +

−∞

∞

∞

(x − μ)2 f (x)dx

μ+Kσ

(x − μ)2 f (x)dx.

μ+Kσ

Note that (x − μ)2 ≥ K2 σ 2 for x ≤ μ − Kσ or x ≥ μ + Kσ. The equation above can be rewritten as ⎡

⎢ σ 2 ≥ K2 σ 2 ⎣

⎤

∞

μ−Kσ

⎥ f (x)dx⎦

f (x)dx + −∞

μ+Kσ

= K2 σ 2 [P{X ≤ μ − Kσ} + P{X ≥ μ + Kσ}] = K2 σ 2 P{|X − μ| ≥ Kσ}.

This implies that P{|X − μ| ≥ Kσ} ≤

1 K2

3.5 Limit Theorems 165

or P (|X − μ| < Kσ) ≥ 1 −

1 . K2

We can also write Chebyshev’s theorem as P{|X − μ| ≥ ε} ≤

# $ E (X − μ)2 ε2

=

Var(X) , ε2

for some ε > 0.

Equivalently, P{|X − μ| ≥ Kσ} ≤

1 . K2

In other words, Chebyshev’s inequality states that the probability that a random variable X differs from its mean by at least K standard deviations is less than or equal to 1/K2 (K ≥ 2). In statistics, if we do not have any idea of the population distribution, Chebyshev’s theorem is used in the following manner. For any data set (regardless of the shape of the distribution), at least (1−(1/k2 ))100% of observations will lie within k(≥ 1) standard deviations of the mean. For example, at least (1−(1/22 ))100% = 75% of the data will fall in the interval (x−2s, x+2s) and at least 88.9% of the observations will lie within three standard deviations of the mean. If the population distribution is bell shaped, we have a better result than Chebyshev’s theorem, namely, the empirical rule that states the following: (i) approximately 68% of the observations lie within one standard deviation of the mean; (ii) approximately 95% of the observations lie within two standard deviations of the mean; and (iii) approximately 99.7% of the observations lie within three standard deviations of the mean.

Example 3.5.1 A random variable X has mean 24 and variance 9. Obtain a bound on the probability that the random variable X assumes values between 16.5 to 31.5.

Solution From Chebyshev’s theorem. P {μ − Kσ < X < μ + Kσ} ≥ 1 − Equating μ + Kσ to 31.5 and μ − Kσ to 16.5 with μ = 24 and σ = Hence, P {16.5 < X < 31.5} ≥ 1 −

1 . K2 √ 9 = 3, we obtain K = 2.5.

1 = 0.84. (2.5)2

166 CHAPTER 3 Additional Topics in Probability

Example 3.5.2 Let X be a random variable that represents the systolic blood pressure of the population of 18- to 74-year-old men in the United States. Suppose that X has mean 129 mm Hg and standard deviation 19.8 mm Hg. (a) Obtain a bound on the probability that the systolic blood pressure of this population will assume values between 89.4 and 168.6 mm Hg. (b) In addition, assume that the distribution of X is approximately normal. Using the normal table, ﬁnd P(89.4 ≤ X ≤ 168.6). Compare this with the empirical rule.

Solution (a) Because we are given only the mean and standard deviation, and no distribution is specified, we use Chebyshev’s theorem. We have P {μ − Kσ < X < μ + Kσ} ≥ 1 −

1 . K2

Equating μ + Kσ to 168.6 and μ − Kσ to 89.4 with μ = 129 and σ = 19.8, we obtain K = 2. Hence, P {89.4 ≤ X ≤ 168.6} ≥ 1 −

1 = 0.75. (2)2

(b) Because X is normally distributed with mean 129 and standard deviation 19.8, using the z-score, we get

168.6 − 129 89.4 − 129 ≤Z≤ P(89.4 ≤ X ≤ 168.6) = P 19.8 19.8 = P(−2 ≤ Z ≤ 2) = 0.9544. Hence, approximately 95.44% of this population will have systolic blood pressure values between 89.4 and 168.6 mm Hg. This compares well with the 95% value from the empirical rule.

We could use Chebyshev’s inequality to prove the following result, which is called the weak law of large numbers. The law of large numbers states that if the sample size n is large, the sample mean rarely deviates from the mean of the distribution of X, which in statistics is called the population mean.

LAW OF LARGE NUMBERS Theorem 3.5.2 Let X1 , . . . , Xn be a set of pairwise independent random variables with E(Xi ) = μ, and var(Xi ) = σ 2 . Then for any c > 0, 0 / σ2 P μ−c ≤X ≤μ+c ≥1− 2 nc

3.5 Limit Theorems 167

and as n → ∞, the probability approaches 1. Equivalently,

Sn − μ < ε → 1 P n as n → ∞.

Proof. Because X1 , . . . , Xn are iid random variables, we know that Var(Sn ) = nσ 2 , and Var(Sn /n) = σ 2 /n. Also, E(Sn /n) = μ. By Chebyshev’s theorem, for any ε > 0,

Sn σ2 − μ ≥ ε ≤ 2 . P n nε

Thus, for any ﬁxed ε,

as n → ∞. Equivalently,

Sn − μ ≥ ε → 0 P n

Sn − μ < ε → 1 P n

as n → ∞. Thus, without any knowledge of the probability distribution function of Sn , the (weak) law of large numbers states that the sample mean, X = Sn /n, will differ from the population mean by less than an arbitrary constant, ε > 0, with probability that tends to 1 as n tends to ∞. Because of this, the law of large numbers is also called the “law of averages.” This result basically states that we can start with a random experiment whose outcome cannot be predicted with certainty, and by taking averages, we can obtain an experiment in which the outcome can be predicted with a high degree of accuracy. The law of large numbers in its simplest form for the Bernoulli random variables was introduced by Jacob Bernoulli toward the end of the 16th century. This result in generality was ﬁrst proved by the Russian mathematician A. Khintchine in 1929. This result is widely used in its applications to insurance, statistics, and the study of heredity.

Example 3.5.3 Let X1 , . . . , Xn be iid Bernoulli random variables with parameter p. Verify the law of large numbers.

Solution For Bernoulli random variables we know that EXi = p, and Var(Xi ) = p(1 − p). Thus, by Chebyshev’s theorem, 1 , Sn 0 / σ2 − p ≤ c ≥ 1 − 2 P p − c ≤ X ≤ p + c = P n nc p(1 − p) =1− → 1, as n → ∞. nc2 This verifies the weak law of large numbers.

168 CHAPTER 3 Additional Topics in Probability

Example 3.5.4 Consider n rolls of a balanced die. Let Xi be the outcome of the ith roll, and let Sn = for any ε > 0,

Sn 7 − ≥ ε → 0 P n 2 as n → ∞.

n

i=1Xi . Show that,

Solution Because the die is balanced, EXi = 7/2. By the law of large numbers, for any ε > 0,

Sn 7 − ≥ε →0 P n 2 as n → ∞, or equivalently,

Sn 7 P − < ε → 1 n 2

as n → ∞.

One of the most important results in probability theory is the Central Limit Theorem. This basically states that the z-transform of the sample mean is asymptotically standard normal. The amazing thing about the Central Limit Theorem is that no matter what the shape of the original distribution is, the (sampling) distribution of the mean approaches a normal probability distribution. We state one version of the Central Limit Theorem. In a restricted case, the proof uses the idea that the momentgenerating functions of Zn converge to the moment-generating function of the standard normal random variable. The general proof is a little bit more involved. Because the proof of the Central Limit Theorem is available in most probability books, we will not give the proof here. CENTRAL LIMIT THEOREM (CLT) Theorem 3.5.3 If X1 , . . . , Xn is a random sample from an inﬁnite population with mean μ, variance σ 2 , √ and the moment-generating function MX (t), then the limiting distribution of Zn = (X − μ)/(σ/ n) as n → ∞ is the standard normal probability distribution. That is, 1 lim P(Zn ≤ z) = √ n→∞ 2π

If Sn =

n

i=1 Xi ,

z

2 e−t /2 dt.

−∞

then we can rewrite Zn as

n X−μ X−μ Zn = √ , √ = nσ/ n σ/ n =

Sn − nμ √ , σ n

since nX =

n i=1

Xi .

3.5 Limit Theorems 169 √ Then the CLT states that Zn = (Sn − nμ) /σ n is approximately N(0, 1) for large n. The Central Limit Theorem basically says that when we repeat an experiment a large number of times, the average (almost always) follows a Gaussian distribution.

Example 3.5.5 X1 , X2 , . . . are iid random variables such that Xi =

1,

with probability p,

0,

with probability 1 − p.

√

Show that Zn = (Sn − np)/ npq is approximately normal for large n, where Sn =

n

i=1 Xi , and q = 1 − p.

Solution We know that E(X) = p; E(X2 ) = p; Var(X) = p − p2 = pq. √ Hence, by the CLT, the limiting distribution of Zn = (Sn − np)/ npq as n → ∞ is the standard normal probability distribution.

Example 3.5.6 A soft-drink vending machine is set so that the amount of drink dispensed is a random variable with a mean of 8 ounces and a standard deviation of 0.4 ounces. What is the approximate probability that the average of 36 randomly chosen ﬁlls exceed 8.1 ounces?

Solution

√ From the CLT, ((X − 8)/(0.4/ 36)) ∼ N(0, 1). Hence, from the normal table, ⎧ ⎫ ⎨ 0 / 8.1 − 8.0 ⎬ P X > 8.1 = P Z > 0.4 ⎩ ⎭ √ 36

= p {Z > 1.5} = 0.0668.

Example 3.5.7 Numbers in decimal form are often approximated by the closest integers. Suppose n numbers X1 , . . . , Xn are approximated by their closest integers J1 , J2 , . . . , Jn . Let Ui = Xi − Ji . Assume that Ui are uniform on (−0.5, 0.5) and that Ui s are independent. n Ui (a) Show that √i=1 ∼ N(0, 1) as n → ∞. n/12 n 1 , Ui 5 −5 i=1 ≤ √ ≤ √ . (b) Find P √ 300/12 300/12 300/12

170 CHAPTER 3 Additional Topics in Probability / 0 (c) Find the value of a such that P −a ≤ Ui ≤ a = 0.95 2 3 (d) For n = 106 , ﬁnd a such that P −a ≤ 106 i=1 Ui ≤ a = 0.99.

Solution (a) Because Ui s are uniform in (−0.5, 0, 5), Kn = ni=1 Ji . Then

Ui = 0, Var(Ui ) = 1/12. Let, Sn =

n

i=1 Xi , and

2 3 (Xi − Ji ) ≤ a P{|Sn − Kn | ≤ a} = P −a ≤ 2 3 = P −a ≤ Ui ≤ a . n Ui − 0 i=1 ∼ N(0, 1) as n → ∞. √ n/12 (b) For n = 300; a = 5. Using the normal table, n , 1 Ui −5 5 P √ ≤ √ i=1 ≤ √ = 0.68. 300/12 300/12 300/12 By the CLT,

(c) Now, 2 3 0.95 = P −a ≤ Ui ≤ a , =P From the normal table, we get √

1 −a a ≤Z≤ √ . √ 300/12 300/12

a = 1.96. This implies, a = 9.8. 300/12

(d) We have 0.99 = P

⎧ ⎨

6

−a ≤

⎩

10 i=1

−a

Ui ≤ a

⎫ ⎬ ⎭ a

4

≤Z≤ . 106 /12 106 /12 Now, using the normal table, we have a/ 106 /12 = 2.58. Hence, a = 745. =P

Example 3.5.8 A casino has a coin, suspected to be biased. Estimate p (probability of heads) such that they can be conﬁdent that their estimate (say, p) ˆ is within 0.01 of p (unknown). What is the minimum number of times we need to toss this coin?

3.5 Limit Theorems 171

Solution Set Xj =

Suppose we decided to use pˆ =

Xi n , that is,

1,

if H as j’th toss,

0,

if T as j’th toss.

#Heads . n

We want P{|X − p| < 0.01} = 0.99. √ Because Y = ni=1 Xi ∼ Bin(n, p), we have EY = np, Var(Y ) = npq. By the CLT, (X − p)/ pq/n ∼ N(0, 1). Now, X−p 0.01 −0.01 < √ < √ 0.99 = P √ pq/n pq/n pq/n 1 , 0.01 −0.01 2), where X100 = (1/100) 3.5.12.

100

i=1 Xi .

Let X1 , . . . , Xn be a sequence of independent Poisson-distributed random variables, with √ parameter λ. Let Sn = ni=1 Xi . Show that Zn = ((Sn − nλ)/ nλ) ∼ N(0, 1).

3.6 Chapter Summary 173

3.5.13.

Let X1 , . . . , Xn be a sequence of independent uniformly-distributed over [0,1) random √ variables. Let Sn = ni=1 Xi . Show that Zn = ((Sn − nλ)/ nλ) ∼ N(0, 1).

3.5.14.

Suppose that 2500 customers subscribe to a telephone exchange. There are 80 trunk lines available. Any one customer has the probability of 0.03 of needing a trunk line on a given call. Consider the situation as 2500 trials with probability of “success” p = 0.03. What is the approximate probability that the 2500 customers will “tie up” the 80 trunk lines at any given time?

3.5.15.

Suppose a group of people have an average IQ of 122 with standard deviation 2. Obtain a bound on the probability that IQ values of this group will be between 104 and 120.

3.5.16.

Let X be a random variable that represents the diastolic blood pressure (DBP) of the population of 18- to 74-year-old men in the United States who are not taking any corrective medication. Suppose that X has mean 80.7 mm Hg and standard deviation 9.2. (a) Obtain a bound on the probability that the DBP of this population will assumes values between 53.1 and 108.3 mm Hg. (b) In addition, assume that the distribution of X is approximately normal. Using the normal table, ﬁnd P(53.1 ≤ X ≤ 108.3). Compare this with the empirical rule.

3.5.17.

Color blindness appears in 2% of the people in a certain population. How large must a random sample be in order to be 99% certain that a color-blind person is included in the sample?

3.5.18.

A shirt manufacturer knows that, on the average, 2% of his product will not meet quality speciﬁcations. Find the greatest number of shirts constituting a lot that will have, with probability 0.95, fewer than ﬁve defectives.

3.5.19.

A random sample of size 100 is taken from a population with mean 1 and variance 0.04. Find the probability that the sample mean is between 0.99 and 1.

3.5.20.

The lifetime X (in hours) of a certain electrical component has the pdf f (x) = (1/3)e−(1/3)x , x > 0. If a random sample of 36 is taken from these components, ﬁnd P(X < 2).

3.5.21.

A drug manufacturer receives a shipment of 10,000 calibrated “eyedroppers” for administering the Sabin poliovirus vaccine. If the calibration mark is missing on 500 droppers, which are scattered randomly throughout the shipment, what is the probability that, at most, two defective droppers will be detected in a random sample of 125?

3.6 CHAPTER SUMMARY In this chapter we looked at some special distribution functions that arise in practice. It should be noted that we discussed only a few of the important probability distributions. There many other discrete and continuous distributions that will be useful and appropriate in particular applications. Some of them are given in Appendix A3. A larger list of probability distributions can be found at http://www.causascientia.org/math_stat/Dists/Compendium.pdf, among many other

174 CHAPTER 3 Additional Topics in Probability

places. For more than one random variable, we learned the joint distributions. We also saw how to ﬁnd the density and cumulative distribution for the functions of a random variable. Limit theorems are a crucial part of probability theory. We have introduced the Chebyshev’s inequality, the law of large numbers, and the Central Limit Theorem for the random variables. We now list some of the key deﬁnitions introduced in this chapter: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Bernoulli probability distribution Binomial experiment Poisson probability distribution Probability distribution Normal (or Gaussian) probability distribution Standard normal random variable Gamma probability distribution Exponential probability distribution Chi-square (χ2 ) distribution Joint probability density function Bivariate probability distributions Marginal pdf Conditional probability distribution Independence of two r.v.s Expected value of a function of bivariate r.v.s Conditional expectation Covariance Correlation coefﬁcient

In this chapter, we have also learned the following important concepts and procedures: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Mean, variance, and moment-generating function (mgf ) of a binomial random variable Mean, variance, and mgf of a Poisson random variable Poisson approximation to the binomial probability distribution Mean, variance, and mgf of a uniform random variable Mean, variance, and mgf of a normal random variable Mean, variance, and mgf of a gamma random variable Mean, variance, and mgf of an exponential random variable Mean, variance, and mgf of a chi-square random variable Properties of expected value Properties of the covariance and correlation coefﬁcient Procedure to ﬁnd the cdf of a function of r.v. using the method of distribution functions The pdf of Y = g(X), where g is differentiable and monotone increasing or decreasing The pdf of Y = g(X), using the probability integral transformation The transformation method to ﬁnd the pdf of Y = g(X1 , . . . , Xn ) Chebyshev’s theorem Law of large numbers Central Limit Theorem (CLT)

3.7 Computer Examples (Optional) 175

3.7 COMPUTER EXAMPLES (OPTIONAL) 3.7.1 Minitab Examples Minitab contains subroutines that can do pdf and cdf computations. For example, for binomial random variables, the pdf and cdf can be respectively computed using the following comments.

MTB > pdf k; SUBC > binomial n p.

and

MTB > cdf; SUBC > binomial n p.

Practice: Try the following and see what you get.

MTB > pdf 3; SUBC > binomial 5 0.40.

will give K 3.00 and

MTB > cdf; SUBC > binomial 5 0.40.

will give BINOMIAL WITH N = 5 P = 0.400000 K P(X LESS OR = K) 0 0.0778 1 0.3370 2 0.6826 3 0.9130 4 0.9898 5 1.0000

P(X = K) 0.2304

176 CHAPTER 3 Additional Topics in Probability

Similarly, if we want to calculate the cdf for a normal probability distribution with mean k and standard deviation s, use the following comments.

MTB > cdf x; SUBC > normal k s.

will give P(X ≤ x). Practice: Try the following.

MTB > cdf 4.20; SUBC > normal 4 2.

We can use the invcdf command to ﬁnd the inverse cdf. For a given probability p, P(X ≤ x) = F (x) = p, we can ﬁnd x for a given distribution. For example, for a normal probability distribution with mean k and standard deviation s, use the following.

MTB > invcdf p; SUBC > normal k s.

We can also use the pull-down menus to compute the probabilities. The following example illustrates this for a binomial probability distribution.

Example 3.7.1 A manufacturer of a color printer claims that only 5% of their printers require repairs within the ﬁrst year. If out of a random sample of 18 of their printers, four required repairs within the ﬁrst year, does this tend to refute or support the manufacturer’s claim? Use Minitab.

Solution Type the numbers 1 through 18 in C1. Then

Calc > Probability Distributions > Binomial. . . > choose Cumulative probability > in Number of trials, enter 18 and in Probability of success, enter 0.05 > in Input column: type C1 > Click OK We will get the following output.

3.7 Computer Examples (Optional) 177

Cumulative Distribution Function Binomial with n=18 and p=0.0500000 x P(X Compute > type in the Target Variable: y > Use the scroll bar beside the Functions box to find CDF.BINOM(q, n, p) > Highlight it and use the up button to load it into the Numeric Expression: box. Set q to 3 (success, the x-value), n to 18 (total trials) and p to 0.05 (probability of success) > OK In the second column, we will get the y-values as 0.99. Hence, P(X ≤ 3) = 0.99.

We can use this procedure for many other distributions.

178 CHAPTER 3 Additional Topics in Probability

3.7.3 SAS Examples Sometimes, we can use computer calculations to ﬁnd out the exact probability of a certain event in lieu of approximations. For example, when n is large in a binomial experiment, we can use normal approximation to calculate the probabilities. The following example shows how to calculate binomial probabilities using SAS codes.

Example 3.7.3 Suppose that a certain drug to treat a disease has a success rate of p = 0.65. This drug is given to n = 500 patients with the disease. (a) What is the probability that 335 or fewer show improvement? (b) What is the probability that more than 320 show improvement? (c) What is the probability that exactly 300 show improvement? (d) What is the probability that the number of improvements lies in the interval (300,350)?

Solution Let X = number of patients showing improvement. Then X is a binomial random variable with parameters n = 500 and p = 0.65. (a) First three lines in the following code are comment lines. In general, it is always helpful to include the comment lines to explain about the program. /*This program can be used to compute probability*/ /* that a Binomial variable with parameters p*/ /*and n is less than or equal to x*/ data binomial; p=0.65; n=500; x=335; y=probbnml(p,n,x); cards; proc print; run; The following is the SAS output from running the foregoing program. Obs 1

p 0.65

n 500

x 335

y 0.83753

Here y = 0.83753 is the P (X ≤ 335). (b) To calculate P(X > 320), we can use the following. data binomial; p=0.65; n=500;

3.7 Computer Examples (Optional) 179

x=320; y=probbnml(p,n,x); z=1–y; cards; proc print; run; The following is the SAS output from running the foregoing program, where the value of z is the probability we are looking for. Obs 1

p 0.65

n 500

x 320

y 0.33516

z 0.66484

Hence, P(X > 320) = 0.66484. (c) To find P(X = 300), we can use the following. data binomial; p=0.65; n= 500; x1=300; y1=probbnml(p,n,x1); x2=299; y2=probbnml(p,n,x2); z=y1−y2; cards; proc print; run; The following is the SAS output from running the foregoing program, where the value of z is the probability we are looking for. Obs p n x1 y1 1 0.65 500 300 0.011327

x2 y2 z 299 .008864418 .002462253

(d) To find P(300 < X < 350), use the following. data binomial; p=0.65; n=500; x1=300; y1=probbnml(p,n,x1); x2=349; y2=probbnml(p,n,x2);

180 CHAPTER 3 Additional Topics in Probability

z=y2−y1; cards; proc print; run; We will get the following output. Obs p n x1 1 0.65 500 300

y1 x2 y2 z 0.011327 349 0.98982 0.97849

Hence, P(300 < X < 350) = 0.97849.

Similar procedures could be used to calculate probabilities for other distributions. In order to test for normality of a given data set using a normal probability plot, we can use PROC UNIVARIATE (see Chapter 1 for explanation) in the following manner. Normal plot is called qqplot in SAS.

proc univariate data=K noprint; /*Specify the name of data set as K*/ qqplot standard; run; quit;

Note that this avoids printing of all the standard output due to the univariate command, and we get only the QQ plot. If we need a straight line in the plot, we can modify the commands as follows.

proc univariate data=K noprint; /*Specify the name of data set as B*/ qqplot standard/ normal (mu=m, sigma=s); run; quit;

PROJECTS FOR CHAPTER 3 3A. Mixture Distribution In statistical modeling, if the data are contaminated by outliers or if the samples are drawn from a population formed by a mixture of two populations, one could use mixture distributions. Mixture distributions are used frequently in medical applications, such as micro array analysis. Suppose a random variable X has pdf f1 (x) with probability p1 and pdf f2 (x) with probability p2 , where p1 + p2 = 1. Then we say that the r.v. X has a mixture distribution. This can be thought of as observing

Projects for Chapter 3 181

a Bernoulli random variable Z that is equal to 1 with probability p1 and 2 with probability p2 . Thus, X=

X1 ∼ f1 (x), X2 ∼ f2 (x),

if Y = 1, if Y = 2.

(a) Show that the pdf of X is given by f (x) = p1 f1 (x) + p2 f2 (x). (b) If (μ1 , σ12 ) and (μ2 , σ22 ) are means and variances of f1 (x) and f2 (x), respectively, show that μ = E(X) = p1 μ1 + p2 μ2 ,

and σ 2 = Var(X) = p1 σ12 + p2 σ12 + p1 μ21 + p2 μ22 − (p1 μ1 + p2 μ2 )2 .

3B. Generating Samples from Exponential and Poisson Probability Distribution (a) Generate a sample from 1θ e−x/θ (θ is chosen). Let Y1 , Y2 , . . . , Yn be a sample from a U(0, 1) distribution. Let F (x) = 1−e−x/θ (cdf of exponential). Then Y = F (x) is uniform. yj = 1−e−x/θ implies xj = − θ ln(1 − yi ) = − θ ln ui , where u1 , u2 , . . . ., un is a sample from U(0, 1). Then X1 , . . . , Xn is a sample from an exponential distribution with parameter θ. (b) Suppose we want to generate a sample from a Poisson probability distribution with parameter λ. X1 , . . . , Xn is a sample from an exponential distribution with parameter 1/λ till ni=1 Xi just exceeds 1. Then yn (n − 1) is a sample values form a Poisson probability distribution with parameter λ.

EXERCISE 3B Let u1 , u2 , . . . , un be a sample from U(0, 1). Show that n 2 , (i) X = −2 ln(ui ) ∼ χ2n (ii) X = −β

i=1 α

ln(ui ) ∼ gamma(α, β), and

i=1 α

(iii) X =

i=1 α+β

ln(ui )

∼ Beta(α, β).

ln(ui )

i=1

3C. Coupon Collector’s Problem Suppose there are n distinct colors of coupons. Each color of coupon is equally likely to occur. When a complete set of coupons with each color represented is assembled, you win a prize. Let X = # coupons for a complete set. Find (a) Distribution of X, (b) E(X), and (c) Var(X).

182 CHAPTER 3 Additional Topics in Probability

3D. Recursive Calculation of Binomial and Poisson Probabilities A simple way to calculate binomial probabilities is as follows: For a given n and p, evaluate b(0, n, p) and then apply the recursive relationship b(x + 1, n, p) = b(x, n, p)

p(n − x) (1 − p)(x + 1)

to obtain other binomial probabilities. (a) Derive this recursion formula. (b) For n = 15, p = 0.4, using the recursive formula, compute all other probabilities starting from x = 0. The following recursive formulas are very useful in calculating successive Poisson probabilities: f (x − 1, λ) = f (x, λ)

x λ

and f (x + 1, λ) =

λ e−λ λx+1 = f (x, λ) . (x + 1)! x+1

For example, if λ = 2.5, we know that f (0, 2.5) = e−2.5 = 0.08208. Using this, calculate (c) f (1, 2.5) and f (2, 2.5).

Chapter

4

Sampling Distributions Objective: In this chapter we study the probability distributions of various sample statistics such as the sample mean and the sample variance and illustrate their usefulness. 4.1 Introduction 184 4.2 Sampling Distributions Associated with Normal Populations 4.3 Order Statistics 207 4.4 Large Sample Approximations 212 4.5 Chapter Summary 218 4.6 Computer Examples 219 Projects for Chapter 4 221

191

Abraham de Moivre (Source: http://en.wikipedia.org/wiki/File:Abraham_de_Moivre.jpg)

Abraham de Moivre (1667–1754) was a French mathematician known for his work on the normal distribution and probability theory. He is famous for de Moivre’s formula, which links complex

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

183

184 CHAPTER 4 Sampling Distributions

numbers and trigonometry. He ﬂed France and went to England to escape the persecution of Protestants. In England he wrote a book on probability theory, titled The Doctrine of Chances. This book was very popular among gamblers. The normal distribution was ﬁrst introduced by de Moivre in an article in 1733 in the context of approximating certain binomial distributions for large n, and is now called the theorem of de Moivre–Laplace.

4.1 INTRODUCTION Sampling distributions play a very important role in statistical analysis and decision making. We begin with studying the distribution of a statistic computed from a random sample. Based on the probabilistic foundation of Chapters 2 and 3, the present study marks the beginning of our learning of statistics beyond the descriptive phase. Because a sample is a set of random variables X1 , . . . , Xn , it follows that a sample statistic that is a function of the sample is also random. We call the probability distribution of a sample statistic its sampling distribution. Sampling distributions provide the link between probability theory and statistical inference. The ability to determine the distribution of a statistic is a critical part in the construction and evaluation of statistical procedures. It is important to observe that there is a difference between the distribution of population from which the sample was taken and the distribution of the sample statistic. In general, a population has a distribution called a population distribution, which is usually unknown, whereas a statistic has a sampling distribution, which is usually different from the population distribution. The sampling distribution of a statistic provides a theoretical model of the relative frequency histogram for the likely values of the statistic that one would observe through repeated sampling. Even though some of the terms in this section have already been deﬁned in Chapter 1, we now present these deﬁnitions in terms of random variables. These abstractions are introduced to develop scientiﬁcally based methods of analyzing the data, and one should always keep in mind the underlying population. Deﬁnition 4.1.1 A sample is a set of observable random variables X1 , . . . , Xn . The number n is called the sample size. In most of the inferential procedures that we study in this book, we are dealing with random samples. We call the random variables X1 , . . . , Xn identically distributed if every Xi has the same probability distribution. Deﬁnition 4.1.2 A random sample of size n from a population is a set of n independent and identically distributed (iid) observable random variables X1 , . . . , Xn . Note that in a sample (not a random sample), Xi s need not be independent or identically distributed. For the results of this book to be applicable, it is important to ensure that the selection of a sample is at least approximately random. The signiﬁcance of random sampling is that the probability distribution of a statistic can be easily derived. Random sampling helps us to control systematic basis. For a ﬁnite population, one can serially number the elements of the population and then select a random sample with the help of a table of random digits. One of the simplest ways to select a random sample of ﬁnite size is to use a table of random numbers. When the population size is very large, such a method can become very taxing and sometimes practically impossible. However, there are excellent computer

4.1 Introduction 185

programs for generating random samples from large populations, and these programs can be used. Now we deﬁne a statistic. Deﬁnition 4.1.3 A function T of observable random variables X1 , . . . , Xn that does not depend on any unknown parameters is called a statistic. The sample mean X = (1/n) ni=1 Xi is a function of X1 , . . . , Xn . The sample median and sample variance S 2 are also examples of statistics. It is important to observe that even with random sampling, there is sampling variability or error. That is, if we select different samples from the same population, a statistic will take different values in different samples. Thus, a sample statistic is a random variable, and hence it has a probability distribution. In order for us to study the behavior of the phenomenon a sample statistic represents, we must identify its probability distribution. Deﬁnition 4.1.4 The probability distribution of a sample statistic is called the sampling distribution. We can illustrate these deﬁnitions with the following example with a ﬁnite population and a ﬁnite sample size. In this case, we take all possible samples of size n from a population of size N.

Example 4.1.1 Let the population consist of the numbers {1, 2, 3, 4, 5}. Consider all possible samples consisting of three numbers randomly chosen without replacement from this population. Obtain the distribution of the sample mean.

Solution Disregarding the order, it is clear that there are

5 = 10 equally likely possible samples of size 3. They are 3

(1,2,3), (1,2,4), (1,2,5), (1,3,4), (1,3,5), (1,4,5), (2,3,4), (2,3,5), (2,4,5), and (3,4,5). Calculating the mean, X, for each of the samples, we will get the sampling distribution of X as x

2 1

7 3

8 3

3 1

10 3

11 3

4 1

1 1 2 2 2 1 1 10 10 10 10 10 10 10 For example, in the table, P X = 8/3 = 2/10 because the two samples (1,2,5) and (1,3,4) both give an x = 8/3, which is an estimate of the population mean, μ. p (x)

In general, sampling distributions are theoretical distributions that consist of possibly an infinite number of sample statistics taken from an infinite number of randomly selected samples of a fixed sample size. For example, if a sample of size n = 30 were taken from a large population an infinite number of times, the combined means taken from all the samples would make up the sampling distribution of the mean. Every sample statistic has a sampling distribution. The next result states that if one selects a random sample from a population with mean μ and variance σ 2 , then regardless of the form of the population distribution, one can obtain the mean and standard deviation of the statistic X in terms of the mean and standard deviation of the population. This is explained in the following result.

186 CHAPTER 4 Sampling Distributions

Theorem 4.1.1 Let X1 , . . . , Xn be a random sample of size n from a population with mean μ and variance σ 2 . Then E(X) = μ and Var(X) = σ 2 /n. Proof. The mean and variance of X is given by, E X =E

1 Xi n n

i=1

=

and

1 n

n i=1

μ=

1 E(Xi ) n n

=

i=1

1 nμ = μ. n

n 1 Var X = Var Xi n i=1

n 1 Var(Xi ) (because Xi s are independent and = 2 n i=1

Var(aXi ) = a2 Var (Xi )) σ2 1 . = 2 nσ 2 = n n

2 . Note that from the previous theorem, μ = μ and We denote E X = μX and Var X = σX X √ σX = σ/ n. Here, σX is called the standard error of the mean. It is important to notice that the variance of each of the random variables X1 , X2 , . . . , Xn is σ 2 , whereas the variance of the sample mean X is σ 2 /n, which is smaller than the population variance σ 2 for n ≥ 2. The implication of Theorem 4.1.1 is that the sample means become more and more reliable as an estimate of μ as the sample size is increased, as we would expect. From Chebyshev’s inequality, 1 P X − μX < kσ X ≥ 1 − 2 . k

√ √ Let ε = (kσ/ n). Then k = (ε n)/σ. Since μX = μ, the above inequality can be written as σ2 P X − μ < ε ≥ 1 − 2 . nε

Thus, for any ε > 0, the probability that the difference between X and μ less than ε can be made arbitrarily close to 1 by choosing the sample size n is sufﬁciently large. We illustrate this result in the following example.

Example 4.1.2 A particular brand of drink has an average of 12 ounces per can. As a result of randomness, there will be small variations in how much liquid each bottle really contains. It has been observed that the amount of liquid in these bottles is normally distributed with σ = 0.8 ounce. A sample of 10 bottles of this brand of

4.1 Introduction 187

soda is randomly selected from a large lot of bottles, and the amount of liquid, in ounces, is measured in each. Find the probability that the sample mean will be within 0.5 ounce of 12 ounces.

Solution Let X1 , X2 , . . . , X10 denote the ounces of liquid measured for each of the bottles. We know that Xi s are normally distributed with mean μ = 12 and variance σ 2 = 0.64. From Theorem 4.1.1, X possesses a normal distribution (actually, for the normality part, we use Corollary 4.2.2) with a mean 12 and variance σ 2 /n = 0.64/10 = 0.064. We find P X −12| ≤ 0.5) = P −0.5 ≤ X − 12 ≤ 0.5 X − 12 0.5 0.5 =P − √ ≤ √ ≤ √ σ/ n σ/ n σ/ n

0.5 0.5 ≤Z ≤ =P − 0.253 0.253 = P(−1.97 ≤Z ≤ 1.97) = 0.9512. using standard normal table . Hence, the chance is about 0.95% that the mean amount of drink in any 10 bottles randomly chosen will be between 11.5 to 12.5 ounces.

4.1.1 Finite Population

Let {c1 , c2 , . . . , cN } be a ﬁnite population. Then the population mean μ = (1/N) N i=1 ci and the 2 population variance σ 2 = (1/N) N i=1 (ci − μ) . The following theorem for the sample mean and variance is stated without proof. Theorem 4.1.2 If X1 , . . . , Xn is a sample of size n (chosen without replacement) from a population {c1 , c2 , . . . , cN }, then E X =μ σ2 Var X = n

N −n . N −1

We remark here that the sample in the theorem is not a random sample and Xi s are not iid random variables. The factor (N − n)/(N − 1) in the foregoing theorem is often called the ﬁnite population correction factor. It is close to 1 unless the sample amounts to a signiﬁcant portion of the population. Note that the sampling without replacement causes dependence among the Xi s. However, if the sample size n is small relative to the population size N, the population correction factor is approximately 1. Hence, we will not use the ﬁnite population correlation factor in the derivation of sampling distribution, unless it is absolutely necessary.

188 CHAPTER 4 Sampling Distributions

Example 4.1.3 Obtain the mean and variance of X in Example 4.1.1.

Solution

First note that for the population in Example 4.1.1, the population mean is μ = (1/N) N i=1 ci = 3 and the 2 population variance is σ 2 = (1/N) N i=1 (ci − μ) = 2. Applying the probability distribution of X given in Example 4.3.1, we obtain

7 1 8 2 2 10 2 1 E X =2 + + +3 + 10 3 10 3 10 10 3 10

1 11 1 +4 + 3 10 10 = 3, and 2 2 Var X = E X − EX = 22

+ 32 =

2 10

+

1 10

+

2 2 1 2 7 8 + 3 10 3 10

2

10 2 2 11 1 1 + + 42 − 32 3 10 3 10 10

2 1 × = 0.3333. 3 2

This is the same as (σ 2 /n). [(N − n)/(N − 1)]. In this case we observe that the variance of X is precisely one sixth of the original variance.

Example 4.1.4 Let X1 , . . . , Xn be a random sample from a population with mean μ and variance σ 2 . Consider the sample variance n 2 1 Xi − X . n−1

S2 =

i=1

Show that E(S 2 ) = σ 2 .

Solution It can be shown that (see Exercise 1.5.8) n

1 n−1

n

Xi − X

i=1

2

=

i=1

Xi2 − nX n−1

2

.

4.1 Introduction 189

Hence,

⎛ n ⎞ 2 2 X − nX n ⎜ ⎟ i n 1 2 2 ⎜ i=1 ⎟ E Xi − E X . E S2 = E ⎜ ⎟= ⎝ ⎠ n−1 n−1 n−1 i=1

Using the fact that E X2 = Var (X) + μ2 and Theorem 4.1.1, we have

E S2

σ2 1 n 2 2 2 n σ +μ − +μ = n−1 n−1 n

1 n n n 2 − σ + − μ2 = n−1 n−1 n−1 n−1 = σ2.

This shows that the expected value of the sample variance is the same as the variance of the population under consideration.

EXERCISES 4.1 4.1.1.

Let the population be given by the numbers {−2, −1, 0, 1, 2}. Take all random samples of size 3. (a) Without replacement, obtain the following in each case. (i) The sampling distribution of the sample mean. (ii) The sampling distribution of the sample median. (iii) The sampling distribution of the sample standard deviation. (iv) The mean and variance of the sample mean. (b) How many samples of size 3 can we get, if we sample with replacement?

4.1.2.

(a) How many different samples of size n = 2 can be chosen from a ﬁnite population of size 12 if the sampling is without replacement? (b) What is the probability of each sample in part (a), if each sample of size 2 is equally likely? (c) Find the value of the ﬁnite population correction factor.

4.1.3.

Let the population be given by {1, 2, 3}. Let p(x) = 1/3 for x = 1, 2, 3. Take samples of size 3 with replacement. (a) Calculate μ and σ 2 . (b) Obtain the sampling distribution of the sample mean. (c) Obtain the mean and variance of the sample mean.

4.1.4.

Find the value of the ﬁnite population correlation factor for (a) n = 8 and N = 60. (b) n = 8 and N = 1000. (c) n = 15 and N = 60.

190 CHAPTER 4 Sampling Distributions n

2 2 Xi − X . Find E[ S ]. Compare

4.1.5.

For a random sample X1 , . . . , Xn , let (S )2 = (1/n) this with E S 2 .

4.1.6.

For a random sample X1 , . . . , Xn with mean μ and variance σ 2 , let Tn = total. Show that E (Tn ) = nμ and Var (Tn ) = nσ 2 .

i=1

n

Xi , the sample

i=1

4.1.7.

A particular brand of sugar is sold in 5-lb packages. The weight of sugar in these packages can be assumed to be normally distributed with mean μ = 5 lb and standard deviation σ = 2 lb. What is the probability that the mean weight of sugar in 15 randomly selected packages will be within 0.2 lb of 5 lb?

4.1.8.

A random sample of size 150 is taken from an inﬁnite population having the mean μ = 15 and standard deviation σ = 2.5. What is the probability that X will be between 10.5 and 18.5?

4.1.9.

The distribution of heights of all students in a large university has a normal distribution with a mean of 66 inches and a standard deviation of 2 inches. What is the probability that the mean height of 26 randomly selected students from this university will be more than 70 inches?

4.1.10.

An image-encoding algorithm, when used to encode images of a certain size, uses a mean of 110 milliseconds with a standard deviation of 15 milliseconds. What is the probability that the mean time (in milliseconds) for encoding 50 randomly selected images of this size will be between 90 milliseconds and 135 milliseconds? What assumptions do we need to make?

4.1.11.

In order to evaluate a new release of a database management system, a database administrator runs a benchmark program several times and measures the time to completion in seconds. Assuming that the distribution of times is normal with mean 95 seconds and with standard deviation of 10 seconds, what proportion of measurement times will fall below 85 seconds?

4.1.12.

A population of disk drives manufactured by a certain company runs with mean seek time of 10 milliseconds with standard deviation of 0.1 milliseconds. What proportion of samples of size 250 would you expect to result in a mean less than 9 milliseconds? What assumptions do we need to make?

4.1.13.

Suppose that the national norm of a science test for 12th graders on a particular year has a mean of 215 and a standard deviation of 35. (a) A random sample of 55 12th graders is selected. What is the probability that this group will average more than 230? (b) A random sample of 200 12th graders is selected. What is the probability that this group will average over 230? (c) A random sample of 35 12th graders is selected. What is the probability that this group will average over 230? (d) How does the sample size inﬂuence the probability?

4.2 Sampling Distributions Associated with Normal Populations 191

4.1.14.

Scores on the Wechsler Adult Intelligence Scale for the 20 to 34 age group are approximately normally distributed with mean equal to 110 and standard deviation equal to 25. If we select 100 people at random, what is the probability that this group will have an average score of 125 or above?

4.1.15.

It is known that a healthy human body has an average temperature of 98.6◦ F, with a standard deviation of 0.95◦ F. Sixty healthy humans are selected at random. What is the probability that their temperatures average at least 99.1◦ F?

4.2 SAMPLING DISTRIBUTIONS ASSOCIATED WITH NORMAL POPULATIONS The sampling distribution of a statistic will depend upon the population distribution from which the samples are taken. In this section we discuss the sampling distributions of some statistics that are based on a random sample drawn from a normal distribution. These statistics are used in many statistical procedures that are very important in solving real-world problems. The following result establishes the distribution of a linear combination of independent normal random variables. Theorem 4.2.1 Let X1 , . . . , Xn be independent random variables with the distribution of Xi being normal with mean μi and variance σi2 . Let a1 , a2 , . . . , an be real constants. Then the distribution of Y = ni=1 ai Xi is normal with mean μY = ni=1 ai μi and variance σY2 = ni=1 ai2 σi2 . Proof. The moment-generating function of Y is given by n

MY (t) = Ee( i=1 ai Xi )t 7 = Ee(ai Xi )t [by independence ofXi s] i

=

7

Ee(ai t)Xi

i

= =

7 i

7

= e[(

MXi (ai t)

[using the deﬁnition of mgf)

2 2 2 e(ai μi t+(1/2)ai σi t )

i

i ai μi )t+(1/2)

[using mgf of a normal] 2

2 i ai σi

t2 ]

which is the mgf of a normal random variable with mean

i a i μi

and variance

2 2 i ai σi .

In Theorem 4.2.1 let ai = 1/n, μi = μ, and σ12 = σ 2 , we obtain the following result, which provides the distribution of the sample mean. Corollary 4.2.2 Let X1 , . . . , Xn be a random sample of size n from a normal population with mean μ and variance σ 2 . Then X = (1/n)

n i=1

Xi

2 = σ 2 /n. is normally distributed with mean μX = μ and variance σX

192 CHAPTER 4 Sampling Distributions

Recall that we have used the notation X ∼ N(μ, σ 2 ) to mean that the random variable X is normally distributed with mean μ and variance σ 2 . From Corollary 4.2.2, X ∼ N(μ, σ 2 /n) and hence by the √ z-transformation we obtain the standard normal random variable, Z = X − μ / σ/ n ∼ N(0, 1).

Example 4.2.1 A company that manufactures cars claims that the gas mileage for its new line of hybrid cars, on the average, is 60 miles per gallon with a standard deviation of 4 miles per gallon. A random sample of 16 cars yielded a mean of 57 miles per gallon. If the company’s claim is correct, what is the probability that the sample mean is less than or equal to 57 miles per gallon? Comment on the company’s claim about the mean gas mileage per gallon of its cars. What assumptions did you make?

Solution Let X represent the gas mileage for the new car (in miles per gallon). If the company’s claim is true, then from Corollary 4.2.2, X is normally distributed with mean μ = 60 and variance σ 2 /n = 16/16 = 1. Hence, 57 − 60 X − 60 P X ≤ 57 = P ≤ 1 1 = P(Z ≤ −3) ≈ 1 − 0.999 = 0.001. Therefore, if the company’s claim is correct, it is very unlikely that the mean value of the random sample of 16 cars will be 57 miles per gallon. Because the mean is indeed 57 miles per gallon, we conclude that the company’s claim is very likely not true. Here we have assumed that the sample of 16 measurements comes from a normal population, so that we could apply the results of Corollary 4.2.2.

Now we introduce some distributions that can be derived from a normal distribution. These distributions play a very important role in inferential problems.

4.2.1 Chi-Square Distribution A chi-square distribution is used in many inferential problems, for example, in inferential problems dealing with the variance. Recall that the chi-square distribution is a special case of a gamma distribution with α = n/2 and β = 2. If n is a positive integer, then the parameter n is called the degrees of freedom. However, if n is not an integer, but β = 2, we still refer to this distribution as a chi-square. The mgf of a χ2 − random variable is M(t) = (1 − 2t)−n/2 . The mean and variance of a chi-square distribution are μ = n and σ 2 = 2n, respectively. That is, the mean of a χ2 (n) random variable is equal to its degree of freedom and the variance is twice the degree of freedom. We now give some useful results for χ2 − random variables. Theorem 4.2.3 Let X1 , . . . , Xk be independent χ2 − random variables with n1 , . . . , nk degrees of freedom, k respectively. Then the sum V = i=1 Xi is chi-square distributed with n1 + n2 + · · · + nk degrees of freedom.

4.2 Sampling Distributions Associated with Normal Populations 193

Proof. The mgf of V is

MV (t) =

k 7

(1 − 2t)−ni /2 = (1 − 2t)

−

k

ni /2

i=1

.

i=1

This implies that V ∼ χ2

k i=1 ni

.

Our next result states that the difference of two chi-square random variables is a chi-square random variable, given by the following theorem. The proof is left as an exercise. Theorem 4.2.4 Let X1 and X2 be independent random variables. Suppose that X1 is χ2 with n1 degrees of freedom, whereas Y = X1 + X2 is chi-square with n degrees of freedom, where n > n1 . Then X2 = Y − X1 is a chi-square random variable with n − n1 degrees of freedom. The following result shows that we can generate a chi-square random variable from a gamma random variable. Theorem 4.2.5 If a random variable X has a gamma distribution with parameters α and β, then 2X ∼χ2 (2α). β

Y=

Proof. Recall that the mgf of the gamma random variable X is (1 − βt)−α .

2X t MY (t) = M 2X (t) = E e β β

X( β2 t)

=E e

= MX

2 t β

= (1 − 2t)−α = (1 − 2t)−

2α 2 .

Hence, Y ∼ χ2 (2α). The following result states that by squaring a standard normal random variable, we can generate a chi-square random variable, with one degree of freedom. Theorem 4.2.6 If X is a standard normal random variable, then X2 is chi-square random variable with 1 d.f. Proof. Because X ∼ N(0, 1) the moment-generating function of X2 is ∞ MX2 (t) = −∞

2 1 2 etx √ e−x /2 dx = (1 − 2t)−1/2 . 2π

This implies that X2 ∼ χ2 (1). Figure 4.1 gives the probability densities of the random variables X and X2 .

194 CHAPTER 4 Sampling Distributions

Densities of Standard normal r.v. and its square 4 3.5 3 2.5 pdf of X 2

2 1.5

pdf of X

1 0.5 0 23

22

21

0

1

2

3

■ FIGURE 4.1 pdf of standard normal r.v. and the pdf of its square.

The following result is a direct consequence of Theorems 4.2.3 and 4.2.6. This result illustrates how to obtain a random sample from chi-square distribution if we have a random sample of n measurements from a normal population. Theorem 4.2.7 Let the random sample X1 , . . . , Xn be from a N(μ, σ 2 ) distributed. Then Zi = (Xi − μ)/ σ, i = 1, . . . , n are independent standard normal random variables and n i=1

Zi2 =

n Xi − μ 2 i=1

σ

has a χ2 -distribution with n degrees of freedom. In particular, if X1 , . . . , Xn are independent standard normal random variables, then Y 2 = ni=1 Xi2 is chi-square distributed with n degrees of freedom. If X ∼ χ2 (n), then from the chi-square table, we can compute the values of χα2 (n) such that P X > χα2 (n) = α,

as shown by Figure 4.2. 2 (15) look in the chi-square table with the row labeled 15 d.f. For example, if X ∼ χ2 (15), to ﬁnd χ0.95 2 and the column headed χ0.950 and obtain the value as 7.26094. Thus, with 15 degrees of freedom, P (X > 7.26094) = 0.95. Also, if X is a chi-square random variable with 11 degrees of freedom, from 2 (11) = 19.675. Therefore, P (X > 19.675) = 0.05. the chi-square table we have χ0.05

4.2 Sampling Distributions Associated with Normal Populations 195

X 2(n ) ■ FIGURE 4.2 Chi-square probability density.

Example 4.2.2 Let the random variables X1 , X2 , . . . , X5 be from an N (5, 1) distribution. Find a number a such that ⎞ ⎛ 5 2 (Xi − 5) ≤ a⎠ = 0.90. P⎝ i=1

Solution By Theorem 4.2.7,

5 i=1

Zi2 =

5 Xi −5 2 i=1

1

=

5

(Xi − 5)2 has a chi-square distribution with 5 degrees of

i=1

freedom. Because the upper tail area is 0.10, looking at the chi-square table with 5 d.f. and the column 2 , we obtain a = 9.23635. Thus, corresponding to χ0.10 ⎞ ⎛ 5 2 (Xi − 5) ≤ 9.23635⎠ = 0.90. P⎝ i=1

Example 4.2.3 Suppose that X is χ2 − random variable with 20 degrees of freedom. Use the chi-square table to obtain the following: (a) Find x0 such that P (X > x0 ) = 0.95. (b) Find P (X ≤ 12.443).

Solution (a) For 20 degrees of freedom, using the chi-square table, we have P (X > 10.851) = 0.95. Hence, x0 = 10.851.

196 CHAPTER 4 Sampling Distributions

(b) From the chi-square table, P (X ≤ 12.443) = 0.10. The following result gives the probability distribution for a function of the sample variance S 2 .

Theorem 4.2.8 If X1 , . . . , Xn is a random sample from a normal population with the mean μ and variance σ 2 , then (a) the random variable n

Xi − X

(b)

i=1

σ2

2 =

(n − 1) S 2 . σ2

has a chi-square distribution with (n − 1) degrees of freedom. (c) X and S 2 are independent. Proof. We will only prove part (a). For part (b), we will give some comments on the proof. n 2 (a) We know from Theorem 4.2.7 that 1/σ 2 i=1 (Xi − μ) has a chi-square distribution with n degrees of freedom. Thus, n n 2 1 2= 1 (X Xi − X + X − μ − μ) i σ2 σ2 i=1 i=1 % n & n 2 2 1 Xi − X + = 2 X−μ σ i=1 i=1 n Since 2 Xi − X X − μ = 0 i=1

(n − 1) S 2 + = σ2

X−μ √ σ/ n

2 .

The left-hand side of this equation has a chi-square distribution with n degrees of freedom. √ # √ $2 ∼ χ2 (1). Also, since X − μ / σ/ n ∼N (0, 1) by Theorem 4.2.6 we have X − μ / σ/ n Now from Theorem 4.2.4, (n − 1) S 2 /σ 2 ∼ χ2 (n − 1). (b) We will accept the result of part (b) without proof here. A rigorous proof depends on geometric properties of the multivariate normal distribution, which is beyond the scope of this book. A proof based on moment-generating functions is relatively straightforward, where essentially we can ﬁrst show that the random variable X and the vector of ran dom variables X1 − X, . . . , Xn − X are independent. Because S 2 is a function of the vector X1 − X, . . . , Xn − X , it is then independent of X.

4.2 Sampling Distributions Associated with Normal Populations 197

Example 4.2.4 Let X1 , X2 , . . . , X10 be a random sample from a normal distribution with σ 2 = 0.8. Find two positive numbers a and b such that the sample variance S 2 satisﬁes P a ≤ S 2 ≤ b = 0.90.

Solution

2 Because (n−1)S ∼ χ2 (n − 1), we have σ2

P a ≤ S2 ≤ b = P

(n − 1) a (n − 1) S 2 (n − 1) b . ≤ ≤ σ2 σ2 σ2

The desired values can be found by setting the upper tail area and lower tail area each equal to 0.05. Using the chi-square table with n − 1 = 9 degrees of freedom, we have (n − 1) b 9b 2 = 16.919 = χ0.05,9 = , 0.8 σ2 which implies b = ((16.919) × (0.8) /9) = 1.50. Similarly, (n − 1) a 9a 2 = 3.325 = χ0.95,9 = . 2 0.8 σ So we have a = ((3.325) × (0.8) /9) = 0.295. Hence, P 0.295 ≤ S 2 ≤ 1.50 = 0.90. It is important to note that this is not the only interval that would satisfy P a ≤ S 2 ≤ b = 0.90 but it is a convenient one.

Example 4.2.5 A fruit-drink company wants to know the variation, as measured by the standard deviation, of the amount of juice in 16-ounce cans. From past experience, it is known that σ 2 = 2. The company statistician decides to take a sample of 25 cans from the production line and compute the sample variance. Assuming that the sample values may be viewed as a random sample from a normal population, ﬁnd a value of b such that P S 2 > b = 0.05.

198 CHAPTER 4 Sampling Distributions

Solution To find the necessary probability, use the fact that (n − 1) S 2 /σ 2 ∼ χ2 (n − 1), with n = 25, 24b 24S 2 2 0.05 = P(S > b) = P > 2 2 = P(χ2 > c). 2 c = 2 (36.4151) = 3.03 and From the chi-square table we obtain, c = 36.4151. Hence, b = 24 24

P S 2 > 3.03 = 0.05.

SUMMARY OF CHI-SQUARE DISTRIBUTION Let X1 , . . . , Xn be iid N μ, σ 2 random variables. Then 1. X has N μ, σ 2 /n distribution, 2. (n − 1)S 2 /σ 2 has a chi-square distribution with (n − 1) degrees of freedom, and 3. X and S 2 are independent. 4. A χ2 − random variable has a mean equal to its degrees of freedom and a variance equal to twice its degrees of freedom.

4.2.2 Student t-Distribution Let the random variables X1 , . . . , Xn follow a normal distribution with mean μ and variance σ 2 . √ If σ is known, then we know that n X − μ /σ is N (0, 1). However, if σ is not known (as is usually the case), then it is routinely replaced by the sample standard deviation s. If the sample size is large, one could suppose that s ≈ σ and apply the Central Limit Theorem and obtain that √ n X − μ /S is approximately an N (0, 1). However, if the random sample is small, then the dis √ tribution of n X − μ /S is given by the so-called Student t-distribution (or simply t-distribution). This was originally developed by W. S. Gosset in 1908. Because his employers, the Guinness brewery, would not permit him to publish this important work in his own name, he used the pseudonym “Student.” Thus, the distribution is known as the Student t-distribution. Deﬁnition 4.2.2 If Y and Z are independent random variables, Y has a chi-square distribution with n degrees of freedom, and Z ∼ N (0, 1), then T = √

Z Y /n

is said to have a (Student) t-distribution with n degrees of freedom. We denote this by T ∼ Tn . The probability density of the random variable T with n degrees of freedom is given by − n+1 2 n+1 t2 2 n 1 + , −∞ < t < ∞. f (t) = √ n πn 2

4.2 Sampling Distributions Associated with Normal Populations 199

T density for n 5 2, n 5 10, n 5 20, n 5 30 0.4

n52

0.35 0.3 0.25

n 5 10 n 5 20

0.2 0.15

n 530

0.1 0.05 24 23

22

21

0

1

2

3

4

■ FIGURE 4.3 The Student t-distribution.

Figure 4.3 illustrates the behavior of the t-distributions for n = 2, 10, 20, and 30. It is clear from Figure 4.3 that as n becomes larger and larger, it is almost impossible to distinguish the graphs. It can be shown that the t-distribution tends to a standard normal distribution as the degrees of freedom (equivalently, the sample size n) tend to inﬁnity. In fact, the standard normal distribution provides a good approximation to the t-distribution for sample sizes of 30 or more. We will use this approximation in the statistical inference problems for n ≥ 30. The t-density is symmetric about zero, and then we have E (T ) = 0. If n > 2, it can be shown that Var (T ) = n/ (n − 2). The value of tα,n is such that P t > tα,n = α (the shaded area in Figure 4.4) is obtained from the t-table. For example, if a random variable X has a t-distribution with 9 degrees of freedom and α = 0.01, then t0.01,9 = 2.821. If we have a random sample from a normal population, the following result involving a t-distribution is useful in applications. Theorem 4.2.9 If X and S 2 are the mean and the variance of a random sample of size n from a normal population with the mean μ and variance σ 2 , then T =

X−μ √ S/ n

has a t-distribution with (n−1) degrees of freedom. Proof. By Corollary 4.2.2, Z=

X−μ √ ∼ N (0, 1) . σ/ n

200 CHAPTER 4 Sampling Distributions

f (t ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 24 23

22

21

0

1

2

3

4

t

■ FIGURE 4.4 Probability of t-distribution.

By Theorem 4.2.8, we have Y=

n 2 (n − 1) S 2 1 Xi − X ∼ χ2 (n − 1) . = 2 2 σ σ i=1

Hence, X−μ √ σ/ n

T = 8

(n−1)S 2 σ 2 (n−1)

∼ 9

Z χ2 (n−1) n−1

.

Also, X and S 2 are independent. Thus, Y and Z are independent, and by Deﬁnition 4.2.2, T follows a t-distribution with (n − 1) degrees of freedom. How can we distinguish between given degrees of freedom and the degrees of freedom from a sample? For the t-distribution, if n is given as the degrees of freedom, we will just use n. However, if a random sample of size n is given, then the corresponding degrees of freedom will be (n − 1), as given in Theorem 4.2.9. The assumption that the sample comes from a normal population is not that onerous. In practice, it is necessary to check that the sampled population is approximately bell shaped and not too much skewed. Construction of the normal-scores plot or histogram is a way to check for approximate normality. See Project 4C.

4.2 Sampling Distributions Associated with Normal Populations 201

Example 4.2.6 A manufacturer of fuses claims that with 20% overload, the fuses will blow in less than 10 minutes on the average. To test this claim, a random sample of 20 of these fuses was subjected to a 20% overload, and the times it took them to blow had the mean of 10.4 minutes and a sample standard deviation of 1.6 minutes. It can be assumed that the data constitute a random sample from a normal population. Do they tend to support or refute the manufacturer’s claim?

Solution Given y = 10.4, s = 1.6, n = 20, and μ = 10. Hence t=

y−μ 10.4 − 10 = 1.118. √ √ = s/ n 1.6/ 20

The degree of freedom is n − 1 = 19. From the t-table, the probability that t exceeds 1.328 is 0.10, and because the observed value of t = 1.118 is less than t0.10 (19) = 1.328 and 0.10 is a pretty large probability, we conclude that the data tend to agree with the manufacturer’s claim.

We will study the problems of the foregoing type in Chapter 7, where we will be learning about hypothesis testing. Prior to Student’s work on the t-distribution, a very large number of observations were necessary for design and analysis of experiments. Today, the use of the t-distribution often makes it possible to draw reliable conclusions from samples as small as 15 to 30 experimental units, provided that the samples are representative of their populations and that normality could reasonably be assumed or justiﬁed for the population.

Example 4.2.7 The human gestation period—the period of time between conception and labor—is approximately 40 weeks (280 days), measured from the ﬁrst day of the mother’s last menstrual period. For a newborn fullterm infant, the length appropriate for gestational age is assumed to be normally distributed with μ = 50 centimeters and σ = 1.25 centimeters. Compute the probability that a random sample of 20 infants born at full term results in a sample mean greater than 52.5 centimeters.

Solution Let X be length (measured in centimeters) of a newborn full-term infant. Then X ∼ N (50, 1.56/20). Hence

52.5 − 50 = 8.94 ≈ 0. P X > 52.5 = P t > √ 1.25/ 20 Thus, the probability of such an occurrence is negligible.

In the previous example, it should be noted that P X > 52.5 ≈ 0 does not imply that the probability of observing a newborn full-term infant with length greater than 52.5 centimeters is zero. In fact, with 19 degrees of freedom, P (X > 52.5) = P (t > 2) ≈ 0.025.

202 CHAPTER 4 Sampling Distributions

4.2.3 F-Distribution The F -distribution was developed by Fisher to study the behavior of two variances from random samples taken from two independent normal populations. In applied problems we may be interested in knowing whether the population variances are equal or not, based on the response of the random samples. Knowing the answer to such a question is also important in selecting the appropriate statistical methods to study their true means. Deﬁnition 4.2.3 Let U and V be chi-square random variables with n1 and n2 degrees of freedom, respectively. Then if U and V are independent, F=

U/n1 V /n2

is said to have an F-distribution with n1 numerator degrees of freedom and n2 denominator degrees of freedom. We denote this by F ∼ F (n1 , n2 ). The pdf for a random variable X ∼ F (n1 , n2 ) is given by ⎧ ⎨ ((n1 + n2 )/2) n1 n1 /2 n21 −1 n1 −(n1+n2 )/2 1 + x x , x>0 n n (n /2) (n /2) 2 2 1 2 f (x) = ⎩ 0, elsewhere.

A graph of f (x) for various values of n is given in Figure 4.5.

F – density with n 1⫽ 3, n 2⫽ 2, and n 1⫽ 12, n 2⫽ 6

0.7 0.6

F (3, 2)

0.5 0.4

F (12, 6)

0.3 0.2 0.1 0

1

2

3

4

■ FIGURE 4.5 pdfs of F -distribution.

5

6

7

4.2 Sampling Distributions Associated with Normal Populations 203

0.7 0.6 0.5 0.4 0.3

F (n 1,n 2)

0.2 0.1 0

1

2

3

4

5

6

7

■ FIGURE 4.6 F -distribution probability.

To ﬁnd Fα (n1 , n2 ) such that P (F > Fα (n1 , n2 )) = α (shaded area in Figure 4.6), we use the F -table. For example, if F has 3 numerator and 6 denominator degrees of freedom, then F0.01 (3, 6) = 9.78. If we know Fα (n1 , n2 ), it is possible to ﬁnd F1−α (n2 , n1 ) by using the identity F1−α (n2 , n1 ) = 1/Fα (n1 , n2 ) .

Using this identity we can obtain F0.99 (6, 3) = 1/F0.01 (3, 6) = 1/9.78 = 0.10225. When we need to compare the variances of two normal populations, we will use the following result. Theorem 4.2.10 Let two independent random samples of size n1 and n2 be drawn from two normal populations with variances σ12 , σ22 , respectively. If the variances of the random samples are given by S12 , S22 , respectively, then the statistic S 2 /σ 2 σ2S2 F = 12 12 = 22 12 S2 /σ2 σ1 S2

has the F-distribution with (n1 − 1) numerator and (n2 − 1) denominator degrees of freedom. Proof. From Theorem 4.2.9, we know that U=

(n1 − 1) S12 σ12

∼ χ2 (n1 − 1)

and V =

(n2 − 1) S22 σ22

∼ χ2 (n2 − 1) .

Also, U and V are independent. From Deﬁnition 4.2.3, F ∼ F (n1 − 1, n2 − 1).

204 CHAPTER 4 Sampling Distributions

Corollary 4.2.11 If σ12 = σ22 , then S2 F = 12 ∼F (n1 − 1, n2 − 1). S2

When σ12 = σ22 , we refer to them as two populations that are homogeneous with respect to their variances.

Example 4.2.8 Let S12 denote the sample variance for a random sample of size 10 from Population I and let S22 denote the sample variance for a random sample of size 8 from Population II. The variance of Population I is assumed to be three times the variance of Population II. Find two numbers a and b such that P a ≤ S12 /S22 ≤ b = 0.90 assuming S12 to be independent of S22 .

Solution From the problem, we can assume that σ12 = 3σ22 with n1 = 10 and n2 = 8. Thus, we can write S12 /σ12

S12 /3σ22 S12 = = , S22 /σ22 S22 /σ22 3S22 this has F -distribution with n1 − 1 = 9 numerator and n2 − 1 = 7 denominator degrees of freedom. Using the F -table, F0.05 (9, 7) = 3.68. Now to find F0.95 such that S12 < F0.95 = 0.05. P 3S22 We proceed as follows:

P

S12 3S22

< F0.95

=P

3S22 S12

>

1 F0.95

= 0.05.

Indexing ν1 = 7 and ν2 = 9 in the F -table, we have 1/F0.95 (7, 9) = 3.29 or F0.95 = 1/3.29 = 0.304. Hence, the entire probability statement is S12 S12 ≤ 3.68 = P 0.912 ≤ 2 ≤ 11.04 = 0.90. P 0.304 ≤ 3S22 S2 Thus, a = 0.912 and b = 11.04.

EXERCISES 4.2 4.2.1.

Let Y have a chi-square distribution with 15 degrees of freedom. Find the following probabilities. (a) P (Y ≤ y0 ) = 0.025 (b) P (a < Y < b) = 0.95 (c) P (Y ≥ 22.307).

4.2 Sampling Distributions Associated with Normal Populations 205

4.2.2.

Let Y have a chi-square distribution with 7 degrees of freedom. Find the following probabilities. (a) P (Y > y0 ) = 0.025 (b) P (a < Y < b) = 0.90 (c) P (Y > 1.239).

4.2.3.

The time to failure T of a microwave oven has an exponential distribution with pdf f (t) =

1 −t/2 e , 2

t > 0.

If three such microwave ovens are chosen and t is the mean of their failure times, ﬁnd the following: (a) Distribution of T . (b) P T > 2 . 4.2.4.

Let X1 , X2 , . . . , X10 be a random sample from a standard normal distribution. Find the numbers a and b such that 10 2 Xi ≤ b = 0.95. P a≤

4.2.5.

Let X1 , X2 , . . . , X5 be a random sample from the normal distribution with mean 55 and variance 223. Let

i=1

Y=

5

(Xi − 55)2 /223

i=1

and Z=

5

Xi − X

2

/223.

i=1

(a) Find the distribution of the random variables Y and Z. (b) Are Y and Z independent? (c) Find (i)P(0.62 ≤ Y ≤ 0.76), and (ii)P(0.77 ≤ Z ≤ 0.95). 4.2.6.

Let X and Y be independent chi-square random variables with 14 and 5 degrees of freedom, respectively. Find (a) P (|X − Y | ≤ 11.15), (b) P (|X − Y | ≥ 3.8).

4.2.7.

A particular type of vacuum-packed coffee packet contains an average of 16 ounces. It has been observed that the number of ounces of coffee in these packets is normally distributed with σ = 1.41 ounce. A random sample of 15 of these coffee packets is selected, and the observations are used to calculate s. Find the numbers a and b such that P a ≤ S 2 ≤ b = 0.90.

4.2.8.

An optical ﬁrm buys glass slabs to be ground into lenses, and it is known that the variance of the refractive index of the glass slabs is to be no more than 1.04 × 10−3 . The ﬁrm rejects a shipment of glass slabs if the sample variance of 16 pieces selected at random exceeds

206 CHAPTER 4 Sampling Distributions

1.15 × 10−3 . Assuming that the sample values may be looked on as a random sample from a normal population, what is the probability that a shipment will be rejected even though σ 2 = 1.04 × 10−3 ? 4.2.9.

Assume that T has a t-distribution with 8 degrees of freedom. Find the following probabilities. (a) P (T ≤ 2.896) (b) P (T ≤ −1.860) (c) The value of a such that P (−a < T < a) = 0.99

4.2.10.

Assume that T has a t-distribution with 15 degrees of freedom. Find the following probabilities. (a) P (T ≤ 1.341) (b) P (T ≥ −2.131) (c) The value of a such that P (−a < T < a) = 0.95

4.2.11.

A psychologist claims that the mean age at which female children start walking is 11.4 months. If 20 randomly selected female children are found to have started walking at a mean age of 11.5 months with standard deviation of 2 months, would you agree with the psychologist’s claim? Assume that the sample came from a normal population.

4.2.12.

Let U1 and U2 be independent random variables. Suppose that U1 is χ2 with ν1 degrees of freedom while U = U1 + U2 is chi-square with ν degrees of freedom, where ν > ν1 . Then prove that U2 is chi-square random variable with ν − ν1 degrees of freedom.

4.2.13.

Show that if X ∼ χ2 (ν), then EX = ν and Var (X) = 2ν.

4.2.14.

Let X1 , . . . , Xn be a random sample with Xi ∼ χ2 (1), for i = 1, . . . , n. Show that the distribution of X−1 Z= √ 2/n

as n → ∞ is standard normal. 4.2.15. 4.2.16.

Find the variance of S 2 , assuming the sample X1 , X2 , . . . , Xn is from N μ, σ 2 .

Let X1 , X2 , . . . , Xn be a random sample an exponential distribution with parameter

from n −1 θ. Show that the random variable 2θ Xi ∼ χ2 (2n). i=1

4.2.17.

Let X and Y be independent random variables from an exponential distribution with common parameter θ = 1. Show that X/Y has an F -distribution. What is the number for degrees of freedom?

4.2.18.

Prove that if X has a t-distribution with n degrees of freedom, then X2 ∼ F (1, n).

4.2.19.

Let X be F distributed with 9 numerator and 12 denominator degrees of freedom. Find (a) P (X ≤ 3.87), (b) P (X ≤ 0.196), (c) The value of a and b such that P (a < Y < b) = 0.95.

4.3 Order Statistics 207

4.2.20.

Prove that if X ∼ F (n1 , n2 ), then 1/X ∼ F (n2 , n1 ).

4.2.21.

Find the mean and variance of F (n1 , n2 ) random variable.

4.2.22.

Let X11 , X12 , . . . , X1n1 be a random sample with sample mean X1 from a normal population with mean μ1 and variance σ12 , and let X21 , X22 , . . . , X2n2 be a random sample with sample mean X2 from a normal population with mean μ2 and variance σ22 . Assume the two samples are independent. Show that the sampling distribution of X1 − X2 is normal with mean μ1 − μ2 and variance σ12 /n1 + σ22 /n2 .

4.2.23.

Let X1 , X2 , . . . , Xn1 be a random sample from a normal population with mean μ1 and variance σ 2 , and Y1 , Y2 , . . . , Yn2 be a random sample from an independent normal population with mean μ2 and variance σ 2 . Show that X − Y − (μ1 − μ2 ) ∼ T(n1 +n2 −2) T = 8 (n1 −1)S12 +(n2 −1)S22 1 1 + n1 n2 n1 +n2 −2

4.2.24.

Show that a t-distribution tends to a standard normal distribution as the degrees of freedom tend to inﬁnity.

4.2.25.

Show that the mgf of a χ2 random variable is M (t) = (1 − 2t)−ν/2 . Using the mgf, show that the mean and variance of a chi-square distribution are ν and 2ν, respectively.

4.2.26.

Let the random variables X1 , X2 , . . . , X10 be normally distributed with mean 8 and variance 4. Find a number a such that ⎛

P⎝

10 Xi − 8 2 i=1

4.2.27.

2

⎞

≤ a⎠ = 0.95

Let X2 ∼ F (1, n). Show that X ∼ t (n).

4.3 ORDER STATISTICS In practice, the random variables of interest may depend on the relative magnitudes of the observed variable. For example, we may be interested in the maximum mileage per gallon of a particular class of cars. In this section, we study the behavior of ordering a random sample from a continuous distribution. Deﬁnition 4.3.1 Let X1 , . . . , Xn be a random sample from a continuous distribution with pdf f (x). Let Y1 , . . . , Yn be a permutation of X1 , . . . , Xn such that Y1 ≤ Y2 ≤ · · · ≤ Yn .

Then the ordered random variables Y1 , . . . , Yn are called the order statistics of the random sample X1 , . . . , Xn . Here Yk is called the kth order statistic. Because of continuity, the equality sign could be ignored.

208 CHAPTER 4 Sampling Distributions

Remark. Although Xi ’s are iid random variables, the random variables Yi ’s are neither independent nor identically distributed. Thus, the minimum of Xi ’s is Y1 = min (X1 , . . . , Xn )

and the maximum is Yn = max (X1 , . . . , Xn ).

The order statistics of the sample X1 , X2 , . . . , Xn can also be denoted by X(1) , X(2) , . . . , X(n) where X(1) < X(2) < · · · < X(n) .

Here X(k) is the kth order statistic and is equal to Yk in Deﬁnition 4.3.1. One of the most commonly used order statistics is the median, the value in the middle position in the sorted order of the values.

Example 4.3.1 (i) The range R = Yn − Y1 is a function of order statistics. (ii) The sample median M equals Ym+1 if n = 2m + 1. Hence, the sample median M is an order statistic, when n is odd. If n is even then the sample median can $ # be obtained using the order statistic, M = (1/2) Yn/2 + Y(n/2)+1 .

The following result is useful in determining the distribution of functions of more than one order statistics. Theorem 4.3.1 Let X1 , . . . , Xn be a random sample from a population with pdf f (x). Then the joint pdf of order statistics Y1 , . . . , Yn is f (y1 , . . . , yn ) =

⎧ ⎨ n!f (y1 )f (y2 ) . . . f (yn ), ⎩

0,

for y1 < · · · < yn otherwise.

The pdf of the kth order statistic is given by the following theorem. Theorem 4.3.2 The pdf of Yk is fk (y) = fYk (y) =

n! f (y) (F (y))k−1 (1 − F (y))n−k , (k − 1)! (n − k)!

for −∞ < y < ∞, where F (y) = P(Xi ≤ y) is the cdf of Xi . In particular, the pdf of Y1 is f1 (y) = nf (y) [1 − F (y)]n−1 and the pdf of Yn is fn (y) = nf (y) [F (y)]n−1 . In the following example, we will derive pdf for Yn .

4.3 Order Statistics 209

Example 4.3.2 Let X1 , . . . , Xn be a random sample from U [0, 1]. Find the pdf of the kth order statistic Yk .

Solution Since the pdf of Xi is f (x) = 1, 0 ≤ x ≤ 1, the cdf is F (x) = x, 0 ≤ x ≤ 1. Using Theorem 4.3.2, the pdf of the kth order statistic Yk reduces to fk (y) =

n! yk−1 (1 − y)n−k , 0 ≤ y ≤ 1 (k − 1)! (n − k)!

which is a beta distribution with α = k and β = n − k + 1.

The next example gives the so-called extreme (i.e., largest) value distribution, which is the distribution of the order statistic Yn .

Example 4.3.3 Find the distribution of the nth order statistic Yn of the sample X1 , . . . , Xn from a population with pdf f (x).

Solution Let the cdf of Yn be denoted by Fn (y). Then

Fn (y) = P(Yn ≤ y) = P

max Xi ≤ y

1≤i≤n

= P(X1 ≤ y, . . . , Xn ≤ y) = [F (y)]n (by independence). Hence, the pdf fn (y) of Yn is fn (y) =

d d [F (y)]n = n[F (y)]n−1 F (y) dy dy

= n[F (y)]n−1 f (y). In particular, if X1 , . . . , Xn is a random sample from U [0, 1], then the cumulative extreme value distribution is given by ⎧ ⎪ 0, ⎪ ⎨ Fn (y) = yn , ⎪ ⎪ ⎩1,

y 1.

210 CHAPTER 4 Sampling Distributions

Example 4.3.4 A string of 10 light bulbs is connected in series, which means that the entire string will not light up if any one of the light bulbs fails. Assume that the lifetimes of the bulbs, τ1 , . . . , τ10 , are independent random variables that are exponentially distributed with mean 2. Find the distribution of the life length of this string of light bulbs.

Solution Note that the pdf of τi is f(t) = 2e−2t , 0 < t < ∞, and the cumulative distribution of τi is Fτi (t) = 1−e−2t . Let T represent the lifetime of this string of light bulbs. Then, T = min(τ1 , . . . , τ10 ). Thus, FT (t) = 1 − [1 − Fτi (t)]10 . Hence, the density of T is obtained by differentiating FT (t) with respect to t, that is, fT (t) = 10fτi (t)[1 − Fτi (t)]9 2(10)e−2t (e−2t )9 = 20e−20t , 0,

=

0 0

0,

otherwise,

212 CHAPTER 4 Sampling Distributions

and fn (yn ) =

⎧ ⎨ n e−yn /θ 1 − e−yn /θ n−1 , ⎩

θ

0,

if yn > 0 otherwise.

(b) Let n = 2l + 1. Show that the sampling distribution of the median, M, is given by ⎧ ⎨ n! e−m(l+1)/θ 1 − e−m/θ l , 2 f (m) = (l!) θ ⎩ 0,

for m > 0 otherwise.

4.3.11.

Let X1 , . . . , Xn be a random sample from a beta distribution with α = 2 and β = 3. Find the joint pdf of Y1 and Yn .

4.3.12.

Let X1 , . . . , Xn be a random sample from a geometric distribution with pmf pi = P (X = i) = pqi−1 , i = 1, 2, . . . , 0 < p < 1, q = 1 − p.

Show that P(Yk = y) =

n i=k

n (y−1)(n−i) n−i q {q [1 − qy ]i − [1 − qy−1 ]i }, i

y = 1, 2, . . . .

4.4 LARGE SAMPLE APPROXIMATIONS If the sample size is large, the normality assumption on the underlying population can be relaxed. A useful generalization of Corollary 4.2.2 follows. Theorem 4.4.1 Suppose that the population (not necessarily normal) from which samples are taken has a probability distribution with mean μ and variance σ 2 . Then the standardized variable (or z-transform) associated with X, given by Z=

X−μ √ σ/ n

is asymptotically standard normal. That is, 1 lim P (Z ≤ z) = √ n→∞ 2π

z

2 e−u /2 du.

−∞

Theorem 4.4.1 follows directly from the Central Limit Theorem. The consequence of this for statistics is that, regardless of the form of the population distribution, the distribution of the z-transform of a sample mean X will be approximately a standard normal random variable whenever n is large. This fact will be used in almost all large sample inference problems. It is important to note that, by

4.4 Large Sample Approximations 213

Theorem 4.2.2, if the random sample came from a normal population, then sampling distribution of the mean is normally distributed regardless of the size of the sample. We could use the foregoing results if the population variance σ 2 is known or when the sample size is large. Even though the required sample size to apply Theorem 4.4.1 will depend on the particular distribution of the population, for practical purposes we will consider the sample size to be large enough if n ≥ 30.

Example 4.4.1 The average SAT score for freshmen entering a particular university is 1100 with a standard deviation of 95. What is the probability that the mean SAT score for a random sample of 50 of these freshmen will be anywhere from 1075 to 1110?

Solution

√ The distribution of X has the mean μX = 1100 and σX = 95/ 50. By Theorem 4.4.3, √ X ∼ N 1100, σX = 95/ 90 . The z-series corresponding to 1075 and 1110 are z = [(1075 − 1100)/ √ √ 95/ 50 = −1.8608 and z = (1110 − 1100)/95/ 50 = 0.74432. Hence P 1075 ≤ X ≤ 1110 = P (−1.8608 ≤ Z ≤ 0.74432) = 0.739 means that we are 73.9% certain based on the given data that the mean SAT score is between 1075 and 1110, inclusive.

4.4.1 The Normal Approximation to the Binomial Distribution We know that a binomial random variable Y , with parameters n and p = P (success), can be viewed as the number of successes in n trials and can be written as Y=

n

Xi

i=1

where, Xi =

1

with probability p

0

with probability (1 − p).

The fraction of successes in n trials is 1 Y = Xi = X. n n n

i=1

Hence, Y /n is a sample mean. Since E (Xi ) = p and Var (Xi ) = p (1 − p), we have n

Y 1 1 E =E Xi = np = p n n n i=1

214 CHAPTER 4 Sampling Distributions

■ FIGURE 4.7 Probability function of discrete r.v.

and

n p (1 − p) 1 Y = 2 . Var Var (Xi ) = n n n i=1

Because Y = nX, by the Central Limit Theorem, Y has an approximate normal distribution with mean μ = np and variance σ 2 = np(1 − p). Because the calculation of the binomial probabilities is cumbersome for large sample sizes n, the normal approximation to the binomial distribution is widely used. A useful rule of thumb for use of the normal approximation to the binomial distribution is to make sure n is large enough if np ≥ 5 and n(1 − p) ≥ 5. Otherwise, the binomial distribution may be so asymmetric that the normal distribution may not provide a good approximation. Other rules, such as np ≥ 10 and n(1 − p) ≥ 10, or np(1 − p) ≥ 10, are also used in the literature. Because all of these rules are only approximations, for consistency’s sake we will use np ≥ 5 and n(1 − p) ≥ 5 to test for largeness of sample size in the normal approximation to the binomial distribution. If need arises, we could use the more stringent condition np(1 − p) ≥ 10. Recall that discrete random variables take no values between integers, and their probabilities are concentrated at the integers as shown in Figure 4.7. However, the normal random variables have zero probability at these integers; they have nonzero probability only over intervals. Because we are approximating a discrete distribution with a continuous distribution, we need to introduce a correction factor for continuity which is explained below.

CORRECTION FOR CONTINUITY FOR THE NORMAL APPROXIMATION TO THE BINOMIAL DISTRIBUTION (a) To approximate P(X ≤ a) or P(X > a), the correction for continuity is (a + 0.5), that is,

(a + 0.5) − np P(X ≤ a) = P Z < √ np(1 − p) and

(a + 0.5) − np . P(X > a) = P Z > √ np(1 − p)

(b) To approximate P(X ≥ a) or P(X < a), the correction for continuity is (a − 0.5), that is,

(a − 0.5) − np P(X ≥ a) = P Z > √ np(1 − p)

4.4 Large Sample Approximations 215

f (x)

x i ⫺1/2 i ⫹1/2 ■ FIGURE 4.8 Continuity correction for P(X = i).

and

(a − 0.5) − np . P(X < a) = P Z < √ np(1 − p)

(c) To approximate P(a ≤ X ≤ b), treat ends of the intervals separately, calculating two distinct z-values according to steps (a) and (b), that is,

(a + 0.5) − np (a − 0.5) − np 225) = P Z > 7.8804 more than 225 drivers will use less than one third of the acceleration lane length before merging.

EXERCISES 4.4 4.4.1.

A random sample size of 150 is taken from an inﬁnite population having mean μ = 8 and variance σ 2 = 4. What is the probability that X will be between 7.5 and 10?

4.4.2.

A machine that is used to ﬁll bottles with soda has been observed to have a true standard deviation in the amounts of ﬁll of approximately σ = 1.25 ounces. However, the mean ounces of ﬁll μ may change from day to day, because of change of operator or adjustments in the machine. If n = 55 observations on ounces of ﬁll are taken on a given day, ﬁnd the probability that the sample mean will be within 0.5 ounce of the true population mean. State any assumptions.

4.4.3.

The times spent by customers coming to a certain gas station to ﬁll up can be viewed as independent random variables with a mean of 3 minutes and a variance of 1.5 minutes. Approximate the probability that a random sample of 75 customers in this gas station will spend a total time less than 3 hours. Interpret your results and state any assumptions.

4.4.4.

Refer to Exercise 4.4.3. Find the number of customers, m, such that the probability that all the m customers can ﬁll up in less than 3 hours is approximately 0.2.

4.4.5.

In the mathematics department of a certain university, in a particular semester, 1250 students took the elementary algebra ﬁnal examination. The mean was 69% with a standard deviation of 5.4%. If a random sample of 60 students is selected from this population, what is the

4.4 Large Sample Approximations 217

probability that the average score of this sample will be at most 75.08? Interpret your results and state any assumptions. 4.4.6.

For a newborn full-term infant, the weight appropriate for gestational age is assumed to be normally distributed with μ = 3025 grams and σ = 165 grams. Compute the probability that a random sample of 50 infants born at full term results in a sample mean of less than 3500 grams.

4.4.7.

Let X1 , . . . , Xn be a random sample, each with mean μ1 and standard deviation σ1 . Also, let Y1 , Y2 , . . . , Ym be a random sample, each with mean μ2 and a standard deviation σ2 . Assume that both the samples are from normal populations. Verify that 1 2 X − Y ∼ N μ1 − μ2 , 1n σ12 + m σ2 .

4.4.8.

Let X1 , . . . , Xn be a random sample, each with mean μ1 and standard deviation σ1 . Also, let Y1 , Y2 , . . . , Yn be a random sample independent of X1 , . . . , Xn , each with mean μ2 and a standard deviation σ2 . Prove that the random variable Vn =

X − Y − (μ1 − μ2 ) 9 2 2 σ1 +σ2 n

satisﬁes the conditions of Theorem 4.4.1 and hence Vn is asymptotically normal. 4.4.9.

Suppose X is a binomial random variable with n = 20 and p = 0.2. Find the probability that X ≤ 10 using binomial tables and compare this to the corresponding value found from normal approximation.

4.4.10.

Using normal approximation, ﬁnd the probability of obtaining 90 heads in 150 tosses of a fair coin. Is the normal approximation valid? Why?

4.4.11.

A car rental company ﬁnds that each day 6% of the persons making reservations will not show up. If the rental company reserves for 215 persons with only 200 automobiles, what is the probability that an automobile will be available for every person who shows up holding a reservation? (Use the normal approximation.)

4.4.12.

The president of the United States is thought to have a positive approval rating of 58% of the people at a certain time. In a random sample of 1200 people, what is the approximate probability that the number of positive approvals will be at least 750? Interpret your results and state any assumptions.

4.4.13.

In the United States, sudden infant death syndrome (SIDS) is one of the leading causes of postneonatal deaths (those occurring between the ages of 28 days and 1 year). Thus far, the most signiﬁcant risk factor discovered for SIDS is placing babies to sleep in a prone position (on their stomachs). Suppose the rate of death due to SIDS is 0.00103 per year. In a random sample of 5000 infants between the ages of 28 days and 1 year, what is the approximate probability that the number of SIDS-related deaths will be at least 10? Interpret your results and state any assumptions.

218 CHAPTER 4 Sampling Distributions

4.4.14.

Let X and Y be independent binomial random variables with parameters (n, p1 ) and (m, p2 ), respectively. X Y (a) Find E − . n n

X Y (b) Find Var − . n n

X Y X Y X Y (c) Show that − ∼N E − , Var − , for large n. n n n n n n

4.5 CHAPTER SUMMARY In this chapter, we learned about sampling distributions. In sampling distributions associated with normal populations, we have seen that we can generate chi-square, t-, and F -distributions. In Section 4.3 we dealt with order statistics. Then in Section 4.4 we looked at large sample approximations such as the normal approximation to the binomial distribution. In the following section, we will give Minitab examples to show how the idea of sampling distribution can be explored using statistical software. We will now list some of the key deﬁnitions introduced in this chapter. ■

Sampling distribution

■

Sample and sample size

■

Random sample

■

Statistic

■

Standard error

■

Finite population correction factor

■

Degrees of freedom

■

t-distribution

■

F -distribution

■

Order statistics

In this chapter, we have also presented the following important concepts and procedures: ■

Sampling distribution associated with normal distribution

■

Results on chi-square distribution

■

Results on Student t-distribution

■

Results on F -distribution

■

Derivation of probability density functions for order statistics

■

Large sample approximations

■

Normal approximation to the binomial

■

Correction for continuity for the normal approximation to the binomial distribution

4.6 Computer Examples 219

4.6 COMPUTER EXAMPLES 4.6.1 Minitab Examples Example 4.6.1 Create three samples of size 30 from standard normal distribution using Minitab, and draw histograms for each sample.

Solution We can use the following procedure: 1. Open a new worksheet. 2. Choose Calc > Random Data > Normal. 3. Generate 30 rows of data. 4. Store results in C1-C3. 5. Enter a mean of 0 and a standard deviation of 1 and click OK. 6. Choose Graph > Character Graphs > Histogram and enter C1-C3 in the variable box and click OK. We will not give the data or any of the three histograms that we will get. These histograms are just lines containing *’s. If we need actual histograms, in step 6 use Graph > Histogram and enter C1 in the graph variable box and click OK If we wish to generate descriptive statistics, then 7. Choose Stat > Basic Statistics > Display Descriptive statistics. . . , enter C1-C3 in the variable box, and click OK. If we would like to see the mean for the three samples, 8. Choose Calc > Row Statistics, then click Mean and in the Input variables type C1-C3. In Store Result in: C4 and Click OK. To see the histogram of these averages, follow step 6 with C4 in the graph variable box. Using a similar procedure, one could generate samples from normal distributions with different means and standard deviations, as well as from other distributions.

4.6.2 SPSS Examples If we have the full version of SPSS, we can write code that can be used to simulate a sampling distribution with different values of p. However, with the student version, it is not easy to simulate. Therefore, we will not give SPSS examples in this chapter.

4.6.3 SAS Examples Example 4.6.2 Generate 50,000 observations from a normal distribution with mean 30 and standard deviation 8. Obtain summary statistics for these data and draw a graph.

220 CHAPTER 4 Sampling Distributions

Solution We could use the following program. title ’50000 Obs Sample from a Normal Distribution’; title2 ’with Mean=30 and Standard Deviation=8’; data normaldat; do n=1 to 50000; X=8*rannor(55)+ 30; output; end; run; proc univariate data=normaldat; var x; run; proc chart; vbar x / midpoints=6 to 54 by 2; format x msd.; run; In the foregoing program, rannor(55), the number 55 is just a seed number to obtain the same series of random numbers each time we run the program. If we use ‘0’, each time we run the program we will get a different set of random numbers. We will not give the output.

Example 4.6.3 From an exponential distribution, draw 10,000 samples, each sample of size 15. Compute the mean of each sample and draw a chart for the means. This will be an approximate sampling distribution of X for a ﬁxed sample of size 15.

Solution Use the following program. title ’10000 Sample Means with 15 Obs per Sample’; title2 ’Drawn from an Exponential Distribution’; data sample15; do Sample=1 to 10000; do n=1 to 15; X=ranexp(3); output; end; end;

Projects for Chapter 4 221

proc means data=sample 15 noprint; output out=mean 15 mean=Mean; var x; by sample; run; proc chart data=mean 15; vbar mean/axis=1800 midpoints=0.10 to 2.05 by .1; run; proc univariate data=mean4 noextrobs=0 normal mu0=1; var mean; run; This will produce an approximate sampling distribution of X. We will not give the output.

PROJECTS FOR CHAPTER 4 4A. A Method to Obtain Random Samples from Different Distributions Most of the statistical software packages contain a random number generator that produces approximations to random numbers from the uniform distribution U [0, 1]. To simulate the observation of any other continuous random variables, we can start with uniform random numbers and associate these to the distribution we want to simulate. For example, suppose we wish to simulate an observation from the exponential distribution F (x) = 1 − e−0.5x ,

0 < x < ∞.

First produce the value of y from the uniform distribution. Then solve for x from the equation y = F (x) = 1 − e−0.5x .

So x = [− ln (1 − y)] /0.5 is the corresponding value of the exponential random variable. For instance, if y = 0.67, then x = [− ln (1 − y)] /0.5 = 2.2173. If we wish to simulate a sample from the distribution F from the different values of y obtained from the uniform distribution, the procedure is repeated for each new observation x. (a) Simulate 10 observations of a random variable having exponential distribution with mean and standard deviation both equal to 2. (b) Select 1500 random samples of size n = 10 measurements from a population with an exponential distribution with mean and standard deviation both equal to 2. Calculate sample mean for each of these 1500 samples and draw a relative frequency histogram. Based on Theorems 4.1.1 and 4.4.1, what can you conclude? It should be noted that in general, if Y ∼ U (0, 1) random variable, then we can show that X = − lnY λ will give an exponential random variable with parameter λ. Uniform random variables could also

222 CHAPTER 4 Sampling Distributions

be used to generate random variables from other distributions. For example, let Ui s be iid U [0, 1] random variables. Then, X = −2

ν

2 , ln (Ui ) ∼ χ2ν

i=1

and Y = −β

α

ln (Ui ) ∼ Gamma (α, β) .

i=1

Of course, these transformations are useful only when ν and α are integers. More efﬁcient methods, such as MCMC methods, are discussed in Chapter 13.

4B. Simulation Experiments When the derivation via probability rules is too difﬁcult or complicated to be carried out, one can use simulation experiments to obtain information about a statistic’s sampling distribution. The following characteristics of the experiment must be speciﬁed: (i) The population distribution (normal with μ = 10 and σ = 2, exponential with λ = 5, etc.) (ii) The sample size n and the statistic of interest (X, S, etc.) (iii) The number of replications k (such as k = 300) Then, using a computer program, obtain k different random samples, each of size n, from the designated population distribution. Calculate the value of the statistic for each of the k replications. Construct a histogram for this k statistic. This histogram gives the approximate sampling distribution of the statistic. The larger the value of k, the better will be the approximation. (a) For your simulation study, use the population distribution as normal with μ = 3.4 and σ = 1.2. For n = 8 perform k = 500 replications and draw a histogram for values of the sample means. Repeat the experiment with n = 15, n = 25, and n = 35 and draw the histograms. Based on this exercise, you will be able to intuitively verify the result that X based on a large n tends to be closer to μ than does X based on a small n. (b) Repeat the experiment of part (a) with different values of k, such as k = 200, k = 750, and k = 1000. (c) Repeat the simulation study with different distributions such as exponential distribution.

4C. A Test for Normality Many statistical procedures require that the population be at least approximately normal. Therefore, a procedure is needed for checking that the sampled data could have come from a normal distribution. There are many procedures, such as the normal-score plot, or Lilliefors test for normality, available in statistics for this purpose. We will describe the normal-score plot, which is an effective way to detect deviations from normality. The normal scores consist of values of z that divide the axes into equal probability intervals. For a sample of size 4, the normal scores are −z0.20 = −0.84, −z0.40 = −0.25, z0.40 = −0.25, and z0.20 = 0.84.

Projects for Chapter 4 223

STEPS TO CONSTRUCT A NORMAL PLOT 1. Rearrange the n data points in ascending order. 2. Obtain the n normal scores. 3. Plot the kth largest observation, versus the k th normal score, for all k . 4. If the data were from a standard normal distribution, the plot would resemble a 45 degree line through the origin. 5. If the observations were from normal (but not from standard normal), the pattern should still be a straight line. However, the line need not pass through the origin or have a slope 1.

In applications, a minimum of 15 to 20 observations is needed to reach a more accurate conclusion.

EXERCISES 1.

For different observations, construct normal plots and check for normality of the corresponding populations.

2.

Using software (such as Minitab), generate 15 observations each from the following distributions: (a) Normal (2, 4), (b) Uniform (0, 1), (c) Gamma (2, 4), and (d) Exponential (2). For each of these data sets, draw a probability plot and note the geometry of the plots.

This page intentionally left blank

Chapter

5

Point Estimation Objective: In this chapter we study some statistical methods to ﬁnd point estimators of population parameters and study their properties. 5.1 Introduction 226 5.2 The Method of Moments 227 5.3 The Method of Maximum Likelihood 235 5.4 Some Desirable Properties of Point Estimators 246 5.5 Other Desirable Properties of a Point Estimator 266 5.6 Chapter Summary 282 5.7 Computer Examples 283 Projects for Chapter 5 285

C. R. Rao (Source: http:www.science.psu.edu/alert/Rao6-2007.htm)

Calyampudi Radhakrishna (C. R.) Rao (1920–) is a contemporary statistician whose work has inﬂuenced not just statistics, but such diverse ﬁelds as anthropology, biometry, demography, economics, Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

225

226 CHAPTER 5 Point Estimation

genetics, geology, and medicine. Several statistical terms and equations are named after Rao. He has worked with many other famous statisticians such as Blackwell, Fisher, and Neyman and has had dozens of theorems named after him. Rao earned an M.A. in mathematics and another M.A. in statistics, both in India, and earned his Ph.D. and Sc.D. at Cambridge University. The following was stated in the Preface to the 1991 special issue of the Journal of Quantitative Economics in Rao’s honor: “Dr. Rao is a very distinguished scientist and a highly eminent statistician of our time. His contributions to statistical theory and applications are well known, and many of his results, which bear his name, are included in the curriculum of courses in statistics at bachelor’s and master’s level all over the world. He is an inspiring teacher and has guided the research work of numerous students in all areas of statistics. His early work had greatly inﬂuenced the course of statistical research during the last four decades. One of the purposes of this special issue is to recognize Dr. Rao’s own contributions to econometrics and acknowledge his major role in the development of econometric research in India.” The importance of statistics can be summarized in Rao’s own words: “If there is a problem to be solved, seek statistical advice instead of appointing a committee of experts. Statistics can throw more light than the collective wisdom of the articulate few.”

5.1 INTRODUCTION In statistical analysis, point estimation of population parameters plays a very signiﬁcant role. In studying a real-world phenomenon we begin with a random sample of size n taken from the totality of a population. The initial step in statistically analyzing these data is to be able to identify the probability distribution that characterizes this information. Because the parameters of a distribution are its deﬁning characteristics, it becomes necessary to know the parameters. In the present chapter, we assume that the form of the population distribution is known (binomial, normal, etc.) but the parameters of the distribution (p for a binomial; μ and σ 2 for a normal, etc.) are unknown. We shall estimate these parameters using the data from our random sample. It is extremely important to have the best possible estimate of the population parameter(s). Having such estimates will lead to a better and more accurate statistical analysis. For example, in the area of phosphate mining in Florida, we may be interested in estimating the average radioactivity from both uranium and radium in a clay settling area of a mining site. Suppose that a random sample of 10 such sites resulted in a sample average of 40 pCi/g (picocuries/gram) of radioactivity. We may use this value as an estimate of the average radioactivity for all of the settling areas of mining sites in Florida. Because many Florida crops are grown on clay settling areas, this type of estimate is important for accessing the radioactivity-associated risks that are due to eating food from the crops grown on these clay settling areas. We will now introduce some of the more useful statistical point estimation methods, discuss their properties, and illustrate their usefulness with a number of applications. The importance of point estimates lies in the fact that many statistical formulas are based on them. For example, the point estimates of mean and standard deviation are used in the calculation of conﬁdence intervals and in many formulas for hypothesis testing. These topics are covered in subsequent chapters. Also, in most applied problems, a certain numerical characteristic of the physical phenomenon may be of interest; however, its value may not be observable directly. Instead, suppose it is possible to observe one or more random variables, the distribution of which depends on the characteristic of interest. Our

5.2 The Method of Moments 227

objective will be to develop methods that use the observed values of random variables (sample data) in order to gain information about the unknown and unobservable characteristic of the population. Let X1 , . . . , Xn be independent and identically distributed (iid) random variables (in statistical language, a random sample) with a pdf or pf f (x, θ1 , . . . θl ), where θ1 , . . . , θl are the unknown population parameters (characteristics of interest). For example, a normal pdf has parameters μ (the mean) and σ 2 (the variance). The actual values of these parameters are not known. The problem in point estimation is to determine statistics gi (X1 , . . . , Xn ), i = 1, . . . , l, which can be used to estimate the value of each of the parameters—that is, to assign an appropriate value for the parameters θ = (θ1 , . . . , θl ) based on observed sample data from the population. These statistics are called estimators for the parameters, and the values calculated from these statistics using particular sample data values are called estimates of the parameters. Estimators of θi are denoted by θˆ i , where θˆ i = gi (X1 , . . . , Xn ), i = 1, . . . , l. Observe that the estimators are random variables. As a result, an estimator has a distribution (which we called the sampling distribution in Chapter 4). When we actually run the experiment and observe the data, let the observed values of the random variables be X1 , . . . , Xn be x1 , . . . , xn ; then, θˆ (X1 , . . . , Xn ) is an estimator, and its value θˆ (x1 , . . . , xn ) is an estimate. For example, in case of the normal distribution, the parameters of interest are θ1 = μ, and θ2 = σ 2 , that is, θ = (μ, σ 2 ). If the estimators of μ and σ 2 are X = (1/n) ni=1 Xi and S 2 = (1/n − 1) ni=1 (Xi − X)2 respectively, then, the corresponding n estimates are x = (1/n) i=1 xi and s2 = (1/n − 1) ni=1 (xi − x)2 , the mean and variance corresponding to the particular observed sample values. In this book, we use capital letters such as X and S 2 to represent the estimators, and lowercase letters such as x and s2 to represent the estimates. There are many methods available for estimating the true value(s) of the parameter(s) of interest. Three of the more popular methods of estimation are the method of moments, the method of maximum likelihood, and Bayes’ method. A very popular procedure among econometricians to ﬁnd a point estimator is the generalized method of moments. In this chapter we study only the method of moments and the method of maximum likelihood for obtaining point estimators and some of their desirable properties. In Chapter 11, we shall discuss Bayes’ method of estimation. There are many criteria for choosing a desired point estimator. Heuristically, some of them can be explained as follows (detailed coverage is given in Sections 5.2 through 5.5). An estimator, θˆ , is unbiased if the mean of its sampling distribution is the parameter θ. The bias of θˆ is given by B = E(θˆ ) − θ. The estimator satisﬁes the consistency property if the sample estimator has a high probability of being close to the population value θ for a large sample size. The concept of efﬁciency is based on comparing variances of the different unbiased estimators. If there are two unbiased estimators, it is desirable to have the one with the smaller variance. The estimator has the sufﬁciency property if it fully uses all the sample information. Minimal sufﬁcient statistics are those that are sufﬁcient for the parameter and are functions of every other set of sufﬁcient statistics for those same parameters. A method due to Lehmann and Scheffé can be used to ﬁnd a minimal sufﬁcient statistic.

5.2 THE METHOD OF MOMENTS How do we ﬁnd a good estimator with desirable properties? One of the oldest methods for ﬁnding point estimators is the method of moments. This is a very simple procedure for ﬁnding an estimator for one or more population parameters. Let μk = E[Xk ] be the kth moment about the origin of a

228 CHAPTER 5 Point Estimation random variable X, whenever it exists. Let mk = (1/n) ni=1 Xik be the corresponding kth sample moment. Then, the estimator of μk by the method of moments is mk . The method of moments is based on matching the sample moments with the corresponding population (distribution) moments and is founded on the assumption that sample moments should provide good estimates of the corresponding population moments. Because the population moments μk = hk (θ1 , θ2 , . . . , θl ) are often functions of the population parameters, we can equate corresponding population and sample moments and solve for these parameters in terms of the moments.

METHOD OF MOMENTS Choose as estimates those values of the population parameters that are solutions of the equations μk = mk , k = 1, 2, . . . , l. Here μk is a function of the population parameters.

For example, the ﬁrst population moment is μ1 = E(X), and the ﬁrst sample moment is X = n X. If k = 2, then the second population and i=1 Xi /n. Hence, the moment estimator of μ1 is 2 sample moments are μ2 = E(X ) and m2 = (1/n) ni=1 Xi2 , respectively. Basically, we can use the following procedure in ﬁnding point estimators of the population parameters using the method of moments.

THE METHOD OF MOMENTS PROCEDURE Suppose there are l parameters to be estimated, say θ = (θ1 , . . . , θl ). 1. Find l population moments, μk , k = 1, 2, . . . , l. μk will contain one or more parameters θ1 , . . . , θl . 2. Find the corresponding l sample moments, mk , k = 1, 2, . . . , l. The number of sample moments should equal the number of parameters to be estimated. 3. From the system of equations, μk = mk , k = 1, 2, . . . , l, solve for the parameter θ = (θ1 , . . . , θl ); this will be a moment estimator of θˆ .

The following examples illustrate the method of moments for population parameter estimation.

Example 5.2.1 Let X1 , . . . , Xn be a random sample from a Bernoulli population with parameter p. (a) Find the moment estimator for p. (b) Tossing a coin 10 times and equating heads to value 1 and tails to value 0, we obtained the following values: 0

1

1

0

1

0

1

Obtain a moment estimate for p, the probability of success (head).

1

1

0

5.2 The Method of Moments 229

Solution

(a) For the Bernoulli random variable, μk = E[X] = p, so we can use m1 to estimate p. Thus, m1 = pˆ =

1 Xi . n n

i=1

Let Y=

n

Xi .

i=1

Then, the method of moments estimator for p is pˆ = Y /n. That is, the ratio of the total number of heads to the total number of tosses will be an estimate of the probability of success. (b) Note that this experiment results in Bernoulli random variables. Thus, using part (a) with Y = 6, we 6 = 0.6. get the moment estimate of p is pˆ = 10 We would use this value pˆ = 0.6, to answer any probabilistic questions for the given problem. For example, what is the probability of exactly obtaining 8 heads out of 10 tosses of this coin? This can be 10 obtained by using the binomial formula, with pˆ = 0.6, that is, P(X = 8) = (0.6)8 (0.4)10−8 . 8

In Example 5.2.1, we used the method of moments to ﬁnd a single parameter. We demonstrate in Example 5.2.2 how this method is used for estimating more than one parameter.

Example 5.2.2 Let X1 , . . . , Xn be a random sample from a gamma probability distribution with parameters α and β. Find moment estimators for the unknown parameters α and β.

Solution For the gamma distribution (see Section 3.2.5), E[X] = αβ

and

E X2 = αβ2 + α2 β2 .

Because there are two parameters, we need to find the first two moment estimators. Equating sample moments to distribution (theoretical) moments, we have 1 Xi = X = αβ, n n

i=1

1 2 Xi = αβ2 + α2 β2 . n n

and

i=1

# $ Solving for α and β we obtain the estimates as α = (x/β) and β = {(1/n) ni=1 xi2 − x2 }/x .

230 CHAPTER 5 Point Estimation

Therefore, the method of moments estimators for α and β are

αˆ =

X βˆ

and

βˆ =

n 1 X2 − X2 n i i=1

X

n

Xi − X

=

2

i=1

,

nX

which implies that 2

αˆ =

2

X X X = = n . n 2 2 βˆ 1 Xi − X Xi2 − X n i=1

i=1

Thus, we can use these values in the gamma pdf to answer questions concerning the probabilistic behavior of the r.v. X.

Example 5.2.3 Let the distribution of X be N(μ, σ 2 ). (a) For a given sample of size n, use the method of moments to estimate μ and σ 2 . (b) The following data (rounded to the third decimal digit) were generated using Minitab from a normal distribution with mean 2 and a standard deviation of 1.5. 3.163 1.883 3.252 3.716 −0.049 −0.653 0.057 2.987 4.098 1.670 1.396 2.332 1.838 3.024 2.706 0.231 3.830 3.349 −0.230 1.496 Obtain the method of moments estimates of the true mean and the true variance.

Solution (a) For the normal distribution, E(X) = μ, and because Var(X) = EX2 − μ2 , we have the second moment as E(X2 ) = σ 2 + μ2 . Equating sample moments to distribution moments we have n 1 Xi = μ1 = μ n i=1

and μ2 =

n 1 2 X i = σ 2 + μ2 . n i=1

5.2 The Method of Moments 231

Solving for μ and σ 2 , we obtain the moment estimators as μ ˆ =X and σˆ 2 =

n n 2 1 2 1 2 Xi − X . Xi − X = n n i=1

i=1

(b) Because we know that the estimator of the mean is μ ˆ = X and the estimator of the variance is σˆ 2 = n 2 ˆ = 2.005, and σˆ 2 = 6.12−(2.005)2 = 2.1. (1/n) i=1 Xi2 −X , from the data the estimates are μ Notice that the true mean is 2 and the true variance is 2.25, which we used to simulate the data.

In general, using the population pdf we evaluate the lower order moments, ﬁnding expressions for the moments in terms of the corresponding parameters. Once we have population (theoretical) moments, we equate them to the corresponding sample moments to obtain the moment estimators.

Example 5.2.4 Let X1 , . . . , Xn be a random sample from a uniform distribution on the interval [a, b]. Obtain method of moment estimators for a and b.

Solution Here, a and b are treated as parameters. That is, we only know that the sample comes from a uniform distribution on some interval, but we do not know from which interval. Our interest is to estimate this interval. The pdf of a uniform distribution is ⎧ ⎨ 1 , a≤x≤b f (x) = b − a ⎩ 0, otherwise. Hence, the first two population moments are b μ1 = E(X) = a

x a+b dx = b−a 2

and

μ2 = E(X2 ) =

b a

x2 a2 + ab + b2 dx = . b−a 3

The corresponding sample moments are μ ˆ1 = X

and

μ ˆ2 =

n 1 2 Xi . n i=1

Equating the first two sample moments to the corresponding population moments, we have μ ˆ1 =

a+b 2

and

μ ˆ2 =

a2 + ab + b2 3

232 CHAPTER 5 Point Estimation

which, solving for a and b, results in the moment estimators of a and b, aˆ = μ ˆ1 −

9 3 μ ˆ2 −μ ˆ 21

bˆ = μ ˆ1 +

and

9 3 μ ˆ2 −μ ˆ 21 .

In Example 5.2.4, if a = −b, that is, X1 , . . . , Xn is a random sample from a uniform distribution on the interval (−b, b), the problem reduces to a one-parameter estimation problem. However, in this case E(Xi ) = 0, so the ﬁrst moment cannot be used to estimate b. It becomes necessary to use the second moment. For the derivation, see Exercise 5.2.3. It is important to observe that the method of moments estimators need not be unique. The following is an example of the nonuniqueness of moment estimators.

Example 5.2.5 Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ > 0. Show that both 2 (1/n) ni=1 Xi and (1/n) ni=1 Xi2 − (1/n) ni=1 Xi are moment estimators of λ.

Solution We know that E(X) = λ, from which we have a moment estimator of λ as (1/n) we have Var(X) = λ, equating the second moments, we can see that

n

i=1 Xi . Also, because

λ = E(X2 ) − (EX)2 , so that n 1 2 Xi − λˆ = n

i=1

2 n 1 Xi . n i=1

Thus, λˆ =

n 1 Xi n i=1

and λˆ =

n 1 2 Xi − n i=1

2 n 1 Xi . n i=1

Both are moment estimators of λ. Thus, the moment estimators may not be unique. We generally choose X as an estimator of λ, for its simplicity.

5.2 The Method of Moments 233

It is important to note that, in general, we have as many moment conditions as the parameters. In Example 5.2.5, we have more moment conditions than parameters, because both the mean and variance of Poisson random variables are the same. Given a sample, this results in two different estimates of a single parameter. One of the questions could be, can these two estimators be combined in some optimal way? This is done by the so-called generalized method of moments (GMM). We will not deal with this topic. As we have seen, the method of moments ﬁnds estimators of unknown parameters by equating the corresponding sample and population moments. This method often provides estimators when other methods fail to do so or when estimators are harder to obtain, as in the case of a gamma distribution. Compared to other methods, method of moments estimators are easy to compute and have some desirable properties that we will discuss in ensuing sections. The drawback is that they are usually not the “best estimators” (to be deﬁned later) available and sometimes may even be meaningless.

EXERCISES 5.2 5.2.1.

Let X1 , . . . , Xn be a random sample of size n from the geometric distribution for which p is the probability of success. (a) Use the method of moments to ﬁnd a point estimator for p. (b) Use the following data (simulated from geometric distribution) to ﬁnd the moment estimator for p: 2 4

5 34

7 19

43 21

18 23

19 6

16 21

11 7

22 12

How will you use this information? [The pdf of a geometric distribution is f (x) = p(1 − p)x−1 , for x = 1, 2, . . . . Also μ = 1/p.] 5.2.2.

Let X1 , . . . , Xn be a random sample of size n from the exponential distribution whose pdf (by taking θ = 1/β in Deﬁnition 2.3.7) is f (x, θ) =

⎧ ⎨θe−θx ,

x≥0

⎩ 0,

x < 0.

(a) Use the method of moments to ﬁnd a point estimator for θ. (b) The following data represent the time intervals between the emissions of beta particles. 0.9 0.1 0.1 0.5 0.4

0.1 0.1 0.5 3.0 0.5

0.1 0.1 0.4 1.0 0.8

0.8 2.3 0.6 0.5 0.1

0.9 0.8 0.2 0.2 0.1

0.1 0.3 0.4 2.0 1.7

0.1 0.2 0.2 1.7 0.1

0.7 0.1 0.1 0.1 0.2

1.0 1.0 0.8 0.3 0.3

0.2 0.9 0.2 0.1 0.1

234 CHAPTER 5 Point Estimation

Assuming the data follow an exponential distribution, obtain a moment estimate for the parameter θ. Interpret. 5.2.3.

Let X1 , . . . , Xn be a random sample from a uniform distribution on the interval (θ − 1, θ + 1). (a) Find a moment estimator for θ. (b) Use the following data to obtain a moment estimate for θ: 11.72

5.2.4.

12.81

12.09

13.47

12.37

The probability density of a one-parameter Weibull distribution is given by 2αxe−αx , 0, 2

f(x) =

x>0 otherwise.

(a) Using a random sample of size n, obtain a moment estimator for α. (b) Assuming that the following data are from a one-parameter Weibull population, 1.87 1.83

1.60 0.64

2.36 1.53

1.12 0.73

0.15 2.26

obtain a moment estimate of α. 5.2.5.

Let X1 , . . . , Xn be a random sample from the truncated exponential distribution with pdf f(x) =

e−(x−θ) , 0,

x≥θ otherwise.

Find the method of moments estimate of θ. 5.2.6.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f(x, α) =

1 + αx , 2

−1 ≤ x ≤ 1, and − 1 ≤ α ≤ 1.

Find the moment estimators for α. 5.2.7.

Let X1 , . . . , Xn be a random sample from a population with pdf f (x) =

⎧ ⎨ 2α2 ,

x≥α

⎩ 0,

otherwise.

x3

Find a method of moments estimator for α. 5.2.8.

Let X1 , . . . , Xn be a random sample from a negative binomial distribution with pmf p(x, r, p) =

x+r−1 x p (1 − p)x , 0 ≤ p ≤ 1, x = 0, 1, 2, . . . . r−1

5.3 The Method of Maximum Likelihood 235 # $ Find method of moments estimators for r and p. [Here E[X] = r(1 − p)/p and E X2 = r(1 − p)(r − rp + 1)/p2 .] 5.2.9.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f (x) =

(θ + 1) xθ ,

0 ≤ x ≤ 1; θ > −1

0,

otherwise.

Use the method of moments to obtain an estimator of θ. 5.2.10.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f (x) =

2β−2x , β2

0<x 0.

5.3 The Method of Maximum Likelihood 239

Hence, the likelihood function is n

xi

n 7 λxi e−λ λi=1 e−nλ L (λ) = . = n : xi ! i=1 xi ! i=1

Then, taking the natural logarithm, we have ln L(λ) =

n

xi ln λ − nλ −

i=1

n

ln (xi !)

i=1

and differentiating with respect to λ results in n

d ln L(λ) i=1 = dλ λ

xi −n

and n

d ln L(λ) i=1 = 0, implies dλ λ

xi − n = 0.

That is, n

λ=

xi

i=1

= x.

n

Hence, the MLE of λ is λˆ = X.

It can be veriﬁed that the second derivative is negative and, hence, we really have a maximum. Sometimes the method of derivatives cannot be used for ﬁnding the MLEs. For example, the likelihood is not differentiable in the range space. In this case, we need to make use of the special structures available in the speciﬁc situation to solve the problem. The following is one such case.

Example 5.3.4 Let X1 , . . . , Xn be a random sample from U(0, θ), θ > 0. Find the MLE of θ.

Solution Note that the pdf of the uniform distribution is ⎧ ⎨1 , f (x) = θ ⎩ 0,

0≤x≤θ otherwise.

240 CHAPTER 5 Point Estimation

L ()

X(n )

■ FIGURE 5.1 Likelihood function for uniform probability distribution.

Hence, the likelihood function is given by ⎧ ⎨ 1 , L (θ, x1 , x2 , . . . , xn ) = θ n ⎩ 0,

0 ≤ x1 , x2 , . . . , xn ≤ θ otherwise.

When θ ≥ max(xi ), the likelihood is (1/θ n ), which is positive and decreasing as a function of θ (for fixed n). However, for θ < max(xi ) the likelihood drops to 0, creating a discontinuity at the point max(xi ) (this is the minimum value of θ that can be chosen which still satisfies the condition 0 ≤ xi ≤ θ), and Figure 5.1 shows that the maximum occurs at this point. Hence, we will not be able to find the derivative. Thus, the MLE is the largest order statistic, θˆ = max (Xi ) = X(n) .

In the previous example, because E(X) = (θ/2), we can see that θ = 2E(X). Hence, the method of moments estimator for θ is θˆ = 2X. Sometimes the method of moments estimator can give meaningless results. To see this, suppose we observe values 3, 5, 6, and 18 from a U(0, θ) distribution. Clearly, the maximum likelihood estimate of θ is 18, whereas the method of moments estimate is 16, which is not quite acceptable, because we have already observed a value of 18. As mentioned earlier, if the unknown parameter θ represents a vector of parameters, say θ = (θ1 , . . . , θl ), then the MLEs can be obtained from solutions of the system of equations ∂ ln L (θ1 , . . . , θn ) = 0, ∂θ

for

i = 1, . . . , l.

These are called the maximum likelihood equations and the solutions are denoted by (θˆ 1 , . . . , θˆ l ).

Example 5.3.5

Let X1 , . . . , Xn be N μ, σ 2 . (a) If μ is unknown and σ 2 = σ02 is known, ﬁnd the MLE for μ. (b) If μ = μ0 is known and σ 2 is unknown, ﬁnd the MLE for σ 2 . (c) If μ and σ 2 are both unknown, ﬁnd the MLE for θ = μ, σ 2 .

5.3 The Method of Maximum Likelihood 241

Solution In order to avoid notational confusion when taking the derivative, let θ = σ 2 . Then, the likelihood function is ⎞ ⎛ n 2 (x − μ) i ⎟ ⎜ ⎟ ⎜ i=1 L (μ, θ) = (2πθ)−n/2 exp⎜− ⎟ ⎠ ⎝ 2θ or n

(xi − μ)2 n n i=1 ln L (μ, θ) = − ln (2π) − ln θ − . 2 2 2θ (a) When θ = θ0 = σ02 is known, the problem reduces to estimating the only one parameter, μ. Differentiating the log-likelihood function with respect to μ,

∂ ln L (μ, θ0 ) = ∂μ

2

n

(xi − μ)

i=1

.

2θ0

Setting the derivative equal to zero and solving for μ, n

(xi − μ) = 0.

i=1

From this, n

xi = nμ

μ = x.

or

i=1

Thus, we get μ ˆ = X. (b) When μ = μ0 is known, the problem reduces to estimating the only one parameter, σ 2 = θ. Differentiating the log-likelihood function with respect to θ, n

(xi − μ)2 −n i=1 ∂ ln L (μ, θ) = + . ∂θ 2θ 2θ 2 Setting the derivative equal to zero and solving for θ, we get n i=1 θˆ = σˆ 2 =

(Xi − μ0 )2 n

.

242 CHAPTER 5 Point Estimation

(c) When both μ and θ are unknown, we need to differentiate with respect to both μ and θ individually:

∂ ln L (μ, θ) = ∂μ

2

n

(xi − μ)

i=1

2θ

and n

(xi − μ)2 −n i=1 ∂ ln L (μ, θ) = + . ∂θ 2θ 2θ 2 Setting the derivatives equal to zero and solving simultaneously, we obtain μ ˆ = X, n

Xi − X

σˆ 2 = θˆ =

i=1

n

2 = S 2 .

Note that in (a) and (c), the estimates for μ are the same; however, in (b) and (c), the estimates for σ 2 are different.

At times, the maximum likelihood estimators may be hard to calculate. It may be necessary to use numerical methods to approximate values of the estimate. The following example gives one such case.

Example 5.3.6 Let X1 , . . . , Xn be a random sample from a population with gamma distribution and parameters α and β. Find MLEs for the unknown parameters α and β.

Solution The pdf for the gamma distribution is given by ⎧ ⎨ xα−1 e−x/β (α)βα , f (x) = ⎩ 0,

x > 0,

α > 0,

β>0

otherwise.

The likelihood function is given by n

n − xi /β 7 1 xiα−1 e i=1 . L = L(α, β) = ( (α) βα )n i=1

Taking the logarithms gives ln L = −n ln (α) − nα ln β + (α − 1)

n i=1

ln xi −

n x . β i=1

5.3 The Method of Maximum Likelihood 243

Now taking the partial derivatives with respect to α and β and setting both equal to zero, we have ∂ (α) ln L = −n − n ln β + ln xi = 0 ∂α (α) n

i=1

∂ α xi ln L = −n + = 0. ∂β β β2 n

i=1

Solving the second one to get β in terms of α, we have β=

x . α

Substituting this β in the first equation, we have to solve (α) x − n ln + ln xi = 0 (α) α n

−n

i=1

for α > 0. There is no closed-form solution for α and β. In this case, one can use numerical methods such as the Newton--Raphson method to solve for α, and then use this value to find β.

There are many references available on the Web. Explaining the Newton–Raphson method, for instance, http://web.as.uky.edu/statistics/users/viele/sta601s08/nummax.pdf gives the algorithm for the gamma distribution. In only a few cases are we able to obtain a simple form for the maximum likelihood equation that can be solved by setting the ﬁrst derivative to zero. Often we cannot write an equation that can be differentiated to ﬁnd the MLE parameter estimates. This is especially true in the situation where the model is complex and involves many parameters. Evaluating the likelihood exhaustively for all values of the parameters becomes almost impossible, even with modern computers. This is why so-called optimization algorithms have become indispensable to statisticians. The purpose of an optimization algorithm is to ﬁnd as fast as possible the set of parameter values that make the observed data most likely. There are many such algorithms available. We describe the Newton–Raphson method in Project 5F, and another powerful algorithm, known as the EM algorithm, is given in Section 13.4. Sometimes, it may be necessary to estimate a function of a parameter. The following invariance property of maximum likelihood estimators is very useful in those cases. Theorem 5.3.1 Let h(θ) be a one-to-one function of θ. If θˆ = (θˆ 1 , . . . , θˆ l ) is the MLE of θ = (θ1 , . . . , θl ), then the MLE of a function h(θ) = (h1 (θ), . . . , hk (θ)) of these parameters is h(θˆ ) = (h1 (θˆ ), . . . , hk (θˆ )) for 1 ≤ k ≤ l. As a consequence of the invariance property, in Example 5.3.5, we can obtain the estimator of the 9 √ 2 true standard deviation as σˆ = σˆ 2 = (1/n) ni=1 Xi − X .

244 CHAPTER 5 Point Estimation

It is also known that, under very general conditions on the joint distribution of the sample and for a large sample size n, the MLE θˆ is approximately the minimum variance unbiased estimator (this concept is introduced in the next section) of θ.

EXERCISES 5.3 5.3.1.

Let X1 , . . . , Xn be a random sample recorded as heads or tails resulting from tossing a coin n times with unknown probability p of heads. Find the MLE pˆ of p. Also using the invariance property, obtain an MLE for q = 1 − p. How would you use the results you have obtained?

5.3.2.

Suppose X1 , . . . , Xn are a random sample from an exponential distribution with parameter θ. Find the MLE of θˆ . Also using the invariance property, obtain an MLE for the variance.

5.3.3.

Let X be a random variable representing the time between successive arrivals at a checkout counter in a supermarket. The values of X in minutes (rounded to the nearest minute) are 1 12

2 7

3 3

7 2

11 11

4 7

13 2

Assume that the pdf of X is f (x) = (1/θ)e−(x/θ) . Use these data to ﬁnd MLE θˆ . How can you use this estimate you have just derived? 5.3.4.

Let X1 , . . . , Xn be a random sample from the truncated exponential distribution with pdf f (x) =

⎧ ⎨e−(x−θ) , ⎩

0,

x≥θ otherwise.

Show that the MLE of θ is min(Xi ). 5.3.5.

The pdf of a random variable X is given by f (x) =

⎧ ⎨ 2x2 e−x2 /α2 , ⎩

x>0

α

0,

otherwise.

Using a random sample of size n, obtain MLE αˆ for α. 5.3.6.

The pdf of a random variable X is given by 1 exp αn − eα , P (X = n) = n!

n = 0, 1, 2, . . . .

Using a random sample of size n, obtain MLE αˆ for α. 5.3.7.

Let X1 , . . . , Xn be a random sample from a two-parameter Weibull distribution with pdf f (x) =

Find the MLEs of α and β.

⎧ α ⎨ αα xα−1 e−(x/β) , β ⎩

0,

x≥0 otherwise.

5.3 The Method of Maximum Likelihood 245

5.3.8.

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

⎧ 2 ⎨ αx e−x /2α , ⎩

x>0

0,

otherwise.

Find the MLEs of α. 5.3.9.

Let X1 , . . . , Xn be a random sample from a two-parameter exponential population with density 1 − (x−υ) θ , e θ

f (x, θ, υ) =

for

x ≥ υ,

θ > 0.

Find MLEs for θ and υ when both are unknown. 5.3.10.

Let X1 , . . . , Xn be a random sample from the shifted exponential distribution with pdf f (x) =

⎧ ⎨λe−λ(x−θ) , ⎩

x≥θ

0,

otherwise.

Obtain the maximum likelihood estimators of θ and λ. 5.3.11.

Let X1 , . . . , Xn be a random sample on [0, 1] with pdf f (x) =

(2θ) [x(1 − x)]θ−1 , (θ)2

θ > 0.

What equation does the maximum likelihood estimate of θ satisfy? 5.3.12.

Let X1 , . . . , Xn be a random sample with pdf f (x) =

⎧ ⎨(α + 1)xα ,

0≤x≤1

⎩

otherwise.

0,

Find the MLE of α. 5.3.13.

Let X1 , . . . , Xn be a random sample from a uniform distribution with pdf f (x) =

⎧ 1 ⎨ 3θ+2 , ⎩

0,

0 ≤ x ≤ 3θ + 2 otherwise.

Obtain the MLE of θ. 5.3.14.

Let X1 , . . . , Xn be a random sample from a Cauchy distribution with pdf f (x) =

Find the MLE for β.

1 $, # π 1 + (x − β)2

−∞ < x < ∞.

246 CHAPTER 5 Point Estimation

5.3.15.

The following data represent the amount of leakage of a ﬂuorescent dye from the bloodstream into the eye in patients with abnormal retinas: 1.6 1.8

1.4 6.3

1.2 2.4

2.2 2.3

1.8 18.9

1.7 22.8

Assuming that these data come from a normal distribution, ﬁnd the maximum likelihood estimate of (μ, σ). 5.3.16.

Let X1 , . . . , Xn be a random sample from a population with gamma distribution and parameters α and β. Show that the MLE of μ = αβ is the sample mean μ ˆ = X.

5.3.17.

The lifetimes X of a certain brand of component used in a machine can be modeled as a random variable with pdf f (x) = (1/θ) e−(x/θ) . The reliability R(x) of the component is deﬁned as R(x) = 1 − F (x). Suppose X1 , X2 , . . . , Xn are the lifetimes of n components randomly selected and tested. Find the MLE of R(x).

5.3.18.

Using the method explained in Project 4A, generate 20 observations of a random variable having an exponential distribution with mean and standard deviation both equal to 2. What is the maximum likelihood estimate of the population mean? How much is the observed error?

5.3.19.

Let X1 , . . . , Xn be a random sample from a Pareto distribution (named after the economist Vilfredo Pareto) with shape parameter a. The density function is given by ⎧ a ⎨ , f (x) = xa+1 ⎩ 0,

x≥1 otherwise.

(The Pareto distribution is a skewed, heavy-tailed distribution. Sometimes it is used to model the distribution of incomes.) Show that the maximum likelihood estimator of a is aˆ = n

n

.

ln (Xi )

i=1

5.3.20.

Let X1 , . . . , Xn be a random sample from N (θ, θ), 0 < θ < ∞. Find the maximum likelihood estimate of θ.

5.4 SOME DESIRABLE PROPERTIES OF POINT ESTIMATORS Two different methods of ﬁnding estimators for population parameters have been introduced in the preceding sections. We have seen that it is possible to have several estimators for the same parameter. For a practitioner of statistics, an important question is going to be which of many available sample statistics, such as mean, median, smallest observation, or largest observation, should be chosen to represent all of the sample? Should we use the method of moments estimator, the maximum

5.4 Some Desirable Properties of Point Estimators 247

likelihood estimator, or an estimator obtained through some other method of least squares (we will see this method in Chapter 8)? Now we introduce some common ways to distinguish between them by looking at some desirable properties of these estimators.

5.4.1 Unbiased Estimators It is desirable to have the property that the expected value of an estimator of a parameter is equal to the true value of the parameter. Such estimators are called unbiased estimators. Deﬁnition 5.4.1 A point estimator θˆ is called an unbiased estimator of the parameter θ if E(θˆ ) = θ for all possible values of θ. Otherwise θˆ is said to be biased. Furthermore, the bias of θˆ is given by B = E(θˆ ) − θ.

Note that the bias is nothing but the expected value of the (random) error, E(θˆ − θ). Thus, the estimator is unbiased if the bias is 0 for all values of θ. The bias occurs when a sample does not accurately represent the population from which the sample is taken. It is important to observe that in order to check whether θˆ is unbiased, it is not necessary to know the value of the true parameter. Instead, one can use the sampling distribution of θˆ . We demonstrate the basic procedure through the following example.

Example 5.4.1 Let X1 , . . . , Xn be a random sample from a Bernoulli population with parameter p. Show that the method of moments estimator is also an unbiased estimator.

Solution We can verify that the moment estimator of p is n

pˆ =

Xi

i=1

n

=

Y . n

Because for binomial random variables, E (Y ) = np, it follows that

1 1 Y = E (Y ) = · np = p. E pˆ = E n n n Hence, pˆ = Y /n is an unbiased estimator for p.

In fact, we have the following result, which states that the sample mean is always an unbiased estimator of the population mean. Theorem 5.4.1 The mean of a random sample X is an unbiased estimator of the population mean μ.

248 CHAPTER 5 Point Estimation

Proof. Let X1 , . . . , Xn be random variables with mean μ. Then, the sample mean is X = (1/n)

n

i=1 Xi .

1 1 EXi = · nμ = μ. n n n

EX =

i=1

Hence, X is an unbiased estimator of μ. How is this interpreted in practice? Suppose that a data set is collected with n numerical observations x1 , . . . , xn . The resulting sample mean may be either less than or greater than the true population mean, μ (remember, we do not know this value). If the sampling experiment was repeated many times, then the average of the estimates calculated over these repetitions of the sampling experiment will equal the true population mean. If we have to choose among several different estimators of a parameter θ, it is desirable to select one 2 that is unbiased. The following result states that the sample variance S 2 = (1/n − 1) ni=1 Xi − X is an unbiased estimator of the population variance σ 2 . This is one of the reasons why in the deﬁnition of the sample variance, instead of dividing by n, we divide by (n − 1). Theorem 5.4.2 If S 2 is the variance of a random sample from an inﬁnite population with ﬁnite variance σ 2 , then S 2 is an unbiased estimator for σ 2 . Proof. Let X1 , . . . , Xn be iid random variables with variance σ 2 < ∞. We have

& % n n / 02 1 1 ¯ 2= (Xi − μ) − X − μ E E Xi − X n−1 n−1 i=1 & % i=1 n / 02 1 2 . E {Xi − μ} − nE X − μ = n−1

E S2 =

i=1

2 Because E{(Xi − μ)2 } = σ 2 and E{ X − μ } = σ 2 /n, it follows that E S2 =

% n & 1 σ2 2 = σ2. σ −n n−1 n i=1

Hence, S 2 is an unbiased estimator of σ 2 .

It is important to observe the following: 1. S 2 is not an unbiased estimator of the variance of a ﬁnite population. 2. Unbiasedness may not be retained under functional transformations, that is; if θˆ is an unbiased estimator of θ, it does not follow that f (θˆ ) is an unbiased estimator of f (θ). 3. Maximum likelihood estimators or moment estimators are not, in general, unbiased. 4. In many cases it is possible to alter a biased estimator by multiplying by an appropriate constant to obtain an unbiased estimator. The following example will show that unbiased estimators need not be unique.

5.4 Some Desirable Properties of Point Estimators 249

Example 5.4.2 Let X1 , . . . , Xn be a random sample from a population with ﬁnite mean μ. Show that the sample mean X and 13 X + 23 X1 are both unbiased estimators of μ.

Solution By Theorem 1, X is unbiased. Now E 13 X + 23 X1 = 13 μ + 23 μ = μ. Hence, 13 X + 23 X1 is also an unbiased estimator of μ.

How many unbiased estimators can we ﬁnd? In fact, the following example shows that if we have two unbiased estimators, there are inﬁnitely many unbiased estimators.

Example 5.4.3 Let θˆ 1 and θˆ 2 be two unbiased estimators of θ. Show that θˆ 3 = aθˆ 1 + (1 − a) θˆ 2 , 0 ≤ a ≤ 1 is an unbiased estimator of θ. Note that θˆ 3 is a convex combination of θˆ 1 and θˆ 2 . In addition, assume that θˆ 1 and θˆ 2 are independent, and Var(θˆ 1 ) = σ12 and Var(θˆ 2 ) = σ22 . How should the constant a be chosen in order to minimize the variance of θˆ 3 ?

Solution We are given that E(θˆ 1 ) = θ and E(θˆ 2 ) = θ. Therefore, E θˆ 3 = E aθˆ 1 + (1 − a) θˆ 2 = aEθˆ 1 + (1 − a) Eθˆ 2 = aθ + (1 − a) θ = θ. Hence θˆ 3 is unbiased. By independence, Var θˆ 3 = Var aθˆ 1 + (1 − a) θˆ 2 = a2 Var θˆ 1 + (1 − a)2 Var θˆ 2 = a2 σ12 + (1 − a)2 σ22 . To find the minimum, d Var θˆ 3 = 2aσ12 − 2(1 − a)σ22 = 0, da

250 CHAPTER 5 Point Estimation

gives us a=

σ22 σ12 + σ22

.

d 2 V (θˆ ) = 2σ 2 + 2σ 2 > 0, V (θˆ ) has a minimum at this value of a . Thus, if σ 2 = σ 2 , then Because da 3 3 2 1 2 1 2 a = 1/2.

Example 5.4.4 Let X1 , . . . , Xn be a random sample from a population with pdf ⎧ ⎨ 1 e−x/β , x>0 β f (x) = ⎩ 0, otherwise. Show that the method of moments estimator for the population parameter β is unbiased.

Solution From Section 5.2, we have seen that the method of moments estimator for β is the sample mean X, and the population mean is β. Because E(X) = μ = β, the method of moments estimator for the population parameter β is unbiased.

As we have seen, there can be many unbiased estimators of a parameter θ. Which one of these estimators can we choose? If we have to choose an unbiased estimator, it will be desirable to choose the one with the least variance. If an estimator is biased, then we should prefer the one with low bias as well as low variance. Generally, it is better to have an estimator that has low bias as well as low variance. This leads us to the following deﬁnition. Deﬁnition 5.4.2 The mean square error of the estimator θˆ , denoted by MSE(θˆ ), is deﬁned as 2 MSE θˆ = E θˆ − θ .

Through the following calculations, we will now show that the MSE is a measure that combines both bias and variance. 2 2 MSE θˆ = E θˆ − θ = E θˆ − E θˆ + E θˆ − θ ! 2 2 " = E θˆ − E θˆ + E θˆ − θ + 2 θˆ − E θˆ E θˆ − θ 2 2 + E E θˆ − θ + 2E θˆ − E θˆ E θˆ − θ = E θˆ − E θˆ 2 = Var θˆ + E θˆ − θ ,

5.4 Some Desirable Properties of Point Estimators 251 because letting B = E(θˆ ) − θ, we get MSE θˆ = Var θˆ + B2 .

B is called the bias of the estimator. Also, E(θˆ − E(θˆ ))(E(θˆ ) − θ) = 0. Because the bias is zero for unbiased estimators, it is clear that MSE(θˆ ) = Var(θˆ ). Mean square error measures, on average, how close an estimator comes to the true value of the parameter. Hence, this could be used as a criterion for determining when one estimator is “better” than another. However, in general, it is difﬁcult to ﬁnd θˆ to minimize MSE(θˆ ). For this reason, most of the time, we look only at unbiased estimators in order to minimize Var(θˆ ). This leads to the following deﬁnition. Deﬁnition 5.4.3 The unbiased estimator θˆ that minimizes the mean square error is called the minimum variance unbiased estimator (MVUE) of θ.

Example 5.4.5 Let X1 , X2 , X3 be a sample of size n = 3 from a distribution with unknown mean μ, −∞ < μ < ∞, where the variance σ 2 is a known positive number. Show that both θˆ 1 = X and θˆ 2 = [(2X1 + X2 + 5X3 ) /8] are unbiased estimators for μ. Compare the variances of θˆ 1 and θˆ 2 .

Solution We have 1 E θˆ 1 = E X = · 3μ = μ, 3 and 1 E θˆ 2 = [2EX1 + EX2 + 5EX3 ] 8 1 = [2μ + μ + 5μ] = μ. 8 Hence, both θˆ 1 and θˆ 2 are unbiased estimators. However, σ2 , Var θˆ 1 = 3 whereas Var θˆ 2 = Var =

2X1 + X2 + 5X3 8

1 2 25 2 30 2 4 2 σ + σ + σ = σ . 64 64 64 64

Because Var(θˆ 1 ) < Var(θˆ 2 ), we see that X is a better unbiased estimator in the sense that the variance of X is smaller.

252 CHAPTER 5 Point Estimation

It is important to observe that the maximum likelihood estimators are not always unbiased, but it can be shown that for such estimators the bias goes to zero as the sample size increases.

5.4.2 Sufficiency In the statistical inference problems on a parameter, one of the major questions is: Can a speciﬁc statistic replace the entire data without losing pertinent information? Suppose X1 , . . . , Xn is a random sample from a probability distribution with unknown parameter θ. In general, statisticians look for ways of reducing a set of data so that these data can be more easily understood without losing the meaning associated with the entire collection of observations. Intuitively, a statistic U is a sufﬁcient statistic for a parameter θ if U contains all the information available in the data about the value of θ. For example, the sample mean may contain all the relevant information about the parameter μ, and in that case U = X is called a sufﬁcient statistic for μ. An estimator that is a function of a sufﬁcient statistic can be deemed to be a “good” estimator, because it depends on fewer data values. When we have a sufﬁcient statistic U for θ, we need to concentrate only on U because it exhausts all the information that the sample has about θ. That is, knowledge of the actual n observations does not contribute anything more to the inference about θ. Deﬁnition 5.4.4 Let X1 , . . . , Xn be a random sample from a probability distribution with unknown parameter θ. Then, the statistic U = g(X1 , . . . , Xn ) is said to be sufﬁcient for θ if the conditional pdf or pf of X1 , . . . , Xn given U = u does not depend on θ for any value of u. An estimator of θ that is a function of a sufﬁcient statistic for θ is said to be a sufﬁcient estimator of θ.

Example 5.4.6 Let X1 , . . . , Xn be iid Bernoulli random variables with parameter θ. Show that U = for θ.

n

i=1 Xi is sufﬁcient

Solution The joint probability mass function of X1 , . . . , Xn is n

f (X1 , . . . , Xn ; θ) = Because U =

θ i=1

Xi

(1 − θ)

n−

n

i=1

Xi

0 ≤ θ ≤ 1.

,

n

i=1 Xi we have

f (X1 , . . . , Xn ; θ) = θ U (1 − θ)n−U ,

0 ≤ U ≤ n.

Also, because U ∼ B(n, θ), we have n U f (u; θ) = θ (1 − θ)n−U . u Also, f (x1 , . . . , xn |U = u ) =

f (x1 , . . . , xn , u) = fU (u)

f (x1 ,...,xn ) fU (u) ,

0,

u=

xi

otherwise.

5.4 Some Desirable Properties of Point Estimators 253

Therefore, ⎧ θ u (1−θ)n−u ⎪ = 1 ⎪ ⎨ n n u n−u θ (1−θ) f (x1 , . . . , xn |U = u) = u u ⎪ ⎪ ⎩ 0,

if u =

Example 5.4.7 Let X1 , . . . , Xn be a random sample from U(0, θ). That is, 1, θ

if 0 < x < θ

0,

otherwise.

Show that U = max X is sufﬁcient for θ. 1≤i≤n

Solution The joint density or the likelihood function is given by

f (x1 , . . . , xn ; θ) =

1 θn ,

if 0 < x1 , . . . , xn < θ

0,

otherwise.

The joint pdf f (x1 , . . . , xn ; θ) can be equivalently written as f (x1 , . . . , xn ; θ) =

1 θn ,

if xmin > 0, xmax < θ

0,

otherwise.

Now, we can compute the pdf of U. F (u) = P (U ≤ u) = P (X1 , . . . , Xn ≤ u) =

n 7 i=1

=

n 7 i=1

P (Xi ≤ u) (because of independence) ⎛ u ⎞ 1 un ⎝ dx⎠ = n , 0 < u < θ. θ θ 0

The pdf of U may now be obtained as f (u) =

nun−1 d F (u) = , du θn

0 0 otherwise

f (X1 , . . . , Xn |U ) is a function of u and xmin which is independent of θ. Hence, U = max Xi is sufficient 1≤i≤n

for θ.

The outcome X1 , . . . , Xn is always sufﬁcient, but we will exclude this trivial statistic from consideration. In the previous two examples, we were given a statistic and asked to check whether it was sufﬁcient. It can often be tedious to check whether a statistic is sufﬁcient for a given parameter based directly on the foregoing deﬁnition. If the form of the statistic is not given, how do we guess what is the sufﬁcient statistic? Now think of working out the conditional probability by hand for each of our guesses! In general, this will be a tedious way to go about ﬁnding sufﬁcient statistics. Fortunately, the Neyman–Fisher factorization theorem makes it easier to spot a sufﬁcient statistic. The following result will give us a convenient way of verifying sufﬁciency of a statistic through the likelihood function. NEYMAN–FISHER FACTORIZATION CRITERIA Theorem 5.4.3 Let U be a statistic based on the random sample X1 , . . . , Xn . Then, U is a sufﬁcient statistic for θ if and only if the joint pdf (or pf ) f (x1 , . . . , xn ; θ) (which depends on the parameter θ) can be factored into two nonnegative functions. f (x1 , . . . , xn ; θ) = g (u, θ) h (x1 , . . . , xn ) ,

for all x1 , . . . , xn ,

where g (u, θ) is a function only of u and θ and h (x1 , . . . , xn ) is a function of only x1 , . . . , xn and not of θ.

Proof. (Discrete case.) We will only give the proof in the discrete case, even though the result is also true for the continuous case. First suppose that U (X1 , . . . , Xn ) is sufﬁcient for θ. Then, X1 = x1 , X2 = x2 , . . . , Xn = xn if and only if X1 = x1 , X2 = x2 , . . . , Xn = xn and U (X1 , . . . , Xn ) = U (x1 , . . . , xn ) = u(say). Therefore f (x1 , . . . , xn ; θ) = Pθ X1 = x1 , X2 = x2 , . . . , Xn = xn and U = u = Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u ) Pθ (U = u) .

5.4 Some Desirable Properties of Point Estimators 255

Because U is assumed to be sufﬁcient for θ, the conditional probability Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u ) does not depend on θ. Let us denote this conditional probability by h(x1 , . . . , xn ). Clearly Pθ (U = u) is a function of u and θ. Let us denote this by g(u, θ). It now follows from the equation above that f (x1 , . . . , xn ; θ) = g (u, θ) h (x1 , . . . , xn )

as was to be shown. To prove the converse, assume that f (x1 , . . . , xn ; θ) = g (u, θ) h (x1 , . . . , xn ) .

Deﬁne the set Au by Au = {(x1 , . . . , xn ) : U (x1 , . . . , xn ) = u} .

That is, Au is the set of all (x1 , . . . , xn ) such that U maps it into u. We note that Au does not depend on θ. Now Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u) Pθ X1 = x1 , X2 = x2 , . . . , Xn = xn and U = u = Pθ (U = u) ⎧ ⎨ Pθ (X1 =x1 ,X2 =x2 ,...,Xn =xn and U=u) , if (x1 , . . . , xn ) ∈ Au Pθ (U=u) = ⎩ 0, if (x1 , . . . , xn ) ∈ / Au .

/ Au , then, clearly, If (x1 , . . . , xn ) ∈ f (x1 , . . . , xn ; θ) = Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u )

which is independent of θ. If (x1 , . . . , xn ) ∈ Au , then, using the factorization criterion, we obtain Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u ) =

Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn ) Pθ (U = u)

=

f (x1 , . . . , xn ; θ) = Pθ (U = u)

g (u, θ) h (x1 , . . . , xn ) g (u, θ) h (x1 , . . . , xn ) (x1 ,...,xn )∈Au

=

g (u, θ) h (x1 , . . . , xn ) = g (u, θ) h (x1 , . . . , xn ) (x1 ,...,xn )∈Au

h (x , . . . , xn ) 1 h (x1 , . . . , xn ) (x1 ,...,xn )∈Au

256 CHAPTER 5 Point Estimation

Therefore, the conditional distribution of X1 , . . . , Xn given U does not depend on θ, proving that U is sufﬁcient. One can use the following procedure to verify that a given statistic is sufﬁcient. This procedure is based on factorization criteria rather than using the deﬁnition of sufﬁciency directly.

PROCEDURE TO VERIFY SUFFICIENCY 1. Obtain the joint pdf or pf fθ (x1 , . . . , xn ). 2. If necessary, rewrite the joint pdf or pf in terms of the given statistic and parameter so that one can use the factorization theorem. 3. Deﬁne the functions g and h, in such a way that g is a function of the statistic and parameter only and h is a function of the observations only. 4. If step 3 is possible, then the statistic is sufﬁcient. Otherwise, it is not sufﬁcient.

In general, it is not easy to use the factorization criterion to show that a statistic U is not sufﬁcient. We now give some examples using the factorization theorem.

Example 5.4.8 Let X1 , . . . , Xn denote a random sample from a geometric population with parameter p. Show that X is sufﬁcient for p.

Solution For the geometric distribution, the pf is given by ⎧ ⎨ p (1 − p)x−1 , f (x, p) = ⎩ 0,

x≥1 otherwise.

Hence, the joint pf is f (x1 , . . . , xn ; p) = pn (1 − p) =

−n+

n

xi

i=1

⎧ ⎨pn (1 − p)nx−n , ⎩

0,

if x1 , . . . , xn ≥ 1 otherwise.

Take, g(x, p) = pn (1 − p)nx−n

Thus, X is sufficient for p.

and

h(x1 , . . . , xn ) =

⎧ ⎨1,

if

⎩0,

otherwise.

xi ≥ 1

5.4 Some Desirable Properties of Point Estimators 257

Example 5.4.9 Let X1 , . . . , Xn denote a random sample from a U (0, θ) with pdf ⎧ 1 ⎪ ⎨ , fθ (x) = θ ⎪ ⎩ 0,

0 < x < θ,

θ>0

otherwise.

Show that X(n)= max Xi is sufﬁcient for θ, using the factorization theorem. 1≤i≤n

Solution The likelihood function of the sample is ⎧ 1 ⎪ ⎨ , θn fθ (x1 , . . . , xn ) = ⎪ ⎩ 0,

if 0 < x1 , . . . , xn < θ, otherwise.

We can now write fθ (x1 , . . . ., xn ) as fθ (x1 , . . . , xn ) = h (x1 , . . . , xn ) g θ, x(n) , for all x1 , . . . , xn where h (x1 , . . . , xn ) =

⎧ ⎨ 1, ⎩

0,

if x1 , . . . , xn > 0 otherwise

and ⎧ ⎪ 1 ⎨ n, θ g θ; x(n) = ⎪ ⎩ 0,

if 0 < x(n) < θ, otherwise.

From the factorization theorem, we now conclude that X(n) is sufficient for θ. In the next definition, we introduce the concept of joint sufficiency.

Deﬁnition 5.4.5 Two statistics U1 and U2 are said to be jointly sufﬁcient for the parameters θ1 and θ2 if the conditional distribution of X1 , . . . , Xn given U1 and U2 does not depend on θ1 or θ2 . In general, the statistic U = (U1 , . . . , Un ) is jointly sufﬁcient for θ = (θ1 , . . . , θn ) if the conditional distribution of X1 , . . . , Xn given U is free of θ. Now we state the factorization criteria for joint sufﬁciency analogous to the single population parameter case.

258 CHAPTER 5 Point Estimation

THE FACTORIZATION CRITERIA FOR JOINT SUFFICIENCY Theorem 5.4.4 The two statistics U1 and U2 are jointly sufﬁcient for θ1 and θ2 if and only if the likelihood function can be factored into two non-negative functions, f (x1 , . . . , xn ; θ1 , θ2 ) = g(u1 , u2 ; θ1 , θ2 ) h(x1 , . . . , xn ) where g (u1 , u2 ; θ1 , θ2 ) is only a function of u1 , u2 ; θ1 and θ2 , and h(x1 , xn ) is free of θ1 or θ2 .

Example 5.4.10 Let X1 , . . . , Xn be a random sample from N(μ, σ 2 ). (a) If μ is unknown and σ 2 = σ02 is known, show that X is a sufﬁcient statistic for μ. (b) If μ = μ0 is known and σ 2 is unknown, show that ni=1 (Xi − μ0 )2 is sufﬁcient for σ 2 . (c) If μ and σ 2 are both unknown, show that ni=1 Xi and ni=1 Xi2 are jointly sufﬁcient for μ and σ 2 .

Solution The likelihood function of the sample is ⎡

⎤ 2 (X − μ) i ⎢ ⎥ 1 ⎢ i=1 ⎥ L= exp − ⎢ ⎥ ⎣ ⎦ 2σ 2 (2π)n/2 σ n n

n & n 1 2 2 exp xi − 2μ xi + nμ = 2σ 2 (2π)n/2 σ n i=1 i=1 ⎞ ⎛ n 2 x

⎜ i ⎟ 2μnx nμ2 ⎜ i=1 ⎟ −n/2 −n exp − 2 . σ exp ⎜− = (2π) ⎟ exp ⎝ 2σ 2 ⎠ 2σ 2 2σ 1

%

(a) When σ 2 = σ02 is known, use the factorization criteria, with 2nμx − nμ2 g(x, μ) = exp 2σ02 and

⎛

⎞ n 2 x ⎜ i ⎟ ⎜ i=1 ⎟ h(x1 , . . . , xn ) = (2π)−n/2 σ −n exp ⎜− ⎟. ⎝ 2σ 2 ⎠

Therefore, X is sufficient for μ.

5.4 Some Desirable Properties of Point Estimators 259

(b) When μ = μ0 is known, let

g

n

(Xi − μ)2 , σ 2

i=1

n 2 (xi − μ) i=1 = σ −n exp − 2σ 2

and h(x1 , . . . , xn ) = Thus, ni=1 (Xi − μ)2 is sufficient for σ 2 . (c) When both μ and σ 2 are unknown, use

g

n

xi ,

i=1

n

xi2 , μ, σ 2

i=1

1 . (2π)n/2

n n 2 2 x − 2μ x + nμ i i i=1 i=1 = σ −n exp − 2 2σ

and h(x1 , . . . , xn ) = Hence,

n

i=1 Xi and

1 . (2π)n/2

n

2 2 i=1 Xi are jointly sufficient for μ and σ .

Example 5.4.11 Suppose that we have a random sample X1 , . . . , Xn from a discrete distribution given by fθ (x) = C (θ) 2−x/θ ,

x = θ, θ + 1, θ + 2, . . . ;

θ>0

where C (θ) > 0 is a normalizing constant. Using the factorization theorem, ﬁnd a sufﬁcient statistic for θ.

Solution The joint density function f (x1 , . . . , xn ; θ) of the sample X1 , . . . , Xn is ⎧ n ⎪ ⎪ − (xi /θ) ⎨ i=1 , x1 , x2 , . . . , xn are integers ≥ θ f (x1 , . . . , xn ; θ) = C (θ) 2 ⎪ ⎪ ⎩ 0, otherwise . The function f (x1 , . . . , xn ; θ) can be written as −

f (x1 , . . . , xn ; θ) = h(x1 , . . . , xn ) C (θ) 2

n

(xi /θ)

i=1

g1 θ, x(1)

260 CHAPTER 5 Point Estimation

where x(1) = min (x1 , . . . , xn ), and i

h(x1 , x2 , . . . , xn ) =

⎧ ⎨1,

if xj − x(1) ≥ 0 is an integer for j = 1, 2, . . . , n

⎩0,

otherwise

and

⎧ ⎨1,

g1 θ, x(1) = ⎩0,

if x(1) ≥ θ otherwise.

Thus, xi , x(1) f (x1 , . . . , xn ; θ) = h(x1 , . . . , xn ) g θ, n

− (xi / θ) g1 θ, x(1) . Using the factorization theorem, we conclude that where g θ, xi , x(1) = C(θ)2 i=1 xi , x(1) is jointly sufﬁcient for θ. This result shows that even for a single parameter, we may need more than one statistic for sufﬁciency.

When using the factorization criterion, one has to be careful in cases where the range space depends on the parameter. Using the factorization criterion, we can prove the following result, which says that if we have a unique maximum likelihood estimator, then that estimator will be a function of the sufﬁcient statistic. Theorem 5.4.5 If U is a sufﬁcient statistic for θ, the maximum likelihood estimator of θ, if unique, is a function of U. Proof. Because U is sufﬁcient, by Theorem 5.4.1, the joint pdf can be factored as f (x1 , . . . , xn ; θ) = g(u, θ) h(x1 , . . . , xn ).

This depends on θ only through the statistic U. To maximize L we need to maximize g(U, θ). Many common distributions such as Poisson, normal, gamma, and Bernoulli are members of the exponential family of probability distributions. The exponential family of distributions has density functions of the form f (x; θ) =

⎧ ⎨exp [k(x)c(θ) + S(x) + d(θ)] , ⎩

where B does not depend on the parameter θ.

0,

if x ∈ B x∈ /B

5.4 Some Desirable Properties of Point Estimators 261

Example 5.4.12 Write the following in exponential form. e−λ λx x! (b) px (1 − p)1−x 1 2 (c) √ e−(x−μ) /2 2π (a)

Solution (a) We have e−λ λx = exp [x ln λ − ln x! − λ] . x! Here k(x) = x, c (λ) = ln λ, S(x) = − ln (x!), and d (λ) = −λ. (b) Similarly, !

px (1 − p)1−x = exp x ln

p 1−p

" + ln (1 − p) ,

x = 0 or 1.

(c) This is the standard normal density. 2 2 1 2 √ e−(x−μ) /2 = exp xμ − x2 − μ2 − 12 ln (2π) , 2π

−∞ < x < ∞.

Note that in the previous example, for each of the cases, ni=1 Xi is a sufﬁcient statistic for the parameter. In the next result, we give a generalization of this fact. Theorem 5.4.6 Let X1 , . . . , Xn be a random sample from a population with pdf or pmf of the exponential form f (x; θ) =

exp [k(x)c (θ) + S(x) + d (θ)] , 0,

where B does not depend on the parameter θ. The statistic

n

i=1 k (Xi )

if x ∈ B x∈ /B

is sufﬁcient for θ.

Proof. The joint density % f (x1 , . . . , xn ; θ) = exp c (θ)

n

k (xi ) +

i=1

% = exp c (θ)

n

& S (xi ) + nd (θ)

i=1

n

&4 k (xi ) + nd (θ)

exp

i=1

Using the factorization theorem, the statistic

n

i=1 k (Xi )

% n i=1

is sufﬁcient.

&4 S (xi )

.

262 CHAPTER 5 Point Estimation

It does not follow that every function of a sufﬁcient statistic is sufﬁcient. However, any one-to-one function of a sufﬁcient statistic is also sufﬁcient. Every statistic need not be sufﬁcient. When they do exist, sufﬁcient estimators are very important, because if one can ﬁnd a sufﬁcient estimator it is ordinarily possible to ﬁnd an unbiased estimator based on the sufﬁcient statistic. Actually, the following theorem shows that if one is searching for an unbiased estimator with minimal variance, it has to be restricted to functions of a sufﬁcient statistics. RAO–BLACKWELL THEOREM Theorem 5.4.7 Let X1 , . . . , Xn be a random sample with joint pf or pdf f (x1 , . . . , xn ; θ) and let U = (U1 , . . . , Un ) be jointly sufﬁcient for θ = (θ1 , . . . , θn ). If T is any unbiased estimator of k (θ), and if T ∗ = E (T |U ), then: (a) T ∗ is an unbiased estimator of k(θ). (b) T ∗ is a function of U, and does not depend on θ. (c) Var T ∗ ≤ Var(T ) for every θ, and Var T ∗ < Var(T ) for some θ unless T ∗ = T with probability 1.

Proof. (a) By the property of conditional expectation and by the fact that T is an unbiased estimator of k(θ), E T ∗ = E(E(T |U)) = E(T ) = k(θ).

Hence, T ∗ is an unbiased estimator of k(θ). (b) Because U is sufﬁcient for θ, the conditional distribution of any statistic (hence, for T ), given U, does not depend on θ. Thus, T ∗ = E(T |U) is a function of U. (c) From the property of conditional probability, we have the following: Var (T ) = E (Var (T |U )) + Var (E (T |U )) = E (Var (T |U )) + Var T ∗ .

Because Var (T |U ) ≥ 0 for all u, it follows that E (Var (T |U )) ≥ 0. Hence, Var T ∗ ≤ Var(T ). We ∗ note that Var T = Var(T ) if and only if Var (T |U) = 0 or T is a function of U, in which case T ∗ = T (from the deﬁnition of T ∗ = E (T |U ) = T ). In particular, if k (θ) = θ, and T is an unbiased estimator of θ, then T ∗ = E (T |U ) will typically give the MVUE of θ. If T is the sufﬁcient statistic that best summarizes the data from a given distribution with parameter θ, and we can ﬁnd some function g of T such that E (g (T )) = θ, it follows from the Rao–Blackwell theorem that g(T ) is the UMVUE for θ.

EXERCISES 5.4 5.4.1.

Let X1 , . . . , Xn be a random sample from a population with density f (x) =

e−(x−θ) , 0,

for x > θ otherwise.

5.4 Some Desirable Properties of Point Estimators 263

(a) Show that X is a biased estimator of θ. (b) Show that X is an unbiased estimator of μ = 1 + θ. 5.4.2.

The mean and variance of a ﬁnite population {a1 , . . . , aN } are deﬁned by μ=

N N 1 1 (ai − μ)2 . ai and σ 2 = N N i=1

i=1

For a ﬁnite population, show that the sample variance S 2 is a biased estimator of σ 2 . 5.4.3.

For an inﬁnite population with ﬁnite variance σ 2 , show that the sample standard deviation S is a biased estimator for σ. Find an unbiased estimator of σ. [We have seen that S 2 is an unbiased estimator of σ 2 . From this exercise, we see that a function of an unbiased estimator need not be an unbiased estimator.]

5.4.4.

Let X1 , . . . , Xn be a random sample from an inﬁnite population with ﬁnite variance σ 2 . Deﬁne S 2 =

n 2 1 Xi − X . n i=1

2

Show that S 2 is a biased estimator for σ 2 , and that the bias of S 2 is − σn . Thus, S 2 is negatively biased, and so on average underestimates the variance. Note that S 2 is the MLE of σ 2 . 5.4.5.

Let X1 , . . . , Xn be a random sample from a population with the mean μ. What condition must be imposed on the constants c1 , c2 , . . . , cn so that c1 X1 + c2 X2 + · · · + cn Xn

is an unbiased estimator of μ? 5.4.6.

Let X1 , . . . , Xn be a random sample from a geometric distribution with parameter θ. Find an unbiased estimate of θ.

5.4.7.

Let X1 , . . . , Xn be a random sample from U (0, θ) distribution. Let Yn = max{X1 , . . . , Xn }. We know (from Example 5.3.4) that θˆ 1 = Yn is a maximum likelihood estimator of θ. (a) Show that θˆ 2 = 2X is a method of moments estimator. (b) Show that θˆ 1 is a biased estimator, and θˆ 2 is an unbiased estimator of θ. (c) Show that θˆ 3 =

5.4.8. 5.4.9.

n+1 ˆ n θ1

is an unbiased estimator of θ.

Let X1 , . . . , Xn be a random sample from a population with mean μ and variance 1. Show 2 that μ ˆ 2 = X is a biased estimator of μ2 , and compute the bias. Let X1 , . . . , Xn be a random sample from an N μ, σ 2 distribution. Show that the estimator μ ˆ = X is the MVUE for μ.

264 CHAPTER 5 Point Estimation

5.4.10.

Let X1 , . . . , Xn1 be a random sample from an N μ1 , σ 2 distribution and let Y1 , . . . , Yn2 be a random sample from a N μ2 , σ 2 distribution. Show that the pooled estimator σˆ 2 =

5.4.11.

(n1 − 1) S12 + (n2 − 1) S22 n1 + n 2 − 2

is unbiased for σ 2 , where S12 and S22 are the respective sample variances. Let X1 , . . . , Xn be a random sample from an N μ, σ 2 distribution. Show that the sample median, M, is an unbiased estimator of the population mean μ. Compare the variances of X and M. [Note: For the normal distribution, the mean, median, and mode all occur at the same location. Even though both X and M are unbiased, the reason we usually use the mean instead of the median as the estimator of μ is that X has a smaller variance than M.]

5.4.12.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. Show that the sample mean X is sufﬁcient for λ.

5.4.13.

Let X1 , . . . , Xn be a random sample from a population with density function fσ (x) =

|x| 1 exp − , 2σ σ

−∞ < X < ∞,

σ > 0.

Find a sufﬁcient statistic for the parameter σ. 5.4.14.

Show that if θˆ is a sufﬁcient statistic for the parameter θ and if the maximum likelihood estimator of θ is unique, then the maximum likelihood estimator is a function of this sufﬁcient statistic θˆ .

5.4.15.

Let X1 , . . . , Xn be a random sample from an exponential population with parameter θ. (a) Show that ni=1 Xi is sufﬁcient for θ. Also show that X is sufﬁcient for θ. (b) The following is a random sample from exponential distribution. 1.5 0.3 5.7

3.0 2.0 0.1

2.6 1.8 0.2

6.8 1.0 0.5

0.7 0.7 0.4

2.2 0.7

1.3 1.6

1.6 3.0

1.1 2.0

6.5 2.5

(i) What is an unbiased estimate of the mean? (ii) Using part (a) and these data, ﬁnd two sufﬁcient statistics for the parameter θ. 5.4.16.

Let X1 , . . . , Xn be a random sample from a one-parameter Weibull distribution with pdf f (x) =

⎧ ⎨2αxe−αx2 , ⎩

(a) Find a sufﬁcient statistic for α. (b) Using part (a), ﬁnd an UMVUE for α.

0,

x>0 otherwise.

5.4 Some Desirable Properties of Point Estimators 265

5.4.17.

Let X1 , . . . , Xn be a random sample from a population with density function ⎧ 1 ⎪ ⎨ , f (x) = θ ⎪ ⎩ 0,

Show that 5.4.18.

−

θ θ ≤x≤ ,θ>0 2 2 otherwise.

min Xi , max Xi is sufﬁcient for θ.

1≤i≤n

1≤i≤n

Let X1 , . . . , Xn be a random sample from a G(1, β) distribution. (a) Show that U = ni=1 Xi is a sufﬁcient statistic for β. (b) The following is a random sample from a G(1, β) distribution. 0.3 0.3

3.4 3.7

0.4 0.1

1.8 1.3

0.7 1.2

1.0 3.3

0.1 0.2

2.3 1.3

3.7 0.6

2.0 0.4

Find a sufﬁcient statistic for β. 5.4.19.

Show that X1 is not sufﬁcient for μ, if X1 , . . . , Xn is a sample from N(μ, 1).

5.4.20.

Let X1 , . . . , Xn be a random sample from the truncated exponential distribution with pdf ⎧ ⎨eθ−x ,

f (x) =

⎩ 0,

x>θ otherwise.

Show that X(1) = min(Xi ) is sufﬁcient for θ. 5.4.21.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f (x) =

⎧ ⎨θxθ−1 , ⎩

0 < x < 1,

0,

θ>0

otherwise.

Show that U = X1 , . . . , Xn is a sufﬁcient statistic for θ. 5.4.22.

Let X1 , . . . , Xn be a random sample of size n from a Bernoulli population with parameter p. Show that pˆ = X is the UMVUE for p.

5.4.23.

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

Show that

n

2 i=1 Xi

⎧ ⎨ 2x e−x2 /α ,

x>0

α

⎩

0,

otherwise.

is sufﬁcient for the parameter α.

266 CHAPTER 5 Point Estimation

5.5 OTHER DESIRABLE PROPERTIES OF A POINT ESTIMATOR In this section, we discuss a few more properties of point estimators that can be used in choosing a particular estimator.

5.5.1 Consistency It is a desirable property that the values of an estimator be closer to the value of the true parameter being estimated as the sample size becomes larger. To this end, we now introduce the notion of consistent estimators. Consistency is a large-sample, or asymptotic, property. That is, it describes the behavior of estimators as the sample size n becomes inﬁnitely large. In this section, we use the notation θˆ n for θˆ to show the dependence of the estimator on the sample size n. Deﬁnition 5.5.1 The estimator θˆ n is said to be a consistent estimator of θ if, for any ε > 0, lim P ˆθn − θ ≤ ε = 1

n→∞

or equivalently, lim P ˆθn − θ > ε = 0. n→∞

The statement “θˆ n is a consistent estimator of θ” is equivalent to “θˆ n converges in probability to θ.” That is, the sample estimator should have a high probability of being close to the population value θ for large sample size n. The idea of consistency can be observed in Figure 5.2, where θˆ n converges to θ. If it did not, θˆ n would not be a consistent estimator of θ. If the estimator is unbiased, we have the following result, which gives a sufﬁcient condition for the consistency of an estimator. However, it is important to note that a consistent estimator need not be unbiased, and hence this result is not a necessary condition.

A SUFFICIENT CONDITION FOR CONSISTENCY OF AN UNBIASED ESTIMATOR Theorem 5.5.1 An unbiased estimator θˆ n of θ is a consistent estimator for θ if lim Var θˆ n = 0. n→∞

x n

■ FIGURE 5.2 Consistency of an estimator.

5.5 Other Desirable Properties of a Point Estimator 267

The proof of this theorem follows directly from Chebyshev’s inequality. A general version of this result is proved in Theorem 5.5.3.

Example 5.5.1 Let X1 , . . . , Xn be a random sample with true mean μ and ﬁnite variance. Then, the sample mean X is a consistent estimator of the population mean μ.

Solution We show this result in two ways. , we obtain (i) Using Chebyshev’s inequality, P{|X − μ| ≥ ε} ≤ Var(x) ε2 σ2 # $ P X − μ ≤ k ≥ 1 − X k2 σ2 = 1 − 2 → 1as n → ∞. k n Hence, X is a consistent estimator of μ. (ii) First note that X is an unbiased estimator of μ. Because Var X = σ 2 /n , we have σ2 = 0. n→∞ n lim

Thus, from the previous theorem, X is a consistent estimator of μ.

We can generalize Theorem 5.5.1 even when the estimator is biased. The following result states that the mean square error of θˆ n decreases to zero as more and more observations are incorporated into its computation. TEST FOR CONSISTENCY Theorem 5.5.2 Let θˆ n be an estimator of θ and let Var θˆ n be ﬁnite. If lim E

n→∞

2 θˆ n − θ =0

then θˆ n is a consistent estimator of θ.

Proof. Using Chebyshev’s inequality, we obtain E P ˆθn − θ ≥ ε ≤

Because lim E

n→∞

2 θˆ n − θ

s2

.

2 θˆ n − θ = 0, [by hypothesis]

268 CHAPTER 5 Point Estimation

the right-hand side converges to zero. Thus, lim P ˆθn − θ ≥ ε = 0. n→∞

Consequently θˆ n is a consistent estimator of θ.

Furthermore, we know that E

2 2 = Var θˆ n + B θˆ n θˆ n − θ ,

and for unbiased estimators, the bias B θˆ n is zero. As a result, Theorem 5.5.1 is a particular case of Theorem 5.5.3. We now summarize the procedure for testing for consistency of an estimator as follows:

PROCEDURE TO TEST FOR CONSISTENCY 1. Check whether the estimator θˆ n is unbiased or not. 2. Calculate Var θˆ n and B θˆ n , the bias of θˆ n . 3. An unbiased estimator is consistent if Var θˆ n → 0 as n → ∞. 4. A biased estimator is consistent if both Var θˆ n → 0 and B θˆ n → 0 as n → ∞.

Example 5.5.2

Let X1 , . . . , Xn be a random sample from N μ, σ 2 population. (a) Show that the sample variance S 2 is a consistent estimator for σ 2 . (b) Show that the maximum likelihood estimators for μ and σ 2 are consistent estimators for μ and σ 2 .

Solution (a) We have already seen that ES 2 = σ 2 , and hence, S 2 is an unbiased estimator of σ 2 . Because $ # the sample is drawn from a normal distribution, we know that (n − 1) S 2 /σ 2 has a chi-square distribution with (n − 1) d.f. and (n − 1) S 2 Var = 2 (n − 1). σ2 Thus,

2 (n − 1) = Var

(n − 1) S 2 σ2

=

(n − 1)2 Var S 2 . σ4

5.5 Other Desirable Properties of a Point Estimator 269

This implies that 2σ 4 Var S 2 = → 0 as n → ∞. n−1 Hence, S 2 is a consistent estimator of the variance of a normal population. 2 ˆ is (b) We have seen that the MLE of μ is μ ˆ = X, and that of σ 2 is σˆ n2 = (1/n) ni=1 Xi − X . Now μ an unbiased estimator of μ, and Var( X ) = (σ 2 /n) → 0 as n → ∞. Therefore, from Theorem 5.5.1, X is a consistent estimator for μ. Now we will use the identity 2 2 = Var θˆ n + B θˆ n E θˆ n − θ to show that the MLE for σ 2 is biased with n−1 2 σ E σˆ n2 = n and n−1 2 1 σ − σ2 = − σ2. B σˆ n2 = n n Thus, σˆ n2 = (1/n)

n

i=1 Xi − X

2

= ((n − 1) /n) S 2 . Using part (a), we get (n − 1)2 Var S 2 Var σˆ n2 = 2 n 2 2 4 2(n − 1) σ 2 (n − 1) 2σ = . = n2 (n − 1) n2

Therefore, −σ 2 = 0, and lim Var σˆ n2 lim B σˆ n2 = lim n→∞ n→∞ n n→∞ 2 2(n − 1) σ 2 = lim = 0. n→∞ n2 By Theorem 5.5.3, σˆ n2 =

n 2 1 Xi − X n i=1

is a consistent estimator of σ 2 .

From the foregoing example we can see that consistent estimators need not be unique. It turns out that most of the MLEs and method of moments estimators derived for important probability distributions are consistent.

270 CHAPTER 5 Point Estimation

5.5.2 Efficiency We have seen that there can be more than one unbiased estimator for a parameter θ. We have also mentioned that the one with the least variance is desirable. Here, we introduce the concept of efﬁciency, which is based on comparing variances of the different unbiased estimators. If there are two unbiased estimators, it is desirable to have the one with a smaller variance. Deﬁnition 5.5.2 If θˆ 1 and θˆ 2 are two unbiased estimators for θ, the efﬁciency of θˆ 1 relative to θˆ 2 is the ratio Var θˆ 2 e θˆ 1 , θˆ 2 = . Var θˆ 1

If Var θˆ 2 > Var θˆ 1 , or equivalently, e θˆ 1 , θˆ 2 > 1, then, θˆ 1 is relatively more efficient than θˆ 2 . That is θˆ 1 has a smaller variance as compared to the variance of θˆ 2 . We summarize the following procedure to compare the efﬁciencies of the different unbiased estimators.

PROCEDURE TO TEST RELATIVE EFFICIENCY 1. Check for unbiasedness of θˆ 1 and θˆ 2 . 2. Calculate the variances of θˆ 1 and θˆ 2 . 3. Calculate the relative efﬁciency as

Var θˆ 2 e θˆ 1 ; θˆ 2 = . Var θˆ 1

4. Conclusion: If e θˆ 1, θˆ 2 < 1, θˆ 2 is more efﬁcient than θˆ 1 , and if e θˆ 1 , θˆ 2 > 1, then, θˆ 1 is more efﬁcient than θˆ 2 . Among the unbiased estimators, the more efﬁcient estimator is preferable.

Example 5.5.3 Let X1 , . . . , Xn , n > 3, be a random sample from a population with a true mean μ and variance σ 2 . Consider the following three estimators of μ: θˆ 1 =

1 (X1 + X2 + X3 ) , 3

θˆ 2 =

1 1 3 X1 + X2 + · · · + Xn−1 + Xn , 8 4 (n − 2) 8

and θˆ 3 = X. (a) Show that each of the three estimators is unbiased. (b) Find e θˆ 2 , θˆ 1 , e θˆ 3 , θˆ 1 , and e θˆ 3 , θˆ 2 . Which of the three estimators is more efﬁcient?

5.5 Other Desirable Properties of a Point Estimator 271

Solution (a) Given E(Xi ) = μ, i = 1, 2, . . . , n. Then, 1 3μ =μ E θˆ 1 = [E(X1 ) + E(X2 ) + E(X3 )] = 3 3 1 1 3 E(X2 ) + · · · + E Xn−1 + E(Xn ) E θˆ 2 = E(X1 ) + 8 4 (n − 2) 8 =

3 1 1 (n − 2) μ + μ = μ μ+ 8 4 (n − 2) 8

E θˆ 3 = E X = μ. Hence, θˆ 1 , θˆ 2 , and θˆ 3 are unbiased estimators of μ. (b) Computing the variances, we have 1 Var θˆ 1 = (Var(X1 ) + Var(X2 ) + Var(X3 )) 9 =

1 2 σ2 3σ = . 9 3

σ2 9 (n − 2) σ 2 σ2 Var θˆ 2 = + + 64 64 16 (n − 2)2 =

9σ 2 n + 16 2 2σ 2 + = σ . 64 16 (n − 2) 32 (n − 2)

σ2 Var θˆ 3 = . n The relative efficiencies are

Var θˆ 2 σ 2 (n + 16) /32 (n − 2) e θˆ 1 , θˆ 2 = = σ 2 /3 Var θˆ 1 =

3 (n + 16) < 1 for n > 3. 32 (n − 2)

Thus, for n ≥ 4, θˆ 2 is more efficient than θˆ 1 . Var θˆ 1 n σ 2 /3 ˆ ˆ e θ3 , θ1 = = 2 = > 1 for n ≥ 4. ˆ 3 σ /n Var θ3

272 CHAPTER 5 Point Estimation

Hence, for n > 3, θˆ 3 is more efficient than θˆ 1 . n+16 2 Var θˆ 2 32(n−2) σ e θˆ 3 , θˆ 2 = = σ 2 /n Var θˆ 3 =

n2 + 16n > 1 for n ≥ 4. 32 (n − 2)

Therefore, even though both θˆ 3 θˆ 2 are based on all the n observations, for n > 3, the sample mean θˆ 3 is more efficient than θˆ 2 .

It is reasonable to compare estimators on the basis of variance alone if they are both unbiased. To facilitate the cases where the estimators are biased, we use the mean square error (MSE) in the deﬁnition of relative efﬁciency. Deﬁnition 5.5.3 An estimator θˆ 1 is more efﬁcient than θˆ 2 if MSEθˆ 1 ≤ MSEθˆ 2

with strict inequality for some θ. Also, the relative efﬁciency of θˆ 1 with respect to θˆ 2 is 2 E θˆ 2 − θ MSE θˆ 2 ˆ ˆ e θ1 , θ2 = . 2 = MSE θˆ 1 E θˆ 1 − θ

Example 5.5.4 Let X1 , . . . , Xn , n ≥ 2 be a random sample from a normal population with a true mean μ and variance σ 2 . Consider the following two estimators of σ 2 : θˆ 1 = S 2 , and θˆ 2 = S 2 . Find e θˆ 1 , θˆ 2 .

Solution

2 2 (n − 1), E(S 2 ) = σ 2 , and MSE(S 2 ) = Var(S 2 ). Also, 2(n − 1) = Var (n−1)S 2 = Because (n−1)S ∼ χ 2 2 σ σ (n−1)2 Var(S 2 ). σ4

Thus, MSE θˆ 1 =

2 σ4. n−1

Also, it can be shown that (2n − 1) 2 MSE S 2 = σ . n2

5.5 Other Desirable Properties of a Point Estimator 273

Thus, the relative efficiency of θˆ 1 with respect to θˆ 2 is MSE θˆ 2 MSE S 2 ˆ ˆ e θ1 , θ2 = = MSE S 2 MSE θˆ 1

=

(2n−1) 2 σ (2n − 1) (n − 1) n2 = . 2 σ2 2n2 (n−1)

For n ≥ 2, it can be seen that e θˆ 1 , θˆ 2 < 1. Hence, S 2 is relatively more efficient than S 2 .

We have seen that it is possible that one unbiased estimator is more efﬁcient than another. This leads to the possibility of having one unbiased estimator more efﬁcient than all the other unbiased estimators. This directs us to the following deﬁnition. Deﬁnition 5.5.4 An unbiased estimator θˆ 0 , is said to be a uniformly minimum variance unbiased estimator (UMVUE) for the parameter θ if, for any other unbiased estimator θˆ Var θˆ 0 ≤ Var θˆ ,

for all possible values of θ. It is not always easy to ﬁnd an UMVUE for a parameter. However, the following result gives a lower bound for the variance of any unbiased estimator.

CRAMÉR–RAO INEQUALITY Theorem 5.5.3 Let X1 , . . . , Xn be a random sample from a population with pdf (or pf ) fθ (x) that depends on a parameter θ. If θˆ is an unbiased estimator of θ, then, under very general conditions, the following inequality is true: Var θˆ ≥

! nE

1

∂ ln fθ (x) 2 ∂θ

".

If θˆ is an unbiased estimator of ψ(θ), then Var θˆ ≥

∂ψ(θ) 2 ∂θ

. ∂ ln f (x) 2 nE ∂θ θ

274 CHAPTER 5 Point Estimation

If L(θ) is the likelihood function, we can rewrite the Cramér–Rao inequality in the form Var θˆ ≥

! E

1

∂ ln L(θ) 2 ∂θ

"

From the Cramér–Rao inequality, we can obtain the following result.

EFFICIENT ESTIMATOR Theorem 5.5.4 If θˆ is an unbiased estimator of θ and if Var θˆ =

! nE

1

∂ ln fθ (x) 2 ∂θ

",

then θˆ is a uniformly minimum variance unbiased estimator (UMVUE) of θ. Sometimes θˆ is also referred to as an efﬁcient estimator.

Note that if the function f (.) is sufﬁciently smooth, it can be shown that ∂ ln fθ (x) 2 ∂2 ln fθ (x) = Var[ln fθ (x)] . E = −E ∂θ ∂θ 2

Hence, the Cramér–Rao inequality in this case can be rewritten as Var θˆ ≥

1

2 −nE ∂ ln∂θf2θ (x)

=

1

.

∂ ln f (x) nVar ∂θ θ

Now, we will give a procedure to apply the Cramér–Rao inequality.

CRAMÉR–RAO PROCEDURE TO TEST FOR EFFICIENCY 2

1. For the pdf (or pf), ﬁnd ∂ lnf∂θ(x) and ∂ ln f2(x) . ∂θ ! ! 2 2 "" 2. Calculate (1/n) E − ∂ ln f2(x) if f (x) is smooth, or else calculate 1/nE ∂ ln∂θf (x) . ∂θ

3. Calculate Var (θˆ ). 4. If the result of step 2 is equal to the result of step 3, then, θˆ is efﬁcient for θ.

Example 5.5.5 Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population with density function f (x). Show that X is an efﬁcient estimator for μ.

5.5 Other Desirable Properties of a Point Estimator 275

Solution To calculate the Cramér–Rao lower bound, we have ln f (x) = c −

(x − μ)2 , 2σ 2

where c is a constant not involving μ. Then ∂ ln f (x) x−μ = ∂μ σ2 and ∂2 ln f (x) 1 =− 2 ∂μ2 σ or 1

% nE −

∂2 ln f (x) ∂θ 2

1

& = nE

1 σ2

=

σ2 = Var X . n

Therefore, X is an efficient estimator of μ. That is, X is an UMVUE of μ.

Example 5.5.6 Suppose p(x) is the Poisson distribution with parameter λ. Show that the sample mean Xn is an efﬁcient estimator for λ.

Solution

−λ

Here the density function is given by p(x) = λx ex! . Taking logarithms, ln p(x) = x ln λ − λ − ln(x!) ∂ ln p(x) x = − 1, ∂λ λ and x ∂2 ln p(x) =− 2 ∂λ2 λ Therefore, using the fact that the expected value of a Poisson r.v. is λ,

1

2 f (x) nE − ∂ ln ∂λ2

=

1 λ = = Var X . n nE X λ2

Hence, X is an efficient estimator of λ.

Example 5.5.7 Let X1 , . . . , Xn be a random sample from a Bernoulli trial with probability of success p. Show that the maximum likelihood estimator is also an efﬁcient estimator.

276 CHAPTER 5 Point Estimation

Solution

Note that the MLE of p is pˆ = (1/n) ni=1 Xi = X/n, the fraction of successes in the total number of trials, n. Because we can view n Bernoulli trials as being a single observation from a binomial distribution with parameters n and p, the likelihood function is n x L(p) = p (1 − p)x . x Then, n + x ln p + (n − x) ln(1 − p). x

ln L(p) = ln Now

∂ ln L(p) x n−x x − np = − = . ∂p p 1−p p(1 − p) Hence,

% E

% & & x − np 2 ∂ ln L(p) 2 =E ∂p p (1 − p) = =

Var(x) [p(1 − p)]2 np(1 − p)

n . [p(1 − p)]2 p(1 − p)

Therefore, the Cramér– Rao bound is ! E Now

1

∂ ln L(p) 2 ∂p

" =

p(1 − p) . n

X Var pˆ = Var n 1 = 2 Var(x) n 1 p(1 − p) = 2 np(1 − p) = . n n

Because the variance of the estimator is equal to the Cramér–Rao lower bound, we conclude that pˆ = X n is an efficient estimator of p.

It is important to note that an UMVUE may not exist for a given problem. Even when an UMVUE exists, it is not necessary that it have a variance equal to the Cramér–Rao lower bound. The term

5.5 Other Desirable Properties of a Point Estimator 277

I(θ) = E

!

∂ ln f (x) ∂θ

2 "

is called the Fisher information. In fact, for a random sample of size n with ! 2 " ∂ ln L(θ) likelihood function L(θ), the Fisher information is deﬁned as In (θ) = E . It can be ∂θ shown that the Fisher information in a sample of size n is n times the Fisher information in one observation. That is, In (θ) = nI(θ).

5.5.3 Minimal Sufficiency and Minimum-Variance Unbiased Estimation In the study of statistics, it is desirable to reduce the data contained in the sample as much as possible without losing relevant information. Our objective is to ﬁnd minimal sufﬁcient statistics and use them to develop uniformly minimum variance unbiased estimators (UMVUEs) for true parameters. Whenever sufﬁcient statistics exist, then a statistician with those summary measures is as well off as the statistician with the entire sample, for point estimation purposes. Minimal sufﬁcient statistics are those that are sufﬁcient for the parameters and are functions of every other set of sufﬁcient statistics for those same parameters. Deﬁnition 5.5.5 A sufﬁcient statistic T (X) is called a minimal sufﬁcient statistic if for any other statistic T (X), T (X) is a function of T (X). That is, T (X) = g T (X) .

Using this deﬁnition, it is difﬁcult to determine whether a set of statistics is, in fact, minimal sufﬁcient. Now we will present a method due to Lehmann and Scheffé that will be of great help in ﬁnding a minimal sufﬁcient statistic. We can summarize the Lehmann and Scheffé method to ﬁnd a minimal sufﬁcient statistic as follows. Let X1 , . . . , Xn be a random sample with pdf or pmf f (x) that depends on a parameter θ. Let (x1 , . . . , xn ) and (y1 , . . . , yn ) be two different sets of values of (X1 , . . . , Xn ). Let L (θ; x1 , . . . , xn ) L (θ; y1 , . . . , yn )

be the ratio of the likelihoods evaluated at these two points. Suppose it is possible to ﬁnd a function g(x1 , . . . , xn ) such that this ratio will be free of the unknown parameter θ if and only if g(x1 , . . . , xn ) = g(y1 , . . . , yn ). If such a function g can be found, then g(X1 , . . . , Xn ) is a minimal sufﬁcient statistic for θ.

Example 5.5.8 Let X1 , . . . , Xn be a random sample from the Bernoulli distribution where P (Xi = 1) = p and P (Xi = 0) = 1 − p, with p unknown. Find a minimal sufﬁcient statistic for p.

278 CHAPTER 5 Point Estimation

Solution The ratio of the likelihoods is

p(x1 , . . . , xn ) p xi (1 − p)n− xi L(x1 , . . . , xn ) = = L(y1 , . . . , yn ) p(y1 , . . . , yn ) p yi (1 − p)n− yi

xi − yi p = . 1−p This ratio is to be independent of p, if and only if n

xi −

i=1

n

yi = 0

i=1

which implies n i=1

xi =

n

yi .

i=1

Therefore, g (X1 , . . . , Xn ) =

n

Xi

i=1

is a minimal sufficient statistic for p.

Example 5.5.9 Let X1 , . . . , Xn be a random sample from a U(0, θ) distribution. Find a minimal sufﬁcient statistic for θ.

Solution The likelihood function is

⎧ ⎨ 1, θn L= ⎩ 0,

if max(x1 , . . . , xn ) ≤ θ otherwise .

Denote by xmax = max(x1 , . . . , xn ), and ymax = max(y1 , . . . , yn ). Then, the ratio of the likelihood functions is ⎧ ⎪ ⎪ ⎪ ⎨

1, L(x1 , . . . , xn ) = 0, L(y1 , . . . , yn ) ⎪ ⎪ ⎪ ⎩ undeﬁned,

if max (xmax , ymax ) ≤ θ, if ymax < xmax , and ymax ≤ θ ≤ xmax , elsewhere.

Thus, the ratio will not depend on θ if and only if xmax = ymax . Therefore, a minimal sufficient statistic for θ is X(n) , the largest order statistic.

5.5 Other Desirable Properties of a Point Estimator 279

It is important to note that although we often can ﬁnd a single statistic that is minimal sufﬁcient for one parameter, this need not be the case (see Exercise 5.5.1). For most of the density functions that we consider, any unbiased estimator that is a function of a minimal sufﬁcient statistic will be a uniformly minimum variance unbiased estimator (UMVUE), that is, it will posses the smallest variance possible among unbiased estimators.

Example 5.5.10 Let X1 , . . . , Xn be a random sample from the normal distribution with known mean μ = μ0 and unknown variance σ 2 . Show that ni=1 (Xi − μ0 )2 is the minimal sufﬁcient statistic for σ 2 . Use this statistic to ﬁnd an MVUE of σ 2 .

Solution The ratio of the likelihoods is # $ exp − (xi − μ0 )2 /2σ 2 L(x1 , . . . , xn ) # $ = L (y1 , . . . , yn ) exp − (yi − μ0 )2 /2σ 2 ! = exp

3" 1 2 2− 2 ) ) (y (x . − μ − μ 0 0 i i 2σ 2

In order for this ratio to be free of σ 2 , we need

(yi − μ0 )2 =

(xi − μ0 )2 .

(Xi − μ0 )2 is minimal sufficient for σ 2 . Hence, Because E(Xi − μ0 )2 = σ 2 , we can see that (1/n) (Xi − μ0 )2 is an unbiased estimator of σ 2 . Because this is a function of a minimal sufficient statistic, (1/n) ni=1 (Xi − μ0 )2 is an MVUE of σ 2 .

EXERCISES 5.5 5.5.1.

Show that the maximum likelihood estimator for p, Yn /n in a binomial distribution is consistent.

5.5.2.

Show that Yn , the nth-order statistic from a U(0, θ) distribution, is a consistent estimator for θ.

5.5.3.

Let X1 , . . . , Xn be a random sample with EXi = μi , EXi2 = μ2 , and EXi4 = μ4 , all ﬁnite. Show that S 2 = (1/n) ni=1 (Xi − X)2 is a consistent estimator of σ 2 = Var(Xi ).

5.5.4.

Let X1 , . . . , Xn be a random sample from a population with pdf f (x) =

αxα−1 , 0,

for 0 < x < 1; α > 0 otherwise.

280 CHAPTER 5 Point Estimation

Is the method of moments estimator for α consistent? 5.5.5.

Let X1 , . . . , Xn be a random sample from an exponential population with parameter θ. Show that X is a consistent estimator of θ.

5.5.6.

Let X1 , . . . , Xn and Y1 , . . . , Yn be independent random samples from populations with means μ1 and μ2 variances σ12 and σ22 , respectively. Show that the difference X − Y is a consistent estimator of μ1 − μ2 .

5.5.7.

Let X1 , . . . , Xn be a random sample from a population with pdf f (x) =

1 (1−α)/α , αx

0,

for 0 < x < 1; α > 0 otherwise.

(a) Show that the maximum likelihood estimator of α is αˆ = − (1/n) (b) Is αˆ of part (a) an unbiased estimator of α? (c) Is αˆ of part (a) a consistent estimator of α? 5.5.8.

n

i=1 ln Xi .

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

x −x2 /(2α) , αe

0,

for x > 0 otherwise.

(a) Determine the maximum likelihood estimator αˆ of α. (b) Is αˆ of part (a) an unbiased estimator of α? (c) Is αˆ of part (a) a consistent estimator of α? 5.5.9.

Let X1 , . . . , Xn be a random sample from the uniform distribution on the interval (θ, θ + 1). Let θˆ 1 = X −

1 n , θˆ 2 = X(n) − , 2 n+1

where X(n) is the nth order statistic. Find the efﬁciency of θˆ 2 relative to θˆ 1 . 5.5.10.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population. Let θˆ 1 be the sample mean and θˆ 2 be the sample median. It is known that Var(θˆ 2 ) = (1.2533)2 (σ 2 /n). Find the efﬁciency of θˆ 2 relative to θˆ 1 .

5.5.11.

Let X1 , . . . , Xn be a random sample from an exponential population with parameter θ. Show that X is efﬁcient for θ.

5.5.12.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population. Show that 2 (n − 1) 4 MSE S 2 = σ , n2

where S 2 = (1/n)

n i=1

2 Xi − X .

5.5 Other Desirable Properties of a Point Estimator 281

5.5.13.

Prove % E

% & & ∂2 ln f (x) ∂ ln f (x) 2 = −E , ∂θ ∂θ 2

making suitable assumptions. 5.5.14.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population. (a) Show that the sample variance S 2 is an UMVUE for σ 2 when the value of μ is not known. (b) Show that the variance of S 2 is greater than the Cramér–Rao lower bound.

5.5.15.

Let X1 , . . . , Xn be a random sample from a U(0, θ) distribution. Let X(n) be the nth order statistic. (a) Show that θˆ 1 = X(n) , θˆ 2=2X, and θˆ 3= n+1 n X(n) are unbiased estimators of θ. (b) Find the efﬁciency of θˆ 1 relative to θˆ 2 . (c) Find the efﬁciency of θˆ 2 relative to θˆ 3 .

5.5.16.

Let X1 , . . . , Xn , (n ≥ 2) be a random sample from a distribution with pdf 1

f (x) = π[1+(x−θ)2 ] ,

−∞ < x < ∞,

−∞ < θ < ∞.

Show that the Cramér–Rao lower bound for a UBE of θ is 2/n. 5.5.17.

Let X1 , . . . , Xn , n > 4, be a random sample from a population with a mean μ and variance σ 2 . Consider the following three estimators of μ: θˆ 1 =

1 (X1 + 2X2 + 5X3 + X4 ) , 9

θˆ 2 =

1 2 1 1 X1 + X2 + X3 + . . . + Xn−1 + Xn , 5 5 5 (n − 3) 5

and θˆ 3 = X.

(a) Show that each of the three estimators is unbiased. (b) Find e(θˆ 2 , θˆ 1 ), e(θˆ 3 , θˆ 1 ), and e(θˆ 3 , θˆ 2 ). 5.5.18.

Find the Cramér–Rao lower bound for the variance of an unbiased estimator of θ, based on a sample of size n for the following pdfs: (i) f (x, θ) = θ12 xe−x/θ , x > 0, θ > 0. (ii) f (x, θ) = θxθ−1 ,

5.5.19.

0 < x < 1, θ > 0.

Let Y1 , . . . , Yn be a random sample from the uniform distribution over the interval (θ − 1, θ + 1). Show that the order statistics X1 = min(Yi ) and Xn = max(Yi ) are jointly sufﬁcient for θ. Also, show that X1 and Xn are jointly minimal for θ.

282 CHAPTER 5 Point Estimation

5.5.20.

Let X1 , . . . , Xn be a random sample from a normal distribution with unknown mean μ and known variance σ 2 . Find the maximum likelihood estimator of μ and show that it is a function of a minimal sufﬁcient statistic.

5.5.21.

Let X1 , . . . , Xn be a random sample from a normal distribution with unknown mean μ and unknown variance σ 2 . Show that ni=1 Xi and ni=1 Xi2 are jointly minimal sufﬁcient for μ and σ 2 . Also show that X and S 2 are UMVUEs for μ and σ 2 .

5.5.22.

Let X1 , . . . , Xn be a random sample from the Weibull density ⎧ ⎨ 2x e−x2 /α, f (x) = α ⎩ 0,

x>0 otherwise.

Find an UMVUE for α. 5.5.23.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. Find a minimal sufﬁcient statistic for λ.

5.5.24.

Let X1 , . . . , Xn be a random sample from a gamma distribution with parameters α and β, both unknown. Find minimal sufﬁcient statistics for the parameters α and β.

5.5.25.

Let X1 , . . . , Xn be a random sample from a distribution with density function f (x) =

ex−β , 0,

x≥β otherwise.

Find an UMVUE for β. 5.5.26.

Let X1 , . . . , Xn be a random sample from the exponential distribution with pdf f (x) =

1 −x/β , βe

x>0

0,

otherwise.

Show that X is an UMVUE for β. Also show that 5.5.27.

n n+1

2

X is an MVUE for β2 .

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

2x e−x2 /β , β

x>0

0,

otherwise.

Find an UMVUE for β.

5.6 CHAPTER SUMMARY In this chapter we have discussed the basic concepts of point estimation. Two methods of ﬁnding point estimators were described—the method of moments and the method of maximum likelihood. We have seen that the maximum likelihood estimators possess the invariance property, which states that if θˆ is a maximum likelihood estimator of the parameter θ, then h(θˆ ) is a maximum likelihood estimator

5.7 Computer Examples 283

for h(θ). Some desirable properties of the point estimators that we have discussed are unbiasedness, consistency, efﬁciency, and sufﬁciency. Unbiasedness means that the expected value of the sample statistic (the mean of its probability distribution) should be equal to the parameter. Unbiasedness guards against consistently producing under- or overestimates of the parameter in repeated sampling. If the estimator is consistent, then, as the sample size increases, the estimator can be expected to get closer and closer to the population parameter. Efﬁcient estimators have the lowest variance among all other estimators. A sufﬁcient estimator is a “good” estimator of the population parameter θ in the sense that it depends on fewer data values. We will now list some of the key deﬁnitions introduced in this chapter. ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Method of moments Likelihood function Maximum likelihood equations Unbiased estimator Mean square error Minimum variance unbiased estimator Consistent estimator Efﬁciency Uniformly minimum variance unbiased estimator Efﬁcient estimator Sufﬁcient estimator Jointly sufﬁcient Minimal sufﬁcient statistic

In this chapter, we have also learned the following important concepts and procedures. ■ ■ ■ ■ ■ ■

The method of moments procedure Procedure to ﬁnd MLE Procedure to test for consistency Procedure to test relative efﬁciency Cramér–Rao procedure to test for efﬁciency Procedure to verify sufﬁciency

5.7 COMPUTER EXAMPLES Because in the earlier chapters we have already given steps to obtain summary statistics such as the mean and variance using SPSS and SAS, we could use those commands to obtain point estimates as we will do with Minitab. Therefore, we will not give separate subsections for SPSS and SAS procedures. The following examples illustrate Minitab procedures.

Example 5.7.1 Generate 50 sample points from an N(4, 4) distribution and ﬁnd the descriptive statistics. Obtain an unbiased and sufﬁcient estimate of μ.

284 CHAPTER 5 Point Estimation

Solution Because we know that the sample mean x is an unbiased and sufficient estimate of the population mean μ, we only need to find the sample mean of the generated data. Calc > Random Data > Normal . . . > Type 50 in Generate __ rows of data > Store in column(s): type C1 > type in Mean: 4.0 and in Standard deviation: 2.0 > click OK . The following is one possible output. C1 4.76039 5.16925 3.25593 2.80955 2.43494 1.76723 2.86329 5.38503

5.07819 3.68845 2.66181 4.19032 2.01465 3.15460 5.97599

4.85263 6.40513 1.01352 4.65449 4.02358 4.81882 7.75170

4.08032 6.13801 5.82506 3.48680 8.22997 0.36250 7.10011

6.77772 7.20015 6.04212 6.39083 2.44516 0.85002 6.61681

4.21677 2.41415 5.22235 6.56357 0.39563 14.47052 0.97982

1.51811 3.50008 5.29924 1.32281 3.78948 0.79586 4.01400

Now follow the procedure to obtain the descriptive statistics from Example 1.8.3 to obtain Descriptive Statistics Variable N Mean Median TrMean StDev SE Mean C1 50 4.116 4.135 4.115 2.047 0.289 Variable Minimum Maximum Q1 Q3 C1 0.362 8.230 2.443 5.863

We can see that the unbiased and sufﬁcient estimate of the mean μ for these data is x = 4.116.

Example 5.7.2 Generate 35 samples from a U(0, 5) distribution and using the descriptive statistics command, ﬁnd the maximum likelihood estimate for this data.

Solution We know that for a random sample X1 , . . . , Xn from U(0, θ), the MLE, θˆ = max(Xi ) = X(n) , the nth order statistic. We can use the following steps to obtain the estimate.

Calc > Random Data > Uniform. . . > Type 35 in Generate __ rows of data > Store in column(s): type C1 > type in Lower end point: 0.0 and in Upper end point: 5.0 > click OK One possible output is given below.

Projects for Chapter 5 285

C1 4.32848 0.07934 2.92537 4.20844 3.25272

4.79402 3.12453 2.39721 3.75506 4.61083

0.34515 1.69073 4.84440 4.56626 3.06527

0.08428 3.44003 1.79129 3.50280 2.34003

1.93000 0.47447 4.38718 1.95689 0.40877

0.27878 2.28072 3.60697 0.56969 2.52708

3.12992 0.49205 0.94159 1.02543 1.44525

Now follow the procedure to obtain the descriptive statistics from Example 1.8.3 to obtain Descriptive Statistics Variable N Mean Median TrMean StDev SE Mean C1 35 2.417 2.397 2.413 1.541 0.260 Variable Minimum Maximum Q1 Q3 C1 0.079 4.844 0.942 3.607 Therefore the MLE θˆ = 4.884.

For the previous example, it should be noted that because we are generating random data, each time we follow this procedure, we will be getting different answers. When we have a particular data set, enter the data in C1 and just use the procedure to ﬁnd the descriptive statistics. For other distributions, click the appropriate distribution in Random Data.

PROJECTS FOR CHAPTER 5 5A. Asymptotic Properties In general, we do not have a single sample with one estimator of the unknown parameter θ. Rather, we will have a general formula that deﬁnes an estimator for any sample size. This gives a sequence of estimators of θ: θˆ = hn (X1 , . . . , Xn ) , n = 1, 2, . . . ..

In this case, we can deﬁne the following asymptotic properties: (i) The sequence of estimators θˆ n is said to be asymptotically unbiased for θ if bias θˆ n → 0 as n → ∞. (ii) Suppose θˆ n and yˆ n are two sequences of estimators that are asymptotically unbiased for θ. The asymptotic relative efﬁciency of θˆ n to yˆ n is deﬁned by Var θˆ n . lim n Var yˆ n

(a) Show that θˆ n is asymptotically unbiased if and only if E θˆ n → θ as n → ∞.

286 CHAPTER 5 Point Estimation

(b) Let X1 , . . . , Xn be a random sample from a distribution with unknown mean μ and variance σ 2 . It is known that the method of moments estimators for μ and σ 2 are, respectively, the 2 sample mean X and S 2n = (1/n) ni=1 Xi − X = ((n − 1)/n) Sn2 , where Sn2 is the sample variance. (i) Show that S 2n is an asymptotically unbiased estimator of σ 2 . (ii) Show that the asymptotic relative efﬁciency of S 2n to Sn2 is 1. (iii) Show that MSE S 2n < MSE Sn2 . Thus, Sn2 is unbiased but S 2n has a smaller mean square error. However, it should be noted that the difference is very small and approaches zero as n becomes large.

5B. Robust Estimation The estimators derived in this chapter are for particular parameters of a presumed underlying family of distributions. However, if the choice of the underlying family of distributions is based on past experience, there is a possibility that the true population will be slightly different from the model used to derive the estimators. Formally, a statistical procedure is robust if its behavior is relatively insensitive to deviations from the assumptions on which it is based. If the behavior of an estimator is taken as its variance, a given estimator may have minimum variance for the distribution used, but it may not be very good for the actual distribution. Hence, it is desirable for the derived estimators to have small variance over a range of distributions. We call such estimators robust estimators. The following illustrates how the variance of an estimator can be affected by deviations from the presumed underlying population model. Consider estimating the mean of a standard normal distribution. Let X1 , . . . , Xn be a random sample from a standard normal distribution. Suppose the population actually follows a contaminated normal distribution. That is, for 0 ≤ δ ≤ 1, 100 (1 − δ) % of the observations come from an N(0, 1) distribution and the remaining 100δ% of observations come from an N(0, 5) distribution. We already know that the minimum variance unbiased estimator of the mean μ of an uncontaminated normal distribution is the sample mean. A less effective alternative would be the sample median. (a) Conduct a simulation study with sample size n that takes, say, 5000 random samples of 100 observations each. Find the mean and median. Also ﬁnd the sample variance of each. For various values of δ, say 0.0, 0.01, 0.05, 0.1, 0.2, 0.3, and 0.4, create a table of variances of sample mean and sample variance. Compare the variances as the value of δ increases. (b) The aim of robust estimation is to derive estimators with variance near that of the sample mean when the distribution is standard normal while having the variance remain relatively stable as δ increases. One such estimator is the α − trimmedmean. Let 0 ≤ α ≤ 0.5, and deﬁne k = [nα], where [x] is the greatest integer that is less than or equal to x. For the ordered sample, discard the k highest and lowest observations and ﬁnd the mean of the remaining n − k observations. That is, let X(1) ≤ X(2) ≤ . . . ≤ X(n) be the ordered sample, and deﬁne Xα =

X(1+k) ≤ X(2+k) ≤ . . . ≤ X(n+k) . n − 2k

For the values of δ and the samples in part (a), compute the mean and the 0.05-, 0.1-, 0.25-, and 0.5-trimmed means. Discuss the robustness.

Projects for Chapter 5 287

5C. Numerical Unbiasedness and Consistency (a) Run the simulation of a normal experiment with increasing sample size. Numerically show the unbiased and consistent properties of the sample mean. Run the experiment at least up until n = 1000. (b) Repeat the experiment of part (a), now with an exponential distribution.

5D. Averaged Squared Errors (ASEs) Generate 25 samples of size 40 from a normal population with μ = 10, and σ 2 = 4. For each of the 25 samples: 40

(a) Compute: x, s2 =

40

(xi −x)2

i=1

,

39

=

s12

40

(xi −x)2

i=1

, and

40

s22

=

(xi −x)2

i=1

41

.

(b) Compute the average squared error (ASE) for each of the estimates s2 , s12 , s22 as follows. !! K " " 2 2 (xi − x)2 /39 for K = 1, 2, . . . , 25; and Ks be the sample variance for the Let Ks = i=1

Kth sample. Then, the average squared error is 2 25 2 Ks − σ 2 ASE =

i=1

.

25

Repeat this procedure for the other two estimators. Compare the three ASEs and check which has the least ASE. (c) Repeat (a) and (b) with a sample size of 15.

5E. Alternate Method of Estimating the Mean and Variance (a) Consider the following alternative method of estimating μ and σ 2 . We sample sequentially, and at each stage we compute the estimates of μ and σ 2 as follows. Let X1 , . . . , Xn , Xn+1 be the sample values. Compute n

Xn =

n+1

Xi

i=1

n

, Xn+1 =

i=1

n+1

n+1

Xi − Xn

2 Sn+1 =

i=1

n

n

Xi − Xn

Xi , Sn2 =

i=1

2 .

The sequential procedure is stopped when 2 2 ≤ 0.01. Sn − Sn+1

This will also determine the sample size. (b) Compare the sample sizes and estimates in 5D and 5E.

n−1

2 , and

288 CHAPTER 5 Point Estimation

5F. Newton–Raphson in One Dimension For a given function g(x), suppose we need to solve g(θ) = 0. Using the ﬁrst-order Taylor expansion, dg , and setting g(θ) = 0, we get θ ≈ x − gg(x) g(θ) ≈ g(x) + (θ − x)g (x), where g (x) = dx (x) . Thus, starting with an initial guess solution x, the guess is updated by θ using the previous formula. This derivation is the basis for the Newton–Raphson iterative method for obtaining the solution of g(θ) = 0. This is given by g (θn ) , θ(n+1) = θn − g (θn )

n ≥ 0,

where θn is the value of θ at the nth iteration, starting with the initial guess, θ0 . For a good approximation of the solution, the choice of θ0 is important. The convergence of this algorithm cannot be guaranteed. For the MLE, we want to ﬁnd a solution of g(θ) =

dL = 0, dθ

where L = L (θ) is the likelihood function of the random sample X1 , . . . , Xn . An iterative algorithm for ﬁnding the MLE can be given by dL (θn ) dθ , θ(n+1) = θn − 2 d L (θn ) dθ 2

n ≥ 0.

Write a computer program to ﬁnd the MLE of α for a gamma distribution with parameters α and β.

5G. The Empirical Distribution Function The estimators in this chapter yield a single real value (point estimate) for each parameter. In Chapter 6, we will learn about so-called interval estimates. In this project, we use an estimation procedure that estimates the whole distribution function, F , of a random variable X. We now deﬁne the empirical distribution. The empirical distribution function for a random sample X1 , . . . , Xn from a distribution F is the function deﬁned by Fn (x) =

1 #{i, 1 ≤ i ≤ n : Xi ≤ x}. n

It can be shown that nFn (x) is a binomial random variable with E [Fn (x)] = F (x) and Var [Fn (x)] =

1 F (x) [1 − F (x)] . n

Also, by the strong law of large numbers, for each real number x, lim Fn (x) = F (x) with probability 1.

n→∞

Projects for Chapter 5 289

One of the tests to determine whether a random sample comes from a speciﬁc distribution is the Kolmogorov–Smirnov (K-S) test. The K-S test is based on the maximum distance between the empirical distribution function and the actual cumulative distribution function of this speciﬁc distribution (such as, say, the normal distribution). Using the method of Project 4A (or using any statistical software), generate 100 sample points from a normal distribution with mean 2 and variance 9. Graph the empirical distribution function for this sample. Compare this graph with the graph of the N(2, 9) distribution.

This page intentionally left blank

Chapter

6

Interval Estimation Objective: To learn some statistical methods that are commonly used to obtain interval estimation or conﬁdence limits of the unknown population parameters. 6.1 Introduction 292 6.2 Large Sample Conﬁdence Intervals: One Sample Case 300 6.3 Small Sample Conﬁdence Intervals for μ 310 6.4 A Conﬁdence Interval for the Population Variance 315 6.5 Conﬁdence Interval Concerning Two Population Parameters 6.6 Chapter Summary 330 6.7 Computer Examples 330 Projects for Chapter 6 334

321

Karl Pearson (Source: http://www-history.mcs.st-and.ac.uk/∼ history/PictDisplay/Pearson.html)

Karl Pearson (1857–1936) is considered the founder of the 20th-century science of statistics. Pearson has contributed in several different ﬁelds such as anthropology, biometry, eugenics, scientiﬁc method, and statistical theory. He applied statistics to biological problems of heredity and evolution.

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

291

292 CHAPTER 6 Interval Estimation

He is the author of The Grammar of Science, the three volumes of The Life, Letters and Labors of Francis Galton, and The Ethic of Free Thought. Pearson was the founder of the statistical journal Biometrika. In 1900, he published a paper on the chi-square goodness of ﬁt test. This is one of Pearson’s most signiﬁcant contributions to statistics. In 1893, Pearson coined the term “standard deviation.”

6.1 INTRODUCTION In the previous chapter, we studied methods for ﬁnding point estimators for the population parameters. In general the estimates will differ from the true parameter values by varying amounts depending on the sample values obtained. In addition, the point estimates do not convey any measure of reliability. In this chapter, we discuss another type of estimation, called an interval estimation. Although point estimators are useful, interval estimators convey more information about the data that are used to obtain the point estimate. The purpose of using an interval estimator is to have some degree of conﬁdence of securing the true parameter. For an interval estimator of a single parameter θ, we will use the random sample to ﬁnd two quantities L and U such that L < θ < U with some probability. Because L and U depend on the sample values, they will be random. This interval (L, U) should have two properties: (1) P(L < θ < U) is high, that is, the true parameter θ is in (L, U) with high probability, and (2) the length of the interval (L, U) should be relatively narrow on the average. In summary, interval estimation goes a step beyond point estimation by providing, in addition to the estimating interval (L, U), a measure of one’s conﬁdence in the accuracy of the estimate. Interval estimators are called conﬁdence intervals and the limits are called U and L, the upper and lower conﬁdence limits, respectively. The associated levels of conﬁdence are determined by speciﬁed probabilities. The width of the conﬁdence interval reﬂects the amount of variability inherent in the point estimate. Thus, our objective is to ﬁnd a narrow interval with high probability of enclosing the true parameter, θ. We will restrict our attention to single parameter estimation. The probability that a conﬁdence interval will contain the true parameter θ is called the conﬁdence coefﬁcient. The conﬁdence coefﬁcient gives the fraction of the time that the constructed interval will contain the true parameter, under repeated sampling. Let L and U be the lower and upper conﬁdence limits for a parameter θ based on a random sample X1 , . . . , Xn . Both L and U are functions of the sample. We can write the interval estimate of θ as P (L ≤ θ ≤ U) = 1 − α

and we read it as we are (1 − α)100% conﬁdent that the true parameter θ is located in the interval (L, U ). The number 1 − α is the conﬁdence coefﬁcient, and the interval (L, U ) is referred to as a (1 − α)100% conﬁdence interval ((1 − α)100% CI) for θ. Thus, if we want a 95% conﬁdence interval for, say, population mean μ, then α = 0.05. Note that for the discrete random variables, we may not be able to ﬁnd a lower bound L and an upper bound U such that the probability, P (L ≤ θ ≤ U ), is exactly (1 − α). In such a case we can choose L and U such that P (L ≤ θ ≤ U ) ≥ 1 − α. How do we ﬁnd the conﬁdence interval? For this, we use the error structure of the point estimator to obtain this interval. For instance, we know that the sample mean, X, is a point estimate (MLE or

6.1 Introduction 293

unbiased estimator) of the population mean μ. In this case, we know that the standard error of X is √ σ/ n. If the sample came from a normal population, then for a 95% conﬁdence interval for the mean, multiply the standard error by 1.96 and then add and subtract this product from the sample mean. From this we can also observe that, if everything else remains the same, the size of the conﬁdence interval reduces as the sample size increases.

Example 6.1.1 As part of a promotion, the management of a large health club wants to estimate average weight loss for its members within the ﬁrst 3 months after joining the club. They took a random sample of 45 members of this health club and found that they lost an average of 13.8 pounds within the ﬁrst 3 months of membership with a sample standard deviation of 4.2 pounds. Find a 95% conﬁdence interval for the true mean. What if a random sample of 200 members of this health club also resulted in the same sample mean and sample standard deviation?

Solution Here a point estimate of the true mean μ is the sample mean x = 13.8 pounds. Because n = 45 is large enough, we can use the Central Limit Theorem and use approximate normality for the distribution of X √ with mean μ and the approximate standard error (4.2/ 45) = 0.626. Thus a 95% confidence interval is 13.8 ± (1.96)(0.626), resulting in the interval (12.57, 15.03). Thus, on average, with 95% confidence, one can expect the true mean to lie in this interval. √ For n = 200, the standard error is (4.2/ 200) ≈ 0.297. Thus a 95% confidence interval is 13.8 ± (1.96)(0.297) resulting in the interval (13.22, 14.38). Thus the more sample values (that is, the more information) we have, the tighter (smaller width) the interval. The previous example was built on our knowledge of the sampling distribution of the sample mean. What if the sampling distribution of the statistic we are interested in is not readily available? More generally, our success in building confidence intervals for an estimate of a parameter depends on identifying a quantity known as the pivot. We now describe this method.

6.1.1 A Method of Finding the Confidence Interval: Pivotal Method The pivotal method is a general method of constructing a conﬁdence interval using a pivotal quantity. This relies on our knowledge of sampling distributions. Here we have to ﬁnd a pivotal quantity with the following two characteristics: (i) It is a function of the random sample (a statistic or an estimator θˆ ) and the unknown parameter θ, where θ is the only unknown quantity, and (ii) It has a probability distribution that does not depend on the parameter θ. From (i) and (ii), it is important to note that the pivotal quantity depends on the parameter, but its distribution is independent of the parameter. Let X1 , . . . , Xn be a random sample and let θˆ be a reasonable point estimate of θ. For instance, θˆ could be the maximum likelihood (or some other) estimator of θ. In general, ﬁnding a pivotal quantity may not be easy. However, if θˆ is the sample mean X or sample variance S 2 , we could ﬁnd a pivotal quantity with known sampling distributions. Suppose p (θˆ , θ) is a pivotal quantity with known probability distribution that is independent of θ.

294 CHAPTER 6 Interval Estimation

(Usually, the probability distribution of the pivotal quantity will be standard normal, t, χ2 , or F -distribution.) The following are some of the standard pivotal quantities: If the sample X1 , . . . , Xn is from N(μ, σ 2 ) √ (i) With μ unknown and σ known, let X be the sample mean. Then the pivot is (X−μ)/(σ/ n), which has an N(0, 1) distribution (see comments after Corollary 4.2.2). √ (ii) With μ unknown and σ unknown, then the pivot is (X − μ)/(S/ n), which has a tdistribution with (n − 1) degrees of freedom (see Theorem 4.2.9). If n is large, using CLT, the distribution of the pivot is approximately N(0, 1). (iii) If σ 2 is unknown, then the pivot is (n − 1)S 2 /σ 2 , which has a χ2 -distribution with (n − 1) degrees of freedom (see Theorem 4.2.8). For a given value of α, (0 < α < 1), and constants a and b, with (a < b), let P (a ≤ p (θˆ , θ) ≤ b) = 1 − α.

Hence, given θˆ , the inequality is solved for θ to obtain a region of θ values, usually an interval corresponding to the observed θˆ -value. The following examples illustrate the pivotal method.

Example 6.1.2 Suppose we have a random sample X1 , . . . , Xn from N(μ, 1). Construct a 95% conﬁdence interval for μ.

Solution Here the confidence coefficient is 0.95. We know that the maximum likelihood estimator of μ is X, which has an N(μ, 1/n) distribution. Note that this distribution depends on the unknown value of μ, and hence X cannot be a pivot. However, taking the z-transform of X, we obtain the pivotal quantity as Z=

X−μ X−μ √ = √ σ/ n 1/ n

which has an N(0, 1) distribution that is a function of the sample measurements and does not depend on μ. Hence, this Z can be taken as a pivot p (θˆ , θ). Now to find a and b such that P (a ≤ Z = p (θˆ , θ) ≤ b) = 0.95. One such choice is to find the value of a such that p (−a ≤ Z ≤ a) = 0.95. From the normal table, P (−zα/2 ≤ Z ≤ zα/2 ) = 0.95, where zα/2 represents the value of z with tail area α/2. This implies a = zα/2 = 1.96. Hence, P (−1.96 ≤ Z ≤ 1.96) = 0.95 or, using the definition of Z and solving for μ, we obtain

1.96 1.96 = 0.95. P X− √ ≤μ≤X+ √ n n

6.1 Introduction 295 √ √ Hence, a 95% confidence interval for μ is (X − (1.96/ n), X + (1.96/ n)). Thus, the lower confidence √ √ limit L is X − (1.96/ n) and the upper confidence limit U is X + (1.96/ n).

From the derivation of Example 6.1.1, it follows that

σ P X − μ < zα/2 √ = 1 − α. n

Thus, for a normal population with known variance σ 2 , if X is used as an estimator of the true mean √ μ, the probability that the error will be less than zα/2 σ/ n is 1 − α. It is important to note that there is some arbitrariness in choosing a conﬁdence interval for a given problem. There may be several pivotals for θˆ that could be used. Also, it is not necessary to allocate equal probability to the two tails of the distribution; however, doing so may result in the shortest length conﬁdence interval for a given conﬁdence coefﬁcient. When we make the statement of the form

1.96 1.96 P X− √ ≤μ≤X+ √ = 0.95, n n

we mean that, in an inﬁnite series of trials in which repeated samples of size n are drawn from the same population and 95% conﬁdence intervals for μ are calculated by the same method for each of the samples, the proportion of intervals that actually include μ will be 0.95. Figure 6.1 illustrates this idea, where the vertical line represents the position of true mean μ and each of the horizontal lines represents a 95% conﬁdence interval of the sample, 20 samples of size n are taken. √ √ A statement of the type P (x − (1.96/ n) ≤ μ ≤ x + (1.96/ n)) = 0.95, where x is the observed sample mean, is misleading. Once we calculate this interval using a particular sample, then either this interval contains the true mean μ or not, and hence the probability will be either 0 or 1. Thus, the correct interpretation of conﬁdence interval for the population mean is that if samples of the same size, n, are drawn repeatedly from a population, and a conﬁdence interval is calculated from each sample, then 95% of these intervals should contain the population mean. This is often stated as √ √ “We are 95% conﬁdent that the true mean is in the interval (X − zα/2 (σ/ n), X + zα/2 (σ/ n)).” Thus,

■ FIGURE 6.1 95% confidence intervals for μ.

296 CHAPTER 6 Interval Estimation

PDF of P

■ FIGURE 6.2 Probability density of the pivot.

the correct interpretation requires the conﬁdence limits to be variables. This concept of conﬁdence interval is attributed to Neyman. We can follow the accompanying procedure to ﬁnd a conﬁdence interval for the parameter θ.

PROCEDURE TO FIND A CONFIDENCE INTERVAL FOR θ USING THE PIVOT 1. Find an estimator θˆ of θ: usually MLE of θ works. 2. Find a function of θ and θˆ , p(θ, θˆ ) (pivot), such that the probability distribution of p(. , . ) does not depend on θ. 3. Find a and b such that P(a ≤ p(θ, θˆ ) ≤ b) = 1 − α. Choose a and b such that P(p(θ, θˆ )≤a) = α/2 and P(p(θ, θˆ ) ≥ b) = α/2 (see Figure 6.2 where the shaded area in each side is α/2). 4. Now, transform the pivot conﬁdence interval to a conﬁdence interval for the parameter θ. That is, work with the inequality in step 3 and rewrite it as P(L ≤ θ ≤ U) = 1 − α, where L is the lower conﬁdence limit and U is the upper conﬁdence limit.

The following example is given to show that the success of ﬁnding a pivotal quantity depends on our ability to ﬁnd the right transformation of the statistic and its distribution so that the transformed variable is a pivot.

Example 6.1.3 Suppose the random sample X1 , . . . , Xn has U(0, θ) distribution. Construct a 90% conﬁdence interval for θ and interpret. Identify the upper and lower conﬁdence limits.

Solution From Example 5.3.4, we know that U = max Xi 1≤i≤n

6.1 Introduction 297

is the MLE of θ. The random variable U has the pdf fU (u) = nu n−1 /θ n ,

0 ≤ u ≤ θ.

This is not independent of the parameter θ. Let Y = U/θ, then (using the Jacobians described in Chapter 3) the pdf of Y is given by fY (y) = ny n−1 ,

0 ≤ y ≤ 1.

Hence, Y satisfies the two characteristics of the pivotal quantity. Thus, Y = U/θ is a pivot. Now, we have to find a and b such that p (a ≤

U ≤ b) = 0.90. θ

pdf of Y

0.05

0.05

y 1

0

To find a and b we use the cdf of Y, FY (y) = yn , 0 ≤ y ≤ 1, as follows. F (y )

0.95

0.05

0

a

y

b1

FY (a) = 0.05

and

FY (b) = 0.95

an = 0.05

and

b n = 0.95

which implies that

298 CHAPTER 6 Interval Estimation

resulting in a=

√ n

0.05

and

b=

√ n 0.95.

Write

P

√ n

0.05

0, x > 0. Construct a 95% conﬁdence interval for θ and interpret. [Hint: Recall that ni=1Xi has a gamma distribution with α = n, β = θ.]

6.1.8.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. (a) Construct a 90% conﬁdence interval for λ. (b) Suppose that the number of raisins in a bowl of a particular brand of cereal is observed to be 25. Assuming that the number of raisins in a bowl is Poisson distributed, estimate the expected number of raisins per bowl with a 90% conﬁdence interval. (c) How many bowls of cereal need to be sampled in order to estimate the expected number of raisins per bowl with a standard error of less than 0.2?

6.1.9.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). (a) Construct a (1 − α)100% conﬁdence interval for μ when the value of σ 2 is known. (b) Construct a (1 − α)100% conﬁdence interval for μ when the value of σ 2 is unknown.

6.1.10.

Let X1 , . . . , Xn be a random sample from an N(μ1 , σ 2 ) population and Y1 , . . . , Yn be an independent random sample from an N(μ2 , σ 2 ) distribution where σ 2 is assumed to be known. Construct a (1 − α)100% interval for (μ1 − μ2 ). Interpret its meaning.

300 CHAPTER 6 Interval Estimation

6.1.11.

Let X1 , . . . , Xn be a random sample from a uniform distribution on [θ, θ + 1]. Find a 99% conﬁdence interval for θ, using an appropriate pivot.

6.2 LARGE SAMPLE CONFIDENCE INTERVALS: ONE SAMPLE CASE If the sample size is large, then by the Central Limit Theorem, certain sampling distributions can be assumed to be approximately normal. That is, if θ is an unknown parameter (such as μ, p, (μ1 − μ2 ), (p1 − p2 )), then for large samples, by the Central Limit Theorem, the z-transform z=

θˆ − θ σθˆ

possesses an approximately standard normal distribution, where θˆ is the MLE of θ and σθˆ is its standard deviation. Then as in Example 6.1.1, the pivotal method can be used to obtain the conﬁdence interval for the parameter θ. For θ = μ, n ≥ 30 will be considered large; for the binomial parameter p, n is considered large if np, and n(1 − p) are both greater than 5. PROCEDURE TO CALCULATE LARGE SAMPLE CONFIDENCE INTERVAL FOR θ 1. Find an estimator (such as the MLE) of θ, say θˆ . 2. Obtain the standard error, σθˆ of θˆ . 3. Find the z-transform z = (θˆ − θ)/σθˆ . Then z has an approximately standard normal distribution. 4. Using the normal table, ﬁnd two tail values −zα/2 and zα/2 . 5. An approximate (1 − α)100% conﬁdence interval for θ is θˆ − zα/2 σθˆ , θˆ + zα/2 σθˆ , that is, P θˆ − zα/2 σθˆ ≤ θ ≤ θˆ + zα/2 σθˆ = 1 − α. 6. Conclusion: We are (1 −α)100% conﬁdent that the true parameter θ lies in the interval θˆ − zα/2 σθˆ , θˆ + zα/2 σθˆ .

Example 6.2.1 Let θˆ be a statistic that is normally distributed with mean θ and standard deviation σθˆ , where σ is assumed to be known. Find a conﬁdence interval for θ that possesses a conﬁdence coefﬁcient equal to 1 − α.

Solution The z-transform of θˆ is Z=

θˆ − θ σθˆ

and has a standard normal distribution. Select two tail values −zα/2 and zα/2 such that P (−zα/2 ≤ Z ≤ zα/2 ) = 1 − α. Because of symmetry, this is the shortest interval that contains the area 1 − α. Then, P (θˆ − zα/2 σθˆ ≤ θ ≤ θˆ + zα/2 σθˆ ) = 1 − α.

6.2 Large Sample Conﬁdence Intervals: One Sample Case 301

Therefore, the confidence limits of θ are θˆ − zα/2 σθˆ and θˆ + zα/2 σθˆ . Hence, (1 − α)100% confidence interval for θ is given by θˆ ± zα/2 σθˆ .

If in particular for a large sample of size n, let θˆ = X be the sample mean. Then the large sample (1 − α)100% conﬁdence interval for the population mean μ is σ S X ± zα/2 √ X ± zα/2 √ n n

where S is a point estimate of σ. That is,

S S P X − zα/2 √ ≤ μ ≤ X + zα/2 √ = 1 − α. n n

As we have seen in Section 6.1, the correct interpretation of this conﬁdence interval is that in a repeated √ sampling, approximately (1 − α)100% of all intervals of the form X ± zα/2 (S/ n) include μ, the true mean. Suppose x and s are the sample mean and the sample standard deviation, respectively, for a particular set of n observed sample values x1 , . . . , xn . Then we do not know whether the particular √ √ interval (x − zα/2 (s/ n), x − zα/2 (s/ n)) contains μ. However, the procedure that produced this interval does capture the true mean in approximately (1 − α)100% of cases. This interpretation will be assumed hereafter, when we make a statement such as, “We are 95% conﬁdent that the true mean will lie in the interval (74.1, 79.8).”

Example 6.2.2 Two statistics professors want to estimate average scores for an elementary statistics course that has two sections. Each professor teaches one section and each section has a large number of students. A random sample of 50 scores from each section produced the following results: (a) Section I: x1 = 77.01, s1 = 10.32 (b) Section II: x2 = 72.22, s2 = 11.02 Calculate 95% conﬁdence intervals for each of these three samples.

Solution Because n = 50 is large, we could use normal approximation. For α = 0.05, from the normal table: zα/2 = z0.025 = 1.96. The confidence intervals are: (a) We have

s1 10.32 x1 ± zα/2 √ = 77.01 ± 1.96 √ n 50 which gives a 95% confidence interval (74.149, 79.871). (b) We can compute

s2 11.02 x2 ± zα/2 √ = 72.22 ± 1.96 √ n 50 which gives the interval (69.165, 75.275).

302 CHAPTER 6 Interval Estimation √ It may be noted that if the population is normal with a known variance σ 2 , we can use X ± zα/2 (σ/ n) as the conﬁdence interval for the population mean μ, irrespective of the sample size. However, if σ 2 is √ unknown, in order to use X ± zα/2 (s/ n) as an approximate conﬁdence interval for μ, the sample size has to be large for the Central Limit Theorem to hold. However to use this approximate procedure, we do not need the condition that samples arise from a normal distribution. We will consider sample size to be large if n ≥ 30 (applicable to estimators of the mean). If not, we shall use the small sample procedure discussed in the next section.

Example 6.2.3 Fifteen vehicles were observed at random for their speeds (in mph) on a highway with speed limit posted as 70 mph, and it was found that their average speed was 73.3 mph. Suppose that from past experience we can assume that vehicle speeds are normally distributed with σ = 3.2. Construct a 90% conﬁdence interval for the true mean speed μ, of the vehicles on this highway. Interpret the result.

Solution Because the population is given to be normal with standard deviation σ = 3.2, sample size need not be large given x = 73.3 and σ = 3.2. Here, n = 15, and α = 0.10. Thus, zα/2 = z0.05 = 1.645. Hence, a 90% confidence interval for μ is given by 3.2 3.2 73.3 − 1.645 √ < μ < 73.3 + 1.645 √ 15 15 or 71.681 < μ < 74.919. Interpretation: We are 90% confident that the true mean speed μ of the vehicles on this highway is between 71.681 and 74.919.

6.2.1 Confidence Interval for Proportion, p Consider a binomial distribution with parameter p. Let X be the number of successes in n trials. Then the maximum likelihood estimator pˆ of p is pˆ = X/n. It can be shown, using the procedure outlined at the beginning of this section, that an approximate large sample (1 − α)100% conﬁdence interval for p is

8 pˆ − zα/2

p(1 ˆ − p) ˆ , pˆ + zα/2 n

8

p(1 ˆ − p) ˆ . n

That is, P pˆ − zα/2

8

p(1 ˆ − p) ˆ < p < pˆ + zα/2 n

8

p(1 ˆ − p) ˆ n

= 1 − α.

A natural question is: “How do we determine the sample size that we have is sufﬁcient for the normal approximation that is used in the foregoing formula?” There are various rules of thumb that are used to determine the adequacy of the sample size for normal approximation. Some of the popular rules

6.2 Large Sample Conﬁdence Intervals: One Sample Case 303 are that np and n(1 − p) should be greater than 10, or that pˆ ± 2 p(1 ˆ − p)/n ˆ should be contained in the interval (0, 1), or np (1 − p) ≥ 10, etc. All of these rules perform poorly when p is nearer to 0 or 1. Recently, there have been many works on coverage analysis for conﬁdence intervals. We refer to a survey article by Lee et al. for more details on this topic. For simplicity of calculations, we will use the rule that np and n(1 − p) are both greater than 5.

Example 6.2.4 An auto manufacturer gives a bumper-to-bumper warranty for 3 years or 36,000 miles for its new vehicles. In a random sample of 60 of its vehicles, 20 of them needed ﬁve or more major warranty repairs within the warranty period. Estimate the true proportion of vehicles from this manufacturer that need ﬁve or more major repairs during the warranty period, with conﬁdence coefﬁcient 0.95. Interpret.

Solution Here we need to find a 95% confidence interval for the true proportion, p. Here, pˆ = 20/60 = 1/3. For α = 0.05, zα/2 = z0.025 = 1.96. Hence, a 95% confidence interval for p is ; < 8 2 < 1 = 3 p(1 ˆ − p) ˆ 1 3 = ± 1.96 pˆ ± zα/2 n 3 60 which gives the confidence interval as (0.21405, 0.45262). That is, we are 95% confident that the true proportion of vehicles from this manufacturer that need five or more major repairs during the warranty period will lie in the interval (0.21405, 0.45262).

6.2.2 Margin of Error and Sample Size In real-world problems, the estimates of the proportion p are usually accompanied by a margin of error, rather than a conﬁdence interval. For example, in the news media, especially leading up to election time, we hear statements such as “The CNN/USA Today/Gallup poll of 818 registered voters taken on June 27–30 showed that if the election were held now, the president would beat his challenger 52% to 40%, with 8% undecided. The poll had a margin of error of plus or minus four percentage points.” What is this “margin of error”? According to the American Statistical Association, the margin of error is a common summary of sampling error that quantiﬁes uncertainty about a survey result. Thus, the margin of error is nothing but a conﬁdence interval. The number quoted in the foregoing statement is half the maximum width of a 95% conﬁdence interval, expressed as a percentage. Let b be the width of a 95% conﬁdence interval for the true proportion, p. Let pˆ = x/n be an estimate for p where x is the number of successes in n trials. Then, 8 8 x (x/n)(1 − (x/n)) (x/n)(1 − (x/n)) x b = + 1.96 − − 1.96 n n n n 8 8 (x/n)(1 − (x/n)) 1 = 3.92 ≤ 3.92 , n 4n

because (x/n)(1 − (x/n)) = p(1 ˆ − p) ˆ ≤ 14 .

304 CHAPTER 6 Interval Estimation

Thus, the margin of error associated with pˆ = (x/n) is 100d%, where 9 1 3.92 1.96 max b 4n = = √ . d= 2 2 2 n

From the foregoing derivation, it is clear that we can compute the margin of error for other values of α by replacing 1.96 by the corresponding value of zα/2 . A quick look at the formula for the conﬁdence interval for proportions reveals that a larger sample would yield a shorter interval (assuming other things being equal) and hence a more precise estimate of p. The larger sample is more costly in terms of time, resources, and money, whereas samples that are too small may result in inaccurate inferences. Then, it becomes beneﬁcial for ﬁnding out the minimum sample size required (thus less costly) to achieve a prescribed degree of precision (usually, the minimum degree of precision acceptable). We have seen that the large sample (1 − α)100% conﬁdence interval for p is 8 pˆ − zα/2

p(1 ˆ − p) ˆ < p < pˆ + zα/2 n

8

p(1 ˆ − p) ˆ . n

Rewriting it, we have pˆ − p ≤ zα/2

8

zα/2 p(1 ˆ − p) ˆ = √ p(1 ˆ − p) ˆ n n

which shows that, with probability (1 − α), the estimate pˆ is within zα/2 p(1 ˆ − p)/n ˆ units of p. Because p(1 ˆ − p) ˆ ≤ 1/4, for all values of p, ˆ we can write the foregoing inequality as zα/2 pˆ − p ≤ √ n

8

zα/2 1 = √ . 4 2 n

If we wish to estimate p at level (1 − α) to within d units of its true value, that is |pˆ − p| ≤ d, the √ sample size must satisfy the condition (zα/2 /(2 n)) ≤ d, or n≥

z2α/2 4d 2

.

Thus, to estimate p at level (1 − α) to within d units of its true value, take the minimal sample size as n = z2α/2 /4d 2 , and if this is not an integer, round up to the next integer. Sometimes, we may have an initial estimate p˜ of the parameter p from a similar process or from a pilot study or simulation. In this case, we can use the following formula to compute the minimum required size of the sample to estimate p, at level (1 − α), to within d units by using the formula n=

z2α/2 p(1 ˜ − p) ˜ d2

and, if this is not an integer, rounding up to the next integer.

6.2 Large Sample Conﬁdence Intervals: One Sample Case 305

A similar derivation for calculation of sample size for estimation of the population mean μ at level (1 − α) with margin of error E is given by n=

z2α/2 σ 2 E2

and, if this is not an integer, rounding up to the next integer. This formula can be used only if we know the population standard deviation, σ. Although it is unlikely to know σ when the population mean itself is not known, we may be able to determine σ from an earlier similar study or from a pilot study/simulation.

Example 6.2.5 A dendritic tree is a branched formation that originates from a nerve cell. In order to study brain development, researchers want to examine the brain tissues from adult guinea pigs. How many cells must the researchers select (randomly) so as to be 95% sure that the sample mean is within 3.4 cells of the population mean? Assume that a previous study has shown σ = 10 cells.

Solution A 95% confidence corresponds to α = 0.05. Thus, from the normal table, zα/2 = z0.025 = 1.96. Given that E = 3.4 and σ = 10, and using the sample size formula, the required sample size n is n=

z2α/2 σ 2 E2

=

(1.96)2 (10)2 = 33.232. (3.4)2

Thus, take n = 34.

Example 6.2.6 Suppose that a local TV station in a city wants to conduct a survey to estimate support for the president’s policies on economy within 3% error with 95% conﬁdence. (a) How many people should the station survey if they have no information on the support level? (b) Suppose they have an initial estimate that 70% of the people in the city support the economic policies of the president. How many people should the station survey?

Solution Here α = 0.05, and thus zα/2 = 1.96. Also, d = 0.03. (a) With no information on p, we use the sample size formula: n=

z2α/2 4d 2

=

(1.96)2 = 1067.1. 4(0.03)2

Hence, the TV station must survey 1068 people.

306 CHAPTER 6 Interval Estimation

(b) Because p˜ = 0.7, the required sample size is calculated from n= =

z2α/2 p(1 ˜ − p) ˜ d2 (1.96)2 (0.70)(0.30) = 896.37. (0.03)2

Thus, the TV station must survey at least 897 people.

In practice, we should realize that one of the key factors of a good design is not sample size by itself; it is getting representative samples. Even if we have a very large sample size, if the sample is not representative of our target population, then sample size means nothing. Therefore, whenever possible, we should use random sampling procedures (or other appropriate sampling procedures) to ensure that our target population is properly represented.

EXERCISES 6.2 6.2.1.

A survey indicates that it is important to pay attention to truth in political advertising. Based on a survey of 1200 people, 35% indicated that they found political advertisements to be untrue; 60% say that they will not vote for candidates whose advertisements are judged to be untrue; and of this latter group, only 15% ever complained to the media or to the candidate about their dissatisfaction. (a) Find a 95% conﬁdence interval for the percentage of people who ﬁnd political advertising to be untrue. (b) Find a 95% conﬁdence interval for the percentage of voters who will not vote for candidates whose advertisements are considered to be untrue. (c) Find a 95% conﬁdence interval for the percentage of those who avoid voting for candidates whose advertisements are considered untrue and who have complained to the media or to the candidate about the falsehood in commercials. (d) For each case above, interpret the results and state any assumptions you have made.

6.2.2.

Many mutual funds use an investment approach involving owning stocks whose price/earnings multiples (P/Es) are less than the P/E of the S&P 500. The following data give P/Es of 49 companies a randomly selected mutual fund owns in a particular year. 6.8 5.6 8.5 8.5 8.4 7.5 9.3 9.4 9.9 9.6 9.0 9.4 13.7 16.6 9.1 10.1 8.9 11.7 12.8 11.5 12.0 10.6 11.1 6.4 11.4 9.9 14.3 11.5 11.8 13.3 12.8 13.7 14.2 14.0 15.5 16.9 18.0 17.9 21.8 18.4

7.8 7.1 10.6 11.1 12.3 12.3 13.9 12.9 34.3

Find a 98% conﬁdence interval for the mean P/E multiples. Interpret the result and state any assumptions you have made.

6.2 Large Sample Conﬁdence Intervals: One Sample Case 307

6.2.3.

Let X1 , . . . , Xn be a random sample from N(μ, σ 2 ) distribution, σ 2 known. (a) Show that μ ˆ = X is a maximum likelihood estimator of the population mean μ. (b) Show that

2σ 2σ P X− √ Basic Statistics > 1-sample t. . . , in variables: enter C1, click Confidence interval, in Level default value is 95, if any other value, enter that value, and click OK

6.7 Computer Examples 331

We will obtain the following output. T Confidence Intervals Variable N C1 6

Mean 5.898

StDev 1.968

SE Mean 0.804

95.0% C.I. (3.832, 7.964)

Example 6.7.2 (Large Sample): For the data 6.8 5.6 8.5 8.5 8.4 7.5 9.3 9.4 7.8 7.1 9.9 9.6 9.0 13.7 9.4 16.6 9.1 10.1 10.6 11.1 8.9 11.7 12.8 11.5 10.6 12.0 11.1 6.4 12.3 12.3 11.4 9.9 15.5 14.3 11.5 13.3 11.8 12.8 13.7 13.9 12.9 14.2 14.0 obtain a 98% conﬁdence interval for μ.

Solution Enter the data in C1. Then click Stat > Basic Statistics > 1-Sample Z. . . > in Variables: type C1 > click Confidence interval, and enter 98 in Level: > enter 5 in Sigma: > OK We will obtain the following output. THE ASSUMED SIGMA = 5.00 Variable N MEAN STDEV SE MEAN 98.0 PERCENT C.I. C1 49 12.124 4.700 0.714 (10.462, 13.787)

Example 6.7.3 For the following data, ﬁnd a 90% conﬁdence interval for μ1 − μ2 Sample 1 Sample 2

1.2 4.2

3.1 2.7

1.7 3.6

2.8 3.9

3.0

Solution Enter sample 1 in C1 and sample 2 in C2. Then click

Stat > Basic Statistics > 2-Sample t. . . > click Sample in different columns > in First: enter C1 and in Second: enter C2 > enter 90 in Confidence Level: (if equality of variance can be assumed, click Assume equal variances) > OK

332 CHAPTER 6 Interval Estimation

We will obtain the following output: TWOSAMPLE T FOR C1 VS C2

C1 C2

N 5 4

MEAN 2.360 3.600

STDEV 0.856 0.648

SE MEAN 0.38 0.32

90 PCT CI FOR MU C1 − MU C2: (−2.22, −0.26) TTEST MU C1 = MU C2 (VS NE): T = −2.39 P = 0.048 DF = 7 POOLED STDEV = 0.774

6.7.2 SPSS Examples Example 6.7.4 Consider the data 66

74

79

80

77

78

65

79

81

69

Using SPSS, obtain a 99% conﬁdence interval for μ.

Solution One easy way to obtain the confidence interval in SPSS is to use the hypothesis testing procedure. The procedure is as follows: First enter the data in C1. Then click Analyze > Compare Means > One-sample t Test. . . , > Move var00001 to Test Variable(s), and Click Options . . . , and enter 99 in Confidence interval:, click Continue, and OK Note that the default value is 95%. We will obtain the following output: One-Sample Statistics N Mean Std. deviation VAR00001 10 74.8000 5.99630

Std. error mean 1.89620

One-Sample Test Test Value = 0 99% Confidence interval of the Mean difference t df Sig.(2-tailed) difference Lower Upper VAR00000 39.447 9 .000 74.8000 68.6377 80.9623 From this, we obtain the 99% confidence interval as (68.6377, 80.9623).

6.7 Computer Examples 333

6.7.3 SAS Examples Example 6.7.5 The following data give P/E for a particular year of 49 mutual fund companies owned by a randomly selected mutual fund. 6.8 9.9 12.8 11.8 34.3

5.6 9.6 11.5 13.3 13.7

8.5 9.0 12.0 13.9 12.3

8.5 16.6 10.6 12.9 18.0

8.4 9.1 11.1 14.2 9.4

7.5 10.1 6.4 14.0 12.3

9.3 10.6 11.4 15.5 16.9

9.4 11.1 9.9 17.9 12.8

7.8 8.9 14.3 21.8 13.7

7.1 11.7 11.5 18.4

Find a 98% conﬁdence interval for the mean P/E multiples. Use SAS procedures.

Solution We could use the following procedure. DATA peratio; INPUT patio @@; DATALINES; 6.8 5.6 8.5 8.5 8.4 7.5 9.3 9.4 7.8 7.1 9.9 9.6 9.0 9.4 13.7 16.6 9.1 10.1 11.1 8.9 11.7 12.8 11.5 12.0 10.6 11.1 6.4 12.3 11.4 9.9 14.3 11.5 11.8 13.3 12.8 13.7 14.2 14.0 15.5 16.9 18.0 17.9 21.8 18.4 34.3 ; PROC MEANS data = peratio lclm uclm alpha = 0.02; var peratio; RUN;

10.6 12.3 13.9

12.9

We will obtain the following output: The MEANS Procedure Analysis Variable : peratio Lower 98% Upper 98% CL for Mean CL for Mean -----------------------------------------------------------10.5084971 13.7404825 -----------------------------------------------------------Hence, we will obtain the 98% confidence interval for the P/E ratios as (10.50, 13.74).

EXERCISES 6.7 6.7.1.

Using any of the software packages (Minitab, SPSS, or SAS), obtain conﬁdence intervals for at least one data set taken from each section of this chapter.

334 CHAPTER 6 Interval Estimation

PROJECTS FOR CHAPTER 6 6A. Simulation of Coverage of the Small Confidence Intervals for μ (a) Generate 25 samples of size 15 from a normal population with μ = 10 and σ 2 = 4. Using a statistical package (such as Minitab), compute the 95% conﬁdence intervals for each of the samples using the small sample formula. From your output, determine the proportion of the 25 intervals that cover the true mean μ = 10. (b) What would you expect if the sample size is increased to 100? Would the width of the interval increase or decrease? Would you expect more or fewer of these intervals to contain the true mean 10? Check your answers with actual computation. (c) Repeat with 20 samples of size 10.

6B. Confidence Intervals Based on Sampling Distributions If we want to obtain a (1 − α)100% conﬁdence interval for θ, begin with an estimator θˆ of θ and determine its sampling distribution. Now select two probability levels, α1 and α2 , so that α = α1 +α2 . Generally we let α1 = α2 . Take a sample and calculate the value of θˆ , say θˆ = k. Now we need to determine the values of the upper and lower conﬁdence limits. Find a value θL such that p (θˆ ≥ k) = α1

and θU such that p (θˆ ≤ k) = α2 .

Then a (1 − α)100% conﬁdence interval for θ will be θL < θ < θU .

(a) Let X1 , . . . , Xn be a random sample from U(0, θ) distribution. Obtain a (1 − α)100% conﬁdence interval for θ, using the method of sampling distribution. (b) Let X have a binomial distribution with parameters n and p. First show that there is no quantity that satisﬁes the conditions of a pivotal quantity. Then using the method of sampling distributions, obtain a (1 − α)100% conﬁdence interval for p.

6C. Large Sample Confidence Intervals: General Case The method of ﬁnding a conﬁdence interval for a parameter θ that we described in this chapter depends on our ability to ﬁnd the pivotal quantity. We have seen that such a quantity may not exist. In those cases, the method of sampling distribution described in the previous project could be used. However, this method can involve some difﬁcult calculations. For large samples, we can utilize the following procedure, which is based on the asymptotic distribution of maximum likelihood estimators. Under fairly general conditions, the maximum likelihood estimators have a limiting distribution that is normal. Also, maximum likelihood estimators are asymptotically efﬁcient. Hence, for a large sample

Projects for Chapter 6 335

the maximum likelihood estimator θˆ of θ will have approximately normal distribution with mean θ. Also, if the Cramér–Rao lower bound exists, the limiting variance of θˆ will be σ 2ˆ = θ

! E

1

∂ ln L 2 ∂θ

".

Hence, Z=

θˆ − θ ∼ N(0, 1). σθˆ

Then a large sample (1 − α)100% conﬁdence interval is obtained from the probability statement

θˆ − θ P −zα/2 < < zα/2 σθˆ

≈ 1 − α.

We summarize the procedure to construct large sample conﬁdence intervals. 1. Determine the maximum likelihood estimator, θˆ , of θ. Also ﬁnd the maximum likelihood estimators of all other unknown parameters. 2. Obtain the variance σθˆ (if possible directly, otherwise by using the Cramér–Rao lower bound). 3. In the expression for σθˆ , substitute θˆ for θ. Replace all other unknown parameters by its maximum likelihood estimators. Let the resulting quantity be denoted by sθˆ . 4. Now construct a (1 − α)100% conﬁdence interval for θ from θˆ − zα/2 sθˆ < θ < θˆ + zα/2 sθˆ .

(a) Using the foregoing procedure, show that a large sample (1 − α)100% conﬁdence interval for the parameter p in a binomial distribution based on n trials is 8 pˆ − zα/2

p(1 ˆ − p) ˆ < p < pˆ + zα/2 n

8

p(1 ˆ − p) ˆ . n

(b) Let X1 , . . . , Xn be a random sample from a normal population with parameters μ and σ 2 . Derive a large sample conﬁdence interval for σ 2 using the above procedure. (c) Let X1 , . . . , Xn be a random sample from a population with a pdf ⎧ ⎪ ⎨ 1 e−x/θ , f (x) = θ ⎪ ⎩0,

Derive a large sample conﬁdence interval for θ.

x>0 otherwise.

336 CHAPTER 6 Interval Estimation

6D. Prediction Interval for an Observation from a Normal Population In many cases, we may be interested in predicting future observations from a population, rather than making an inference. A (1 − α)100% prediction interval for a future observation X is an interval of the form (XL , XU ) such that p (XL < X < XU ) = 1 − α. Similarly to conﬁdence intervals, we can also deﬁne one-sided prediction intervals. Assume that the population is normal with known variance σ 2 . Let X1 , . . . , Xn be a random sample from this population. Then the sampling distribution of 2 = the difference X − X (we use X to denote Xn ) is normal with mean zero and variance σ 2 + σX (1 + (1/n))σ 2 . Then a (1 − α)100% prediction interval for X is given by +

X − zα/2

+ 1 1 1+ 1+ σ 2 , X + zα/2 σ2 . n n

Thus, we are (1 − α)100% conﬁdent that the next observation, Xn+1 , will lie in this interval. As in conﬁdence intervals, if the sample size is large, replace σ by sample standard deviation s. In case, where both μ and σ are not known, and the sample size is small (so that the Central Limit $ # √ Theorem cannot be applied), it can be shown that (Xn+1 − Xn )/(Sn 1 + (1/n)) has a t-distribution with (n − 1) degrees of freedom. Thus, a (1 − α)100% prediction interval for Xn+1 is given by

9 9 X − tα/2,n−1 (1 + (1/n))S 2 , X + tα/2,n−1 (1 + (1/n))S 2 .

A standard measure of the capacity of lungs to expel air in breathing is called forced expiratory volume (FEV). The FEV1 is the volume exhaled during the ﬁrst second of a forced expiratory maneuver started from the level of total lung capacity. The following data (source: M. Bland, An Introduction to Medical Statistics, Oxford University Press, 1995) represents FEV measurements (in liters) from 57 male medical students. 4.47 4.47 3.48 5.00 3.42 3.78

3.10 3.57 4.20 4.50 3.60 3.75

4.50 4.90 3.50 2.85 5.10 5.20 3.70 5.30 4.71 4.20 4.16 3.70 3.20 4.56 4.78 4.05 3.54 4.14

4.14 4.32 4.80 5.10 4.10 4.30 3.83 3.90 3.60 3.96 2.98 3.54

Obtain a 95% prediction interval for a future observation Xn+1 .

4.80 3.10 4.30 4.70 3.39 3.69 4.47 3.30 3.19 2.85

4.68 4.08 4.44 5.43 3.04

Chapter

7

Hypothesis Testing Objective: In this chapter, various methods of testing hypotheses will be discussed. 7.1 Introduction 338 7.2 The Neyman–Pearson Lemma 349 7.3 Likelihood Ratio Tests 355 7.4 Hypotheses for a Single Parameter 361 7.5 Testing of Hypotheses for Two Samples 372 7.6 Chi-Square Tests for Count Data 388 7.7 Chapter Summary 399 7.8 Computer Examples 399 Projects for Chapter 7 408

Jerzy Neyman (Source: http://sciencematters.berkeley.edu/archives/volume2/issue12/legacy.php)

Jerzy Neyman (1894–1981) made far-reaching contributions in hypothesis testing, conﬁdence intervals, probability theory, and other areas of mathematical statistics. His work with Egon Pearson gave

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

337

338 CHAPTER 7 Hypothesis Testing

logical foundation and mathematical rigor to the theory of hypothesis testing. Their ideas made sure that samples were large enough to avoid false representation. Neyman made a broader impact in statistics throughout his lifetime.

7.1 INTRODUCTION Statistics plays an important role in decision making. In statistics, one utilizes random samples to make inferences about the population from which the samples were obtained. Statistical inference regarding population parameters takes two forms: estimation and hypothesis testing, although both hypothesis testing and estimation may be viewed as different aspects of the same general problem of arriving at decisions on the basis of observed data. We already saw several estimation procedures in earlier chapters. Hypothesis testing is the subject of this chapter. Hypothesis testing has an important role in the application of statistics to real-life problems. Here we utilize the sampled data to make decisions concerning the unknown distribution of a population or its parameters. Pioneering work on the explicit formulation as well as the fundamental concepts of the theory of hypothesis testing are due to J. Neyman and E. S. Pearson. A statistical hypothesis is a statement concerning the probability distribution of a random variable or population parameters that are inherent in a probability distribution. The following example illustrates the concept of hypothesis testing. An important industrial problem is that of accepting or rejecting lots of manufactured products. Before releasing each lot for the consumer, the manufacturer usually performs some tests to determine whether the lot conforms to acceptable standards. Let us say that both the manufacturer and the consumer agree that if the proportion of defectives in a lot is less than or equal to a certain number p, the lot will be released. Very often, instead of testing every item in the lot, we may test only a few items chosen at random from the lot and make decisions about the proportion of defectives in the lot; that is, we make the decisions about the population on the basis of sample information. Such decisions are called statistical decisions. In attempting to reach decisions, it is useful to make some initial conjectures about the population involved. Such conjectures are called statistical hypotheses. Sometimes the results from the sample may be markedly different from those expected under the hypothesis. Then we can say that the observed differences are signiﬁcant and we would be inclined to reject the initial hypothesis. These procedures that enable us to decide whether to accept or reject hypotheses or to determine whether observed samples differ signiﬁcantly from expected results are called tests of hypotheses, tests of signiﬁcance, or rules of decision. In any hypothesis testing problem, we formulate a null hypothesis and an alternative hypothesis such that if we reject the null, then we have to accept the alternative. The null hypothesis usually is a statement of either the “status quo” or “no effect.” A guideline for selecting a null hypothesis is that when the objective of an experiment is to establish a claim, the nulliﬁcation of the claim should be taken as the null hypothesis. The experiment is often performed to determine whether the null hypothesis is false. For example, suppose the prosecution wants to establish that a certain person is guilty. The null hypothesis would be that the person is innocent and the alternative would be that the person is guilty. Thus, the claim itself becomes the alternative hypothesis. Customarily, the alternative hypothesis is the statement that the experimenter believes to be true. For example, the alternative hypothesis is the reason a person is arrested (police suspect the person is not innocent). Once the hypotheses

7.1 Introduction 339

have been stated, appropriate statistical procedures are used to determine whether to reject the null hypothesis. For the testing procedure, one begins with the assumption that the null hypothesis is true. If the information furnished by the sampled data strongly contradicts (beyond a reasonable doubt) the null hypothesis, then we reject it in favor of the alternative hypothesis. If we do not reject the null, then we automatically reject the alternative. Note that we always make a decision with respect to the null hypothesis. Note that the failure to reject the null hypothesis does not necessarily mean that the null hypothesis is true. For example, a person being judged “not guilty” does not mean the person is innocent. This basically means that there is not enough evidence to reject the null hypothesis (presumption of innocence) beyond “a reasonable doubt.” We summarize the elements of a statistical hypothesis in the following. THE ELEMENTS OF A STATISTICAL HYPOTHESIS 1. The null hypothesis, denoted by H0 , is usually the nulliﬁcation of a claim. Unless evidence from the data indicates otherwise, the null hypothesis is assumed to be true. 2. The alternate hypothesis, denoted by Ha (or sometimes denoted by H1 ), is customarily the claim itself. 3. The test statistic, denoted by TS, is a function of the sample measurements upon which the statistical decision, to reject or not reject the null hypothesis, will be based. 4. A rejection region (or a critical region) is the region (denoted by RR) that speciﬁes the values of the observed test statistic for which the null hypothesis will be rejected. This is the range of values of the test statistic that corresponds to the rejection of H0 at some ﬁxed level of signiﬁcance, α, which will be explained later. 5. Conclusion: If the value of the observed test statistic falls in the rejection region, the null hypothesis is rejected and we will conclude that there is enough evidence to decide that the alternative hypothesis is true. If the TS does not fall in the rejection region, we conclude that we cannot reject the null hypothesis.

In practice one may have hypotheses such as H0 : μ = μ0 against one of the following alternatives: ⎧ ⎪ ⎪ ⎪ ⎨ or ⎪ or ⎪ ⎪ ⎩

Ha : μ = μ0 , Ha : μ < μ0 , Ha : μ > μ0 ,

called a two-tailed alternative called a lower (or left) tailed alternative called an upper (or right) tailed alternative

A test with a lower or upper tailed alternative is called a one-tailed test. In an applied hypothesis testing problem, we can use the following general steps. GENERAL METHOD FOR HYPOTHESIS TESTING 1. From the (word) problem, determine the appropriate null hypothesis, H0 , and the alternative, Ha . 2. Identify the appropriate test statistics and calculate the observed test statistic from the data. 3. Find the rejection region by looking up the critical value in the appropriate table. 4. Draw the conclusion: Reject or fail to reject the null hypothesis, H0 . 5. Interpret the results: State in words what the conclusion means to the problem we started with.

340 CHAPTER 7 Hypothesis Testing

It is always necessary to state a null and an alternate hypothesis for every statistical test performed. All possible outcomes should be accounted for by the two hypotheses.

Example 7.1.1 In a coin-tossing experiment, let p be the probability of heads. We start with the claim that the coin is fair, that is, H0 : p = 1/2. We test this against one of the following alternatives: (a) Ha : The coin is not fair (p = 1/2). This is a two-tailed alternative. (b) Ha : The coin is biased in favor of heads (p > 1/2). This is an upper tailed alternative. (c) Ha : The coin is biased in favor of tails (p < 1/2). This is a lower tailed alternative.

It is important to observe that the test statistic is a function of a random sample. Thus, the test statistic itself is a random variable whose distribution is known under the null hypothesis. The value of a test statistic when speciﬁc sample values are substituted is called the observed test statistic or simply test statistic. For example consider the hypothesis H0 : μ = μo versus Ha : μ = μo , where μo is known. Assume that the population is normal with a known variance σ 2 . Consider X, an unbiased estimator of μ √ based on the random sample X1 , . . . , Xn . Then Z = (X − μ0 )/(σ/ n) is a function of the random sample X1 , . . . , Xn , and has a known distribution, a standard normal, under H0 . If x1 , x2 , . . . , xn are √ speciﬁc sample values, then z = (x − μ0 )/(σ/ n) is called the observed sample statistic or simply sample statistic. Deﬁnition 7.1.1 A hypothesis is said to be a simple hypothesis if that hypothesis uniquely speciﬁes the distribution from which the sample is taken. Any hypothesis that is not simple is called a composite hypothesis.

Example 7.1.2 Refer to Example 7.1.1. The null hypothesis p =1/2 is simple, because the hypothesis completely speciﬁes the distribution, which in this case will be a binomial with p = 1/2 and with n being the number of tosses. The alternative hypothesis p = 1/2 is composite because the distribution now is not completely speciﬁed (we do not know the exact value of p).

Because the decision is based on the sample information, we are prone to commit errors. In a statistical test, it is impossible to establish the truth of a hypothesis with 100% certainty. There are two possible types of errors. On the one hand, one can make an error by rejecting H0 when in fact it is true. On the other hand, one can also make an error by failing to reject the null hypothesis when in fact it is false. Because the errors arise as a result of wrong decisions, and the decisions themselves are based on random samples, it follows that the errors have probabilities associated with them. We now have the following deﬁnitions.

7.1 Introduction 341

Table 7.1 Statistical Decision and Error Probabilities Statistical

True state of null hypothesis

decision

H 0 true

H 0 false

Do not reject H0

Correct decision

Type II error (β)

Reject H0

Type I error (α)

Correct decision

The decision and the errors are represented in Table 7.1. Deﬁnition 7.1.2 (a) A type I error is made if H0 is rejected when in fact H0 is true. The probability of type I error is denoted by α. That is, P (rejecting H0 |H0 is true) = α.

The probability of type I error, α, is called the level of signiﬁcance. (b) A type II error is made if H0 is accepted when in fact Ha is true. The probability of a type II error is denoted by β. That is, P (not rejecting H0 |H0 is false) = β.

It is desirable that a test should have a = β = 0 (this can be achieved only in trivial cases), or at least we prefer to use a test that minimizes both types of errors. Unfortunately, it so happens that for a ﬁxed sample size, as α decreases, β tends to increase and vice versa. There are no hard and fast rules that can be used to make the choice of α and β. This decision must be made for each problem based on quality and economic considerations. However, in many situations it is possible to determine which of the two errors is more serious. It should be noted that a type II error is only an error in the sense that a chance to correctly reject the null hypothesis was lost. It is not an error in the sense that an incorrect conclusion was drawn, because no conclusion is made when the null hypothesis is not rejected. In the case of type I error, a conclusion is drawn that the null hypothesis is false when, in fact, it is true. Therefore, type I errors are generally considered more serious than type II errors. For example, it is mostly agreed that ﬁnding an innocent person guilty is a more serious error than ﬁnding a guilty person innocent. Here, the null hypothesis is that the person is innocent, and the

Prob (TYPE II Error) 5 Beta Under H 0

Prob (TYPE I Error) 5 Alpha Under Ha

Critical value

342 CHAPTER 7 Hypothesis Testing

alternate hypothesis is that the person is guilty. “Not rejecting the null hypothesis” is equivalent to acquitting a defendant. It does not prove that the null hypothesis is true, or that the defendant is innocent. In statistical testing, the signiﬁcance level α is the probability of wrongly rejecting the null hypothesis when it is true (that is, the risk of ﬁnding an innocent person guilty). Here the type II risk is acquitting a guilty defendant. The usual approach to hypothesis testing is to ﬁnd a test procedure that limits α, the probability of type I error, to an acceptable level while trying to lower β as much as possible. The consequences of different types of errors are, in general, very different. For example, if a doctor tests for the presence of a certain illness, incorrectly diagnosing the presence of the disease (type I error) will cause a waste of resources, not to mention the mental agony to the patient. On the other hand, failure to determine the presence of the disease (type II error) can lead to a serious health risk. To formulate a hypothesis testing problem, consider the following situation. Suppose a toy store chain claims that at least 80% of girls under 8 years old prefer dolls over other types of toys. We feel that this claim is inﬂated. In an attempt to dispose of this claim, we observe the buying pattern of 20 randomly selected girls under 8 years old, and we observe X, the number of girls under 8 years old who buy stuffed toys or dolls. Now the question is, how can we use X to conﬁrm or reject the store’s claim? Let p be the probability that a girl under 8 chosen at random prefers stuffed toys or dolls. The question now can be reformulated as a hypothesis testing problem. Is p ≥ 0.8 or p < 0.8? Because we would like to reject the store’s claim only if we are highly certain of our decision, we should choose the null hypothesis to be H0 : p ≥ 0.8, the rejection of which is considered to be more serious. The null hypothesis should be H0 : p ≥ 0.8, and the alternative Ha : p < 0.8. In order to make the null hypothesis simple, we will use H0 : p = 0.8, which is the boundary value with the understanding that it really represents H0 : p ≥ 0.8. We note that X, the number of girls under 8 years old who prefer stuffed toys or dolls, is a binomial random variable. Clearly a large sample value of X would favor H0 . Suppose we arbitrarily choose to accept the null hypothesis if X > 12. Because our decision is based on only a sample of 20 girls under 8, there is always a possibility of making errors whether we accept or reject the store chain’s claim. In the following example, we will now formally state this problem and calculate the error probabilities based on our decision rule.

Example 7.1.3 A toy store chain claims that at least 80% of girls under 8 years old prefer dolls over other types of toys. After observing the buying pattern of many girls under 8 years old, we feel that this claim is inﬂated. In an attempt to dispose of this claim, we observe the buying pattern of 20 randomly selected girls under 8 years old, and we observe X, the number of girls who buy stuffed toys or dolls. We wish to test the hypothesis H0 : p = 0.8 against Ha : p < 0.8. Suppose we decide to accept the H0 if X > 12 (that is X ≥ 13). This means that if {X ≤ 12} (that is X < 13) we will reject H0 . (a) Find α. (b) Find β for p = 0.6. (c) Find β for p = 0.4. (d) Find the rejection region of the form {X ≤ K} so that (i) α = 0.01; (ii) α = 0.05. (e) For the alternative Ha :p = 0.6, ﬁnd β for the values of α in part (d).

7.1 Introduction 343

Solution The TS X is the number of girls under 8 years old who buy dolls. X follows the binomial distribution with n = 20 and p, the unknown population proportion of girls under 8 who prefer dolls. We now calculate α and β. (a) For p = 0.8, the probability of type I error is α = P{reject H0 |H0 is true} = P{X ≤ 12|p = 0.8} =

12 20 x=0

x

(0.8)x (0.2)20−x

= 0.0321. If we calculate α for any other value of p > 0.8, then we will find that it is smaller than 0.0321. Hence, there is at most a 3.21% chance of rejecting a true null hypothesis. That is, if the store’s claim is in fact true, then the chance that our test will erroneously reject that claim is at most 3.21%. (b) Here p = 0.6. The probability of type II error is β = P{accept H0 |H0 false} = P{X > 12|p = 0.6} = 1 − P{X ≤ 12|p = 0.6} = 1 − 0.584 = 0.416 so there is a 4.2% chance of accepting a false null hypothesis. Thus, in case the store’s claim is not true, and the truth is that only 60% of girls under 8 years old prefer dolls over other types of toys, then there is a 4.2% chance that our test will erroneously conclude that the store’s claim is true. (c) If p = 0.4, then β = P{accept H0 |H0 false} = P{X > 12|p = 0.4} = 1 − P{X ≤ 12|p = 0.4} = 1 − 0.979 = 0.021. That is, there is a 2.1% chance of accepting a false null hypothesis. (d) (i) To find K such that α = P{X ≤ K|p = 0.8} = 0.01 from the binomial table, K = 11. Hence, the rejection region is: Reject H0 if {X ≤ 11}. (ii) To find K such that α = P{X ≤ K|p = 0.8} = 0.05

344 CHAPTER 7 Hypothesis Testing

from the binomial table, α = 0.05 falls between K = 12 and K = 13. However, for K = 13, the value for α is 0.087, exceeding 0.05. If we want to limit α to be no more than 0.05, we will have to take K = 12. That is, we reject the null hypothesis if X ≤ 12, yielding an α = 0.0321 as shown in (a). (e) (i) When a = 0.01, from (d), the rejection region is of the form {X ≤ 11}. For p = 0.6, β = P{accept H0 |H0 false} = P{Y > 11|p = 0.6} = 1 − P{Y ≤ 11|p = 0.6} = 1 − 0.404 = 0.596. (ii) From (a) and (b) for testing the hypothesis H0 : p = 0.8 against Ha : p < 0.8 with n = 20. We see that when α is 0.0321, β is 0.416. From (d)(i) and (e)(i) for the same hypothesis, we see that when α is 0.01, β is 0.596. This holds in general. Thus, we observe that for fixed n as α decreases, β increases and vice versa.

In the next example, we explore what happens to β as the sample size increases, with α ﬁxed.

Example 7.1.4 Let X be a binomial random variable. We wish to test the hypothesis H0 : p = 0.8 against Ha : p = 0.6. Let α = 0.03 be ﬁxed. Find β for n = 10 and n = 20.

Solution

For n = 10, using the binomial tables, we obtain P{X ≤ 5|p = 0.8} ∼ = 0.03. Hence the rejection region for the hypothesis H0 : p = 0.8 vs. Ha : p = 0.6 is given by reject H0 if X ≤ 5. The probability of type II error is β = P{accept H0 |p = 0.6} β = P{X > 5|p = 0.6} = 1 − P{X ≤ 5|p = 0.6} = 0.733. For n = 20, as shown in Example 7.1.3, if we reject H0 for X ≤ 12, we obtain P(X ≤ 12|p = 0.8) ∼ = 0.03 and β = P(X > 12|p = 0.6) = 1 − P{X ≤ 12|p = 0.6} = 0.416. We see that for a fixed α, as n increases β decreases and vice versa. It can be shown that this result holds in general.

7.1 Introduction 345

In order for us to compute the value of β, it is necessary that the alternate hypothesis is simple. Now we will discuss a three-step procedure to calculate β.

STEPS TO CALCULATE β 1. Decide an appropriate test statistic (usually this is a sufﬁcient statistic or an estimator for the unknown parameter, whose distribution is known under H0 ). 2. Determine the rejection region using a given α, and the distribution of the test statistic (TS). 3. Find the probability that the observed test statistic does not fall in the rejection region assuming Ha is true. This gives β. That is, β = P(T .S. falls in the complement of the rejection region|Ha is true).

Example 7.1.5 A random sample of size 36 from a population with known variance, σ 2 = 9, yields a sample mean of x = 17. Find β, for testing the hypothesis H0 : μ = 15 versus Ha : μ = 16. Assume α = 0.05.

Solution Here n = 36, x = 17, and σ 2 = 9. In general, to test H0 : μ = μ0 versus Ha : μ > μ0 , we proceed as follows. An unbiased estimator of μ is X. Intuitively we would reject H0 if X is large, say X > c. Now using α = 0.05, we will determine the rejection region. By the definition of α, we have P (X > c |μ = μ0 ) = 0.05 or P

X − μ0 c − μ0 √ > √ |μ = μ0 σ/ n σ/ n

= 0.05

√ √0 > But if μ = μ0 , because the sample size n ≥ 30, [(X − μ0 )/(σ/ n)] ∼ N(0, 1). Therefore, P X−μ (σ/ n) c−μ √ 0 = 0.05 is equivalent to P Z > c−μ √ 0 = 0.05. From standard normal tables, we obtain P (Z > (σ/ n) (σ/ n) √ √ 0 = 1.645 or c = μ0 + 1.645(σ/ n). 1.645) = 0.05. Hence c−μ (σ/ n)

Therefore, the rejection region is the set of all sample means x such that

σ x > μ0 + 1.645 √ . n Substituting μ0 = 15, and σ = 3, we obtain √ μ0 + 1.645(σ/ n) = 15 + 1.645 The rejection region is the set of x such that x ≥ 15.8225.

3 36

= 15.8225.

346 CHAPTER 7 Hypothesis Testing

Then by definition, β = P (X ≤ 15.8225 when μ = 16). Consequently, for μ = 16,

β=P

X − 16 15.8225 − 16 √ √ ≤ σ/ n 3/ 36

= P (Z ≤ −0.36) = 0.3594. That is, under the given information, there is a 35.94% chance of not rejecting a false null hypothesis.

7.1.1 Sample Size It is clear from the preceding example that once we are given the sample size n, an α, a simple alternative Ha , and a test statistic, we have no control over β and it is exactly determined. Hence, for a given sample size and test statistic, any effort to lower β will lead to an increase in α and vice versa. This means that for a test with ﬁxed sample size it is not possible to simultaneously reduce both α and β. We also notice from Example 7.1.4 that by increasing the sample size n, we can decrease β (for the same α) to an acceptable level. The following discussion illustrates that it may be possible to determine the sample size for a given α and β. Suppose we want to test H0 : μ = μ0 versus Ha : μ > μ0 . Given α and β, we want to ﬁnd n, the sample size, and K, the point at which the rejection begins. We know that α = P (X > K when μ = μ0 ) X − μ0 K − μ0 =P √ > √ , when μ = μ0 σ/ n σ/ n

(7.1)

= P (Z > za )

and β = P (X ≤ K, when μ = μa ) X − μa K − μa =P √ ≤ √ , when μ = μa σ/ n σ/ n = P (z ≤ −zβ ).

From Equations (7.1) and (7.2), zα =

K − μ0 √ σ/ n

(7.2)

7.1 Introduction 347

and −zβ =

K − μa √ . σ/ n

This gives us two equations with two unknowns (K and n), and we can proceed to solve them. Eliminating K, we get

μ0 + zα

σ √ n

= μa − zβ

σ . √ n

From this we can derive √

n=

(zα + zβ )σ . μa − μ 0

Thus, the sample size for an upper tail alternative hypothesis is n=

(zα + zβ )2 σ 2 (μa − μ0 )2

.

The sample size increases with the square of the standard deviation and decreases with the square of the difference between mean value of the alternative hypothesis and the mean value under the null hypothesis. Note that in real-world problems, care should be taken in the choice of the value of μa for the alternative hypothesis. It may be tempting for a researcher to take a large value of μa in order to reduce the required sample size. This will seriously affect the accuracy (power) of the test. This alternative value must be realistic within the experiment under study. Care should also be taken in the choice of the standard deviation σ. Using an underestimated value of the standard deviation to reduce the sample size will result in inaccurate conclusions similar to overestimating the difference of means. Usually, the value of σ is estimated using a similar study conducted earlier. The problem could be that the previous study may be old and may not represent the new reality. When accuracy is important, it may be necessary to conduct a pilot study only to get some idea on the estimate of σ. Once we determine the necessary sample size, we must devise a procedure by which the appropriate data can be randomly obtained. This aspect of the design of experiments is discussed in Chapter 9.

Example 7.1.6 Let σ = 3.1 be the true standard deviation of the population from which a random sample is chosen. How large should the sample size be for testing H0 : μ = 5 versus Ha : μ = 5.5, in order that α = 0.01 and β = 0.05?

Solution We are given μ0 = 5 and μa = 5.5. Also, zα = z0.01 = 2.33 and zβ = z0.05 = 1.645. Hence, the sample size n=

(zα + zβ )2 σ 2 (μa − μ0 )2

=

(2.33 + 1.645)2 (3.1)2 = 607.37. (0.5)2

348 CHAPTER 7 Hypothesis Testing

So, n = 608 will provide the desired levels. That is, in order for us to test the foregoing hypothesis, we must randomly select 608 observations from the given population.

From a practical standpoint, the researcher typically chooses α, and the sample size β is ignored. Because a trade-off exists between α and β, choosing a very small value of α will tend to increase β in a serious way. A general rule of thumb is to pick reasonable values of α, possibly in the 0.05 to 0.10 range so that β will remain reasonably small.

EXERCISES 7.1 7.1.1.

An appliance manufacturer is considering the purchase of a new machine for stamping out sheet metal parts. If μ0 (given) is the true average of the number of good parts stamped out per hour by their old machine and μ is the corresponding true unknown average for the new machine, the manufacturer wants to test the null hypothesis μ = μ0 versus a suitable alternative. What should the alternative be if he does not want to buy the new machine unless it is (a) more productive than the old one? (b) At least 20% more productive than the old one?

7.1.2.

Formulate an alternative hypothesis for each of the following null hypotheses. (a) H0 : Support for a presidential candidate is unchanged after the start of the use of TV commercials. (b) H0 : The proportion of viewers watching a particular local news channel is less than 30%. (c) H0 : The median grade point average of undergraduate mathematics majors is 2.9.

7.1.3.

It is suspected that a coin is not balanced (not fair). Let p be the probability of tossing a head. To test H0 : p = 0.5 against the alternative hypothesis Ha : p > 0.5, a coin is tossed 15 times. Let Y equal the number of times a head is observed in the 15 tosses of this coin. Assume the rejection region to be {Y ≥ 10}. (a) Find α. (b) Find β for p = 0.7. (c) Find β for p = 0.6. (d) Find the rejection region for {Y ≥K} for α = 0.01, and α = 0.03. (e) For the alternative Ha : p = 0.7, ﬁnd β for the values of α given in (d).

7.1.4.

In Exercise 7.1.3: (a) Assume that the rejection region is {Y ≥ 8}. Calculate α and β if p = 0.6. Compare the results with the corresponding values obtained in Exercise 7.1.3. (This gives the effect of enlarging the rejection region on α and β.) (b) Assume that the rejection region is {Y ≥ 8}. Calculate α and β if p = 0.6 and (i) the coin is tossed 20 times, or (ii) the coin is tossed 25 times. (This shows the effect of increasing the sample size on α and β for a ﬁxed rejection region.)

7.1.5.

Suppose we have a random sample of size 25 from a normal population with an unknown mean μ and a standard deviation of 4. We wish to test the hypothesis H0 : μ = 10 vs.

7.2 The Neyman–Pearson Lemma 349

Ha : μ > 10. Let the rejection region be deﬁned by: reject H0 if the sample mean X > 11.2. (a) Find α. (b) Find β for Ha : μ = 11. (c) What should the sample size be if α = 0.01 and β = 0.8? 7.1.6.

A process for making steel pipe is under control if the diameter of the pipe has mean 3.0 in. with standard deviation of no more than 0.0250 in. To check whether the process is under control, a random sample of size n = 30 is taken each day and the null hypothesis μ = 3.0 is rejected if X is less than 2.9960 or greater than 3.0040. Find (a) the probability of type I error; (b) the probability of type II error when μ = 3.0050 in. Assume σ = 0.0250 in.

7.1.7.

A bowl contains 20 balls, of which x are green and the remain- der red. To test H0 : x = 10 versus Ha : x = 15, three balls are selected at random without replacement, and H0 is rejected if all three balls are green. Calculate α and β for this test.

7.1.8.

Suppose we have a sample of size 6 from a population with pdf f (x) = (1/θ)e−x/θ , x > 0, θ > 0. We wish to test H0 : θ = 1 vs. Ha : θ > 1. Let the rejection region be deﬁned by reject H0 if 6 i=1 Xi > 8. (a) Find α. (b) Find β for Ha : θ = 2.

7.1.9.

Let σ 2 = 16 be the variance of a normal population from which a random sample is chosen. How large should the sample size be for testing H0 : μ = 25 versus Ha : μ = 24, in order that α = 0.05 and β = 0.05?

7.2 THE NEYMAN–PEARSON LEMMA In practical hypothesis testing situations, there are typically many tests possible with signiﬁcance level α for a null hypothesis versus alternative hypothesis (see Project 7A). This leads to some important questions, such as (1) how to decide on the test statistic and (2) how to know that we selected the best rejection region. In this section, we study the answer to these questions using the Neyman–Pearson approach. Deﬁnition 7.2.1 Suppose that W is the test statistic and RR is the rejection region for a test of hypothesis concerning the value of a parameter θ. Then the power of the test is the probability that the test rejects H0 when the alternative is true. That is, π = Power(θ) = P(W in RR when the parameter value is an alternative θ).

If H0 : θ = θ0 and Ha : θ = θ0 , then the power of the test at some θ = θ1 = θ0 is Power(θ1 ) = P(reject H0 |θ = θ1 ).

But, β(θ1 ) = P(accept H0 |θ = θ1 ). Therefore, Power(θ1 ) = 1 − β(θ1 ).

A good test will have high power.

350 CHAPTER 7 Hypothesis Testing

Note that the power of a test H0 cannot be found until some true situation Ha is speciﬁed. That is, the sampling distribution of the test statistic when Ha is true must be known or assumed. Because β depends on the alternative hypothesis, which being composite most of the time does not specify the distribution of the test statistic, it is important to observe that the experimenter cannot control β. For example, the alternative Ha : μ < μ0 does not specify the value of μ, as in the case of the null hypothesis, H0 : μ = μ0 .

Example 7.2.1 Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ, that is, the pdf is given by f (x) = e−λ λx /(x!). Then the hypothesis H0 : λ = 1 uniquely speciﬁes the distribution, because f (x) = e−1 /(x!) and hence is a simple hypothesis. The hypothesis Ha : λ > 1 is composite, because f (x) is not uniquely determined.

Deﬁnition 7.2.2 A test at a given α of a simple hypothesis H0 versus the simple alternative Ha that has the largest power among tests with the probability of type I error no larger than the given α is called a most powerful test. Consider the test of hypothesis H0 : θ = θ0 versus Ha : θ = θ1 . If α is ﬁxed, then our interest is to make β as small as possible. Because β = 1 − Power(θ1 ), by minimizing β we would obtain a most powerful test. The following result says that among all tests with given probability of type I error, the likelihood ratio test given later minimizes the probability of a type II error, in other words, it is most powerful. Theorem 7.2.1 (Neyman–Pearson Lemma) Suppose that one wants to test a simple hypothesis H0 : θ = θ0 versus the simple alternative hypothesis Ha : θ = θ1 based on a random sample X1 , . . . , Xn from a distribution with parameter θ. Let L(θ) ≡ L(θ; X1 , . . . , Xn ) > 0 denote the likelihood of the sample when the value of the parameter is θ. If there exist a positive constant K and a subset C of the sample space Rn (the Euclidean n-space) such that 1.

L(θ0 ) ≤ K for (x1 , x2 , . . . , xn ) ∈ C L(θ1 )

2.

L(θ0 ) ≥ K for (x1 , x2 , . . . , xn ) ∈ C , where C is the complement of C, and L(θ1 )

3. P [(X1 , . . . , Xn ) ∈ C; θ0 ] = α. Then the test with critical region C will be the most powerful test for H0 versus Ha . We call α the size of the test and C the best critical region of size α. Proof. We prove this theorem for continuous random variables. For discrete random variables, the proof is identical with sums replacing the integral. Let S be some region in Rn , an n-dimensional Euclidean space. For simplicity we will use the following notation:

L(θ) = S

...

S

L(θ; x1 , x2 , . . . , xn )dx1 dx2 , . . . , dxn S

7.2 The Neyman–Pearson Lemma 351

Note that P((X1 , . . . , Xn ) ∈ C; θ0 ) =

f (x1 , . . . , xn ; θ0 )dx1 , . . . , dxn C

=

L(θ0 ; x1 , . . . , xn )dx1 , . . . , dxn . C

Suppose that there is another critical region, say B, of size less than or equal to α, that is B L(θ0 ) ≤ α. Then 0≤

L(θ0 ) −

B

C

L(θ0 ) = α by assumption 3.

L(θ0 ), because C

Therefore, 0≤

L(θ0 ) −

L(θ0 ) B

C

L(θ0 ) +

=

C∩B

C∩B

=

L(θ0 ) −

L(θ0 ) −

L(θ0 )

C ∩B

C∩B

L(θ0 ) −

C∩B

L(θ0 ).

C ∩B

Using assumption 1 of Theorem 7.2.1, KL(θ1 ) ≥ L(θ0 ) at each point in the region C and hence in C ∩ B . Thus

L(θ0 ) ≤ K C∩B

L(θ1 ).

C∩B

By assumption 2 of the theorem, KL(θ1 ) ≤ L(θ0 ) at each point in C , and hence in C ∩ B. Thus,

L(θ0 ) ≥ K

C ∩B

L(θ1 ).

C ∩B

Therefore,

0≤

L(θ0 ) − C∩B

≤K

⎧ ⎪ ⎨ ⎪ ⎩

C∩B

L(θ0 )

C ∩B

L(θ1 ) − C ∩B

⎫ ⎪ ⎬ L(θ1 ) . ⎪ ⎭

352 CHAPTER 7 Hypothesis Testing

That is, 0≤K

⎧ ⎪ ⎨ ⎪ ⎩

L(θ1 ) +

C∩B

=K

⎧ ⎨ ⎩

L(θ1 ) −

L(θ1 )−

C∩B

⎫ ⎬

L(θ1 ) −

L(θ1 )

C ∩B

C∩B

⎫ ⎪ ⎬

⎪ ⎭

L(θ1 ) . ⎭ B

C

As a result,

L(θ1 ) ≥

L(θ1 ). B

C

Because this is true for every critical region B of size ≤ α, C is the best critical region of size α, and the test with critical region C is the most powerful test of size α. When testing two simple hypotheses, the existence of a best critical region is guaranteed by the Neyman–Pearson lemma. In addition, the foregoing theorem provides a means for determining what the best critical region is. However, it is important to note that Theorem 7.2.1 gives only the form of the rejection region; the actual rejection region depends on the speciﬁc value of α. In real-world situations, we are seldom presented with the problem of testing two simple hypotheses. There is no general result in the form of Theorem 7.4.1 for composite hypotheses. However, for hypotheses of the form H0 : θ = θ0 versus Ha : θ > θ0 , we can take a particular value θ1 > θ0 and then ﬁnd a most powerful test for H0 : θ = θ0 versus Ha : θ > θ1 . If this test (that is, the rejection region of the test) does not depend on the particular value θ1 , then this test is said to be a uniformly most powerful test for H0 : θ = θ0 versus Ha : θ > θ0 . The following example illustrates the use of the Neyman–Pearson lemma.

Example 7.2.2 Let X1 , . . . , Xn denote an independent random sample from a population with a Poisson distribution with mean λ. Derive the most powerful test for testing H0 : λ = 2 versus Ha : λ = 1/2.

Solution Recall that the pdf of Poisson variable is p(x) =

e−λ λx , x!

λ > 0, x = 0, 1, 2, . . .

0,

otherwise.

Thus, the likelihood function is n " ! ( xi ) λ i=1 e−λn

L=

n

(xi !)

i=1

.

7.2 The Neyman–Pearson Lemma 353

For λ = 2,

!

2

n

xi

e−2n

i=1

L(θ0 ) = L(λ = 2) =

"

n

(xi !)

i=1

and for λ = 1/2,

⎡

⎣(1/2) L(θ1 ) = L(λ = 1/2) =

n

xi

i=1

⎤ e−(1/2)n ⎦

n

(xi !)

i=1

Thus, 2( xi ) e−n2 L(θ0 ) = σ02 ?

354 CHAPTER 7 Hypothesis Testing

Solution To test H0 : σ 2 = σ02 versus Ha : σ 2 > σ12 . We have

L(σ02 ) =

n 7

√

i=1

−

1 2πσ0n

(xi − μ)2 2σ02

e

(xi − μ)2 2σ0n .

−

1 = √ e ( 2π)n σ0n Similarly,

(xi − μ)2

−

2σ12

1 L(σ12 ) = √ e ( 2π)n σ1n

.

Therefore, the most powerful test is, reject H0 if, L(σ02 ) L(σ12 )

=

σ12

n ! (σ12 −σ02 )2

σ02

e

−

(xi − μ)2

2σ12 σ02

"

≤K

for some K. Taking the natural logarithms, we have

n ln

σ1 σ0

−

(σ12 − σ02 )

(xi − μ)2 ≤ ln K

2σ12 σ02

or

!

σ1 (xi − μ)2 ≥ n ln σ0

"

− ln K

2σ12 σ02 σ12 − σ02

= C.

To find the rejection region for a fixed value of α, write the region as (xi − μ)2 σ02

≥

C σ02

= C.

Note that (xi − μ)2 /σ02 has a χ2 -distribution with n degrees of freedom. Under the H0 because the same rejection region (does not depend upon the specific value of σ12 in the alternative) would be used for any σ12 > σ02 , the test is uniformly most powerful.

The foregoing example shows that, in order to test for variance using a sample from a normal distribution, we could use the chi-square table to obtain the critical value for the rejection region given α.

7.3 Likelihood Ratio Tests 355

EXERCISES 7.2 7.2.1.

Suppose X1 , . . . , Xn is a random sample from a normal distribution with a known variance of σ 2 and an unknown mean of μ. Find the most powerful α-level test of H0 : μ = μ0 versus Ha : μ = μa if (a) μ0 > μa , and (b) μa > μ0 .

7.2.2.

Show that the most powerful test obtained in Example 7.2.1 is uniformly most powerful for testing H0 : μ ≤ μ0 versus Ha : μ > μa , but not uniformly most powerful for testing H0 : μ = μ0 versus Ha : μ = μ0 .

7.2.3.

Suppose X1 , . . . , Xn is a random sample from a U(0, θ) distribution. Find the most powerful α-level test for testing H0 : θ = θ0 versus Ha : θ = θ1 , where θ0 < θ1 .

7.2.4.

Let X1 , . . . , Xn be a random sample from a geometric distribution with parameter p. Find the most powerful test of H0 : p = p0 versus Ha : p = pa (> p0 ). Is this uniformly most powerful test for H0 : p = p0 versus Ha : p > p0 ?

7.2.5.

Let X1 , . . . , Xn be a random sample from a distribution having a pdf of ⎧ 2 ⎨ 2y − y2 η , e f (y) = η2 ⎩ 0,

if x > 0 otherwise.

Find a uniformly most powerful test for testing H0 : η = η0 versus Ha : η < η0 . 7.2.6.

Let X be a single observation from the pdf f (x) =

θxθ−1 ,

0<x p0 .

7.2.8.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with mean λ. Find a best critical region for testing H0 : λ = 3 against Ha : λ = 6.

7.3 LIKELIHOOD RATIO TESTS The Neyman–Pearson lemma provides a method for constructing most powerful tests for simple hypotheses. We also have seen that in some instances when a hypothesis is not simple, it is possible to ﬁnd uniformly most powerful tests. In general, uniformly most powerful (UMP) tests do not exist for composite hypotheses. As an example, consider the two-sided hypothesis, at level α, given by H0 : μ = μ0

vs.

Ha : μ = μ0

where μ is the mean of a normal population with known variance σ 2 . If X is the sample mean of a random sample of size n, then as shown earlier, we can use the test statistic

356 CHAPTER 7 Hypothesis Testing

Z=

X − μ0 >√ . σ n

For Ha : μ = μ1 > μ0 , the rejection region for the most powerful test would be Reject H0 if z > zα .

On the other hand for Ha : μ = μ2 < μ0 , the rejection region for the most powerful test would be Reject H0 if z < −zα .

Thus, the rejection region depends on the speciﬁc alternative. Consequently, the two-sided hypothesis just given has no UMP test. In this section, we shall study a general procedure that is applicable when one or both H0 and Ha are composite. In fact, this procedure works for simple hypotheses as well. This method is based on the maximum likelihood estimation and the ratio of likelihood functions used in the Neyman–Pearson lemma. We assume that the pdf or pmf of the random variable X is f (x, θ), where θ can be one or more unknown parameters. Let represent the total parameter space that is the set of all possible values of the parameter θ given by either H0 or H1 . Consider the hypotheses H0 : θ ∈ 0 vs. Ha : θ ∈ a = − 0 .

where θ is the unknown population parameter (or parameters) with values in , and 0 is a subset of . Let L(θ) be the likelihood function based on the sample X1 , . . . , Xn . Now we deﬁne the likelihood ratio corresponding to the hypotheses H0 and Ha . This ratio will be used as a test statistic for the testing procedure that we develop in this section. This is a natural generalization of the ratio test used in the Neyman–Pearson lemma when both hypotheses were simple. Deﬁnition 7.3.1 The likelihood ratio λ is the ratio max L(θ; x1 , . . . , xn )

λ=

θ∈0

max L(θ; x1 , . . . , xn ) θ∈

=

L∗0 . L∗

We note that 0 ≤ λ ≤ 1. Because λ is the ratio of nonnegative functions, λ ≥ 0. Because 0 is a subset of , we know that max L(θ) ≤ max L(θ). Hence, λ ≤ 1. θ∈0

θ∈

If the maximum of L in 0 is much smaller as compared with the maximum of L in , that is, if λ is small, it would appear that the data X1 , . . . , Xn do not support the null hypothesis θ ∈ 0 . On the other hand, if λ is close to 1, one could conclude that the data support the null hypothesis, H0 . Therefore, small values of λ would result in rejection of the null hypothesis, and large values nearer to 1 will result a decision in support of the null hypothesis.

7.3 Likelihood Ratio Tests 357 For the evaluation of λ, it is important to note that max θ ∈ L(θ) = L(θˆ ml. ), where θˆ ml. is the maximum likelihood estimator of θ ∈ , and max θ ∈ 0 L(θ) is the likelihood function with unknown parameters replaced by their maximum likelihood estimators subject to the condition that θ ∈ 0 . We can summarize the likelihood ratio test as follows. LIKELIHOOD RATIO TESTS (LRTs) To test H0 : θ ∈ 0 vs. Ha : θ ∈ a max L(θ; x1 , . . . , xn )

λ=

L∗ = 0 max L(θ; x1 , . . . , xn ) L∗

θ∈0 θ∈

will be used as the test statistic. The rejection region for the likelihood ratio test is given by Reject H0 if λ ≤ K . K is selected such that the test has the given signiﬁcance level α.

Example 7.3.1 Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). Assume that σ 2 is known. We wish to test, at level α, H0 : μ = μ0 vs. Ha : μ = μ0 . Find an appropriate likelihood ratio test.

Solution We have seen that to test H0 : μ = μ0

vs.

Ha : μ = μ0

there is no uniformly most powerful test for this case. The likelihood function is n

L(μ) =

√

1 2πσ

n − e

2σ 2

Here, 0 = {μ0 } and a = R − {μ0 }. Hence,

n

L∗0 = max

μ=μ0

n − 1 e √ 2πσ n

=

(xi − μ)2

i=1

n − 1 e √ 2πσ

.

(xi − μ)2

i=1

2σ 2

(xi − μ0 )2

i=1

2σ 2

.

358 CHAPTER 7 Hypothesis Testing

Similarly, n

L∗ =

max

−∞ K (note that L L1 = 2 ( 19 ) That is, reject H0 if

10−x 18 >K 2x 19 x 2 ⇔ 18 > K1

⇔

19

19 x > K1 . 9

Hence, reject H0 if X > C; P(X > C|H0 : θ = 0.05) ≤ 0.05. Using the binomial tables, we have P(X > 2|θ = 0.05) = 0.0116

and P(X ≥ 2|θ = 0.05) = 0.0862.

Reject H0 if X > 2. If we want α to be exactly 0.05, we have to use randomized test. Reject with probability 0.0384 0.0762 = 0.5039 if X = 2. The likelihood ratio tests do not always produce a test statistic with a known probability distribution such as the z-statistic of Example 7.3.1. If we have a large sample size, then we can obtain an approximation to the distribution of the statistic λ, which is beyond the level of this book.

EXERCISES 7.3 7.3.1.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). Assume that σ 2 is unknown. We wish to test, at level α, H0 : μ = μ0 vs Ha : μ < μ0 . Find an appropriate likelihood ratio test.

7.3.2.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). Assume that both μ and σ 2 are unknown. We wish to test, at level α, H0 : σ 2 = σ02 vs. Ha : σ 2 > σ02 . Find an appropriate likelihood ratio test.

7.3.3.

Let X1 , . . . , Xn be a random sample from an N(μ1 , σ 2 ) and let Y1 , Y2 , . . . , Yn be an independent sample from an N(μ2 , σ 2 ), where σ 2 is unknown. We wish to test, at level α, H0 : μ1 = μ2 vs. Ha : μ1 = μ2 . Find an appropriate likelihood ratio test.

7.3.4.

Let X1 , . . . , Xn be a sample from a Poisson distribution with parameter λ. Show that a likelihood ratio test of H0 : λ = λ0 vs. Ha : λ = λ0 rejects the null hypothesis if X ≥ m1 or X ≤ m2 .

7.4 Hypotheses for a Single Parameter 361

7.3.5.

Let X1 , . . . , Xn be a sample from an exponential distribution with parameter θ. Show that a likelihood ratio test of H0 : θ = θ0 vs. Ha : θ = θ0 rejects the null hypothesis if ni=1 Xi ≥ m1 n or i=1 Xi ≤ m2 .

7.3.6.

A clinical oncology program developed a set of guidelines for their cancer patients to follow. It is believed that the proportion of patients who are still living after 24 months is greater for those who follow the guidelines. Of the 40 patients who followed the guidelines, 30 are still living after 24 months, whereas of 32 patients who did not follow the guidelines, 21 are living after 24 months. Find a likelihood ratio test at α = 0.01 to decide whether the program is effective.

7.4 HYPOTHESES FOR A SINGLE PARAMETER In this section, we ﬁrst introduce the concept of p-value. After that, we study hypothesis testing concerning a single parameter.

7.4.1 The p-Value In hypothesis testing, the choice of the value of α is somewhat arbitrary. For the same data, if the test is based on two different values of α, the conclusions could be different. Many statisticians prefer to compute the so-called p-value, which is calculated based on the observed test statistic. For computing the p-value, it is not necessary to specify a value of α. We can use the given data to obtain the p-value. Deﬁnition 7.4.1 Corresponding to an observed value of a test statistic, the p-value (or attained signiﬁcance level) is the lowest level of signiﬁcance at which the null hypothesis would have been rejected. For example, if we are testing a given hypothesis with α = 0.05 and we make a decision to reject H0 and we proceeded to calculate the p-value equal to 0.03, this means that we could have used an α as low as 0.03 and still maintain the same decision, rejecting H0 . Based on the alternative hypothesis, one can use the following steps to compute the p-value.

STEPS TO FIND THE p-VALUE 1. Let TS be the test statistic. 2. Compute the value of TS using the sample X1 , . . . , Xn . Say it is a. 3. The p-value is given by

p-value =

⎧ ⎪ P (T S < a|H0 ), ⎪ ⎪ ⎨ P (T S > a|H0 ), ⎪ ⎪ ⎪ ⎩ P (|T S| > |a||H0 ),

if lower tail test if upper tail test if two tail test.

362 CHAPTER 7 Hypothesis Testing

Example 7.4.1 To test H0 : μ = 0 vs. Ha : μ = 0, suppose that the test statistic Z results in a computed value of 1.58. Then, the p-value = P (|Z| > 1.58) = 2(0.0571) = 0.1142. That is, we must have a type I error of 0.1142 in order to reject H0 . Also, if Ha : μ > 0, then the p-value would be P (Z > 1.58) = 0.0582. In this case we must have an α of 0.0582 in order to reject H0 .

The p-value can be thought of as a measure of support for the null hypothesis: The lower its value, the lower the support. Typically one decides that the support for H0 is insufﬁcient when the p-value drops below a particular threshold, which is the signiﬁcance level of the test. REPORTING TEST RESULT AS p-VALUES 1. Choose the maximum value of α that you are willing to tolerate. 2. If the p-value of the test is less than the maximum value of α, reject H0 .

If the exact p-value cannot be found, one can give an interval in which the p-value can lie. For example, if the test is signiﬁcant at α = 0.05 but not signiﬁcant for α = 0.025, report that 0.025 ≤ p-value ≤ 0.05. So for α > 0.05, reject H0 , and for α < 0.025, do not reject H0 . In another interpretation, 1−(p-value) is considered as an index of the strength of the evidence against the null hypothesis provided by the data. It is clear that the value of this index lies in the interval [0, 1]. If the p-value is 0.02, the value of index is 0.98, supporting the rejection of the null hypothesis. Not only do p-values provide us with a yes or no answer, they provide a sense of the strength of the evidence against the null hypothesis. The lower the p-value, the stronger the evidence. Thus, in any test, reporting the p-value of the test is a good practice. Because most of the outputs from statistical software used for hypothesis testing include the p-value, the p-value approach to hypothesis testing is becoming more and more popular. In this approach, the decision of the test is made in the following way. If the value of α is given, and if the p-value of the test is less than the value of α, we will reject H0 . If the value of α is not given and the p-value associated with the test is small (usually set at p-value < 0.05), there is evidence to reject the null hypothesis in favor of the alternative. In other words, there is evidence that the value of the true parameter (such as the population mean) is signiﬁcantly different (greater, or lesser) than the hypothesized value. If the p-value associated with the test is not small (p > 0.05), we conclude that there is not enough evidence to reject the null hypothesis. In most of the examples in this chapter, we give both the rejection region and p-value approaches.

Example 7.4.2 The management of a local health club claims that its members lose on the average 15 pounds or more within the ﬁrst 3 months after joining the club. To check this claim, a consumer agency took a random sample of 45 members of this health club and found that they lost an average of 13.8 pounds within the ﬁrst 3 months of membership, with a sample standard deviation of 4.2 pounds.

7.4 Hypotheses for a Single Parameter 363

(a) Find the p-value for this test. (b) Based on the p-value in (a), would you reject the null hypothesis at α = 0.01?

Solution (a) Let μ be the true mean weight loss in pounds within the first 3 months of membership in this club. Then we have to test the hypothesis H0 : μ = 15 versus Ha : μ < 15 Here n = 45, x = 13.8, and s = 4.2. Because n = 45 > 30, we can use normal approximation. Hence, the test statistic is 13.8 − 15 = −1.9166 z= √ 4.2/ 45 and p-value = P (Z < −1.9166) P (Z < −1.92) = 0.0274. Thus, we can use an α as small as 0.0274 and still reject H0 . (b) No. Because the p-value = 0.0274 is greater than α = 0.01, one cannot reject H0 .

In any hypothesis testing, after an experimenter determines the objective of an experiment and decides on the type of data to be collected, we recommend the following step-by-step procedure for hypothesis testing.

STEPS IN ANY HYPOTHESIS TESTING PROBLEM 1. State the alternative hypothesis, Ha (what is believed to be true). 2. State the null hypothesis, H0 (what is doubted to be true). 3. Decide on a level of signiﬁcance α. 4. Choose an appropriate TS and compute the observed test statistic. 5. Using the distribution of TS and α, determine the rejection region(s) (RR). 6. Conclusion: If the observed test statistic falls in the RR, reject H0 and conclude that based on the sample information, we are (1 − α)100% conﬁdent that Ha is true. Otherwise, conclude that there is not sufﬁcient evidence to reject H0 . In all the applied problems, interpret the meaning of your decision. 7. State any assumptions you made in testing the given hypothesis. 8. Compute the p-value from the null distribution of the test statistic and interpret it.

7.4.2 Hypothesis Testing for a Single Parameter Now we study the testing of a hypothesis concerning a single parameter, θ, based on a random sample X1 , . . . , Xn . Let θˆ be the sample statistic. First, we deal with tests for the population mean μ for large and small samples. Next, we study procedures for testing the population variance σ 2 . We conclude the section by studying a test procedure for the true proportion p.

364 CHAPTER 7 Hypothesis Testing

To test the hypothesis H : μ = μ0 concerning the true population mean μ, when we have a large sample (n ≥ 30) we use the test statistic Z given by Z=

X − μ0 √ S/ n

where S is the sample standard deviation and μ0 is the claimed mean under H0 (if the population variance is known, we replace S with σ. For a small random sample (n < 30), the test statistic is T =

X − μ0 √ S/ n

where μ0 is the claimed value of the true mean, and X and S are the sample mean and standard deviation, respectively. Note that we are using the lowercase letters, such as z and t, to represent the observed values of the test statistics Z and T , respectively. In practice, with raw data, it is important to verify the assumptions. For example, in the small sample case, it is important to check for normality by using normal plots. If this assumption is not satisﬁed, the nonparametric methods described in Chapter 12 may be more appropriate. In addition, because the sample statistic such as X and S will be greatly affected by the presence of outliers, drawing a box plot to check for outliers is a basic practice we should incorporate in our analysis. We now summarize the typical test of hypothesis for tests concerning population (true) mean. In order to compute the observed test statistic, z in the large sample case and t in the small sample √ √ case, calculate the values of z = (x − μ0 )/(s/ n) and t = [(x − μ0 )/(s/ n)], respectively.

SUMMARY OF HYPOTHESIS TESTS FOR μ Large Sample (n ≥ 30) To test H0 : μ = μ0 versus μ > μ0 , upper tail test μ < μ0 , lower tail test Ha : μ = μ0 , two-tailed test

Small Sample (n < 30) To test H0 : μ = μ0 versus μ > μ0 , upper tail test Ha : μ < μ0 , lower tail test μ = μ0 , two-tailed test

X − μ0 √ σ/ n Replace σ by S, if σ is unknown. Test statistic: Z =

⎧ ⎪ ⎪ ⎨z > zα , Rejection region : z < −zα , ⎪ ⎪ ⎩|z| > z , α/2

Test statistic: T =

upper tail RR lower tail RR two tail RR

X − μ0 √ S/ n

⎧ ⎪ ⎪ ⎨t > tα,n−1 , RR : t < −tα,n−1 , ⎪ ⎪ ⎩|t| > t α/2,n−1 ,

upper tail RR lower tail RR two tail RR

7.4 Hypotheses for a Single Parameter 365 Assumption: n ≥ 30

Assumption: Random sample comes from a normal population

Decision: Reject H0 , if the observed test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, keep H0 so that there is not enough evidence to conclude that Ha is true for the given α and more experiments may be needed.

Example 7.4.3 It is claimed that sports-car owners drive on the average 18,000 miles per year. A consumer ﬁrm believes that the average mileage is probably lower. To check, the consumer ﬁrm obtained information from 40 randomly selected sports-car owners that resulted in a sample mean of 17,463 miles with a sample standard deviation of 1348 miles. What can we conclude about this claim? Use α = 0.01.

Solution Let μ be the true population mean. We can formulate the hypotheses as H0 : μ = 18,000 versus Ha : μ < 18,000. The observed test statistic (for n ≥ 30) is z=

x − μ0 ∼ 17,463 − 18,000 √ √ = σ/ n 1348/ 40

= −2.52. Rejection region is {z < −z0.01 } = {z < −2.33}. Decision: Because z = −2.52 is less than −2.33, the null hypothesis is rejected at α = 0.01. There is sufficient evidence to conclude that the mean mileage on sport cars is less than 18,000 miles per year.

Example 7.4.4 In a frequently traveled stretch of the I-75 highway, where the posted speed is 70 mph, it is thought that people travel on the average of at least 75 mph. To check this claim, the following radar measurements of the speeds (in mph) is obtained for 10 vehicles traveling on this stretch of the interstate highway. 66

74

79

80

69

77

78

65

79

81

Do the data provide sufﬁcient evidence to indicate that the mean speed at which people travel on this stretch of highway is at most 75 mph? Test the appropriate hypothesis using α = 0.01. Draw a box plot and normal plot for this data, and comment.

Solution We need to test H0 : μ = 75 vs. Ha : μ > 75

366 CHAPTER 7 Hypothesis Testing

Speed

80 75 70 65

■ FIGURE 7.1 Box plot of speed data.

For this sample, the sample mean is x = 74.8 mph and the standard deviation is σ = 5.9963 mph. Hence, the observed test statistic is t=

x − μ0 74.8 − 75 √ = √ σ/ n 5.9963/ 10

= −0.10547. From the t-table, t0.019 = 2.821. Hence, the rejection region is {t > 2.821}. Because, t = −0.10547 does not fall in the rejection region, we do not reject the null hypothesis at α = 0.01. Note that we assumed that the vehicles were randomly selected and that collected data follow the normal distribution, because of the small sample size, n < 30, we use the t-test. Figures 7.1 and 7.2 are the box plot and the normal plot of the data, respectively.

99

ML Estimates Mean : 74.8 Std Dev: 5.68858

Percent

95 90 80 70 60 50 40 30 20 10 5 1 55

65

75

85

95

Data ■ FIGURE 7.2 Normal probability plot for speed.

The box plot suggests that there are no outliers present. However, the normal plot indicates that the normality assumption for this data set is not justified. Hence, it may be more appropriate to do a nonparametric test.

7.4 Hypotheses for a Single Parameter 367

Example 7.4.5 In attempting to control the strength of the wastes discharged into a nearby river, an industrial ﬁrm has taken a number of restorative measures. The ﬁrm believes that they have lowered the oxygen consuming power of their wastes from a previous mean of 450 manganate in parts per million. To test this belief, readings are taken on n = 20 successive days. A sample mean of 312.5 and the sample standard deviation 106.23 are obtained. Assume that these 20 values can be treated as a random sample from a normal population. Test the appropriate hypothesis. Use α = 0.05.

Solution Here we need to test the following hypothesis: H0 : μ = 450 vs. Ha : μ < 450 Given n = 20, x = 312.5, and s = 106.23. The observed test statistic is t=

312.5 − 450 = −5.79. √ 106.23/ 20

The rejection region for α = 0.05 and with 19 degrees of freedom is the set of t-values such that {t < −t0.05,19 } = {t < −1.729}. Decision: Because t = −5.79 is less than −1.729, reject H0 . There is sufficient evidence to confirm the firm’s belief. For large random samples, the following procedure is used to perform tests of hypotheses about the population proportion, p.

Example 7.4.6 A machine is considered to be unsatisfactory if it produces more than 8% defectives. It is suspected that the machine is unsatisfactory. A random sample of 120 items produced by the machine contains 14 defectives. Does the sample evidence support the claim that the machine is unsatisfactory? Use α = 0.01.

Solution Let Y be the number of observed defectives. This follows a binomial distribution. However, because np0 and nq0 are greater than 5, we can use a normal approximation to the binomial to test the hypothesis. So we need to test H0 : p = 0.08 versus Ha : p > 0.08. Let the point estimate of p be pˆ = (Y /n) = 0.117, the sample proportion. Then the value of the TS is pˆ − p0 0.117 − 0.08 = 0.137. z= 8 = 8 p0 q0 (0.08)(0.92) n 120 For α = 0.01, z0.01 = 2.33. Hence, the rejection region is {z > 2.33}.

368 CHAPTER 7 Hypothesis Testing

Decision: Because 0.137 is not greater than 2.33, we do not reject H0 . We conclude that the evidence does not support the claim that the machine is unsatisfactory.

SUMMARY OF LARGE SAMPLE HYPOTHESIS TEST FOR p To test H0 : p = p0 versus p > p0 ,

upper tail test

Ha : p < p0 ,

lower tail test.

Test statistic: pˆ − p0 Z= , σpˆ

8 where

σpˆ =

p0 q0 , n

⎧ ⎪ ⎨ z > zα , Rejection region : z < −zα , ⎪ ⎩|z| > z , α/2

where

q0 = 1 − p 0 .

upper tail RR lower tail RR two tail RR,

where z is the observed test statistic. Assumption: n is large. A good rule of thumb is to use the normal approximation to the binomial distribution only when np0 and n(1 − p0 ) are both greater than 5. Decision: Reject H0 , if the observed test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α and more data are needed.

Note that this an approximate test, and the test can be improved by increasing the sample size. Now we give the procedure for testing the population variance when the samples come from a normal population. SUMMARY OF HYPOTHESIS TEST FOR THE VARIANCE σ 2 To test H0 : σ 2 = σ02 versus σ 2 > σ02 , Ha : σ 2 < σ02 , σ 2 = σ02 ,

upper tail test lower tail test two-tailed test.

7.4 Hypotheses for a Single Parameter 369

Test statistic: χ2 =

(n − 1)S 2 σ02

where S 2 is the sample variance. Observed value of test statistic: (n − 1)s 2 σ02 ⎧ 2 χ2 > χα,n−1 , ⎪ ⎪ ⎨ 2 2 χ < χ1−α,n−1 , Rejection region : ⎪ ⎪ 2 2 ⎩χ2 > χ2 α/2,n−1 or χ < χ1−α/2,n−1 ,

upper tail RR lower tail RR two tail RR

2 where χα,n−1 is such that the area under the chi-square distribution with (n − 1) degrees of freedom to its right is equal to α.

Assumption: Sample comes from a normal population. Decision: Reject H0 , if the observed test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α and more data are needed.

Because the chi-square distribution is not symmetric, the “equal tails” used for the two-sided alternative may not be the best procedure. However, in real-world problems we seldom use a two tail test for the population variance.

Example 7.4.7 A physician claims that the variance in cholesterol levels of adult men in a certain laboratory is at least 100. A random sample of 25 adult males from this laboratory produced a sample standard deviation of cholesterol levels as 12. Test the physician’s claim at 5% level of signiﬁcance.

Solution To test H0 : σ 2 = 100 versus Ha : σ 2 < 100 for α = 0.05, and 24 degrees of freedom, the rejection region is 2 RR = {χ2 < χ1−α,n−1 } = {χ2 < 13.484}.

The observed value of the TS is χ2 =

(n − 1)S 2 σ02

=

(24)(144) = 34.56. 100

370 CHAPTER 7 Hypothesis Testing

Because the value of the test statistic does not fall in the rejection region, we cannot reject H0 at 5% level of significance. Here, we assumed that the 25 cholesterol measurements follow the normal distribution.

EXERCISES 7.4 7.4.1.

A random sample of 50 measurements resulted in a sample mean of 62 with a sample standard deviation 8. It is claimed that the true population mean is at least 64. (a) Is there sufﬁcient evidence to refute the claim at the 2% level of signiﬁcance? (b) What is the p-value? (c) What is the smallest value of α for which the claim will be rejected?

7.4.2.

A machine in a certain factory must be repaired if it produces more than 12% defectives among the large lot of items it produces in a week. A random sample of 175 items from a week’s production contains 45 defectives, and it is decided that the machine must be repaired. (a) Does the sample evidence support this decision? Use α = 0.02. (b) Compute the p-value.

7.4.3.

A random sample of 78 observations produced the following sums: 78 i=1

xi = 22.8,

78

(xi − x)2 = 2.05.

i=1

(a) Test the null hypothesis that μ = 0.45 against the alternative hypothesis that μ < 0.45 using α = 0.01. Also ﬁnd the p-value. (b) Test the null hypothesis that μ = 0.45 against the alternative hypothesis that μ = 0.45 using α = 0.01. Also ﬁnd the p-value. (c) What assumptions did you make for solving (a) and (b)? 7.4.4.

Consider the test H0 : μ = 35 vs. Ha : μ > 35 for a population that is normally distributed. (a) A random sample of 18 observations taken from this population produced a sample mean of 40 and a sample standard deviation of 5. Using α = 0.025, would you reject the null hypothesis? (b) Another random sample of 18 observations produced a sample mean of 36.8 and a sample standard deviation of 6.9. Using α = 0.025, would you reject the null hypothesis? (c) Compare and discuss the decisions of parts (a) and (b).

7.4.5.

According to the information obtained from a large university, professors there earned an average annual salary of $55,648 in 1998. A recent random sample of 15 professors from this university showed that they earn an average annual salary of $58,800 with a sample standard deviation of $8300. Assume that the annual salaries of all the professors in this university are normally distributed.

7.4 Hypotheses for a Single Parameter 371

(a) Suppose the probability of making a type I error is chosen to be zero. Without performing all the steps of test of hypothesis, would you accept or reject the null hypothesis that the current mean annual salary of all professors at this university is $55,648? (b) Using the 1% signiﬁcance level, can you conclude that the current mean annual salary of professors at this university is more than $55,648? 7.4.6.

A check-cashing service company found that approximately 7% of all checks submitted to the service were without sufﬁcient funds. After instituting a random check veriﬁcation system to reduce its losses, the service company found that only 70 were rejected in a random sample of 1125 that were cashed. Is there sufﬁcient evidence that the check veriﬁcation system reduced the proportion of bad checks at α = 0.01? What is the p-value associated with the test? What would you conclude at the α = 0.05 level?

7.4.7.

A manufacturer of washers provides a particular model in one of three colors, white, black, or ivory. Of the ﬁrst 1500 washers sold, it is noticed that 550 were of ivory color. Would you conclude that customers have a preference for the ivory color? Justify your answer. Use α = 0.01.

7.4.8.

A test of the breaking strength of six ropes manufactured by a company showed a mean breaking strength of 6425 lb and a standard deviation of 120 lb. However, the manufacturer claimed a mean breaking strength of 7500 lb. (a) Can we support the manufacturer’s claim at a level of signiﬁcance of 0.10? (b) Compute the p-value. What assumptions did you make for this problem?

7.4.9.

A sample of 10 observations taken from a normally distributed population produced the following data: 44

31

52

48

46

39

43

36

41

49

(a) Test the hypothesis that H0 : μ = 44 vs. Ha : μ = 44 using α = 0.10. Draw a box plot and normal plot for this data, and comment. (b) Find a 90% conﬁdence interval for the population mean μ. (c) Discuss the meanings of (a) and (b). What can we conclude? 7.4.10.

The principal of a charter school in Tampa believes that the IQs of its students are above the national average of 100. From the past experience, IQ is normally distributed with a standard deviation of 10. A random sample of 20 students is selected from this school and their IQs are observed. The following are the observed values. 95 113

91 100

110 100

93 133 124 116

119 113 113 110

107 106

110 115

89 113

(a) Test for the normality of the data (b) Do the IQs of students at the school run above the national average at α = 0.01? 7.4.11.

In order to ﬁnd out whether children with chronic diarrhea have the same average hemoglobin level (Hb) that is normally seen in healthy children in the same area, a random

372 CHAPTER 7 Hypothesis Testing

sample of 10 children with chronic diarrhea are selected and their Hb levels (g/dL) are obtained as follows. 12.3 11.4 14.2 15.3 14.8 13.8 11.1 15.1 15.8 13.2

Do the data provide sufﬁcient evidence to indicate that the mean Hb level for children with chronic diarrhea is less than that of the normal value of 14.6 g/dL? Test the appropriate hypothesis using α = 0.01. Draw a box plot and normal plot for this data, and comment. 7.4.12.

A company that manufactures precision special-alloy steel shafts claims that the variance in the diameters of shafts is no more than 0.0003. A random sample of 10 shafts gave a sample variance of 0.00027. At the 5% level of signiﬁcance, test whether the company’s claim can be substantiated.

7.4.13.

It was claimed that the average annual expenditures per consumer unit had continued to rise, as measured by the Consumer Price Index annual averages (Bureau of Labor Statistics report, 1995). To test this claim, 100 consumer units were randomly selected in 1995 and found to have an average annual expenditure of $32,277 with a standard deviation of $1200. Assuming that the average annual expenditure of all consumer units was $30,692 in 1994, test at the 5% signiﬁcance level whether the annual expenditure per consumer unit had really increased from 1994 to 1995.

7.4.14.

It is claimed that two of three Americans say that the chances of world peace are seriously threatened by the nuclear capabilities of other countries. If in a random sample of 400 Americans, it is found that only 252 hold this view, do you think the claim is correct? Use α = 0.05. State any assumptions you make in solving this problem.

7.4.15.

According to the Bureau of Labor Statistics (1996), the average price of a gallon of gasoline in all U.S. cities in the United States in January 1996 was $1.129. A later random sample in 24 cities found the mean price to be $1.24 with a standard deviation of 0.01. Test at α = 0.05 to see whether the average price of a gallon of gas in the cities had recently changed.

7.4.16.

A manufacturer claims that the mean life of batteries manufactured by his company is at least 44 months. A random sample of 40 of these batteries was tested, resulting in a sample mean life of 41 months with a sample standard deviation of 16 months. Test at α = 0.01 whether the manufacturer’s claim is correct.

7.5 TESTING OF HYPOTHESES FOR TWO SAMPLES In this section we study the hypothesis testing procedures for comparing the means and variances of two populations. For example, suppose that we want to determine whether a particular drug is effective for a certain illness. The sample subjects will be randomly selected from a large pool of people with that particular illness and will be assigned randomly to the two groups. To one group we will administer a placebo; to the other we will administer the drug of interest. After a period of time, we measure a physical characteristic, say the blood pressure, of each subject that is an indicator of the severity of the illness. The question is whether the drug can be considered effective on the population from which our samples have been selected. We will consider the cases of independent and dependent samples.

7.5 Testing of Hypotheses for Two Samples 373

7.5.1 Independent Samples Two random samples are drawn independently of each other from two populations, and the sample information is obtained. We are interested in testing a hypothesis about the difference of the true means. Let X11 , . . . , X1n be a random sample from population 1 with mean μ1 and variance σ12 , and X21 , . . . , X2n be a random sample from population 2 with mean μ2 and variance σ22 . Let Xi , i = 1, 2, represent the respective sample means and Si2 , i = 1, 2, represent the sample variances. In this case, we shall consider following three cases in testing hypotheses about μ1 and μ2 : (i) when σ12 and σ22 are known, (ii) when σ12 and σ22 are unknown and n1 ≥ 30 and n2 ≥ 30, and (iii) when σ12 and σ22 are unknown and n1 < 30 and n2 < 30. In case (iii) we have the following two possibilities, (a) σ12 = σ22 , and (b) σ12 = σ22 . In the large sample case, knowledge of population variances σ12 and σ22 does not make much difference. If the population variances are unknown, we could replace them with sample variances as an approximation. If both n1 ≥ 30 and n2 ≥ 30 (large sample case), we can use normal approximation. The following box sums up a large sample hypothesis testing procedure for the difference of means for the large sample case. SUMMARY OF HYPOTHESIS TEST FOR μ1 − μ2 FOR LARGE SAMPLES (n1 & n2 ≥ 30) To test H0 : μ1 − μ2 = D0 versus

⎧ ⎪ ⎨μ1 − μ2 > D0 , Ha : μ1 − μ2 < D0 , ⎪ ⎩ μ1 − μ2 = D0 ,

upper tailed test lower tailed test two-tailed test.

The test statistic is Z=

X 1 − X 2 − D0 . + σ12 σ22 + n2 n1

Replace σi by Si , if σi ,i = 1,2 are not known. Rejection region is ⎧ ⎪ ⎪ ⎨z > zα , RR :

z < −zα , ⎪ ⎪ ⎩ |z| > zα/2 ,

upper tail RR lower tail RR two tail RR,

where z is the observed test statistic given by x 1 − x 2 − D0 . z= + σ22 σ12 + n2 n1

374 CHAPTER 7 Hypothesis Testing

Assumption: The samples are independent and n1 and n2 ≥ 30. Decision: Reject H0 , if test statistic falls in the RR and conclude that Ha is true with (1 − a)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α and more experiments are needed.

Example 7.5.1 In a salary equity study of faculty at a certain university, sample salaries of 50 male assistant professors and 50 female assistant professors yielded the following basic statistics.

Male assistant professor Female assistant professor

Sample mean salary $36,400 $34,200

Sample standard deviation 360 220

Test the hypothesis that the mean salary of male assistant professors is more than the mean salary of female assistant professors at this university. Use α = 0.05.

Solution Let μ1 be the true mean salary for male assistant professors and μ2 be the true mean salary for female assistant professors at this university. To test H0 : μ1 − μ2 = 0 vs. Ha : μ1 − μ2 > 0 the test statistic is 36,400 − 34,200 x 1 − x 2 − D0 = 36.872. = + z= + 2 2 s2 s1 (360)2 (220)2 + + n1 n2 50 50 The rejection region for α = 0.05 is {z > 1.645}. Because z = 36.872 > 1.645, we reject the null hypothesis at α = 0.05. We conclude that the salary of male assistant professors at this university is higher than that of female assistant professors for α = 0.05. Note that even though σ12 and σ22 are unknown, because n1 ≥ 30 and n2 ≥ 30, we could replace σ12 and σ22 by the respective sample variances. We are assuming that the salaries of male and female are sampled independently of each other.

Given next is the procedure we follow to compare the true means from two independent normal populations when n1 and n2 are small (n1 < 30 or n2 < 30) and we can assume homogeneity in the population variances, that is, σ12 = σ22 . In this case, we pool the sample variances to obtain a point estimate of the common variance.

7.5 Testing of Hypotheses for Two Samples 375

COMPARISON OF TWO POPULATION MEANS, SMALL SAMPLE CASE (POOLED t-TEST) To test H0 : μ1 − μ2 = D0 versus μ1 − μ2 > D0 ,

upper tailed test

Ha : μ1 − μ2 < D0 , μ1 − μ2 = D0 ,

lower tailed test two-tailed test.

The test statistic is T =

X 1 − X 2 − D0 9 Sp n11 + n12

Here the pooled sample variance is Sp2 =

(n1 − 1)S12 + (n2 − 1)S22 . n1 + n2 − 2

Then the rejection region is ⎧ ⎪ t > tα , ⎪ ⎨ RR : t < −tα , ⎪ ⎪ ⎩|t| > t

α/2 ,

upper tailed test lower tail test two-tailed test

where t is the observed test statistic and tα is based on (n1 + n2 − 2) degrees of freedom, and such that P(T > tα ) = α. Decision: Reject H0 , if test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α. Assumptions: The samples are independent and come from normal populations with means μ1 and μ2 , and with the (unknown) but equal variances, that is, σ12 = σ22 .

Now we shall consider the case where σ12 and σ22 are unknown and cannot be assumed to be equal. In such a case the following test is often used. For the hypothesis ⎧ ⎪ ⎨μ1 − μ2 > D0 H0 : μ1 − μ2 = D0 vs. H0 : μ1 − μ2 < D0 ⎪ ⎩μ − μ = D 1 2 0

376 CHAPTER 7 Hypothesis Testing

deﬁne the test statistic Tν as Tν =

X 1 − X 2 − D0 8 S12 S22 n1 + n2

where Tν has a t-distribution with ν degrees of freedom, and ν=

# 2 $2 (s1 /n1 ) + (s22 /n2 ) (s2 /n2 )2 (s12 /n1 )2 + 2 n1 − 1 n2 − 1

.

The value of ν will not necessarily be an integer. In that case, we will round it down to the nearest integer. This method of hypothesis testing with unequal variances is called the Smith–Satterthwaite procedure. Even though this procedure is not widely used, some simulation studies have shown that the Smith–Satterthwaite procedure perform well when variances are unequal and it gives results that are more or less equivalent to those obtained with the pooled t-test when the variances are equal. However, when the sample sizes are approximately equal, the pooled t-test may still be used. Note that in addressing the question which of the cases (iii)(a) or (iii)(b) to use in a given problem, we suggest that if the point estimates S12 of σ12 , and S22 of σ22 are approximately the same, then it is logical to assume homogeneity, σ12 = σ22 and use (iii)(a), whereas if S12 and S22 are signiﬁcantly different we use (iii)(b). More appropriately, we have tests that can be used to test hypotheses concerning σ12 = σ22 or σ12 = σ22 , known as the F -test, which we discuss at the end of this subsection.

Example 7.5.2 The intelligence quotients (IQs) of 17 students from one area of a city showed a sample mean of 106 with a sample standard deviation of 10, whereas the IQs of 14 students from another area chosen independently showed a sample mean of 109 with a sample standard deviation of 7. Is there a signiﬁcant difference between the IQs of the two groups at α = 0.02? Assume that the population variances are equal.

Solution We test H0 : μ1 − μ2 = 0 vs. Ha : μ1 − μ2 = 0 Here n1 = 17, x1 = 106, and s1 = 10. Also, n2 = 14, x2 = 109, and s2 = 7. We have 2 = sp

=

(n1 − 1)s12 + (n2 − 1)s22 n1 + n2 − 2 (16)(10)2 + (13)(7)2 = 77.138. 29

7.5 Testing of Hypotheses for Two Samples 377

The test statistic is T =

106 − 109 X 1 − X 2 − D0 = −0.94644. = 8 8 1 √ 1 1 1 sp + + 77.138 n1 n2 17 14

For α = 0.02, t0.01,29 = 2.462. Hence, the rejection region is t < − 2.462 or t > 2.462. Because the observed value of the test statistic, T = −0.94644, does not fall in the rejection region, there is not enough evidence to conclude that the mean IQs are different for the two groups. Here we assume that the two samples are independent and taken from normal populations.

Example 7.5.3 Assume that two populations are normally distributed with unknown and unequal variances. Two independent samples were drawn from these populations and the data obtained resulted in the following basic statistics: n1 = 18

x1 = 20.17

s1 = 4.3

n2 = 12

x2 = 19.23

s2 = 3.8

Test at the 5% signiﬁcance level whether the two population means are different.

Solution We need to test the hypothesis H0 : μ1 − μ2 = 0 versus Ha : μ1 − μ2 = 0. Here n1 = 18, x1 = 20.17, and s1 = 4.3. Also, n2 = 12, x2 = 19.23, and s2 = 3.8. The degrees of freedom for the t-distribution are given by ν=

2 2 s1 /n1 + s22 /n2 (s2 /n2 )2 (s12 /n1 )2 + 2 n1 − 1 n2 − 1 2 (4.3) (3.8)2 2 18 + 12

=

(4.3)2 18 17

2

+

(3.8)2 12 11

2 = 25.685.

Hence, we have ν = 25 degrees of freedom. For α = 0.05, t0.025,25 = 2.060. Thus, the rejection region is t < −2.060 or t > 2.060. The test statistic is given by Tν =

x 1 − x 2 − D0 8 S12 S22 n1 + n2

378 CHAPTER 7 Hypothesis Testing

= +

20.17 − 19.23 (3.8)2 (4.3)2 + 18 12

= 0.62939.

Because the observed value of the test statistic, Tν = 0.62939, does not fall in the rejection region, we do not reject the null hypothesis. At α = 0.05 there is not enough evidence to conclude that the population means are different. Note that the assumptions we made are that the samples are independent and came from two normal populations. No homogeneity assumption is made.

Example 7.5.4 Infrequent or suspended menstruation can be a symptom of serious metabolic disorders in women. In a study to compare the effect of jogging and running on the number of menses, two independent subgroups were chosen from a large group of women, who were similar in physical activity (aside from running), heights, occupations, distribution of ages, and type of birth control methods being used. The ﬁrst group consisted of a random sample of 26 women joggers who jogged “slow and easy” 5 to 30 miles per week, and the second group consisted of a random sample of 26 women runners who ran more than 30 miles per week and combined long, slow distance with speed work. The following summary statistics were obtained (E. Dale, D. H. Gerlach, and A. L. Wilhite, “Menstrual Dysfunction in Distance Runners,” Obstet. Gynecol. 54, 47–53, 1979). Joggers x1 = 10.1, s1 = 2.1 Runners x2 = 9.1, s2 = 2.4 Using α = 0.05, (a) test for differences in mean number of menses for each group assuming equality of population variances, and (b) test for differences in mean number of menses for each group assuming inequality of population variances.

Solution Here we need to test H0 : μ1 − μ2 = 0 versus Ha : μ1 − μ2 = 0. Here, n1 = 26, x1 = 10.1, and s1 = 2.1. Also, n2 = 26, x2 = 9.1, and s2 = 2.4. (a) Under the assumption σ12 = σ22 , we have 2 = sp

=

(n1 − 1)s12 + (n2 − 1)s22 n1 + n2 − 2 (25)(2.1)2 + (25)(2.4)2 = 5.085. 50

The test statistic is T =

X 1 − X 2 − D0 9 sp n11 + n12

7.5 Testing of Hypotheses for Two Samples 379 10.1 − 9.1 = √ = 1.5989. 9 1 + 1 5.085 26 26 For α = 0.05, t0.025,50 ≈ 1.96. Hence, the rejection region is t < −1.96 and t > 1.96. Because T = 1.589 does not fall in the rejection region, we do not reject the null hypothesis. At α = 0.05 there is not enough evidence to conclude that the population mean number of menses for joggers and runners are different. (b) Under the assumption σ12 = σ22 , we have ν=

2 2 s1 /n1 + s22 /n2 (s12 /n1 )2 (s22 /n2 )2 n1 −1 + n2 −1

(2.1)2 (2.4)2 2 26 + 26 =

2 2 = 49.134. (2.1)2 (2.4)2 26 26 + 25 25

Hence, we have ν = 49 degrees of freedom. Because this value is large, the rejection region is still approximately t < − 1.96 and t > 1.96. Hence, the conclusion is the same as that of part (a). In both parts (a) and (b), we assumed that the samples are independent and came from two normal populations.

Now we present the summary of the test procedure for testing the difference of two proportions, inherent in two binomial populations. Here, again we assume that the binomial distribution is approximated by the normal distribution and thus it is an approximate test.

SUMMARY OF HYPOTHESIS TEST FOR (p1 − p2 ) FOR LARGE SAMPLES (ni pi > 5 AND ni qi > 5, FOR i = 1, 2) To test H0 : p1 − p2 = D0 versus p1 − p2 < D0 , Ha : p1 − p2 > D0 , p1 − p2 = D0 ,

upper tailed test lower tailed test two-tailed test

at signiﬁcance level α, the test statistic is pˆ 1 − pˆ 2 − D0 Z= 9 pˆ 1 qˆ 1 pˆ 2 qˆ 2 n1 + n2 where z is the observed value of Z .

380 CHAPTER 7 Hypothesis Testing

The rejection region is ⎧ ⎪ ⎨ z > zα , RR : z < −zα , ⎪ ⎩|z| > z , α/2

upper tailed RR lower tailed RR two-tailed RR

Assumption: The samples are independent and ni pi > 5 and ni qi > 5, for i = 1,2. Decision: Reject H0 if the test statistic falls in the RR and conclude that Ha is true with (1 − a)100% conﬁdence. Otherwise, do not reject H0 , because there is not enough evidence to conclude that Ha is true for given α and more experiments are needed.

Example 7.5.5 Because of the impact of the global economy on a high-wage country such as the United States, it is claimed that the domestic content in manufacturing industries fell between 1977 and 1997. A survey of 36 randomly picked U.S. companies gave the proportion of domestic content total manufacturing in 1977 as 0.37 and in 1997 as 0.36. At the 1% level of signiﬁcance, test the claim that the domestic content really fell during the period 1977–1997.

Solution Let p1 be the domestic content in 1977 and p2 be the domestic content in 1997. Given n1 = n2 = 36, pˆ 1 = 0.37 and pˆ 2 = 0.36. We need to test H0 : p1 − p2 = 0 vs. Ha : p1 − p2 > 0. The test statistic is z= 9

= 9

pˆ 1 − pˆ 2 pˆ 1 qˆ 2 pˆ 1 qˆ 2 n1 + n2 0.37 − 0.36 (0.37)(0.63) + (0.36)(0.64) 36 36

= 0.08813.

For α = 0.01, z0.01 = 2.325. Hence, the rejection region is z > 2.325. Because the observed value of the test statistic does not fall in the rejection region, at α = 0.01, there is not enough evidence to conclude that the domestic content in manufacturing industries fell between 1977 and 1997.

Let X1 , . . . , Xn and Y1 , . . . , Yn be two independent random samples from two normal populations with sample variances s12 and s22 , respectively. The problem here is of testing for the equality of the

7.5 Testing of Hypotheses for Two Samples 381

variances, H0 : σ12 = σ22 . We have already seen in Chapter 4 that S 2 /σ 2 F = 12 12 S2 /σ2

follows the F -distribution with ν1 = n1 − 1 numerator and ν2 = n2 − 1 degrees of freedom. Under the assumption H0 : σ12 = σ22 , we have S2 F = 12 S2

which has an F -distribution with (ν1 , ν2 ) degrees of freedom. We summarize the test procedure for the equality of variances.

TESTING FOR THE EQUALITY OF VARIANCES To test H0 : σ12 = σ22 versus σ12 > σ22 ,

lower tailed test

Ha : σ12 < σ22 , σ12 = σ22 ,

upper tailed test two-tailed test

at signiﬁcance level α, the test statistic is S2 F = 12 . S2 The rejection region is ⎧ ⎪ ⎨ RR :

f > Fα (ν1 ,ν2 ), f < F1−α (ν1 ,ν2 ), ⎪ ⎩f > F (ν ,ν ) or f < F α/2 1 2 1−α/2 (ν1 ,ν2 ),

where f is the observed test statistic given by f =

upper tailed RR lower tailed RR two-tailed RR

s12 . s22

Decision: Reject H0 if the test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, keep H0 , because there is not enough evidence to conclude that Ha is true for a given α and more experiments are needed. Assumption: (i) The two random samples are independent. (ii) Both populations are normal.

Recall from Section 4.2 that in order to ﬁnd F1−α (ν1 , ν2 ), we use the identity F1−α (ν1 , ν2 ) = (1/Fα (ν2 , ν1 )).

382 CHAPTER 7 Hypothesis Testing

Example 7.5.6 Consider two independent random samples X1 , . . . , Xn from an N(μ1 , σ12 ) distribution and Y1 , . . . , Yn from an N(μ2 , σ22 ) distribution. Test H0 : σ12 = σ22 versus Ha : σ12 = σ22 for the following basic statistics: n1 = 25, x1 = 410, s12 = 95, and n2 = 16, x2 = 390, s22 = 300 Use α = 0.20.

Solution Test H0 : σ12 = σ22 versus Ha : σ12 = σ22 . This is a two-tailed test. Here the degrees of freedom are ν1 = 24 and ν2 = 15. The test statistic is s2 95 = 0.317. F = 12 = 300 s2 From the F -table, F0.10 (24, 15) = 1.90 and F0.90 (24, 15) =(1/F0.10 (15, 24)) = 0.50. Hence, the rejection region is F > 1.90 or F < 0.56. Because the observed value of the test statistic, 0.317, is less than 0.56, we reject the null hypothesis. There is evidence that the population variances are not equal.

7.5.2 Dependent Samples We now consider the case where the two random samples are not independent. When two samples are dependent (the samples are dependent if one sample is related to the other), then each data point in one sample can be coupled in some natural, nonrandom fashion with each data point in the second sample. This situation occurs when each individual data point within a sample is paired (matched) to an individual data point in the second sample. The pairing may be the result of the individual observations in the two samples: (1) representing before and after a program (such as weight before and after following a certain diet program), (2) sharing the same characteristic, (3) being matched by location, (4) being matched by time, (5) control and experimental, and so forth. Let (X1i , X2i ), for i = 1, 2, . . . , n, be a random sample. X1i , and X2j (i = j) are independent. To test the signiﬁcance of the difference between two population means when the samples are dependent, we ﬁrst calculate for each pair of scores the difference, Di = X1i − X2i , i = 1, 2, . . . , n, between the two scores. Let μD = E(Di ). Because pairs of observations form a random sample D1 , . . . , Dn are independent and identically distributed random variables, if d1 , . . . , dn are the observed values of D1 , . . . , Dn , then we deﬁne

d=

1 n

n i=1

1 di and sd2 = n−1

n i=1

(di − d)2 =

2 n n 1 di2 − di n i=1 i=1 n−1

.

Now the testing for these n observed differences will proceed as in the case of a single sample. If the number of differences is large (n ≥ 30), large sample inferential methods for one sample case can be used for the paired differences. We now summarize the hypothesis testing procedure for small samples.

7.5 Testing of Hypotheses for Two Samples 383

SUMMARY OF TESTING FOR MATCHED PAIRS EXPERIMENT To test μD > d0 , H0 : μD = d0 versus Ha : μD < d0 , μD = d0 ,

upper tail test lower tail test two-tailed test

√0 (this approximately follows a Student t-distribution with (n − 1) degrees of the test statistic: T = D−D SD / n

freedom). The rejection region is

⎧ ⎪ ⎨ t > tα,n−1 , t < −tα,n−1 , ⎪ ⎩|t| > t α/2,n−1 ,

upper tail RR lower tail RR two-tailed RR

where t is the observed test statistic. Assumptions: The differences are approximately normally distributed. Decision: Reject H0 if the test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 , because there is not enough evidence to conclude that Ha is true for a given α and more data are needed.

Example 7.5.7 A new diet and exercise program has been advertised as remarkable way to reduce blood glucose levels in diabetic patients. Ten randomly selected diabetic patients are put on the program, and the results after 1 month are given by the following table: Before 268 225 252 192 307 228 246 298 231 185 After 106 186 223 110 203 101 211 176 194 203 Do the data provide sufﬁcient evidence to support the claim that the new program reduces blood glucose level in diabetic patients? Use α = 0.05.

Solution We need to test the hypothesis H0 : μD = 0

vs.

Ha : μD < 0.

First we calculate the difference of each pair given in the following table. Before 268 225 252 192 307 228 246 298 231 185 After 106 186 223 110 203 101 211 176 194 203 Difference −162 −39 −29 −82 −104 −127 −35 −122 −37 18 (after−before)

384 CHAPTER 7 Hypothesis Testing

From the table, the mean of the differences is d = −71.9 and the standard deviation sd = 56.2. The test statistic is t=

d − d0 −71.9 √ = −4.0457 ≈ −4.05. √ = sd / n 56.2/ 10

From the t-table, t0,05,9 = 1.833. Because the observed value of t = − 4.05 < −t0,05,9 = −1.833, we reject the null hypothesis and conclude that the sample evidence suggests that the new diet and exercise program is effective.

We can also obtain a (1 − α)100% conﬁdence interval for μD using the formula

S S D − tα/2 √d , D + tα/2 √d n n

where tα/2 is obtained from the t-table with (n − 1) degrees of freedom. The interpretation of the conﬁdence interval is identical to the earlier interpretation.

Example 7.5.8 For the data in Example 7.5.7, obtain a 95% conﬁdence interval for μD and interpret its meaning.

Solution We have already calculated d = − 71.9 and sd = 56.2. From the t-table, t0.025,9 = 2.262. Hence, a 95% confidence interval for μD is (−112.1, −31.7). That is, P(−112.1 ≤ μD ≤ −31.7) = 0.95. Note that μD = μ1 − μ2 , and from the confidence limits we can conclude with 95% confidence that μ2 is always greater than μ1 , that is, μ2 > μ1 .

It is interesting to compare the matched pairs test with the corresponding two independent sample test. One of the natural questions is, why must we take paired differences and then calculate the mean and standard deviation for the differences—why can’t we just take the difference of means of each 2 need not be equal to sample, as we did for independent samples? The answer lies in the fact that σD 2 σ(X . Assume that −X ) 1

2

E(Xji ) = μj , Var(Xji ) = σj2 , for j = 1, 2,

and Cov(X1i , X2i ) = ρσ1 σ2

where ρ denotes the assumed common correlation coefﬁcient of the pair (X1i , X2i ) for i = 1, 2, . . . , n. Because the values of Di , i = 1, 2, . . . , n, are independent and identically distributed, μD = E(Di ) = E(X1i ) − E(X2i ) = μ1 − μ2

7.5 Testing of Hypotheses for Two Samples 385

and 2 = Var(D ) = Var(X ) + Var(X ) − 2Cov(X , X ) σD i 1i 2i 1i 2i

= σ12 + σ22 − 2ρσ1 σ2 .

From these calculations, E(D) = μD = μ1 − μ2

and σ2 1 σ 2 = Var(D) = D = (σ12 + σ22 − 2ρσ1 σ2 ). D n n

Now, if the samples were independent with n1 = n2 = n, E(X1 − X2 ) = μ1 − μ2

and σ2

(X1 −X2 )

=

1 2 (σ + σ22 ). n 1

2 < σ2 Hence, if ρ > 0, then σD . As a result, we can see that the matched pairs test reduces any (X1 −X2 ) variability introduced by differences in physical factors in comparison to the independent samples test when ρ > 0. It is also important to observe that normality assumption for the difference does not imply that the individual samples themselves are normal. Also, in a matched pairs experiment, there is no need to assume the equality of variances for the two populations. Matching also reduces degrees of freedom, because in case of two independent samples, the degrees of freedom is (n1 + n2 − 2), whereas for the case of two dependent samples it is only (n − 1).

EXERCISES 7.5 7.5.1.

Two sets of elementary school children were taught to read by different methods, 50 by each method. At the conclusion of the instructional period, a reading test gave results y1 = 74, y2 = 71, s1 = 9, and s2 = 10. What is the attained signiﬁcance level if you wish to see if there is evidence of a real difference between the two population means? What would you conclude if you desired an α-value of 0.05?

7.5.2.

The following information was obtained from two independent samples selected from two normally distributed populations with unknown but equal variances. Sample 1 14 15 11 14 10 8 13 10 12 16 15 Sample 2 17 16 21 12 20 18 16 14 21 20 13 20 13

Test at the 2% signiﬁcance level whether μ1 is lower than μ2 .

386 CHAPTER 7 Hypothesis Testing

7.5.3.

In the academic year 1997–1998, two random samples of 25 male professors and 23 female professors from a large university produced a mean salary for male professors of $58,550 with a standard deviation of $4000 and an average for female professors of $53,700 with a standard deviation of $3200. At the 5% signiﬁcance level, can you conclude that the mean salary of all male professors for 1997–1998 was higher than that of all female professors? Assume that the salaries of male and female professors are both normally distributed with equal standard deviations.

7.5.4.

It is believed that the effects of smoking differ depending on race. The following table gives the results of a statistical study for this question.

Whites African Americans

Number in the study 400 280

Average number of cigarettes per day 15 15

Number of lung cancer cases 78 70

Do the data indicate that African Americans are more likely to develop lung cancer due to smoking? Use α = 0.05. 7.5.5.

A supermarket chain is considering two sources A and B for the purchase of 50-pound bags of onions. The following table gives the results of a study.

Number of bags weighed Mean weight Sample variance

Source A 80 105.9 0.21

Source B 100 100.5 0.19

Test at α = 0.05 whether there is a difference in the mean weights. 7.5.6.

In order to compare the mean Hemoglobin (Hb) levels of well-nourished and undernourished groups of children, random samples from each of these groups yielded the following summary.

Well nourished Undernourished

Number of children 95 75

Sample mean 11.2 9.8

Sample standard deviation 0.9 1.2

Test at α = 0.01 whether the mean Hb levels of well-nourished children were higher than those of undernourished children. 7.5.7.

An aquaculture farm takes water from a stream and returns it after it has circulated through the ﬁsh tanks. In order to ﬁnd out how much organic matter is left in the waste water after the circulation, some samples of the water are taken at the intake and other samples are taken at the downstream outlet and tested for biochemical oxygen demand (BOD). BOD is a common environmental measure of the quantity of oxygen consumed by microorganisms during the decomposition of organic matter. If BOD increases, it can be said that the waste

7.5 Testing of Hypotheses for Two Samples 387

matter contains more organic matter than the stream can handle. The following table gives data for this problem. Upstream 9.0 6.8 6.5 8.0 7.7 8.6 6.8 8.9 7.2 7.0 Downstream 10.2 10.2 9.9 11.1 9.6 8.7 9.6 9.7 10.4 8.1

Assuming that the samples come from a normal distribution, (a) Test that the mean BOD for the downstream samples is less than for the samples upstream at α = 0.05. Assume that the variances are equal. (b) Test for the equality of the variances at α = 0.05. (c) In parts (a) and (b), we assumed samples are independent. Now, we feel this assumption is not reasonable. Assuming that the difference of each pair is approximately normal, test that the mean BOD for the downstream samples is less than for the upstream samples at α = 0.05. 7.5.8.

Suppose we want to know the effect on driving of a drug for cold and allergy, in a study in which the same people were tested twice, once after 1 hour of taking the drug and once when no drug is taken. Suppose we obtain the following data, which represent the number of cones (placed in a certain pattern) knocked down by each of the nine individuals before taking the drug and after an hour of taking the drug. No drug After drug

0 1

0 5

3 6

2 5

0 5

0 5

3 6

3 1

1 6

Assuming that the difference of each pair is coming from an approximately normal distribution, test if there is any difference in the individuals’ driving ability under the two conditions. Use α = 0.05. 7.5.9.

Suppose that we want to evaluate the role of intravenous pulse cyclophosphamide (IVCP) infusion in the management of nephrotic syndrome in children with steroid resistance. Children were given a monthly infusion of IVCP in a dose of 500 to 750 mg/m2 . The following data (source: S. Gulati and V. Kher, “Intravenous pulse cyclophosphamide—A new regime for steroid resistant focal segmental glomerulosclerosis,” Indian Pediatr. 37, 2000) represent levels of serum albumin (g/dL) before and after IVCP in 14 randomly selected children with nephrotic syndrome. Pre-IVCP 2.0 2.5 1.5 2.0 2.3 2.1 2.3 1.0 2.2 1.8 2.0 2.0 1.5 3.4 Post-IVCP 3.5 4.3 4.0 4.0 3.8 2.4 3.5 1.7 3.8 3.6 3.8 3.8 4.1 3.4

Assuming that the samples come from a normal distribution: (a) Test whether the mean Pre-IVCP is less than the mean Post-IVCP at α = 0.05. Assume that the variances are equal. (b) Test for the equality of the variances at α = 0.05. (c) In parts (a) and (b), we assumed that the samples are independent. Now, we feel this assumption is not reasonable. Assuming that the difference of each pair is approximately normal, test that the mean Pre-IVCP is less than the Post-IVCP at α = 0.05.

388 CHAPTER 7 Hypothesis Testing

7.5.10.

2 is an unbiased estimator of σ 2 . Show that SD D

7.5.11.

Test H0 : σ12 = σ22 versus Ha : σ12 = σ22 for the following data. n1 = 10, x1 = 71, s12 = 64

and

n2 = 25, x2 = 131, s22 = 96.

Use α = 0.10. 7.5.12.

The IQs of 17 students from one area of a city showed a mean of 106 with a standard deviation of 10, whereas the IQs of 14 students from another area showed a mean of 109 with a standard deviation of 7. Test for equality of variances between the IQs of the two groups at α = 0.02.

7.5.13.

The following data give SAT mean scores for math by state for 1989 and 1999 for 20 randomly selected states (source: The World Almanac and Book of Facts 2000). State Arizona Connecticut Alabama Indiana Kansas Oregon Nebraska New York Virginia Washington Illinois North Carolina Georgia Nevada Ohio New Hampshire

1989 523 498 539 487 561 509 560 496 507 515 539 469 475 512 520 510

1999 525 509 555 498 576 525 571 502 499 526 585 493 482 517 568 518

Assuming that the samples come from a normal distribution: (a) Test that the mean SAT score for math in 1999 is greater than that in 1989 at α = 0.05. Assume the variances are equal. (b) Test for the equality of the variances at α = 0.05.

7.6 CHI-SQUARE TESTS FOR COUNT DATA In this section, we study several commonly used tests for count data. These are basically large sample tests based on a χ2 -approximation. Suppose that we have outcomes of a multinomial experiment that consists of K mutually exclusive and exhaustive events A1 , . . . , Ak . Let P(Ai ) = pi , i = 1, 2, . . . , k. Then ni=1 pi = 1. Let the experiment be repeated n times, and let Xi (i = 1, 2, . . . , k) represent the number of times the event Ai occurs. Then (X1 , . . . , Xk ) have a multinomial distribution with parameters n, p1 , . . . , pk .

7.6 Chi-Square Tests for Count Data 389

Let Q2 =

k (Xi − npi )2 . (Xi − npi )2 i=1

It can be shown that for large n, the random variable Q2 is approximately χ2 -distributed with (k − 1) degrees of freedom. It is usual to demand npi ≥ 5 (i = 1, 2, . . . , k) for the approximation to be valid, although the approximation generally works well if for only a few values of i (about 20%), npi ≥ 1 and the rest (about 80%) satisfy the condition npi ≥ 5. This statistic was proposed by Karl Pearson in 1900. It should be noted that the χ2 -tests that we discuss in this section are approximate tests valid for large samples. Often Xi is called the observed frequency and is denoted by Oi (this is the observed value in class i), and npi is called the expected frequency and is denoted by Ei (this is the theoretical distribution frequency under the null hypothesis). Thus, with these notations, we get Q2 =

k (Oi − Ei )2 i=1

Ei

.

Example 7.6.1 A plant geneticist grows 200 progeny from a cross that is hypothesized to result in a 3 : 1 phenotypic ratio of red-ﬂowered to white-ﬂowered plants. Suppose the cross produces 170 red- to 30 white-ﬂowered plants. Calculate the value of Q2 for this experiment.

Solution There are two categories of data totaling n = 200. Hence, k = 2. Let i = 1 represent red-flowered and i = 2 represent white-flowered plants. Then O1 = 170, and O2 = 30. Here, H0 : The flower color population ratio is not different from 3 : 1, and the alternate is Ha : The flower color population sampled has a flower color ratio that is not 3 red : 1 white. Under the null hypothesis, the expected frequencies are E1 = (200)(3/4) = 150, and E2 = (200)(1/4) = 50. Hence, Q2 =

2 (Oi − Ei )2 Ei i=1

=

(30 − 50)2 (170 − 150)2 + = 10.667. 150 50

The type of calculation in Example 7.6.1 gives a measure of how close our observed frequencies come to the expected frequencies and is referred to as a measure of goodness of ﬁt. Smaller values of Q2 values indicate better ﬁt. One of the most frequent uses of the χ2 -test is in comparison of observed frequencies. Unless the sample size is exactly 100, percentages cannot be used. These are approximate tests. Let the random

390 CHAPTER 7 Hypothesis Testing

variables (X1 , . . . , Xk ) have a multinomial distribution with parameters n, p1 , . . . , pk . Let n be known. We will now present some important tests based on the chi-square statistic.

7.6.1 Testing the Parameters of Multinomial Distribution: Goodness-of-Fit Test Let an experiment have k mutually exclusive and exhaustive outcomes A1 , A2 , . . . , Ak . We would like to test the null hypothesis that all the pi = p(Ai ), i = 1, 2, . . . , k are equal to known numbers pi0 , i = 1, 2, . . . , k. We now summarize the test procedure.

TESTING THE PARAMETERS OF A MULTINOMIAL DISTRIBUTION (SUMMARY) To test H0 : p1 = p10 , . . . , pk = pk 0 versus Ha : At least one of the probabilities is different from the hypothesized value. The test is always a one-sided upper tail test. Let Oi be the observed frequency, Ei = npi0 be the expected frequency (frequency under the null hypothesis), and k be the number of classes. The test statistic is Q2 =

k (Oi − Ei )2 . Ei i=1

The test statistic Q 2 has an approximate chi-square distribution with k − 1 degrees of freedom. The rejection region is 2 Q 2 ≥ χα,k −1 .

Assumption: Ei ≥ 5: Exact methods are available. Computing the power of this test is difﬁcult.

This test is known as the goodness-of-ﬁt test. It implies that if the observed data are very close to the expected data, we have a very good ﬁt and we accept the null hypothesis. That is, for small Q2 values, we accept H0 .

Example 7.6.2 A TV station broadcasts a series of programs on the ill effects of smoking marijuana. After the series, the station wants to know whether people have changed their opinion about legalizing marijuana. Given in the following tables are the data based on a survey of 500 randomly chosen people:

7.6 Chi-Square Tests for Count Data 391

Before the Series Was Shown For legalization Decriminalization Existing law No opinion (ﬁne or imprisonment) 7% 18% 65% 10% After the Series Was Shown For legalization Decriminalization Existing law No opinion (ﬁne or imprisonment) 39% 9% 36% 16% Here, n = 4, and we wish to test H0 : p1 = 0.07; p2 = 0.18; p3 = 0.65; p4 = 0.1 versus Ha : At least one of the probabilities is different from the hypothesized value. The test is always an upper tail test. Test this hypothesis using α = 0.01.

Solution We have E1 = (500)(0.07) = 35; E2 = 90; E3 = 325; E4 = 50. The observed frequencies are O1 = (500)(0.39) = 195; O2 = 45; O3 = 180; O4 = 80. The test statistic is

Q2 =

4 (Oi − Ei )2 i=1

% =

Ei

(45 − 90)2 (180 − 325)2 (80 − 50)2 (195 − 35)2 + + + 35 90 325 50

&

= 836.62. 2 From the χ2 -table, χ0.01,3 = 11.3449. Because the test statistic Q2 = 836.62 > 11.3449, we reject H0 at α = 0.01. Hence, the data suggest that people have changed their opinion after the series on the ill effects of smoking marijuana was shown.

392 CHAPTER 7 Hypothesis Testing

Example 7.6.3 A die is rolled 60 times and the face values are recorded. The results are as follows. Up face Frequency

1 8

2 11

3 5

4 12

5 15

6 9

Is the die balanced? Test using α = 0.05.

Solution If the die is balanced, we must have p1 = p2 = . . . = p6 =

1 6

where pi = P(face value on the die is i), i = 1, 2, . . . , 6. This has the discrete uniform distribution. Hence, H0 : p1 = p2 = . . . = p6 =

1 6

versus Ha : At least one of the probabilities is different from the hypothesized value of 1/6 E1 = n1 p1 = (60)(1/6) = 10, . . . , E6 = 10. We summarize the calculations in the following table: Face value Frequency, Oi Expected value, Ei

1 8 10

2 11 10

3 5 10

4 12 10

5 15 10

6 9 10

The test statistic value is given by Q2 =

6 (Oi − Ei )2 i=1

Ei

= 6.

2 From the chi-square table with 5 d.f., χ0.05,5 = 11.070.

Because the value of the test statistic does not fall in the rejection region, we do not reject H0 . Therefore, we conclude that the die is balanced.

7.6.2 Contingency Table: Test for Independence One of the uses of the χ2 -statistic is in contingency (dependence) testing where n randomly selected items are classiﬁed according to two different criteria, such as when data are classiﬁed on the basis of two factors (row factor and column factor) where the row factor has r levels and the column factor has c levels. The obtained data are displayed as shown in the following table, where nij represents

7.6 Chi-Square Tests for Count Data 393

the number of data values under row i and column j. Our interest here is to test for independence of two methods of classiﬁcation of observed events. For example, we might classify a sample of students by sex and by their grade on a statistics course in order to test the hypothesis that the grades are dependent on sex. More generally the problem is to investigate a dependency (or contingency) between two classiﬁcation criteria. Levels of column factor 1 2 … c n11 n12 n1c n21 n21 n2c

Row total 1 n1 2 n2 . . r nr1 nr2 anrc nr Column total n.1 n.2 n.c N c c r r where N = n.j = ni. = nij is the grand total. Row levels

j=1

i=1

i=1 j=1

We wish to test the hypothesis that the two factors are independent. We summarize the procedure in the following table for testing that the factors represented by the rows are independent with that represented by the columns.

TESTING FOR THE INDEPENDENCE OF TWO FACTORS To test H0 : The factors are independent versus Ha : The factors are dependent the test statistic is, Q2 =

r c (Oij − Eij )2 Eij i=1 j=1

where Oij = nij and Eij =

ni nj N

.

Then under the null hypothesis the test statistic Q 2 has an approximate chi-square distribution with (r − 1)(c − 1) degrees of freedom. 2 Hence, the rejection region is Q 2 > χα,(r −1)(c−1) . Assumption: Eij ≥ 5.

394 CHAPTER 7 Hypothesis Testing

Example 7.6.4 The following table gives a classiﬁcation according to religious afﬁliation and marital status for 500 randomly selected individuals.

Marital status

Single With spouse Total

A 39 172 211

Religious afﬁliation B C D None 19 12 28 18 61 44 70 37 80 56 98 55

Total 116 384 500

For α = 0.01, test the null hypothesis that marital status and religious afﬁliation are independent.

Solution We need to test the hypothesis H0 : Marital status and religious affiliation are independent versus Ha : Marital status and religious affiliation are dependent. Here, c = 5, and r = 2. For α = 0.01, and for (c − 1)(r − 1) = 4 degrees of freedom, we have 2 χ0.01,4 = 13.2767

Hence, the rejection region is Q2 > 13.2767. ni nj . Thus, We have Eij = N E11 =

(116)(211) (116)(80) = 48.952; E12 = = 18.5; 500 500

E13 =

(116)(56) (116)(98) = 12.992, E14 = = 22.736; 500 500

E15 =

(116)(55) (384)(211) = 12.76, E21 = = 162.05; 500 500

E22 =

(384)(80) (384)(56) = 61.44; E23 = = 43.008; 500 500

and E24 =

(384)(98) = 75.264; 500

The value of the test statistic is Q2 =

r c (Oij −Eij )2 Eij i=1 j=1

E25 =

(384)(55) = 42.24. 500

7.6 Chi-Square Tests for Count Data 395 % =

(19 − 18.5)2 (12 − 12.992)2 (28 − 22.736)2 (39 − 48.952)2 + + + 48.952 18.5 12.992 22.736

&

+

(172 − 162.05)2 (61 − 61.44)2 (44 − 43.08)2 (18 − 12.76)2 + + + 12.76 162.05 61.44 43.08

+

(37 − 42.24)2 (70 − 75.264)2 + 75.264 42.24

= 7.1351. Because the observed value of Q2 does not fall in the rejection region, we do not reject the null hypothesis at α = 0.01. Therefore, based on the observed data, the marital status and religious affiliation are independent.

7.6.3 Testing to Identify the Probability Distribution: Goodness-of-Fit Chi-Square Test Another application of the chi-square statistic is using it for goodness-of-ﬁt tests in a different context. In hypothesis testing problems we often assume that the form of the population distribution is known. For example, in a χ2 -test for variance, we assume that the population is normal. The goodness-of-ﬁt tests examine the validity of such an assumption if we have a large enough sample. We now describe the goodness-of-ﬁt test procedure for such applications. GOODNESS-OF-FIT TEST PROCEDURES FOR PROBABILITY DISTRIBUTIONS Let X1 , . . . ,Xn be a sample from a population with cdf F (x), which may depend on the set of unknown parameters θ. We wish to test H0 : F (x) = F0 (x), where F0 (x ) is completely speciﬁed. 1. Divide the range of values of the random variables X1 into K nonoverlapping intervals I1 , I2 , . . . , IK . Let Oj be the number of sample values that fall in the interval Ij (j = 1, 2, . . . , K ). 2. Assuming the distribution of X to be F0 (x), ﬁnd P(X ∈ Ij ). Let P(X ∈ Ij ) = πi . Let ej = nπj be the expected frequency. 3. Compute the test statistic Q 2 given by Q2 =

K (Oi − Ei )2 . Ei i=1

The test statistic Q 2 has an approximate χ2 -distribution with (K − 1) degrees of freedom. 2 4. Reject the H0 if Q 2 ≥ χα, (K −1) . 5. Assumptions: ej ≥ 5, j = 1, 2, . . . , K .

If the null hypothesis does not specify F0 (x) completely, that is, if F0 (x) contains some unknown parameters θ1 , θ2 , . . . , θp , we estimate these parameters by the method of maximum likelihood. Using

396 CHAPTER 7 Hypothesis Testing

these estimated values we specify F0 (x) completely. Denote the estimated F0 (x) by Fˆ 0 (x). Let 2 3 πˆ i = P X ∈ Ii |Fˆ 0 (x)

and

ˆ i = nπˆ i . E

The test statistic is Q2 =

K (Oi − eˆ i )2 . eˆ i i=1

The statistic Q2 has an approximate chi-square distribution with (K − 1 − p) degrees of freedom. We 2 . reject H0 if Q2 ≥ χa,(K−1−p) We now illustrate the method of goodness-of-ﬁt with an example.

Example 7.6.5 The grades of students in a class of 200 are given in the following table. Test the hypothesis that the grades are normally distributed with a mean of 75 and a standard deviation of 8. Use α = 0.05. Range Number of students

0–59 12

60–69 36

70–79 90

80–89 44

90–100 18

Solution We have O1 = 12, O2 = 36, O3 = 90, O4 = 44, O5 = 18. We now compute πi (i = 1, 2, . . . , 5), using the continuity correction factor, π1 = P{X ≤ 59.5|H0 } = P{z ≤ 59.5−75 } = 0.0262, 8 π2 = 0.2189, π3 = 0.4722, π4 = 0.2476, π5 = 0.0351, and E1 = 5.24, E2 = 43.78, E3 = 94.44, E4 = 49.52, E5 = 7.02. The test statistic results in Q2 =

n (Oi − ei )2 i=1

=

ei

(36 − 43.78)2 (90 − 94.44)2 (44 − 49.52)2 (18 − 7.02)2 (12 − 5.74)2 + + + + 5.74 43.78 94.44 49.52 7.02

= 26.22.

7.6 Chi-Square Tests for Count Data 397

2 Q2 has a chi-square distribution with (5 − 1) = 4 degrees of freedom. The critical value is χ0.05,4 = 7.11. Hence, the rejection region is Q2 > 7.11. Because the observed value of Q2 = 26.22 > 7.11, we reject H0 at α = 0.05. Thus, we conclude that the population is not normal.

EXERCISES 7.6 7.6.1.

The following table gives the opinion on collective bargaining by a random sample of 200 employees of a school system, belonging to a teachers’ union. Opinion on Collective Bargaining by Teachers’ Union For Against Undecided Total Staff 30 15 15 60 Faculty 50 10 40 100 Administration 10 25 5 40 Column totals 90 50 60 200

Test the hypotheses H0 : Opinion on collective bargaining is independent of employee classiﬁcation versus Ha : Opinion on collective bargaining is dependent on employee classiﬁcation using α = 0.05. 7.6.2.

A random sample was taken of 300 undergraduate students from a university. The students in the sample were classiﬁed according to their gender and according to the choice of their major. The result is given in the following table.

Gender Male Female Total

Arts and sciences 75 45 120

College Engineering 40 12 52

Business 24 15 39

Other 66 23 89

Total 205 95 300

Test the hypothesis that the choice of the major by undergraduate students in this university is independent of their gender. Use α = 0.01. 7.6.3.

The speeds of vehicles (in mph) passing through a section of Highway 75 are recorded for a random sample of 150 vehicles and are given below. Test the hypothesis that the speeds are normally distributed with a mean of 70 and a standard deviation of 4. Use a = 0.01. Range Number

7.6.4.

40–55 12

56–65 14

66–75 78

76–85 40

> 85 6

Based on the sample data of 50 days contained in the following table, test the hypothesis that the daily mean temperatures in the city are normally distributed with mean 77 and variance 6. Use α = 0.05.

398 CHAPTER 7 Hypothesis Testing

Temperature Number of days

7.6.5.

46–55 4

56–65 6

66–75 13

76–85 23

86–95 4

A presidential candidate advertises on TV by comparing his positions on some important issues with those of his opponent. After a series of advertisements, a pollster wants to know whether people have changed their opinion about the candidate. The following are the data based on a survey of 950 randomly chosen people: Before the Advertisement Was Shown Support the Oppose the Need to know more Undecided candidate candidate about the candidate 40% 20% 5% 35% After the Advertisement Was Shown Support the Oppose the Need to know more Undecided candidate candidate about the candidate 45% 25% 2% 28%

Let pi , i = 1, 2, 3, 4, represent the respective true proportions. Test H0 : p1 = 0.35; p2 = 0.20; p3 = 0.15; p4 = 0.3

versus Ha : At least one of the probabilities is different from the hypothesized value. Test this hypothesis using α = 0.05. 7.6.6.

A survey of footwear preferences of a random sample of 100 undergraduate students (50 females and 50 males) from a large university resulted in the following data. Boots Female Male

12 10

Leather shoes 9 12

Sneakers

Sandals

Other

12 17

10 7

7 4

(a) Let pi , i = 1, 2, 3, 4, 5, represent the respective true proportions of students with a particular footwear preference, and let H0 : p1 = 0.20; p2 = 0.20; p3 = 0.30; p4 = 0.20; p5 = 0.10

versus Ha : At least one of the probabilities is different from the hypothesized value. Test this hypothesis using α = 0.05. (b) Test the hypothesis that the choice of footwear by undergraduate students in this university is independent of their gender, using α = 0.05.

7.8 Computer Examples 399

7.7 CHAPTER SUMMARY In this chapter, we have learned various aspects of hypothesis testing. First, we dealt with hypothesis testing for one sample where we used test procedures for testing hypotheses about true mean, true variance, and true proportion. Then we discussed the comparison of two populations through their true means, true variances, and true proportions. We also introduced the Neyman–Pearson lemma and discussed likelihood ratio tests and chi-square tests for categorical data. We now list some of the key deﬁnitions in this chapter. ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Statistical hypotheses Tests of hypotheses, tests of signiﬁcance, or rules of decision Simple hypothesis Composite hypothesis Type I error Type II error The level of signiﬁcance The p-value or attained signiﬁcance level The Smith–Satterthwaite procedure Power of the test Most powerful test Likelihood ratio

In this chapter, we also learned the following important concepts and procedures: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

General method for hypothesis testing Steps to calculate β Steps to ﬁnd the p-value Steps in any hypothesis testing problem Summary of hypothesis tests for μ Summary of large sample hypothesis tests for p Summary of hypothesis tests for the variance σ 2 Summary of hypothesis tests for μ1 − μ2 for large samples (n1 & n2 ≥ 30) Summary of hypothesis tests for p1 − p2 for large samples Testing for the equality of variances Summary of testing for a matched pairs experiment Procedure for applying the Neyman–Pearson lemma Procedure for the likelihood ratio test Testing the parameters of a multinomial distribution (summary) Testing the independence of two factors Goodness-of-ﬁt test procedures for probability distributions

7.8 COMPUTER EXAMPLES In the following examples, if the value of α is not speciﬁed, we will always take it as 0.05.

400 CHAPTER 7 Hypothesis Testing

7.8.1 Minitab Examples Example 7.8.1 (t-Test): Consider the data 66

74

79

80

69

77

78

65

79

81

Using Minitab, test H0 : μ = 75 vs. H1 : μ > 75.

Solution Enter the data in C1. Then Stat > Basic Statistics > 1-sample t. . . > In Variables: enter C1 > choose Test Mean > enter 75 > in Alternative: choose greater than and click OK

We obtain the following output. T-Test of the Mean Test of mu = 75.00 vs mu > 75.00 Variable C1

N 10

Mean 74.80

StDev 6.00

SE Mean 1.90

T −0.11

P 0.54

Example 7.8.2 For the following data: Sample 1: 16 18 21 13 19 16 18 15 20 19 14 21 14 Sample 2: 14 15 10 13 11 7 12 11 12 15 14 Test H0 : μ1 = μ2 vs. H1 : μ1 < μ2 . Use α = 0.02.

Solution Enter sample 1 data in C1 and sample 2 data in C2. Then Stat > Basic Statistics > 2-sample t. . . > Choose Samples in different columns > in Alternative: choose less than > in Confidence level: enter 98 > click Assumed equal variances and click OK

We obtain the following output. Two Sample T-test and Confidence Interval Two sample T for C1 vs C2

7.8 Computer Examples 401

N 13 11

C1 C2

Mean 17.23 12.18

StDev 2.74 2.40

SE Mean 0.76 0.72

98% CI for mu C1 − mu C2: (2.38, 7.71) T-Test mu C1 = mu C2 (vs 1-sample z. . . > in Variables: Type C1 > choose Test Mean and enter 12 > choose not equal in Alternative, and Type 4.7 for sigma > Click OK We obtain the following output. Z-Test Test of mu = 12.000 vs mu not = 12.000 The assumed sigma = 4.70 Variable C1

N 49

Mean 12.124

StDev 4.700

SE Mean 0.671

Z 0.19

P 0.85

Here the test statistic is 0.19 and the p-value is 0.85, which is larger than 0.05. Hence, we cannot reject the null hypothesis.

402 CHAPTER 7 Hypothesis Testing

Example 7.8.4 (Contingency Table): Consider the following data with ﬁve levels and two factors. Test for dependence of the factors. Factors 1 39 172

1 2

2 19 61

Levels 3 4 12 28 44 70

5 18 37

Solution In C1 enter the data in column 1 (39 and 172), and continue to C5. Then Stat > Tables > Chi-Square-Test. . . > in Columns containing the table: Type C1 C2 C3 C4 C5 > click OK

We will obtain the following output. Chi-Square Test Expected counts are printed below observed counts C1 C2 C3 C4 C5 Total 1 39 19 12 28 18 116 48.95 18.56 12.99 22.74 12.76 2 Total

172 162.05 211

61 61.44 80

44 43.01 56

70 75.26 98

37 42.24 55

384 500

Chi-Sq = 2.023 + 0.010 + 0.076 + 1.219 + 2.152 + 0.611 + 0.003 + 0.023 + 0.368 + 0.650 = 7.135 DF = 4, p-value = 0.129

Example 7.8.5 (Paired t-Test): Consider the data of Example 7.5.7. Using Minitab, perform a paired t-test.

Solution Enter sample 1 in column C1 and sample 2 in column C2. Then: Stat > Basic Statistics > Paired t. . . > in First Sample: Type C2, and in the Second sample: Type C1 > click options > and click less than (if α is other than 0.05, enter appropriate percentage in Confidence level: and enter appropriate number if it is not zero in Test mean:) > click OK > OK

7.8 Computer Examples 403

We obtain the following output. Paired T-test and Confidence Interval T for C2 − C1 N Mean StDev SE Mean C2 10 171.3 47.1 14.9 C1 10 243.2 40.1 12.7 Difference 10 −71.9 56.2 17.8 95% CI for mean difference: (−112.1, −31.7) T-Test of mean difference = 0 (vs < 0): T-Value = −4.05 p-value = 0.001 Paired

because the p-value 0.001 < 0.05 = α.

7.8.2 SPSS Examples Example 7.8.6 Consider the data 66

74

79

80

69

77

78

65

79

81

Using SPSS, test H0 : μ = 75 vs. H1 : μ > 75.

Solution Use the following procedure: 1. Enter the data in column 1. 2. Click Analyze > Compare Means > One-sample t Test. . . , Move var00001 to Test Variable(s), and change Test Value: 0 to 75. Click OK We obtain the following output. One-Sample Statistics

VAR00001

N 10

Mean 74.8000

Std. Deviation 5.99630

Std. Error Mean 1.89620

One-Sample Test Test Value = 75 95% Confidence Interval of the Sig. Mean Difference t df (2-tailed) Difference Lower Upper VAR00001 −.105 9 .918 −.2000 −4.4895 4.0895 For the one sample t-test H0 : μ = 75 vs. H1 : μ > 75, the t-statistic is −0.105 with 9 degrees of freedom. The p-value is 0.46 > 0.02. Hence, we will not reject the null hypothesis.

404 CHAPTER 7 Hypothesis Testing

If we want the computer to calculate the p-value in the previous example, use the following procedure. 1. Enter the test statistic (−0.105) in the data editor using ‘teststat’. 2. Click Transform > compute. . . 3. Type ‘p-value’ in the box called Tarobtain value. In the box called Functions: scroll and click on CDF.T(q,df) and move to Numeric Expressions. 4. The CDF(q,df) will appear as CDF(?,?) in the Numeric Expressions box. Replace teststat for q and 9 for df (the degree of freedom in this example is 9). Click OK

We obtain the p-value as 0.46.

Example 7.8.7 For the following data Sample 1: Sample 2:

16 14

18 15

21 10

13 13

19 11

16 7

18 12

15 11

20 12

19 15

14 14

21

14

Test H0 : μ1 = μ2 vs. H1 : μ1 < μ2 . Use α = 0.02.

Solution In column 1, under the title ‘‘group’’ enter 1s to identify the sample 1 data and 2s to identify sample 2 data. In column C2, under the title ‘‘data’’ enter the data corresponding to samples 1 and 2. Then: Analyze > Compare Means > Independent Samples t-test. . . > bring Data to Test Variable(s): and group to Grouping Variable:, click Define Groups. . . , and enter 1 for sample 1, 2 for sample 2 > click continue > click Options. . . . Enter 98 in Confidence interval: > click continue > OK We obtain the following output. Group Statistics GROUP N DATA 1.00 13 2.00 11

Mean 17.2308 12.1818

Independent Samples Test Levene’s Test t-test for for Equality Equality of Variances of Means F Sig. t

DATA Equal variances assumed Equal variances not assumed

.975

.334

df

4.753

22

4.808

21.963

Std. Deviation 2.74329 2.40076

Std. Error Mean .76085 .72386

Sig. Mean Std. Error 98% Confidence (2-tailed) Difference Difference Interval of the Difference Lower Upper .000 5.0490 1.06237 2.38419 7.71372 .000

5.0490

1.05017

2.41443

7.68347

7.8 Computer Examples 405

Looking at the statistical significance values, which are greater than 0.05, we do not reject the null hypothesis.

Example 7.8.8 (Paired t-Test) For the data of Example 7.5.7, use SPSS to test whether the data provide sufﬁcient evidence for the claim that the new program reduces blood glucose level in diabetic patients. Use α = 0.05.

Solution Enter after data in column C1 and before data in column C2. Then:

Analyze > Compare Means > Paired-Sample T-Test > bring after and before to Paired Variables: so that it will look after-before > click OK We obtain the following output. Paired Samples Statistics Mean N Std. Deviation Std. Error Mean Pair 1 AFTER 171.3000 10 47.11228 14.89821 BEFORE 243.2000 10 40.12979 12.69015 Paired Samples Correlations N Correlation Sig. Pair 1 AFTER & BEFORE 10 .179 .621 Paired Samples Test Paired Differences Mean

t

df Sig. (2-tailed)

Std. Deviation

Std. Error 95% Confidence Mean Interval of the Difference Upper

Pair 1 AFTER -- −71.9000 BEFORE

17.75791 56.15544

Lower −112.0712

−31.7288 −4.049 9

.003

Because the significance level for the test is 0.003, which is less than α = 0.05, we reject the null hypothesis.

7.8.3 SAS Examples To conduct a hypothesis test using SAS, we could use proc ttest, or proc means with option of computing the t-value and corresponding probability. However, to use this, we need a hypothesis of the form H0 : μ = 0. For testing nonzero values, H0 : μ = μ0 , we must create a new variable

406 CHAPTER 7 Hypothesis Testing

by subtracting μ0 from each observation, and then use the test procedure for this new variable. The following example illustrates this concept.

Example 7.8.9 (t-Test): The following radar measurements of speed (in miles per hour) are obtained for 10 vehicles traveling on a stretch of interstate highway. 66

74

79

80

69

77

78

65

79

81

Do the data provide sufﬁcient evidence to indicate that the mean speed at which people travel on this stretch of highway is at least 75 mph? Test using α = 0.01. Use an SAS procedure to do the analysis.

Solution In the SAS editor, type in the following commands. data speed; title ’Test on highway speed’; input X @@; Y=X-75; datalines; 66 74 79 80 69 77 78 65 79 81 ; PROC TTEST data=speed; run; We obtain the following output. Test on highway speed The TTEST Procedure Statistics Lower CL

Upper CL

Variable N

Mean

Mean

Mean

X Y

70.511 −4.489

74.8 −0.2

79.089 4.0895

10 10

Std Dev 4.1245 4.1245 Variable X Y

Lower CL Std Dev 5.9963 5.9963 T-Tests DF 9 9

Upper CL Std Dev 10.947 10.947 t Value 39.45 −0.11

Std Err 1.8962

Pr > |t| |t| −4.05 0.0029 Because the p-value 0.0029 is less than α = 0.05, we reject the null hypothesis.

PROJECTS FOR CHAPTER 7 7A. Testing on Computer-Generated Samples (a) Small sample test: Generate a sample of size 20 from a normal population with μ = 10, and σ 2 = 4. (i) Perform a t-test for the test H0 : μ = 10 versus Ha : μ = 10 at level α = 0.05. (ii) Perform the test H0 : σ 2 = 4 versus Ha : σ 2 = 4 at level α = 0.05. Repeat the procedure 10 times, and comment on the results. (b) Large sample test:

Projects for Chapter 7 409

Generate a sample of size 50 from a normal population with μ = 10, and σ 2 = 4. Perform a z-test for the test H0 : μ = 10 versus Ha : μ = 10 at level α = 0.05. Repeat the procedure 10 times and comment on the results.

7B. Conducting a Statistical Test with Confidence Interval Let θ be any population parameter. Consider the three tests of hypotheses H0 : θ = θ0 vs. Ha : θ > θ0

(1)

H0 : θ = θ0 vs. Ha : θ < θ0

(2)

H0 : θ = θ0 vs. Ha : θ = θ0

(3)

The following procedure can be exploited to test a statistical hypothesis utilizing the conﬁdence intervals.

Procedure to Use Confidence Interval for Hypothesis Testing Let θ be any population parameter. (a) For test (1), that is, H0 : θ = θ0 vs. Ha : θ > θ0

choose a value for α. From a random sample, compute a conﬁdence interval for θ using a conﬁdence coefﬁcient equal to 1 − 2α. Let L be the lower end point of this conﬁdence interval. Reject H0 if θ0 < L. That is, we will reject the null hypothesis if the conﬁdence interval is completely to the right of θ0 . (b) For test (2), that is, H0 : θ = θ0 vs. Ha : θ < θ0

choose a value for α. From a random sample, compute a conﬁdence interval for θ using a conﬁdence coefﬁcient equal to 1 − 2α. Let U be the upper end point of this conﬁdence interval. Reject H0 if U < θ0 . That is, we will reject the null hypothesis if the conﬁdence interval is completely to the left of θ0 . (c) For test (3), that is, H0 : θ = θ0 vs. Ha : θ = θ0

410 CHAPTER 7 Hypothesis Testing

choose a value for α. From a random sample, compute a conﬁdence interval for θ using a conﬁdence coefﬁcient equal to 1 − α. Let L be the lower end point and U be the upper end point of this conﬁdence interval. Reject H0 if θ0 < L or U < θ0 . That is, we will reject the null hypothesis if the conﬁdence interval does not contain θ0 . (i) For any large data set, conduct all three of these hypothesis tests using a conﬁdence interval for the population mean. (ii) For any small data set, conduct all three of these hypothesis tests using a conﬁdence interval for the population mean.

Chapter

8

Linear Regression Models Objective: In this chapter we will study linear relationships in sample data and use the method of least squares to estimate the necessary parameters. 8.1 Introduction 412 8.2 The Simple Linear Regression Model 413 8.3 Inferences on the Least-Squares Estimators 428 8.4 Predicting a Particular Value of Y 437 8.5 Correlation Analysis 440 8.6 Matrix Notation for Linear Regression 445 8.7 Regression Diagnostics 451 8.8 Chapter Summary 454 8.9 Computer Examples 455 Projects for Chapter 8 461

Sir Francis Galton (Source: http://en.wikipedia.org/wiki/Francis_Galton)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

411

412 CHAPTER 8 Linear Regression Models

English scientist Sir Francis Galton (1822–1911), a cousin of Charles Darwin, made signiﬁcant contributions to both genetics and psychology. He is the inventor of regression and a pioneer in applying statistics to biology. One of the data sets that he considered consisted of the heights of fathers and ﬁrst sons. He was interested in predicting the height of son based on the height of father. Looking at the scatterplots of these heights, Galton saw that the trend was linear and increasing. After ﬁtting a line to these data (using the techniques described in this chapter), he observed that for fathers whose heights were taller than the average, the regression line predicted that taller fathers tended to have shorter sons and shorter fathers tended to have taller sons. There is a regression toward the mean. That is how the method of this chapter got its name: regression.

8.1 INTRODUCTION In earlier chapters, we were primarily concerned about inferences on population parameters. In this chapter, we examine the relationship between one or more variables and create a model that can be used for predictive purposes. For example, consider the question “Is there statistical evidence to conclude that the countries with the highest average blood-cholesterol levels have the greatest incidence of heart disease?” It is important to answer this if we want to make appropriate lifestyle and medical choices. We will study the relationship between variables using regression analysis. Our aim is to create a model and study inferential procedures when one dependent and several independent variables are present. We denote by Y the random variable to be predicted, also called the dependent variable (or response variable) and by xi the independent (or predictor) variables used to model (or predict) Y . For example, let (x, y) denote the height and weight of an adult male. Our interest may be to ﬁnd the relationship between height and weight from a sample measurements of n individuals. The process of ﬁnding a mathematical equation that best ﬁts the noisy data is known as regression analysis. In his book Natural Inheritance, Sir Francis Galton introduced the word regression in 1889 to describe certain genetic relationships. The technique of regression is one of the most popular statistical tools to study the dependence of one variable with respect to another. There are different forms of regression: simple linear, nonlinear, multiple, and others. The primary use of a regression model is prediction. When using a model to predict Y for a particular set of values of x1 , . . . , xk , one may want to know how large the error of prediction might be. Regression analysis, in general after collecting the sample data, involves the following steps.

PROCEDURE FOR REGRESSION MODELING 1. Hypothesize the form of the model as Y = f (x1 , . . . , xk ; β0 , β1 , . . . , βk ) + ε. Here ε represents the random error term. We assume that E(ε) = 0 but Var (ε) = σ 2 is unknown. From this we can obtain E(Y ) = f (x1 , . . . , xk ; β0 , β1 , . . . , βk ). 2. Use the sample data to estimate unknown parameters in the model. 3. Check for goodness of ﬁt of the proposed model. 4. Use the model for prediction.

8.2 The Simple Linear Regression Model 413

The function f (x1 , . . . , xk ; β0 , β1 , . . . , βk )(k ≥ 1) contains the independent or predictor variables x1 , . . . , xn (assumed to be nonrandom) and unknown parameters or weights β0 , β1 , . . . , βk and ε representing the random or error variable. We now proceed to introduce the simplest form of a regression model, called simple linear regression.

8.2 THE SIMPLE LINEAR REGRESSION MODEL Consider a random sample of n observations of the form (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ), where X is the independent variable and Y is the dependent variable, both being scalars. A preliminary descriptive technique for determining the form of relationship between X and Y is the scatter diagram. A scatter diagram is drawn by plotting the sample observations in Cartesian coordinates. The pattern of the points gives an indication of a linear or nonlinear relationship between the variables. In Figure 8.1a, the relationship between x and y is fairly linear, whereas the relationship is somewhat like a parabola in Figure 8.1b, and in Figure 8.1c there is no obvious relationship between the variables. Once the scatter diagram reveals a linear relationship, the problem then is to ﬁnd the linear model that best ﬁts the given data. To this end, we will ﬁrst give a general deﬁnition of a linear statistical model, called a multiple linear regression model.

y

y

x

x (a) Linear relationship

(b) Quadratic relationship

y

x (c) No relationship ■ FIGURE 8.1 Scatter diagram.

414 CHAPTER 8 Linear Regression Models

Deﬁnition 8.2.1 A multiple linear regression model relating a random response Y to a set of predictor variables x1 , . . . , xk is an equation of the form Y = β0 + β1 x1 + β2 x2 + · · · + βk xk + ε

where β0 , . . . , βk are unknown parameters, x1 , . . . , xk are the independent nonrandom variables, and ε is a random variable representing an error term. We assume that E(ε) = 0, or equivalently, E(Y ) = β0 + β1 x1 + β2 x2 + · · · + βk xk .

To understand the basic concepts of regression analysis we shall consider a single dependent variable Y and a single independent nonrandom variable x. We assume that there are no measurement errors in xi . The possible measurement errors in y and the uncertainties in the assumed model are expressed through the random error ε. Our inability to provide an exact model for a natural phenomenon is expressed through the random term ε, which will have a speciﬁed probability distribution (such as a normal) with mean zero. Thus, one can think of Y as having a deterministic component, E(Y ), and a random component, ε. If we take k = 1 in the multiple linear regression model, we have a simple linear regression model. Deﬁnition 8.2.2 If Y = β0 + β1 x + ε, this is called a simple linear regression model. Here, β0 is the y-intercept of the line and β1 is the slope of the line. The term ε is the error component. This basic linear model assumes the existence of a linear relationship between the variables x and y that is disturbed by a random error ε. The known data points are the pairs (x1 , y2 ), (x2 , y2 ), . . . , (xn , yn ); the problem of simple linear regression is to ﬁt a straight line optimal in some sense to the set of data, as shown in Figure 8.2.

25 20 15 10 5 0 ⫺5 ⫺10 ⫺5

0

5

10

■ FIGURE 8.2 Scatterplot and least-squares regression line.

15

8.2 The Simple Linear Regression Model 415

Now, the problem becomes one of ﬁnding estimators for β0 and β1 . Once we obtain the “good” estimators βˆ 0 and βˆ 1 , we can ﬁt a line to the data given by the prediction equation Yˆ = βˆ 0 + βˆ 1 x. The question then becomes whether this predicted line gives the “best” (in some sense) description of the data. We now describe the most widely used technique, called the method of least squares, to obtain the estimators or weights of the parameters.

8.2.1 The Method of Least Squares As stated (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) are the n observed data points, with corresponding errors εi , i = 1, . . . , n. That is, Yi = β0 + β1 xi + εi ,

i = 1, 2, . . . , n.

We assume that the errors εi , i = 1, . . . , n are independent and identically distributed with E(εi ) = 0, i = 1, . . . , n, and Var(εi ) = σ 2 , i = 1, . . . , n. One of the ways to decide on how well a straight line ﬁts the set of data is to determine the extent to which the data points deviate from the line. The straight line model for the response Y for a given x is Y = β0 + β1 x + ε.

Because we assumed that E(ε) = 0, the expected value of Y is given by E(Y ) = β0 + β1 x.

The estimator of the E(Y ), denoted by Yˆ , can be obtained by using the estimators βˆ 0 and βˆ 1 of the parameters β0 and β1 , respectively. Then, the ﬁtted regression line we are looking for is given by Yˆ = βˆ 0 + βˆ 1 x.

For observed values (xi , yi ), we obtain the estimated value of yi as yˆ i = βˆ 0 + βˆ 1 xi .

The deviation of observed yi from its predicted value yˆ i , called the ith residual, is deﬁned by ei = yi − yˆ i = yi − βˆ 0 + βˆ 1 xi .

The residuals, or errors ei , are the vertical distances between observed and predicted values of yi s (Figure 8.3). y .

ei

. . .

. . .

x ■ FIGURE 8.3 Illustration of ei .

416 CHAPTER 8 Linear Regression Models

Deﬁnition 8.2.3 The sum of squares for errors (SSE) or sum of squares of the residuals for all of the n data points is SSE =

n

e21 =

i=1

n

2 yi − βˆ 0 + βˆ 1 xi

i=1

The least-squares approach to estimation is to ﬁnd βˆ 0 and βˆ 1 that minimize the sum of squared residuals, SSE. Thus, in the method of least squares, we choose β0 and β1 so that SSE is a minimum. The quantities βˆ 0 and βˆ 1 that make the SSE a minimum are called the least-squares estimates of the parameters β0 and β1 , and the corresponding line yˆ = βˆ 0 + βˆ 1 x is called the least-squares line. Deﬁnition 8.2.4 The least-squares line yˆ = βˆ 0 + βˆ 1 x is one that satisﬁes the following property: SSE =

n

yi − yˆ i

2

i=1

is a minimum for any other straight line model with SE =

n

yi − yˆ i = 0

i=1

Thus, the least-squares line is a line of the form y = b0 + b1 x for which the error sum of squares n 2 i=1 (yi − b0 − b1 x) is a minimum. The minimum is taken over all values of b0 and b1 , and (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) are observed data pairs. The problem of ﬁtting a least-squares line now reduces to ﬁnding the quantities βˆ 0 and βˆ 1 that minimize the error sum of squares.

8.2.2 Derivation of βˆ 0 and βˆ 1

Now we derive expressions for βˆ 0 and βˆ 1 . If SSE attains a minimum, then the partial derivatives of SSE with respect to β0 and β1 are zeros. That is, n

∂ ∂SSE = ∂β0

i=1

∂β0

=− =2

4

[yi − (β0 + β1 xi )]2

n

2 [yi − (β0 + β1 xi )]

i=1n

yi − nβ0 − β1

i=1

i=1

and ∂ ∂SSE = ∂β1

n

n

4 [yi − (β0 + β1 xi )]2

i=1

∂β1

(8.1)

xi

=0

8.2 The Simple Linear Regression Model 417

=−

n

2 [yi − (β0 + β1 xi )]xi

(8.2)

i=1

= −2

n

xi yi − β0

i=1

n

n

xi − β1

i=1

x12

= 0.

i=1

Equations (8.1) and (8.2) are called the least squares equations for estimating the parameters of a line. From (8.1) and (8.2) we obtain a set of linear equations called the normal equations, n

yi = nβ0 + β1

i=1

n

(8.3)

xi

i=1

and n

xi yi = β0

i=1

n

xi + β1

i=1

n

xi2 .

(8.4)

i=1

Solving for β0 and β1 from Equations (8.3) and (8.4), we obtain n

(xi − x) (yi − y)

i=1 βˆ 1 = n

n =

(xi − x)2

i=1

n

xi yi −

i=1 n n x12 − i=1

n

xi

i=1

n

n

yi

i=1 2

n

n

=

xi yi −

i=1

xi

i=1

i=1

x12 −

n

yi

i=1

n

n

xi

i=1 n

2

(8.5)

xi

i=1

n

and βˆ 0 = y − βˆ 1 x.

(8.6)

To simplify the formula for βˆ 1 , set Sxx =

n i=1

xi2 −

n

2

xi

i=1

n

, Sxy =

n i=1

xi yi −

n

xi

i=1

n

yi

i=1

n

we can rewrite (8.5) as Sxy βˆ 1 = . Sxx

It can be shown (by using the second derivatives) that (8.5) and (8.6) do indeed minimize SSE. Now we will summarize the procedure for ﬁtting a least-squares line.

418 CHAPTER 8 Linear Regression Models

PROCEDURE FOR FITTING A LEAST-SQUARES LINE 1. Form the n data points (x1 , y1 ),(x2 , y2 ), . . . ,(xn , yn ), and compute the following quantities: n n n n 2 n 2 i=1 xi , i=1 xi , i=1 yi , i=1 yi , and i=1 xi yi . Also compute the sample means, x = (1/n) ni=1 xi and y = (1/n) ni=1 yi . 2. Compute Sxx =

n

x12 −

n

2 xi

i=1

=

n

i=1

n

xi − x

2

i=1

and Sxy =

n i=1

xi yi −

n

xi

i=1

n

yi

i=1

n

=

n

xi − x

yi − y .

i=1

3. Compute βˆ 0 and βˆ 1 by substituting the computed quantities from step 1 into the equations Sxy βˆ 1 = Sxx and βˆ 0 = y − βˆ 1 x . 4. The ﬁtted least-squares line is yˆ = βˆ 0 + βˆ 1 x . 5. For a graphical representation, in the xy -plane, plot all the data points and draw the least-squares line obtained in step 4.

Once we have accomplished the best-ﬁt combination of the two parameters β0 and β1 , any deviation of either parameter away from its optimum value will cause the sum of squares error to increase. Thus, the optimum combination of the pairs (βˆ 0 , βˆ 1 ) forms a global minimum point of the error sum of squares among all possible values of β0 and β1 for the given data set.

8.2 The Simple Linear Regression Model 419

Example 8.2.1 Use the method of least squares to ﬁt a straight line to the accompanying data points. Give the estimates of β0 and β1 . Plot the points and sketch the ﬁtted least-squares line. The observed data values are given in the following table. −1 −5

x y

0 −4

−2 −7

2 2

5 6

6 9

8 13

11 21

12 20

−3 −9

Solution Form a table to compute various terms

xi −1 0 2 −2 5 6 8 11 12 −3

xi = 38

yi −5 −4 2 −7 6 9 13 21 20 −9

Sxx =

n

x12 −

n

Sxy =

n i=1

xi yi −

n

= 408 −

xi

n

2 xi = 408

(38)2 = 263.6 10

yi

i=1

n x = 3.8

xi yi = 709

xi

n

i=1

1 0 4 4 25 36 64 121 144 9

2

i=1

i=1

yi = 46

xi2

xi yi 5 0 4 14 30 54 104 231 240 27

and

= 709 −

(38)(46) = 534.2 10

y = 4.6.

Therefore, Sxy 534.2 = 2.0266 βˆ 1 = = Sxx 263.6

420 CHAPTER 8 Linear Regression Models

25 20 15 10 5 0 ⫺5 ⫺10 ⫺5

0

5

10

15

■ FIGURE 8.4 Simple regression line.

and βˆ 0 = y − βˆ 1 x = 4.6 − (2.0266)(3.8) = −3.1011.

Hence, the least-squares line for these data is yˆ = βˆ 0 + βˆ 1 x = −3.1011 + 2.0266x

and its plot is shown in Figure 8.4. Recall that for the regression line yˆ = βˆ 0 + βˆ 1 x. we have deﬁned SSE to be SSE =

n

yi − yˆ i

2

=

i=1

n

yi − βˆ 0 − βˆ 1 xi

SSE = Syy − βˆ 1 Sxy , where Syy =

n i=1

We know that n

yi − βˆ 0 − βˆ 1 xi

2

i=1

=

n

yi − y + βˆ 1 x − βˆ 1 xi

i=1

.

i=1

We now show that

SSE =

2

2

y12 −

n

2 yi

i=1

n

=

n i=1

(yi − y)2 .

8.2 The Simple Linear Regression Model 421

=

n

(yi − y) − βˆ 1 (xi − x)

2

i=1

=

n

(yi − y)2 + βˆ 12

i=1

n

(xi − x)2 − 2βˆ 1

i=1

n

(xi − x) (yi − y)

i=1

= Syy + βˆ 12 Sxx − 2βˆ 1 Sxy .

Recall that βˆ 1 =

Sxy Sxx .

Substituting for βˆ 1 , we obtain

SSE = Syy − = Syy −

Sxy 2 Sxy Sxx − 2 Sxy Sxx Sxx

Sxy Sxy Sxx

= Syy − βˆ 1 Sxy .

8.2.3 Quality of the Regression Once we obtain the linear model, the question is, How well does this line ﬁt the data? We could make use of the residuals eˆ i = yi − βˆ 0 − βˆ 1 xi

to answer the question and to assess the quality of the ﬁt. If our model is good, then the residual eˆ i should be close to the random error ε with mean zero. Furthermore, the residuals should contain little or no information about the model, and there should be no recognizable pattern. If we plot the residuals versus the independent variables on the x-axis, ideally, the plot should look like a horizontal blur, the residuals showing no relationship to the x-values, as shown by Figure 8.5. Otherwise, these plots reveal a not very good ﬁt of the given data, as shown by Figure 8.6, and we need to improve our model speciﬁcations. Thus, a symmetric trend in the plot of residuals ei versus xi or yˆ i (i = 1, . . . , n) indicates that the assumed regression model is not correct.

e

y

■ FIGURE 8.5 Good fit.

422 CHAPTER 8 Linear Regression Models

e

y

■ FIGURE 8.6 Not a good fit.

Whereas the residual plots give us a visual representation of the quality of ﬁt, a numerical measure of how well the regression explains the data is obtained by calculating the coefﬁcient of determination, also called the R2 of the regression. This is discussed in Project 8B. Regression analysis with any of the standard statistical software packages will contain an output value of the R2 . This value will be between 0 and 1; closer to 1 means a better ﬁt. For example, if the value of R2 is 0.85, the regression captures 85% of the variation in the dependent variable. This is generally considered good regression.

8.2.4 Properties of the Least-Squares Estimators for the Model Y = β0 + β1 x + ε We discussed in Chapter 4 the concept of sampling distribution of sample statistics such as that of X. Similarly, knowledge of the distributional properties of the least-squares estimators βˆ 0 and βˆ 1 is necessary to allow any statistical inferences to be made about them. The following result gives the sampling distribution of the least-squares estimators. Theorem 8.2.1 Let Y = β0 + β1 x + ε be a simple linear regression model with ε ∼ N(0, σ 2 ), and let the errors εi associated with different observations yi (i = 1, . . . , N) be independent. Then (a) βˆ 0 and βˆ 1 have normal distributions. (b) The mean and variance are given by E βˆ 0 = β0 ,

Var βˆ 0 =

1 x2 + n Sxx

σ2,

and E βˆ 1 = β1 ,

σ2 Var βˆ 1 = , Sxx

n 2 1 xi . In particular, the least-squares estimators βˆ 0 and βˆ 1 are unbiased n i=1 i=1 estimators of β0 and β1 , respectively.

where Sxx =

n

xi2 −

8.2 The Simple Linear Regression Model 423

Proof. We know that Sxy βˆ 1 = Sxx =

n 1 (xi − x) Yi − Y Sxx i=1

% n & n 1 (xi − x) Yi − Y (xi − x) = Sxx i=1

i=1

n 1 (xi − x) Yi = Sxx i=1

where the last equality follows from the fact that

n

(xi − x) =

i=1

distributed, the sum

n

xi − nx = 0. Because Yi is normally

i=1

n 1 (xi − x)Yi is also normal. Furthermore, Sxx i=1

E[βˆ 1 ] =

n 1 (xi − x)E[Yi ] Sxx i=1

=

n 1 (xi − x)(β0 + β1 xi ) Sxx i=1

=

n n β1 β0 (xi − x) + (xi − x)xi Sxx Sxx i=1

= β1

1 Sxx

i=1

n

(xi − x)xi

i=1

% n & n 1 2 = β1 x1 − x xi Sxx i=1

⎡ = β1

n ⎢

1 ⎢ ⎢ Sxx ⎣

i=1

i=1

⎛ n ⎞⎤ n x i ⎜ ⎟⎥ ⎜ i=1 ⎟⎥ x12 − xi ⎜ ⎟⎥ ⎝ n ⎠⎦

⎡ ⎢ n ⎢ ⎢ x12 − ⎢ ⎢ ⎣i=1

= β1

1 Sxx

= β1

1 Sxx = β1 . Sxx

i=1

n

2 ⎤ xi

i=1

n

⎥ ⎥ ⎥ ⎥ ⎥ ⎦

424 CHAPTER 8 Linear Regression Models

For the variance we have, Var βˆ 1 = Var

%

n 1 (xi − x) Yi Sxx

&

i=1

n 1 (xi − x)2 Var [Yi ] = 2 Sxx

(since the Yi ’s are independent)

i=1

n 1 (xi − x)2 Var (Yi ) = Var (β0 + β1 + εi ) = Var (εi ) = σ 2 = σ2 2 Sxx i=1

=

σ2 Sxx

.

Note that both Y and βˆ 1 are normal random variables. It can be shown that they are also independent (see Exercise 8.3.3). Because βˆ 0 = y − βˆ 1 x is a linear combination of Y and βˆ 1 , it is also normal. Now, # $ E βˆ 0 = E Y − βˆ 1 x = E Y − xE βˆ 1 & % n n 1 1 (β0 + β1 x) − xβ1 Yi − xβ1 = =E n n i=1

i=1

= β0 + xβ1 − xβ1 = β0 .

The variance of βˆ 0 is given by Var βˆ 0 = Var Y − βˆ 1 x # $ = Var Y + x2 Var βˆ 1 (since Y and βˆ 1 are independent) 1 x2 σ 2 x2 σ2 = σ2. + + = n Sxx n Sxx

If an estimator θˆ is a linear combination of the sample observations and has a variance that is less than or equal to that of any other estimator that is also a linear combination of the sample observations, then θˆ is said to be a best linear unbiased estimator (BLUE) for θ. The following result states that among all unbiased estimators for β0 and β1 which are linear in Yi , the least-square estimators have the smallest variance.

GAUSS–MARKOV THEOREM Theorem 8.2.2 Let Y = β0 + β1 x + ε be the simple regression model such that for each xi ﬁxed, each Yi is an observable random variable and each ε = εi , i = 1, 2, . . . , n is an unobservable random variable. Also, let the random variable εi be such that E[εi ] = 0, Var(εi ) = σ 2 and Cov(εi , εj ) = 0, if i = j. Then the least-squares estimators for β0 and β1 are best linear unbiased estimators.

8.2 The Simple Linear Regression Model 425

It is important to note that even when the error variances are not constant, there still can exist unbiased least-square estimators, but the least-squares estimators do not have minimum variance.

8.2.5 Estimation of Error Variance σ 2 The greater the variance, σ 2 , of the random error ε, the larger will be the errors in the estimation of model parameters β0 and β1 . We can use already-calculated quantities to estimate this variability of errors. It can be shown that (see Exercise 8.2.1(b)) that E(SSE) = (n − 2)σ 2 .

Thus, an unbiased estimator of the error variance, σ 2 , is σˆ 2 = (SSE)/(n − 2). We will denote (SSE)/ (n − 2) by MSE (Mean Square Error).

EXERCISES 8.2 8.2.1.

For a random sample of size n, (a) Show that the error sum of squares can be expressed by SSE = Syy − βˆ 1 Sxy .

(b) Show that E[SSE] = (n − 2)σ 2 . 8.2.2.

The following are midterm and ﬁnal examination test scores for 10 students from a calculus class, where x denotes the midterm score and y denotes the ﬁnal score for each student. x y

68 74

87 79

75 80

91 93

82 88

77 79

86 97

82 95

75 89

79 92

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.3.

The following data give the annual incomes (in thousands of dollars) and amounts (in thousands of dollars) of life insurance policies for eight persons. Annual income 42 58 27 36 70 24 53 37 Life insurance 150 175 25 75 250 50 250 100 (a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph.

8.2.4.

Consider a simple linear model Y = β0 + β1 x + ε, with ε ∼ N(0, σ 2 ). Show that n

−σ 2 cov(βˆ 0 , βˆ 1 ) =

xi

i=1 n n xi2 − i=1

n i=1

2 . xi

426 CHAPTER 8 Linear Regression Models

8.2.5.

(a) Show that the least-squares estimates of β0 and β1 of a line can be expressed as βˆ 0 = y − βˆ 1 x

and

n

(xi − x) (yi − y) i=1 . βˆ 1 = n (xi − x)2 i=1

(b) Using part (a), show that the line ﬁtted by the method of least squares passes through the point (x, y). 8.2.6.

Crickets make their chirping sounds by rapidly sliding one wing over the other. The faster they move their wings, the higher the number of chirping sounds that are produced. Scientists have noticed that crickets move their wings faster in warm temperatures than in cold temperatures (they also do this when they are threatened). Therefore, by listening to the pitch of the chirp of crickets, it is possible to tell the temperature of the air. The following table gives the number of cricket chirps per 13 seconds recorded at 10 different temperatures. Assume that the crickets are not threatened. Temperature Number of chirps

60 20

66 25

70 31

73 33

78 36

80 39

82 42

87 48

90 49

92 52

Calculate the least-squares regression line for these data and discuss its usefulness. 8.2.7.

Consider the regression model y = β1 x + ε

where ε ∼ N(0, σ 2 ). Show that n

xi yi i=1 . βˆ 1 = n 2 xi i=1

8.2.8.

A farmer collected the following data, which show crop yields for various amounts of fertilizer used. Fertilizer (pounds/100 sq. ft) Yield (bushels)

0 6

4 7

8 10

10 13

15 17

18 18

20 22

25 23

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.9.

An economist desires to estimate a line that relates personal disposable income (DI) to consumption expenditures (CE). Both DI and CE are in thousands of dollars. The following gives the data for a random sample of nine households of size four.

8.2 The Simple Linear Regression Model 427

DI CE

25 21

22 20

19 17

36 28

40 34

47 41

28 25

52 45

60 51

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.10.

The following data represent systolic blood pressure readings on 10 randomly selected females between ages 40 and 82. Age (x) Systolic (y)

63 151

70 149

74 164

82 157

60 144

44 130

80 157

71 160

71 121

41 125

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.11.

Itisbelievedthatexposuretosolarradiationincreasesthepathogenesisofmelanoma.Suppose that the following data give sunspot relative number and age-adjusted total incidence (incidence is the number of cases per 100,000 population) for 8 different years in a certain region. Sunspot relative number Incidence total

104 4.7

12 1.9

40 3.8

75 2.9

110 0.9

180 2.7

175 3.9

30 1.6

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.12.

It is believed that the average size of a mammal species is a major factor in the period of gestation (the period of development in the uterus from conception until birth). In general, it is observed that the bigger the mammal is, the longer the gestation period. Table 8.2.1 gives adult mass in kilograms and gestation period in weeks of some species (source: http://www.saburchill.com/chapters/chap0037.html).

Table 8.2.1 Species

Adult mass (kg)

Gestation period (weeks)

African elephant

6000

88

Horse

400

48

Grizzly bear

400

30

Lion

200

17

Wolf

34

9

Badger

12

8

Rabbit

2

4.5

Squirrel

0.5

3.5

428 CHAPTER 8 Linear Regression Models

Table 8.2.2 Species

Gestation period (weeks)

Indian elephant

89.0

Camel

57.0

Sea lion

51.4

Dog

8.7

Rat

3.0

Hamster

2.3

(a) Calculate the least-squares regression line for these data with adult mass as the independent variable. (b) Plot the points and the least-squares regression line on the same graph. (c) Calculate the least-squares regression line for these data with gestation period as the independent variable. (d) Assuming that the regression model of part (c) holds for all mammals, estimate the adult mass in kilograms for the mammals given in Table 8.2.2.

8.3 INFERENCES ON THE LEAST-SQUARES ESTIMATORS Once we obtain the estimators of the slope β1 and intercept β0 of the model regression line, we are in a position to use Theorem 8.2.1 to make inferences regarding these model parameters. Using the properties of βˆ 0 and βˆ 1 , in this section we study the conﬁdence intervals and hypothesis tests concerning these parameters. From Theorem 8.2.1, we can write Z1 =

βˆ 1 − β1 √σ Sxx

∼ N(0, 1).

Also, it can be shown that SSE/σ 2 is independent of βˆ 1 and has a chi-square distribution with n − 2 degrees of freedom. Let the mean square error be deﬁned by MSE =

n 1 SSE = [yi − (βˆ 0 + βˆ 1 xi )]2 . n−2 n−2 i=1

Then using Deﬁnition 4.2.2, we have Z βˆ 1 − β1 tβ1 = 8 = 9 SSE σ2

n−2

MSE Sxx

8.3 Inferences on the Least-Squares Estimators 429

which follows the t-distribution with n − 2 degrees of freedom. Similarly, let Z0 =

βˆ − β0 0 ∼ N(0, 1). 2 σ 1n + Sx yy

Also, it can be shown that βˆ 0 and SSE are independent. Hence, tβ0 = 8

z0 SSE σ2

βˆ 0 − β0 1/2 2 MSE 1n + Sx

=

xx

n−2

follows the t-distribution with n − 2 degrees of freedom. From these derivations, we can obtain the following procedure about the conﬁdence intervals for the slopes β1 and for the intercept β0 .

PROCEDURE FOR OBTAINING CONFIDENCE INTERVALS FOR β0 AND β1 1. Compute Sxx , Sxy , Sxy , y , and x as in the procedure for ﬁtting a least-squares line. 2. Compute βˆ 1 , βˆ 0 using equations βˆ 1 = (Sxy )/(Sxx ) and βˆ 0 = y − βˆ 1 x , respectively. 3. Compute SSE by SSE = Syy − βˆ 1 Sxy . 4. Deﬁne MSE (mean square error) to be MSE =

SSE , n−2

where n = Number of pairs of observations x1 , y1 , . . . , xn , yn . 5. A (1 − α)100% conﬁdence interval for β1 is given by +

βˆ 1 − tα/2,n−2

+ MSE , βˆ 1 + tα/2,n−2 Sxx

MSE Sxx

where ta/2 is the upper tail α/2-point based on a t-distribution with (n − 2) degrees of freedom. 6. A (1 − α)100% conﬁdence interval for β0 is given by ⎛

%

⎝βˆ 0 − tα/2, n−2 MSE

x2 1 + n Sxx

& 2

% , βˆ 0 + tα/2,n−2 MSE

x2 1 + n Sxx

We illustrate this procedure for obtaining conﬁdence limits with an example.

& 1/2 ⎞ ⎠.

430 CHAPTER 8 Linear Regression Models

Example 8.3.1 For the data of Example 8.2.1: (a) Construct a 95% conﬁdence interval for β0 and interpret. (b) Construct a 95% conﬁdence interval for β1 and interpret.

Solution The following calculations were obtained in Example 8.2.1: Sxx = 263.6, Sxy = 534.2, y = 4.6 and x = 3.8. Also, βˆ 1 = 2.0266, βˆ 0 = −3.1011. In addition to those calculations, we can compute n

y12 = 1302 and Syy =

i=1

n

y12 −

n

2 yi

i=1

n

i=1

= 1302 −

(46)2 = 1090.4. 10

Now, SSE = Syy − βˆ 1 Sxy = 1090.4 − (2.0266)(534.2) = 7.79028. Hence, MSE =

SSE 7.79028 = = 0.973785. n−2 8

Now from the t-table, we have t0.025,8 = 2.306. (a) A 95% confidence interval for β0 is given by ⎛

%

2 ⎝βˆ 0 − tα/2,n−2 MSE 1 + x n Sxx

&1/2

%

x2 1 + , βˆ 0 + tα/2,n−2 MSE n Sxx

&1/2 ⎞ ⎠

⎛

&1/2 % (3.8)2 1 ⎝ + = −3.1011 − (2.306) (0.973785) 10 263.6 %

1 (3.8)2 −3.1011 + (2.306) (0.973785) + 10 263.6

&1/2 ⎞ ⎠

From which we obtain a 95% confidence interval for β0 as (−3.9846, −2.2176). Thus, we can conclude with 95% confidence that the true value of the intercept, β0 , is between −3.9846 and −2.2176.

8.3 Inferences on the Least-Squares Estimators 431

(b) A 95% confidence interval for β1 is given by 8 8

MSE MSE ˆ ˆβ1 − tα/2,n−2 , β1 + tα/2,n−2 Sxx 8 Sxx 8 0.973785 0.973785 , 2.0266 + (2.306) = 2.0266 − (2.306) 236.6 236.6 from which we obtain a 95% confidence interval for β1 as (1.8864, 2.1668). Thus, we can conclude with 95% confidence that the true value of the slope of the linear regression model is between 1.8864 and 2.1663.

One of the assumptions for linear regression model that we have made is that the variance of the errors is a constant and independent of x. Errors with this property are called homoscedastic. If the variance of the errors is not constant, the errors are called heteroscedastic. In the heteroscedastic case, standard errors and conﬁdence intervals based on the assumption that s2 is an estimate of σ 2 may be somewhat deceptive. Now we introduce hypothesis testing concerning the slope and intercept of the ﬁtted least-squares line. We use tβ0 and tβ1 deﬁned earlier as the test statistic for testing hypotheses concerning β0 and β1 , respectively. The usual one- and two-sided alternatives apply. We proceed to summarize these test procedures. HYPOTHESIS TEST FOR β0 One-sided test

Two-sided test

H0 : β0 = β00 (β00 is a speciﬁc value of β0 )

H0 : β0 = β00

Ha : β0 > β00 or β0 < β00

Ha : β0 = β00

Test statistic:

Test statistic:

βˆ 0 − β00

" 1/2 x 1 MSE + n Sxx

βˆ 0 − β00

" 1/2 x 1 MSE + n Sxx

tβ0 = !

tβ0 = !

Rejection region:

Rejection region:

t > tα, (n−2) (upper tail region) t < −tα, (n−2) (lower tail region)

|t| > tα/2,(n−2)

Decision: If tβ0 falls in the rejection region, reject the null hypothesis at level of signiﬁcance α. Assumptions: Assume that the errors εi , i = 1, . . . , n are independent and normally distributed with E (εi ) = 0, i = 1, . . . , n, and Var (εi ) = σ 2 , i = 1, . . . , n.

We now illustrate this procedure with the following example.

432 CHAPTER 8 Linear Regression Models

Example 8.3.2 Using the data given in Example 8.2.1, test the hypothesis H0 : β0 = −3 versus Ha : β0 = −3 using the 0.05 level of signiﬁcance.

Solution We test H0 : β0 = −3 versus Ha : β0 = −3. Here β00 = −3. The rejection region is t < −2.306 or t > 2.306. From the calculations of the previous example, we have βˆ 0 − β00 1/ 2 2 MSE 1n + Sx

tβ0 =

xx

−3.1011 − (−3) 1 + (3.8)2 1/2 (0.973785) 10 263.6

=

= −0.26041. Because the test statistic does not fall in the rejection region, at α = 0.05, we do not reject H0 .

HYPOTHESIS TEST FOR β1 One-sided test H0 : β1 = β10 β10 is a speciﬁc value of β1

Two-sided test H0 : β1 = β10

Ha : β1 > β10 or β1 < β10

Ha : β1 = β10

Test statistic:

Test statistic:

βˆ 1 − β10 tβ1 = 8 MSE Sxx

βˆ 1 − β10 tβ1 = 8 MSE Sxx

Rejection region:

Rejection region:

t > tα,(n−2) (upper tail region) t < −tα,(n−2) (lower tail region)

|t| > tα/2,(n−2)

Decision: If tβ1 falls in the rejection region, reject the null hypothesis at conﬁdence level α. Assumptions: Assume that the errors εi , i = 1, . . . , n are independent and normally distributed with E (εi ) = 0, i = 1, . . . , n, and Var (εi ) = σ 2 , i = 1, . . . , n.

The test of hypothesis H0 : β1 = 0 answers the question, Is the regression signiﬁcant? If β1 = 0, we conclude that there is no signiﬁcant linear relationship between X and Y , and hence, the independent

8.3 Inferences on the Least-Squares Estimators 433

variable X is not important in predicting the values of Y if the relationship of Y and X is not linear. Note that if β1 = 0, then the model becomes y = β0 + ε. Thus, the question of the importance of the independent variable in the regression model translates into a narrower question of the test of hypothesis H0 : β1 = 0. That is, the regression line is actually a horizontal line through the intercept, β0 .

Example 8.3.3 Using the data given in Example 8.2.1, test the hypothesis H0 : β1 = 2 versus Ha : β1 = 2 using the 0.05 level of signiﬁcance.

Solution We test H0 : β1 = 2 vs. Ha : β1 = 2. We know that βˆ 1 = 2.0266. For α = 0.05 and n = 10, the rejection region is t < −2.306 or t > 2.306. The test statistic is βˆ 1 − β10 tβ1 = 8 MSE Sxx 2.0266 − 2 = 0.4376. = 8 2.0266 − 2 263.6 Because the test statistic does not fall in the rejection region, at α = 0.05, we do not reject H0 . Thus, for α = 0.05, the given data support the null hypothesis that the true value of the slope, β1 , of the regression line is equal to 2.

Another problem closely related to the problem of estimating the regression coefﬁcients β0 and β1 is that of estimating the mean of the distribution of Y for a given value of x, that is, estimating β0 + β1 x. For a ﬁxed value of x, say x0 , we have the following conﬁdence limits. A (1 − α)100% conﬁdence interval for β0 + β1 x is given by + 2 x0 − x 1 ˆβ0 + βˆ 1 x ± tα 2 se + / n Sxx where

+ se =

2 Syy − Sxy . (n − 2)Sxx

We could use the data from the previous example to easily calculate a conﬁdence interval for β0 +β1 x.

434 CHAPTER 8 Linear Regression Models

8.3.1 Analysis of Variance (ANOVA) Approach to Regression Another approach to hypothesis testing is based on ANOVA. A detailed explanation of this approach is given in Chapter 10. Here we present necessary steps for regression. The main reason for this presentation is the fact that most of the major statistical software outputs for regression analysis (see Section 8.9) are given in the form of ANOVA tables. It can be veriﬁed that (see Exercise 8.3.7) n

(yi − y)2 =

i=1

n

yi − yˆ i

2

+

i=1

n

yˆ i − y

2

.

i=1

Denoting SST =

n

(yi − y)2 , SSE =

i=1

n

yi − yˆ i

2

, and SSR =

i=1

n

yˆ i − y

2

,

i=1

the foregoing equation can be written as SST = SSR + SSE.

Note that the total sum of squares (SST ) is a measure of the variation of yi ’s around the mean y, and SSE is the residual or error sum of squares that measures the lack of ﬁt of the regression model. Hence, SSR (sum of squares of regression or model) measures the variation that can be explained by the regression model. We saw that to test the hypothesis H0 : β1 = 0 vs. Ha : β1 = 0, the statistic tβ1 = 8

βˆ 1 MSE Sxx

was used, where tβ1 follows a t-distribution with (n − 2) degrees of freedom. From Exercise 4.2.18, we know that tβ21 =

βˆ 12 MSE Sxx

follows an F -distribution with numerator degrees of freedom 1 and denominator degrees of freedom (n − 2). We can also verify that tβ21 =

MSR . MSE

8.3 Inferences on the Least-Squares Estimators 435

Table 8.1 ANOVA Table for Simple Regression Source of variation

Degrees of freedom

Sum of squares

Regression (model)

1

SSR

Error (residuals)

n−2

SSE

Total

n−1

SST

Mean sum of squares MSR =

F-ratio MSR MSE

SSR d.f.

SSE d.f.

Thus, to test H0 : β1 = 0 vs. Ha : β1 = 0, we could use the statistic MSR ∼ F (1, n − 2) MSE

and reject H0 if MSR ≥ Fα (1, n − 2). MSE

These can be summarized by Table 8.1, known as the ANOVA table. The last column in the ANOVA table gives the statistic (MSR)/(MSE). It is also customary to give another column with the p-value of the test.

Example 8.3.4 In a study of baseline characteristics of 20 patients with foot ulcers, we want to see the relationship between the stage of ulcer (determined using the Yarkony-Kirk scale, a higher number indicating a more severe stage, with range 1 to 6), and duration of ulcer (in days). Suppose we have the data shown in Table 8.2. (a) Give an ANOVA table to test H0 : β1 = 0 vs. Ha : β1 = 0. What is the conclusion of the test based on α = 0.05? (b) Write down the expression for the least-squares line.

Table 8.2 Stage of Ulcer (x)

4

3

5

4

4

3

3

4

6

3

Duration (d)

18

6

20

15

16

15

10

18

26

15

Stage of Ulcer (x)

3

4

3

2

3

2

2

3

5

6

Duration (d)

8

16

17

6

7

7

8

11

21

24

436 CHAPTER 8 Linear Regression Models

Table 8.3 Source of variation

Sum of squares

Mean sum of squares

F-ratio

p-Value

1

570.04

570.04

77.05

0.000

Error (residuals)

18

133.16

7.40

Total

19

703.20

Regression (model)

Degrees of freedom

Solution (a) We test H0 : β1 = 0 vs. Ha : β1 = 0. We will use Minitab to generate the ANOVA table (Table 8.3). Because the p-value is less than 0.001, for α = 0.05, we reject the null hypothesis that β1 = 0 and conclude that there is a relationship between the stage of ulcer and its duration. (b) Again, using the Minitab output, we get the least-squares line as d = 4.61x − 2.40.

EXERCISES 8.3 8.3.1.

An experiment was conducted to observe the effect of an increase in temperature on the potency of an antibiotic. Three one ounce portions of the antibiotic were stored for equal lengths of time at each of the following Fahrenheit temperatures: 40◦ , 55◦ , 70◦ , and 90◦ . The potency readings observed at the end of the experimental period were Potency reading, y 49 38 27 24 38 33 19 28 16 18 23 Temperature, x 40◦ 55◦ 70◦ 90◦ (a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line as a check on your calculations. (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.3.2.

Consider the data x 38 26 48 22 40 15 30 33 y 10 11 16 8 12 5 10 11 (a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line as a check on your calculations. (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.3.3.

Show that Y and βˆ 1 are independent, under the usual assumptions of a simple linear regression model.

8.3.4.

Using the data of Exercise 8.2.10, calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.4 Predicting a Particular Value of Y 437

8.3.5.

The following data represent survival time in days after a heart transplant and patient age in years at the time of transplant for 10 randomly selected patients. Age at transplant 28 41 46 53 39 36 47 29 48 44 Survival time, in days 7 278 44 48 406 382 1995 176 323 1846

(a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line. (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.3.6.

The following data represent weights of cigarettes (g) from different manufacturers and their nicotine contents (mg). Weight 15.8 14.9 9.0 4.5 15.0 17.0 8.6 12.0 4.1 16.0 Nicotine 0.957 0.886 0.852 0.911 0.889 0.919 0.969 1.118 0.946 1.094

(a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line. Do you think the linear regression is appropriate? (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively. 8.3.7.

Show that n n n 2 2 2 yi − y = yi − yˆ i + yˆ i − y . i=1

i=1

i=1

8.4 PREDICTING A PARTICULAR VALUE OF Y In the earlier sections, we have seen how to ﬁt a least-squares line for a given set of data. Also using this line, we could ﬁnd E(Y ), for any given value of x. Instead of obtaining this mean value, we may be interested in predicting the particular value of Y for a given x. In fact, one of the primary uses of the estimated regression line is to predict the response value of Y for a given value of x. Prediction problems are very important in several real-world problems; for example, in economics one may be interested in a particular gain associated with an investment. Let Yˆ 0 denote a predictor of a particular value of Y = Y0 and let the corresponding values of x be x0 . We shall choose Yˆ 0 to be E(Yˆ |x0 ). Let Yˆ denote a predictor of a particular value of Y . Then the error η of the predictor in comparison to a particular value of Y is η = Y − Yˆ 0 .

438 CHAPTER 8 Linear Regression Models

Both Y and Yˆ are normal random variables, and the error is a linear function of Y and Yˆ . This means that η itself is normally distributed. Also, because E(Yˆ ) = E(Y ), we have E(η) = E(Y |x0 ) − E Yˆ = 0.

Furthermore, Var(η) = Var Y − Yˆ = Var(Y ) + Var Yˆ − 2Cov Y, Yˆ .

We can consider Y and Yˆ as independent, because we are predicting a different value of Y , not used in the calculation of Yˆ . Therefore, Cov(Y, Yˆ ) = 0. In that case Var(η) = Var(Y0 ) + Var Yˆ 0 % & 1 (x − x)2 2 2 + =σ +σ n Sxx & % 1 (x − x)2 σ2. = 1+ + n Sxx

Hence, the error of predicting a particular value of Y , given x, is normally distributed with mean zero (x−x)2 1 and variance 1 + n + Sxx σ 2 . That is,

& 1 (x − x)2 σ2 , η ∼ N 0, 1 + + n Sxx

%

and Z=

Y − Yˆ

8 σ

2 1 + 1n + (x−x) Sxx

∼ N(0, 1).

If we substitute the sample standard deviation S for σ, then we can show that Y − Yˆ T = 8 2 S 1 + 1n + (x−x) S xx

follows the t-distribution with [n − (k + 1)] degrees of freedom. Using this fact, we now give a prediction interval for the random variable Y , the response of a given situation. We know that P −tα/2 < T < tα/2 = 1 − α.

8.4 Predicting a Particular Value of Y 439

Substituting for T , we have ⎞

⎛ ⎜ P⎜ ⎝−tα/2

0, there is a positive relation between X and Y (increasing slope); and when ρ < 0, we have a negative relationship (decreasing slope). Thus, the correlation coefﬁcient can be used to measure how well the linear regression model ﬁts the data. Let (X1 , Y1 ), (X2 , Y2 ), . . . , (Xn , Yn ) be a random sample from a bivariate normal distribution. The maximum likelihood estimator of ρ is the sample correlation coefﬁcient deﬁned by ρˆ or r, n

Xi − X Yi − Y

r= +

i=1 n

Xi − X

n 2

i=1

Yi − Y

(8.7)

2

i=1

Sxy . = Sxx Syy

Equivalently, we can rewrite (8.7) by n

n

Xi Yi −

i=1

n i=1

Xi

n

Yi

i=1

r = ;⎡ . < 2 ⎤ ⎡ 2 ⎤ < n n n n 2 0, values of y increase as the values of x increase, and the data set is said to be positively correlated. When r < 0,

442 CHAPTER 8 Linear Regression Models

values of y decrease as the values of x increase, and the data set is said to be negatively correlated. In this book, we use the term correlation only when referring to linear relationships. In actual practice we can use the value of r to decide whether it is appropriate to develop linear regression models in a given situation. As a rule of thumb, if r > 0.30 or r < −0.30, we proceed with developing a linear regression model. However, a much higher or lower value is desirable. For example, if in a given problem where r = 0.77, it conveys to us that approximately 77% of the data we have are linearly related. The probability distribution for r is difﬁcult to obtain. For large samples, this difﬁculty could be overcome by using the fact that the Fisher z-transform, given by z = (1/2) ln[(1 + r)/(1 − r)]

is approximately normally distributed with mean μz = (1/2) ln[(1 + ρ)(1 − ρ)] and variance σz = 1/(n − 3). Thus, for large random samples, we can test hypotheses about ρ using the approximate test statistic: Z=

=

z − μz σz

1+ρ (1/2) ln 1+r 1−r − (1/2) 1−ρ √1 n−3

.

For example, suppose we are interested in testing the hypothesis that the true value of ρ is a speciﬁc number, say, ρ0 , with a certain value of α. We can proceed to make a decision by following the procedure given next.

HYPOTHESIS TEST FOR ρ One-sided test

Two-sided test

H0 : ρ = ρ0

H0 : ρ = ρ0

Ha : ρ > ρ0 or

Ha : ρ = ρ0

Ha : ρ < ρ0 Test statistic:

Z=

(1/2) ln

1+r 1−r

√

1+ρ −(1/2) 1−ρ0

1

0

Test statistic:

Z=

(1/2) ln

n−3

1+r 1−r

√

−(1/2)

1

1+ρ0 1−ρ0

n−3

Rejection region:

Rejection region:

z > za (upper tail region) z < −za (lower tail region)

|z| > za/2

Decision: If Z falls in the rejection region, reject the null hypothesis at conﬁdence level α. Assumption: (X ,Y ) follow the bivariate normal, and this test procedure is approximate.

8.5 Correlation Analysis 443

Example 8.5.1 For the data given in Example 8.2.1, would you say that the variables X and Y are independent? Use α = 0.05.

Solution We test H0 : ρ = 0 vs. Ha : ρ = 0. From Example 8.2.1, we have the following summary: n

xi = 38;

i=1

n

yi = 46;

i=1

n

xi yi = 709

i=1

and n

xi2 = 408;

i=1

n

yi2 = 1302; n = 10.

i=1

Hence, n

n

Xi Yi −

i=1

n i=1

Xi

n

Yi

i=1

r = ;⎡ < 2 ⎤ ⎡ 2 ⎤ < n n n n 1.96. Because the observed value of the test statistic falls in the rejection region, we reject the null hypothesis and conclude that at α = 0.05, the variables X and Y are dependent.

444 CHAPTER 8 Linear Regression Models

EXERCISES 8.5 8.5.1.

The table shows the midterm and ﬁnal examination test scores for 10 students from a differential equations class, where x denotes the midterm scores and y denotes the ﬁnal scores. x y

68 74

87 89

75 80

91 93

82 88

77 79

86 97

82 95

75 89

79 92

(a) At 95% conﬁdence level, test whether X and Y are independent. (b) Find the p-value. (c) State any assumptions you have made in solving the problem. 8.5.2.

The following table gives the annual incomes (in thousands of dollars) and amounts (in thousands of dollars) of life insurance policies for eight persons. Annual income 42 58 27 36 70 24 53 37 Life insurance 150 175 25 75 250 50 250 100 (a) At the 98% conﬁdence level, test whether annual income and the amount of life insurance policies are independent. (b) Find the attained signiﬁcance level. (c) State any assumptions you have made in solving the problem.

8.5.3.

Show that n

n

n

Xi Yi −

i=1

i=1

Xi

n

Yi

i=1

r = ;⎡ < 2 ⎤ ⎡ 2 ⎤ < n n n n 1) independent variables are used to predict the dependent variable. The model to be studied is of the form Y = β0 + β1 x1 + β1 x2 + · · · + βk xk + ε.

Here, ε ∼ N 0, σ 2 . This model is called a multiple regression model. Let y1 , y2 , . . . , yn be n independent observations on Y . Then each observation yi can be written as yi = β0 + β1 xi1 + β2 xi2 + · · · + βk xik + ε

where xij is the jth independent variable for the ith observation, i = 1, 2, . . . , n, and εi s are independent as in the simple linear regression case. It is sometimes advantageous to introduce matrices to study the linear equations. Let x0 = 1. Deﬁne the following matrices: ⎡ ⎢ ⎢ ⎢ ⎢ X=⎢ ⎢ ⎢ ⎣

x0 x0 . . . x0

⎢ ⎢ ⎢ ⎢ Y =⎢ ⎢ ⎢ ⎣

⎤

⎡

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

⎢ ⎢ ⎢ ⎢ ε=⎢ ⎢ ⎢ ⎢ ⎣

and

x1k x2k . . . xnk

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

⎡ ⎢ ⎢ ⎢ ⎢ β=⎢ ⎢ ⎢ ⎣

. . . . . .

⎡

x12 x22 . . . xn2

β0 β1 . . . βk

. . . . . .

⎤

x11 x21 . . . xn1

ε1 ε2 . . . εn

y1 y2 . . . yn

⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

⎤ ⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎥ ⎦

Thus the n equations representing the linear equations can be rewritten in the matrix form as Y = Xβ + ε.

In particular, for the n observations from the simple linear model of the form Y = β0 + β1 x + ε

we can write Y = Xβ + ε,

(8.8)

446 CHAPTER 8 Linear Regression Models

where ⎡ ⎢ ⎢ ⎢ ⎢ Y =⎢ ⎢ ⎢ ⎣

⎤

y1 y2 . . . yn

⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

⎡

1 1 1 1 1 1

⎢ ⎢ ⎢ ⎢ X=⎢ ⎢ ⎢ ⎣

x1 x2 . . . xn

⎤

⎡

⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

We can see that

⎢ ⎢ ⎢ ⎢ ε=⎢ ⎢ ⎢ ⎣

⎡ % X X =

1 x1

1 x2

. .

. .

. .

ε1 ε2 . . . εn

⎢ &⎢ ⎢ 1 ⎢ ⎢ xn ⎢ ⎢ ⎣

1 1 . . . 1

x1 x2 . . . xn

⎤ ⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

%

and

& β0 β= . β1

⎤

⎡ ⎥ ⎥ n ⎥ ⎢ ⎥ ⎢ ⎥=⎢ n ⎥ ⎣ ⎥ xi ⎦ i=1

n i=1 n i=1

⎤ xi

⎥ ⎥ ⎥, ⎦ 2 x1

where denotes the transpose of a matrix. Also, ⎡

n

⎤ yi

⎢ i=1 ⎢ X Y = ⎢ n ⎣ xi yi

⎥ ⎥ ⎥. ⎦

i=1

Let us now go back to the multiple regression model Y = β0 + β1 x1 + β1 x2 + · · · + βk xk + ε.

The least-squares estimators βˆ i of βi for i = 0, 1, 2, . . . , k are the ones that minimize the sum of squares SSE =

n

e2i =

i=1

n

2 yi − βˆ 0 + βˆ 1 x1 + βˆ 2 x2 + · · · + βˆ k xk

i=1

y − Xβˆ ˆ ˆ = y y − y Xβˆ − Xβˆ y + βX Xβ.

= y − Xβˆ

To minimize SSE with respect to β, we differentiate SSE with respect to β and equate it to zero. Thus, ∂ y y − y X β − β X y + X β Xβ = 0 ∂β

yielding (X X)βˆ = X Y.

8.6 Matrix Notation for Linear Regression 447

Assuming the matrix (X X) is invertible, we obtain βˆ = (X X−1 )X Y.

Now we summarize the procedure to obtain a multiple linear regression equation. PROCEDURE TO OBTAIN A MULTIPLE LINEAR REGRESSION EQUATION 1. Rewrite the n observations Yi = β0 + β1 x1i + β1 x2i + · · · + βk xki , i = 1, 2, . . . , n in the matrix notation as Y = Xβ + ε where X , Y , and β are deﬁned in (1). 2. Compute (X X )−1 and obtain the estimators of β as βˆ = (X X )−1 X Y . 3. Then the regression equation is ˆ Yˆ = X β.

Example 8.6.1 Using the data given in Example 8.2.1, use the matrix approach to solve the problem of operations.

Solution From the data of Example 8.2.1 we have ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ Y =⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣

−9 −7 −5 −4 2 6 9 13 21 20

⎤

⎡

⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ and X = ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎢ ⎥ ⎥ ⎢ ⎣ ⎦

1 1 1 1 1 1 1 1 1 1

−3 −2 −1 0 2 5 6 8 11 12

⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

Thus, we can write % X X =

10 38

& % & % 38 46 0.1548 X Y = (X X)−1 = 408 709 −0.0144

& −0.0144 . 0.0038

448 CHAPTER 8 Linear Regression Models

Hence, %

0.1548 −0.0144 % & % & −3.1009 βˆ = = 0 . βˆ 1 2.0266

βˆ = (X X)−1 (X Y )

−0.0144 0.0038

&%

46 709

&

Thus, the least-squares line is given by yˆ = −3.1009 + 2.0266X, which is identical to the regression line we obtained in Example 8.2.1.

Example 8.6.2 The following data relate to the prices (Y ) of ﬁve randomly chosen houses in a certain neighborhood, the corresponding ages of the houses (x1 ), and square footage (x2 ). Price y in thousands Age x1 in Square footage x2 in thousands of dollars years of square feet 100 1 1 80 5 1 104 5 2 94 10 2 130 20 3 Fit a multiple linear regression model Y = β0 + β1 x1 + β2 x2 + ε to the foregoing data.

Solution We have ⎡

⎤ ⎡ 100 1 1 ⎢ 80 ⎥ ⎢1 5 ⎢ ⎥ ⎢ ⎢ ⎥ ⎢ Y = ⎢ 104 ⎥ ; X = ⎢1 5 ⎢ ⎥ ⎢ ⎣ 94 ⎦ ⎣1 0 130 1 20 ⎡ ⎤ 508 ⎢ ⎥ X Y = ⎣4560⎦ 966

⎤ 1 ⎡ 1⎥ 5 ⎥ ⎥ ⎢ 2⎥ ; X X = ⎣41 ⎥ 2⎦ 9 3

41 551 96

⎤ 9 ⎥ 96⎦ ; 19

8.6 Matrix Notation for Linear Regression 449

and

⎡

2.3076 ⎢ (X X)−1 = ⎣ 0.1565 −1.8840 Hence,

0.1565 0.0258 −0.2044

⎤ −1.8840 ⎥ −0.2044⎦ . 1.9779

⎡

⎤ 66.1252 ⎢ ⎥ (X X)−1 (X Y ) = ⎣−0.3794⎦ . 21.4365

Thus, the regression model is y = 66.12 − 0.3794x1 + 21.4365x2 .

8.6.1 ANOVA for Multiple Regression As in Section 8.3, we can obtain an ANOVA table for multilinear regression (with k independent or explanatory variables) to test the hypothesis H0 : β1 = β2 = · · · = βk = 0

versus Ha : At least one of the parameters βj = 0, j = 1, . . . , k.

The calculations for multiple regression are almost identical to those for simple linear regression, except that the test statistic (MSR)/(MSE) has an F (k, n − k − 1) distribution. Note that the F -test does not indicate which of the parameters βj = 0, except to say that at least one of them is not zero. The ANOVA table for multiple regression is given by Table 8.4.

Table 8.4 ANOVA Table for Multiple Regression Source of variation

Degrees of freedom

Sum of squares

Regression (Model)

k

SSR

Error (Residuals)

n−k−1

SSE

n−1

SST

Total

Mean sum of squares MSR =

SSR d.f.

SSE d.f.

F-ratio MSR MSE

450 CHAPTER 8 Linear Regression Models

Example 8.6.3 For the data of Example 8.6.2, obtain an ANOVA table and test the hypothesis H0 : β1 = β2 = 0 vs. Ha : at least one of the βi = 0, i = 1, 2. Use α = 0.05.

Solution We test H0 : β1 = β2 = 0 vs. Ha : At least one of the βi = 0, i = 1, 2. Here n = 5, k = 2. Using Minitab, we obtain the ANOVA table (Table 8.5). Based on the p-value, we cannot reject the null hypothesis at α = 0.05.

Table 8.5 Source of Degrees of Sum of Mean sum of F-ratio p-Value variation freedom squares squares Regression (Model)

2

956.5

478.2

Error (Residuals)

2

382.7

191.4

Total

4

1339.2

2.50

0.286

EXERCISES 8.6 8.6.1.

Given the data X1 3 2 3 1

(a) (b) (c) (d) 8.6.2.

X2 1 5 3 2

y 4 3 6 5

Write the multiple regression model in matrix form. Find X X, (X X)−1 , and X y. Estimate β. Estimate the error variance.

A study is conducted to estimate the demand for housing (y) based on current interest rate X1 and the rate of unemployment. The data in Table 8.6.1 are obtained. (a) Fit the multiple regression model y = β0 + β1 x1 + β1 x2 + ε.

8.7 Regression Diagnostics 451

Table 8.6.1 Units sold

Interest rate (%)

Unemployment rate (%)

65

9.0

10.0

59

9.3

8.0

80

8.9

8.2

90

9.1

7.7

100

9.0

7.1

105

8.7

7.2

(b) Test whether the model is signiﬁcant. 8.6.3.

The following data give the annual incomes (in thousands of dollars) and amounts (in thousands of dollars) of life insurance policies for eight persons. Annual income 42 58 27 36 70 24 53 37 Life insurance 150 175 25 75 250 50 250 100 Calculate the least-squares regression line for these data using matrix operations.

8.6.4.

The following is a random sample of height (in inches) and weight (in pounds) of seven basketball players. Height Weight

73 186

83 234

77 208

80 237

85 265

71 190

80 220

Calculate the least-squares regression line for these data using matrix operations.

8.7 REGRESSION DIAGNOSTICS In the previous sections, we derived least-squares estimators for the parameters in the linear regression model. These estimators are useful as long as we can determine (1) how well the model ﬁts the data and (2) how good our estimates are in providing possible relationships between variables of interest. Some of these problems are discussed in Chapter 14 in a uniﬁed manner. We now brieﬂy discuss some aspects of the adequacy of the simple linear regression model. In multiple regression, in addition to the problems discussed here, there are other problems, such as collinearity and model speciﬁcation (inclusion of all relevant variables, as well as exclusion of irrelevant variables), that need to be examined. They are beyond the level of this text. Many graphical methods and numerical tests dealing with these problems are available in the literature and are often called regression diagnostics. Most of the major statistical software packages incorporate these tests, making it easier to perform regression diagnostics so as to detect potential problems. We have seen that the (ordinary) least-squares regression model must meet the following assumptions.

452 CHAPTER 8 Linear Regression Models

1. Linearity. The existence of a linear relationship between x and y is the basis of the simple linear regression model. A simple method to test for linearity is to draw a scatterplot of data points. As we explained in Section 8.2, we could also plot residual ei versus xi or Yˆ i . A symmetric trend in the plot of the residuals versus the explanatory variable or the ﬁtted values indicates there is a problem with the obtained regression model. For a correct model, the residuals should center around zero across the explanatory variables and the ﬁtted values. The degree of linear relationship can be ascertained by the correlation coefﬁcient, r, given in Section 8.5 or by using the value of the coefﬁcient of determination r 2 , explained in Project 8B. Most statistical software packages give the value of r 2 (refer to outputs given in Section 8.9). The closer the value of r 2 is to 1, the better the least-squares equation yˆ = βˆ 1 x + βˆ 0 performs as a predictor of y. 2. Homoscedasticity (homogeneity of variance). This assumption says that the variance of the error term remains constant across all values of x. In this case we know by the Gauss–Markov theorem that the least-squares estimators βˆ 0 and βˆ 1 are the best linear unbiased estimators of β0 and β1 . A frequently used graphical method is to draw the residuals versus a ﬁtted plot. This can be easily done using statistical software packages. The graph of residuals ei versus ﬁtted values Yˆ i or explanatory variable xi indicates a change in the spread of residuals as Yˆ or x changes. It may look like Figure 8.7. If the variances of yi values are not constant, the inferences we made, such as conﬁdence intervals on means, prediction, and so forth, are off. The severity of this discrepancy depends on the degree of the assumption violation. If we see that the pattern of data points only changes slightly, that will indicate a mild heteroscedasticity. Two numerical tests for heteroscedasticity are explained in Section 14.4.3.

Residuals versus the fitted values (response is C2)

20

Residual

10

0 ⫺10 ⫺20 0

10

20

30

40 50 Fitted value

■ FIGURE 8.7 Scatterplot of fitted values versus residuals.

60

70

80

90

8.7 Regression Diagnostics 453

3. Independence of εi and εj , for i = j. This assumption speciﬁes that the errors associated with one observation should not be correlated with the errors of any other observation. In general, whether the two samples are independent of each other is decided by the structure of the experiment from which they arise. Violation of the independence assumption can occur in a variety of situations. For example, if we take a survey on a certain issue on children’s education from one particular school, these observations may reﬂect some pattern, thus violating the independence assumption. If data are collected on the same variable over time, then the assumption of independence will be violated. Project 12B explains a run test for check of this assumption. Also, see Section 14.4.4. 4. Normality of the errors. This assumption speciﬁes that the distribution of the εi values should be normal. This assumption is crucial when sample size is small if the p-value for the test is to be valid. For large samples, by the Central Limit Theorem this assumption becomes less important unless the prediction of a single value of y is involved. Thus a test of normality is necessary mainly when the t-test is used. Section 14.4.1 explains some of the tests for normality. A simple way is to draw a probability plot for the errors to conform to the assumption of normality. If we observe nonnormality, one of the ways to overcome the problem is to use data transformation such as logarithmic transformation, as explained in Section 14.4.2, and perform the regression analysis on the transformed data. Sometimes nonparametric methods may be more appropriate, but we will not deal with this topic in this book. Another important issue is the existence of inﬂuential observations, individual observations that have a strong inﬂuence on estimated coefﬁcients. If a single observation substantially changes our results, we need to do further investigation. The ordinary least-squares method is quite sensitive for outlying observations, both for independent variables and for dependent variables, and can have an adverse effect on the estimate. In higher dimensional data, these outlying observations can remain unnoticed. This aspect in one explanatory variable case is discussed in Project 8C. One of the simple ways to identify such observations is to draw a scatterplot. In the scatterplot, if we see a data point that is farther away from the rest of the data points, that is an indication of possible inﬂuential points. The natural question is, if we ﬁnd that the data violate one or more of the assumptions, what can we do about it? We have already explained that violation of the normality assumption in large samples is not an issue unless prediction is involved, because prediction depends on normality of an individual observation. Thus, if the inferences are based on the t- or F -tests or prediction is involved, we may be able to transform Y to Y to achieve normality. If we have predicted Y , then back-transform to predict Y . If we observe nonlinearity of data, we may be able to transform x to x = h(x) such that Y is linear in x , or consider a polynomial model in x, in which case the ideas of multiple linear regression may be utilized. Robust estimates of variances of β0 and β1 or the method of weighted least squares may be used to deal with the case of nonconstant variance. Often careful experimental design could be done to remove possible correlation in errors. There are also robust methods available for correlation analysis. We refer to specialized books on regression methods for further details on these issues. If we detect inﬂuential observations, there are statistical techniques available, such as least trimmed squares estimators, to deal with outlying observations.

454 CHAPTER 8 Linear Regression Models

8.8 CHAPTER SUMMARY In this chapter, we ﬁrst derived the least-squares line and its properties. Then we learned about the conﬁdence intervals for the coefﬁcients in the regression model and did hypothesis tests on the values of the coefﬁcients. We introduced the matrix notation for linear regression as well as for multiple regression. We discussed how to predict a particular value of Y for a given value of X. In order to study the dependence of X and Y , we presented correlation analysis. The following are some of the key deﬁnitions we have used in this chapter. ■

Predictors

■

Response variable

■

Regression analysis

■

Multiple linear regression model

■

Simple linear regression model

■

Sum of squares for errors (SSE)

■

Sum of squares of the residuals

■

Least-squares line

■

Least-squares equations

■

Normal equations

■

Best linear unbiased estimator (BLUE)

■

Correlation analysis

The following important concepts and procedures were discussed in this chapter: ■

Procedure for regression modeling

■

Procedure for ﬁtting a least-squares line

■

Properties of the least-squares estimators for the model Y = β0 + β1 x + ε

■

The Gauss–Markov theorem

■

Procedure for obtaining conﬁdence intervals of β0 and β1

■

Procedure to obtain a multiple linear regression equation

■

Prediction interval for the response variable Y

■

Hypothesis testing for correlation, ρ

■

Linearity

■

Homoscedasticity

■

Independence of εi and εj , for i = j

■

Normality of the errors

■

Inﬂuential observations

8.9 Computer Examples 455

8.9 COMPUTER EXAMPLES 8.9.1 Minitab Examples Example 8.9.1 For the data in Example 8.2.1, use the method of least squares to ﬁt a straight line to the accompanying data points. Give the estimates of β0 and β1 . Plot the points and sketch the ﬁtted least-squares line.

Solution Enter independent variable, x, in C1 and the response variable, y, in C2. Then: Stat > Regression > Regression. . . > in Response: type C2, and in Predictors: type C1 > click OK We obtain the following output.

Regression Analysis The regression equation is

C2 = –3.10 + 2.03 C1 Predictor Constant C1

Coef –3.1009 2.02656

StDev 0.3888 0.06087

S = 0.9883 R-Sq = 99.3% Analysis of Variance Source DF SS Regression 1 1082.6 Residual Error 8 7.8 Total 9 1090.4 Unusual Observations Obs C1 C2 8 11.0 21.000 Residual St Resid 1.809 2.18R

T –7.98 33.29

P 0.000 0.000

R-Sq(adj) = 99.2% MS 1082.6 1.0

Fit 19.191

F 1108.34

P 0.000

StDev Fit 0.538

R denotes an observation with a large standardized residual

From this the estimate of β0 is −3.1009, and the estimate of β1 is 2.02656. Hence, the regression line is yˆ = −3.1009 + 2.02656x. Now to obtain the ﬁtted regression line, use the following procedure: Stat > Regression > Fitted Line Plot. . . > in Response(Y): type C2, and in Predictors(X): type C1 > click Linear OK

456 CHAPTER 8 Linear Regression Models

We obtain the following graph. Regression plot Y ⫽ ⫺3.1009 ⫹ 2.02656x R⫺Sq ⫽ 99.3%

20

C2

10

0

⫺20

0

5 C1

10

If in addition, we need, say, 95% conﬁdence and predictor bands, then use Stat > Regression > Fitted Line Plot. . . > in Response(Y): type C2, and in Predictor(X): type C1 > click Linear > click options. . . > click Display confidence bands and Display predictor bands > in Title: type a title for the graph and OK > OK

We obtain the following graph. Regression line with 95% confidence and predictor bands Y ⫽ ⫺3.1009 ⫹ 2.02656 x R⫺Sq ⫽ 99.3%

20

C2

10

0 Regression 95% C 95% R

⫺20

0

5 C1

10

8.9 Computer Examples 457

8.9.2 SPSS Examples A detailed explanation of regression methods including diagnostics using SPSS can be obtained at the site: http://www.ats.ucla.edu/stat/spss/webbooks/reg/. We will just demonstrate a simple case with an example.

Example 8.9.2 The following is a random sample of height (in inches) and weight (in pounds) of seven basketball players. Height

73

83

77

80

85

71

80

Weight

186

234

208

237

265

190

220

Calculate the least-squares regression line for these data using SPSS.

Solution Enter height in column 1 and weight in column 2. Then Analyze > Regression > Linear. . . > move var00002 to dependent:, and var00001 to Independent(s): > click OK

We obtain the following output: Regression: Variables Entered/Removed Model 1

Variables Entered VAR00001

Variables Removed .

Method Enter

a All requested variables entered. b Dependent Variable: VAR00002 Model Summary: Model R R Square Adjusted R Square Std. Error of the Estimate 1 .947 .897 .876 9.86006 a Predictors: (Constant), VAR00001 ANOVA: Model Sum of Squares df Mean Square F Sig. 1 Regression 4223.896 1 4223.896 43.446 .001 Residual 486.104 5 97.221 Total 4710.000 6 a Predictors: (Constant), VAR00001 b Dependent Variable: VAR00002

458 CHAPTER 8 Linear Regression Models

Coefficients: Unstandardized Coefficients Model B Std. Error 1 (Constant) −188.476 62.083 VAR00001 5.208 .790

Standardized Coefficients Beta .947

t

Sig.

−3.036 .029 6.591 .001

a Dependent Variable: VAR00002 Looking at the coefficients, we see that βˆ 0 = −188.476 and βˆ 1 = 5.208. Hence, the regression line is given by yˆ = −188.476 + 5.208x. Because the coefficient of determination r 2 is 0.897, and the p-value is small, the model fit looks pretty good.

8.9.3 SAS Examples For regression analysis, we can use the SAS procs called GLM, which stands for General Linear Model, and REG, which stands for regression. In the following example we will give a simpliﬁed version of the foregoing procedure. A good explanation of regression methods including diagnostics using SAS can be obtained at http://www.ats.ucla.edu/stat/sas/webbooks/reg/.

Example 8.9.3 Using the SAS commands, redo Example 8.9.1.

Solution We can use the following commands. options nodate nonumber; data exreg; INPUT x y @@; datalines; –1 –5 0 –4 2 2 –2 –7 5 6 6 9 8 13 11 21 12 20 –3 –9 ; proc reg data=exreg; title ‘Regression of Y on X’; model y=x / p clm; run;

8.9 Computer Examples 459

We obtain the following output.

Regression of Y on X The REG Procedure Model: MODEL1 Dependent Variable: y Analysis of Variance

Source

DF

Sum of Squares

Mean Square

Model 1 1082.58589 Error 8 7.81411 Corrected Total 9 1090.40000

Root MSE Dependent Mean Coeff Var

F Value

1082.58589 0.97676

1108.34

0.98831 R-Square 4.60000 Adj R-Sq 21.48508

Pr > F

1, otherwise a2 < 0; both are not admissible because a is a fraction. Hence, a=

σ1 σ1 + σ2

and

1−a=

σ2 . σ1 + σ2

Using the second derivative test, we can verify that this indeed is a minimum for var(X1 − X2 ). From this analysis we can see that the sample sizes that maximize the information in the data relevant to the parameter μ1 − μ2 subject to the constraint n1 + n2 = n are n1 =

σ1 n σ1 + σ2

and

n2 =

σ2 n. σ1 + σ2

9.5 The Taguchi Methods 489

As a special case, we can see that when σ12 = σ22 , the optimal design is to take n1 = n2 .

EXERCISES 9.4 9.4.1.

A total of 100 sample points were taken from two populations with variances σ12 = 4 and σ22 = 9. Find n1 and n2 that will result in the maximum amount of information about (μ1 − μ2 ).

9.4.2.

Suppose in Exercise 9.4.1 we want to take n = n1 = n2 . How large should n be to obtain the same information as that implied by the solution of Exercise 9.4.1?

9.5 THE TAGUCHI METHODS Taguchi methods were developed by Genichi Taguchi to improve the implementation of total quality control in Japan. These methods are claimed to have provided as much as 80% of Japanese quality gains. They are based on the design of experiments to provide near-optimal quality characteristics for a speciﬁc objective. A special feature of Taguchi methods is that they integrate the methods of statistical design of experiments into a powerful engineering process. The Taguchi methods are in general simpler to implement. Taguchi methods are often applied on the Japanese manufacturing ﬂoor by technicians to improve their processes and their product. The goal is not just to optimize an arbitrary objective function, but also to reduce the sensitivity of engineering designs to uncontrollable factors or noise. The objective function used is the signal-to-noise ratio, which is then maximized. This moves design targets toward the middle of the design space so that external variation affects the behavior of the design as little as possible. This permits large reductions in both part and assembly tolerances, which are major drivers of manufacturing cost. Linking quality characteristics to cost through the Taguchi loss function (Taguchi and Yokoyama, 1994) was a major advance in quality engineering, as well as in the ability to design for cost. Taguchi methods are also called robust design. In 1982, the American Supplier Institute introduced Dr. Taguchi and his methods to the U.S. market. Using a well-planned experimental design, such as a fractional factorial design, it is possible to efﬁciently obtain information about the model and the underlying process. Clearly, the purpose of these methods is to control and ensure the quality of the end product. In the conventional approach, this is achieved by further testing a few end products that are randomly chosen or using control charts and making decisions based on certain preset criteria, such as acceptable or unacceptable. Thus, “quality” of the product is thought of as inside or outside of speciﬁcations. Instead, Taguchi suggested that we should specify a target value, and the quality should be thought of as the variation from the target. As an example, suppose we make n observations of the output x1 , . . . , xn of a process at times 1, 2, . . . , n, as shown in Figure 9.3. The control chart consists of a plot of observed output values (xi ’s) on the y-axis and the times of observation, 1, 2, . . . , n on the x-axis, as shown in the ﬁgure. The letter T represents the target value. If

490 CHAPTER 9 Design of Experiments

. TU x

.

T ⫽ Target value

.

1

TL

n

2 Time

■ FIGURE 9.3 Control plot of processing times and outputs.

L

L TL

T

x1

x2

TU

x3

■ FIGURE 9.4 Loss function.

the output value is between TL and TU , the process is deemed to be operating satisfactorily; otherwise the process is said to be out of control and the output value is considered unsatisfactory. Some other examples are (1) deﬁning speciﬁcation limits for acceptance, such as stating that the diameter of bolts must be between 9.8 mm and 10.2 mm with mean 10 mm, and (2) that the waiting time in a line should be less than 30 minutes for at least 90% of customers. In all these situations, the speciﬁcations partition the state of the process as acceptable or unacceptable, that is, it classiﬁes the state as a dichotomy. This is often called the “goal post mentality.” The basic idea of the Taguchi approach is a shift in mindset from demarking the quality as acceptable or unacceptable to a more ﬂexible and realistic classiﬁcation. The traditional approach to quality control does not take into account the size of departure from the target value. To accommodate the size of such departure as a signiﬁcant factor in quality control, let us introduce the concept of loss function (see Chapter 11). If an output value x differs from the target value T , let L(T, x) denote the loss incurred, say in dollars. Other possible losses could also be reputation or customer satisfaction. For the control chart example, we can assign the loss function L(T, x) =

0, L,

if TU < x < TL if x > TL or x < TU

where L is a constant and x is the measured value. This is schematically shown in Figure 9.4. From Figure 9.4, it is seen that we view outputs x1 and x2 as having equal quality, whereas x2 and x3 are considered to have vastly differing quality (x2 is acceptable and x3 is not acceptable). A more

9.5 The Taguchi Methods 491

L(T, x )

L TL

x

T

TU

■ FIGURE 9.5 Quadratic loss function.

reasonable conclusion would be that x1 has excellent quality, whereas x2 and x3 are similar, both being poor. In Taguchi’s approach, the loss function takes into account the size of departure from the target value. For example, a popular choice for the loss function is L(T, X) = k(X − T )2 ,

where L = loss incurred, k = constant, X = actual value of the measured output, and T = target value.

We can schematically represent the behavior as shown by Figure 9.5. This form of loss function is called the quadratic loss function. The choice of k depends on the particular problem. For example, the scaling factor k can be used to convert loss into monetary units to accommodate comparisons of systems with different capital loss. Or, in product manufacturing, let D denote the allowed deviation from the target, and let A denote the loss due to a defective product. Then a choice of k can be k = (A/D)2 . As shown earlier, the average loss is E(L) and is given by E(L) = k[(E(X) − T )2 + σ 2 ] = k[(bias)2 + variance]

where σ 2 is the variance of X (measured quality, which is assumed to be random). In Taguchi, the variation from the target can be broken into components containing bias and product variation. Thus, if our aim is to minimize the expected loss, E(L), we should not only require E(X) = μ to be close to T but also should reduce the variance. It turns out that often these requirements are contradictory. The objective is to choose the design parameters (the factors that inﬂuence the quality) optimally to obtain the best quality product. In practice, the parameters μ and σ 2 are not known and are being estimated by X and S 2, respectively. This results in the Taguchi loss function L = k[(X − T )2 + S 2 ].

This loss function penalizes small deviations from T only slightly, while assessing a larger penalty for responses far from the target. The expected loss is similar to a mean squared error loss, which we have seen in regression analysis in the form of least squares.

492 CHAPTER 9 Design of Experiments

Why is controlling both bias and variance important? Suppose you want your community swimming pool temperature at 80◦ F, which is the T here. Suppose the temperature varies between 60◦ F and 100◦ F. Clearly the average (bias) is zero; however, it will be pretty uncomfortable to swim at 60◦ F or 100◦ F. Here the bias takes the ideal value of zero, but the variance is large. In another scenario, the variance may be small, but the average temperature may be farther away from the target value of 80◦ F (for example, the temperature is constant at 60◦ F). Hence, we want the pool temperature to be near to the target value of 80◦ F, with as small variance as possible (say, within 1◦ F to 2◦ F). Taguchi coined the term design parameters as the generic description for factors that may inﬂuence the quality and whose levels we want to optimize. Taguchi’s philosophy is to “design quality in” rather than to weed out the defective items after manufacturing. In order to obtain an optimal set of design parameters that affect the quality of the end product, the Taguchi method utilizes appropriately designed experiments. More speciﬁcally, orthogonal arrays are used for fractional factorial designs. Taguchi provides tables for these designs so that even a nonspecialist can use them. For two-level designs (high, low), we have a table for an L4 orthogonal array up to three factors; a table for an L8 orthogonal array up to seven factors; and so forth. Similar tables are available for three-level designs. We will not describe these design issues in this section. We refer the reader to specialized books on the subject for further details. We can summarize the Taguchi approach to quality design as follows: 1. Taguchi’s methods for experimental design are ready made and simple to use in the design of efﬁcient experiments, even by nonexperts. 2. Taguchi’s approach to total quality management is holistic and tries to design quality into a product rather than inspecting defects in the ﬁnal product. 3. Taguchi’s techniques can readily be applied to other ﬁelds such as management problems.

EXERCISES 9.5 9.5.1.

Suppose the following data represent thickness between and within silicon wafers (in microns), with a target value of 14.5 microns. 13.688 13.925

13.788 14.545

14.173 13.797

14.557 14.778

Compute the Taguchi loss function. 9.5.2.

One of the commonly used performance measures in the Taguchi method is log

(mean)2 s2

,

where s2 is the sample variance. In general, the higher the performance measure, the better the design. This measure is called robustness statistics. For the problem of Exercise 9.5.1, suppose that we run the experiment by controlling various factors affecting the thickness. Table 9.5.1 shows the data obtained in four different runs.

9.6 Chapter Summary 493

Table 9.5.1 Run 1: 14.158 14.754 14.412 14.065 13.802 14.424 14.898 14.187 Run 2: 13.676 14.177 14.201 14.557 13.827 14.514 13.897 14.278 Run 3: 13.868 13.898 14.773 13.597 13.628 14.655 14.597 14.978 Run 4: 13.668 13.788 14.173 14.557 13.925 14.545 13.797 14.778

(a) Using the robustness statistics given earlier, which of the processes gives us an improved performance? (b) Another commonly used performance statistic is − log(s2 ).

Using this robustness statistic, which of the processes gives us an improved performance? Compare this with the results of part (a).

9.6 CHAPTER SUMMARY In this chapter, we have learned some basic aspects of experimental design. Some fundamental deﬁnitions and tools for developing experimental designs such as randomization, replication, and blocking were introduced in Section 9.2. Basic concepts of factorial design were given in Section 9.3. In Section 9.4, we saw an example of optimal design. The Taguchi method was introduced in Section 9.5. In the next chapter, we introduce the analysis component. We have discussed only a very small collection of experimental designs in this chapter. There exist a wide variety of experimental designs to deal with a large number of treatments and to suit speciﬁc needs of research experiments in diverse ﬁelds. It is an exciting and growing area for the interested student to apply and explore. We list some of the key deﬁnitions introduced in this chapter: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Response variable (output variable) Independent variables (treatment variables or input variables or factors) Nuisance variables Noise Observational Experimental units Single-factor experiments Multifactor experiments Experimental error Blinding, double-blinding, and placebo Replication Block Randomization Completely randomized design Randomized complete block design k × k Latin square design

494 CHAPTER 9 Design of Experiments

■ ■

Greco-Latin square design parameters

In this chapter, we have also learned the following important concepts and procedures. ■ ■ ■ ■ ■ ■ ■ ■ ■

Procedure for random assignment Procedure for randomization in a randomized complete block design Procedure for a randomized complete block design with r replications Procedure for constructing a 4 × 4 Latin square One-factor-at-a-time design Full factorial design Fractional factorial design Choice of optimal sample size The Taguchi methods

9.7 COMPUTER EXAMPLES In this chapter, we present Minitab and SAS commands only. SPSS commands can be performed similarly to Minitab.

9.7.1 Minitab Examples Example 9.7.1 Obtain a random permutation of numbers 1 to n.

Solution Enter in C1 the numbers 1 to n, say n = 10. Then Calc > random data > samples from column. . . > enter sample 10 > rows from column(s) C1 > Store samples in: C2 > OK The result is a random permutation of numbers 1 to n(= 10). One such permutation is given by 8 5 9 7 10 6 4 3 2 1 Now if we need to generate blocks of random permutations of numbers 1 to n(= 10), in the foregoing steps, just store samples in C3, C4, . . . .

9.7.2 SAS Examples Example 9.7.2 For the data of Example 9.2.4, conduct a randomized complete block design using SAS.

9.7 Computer Examples 495

Solution We represent blocks that are reasons for pain by H = 1, M = 2, and CB = 3. Similarly five brands which are treatments by A = 1, B = 2, C = 3, D = 4, and E = 5. Then we can use the following code to generate a randomized complete block design.

options nodate nonumber; data a; do block = 1 to 3 ; do subject = 1 to 5; x = ranuni(0); output; end; end ; proc sort; by block x; data c; set a; trt = 1 + mod(N − 1, 5); /* mod = remainder of N/5 */ proc sort; by block subject; proc print; var block subject trt; run;

We get the following output. Completely randomized 2 × 3 design, 4 subjects per cell Obs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

block 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3

subject 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

trt 5 4 3 2 1 2 5 3 4 1 4 5 1 2 3

496 CHAPTER 9 Design of Experiments

Note that the numbers in the column corresponding to a block identify the type of pain, the numbers in the subject column correspond to the subjects, and the numbers in the column corresponding to trt identify the brands. Using the corresponding letters, we can rewrite the foregoing table in the familiar form shown in Table 9.14.

Table 9.14 H

M

CB

1(E)

1(B)

1(D)

2(D)

2(E)

2(E)

3(C)

3(C)

3(A)

4(B)

4(D)

4(B)

5(A)

4(A)

5(C)

The PLAN procedure constructs experimental designs. The PLAN procedure does not have a DATA= option in the PROC statement; in this procedure, both the input and output data sets are speciﬁed in the OUTPUT statement. We will use this to construct a Latin square design.

Example 9.7.3 A gasoline company is interested in comparing the effect of four gasoline additives (A, B, C, D) on the gas mileage achieved per gallon. Four cars (1, 2, 3, 4) and four drivers (I, II, III, IV) will be used in the experiment. Create a Latin square design.

Solution We can use the following program, where we represent the additives by 1 = A, 2 = B, 3 = C, and 4 = D. Options nodate nonumber; title ’Latin Square design for 4 additives’; proc plan seed=37432; factors rows=4 ordered cols=4 ordered/NOPRINT; treatments tmts=4 cyclic; output out=g rows cvals=(’car 1’ ’car 2’ ’car 3’ ’car 4’) random cols cvals=(’Driver 1’ ’Driver 2’ ’Driver 3’ ’Driver 4’) random tmts nvals=(1 2 3 4) random; run; proc tabulate; class rows cols;

Projects for Chapter 9 497

var tmts; table rows, cols*(tmts*f=1.); keylabel sum=’ ’; run;

PROJECTS FOR CHAPTER 9 9A. Sample Size and Power Suppose that the experimenter is interested in comparing the true means of two independent populations. If two similar treatments are to be compared, the assumption of equality of variances is not unreasonable. Hence, assume that the common variance of the two populations is σ 2 , and the experimenter has a prior estimate of the variance. We learned in Section 9.4 that in this case, the optimal design will be to take sample sizes n1 and n2 to be equal. Let n = n1 = n2 be the size of the random sample that the experimenter should take from each population. Now, suppose that the experimenter has decided to use the one-sided large sample test, H0 : μ1 = μ2 vs. Ha : μ1 > μ2 with a ﬁxed α = P(Type I error). He wants to choose n to be so large that if μ1 = μ2 + kσ, he will get a ﬁxed power (1 − β) of deciding μ1 > μ2 . Recall that power of a test is the probability of (correctly) rejecting H0 when H0 is false. Find the approximate value of n. Note that, for a given α, this will be an optimal sample size with a desired value of the power. In particular, what should be the sample size in the hypothesis testing problem, H0 : μ1 − μ2 = 0 vs. Ha : μ1 − μ2 = 3, if α = β = 0.05. Assume that σ = 7.

9B. Effect of Temperature on Spoilage of Milk Suppose you have observed that milk in your refrigerator spoils very fast. You may be wondering whether it has anything to do with the temperature settings. Design an experiment to study the effect of temperature on spoiled milk, with at least three meaningful settings of the temperature. (i) Write a possible hypothesis for your experiment. (ii) What are the independent and dependent variables? (iii) Which variables are being controlled in this experiment? (iv) Discuss how you used the three basic principles of replication, blocking, and randomization. (v) What conclusions can you make? Think through any possible ﬂaws in the design that may affect the integrity of your ﬁndings.

This page intentionally left blank

Chapter

10

Analysis of Variance Objective: To analyze the means of several populations by identifying sources of variability of the data. 10.1 Introduction 500 10.2 Analysis of Variance Method for Two Treatments (Optional) 501 10.3 Analysis of Variance for Completely Randomized Design 510 10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 10.5 Multiple Comparisons 536 10.6 Chapter Summary 543 10.7 Computer Examples 543 Projects for Chapter 10 554

526

John Wilder Tukey (Source: http://en.wikipedia.org/wiki/John_Tukey)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

499

500 CHAPTER 10 Analysis of Variance

John W. Tukey (1915–2000), a chemist-turned-topologist-turned statistician, was one of the most inﬂuential statisticians of the past 50 years. He is credited with inventing the word software. He worked as a professor at Princeton University and a senior researcher at AT&T’s Bell Laboratories. He made signiﬁcant contributions to the ﬁelds of exploratory data analysis and robust estimation. His works on the spectrum analysis of time series and other aspects of digital signal processing have been widely used in engineering and science. He coined the word bit, which refers to a unit of information processed by a computer. In collaboration with Cooley, in 1965, Tukey introduced the fast Fourier transform (FFT) algorithm that greatly simpliﬁed computation for Fourier series and integrals. Tukey authored or coauthored many books in statistics and wrote more than 500 technical papers. Among Tukey’s most far-reaching contributions was his development of techniques for “robust analysis,” an approach to statistics that guards against wrong answers in situations where a randomly chosen sample of data happens to poorly represent the rest of the data set. Tukey also made signiﬁcant contributions to the analysis of variance.

10.1 INTRODUCTION Suppose that we are interested in the effect of four different types of chemical fertilizers on the yield of rice, measured in pounds per acre. If there is no difference between the different types of fertilizers, then we would expect all the mean yields to be approximately equal. Otherwise, we would expect the mean yields to differ. The different types of fertilizers are called treatments and their effects are the treatment effects. The yield is called the response. Typically we have a model with a response variable that is possibly affected by one or more treatments. The study of these types of models falls under the purview of design of experiments, which we discussed in Chapter 9. In this chapter we concentrate on the analysis aspect of the data obtained from the designed experiments. If the data came from one or two populations, we could use the techniques learned in Chapters 6 and 7. Here, we introduce some tests that are used to analyze the data from more than two populations. These tests are used to deal with treatment effects, including tests that take into account other factors that may affect the response. The hypothesis that the population means are equal is considered equivalent to the hypothesis that there is no difference in treatment effects. The analytical method we will use in such problems is called the analysis of variance (ANOVA). Initial development of this method could be credited to Sir Ronald A. Fisher who introduced this technique for the analysis of agricultural ﬁeld experiments. The “green revolution” in agriculture would have been impossible without the development of theory of experimental design and the methods of analysis of variance. Analysis of variance is one of the most ﬂexible and practical techniques for comparing several means. It is important to observe that analysis of variance is not about analyzing the population variance. In fact, we are analyzing treatment means by identifying sources of variability of the data. In its simplest form, analysis of variance can be considered as an extension of the test of hypothesis for the equality of two means that we learned in Chapter 7. Actually, the so-called one-way analysis of variance is a generalization of the two-means procedure to a test of equality of the means of more than two independent, normally distributed populations.

10.2 Analysis of Variance Method for Two Treatments (Optional) 501

Recall that the methods of testing H0 : μ1 − μ2 = 0, such as the t-test, were discussed earlier. In this chapter, we are concerned with studying situations involving the comparison of more than two population or treatment means. For example, we may be interested in the question “Do the rates of heart attack and stroke differ for three different groups of people with high cholesterol levels (borderline high such as 150–199 mg/dL, high such as 200–239 mg/dL, very high such as greater than 240 mg/dL) and a control group given different dosage levels of a particular cholesterol-lowering drug (say, a particular statin drug)?” Let us consider four populations with means μ1 , μ2 , μ3 , and μ4 , and say that we wish to test the hypotheses μ1 = μ2 = μ3 = μ4 . That is, the mean rate is the same for all the four groups. The question here is: Why do we need a new method to test for differences among the four procedure population means? Why not use z- or t-tests for all possible pairs and test for differences in each pair? If any one of these tests leads to the rejection of the hypothesis of equal means, then we might conclude that at least two of the four population means differ. The problem with this approach is that our ﬁnal decision is based on results of 42 = 6 different tests, and any one of them can be wrong. For each of the six tests, let α = 0.10 be the probability of being wrong (type I error). Then the probability that at least one of the six tests leads to the conclusion that there is a difference leads to an error 1 − (0.9)6 = 0.46856, which clearly is much larger than 0.10, thus resulting in a large increase in the type I error rate. Hence, if an ordinary t-test is used to make several treatment comparisons from the same data, the actual α-value applying to the tests taken as a group will be larger than the speciﬁed value of α, and one is likely to declare signiﬁcance when there is none. Analysis of variance procedures were developed to eliminate the increase in error rates resulting from multiple t-tests. With ANOVA, we are able to set one alpha level and test whether any of the group means differ from one another. Given a sample from each of the populations, our interest is to answer the question: Are the observed discrepancies among the different sample means merely due to chance ﬂuctuations, or are they due to inherent differences among the populations? Analysis of variance separates the effect of purely random variations from those caused by existing differences among population means: The phrase “analysis of variance” springs from the idea of analyzing variability in the data to see how much can be attributed to differences in μ and how much is due to variability in the individual populations. The ANOVA method incorporates information on variability from all of the samples simultaneously. At the heart of ANOVA is the fact that variances can be partitioned, with each partition attributable to a speciﬁc source. The method inspects various sums of squares (which are measures of variation in a sample) calculated from the data. ANOVA looks at two types of sums of squares: sums of squares within groups and sums of squares between groups. That is, it looks at each of the distributions and compares the between-group differences (variation in group means) with the within-group differences (variation in individuals’ scores within groups).

10.2 ANALYSIS OF VARIANCE METHOD FOR TWO TREATMENTS (OPTIONAL) In this section, we present the simplest form of the analysis of variance procedure, the case of studying the means of two populations I and II. For comparing only two means, the ANOVA will result in the same conclusions as the t-test for independent random samples. The basic purpose of this section is to introduce the concept of ANOVA in simpler terms. Let us consider two random samples of size n1 and

502 CHAPTER 10 Analysis of Variance

n2 , respectively. That is, y11 , y12 , . . . , y1n1 from population I and y21 , y22 , . . . , y2n2 from population II. Let y1 =

y11 + y12 + · · · + y1n1 (sample mean from population I) n1

and y2 =

y21 + y22 + · · · + y2n2 (sample mean from population II). n2

These samples are assumed to be independent and come from normal populations with respective means μ1 , μ2 , and variances σ12 = σ22 . We wish to test the hypothesis H0 : μ1 = μ2 vs. Ha : μ1 = μ2 .

The total variation of the two combined response measurements about y (the sample mean of all n = n1 + n2 observations) is (SS is used for sum of squares) deﬁned by Total SS =

ni 2

2 yij − y .

(10.1)

i=1 j=1

That is, y=

y11 + y12 + · · · + y1n1 + y21 + y22 + · · · + y2n2 . n

The total sums of squares measures the total spread of scores around the grand mean, y. We can rewrite (10.1) as

Total SS =

ni 2

yij − y

2

i=1 j=1

=

n1

y1j − y

2

+

j=1

=

n1

n2

y2j − y

j=1

y1j − y1 + y1 − y

j=1

=

2

+

n2

y2j − y2 + y2 − y

2

j=1

n1

y1j − y1

2

n1 2 y1j − y1 + n1 y 1 − y + 2 y 1 − y

j=1

+

2

n2 j=1

j=1

y2j − y2

2

n2 2 + n2 y 2 − y + 2 y 2 − y y2j − y2 . j=1

10.2 Analysis of Variance Method for Two Treatments (Optional) 503

Note that

n1 j=1

n2 y1j − y1 = 0 = y2j − y2 . We obtain j=1

Total SS =

n1

y1j − y1

2

+

j=1

y2j − y2

2

j=1

+ n1 y 1 − y =

n2

2

2 + n2 y 2 − y

ni 2

yij − yi

2

i=1 j=1

+

2

2 ni y i − y .

(10.2)

i=1

Deﬁne SST, the sum of squares for treatment by 2

SST =

ni (yi − y)2 .

i=1

The SST measures the total spread of the group means yi with respect to the grand mean, y. Also, SSE represents the sum of squares of errors given by SSE =

ni 2

yij − yi

2

i=1 j=1

=

n1

y1j − y1

j=1

2

+

n2

y2j − y2

2

j=1

= (n1 − 1)s12 + (n2 − 1)s22

where s12 and s22 are the unbiased sample variances of the two random samples. Note that this connects the sum of squares to the concept of variance we have been using in previous chapters. We can now rewrite (10.2) as Total SS = SSE + SST.

It should be clear that the SSE measures the within-sample variation of the y-values (effects), whereas SST measures the variation among the two sample means. The logic by which the analysis of variance tests is as follows: If the null hypothesis is true, then SST as compared to SSE should be about the same, or less. The larger SST, the greater will be the weight of evidence to indicate a difference in the means μ1 and μ2 . The question then is, how large? To answer this question, let us suppose we have two populations that are normal. That is, let Yij be N μi , σ 2 distributed with values yij . Then the pooled unbiased estimate of σ 2 is given by 2 = sp

(n1 − 1) s12 + (n2 − 1) s22 SSE = . n1 + n 2 − 2 n1 + n 2 − 2

504 CHAPTER 10 Analysis of Variance

Hence,

2 =E σ 2 = E sp

SSE . n1 + n 2 − 2

Also, we can write 2 2 n1 n2 Y1j − Y1 Y2j − Y2 SSE = + σ2 σ2 σ2 j=1

j=1

which has a χ2 -distribution with (n1 + n2 − 2) degrees of freedom. Under the hypothesis that μ1 = μ2 , E (SST ) = σ 2 . Furthermore, Y1 − Y2 Z= 8 ∼ N (0, 1) . σ 2 n11 + n12

This implies that

Z2 =

1 1 + n1 n2

%

Y1 − Y2 σ2

& =

SST σ2

has a χ2 −distribution with 1 degree of freedom. It can be shown that SST and SSE are independent. From Chapter 4, we restate the following result. Theorem 10.2.1 If χ12 has υ1 degrees of freedom χ22 has υ2 degrees of freedom, and χ12 and χ22 are indeχ 2 /υ pendent, then F = χ12 υ1 has an F -distribution with υ1 numerator degrees of freedom and υ2 denominator 2/ 2 degrees of freedom. Using the foregoing result, we have > SST (1) σ 2 SST /1 = > SSE/(n1 + n2 − 2) SSE (n1 + n2 − 2) σ 2

which has an F -distribution with υ1 = 1 numerator degrees of freedom and υ2 = (n1 + n2 − 2) denominator degrees of freedom. Now, we introduce the mean square error (MSE), deﬁned by MSE = =

SSE (n1 + n2 − 2) (n1 − 1) s12 + (n2 − 1) s22 (n1 + n2 − 2)

10.2 Analysis of Variance Method for Two Treatments (Optional) 505

and the mean square treatment (MST) given by SST 1 2 2 . = n1 y1 − y + n2 y2 − y

MST =

Under the null hypothesis, H0 : μ1 = μ2 , both MST and MSE estimate σ 2 without bias. When H0 is false and μ1 = μ2 , MST estimates something larger than σ 2 and will be larger than MSE. That is, if H0 is false, then E(MST ) > E(MSE) and the greater the differences among the values of μ, the larger E(MST ) will be relative to E(MSE). Hence, to test H0 : μ1 = μ2 vs. Ha : μ1 = μ2 , we use the F -test given by F=

MST MSE

as the test statistic. Thus, for given α, the rejection region is {F > Fα }. It is important to observe that compared to the small sample t-test, here we work with variability. Now we summarize the analysis of variance procedure for the two-sample case.

ANALYSIS OF VARIANCE PROCEDURE FOR TWO TREATMENTS For equal sample sizes n = n1 = n2 , assume σ12 = σ22 . We test H0 : μ1 = μ2 vs. Ha : μ1 = μ2 . 1. Calculate: y1 , y2 ,

2 yij , yij , and ﬁnd ij

ij

SST =

2

2 ni y i − y .

i=1

Also calculate Total SS =

i

j

yij2 −

i

2 yij

j

n1 + n2

Then SSE = Total SS − SST .

.

506 CHAPTER 10 Analysis of Variance

2. Compute MST =

SST 1

MSE =

SSE . n1 + n2 − 2

3. Compute the test statistic, F=

MST . MSE

4. For a given α, ﬁnd the rejection region as RR : F > Fα , based on 1 numerator and (n1 + n2 − 2) denominator degrees of freedom. 5. Conclusion: If the test statistic F falls in the rejection region, conclude that the sample evidence supports the alternative hypothesis that the means are indeed different for the two treatments. Assumptions: Populations are normal with equal but unknown variances.

Example 10.2.1 The following data represent a random sample of end-of-year bonuses for lower-level managerial personnel employed by a large ﬁrm. Bonuses are expressed in percentage of yearly salary. Female 6.2 9.2 8.0 7.7 8.4 9.1 7.4 6.7 Male 8.9 10.0 9.4 8.8 12.0 9.9 11.7 9.8 The objective is to determine whether the male and female bonuses are the same. We can answer this question by connecting the following. (a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.05. (b) What assumptions are necessary for the test in part (a)? (c) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a).

Solution (a) We need to test H0 : μ1 = μ2 vs. Ha : μ1 = μ2 From the random sample, we obtain the following needed estimates, n1 = n2 = 8: y1 = 7.8375, y2 = 10.0625,

2 = 1319.34, yij

ij

SST =

2 i=1

ni (yi − y2 )2 = 19.8025.

ij

yij = 143.20

10.2 Analysis of Variance Method for Two Treatments (Optional) 507

Therefore, Total SS =

i

2 yij

2 − yij

i

j

= 1391.34 −

2

j

2n

(143.2)2 = 109.70. 16

Then SSE = Total SS − SST = 109.7 − 19.8025 = 89.8975, MST =

SST = 19.8025 1

and MSE =

89.8975 SSE = 2n1 − 2 14

= 6.42125. Hence, the test statistic F=

19.8025 MST = MSE 6.42125

= 3.0839. For α = 0.05, F0.05,14 = 4.60. Hence the rejection region is {F > 4.60}. Because 3.0839 is not greater than 4.60, H0 is not rejected. There is not enough evidence to indicate that the average bonuses are different for men and women at α = 0.05. (b) To solve the problem, we assumed that the samples are random and independent with n1 = n2 = 8, drawn from two normal populations with means μ1 and μ2 and common variance σ 2 . 2 = 6.42125. Also, y = 7.8375 and y = 10.0625. Then, (c) The value of MSE is the same as s2 = sp 1 2 the t-statistic is 7.8375 − 10.0625 y1 − y2 t= 8 = 8 = −1.756. 1 1 1 1 2 s n1 + n2 6.42125 8 + 8 Now, t0.025,14 = 2.415 and the rejection region is {t < −2.145}. Because −1.756 is not less than −2.45, H0 is not rejected, which implies that there is no significant difference between the bonuses for the males and the females. Note also that t 2 = F , that is, (−1.756)2 = 3.083 implying that in the two-sample case, the t-test and F -test lead to the same result.

508 CHAPTER 10 Analysis of Variance

It is not surprising that in the previous example, the conclusions reached using ANOVA and two sample t-tests are the same. In fact, it can be shown that for two sets of independent and normally distributed random variables, the two procedures are entirely equivalent for a two-sided hypothesis. However, a t-test can also be applied to a one-sided hypothesis, whereas ANOVA cannot. The purpose of this section is only to illustrate the computations involved in the analysis of variance procedures as opposed to simple t-tests. The analysis of variance procedure is effectively used for three or more populations, which is described in the next section.

EXERCISES 10.2 10.2.1.

The following information was obtained from two independent samples selected from two normally distributed populations with unknown but equal standard deviations. Do the data present sufﬁcient evidence to indicate that there is a difference in the mean for the two populations? Sample 1 Sample 2

1 2

2 5

3 2

3 4

1 3

2 1

1 2

3 3

1 3

(a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.05. (b) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a). 10.2.2.

The following information was obtained from two independent samples selected from two normally distributed populations with unknown but equal standard deviations. Do the data present sufﬁcient evidence to indicate that there is a difference in the mean for the two populations? Sample 1: Sample 2:

15 18

13 16

11 13

14 21

10 16

12 19

7 15

12 18

11 19

14 20

15 21

14

(a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.01. (b) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a). 10.2.3.

A company claims that its medicine, brand A, provides faster relief from pain than another company’s medicine, brand B. A random sample from each brand gave the following times (in minutes) for relief. Do the data present sufﬁcient evidence to indicate that there is a difference in the mean time to relief for the two populations? Brand A: Brand B:

47 44

51 48

45 42

53 45

41 44

55 42

50 49

46 46

45 45

51 48

53 39

50 49

48

(a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.01. (b) What assumptions are necessary for the conclusion in part (a)?

10.2 Analysis of Variance Method for Two Treatments (Optional) 509

(c) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a). 10.2.4.

Table 10.2.1 gives mean SAT scores for math by state for 1989 and 1999 for 20 randomly selected states (source: The World Almanac and Book of Facts 2000).

Table 10.2.1 State

1989

1999

Arizona

523

525

Connecticut

498

509

Alabama

539

555

Indiana

487

498

Kansas

561

576

Oregon

509

525

Nebraska

560

571

New York

496

502

Virginia

507

499

Washington

515

526

Illinois

539

585

North Carolina

469

493

Georgia

475

482

Nevada

512

517

Ohio

520

568

New Hampshire

510

518

Using the ANOVA procedure, test that the mean SAT score for math in 1999 is greater than that in 1989 at α = 0.05. Assume that the variances are equal and the samples come from a normal distribution. 10.2.5.

Let X1 , . . . , Xn1 and Y1 , . . . , Yn2 be two sets of independent, normally distributed random variables with means μ1 and μ2 , and the common variance σ 2 . Show that the two-sample t-test and the analysis of variance are equivalent for testing H0 : μ1 = μ2 versus Ha : μ1 > μ2 .

510 CHAPTER 10 Analysis of Variance

10.3 ANALYSIS OF VARIANCE FOR COMPLETELY RANDOMIZED DESIGN In this section, we study the hypothesis testing problem of comparing population means for more than two independent populations, where the data are about several independent groups (different treatments being applied, or different populations being sampled). We have seen in Chapter 9 that the random selection of independent samples from k populations is known as a completely randomized experimental design or one-way classiﬁcation. Let μ1 , . . . , μk be the means of k normal populations with unknown but equal variance σ 2 . The question is whether the means of these groups are different or are all equal. The idea is to consider the overall variability in the data. We partition the variability into two parts: (1) between-groups variability and (2) within-groups variability. If between groups is much larger than that within groups, this will indicate that differences between the groups are real, not merely due to the random nature of sampling. Let independent samples be drawn of sizes ni , i = 1, 2, . . . , k and let N = n1 + · · · + nk . Let yij be the measured response on the jth experimental unit in the ith sample. That is, Yij is the jth observation from population i, i = 1, 2, . . . , k, and j = 1, 2, . . . , ni . Let y be the overall mean of all observations. The problem can be formulated as a hypothesis testing problem, where we need to test H0 : μ1 = μ2 = . . . = μk vs. Ha : Not all the μi s are equal.

The method of analysis of variance tests the null hypothesis H0 by comparing two unbiased estimates of the variance, σ 2 , an estimate based on variations from sample to sample and the other one based on variations within the samples. We will be rejecting H0 if the ﬁrst estimate is signiﬁcantly larger than the second, so that the samples cannot be assumed to come from the same population. We can write the total sum of squares of deviations of the response measurements about their overall mean for the k samples into two parts, from the treatment (SST) and from the error (SSE). This partition gives the fundamental relationship in ANOVA, where total variation is divided into two portions: between-sample variation and within-sample variation. That is, Total SS = SST + SSE.

The following derivations will make computation of these quantities simpler. The total SS can be written as Total SS =

ni k

yij − y

2

=

i=1 j=1 ni k

Note that y =

i=1 j=1

N

ni k

2 − 2y yij

i=1 j=1

yij

, and then we have Total SS =

ni k i=1 j=1

2 − CM yij

ni k i=1 j=1

yij + Ny2 .

10.3 Analysis of Variance for Completely Randomized Design 511

where CM is the correction factor for the correction for the means and is given by

ni k

2 yij

i=1 j=1

CM =

= Ny2 .

N

Let Ti =

ni

yij , be the sum of all the observations in the ith sample

j=1

and ni

Ti =

yij

j=1

ni

, the mean of the observations in the ith sample.

We can rewrite y as ni k

y=

k

yij

i=1 j=1

=

N

ni T i

i=1

N

Now, we introduce SST, the sum of squares for treatment (sometimes known as between group sum of squares, SSB) by SST =

k

2 ni Ti − y .

i=1

We note that Ti is the mean response due to its ith treatment and y is the overall mean. A large value of Ti − y is likely to be caused by the ith treatment effect being much different from the rest. Hence SST can be used to measure the differences in the treatment effects. Thus, the sum of squares of errors (SSE) is SSE = Total SS − SST.

We must state that the SSE is the sum of squares within groups (thus, sometimes SSE is referred to as within group sum of squares, SSW) and this can be seen from rewriting the expression as SSE =

ni k

2 yij − T i .

i=1 j=1

The decomposition of total sum of squares can be easily seen in Figure 10.1. Figure 10.2 represents one point for each observation against each sample, with SM representing the sample means and GM representing the grand mean. The dotted line between SMs and GM is the

512 CHAPTER 10 Analysis of Variance

Total sum of squares

SST (or between group sum of squares

SSE (or within group sum of squares ni

k

⫽

k

(yij ⫺Ti )2

i ⫽1 j ⫽1

2

ni (Ti ⫺ y )

⫽ i⫽1

Observations

■ FIGURE 10.1 Decomposition of total SS.

GM SM

0

SM

0 0

I

SM

II

III Sample

■ FIGURE 10.2 ANOVA decomposition.

distance between them. Taking this distances, squaring, multiplying by the corresponding sample sizes, and summing, we get SST. To obtain SSE, we take the distance from each group mean, SM, to each member of the group, square them, and add them. In addition, to give an idea of within-group variations, it is customary to draw side-by-side box plots. As mentioned earlier, SST estimates the variation among the μi s, and hence if all the μi s were equal, the Ti s would be similar and the SST would be small. It can be veriﬁed that the unbiased estimator of σ 2 based on (n1 + n2 + · · · + nk − k) degrees of freedom is S 2 = MSE = =

SSE . N −k

SSE (n1 + n2 + · · · + nk − k)

10.3 Analysis of Variance for Completely Randomized Design 513

Note that the quantity MSE is a measure of variability within the groups. If there were only one group with n observations, then the MSE is nothing but the sample variance, s2 . The fact that ANOVA deals simultaneously with all the k groups can be seen by rewriting MSE in the following form: MSE =

(n1 − 1) s12 + (n2 − 1) s22 + · · · + (nk − 1) sk2 (n1 − 1) + (n2 − 1) + · · · + (nk − 1)

.

The mean square for treatments with (k − 1) degrees of freedom is MST =

SST . k−1

The MST is a measure of the variability between the sample means of the groups. We now summarize the analysis of variance hypothesis testing method for two or more populations. ONE-WAY ANALYSIS OF VARIANCE FOR k ≥ 2 POPULATIONS We test H0 : μ1 = μ2 = . . . = μk

versus

Ha : At least two of the μi s are different. When H0 is true, we have E(MST ) = E(MSE ) The greater the differences among the μ s, the larger the E(MST ) will be relative to E(MSE ). Test statistic: MST . F= MSE Rejection region is RR : F > Fα with υ1 = (k − 1) numerator degrees of freedom and υ2 = ki=1 ni − k = N − k denominator degrees of k freedom, where N = i=1 ni . Assumptions: The observations Yij s are assumed to be independent and normally distributed with mean μi , i = 1, 2, . . . , k , and variance σ 2 .

Now we give a ﬁve-step computational procedure that we could follow for analysis of variance for the completely randomized design. ONE-WAY ANALYSIS OF VARIANCE PROCEDURE FOR k ≥ 2 POPULATIONS We test H0 : μ1 = μ2 = . . . = μk versus Ha : At least two of the μi s are different.

514 CHAPTER 10 Analysis of Variance

1. Compute Ti =

CM =

ni

ni k

yij , T =

j=1

i=1 j=1

2

ni k

yij

yij , and

ni k

yij2 ,

i=1 j=1

T2 , where N = ni , N k

i=1 j=1

=

N

i=1

T Ti = i , ni and Total SS =

ni k

yij2 − CM.

i=1 j=1

2. Compute the sum of squares between samples (treatments), SST =

k Ti2 i=1

=

k

ni

− CM

Ti − CM.

i=1

and the sum of squares within samples, SSE = Total SS − SST Let MST =

SST , k −1

MSE =

SSE . n−k

and

3. Compute the test statistic: F=

MST . MSE

4. For a given α, ﬁnd the rejection region as RR : F > Fα

10.3 Analysis of Variance for Completely Randomized Design 515 k with υ1 = k − 1 numerator degrees of freedom and υ2 = i=1 ni − k = N − k denominator degrees of freedom, where N = ki=1 ni . 5. Conclusion: If the test statistic F falls in the rejection region, conclude that the sample evidence supports the alternative hypothesis that the means are indeed different for the k treatments and are not all equal. Assumptions: The samples are randomly selected from the k populations in an independent manner. The populations are assumed to be normally distributed with equal variances σ 2 and means μ1 , . . . , μk .

10.3.1 The p-Value Approach Note that if we are using statistical software packages, the p-value approach can be used for the testing. Just compare the p-value and α to arrive at a conclusion. Refer to the computer examples in Section 10.7. The following example illustrates the ANOVA procedure.

Example 10.3.1 The three random samples in Table 10.1 represent test scores from three classes of statistics taught by three different instructors and are independently obtained. Assume that the three different populations are normal with equal variances. At the α = 0.05 level of signiﬁcance, test for equality of population means.

Table 10.1 Sample 1

Sample 2

Sample 3

64

56

81

84

74

92

75

69

84

77 80

Solution We test H0 : μ1 = μ2 = μ3 versus Ha : At least two of the μ s are different. Here, k = 3, n1 = 5, n2 = 3, and N = n1 + n2 + n3 = 11.

516 CHAPTER 10 Analysis of Variance

Also, Ti ni Ti

380 5 76

199 3 66.33

257 3 85.67

Clearly, the sample means are different. The question we are going to answer is: Is this difference due to just chance, or is it due to a real difference caused by different teaching styles? For this, we now compute the following: 2 yij (836)2 i j CM = = = 63,536 N 11 2 − CM yij Total SS = i

j

= 64,558 − 63,536 = 1022 SST =

T2

i − CM

i

ni

(199)2 (257)2 (380)2 + + − CM = 5 3 3 = 64,096.66 − 63,536 = 560.66 SSE = Total SS − SST = 1022 − 560.66 = 461.34. Hence, MST =

SST 560.66 = = 280.33, k−1 2

MSE =

461.34 SSE = = 57.67. N −k 8

and

The test statistic is F=

280.33 MST = = 4.86. MSE 57.67

From the F -table, F0.05,2,8 = 4.46. Therefore, the rejection region is given by RR : F > 4.46. Decision: Because the observed value of F = 4.86 falls in the rejection region, we do reject H0 and conclude that there is sufficient evidence to indicate a difference in the true means.

10.3 Analysis of Variance for Completely Randomized Design 517

If we want the p-value, we can see from the F -table that 0.025 < p-value < 0.05, indicating the rejection of the null hypothesis with α = 0.05. Using statistical software packages, we can get the exact p-value.

The calculations obtained in analyzing the total sum of squares into its components are usually summarized by the analysis-of-variance table (ANOVA table), given in Table 10.2. Sometimes, one may also add a column for the p-value, P(Fk−1,n−k ≥ observed F ), in the ANOVA table. For the previous example, we can summarize the computations by the ANOVA table shown in Table 10.3.

10.3.2 Testing the Assumptions for One-Way ANOVA The randomness assumption could be tested using the Wald–Wolfowitz test (see Project 12B). The assumption of independence of the samples is hard to test without knowing how the data are collected and should be implemented during collection of data in the design stage. Normality can be tested (this should be performed separately for each sample, not for the total data set) using probability plots or other tests such as the chi-square goodness-of-ﬁt-test. ANOVA is fairly robust against violation of this assumption if the sample sizes are equal. Also, if the sample sizes are fairly large, the central limit theorem helps. The presence of outliers is likely to increase the sample variance, thus decreasing

Table 10.2 Source of variation

Degree of freedom

Treatments

k−1

Sum of squares SST =

k i=1

Error

n−k

Total

n−k

Ti2 ni

Mean squares

− CM

SSE = Total SS − SST Total SS =

ni k

yij − y

Fstatistic

MST =

SST k−1

MSE =

SSE n−k

2

i=1 i=1

Table 10.3 Source of variation

Degree of freedom

Sum of squares

Mean square

F-statistic

p-Value

Treatments

2

560.66

280.33

4.86

0.042

Error

8

461.34

57.67

Total

10

1022

MST MSE

518 CHAPTER 10 Analysis of Variance

the value of the F -statistic for ANOVA, which will result in a lower power of the test. Box plots or probability plots could be used to identify the outliers. If the normality test fails, transforming the data (see Section 14.4.2) or a nonparametric test such as the Kruskal–Wallis test described in Section 12.5.1 may be more appropriate. If the sample sizes of each sample are equal, ANOVA is mostly robust for violation of homogeneity of the variances. A rule of thumb used for robustness for this condition is that the ratio of sample variance of the largest sample variance s2 to the smallest sample variance s2 should be no more than 3 : 1. Another popular rule of thumb used in one-way ANOVA to verify the requirement of equality of variances is that the largest sample standard deviation not be larger than two times the smallest sample standard deviation. Graphically, representing side-byside box plots of the samples can also reveal lack of homogeneity of variances if some box plots are much longer than others (see Figure 10.3e). For a signiﬁcance test on the homogeneity of variances (Levene’s test), refer to Section 14.4.3. If these tests reveal that the variances are different, then the populations are different, in spite of what ANOVA concludes about differences of the means. But this itself is signiﬁcant, because it shows that the treatments had an effect.

Example 10.3.2 In order to study the effect of automobile size on the noise pollution, the following data are randomly chosen from the air pollution data (source: A. Y. Lewin and M. F. Shakun, Policy Sciences: Methodology and Cases, Pergamon Press, 1976, p. 313). The automobiles are categorized as small, medium, large, and noise level reading (decibels) are given in Table 10.4.

Table 10.4 Size of automobile Small

Medium

Large

820

840

785

Noise level

820

825

775

(decibels)

825

815

770

835

855

760

825

840

770

At the α = 0.05 level of signiﬁcance, test for equality of population mean noise levels for different sizes of the automobiles. Comment on the assumptions.

Solution Let μ1 , μ2 , μ3 be population mean noise levels for small, medium, and large automobiles, respectively. First we test for the assumptions. Using Minitab, run tests for each of the samples; we can justify the assumption of randomness of the sample values. A normality test for each column gives the graphs shown in Figures 10.3a through 10.3c, through which we can reasonably assume the normality. Because the sample sizes are equal, we will use the one-way ANOVA method to analyze these data.

10.3 Analysis of Variance for Completely Randomized Design 519

Figure 10.3d indicates that the relative positions of the sample means are different, and Figure 10.3e (Minitab steps for creating side-by-side box plots are given at the end of Example 10.7.1) gives an indication of withingroup variations; perhaps the group 2 (medium-size) variance is larger. Now, we will do the analytic testing. Noise level for small size automobiles 0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 820

825

830

835

Small Average: 825 Std Dev: 6.12372 N: 5

Kolmogorov-Smirnov Normality Test D⫹: 0.200 D⫺: 0.149 D: 0.200 Approximate P -Value ⬎ 0.15

■ FIGURE 10.3(a) Normal plot for noise level of small automobiles.

Noise level for medium size automobiles 0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 815

Average: 835 Std Dev: 15.4110 N: 5

825

835 Medium

845

855

Kolmogorov-Smirnov Normality Test D⫹: 0.142 D ⫺: 0.127 D: 0.142 Approximate P -Value ⬎ 0.15

■ FIGURE 10.3(b) Normal plot for noise level of medium-sized automobiles.

520 CHAPTER 10 Analysis of Variance

Noise level for large size automobiles

0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 760

770

780 Large

Average: 772 Std Dev: 9.08295 N: 5

Kolmogorov-Smirnov Normality Test D⫹: 0.171 D⫺: 0.124 D: 0.171 Approximate P -Value ⬎ 0.15

■ FIGURE 10.3(c) Normal plot for noise level of large automobiles.

840 830

Mean

820 810 800 790 780 770 1

2 Sample

■ FIGURE 10.3(d) Mean decibel levels for three sizes of automobiles.

3

10.3 Analysis of Variance for Completely Randomized Design 521

860 850 840

Decibels

830 820 810 800 790 780 770 760 1

2 Size of auto

3

■ FIGURE 10.3(e) Side-by-side box plots for decibel levels for three sizes of automobiles.

We test H0 : μ1 = μ2 = μ3 versus Ha : At least two of the μ s are different. Here, k = 3, n1 = 5, n2 = 5, n3 = 5 and N = n1 + n2 + n3 = 15. Also Ti ni Ti

4125 5 825

4175 5 835

3860 5 772

In the following calculations, for convenience we will approximate all values to the nearest integer. 2 yij (12,160)2 i j = = 9,857,707 CM = N 15 2 − CM Total SS = yij i

j

= 12,893 SST =

T2

i − CM

i

ni

= 11,463 SSE = Total SS − SST = 1430.

522 CHAPTER 10 Analysis of Variance

Hence, MST =

SST 11,463 = = 5732 k−1 2

and MSE =

SSE 1430 = = 119. N −k 12

The test statistic is F=

MST 5732 = = 48.10. MSE 119

From the table, we get F0.05,2,12 = 3.89. Because the test statistic falls in the rejection region, we reject at α = 0.05 the null hypothesis that the mean noise levels are the same. We conclude that size of the automobile does affect the mean noise level.

It should be noted that the alternative hypothesis Ha in this section covers a wide range of situations, from the case where all but one of the population means are equal to the case where they are all different. Hence, with such an alternative, if the samples lead us to reject the null hypothesis, we are left with a lot of unsettled questions about the means of the k populations. These are called post hoc testing. This problem of multiple comparisons is the topic of Section 10.5.

10.3.3 Model for One-Way ANOVA (Optional) We conclude this section by presenting the classical model for one-way ANOVA. Because the variables Yij values are random samples from normal populations with E(Yij ) = μi and with common variance Var(Yij ) = σ 2 , for i = 1, . . . , k and j = 1, . . . , ni , we can write a model as Yij = μi + εij , j = 1, . . . , ni

where the error terms εij are independent normally distributed random variables with E(εij ) = 0 and Var(εij ) = σ 2 . Let αi = μ − μi be the difference of μi (ith population mean) from the grand mean μ. Then αi can be considered as the ith treatment effect. Note that the αi values are nonrandom. Because μ = i (ni μi /N), it follows that ki=1 αi = 0. This will result in the following classical model for one-way layout: Yij = μ + αi + εij ,

i = 1, . . . , k,

j = 1, . . . , ni .

With this representation, the test H0 : μ1 = μ2 = . . . . = μk reduces to testing the null hypothesis that there is no treatment effect, H0 : αi = 0, for i = 1, . . . , k.

EXERCISES 10.3 10.3.1.

In an effort to investigate the premium charged by insurance companies for auto insurance, an agency randomly selects a few drivers who are insured by one of three different companies. These individuals have similar cars, driving records, and levels of coverage.

10.3 Analysis of Variance for Completely Randomized Design 523

Table 10.3.1 gives the premiums paid per 6 months by these drivers with these three companies.

Table 10.3.1 Company I

Company II

Company III

396

348

378

438

360

330

336

522

294

318

474

432

(a) Construct an analysis-of-variance table and interpret the results. (b) Using the 5% signiﬁcance level, test the null hypothesis that the mean auto insurance premium paid per 6 months by all drivers insured for each of these companies is the same. Assume that the conditions of completely randomized design are met. 10.3.2.

Three classes in elementary statistics are taught by three different persons: a regular faculty member, a graduate teaching assistant, and an adjunct from outside the university. At the end of the semester, each student is given a standardized test. Five students are randomly picked from each of these classes, and their scores are as shown in Table 10.3.2.

Table 10.3.2 Faculty

Teaching assistant

Adjunct

93

88

86

61

90

56

87

76

73

75

82

90

92

58

47

(a) Construct an analysis-of-variance table and interpret your results. (b) Test at the 0.05 level whether there is a difference between the mean scores for the three persons teaching. Assume that the conditions of completely randomized design are met. 10.3.3.

Let n1 = n2 = . . . = nk = n . Show that

k n

yij − y

i=1 j=1

2

=

k n

yij − Ti

i=1 j=1

2

+n

k i=1

2 Ti − y .

524 CHAPTER 10 Analysis of Variance

10.3.4.

For the sum of squares for treatment SST =

k

2 n i Ti − y

i=1

show that E (SST ) = (k − 1) σ 2 +

k

ni (μi − μ)2

i=1

where μ =

1 N

k

ni μi .

i=1

[This exercise shows that the expected value of SST increases as the differences among the μi s increase.] 10.3.5.

(a) Show that SSE = 1 n−1

k i=1

ni

(ni − 1) Si2 =

ni k

2 Yij − Ti ,

i=1 j=1

2

where = provides an independent, unbiased estimator for j=1 Yij − Ti σ 2 in each of the k samples. > (b) Show that SSE σ 2 has a chi-square distribution with N − k degrees of freedom, where N = ki=1 ni . Si2

10.3.6.

Let each observation in a set of k independent random samples be normally distributed with means μ1 , . . . , μk and common variance σ 2 . If H0 = μ1 = μ2 = . . . = μk is true, show that F=

MST SST /(k − 1) = SSE/(n − k) MSE

has an F -distribution with k − 1 numerator and n − k denominator degrees of freedom. 10.3.7.

The management of a grocery store observes various employees for work productivity. Table 10.3.3 gives the number of customers served by each of its four checkout lanes per hour.

Table 10.3.3 Lane 1

Lane 2

Lane 3

Lane 4

16

11

8

21

18

14

12

16

22

10

17

17

21

10

10

23

15

14

13

17

10

15

10.3 Analysis of Variance for Completely Randomized Design 525

(a) Construct an analysis-of-variance table and interpret the results. Indicate any assumptions that were necessary. (b) Test whether there is a difference between the mean number of customers served by the four employees at the 0.05 level. Assume that the conditions of completely randomized design are met. 10.3.8.

Table 10.3.4 represents immunoglobulin levels (with each observation being the IgA immunoglobulin level measured in international units) of children under 10 years of age of a particular group. The children are grouped as follows: A: ages 1 to less than 3, B: ages 3 to less than 6, C: ages 6 to less than 8, and D: ages 8 to less than 10. Test whether there is a difference between the means for each of the age groups. Use α = 0.05. Interpret your results and state any assumptions that were necessary to solve the problem.

Table 10.3.4

10.3.9.

A

35

8

12

19

56

64

75

25

B

31

79

60

45

39

44

45

62

20

C

74

56

77

35

95

81

28

D

80

42

48

69

95

40

86

79

51

66

Table 10.3.5 gives rental and homeowner vacancy rates by U.S. region (source: U.S. Census Bureau) for 5 years.

Table 10.3.5 Rental units

1995

1996

1997

1998

1999

Northeast

7.2

7.4

6.7

6.7

6.3

Midwest

7.2

7.9

8.0

7.9

8.6

South

8.3

8.6

9.1

9.6

10.3

West

7.5

7.2

6.6

6.7

6.2

Test at the 0.01 level whether the true rental and homeowner vacancy rates by area are the same for all 5 years. Interpret your results and state any assumptions that were necessary to perform the analysis. 10.3.10.

Table 10.3.6 gives lower limits of income (approximated to the nearest $1000 and calculated as of March of the following year) of the top 5% of U.S. households by race from 1994 to 1998 (Source: U.S. Census Bureau). Test at the 0.05 level whether the true lower limits of income for the top 5% of U.S. households for each race are the same for all 5 years.

526 CHAPTER 10 Analysis of Variance

Table 10.3.6 Race

10.3.11.

Year 1994

1995

1996

1997

1998

All Races

110

113

120

127

132

White

113

117

123

130

136

Black

81

80

85

87

94

Hispanic

82

80

86

93

98

Table 10.3.7 gives mean serum cholesterol levels (given in milligrams per deciliter) by race and age in the United States between 1978 and 1980 (source: “Report of the National Cholesterol Education Program Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults,” Arch. Intern. Med. 148, January 1988).

Table 10.3.7 Race

Age 20–24

25–34

35–44

45–54

55–64

65–74

All Races

180

199

217

227

229

221

White

180

199

217

227

230

222

Black

171

199

218

229

223

217

Test at the 0.01 level whether the true mean cholesterol levels for all races in the United States between 1978 and 1980 are the same.

10.4 TWO-WAY ANALYSIS OF VARIANCE, RANDOMIZED COMPLETE BLOCK DESIGN A randomized block design, or the two-way analysis of variance, consists of b blocks of k experimental units each. In many cases we may be required to measure response at combinations of levels of two or more factors considered simultaneously. For example, we might be interested in gas mileage per gallon among four different makes of cars for both in-city and highway driving, or to examine weight loss comparing ﬁve different diet programs among whites, African Americans, Hispanics, and Asians according to their gender. In studies involving various factors, the effect of each factor on the response variable may be analyzed using one-way classiﬁcation. However, such an analysis will not be efﬁcient with respect to time, effort, and cost. Also, such a procedure would give no knowledge about the likely

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 527

interactions that may exist among different factors. In such cases, the two-way analysis of variance is an appropriate statistical method to use. In a randomized block design, the treatments are randomly assigned to the units in each block, with each treatment appearing exactly once in every block (that is, there is no interaction between factors). Thus, the total number of observations obtained in a randomized block design is n = bk. The purpose of subdividing experiments into blocks is to eliminate as much variability as possible, that is, to reduce the experimental error or the variability due to extraneous causes. Refer to Section 9.2.3 for a procedure to obtain completely randomized block design. The goal of such an experiment is to test the equality of levels for the treatment effect. Sometimes, it may also be of interest to test for a difference among blocks. We proceed to give a formal statistical model for the completely randomized block design. For i = 1, 2, . . . , k and j = 1, 2, . . . , b, let Yij = μ + αi + βj + εij , where Yij is the observation on treatment i in block j, μ is the overall mean, αi is the nonrandom effect of treatment i, βj is the nonrandom effect of block j, and εij are the random error terms such that εij are independent αi = 0, normally distributed random variables with E εij = 0 and Var εij = σ 2 . In this case, and βj = 0. The analysis of variance for a randomized block design proceeds similarly to that for a completely randomized design, the main difference being that the total sum of squares of deviations of the response measurements from their means may be partitioned into three parts: the sum of squares of blocks (SSB), treatments (SST), and error (SSE). Let Bj = ki=1 yij and Bj denote, respectively, the total sum and mean of all observations in block j. Represent the total for all observations receiving treatment i by Ti = bj=1 yij , and mean and T i , respectively. Let y = average of n = bk observations =

b k 1 yij n j=1 i=1

and 2 1 total of all observations n ⎛ ⎞2 b k 1 ⎝ ⎠ = yij . n

CM =

j=1 i=1

For convenience, we can represent the two-way classiﬁcation as in Table 10.5. Note that from the table we can obtain

k b j=1 i=1

yij =

b j=1

Bj . Hence, CM = (1/n)

b j=1 Bj

2

.

528 CHAPTER 10 Analysis of Variance

Table 10.5 Blocks 1

2

...

j

...

b

Total T i

Mean T i

Treatment 1

y11

y12

...

y1j

...

y1b

T1

T1

Treatment 2

y21

y22

...

y2j

...

y2b

T2

T2

yij

...

yib

Ti

Ti

Tk

Tk

. . . Treatment i

. . .

. . .

yi1

yi2

...

. . .

. . .

Treatment k

yk1

yk2

...

ykj

...

ykb

Total Bj

B1

B2

...

Bj

...

Bb

Mean Bj

B1

B2

...

Bj

...

Bb

y

Then for a randomized block design with b blocks and k treatments, we need to compute the following sums of squares. They are Total SS = SSB + SST + SSE =

b k

yij − y

2

b k

=

j=1 i=1

j=1 i=1 b

SSB = k

b

2 − CM yij

Bj − y

2

=

Bj2

j=1

k

j=1

− CM

and k

SST = b

k

Ti − y

2

=

i=1

i=1

Ti2

b

SSE = Total SS − SSB − SST.

We deﬁne SSB , b−1 SST , MST = k−1

MSB =

− CM

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 529

Table 10.6 Source

d.f.

SS

MS

Blocks

b−1

SSB

SSB b−1

Treatments

k−1

SST

SST k−1

Error

(b − 1)(k − 1) =n−b−k+1

SSE

SSE n−b−k+1

Total

n−1

Total SS

and MSE =

SSE . n−b−k+1

The analysis of variance for the randomized block design is presented in Table 10.6. The column corresponding to d.f. represents the degrees of freedom associated with each sum of squares. MS denotes the mean square. To test the null hypothesis that there is no difference in treatment means, that is, to test H0 : αi = 0, i = 1, . . . , k versus Ha : Not all αi s are zero

we use the F -statistic F=

MST MSE

and reject H0 if F > Fα based on (k − 1) numerator and (n − b − k + 1) denominator degrees of freedom. Although blocking lowers the experimental error, it also furnishes a chance to see whether evidence exists to indicate a difference in the mean response for blocks. In this case we will be testing the hypothesis H0 : βj = 0, j = 1, . . . , b versus Ha : Not all βj s are zero.

Under the assumption that there is no difference in the mean response for blocks, MSB provides an unbiased estimator for σ 2 based on (b − 1) degrees of freedom. If there is a real difference that exists among block means, MSB will be larger in comparison with MSE and F=

MSB MSE

will be used as a test statistic. The rejection region will be if F > Fα based on (b − 1) numerator and (n − b − k + 1) denominator degrees of freedom.

530 CHAPTER 10 Analysis of Variance

We now summarize the foregoing methodology in a step-by-step computational procedure. For a reasonable data size, we could use scientiﬁc calculators for handling the ANOVA calculations. For larger data sets, the use of statistical software packages is recommended.

COMPUTATIONAL PROCEDURE FOR RANDOMIZED BLOCK DESIGN 1. Calculate the following quantities: (i) Sum the observations for each row to form row totals: T1 , T2 , . . . , Tk , where Ti =

b

yij .

j=1

(ii) Sum the observations for each column to form column totals: B1 , B2 , . . . , Bb , where Bj =

k

yij .

i=1

(iii) Find the sum of all observations: b k

yij =

j=1 i=1

b

Bj .

j=1

2. Calculate the following quantities: (i) Square the sum of the totals for each column and divide it by n = bk to obtain ⎛ ⎞ b 1 ⎝ 2 ⎠ Bj . CM = n j=1

(ii) Find the sum of squares of the totals of each column and divide it by k to obtain b 1 2 Bj k j=1

and b Bj2

SSB =

j=1

k

− CM

and

MSB =

SSB . b−1

(iii) Find the sum of squares of the totals of each row and divide it by b to obtain k Ti2

i=1

b

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 531

and k Tj2

SST =

i=1

b

− CM

and

MSB =

SST . k −1

(iv) Find the sum of squares of individual observations: b k

yij2

j=1 i=1

Also compute Total SS =

b k

yij2 − CM.

j=1 i=1

(v) Using (ii), (iii), and (iv), ﬁnd SSE = Total SS − SSB − SST

and

MSE =

SSE . n−b−k +1

3. To test the null hypothesis that there is no difference in treatment means: (i) Compute the F -statistic, F=

MST . MSE

(ii) From the F-table, ﬁnd the value of Fα, υ1 , υ2 , where υ1 = (k − 1) is the numerator and υ2 = (n − b − k + 1) the denominator degrees of freedom. (iii) Decision: Reject H0 if F > Fα, υ1 , υ2 and conclude that there is evidence to conclude that there is a difference in treatment means at level α. 4. To test the null hypothesis that there is no difference in the mean response for blocks, (i) Compute the F-statistic, F=

MSB . MSE

(ii) From the F -table, ﬁnd the value of Fα, υ1 , υ2 , where υ1 = (b − 1) is the numerator and υ2 = (n − b − k + 1) the denominator degrees of freedom. (iii) Decision: Reject H0 if F > Fα, υ1 , υ2 and conclude that there is evidence to conclude there is a difference in the mean response for blocks at level α. Assumptions: The samples are randomly selected in an independent manner from n = bk populations. The populations are assumed to be normally distributed with equal variances σ 2 . Also, there are no interactions between the variables (two factors).

532 CHAPTER 10 Analysis of Variance

We have already discussed the assumptions and how to verify those assumptions in one-way analysis. The only new assumption in the randomized blocked design is about the interactions. One of the ways to verify the assumption of no interaction is to plot the observed values against the sample number. If there is no interaction, the line segments (one for each block) will be parallel or nearly parallel; see Figure 9.2. If the lines are not approximately parallel, then there is likely to be interaction between blocks and treatments. In the presence of interactions, the analysis of this section need to be modiﬁed. For details on those procedures, refer to more specialized books on ANOVA methods. We illustrate the randomized block design procedure with the following example.

Example 10.4.1 A furniture company wants to know whether there are differences in stain resistance among the four chemicals used to treat three different fabrics. Table 10.7 shows the yields on resistance to stain (a low value indicates good stain resistance). At the α = 0.05 level of signiﬁcance, is there evidence to conclude that there is a difference in mean resistance among the four chemicals? Is there any difference in the mean resistance among the materials? Give bounds for the p-values in each case.

Table 10.7 Chemical

Material I

II

III

Total

C1

3

7

6

16

C2

9

11

8

28

C3

2

5

7

14

C4

7

9

8

24

Total

21

32

29

82

Solution Here T1 = 16, T2 = 28, T3 = 14, and T4 = 24. Also, B1 = 21, B2 = 32, and B3 = 29. In addition, b = 3, k = 4, and n = bk = 12. Now ⎞2 ⎛ b 1 1 ⎝ ⎠ (82)2 = 560.3333. Bj = CM = n 12 j=1

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 533

We can compute the following quantities: b

SSB =

Bj2

j=1

k

SST =

− CM =

k

MSB =

16.1667 SSB = = 8.0834, b−1 2

− CM =

1812 − 560.3333 = 43.6667, 3

Ti2

i=1

b

2306 − 560.3333 = 16.1667, 4

and MST =

We have

SST 43.6667 = = 14.5556. k−1 3

k b 2 = 632. From this yij

j=1 i=1

Total SS =

b k

2 − CM = 632 − 560.3333 = 71.666 yij

j=1 i=1

SSE = Total SS − SSB − SST = 71.6667 − 16.1667 − 43.6667 = 11.8333 and MSE =

11.8333 SSE = = 1.9722. n−b−k+1 6

The F -statistic is F=

MST 14.5556 = = 7.3804 MSE 1.9722

From the F -table, F0.05,3,6 = 4.76. Because the observed value F = 7.3804 > 4.76, we reject the null hypothesis and conclude that there is a difference in mean resistance among the four chemicals. Because the F -value falls between α = 0.025 and α = 0.01, the p-value falls between 0.01 and 0.025. To test for the difference in the mean resistance among the materials, F=

MSB 8.0834 = = 4.0987. MSE 1.9722

From the F -table, F0.05,2,6 = 5.14. Because the observed value of F = 4.098 < 5.14, we conclude that there is no difference in the mean resistance among the materials. Because the F -value falls between α = 0.10 and 0.05, the p-value falls between 0.05 and 0.10.

534 CHAPTER 10 Analysis of Variance

EXERCISES 10.4 10.4.1.

Show that b k

yij − y

2

j=1 i=1

=

k b

yij − Ti − Bj − y

2

i=1 j=1

+b

k b 2 2 Ti − y + k Bj − y . i=1

j=1

[Hint: Use the identity yij − y = yij − Ti − Bj − y + Ti − y + Bj − y .] 10.4.2.

Show the following: (a) E(MSE) = σ 2 , (b) E(MSB) = (c) E(MST ) =

10.4.3.

k b−1 b k−1

b j=1 k i=1

Bj2 + σ 2 , τi2 + σ 2 .

The least-square estimators of the parameters μ, τi ’s, and βj ’s are obtained by minimizing the sum of squares W=

k b

yij − μ − τi − βj

2

i=1 j=1

with respect to μ, τi ’s, and βj ’s; subject to the restrictions:

k i=1

τi =

b

βj = 0. Show that the

j=1

resultant estimators are μ ˆ = y, τˆi = Ti − y, i = 1, 2, . . . , k,

and βˆ j = Bj − y,

10.4.4.

j = 1, . . . , b.

In order to test the wear on four hyperalloys, a test piece of each alloy was extracted from each of the three positions of a test machine. The reduction of weight in milligrams due to wear was determined on each piece, and the data are given in Table 10.4.1. At α = 0.05, test the following hypotheses, regarding the positions as blocks: (a) There is no difference in average wear for each material. (b) There is no difference in average wear for each position. (c) Interpret your ﬁnal result and state any assumptions that were necessary to solve the problem.

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 535

Table 10.4.1 Loss in Weights Due to Wear Testing of Four Materials (in mg) Position Type of alloy

1

2

3

1

241

270

274

2

195

241

218

3

235

273

230

4

234

236

227

10.4.5.

For the data of Exercise 10.3.10, test at the 0.05 level that the true income lower limits of the top 5% of U.S. households for each race are the same for all 5 years. Also, test at the 0.05 level that the true income lower limits of the top 5% of U.S. households for each year between 1994 and 1998 are the same.

10.4.6.

For the data of Exercise 10.3.11, test at the 0.01 level that the true mean cholesterol levels for all races in the United States during 1978–1980 are the same. Also, test at the 0.01 level that the true mean cholesterol levels for all ages in the United States during 1978–1980 are the same.

10.4.7.

In order to see the effect of hours of sleep on tests of different skill categories (vocabulary, reasoning, and arithmetic), tests consisting of 20 questions each in each category were given to 16 students, four each based on the hours of sleep they had on the previous night. Each right answer is given one point. Table 10.4.2 gives the cumulative scores of the each of the four students in each category.

Table 10.4.2 Hours of sleep

Category Vocabulary

Reasoning

Arithmetic

0

44

33

35

4

54

38

18

6

48

42

43

8

55

52

50

Test at the 0.05 level whether the true mean performance for different hours of sleep is the same. Also, test at the 0.05 level whether the true mean performance for each category of the test is the same.

536 CHAPTER 10 Analysis of Variance

10.5 MULTIPLE COMPARISONS The analysis of variance procedures that we have used so far showed whether differences among several means are signiﬁcant. However, if the equality of means is rejected, the F -test did not pinpoint for us which of the given means or group of means differs signiﬁcantly from another given mean or group of means. With ANOVA, when the null hypothesis of equality of means is rejected, the problem is to see whether there is some way to follow up (post hoc) this initial test H0 : μ1 = μ2 = . . . = μk by looking at subhypotheses, such as H0 : μ1 = μ2 . This involves multiple tests. However, the solution is not to use a simple t-test repeatedly for every possible combination taken two at a time. That, apart from introducing many tests, will considerably increase the signiﬁcance level, the probability of type I error. For example, to test four samples we will need 42 = 6 tests. If each one of the comparisons is tested with the same value of α = P (type I error), and if all the null hypotheses involving six comparisons are true, then the probability of rejecting at least one of them is P(at least one type I error) = 1 − (1 − α)6 .

In particular, if α = 0.01, then P(at least one type I error) = 0.077181, which is signiﬁcantly higher than the original error value of 0.01. One way to investigate the problem is to use a multiple comparison procedure. A good deal of work has been done on problems of multiple comparisons. There are a variety of techniques available in the literature, such as the Bonferroni procedure, Tukey’s method, and Scheffe’s method. We now describe one of the more popular procedures called Tukey’s method for completely randomized, one factor design. In this multiple comparison problem, we would like to test H0 : μi = μj versus Ha : μi = μj , for all i = j. Tukey’s method will be used to test all possible differences of means to decide whether at least one of the differences μi −μj is considerably different from zero. In this comparison problem, Tukey’s method makes use of conﬁdence intervals for μi − μj . If each conﬁdence interval has a conﬁdence level 1 − α, then the probability that all conﬁdence intervals include their respective parameters is less than 1 − α. We now describe this method where each of the k sample means is based on the common number of observations, n. Let N = kn be the total number of observations and let S2 =

k ni =n 2 1 Yij − Ti . N −k i=1 j=1

Let T max

= max T1 , . . . , Tk and T min = min T1 , . . . , Tk . Deﬁne the random variable Q=

T max − T min . √ S n

The distribution of Q under the null hypothesis H0 : μ1 = . . . = μk is called the Studentized range distribution, which depends on the number of samples k and the degrees of freedom υ = N − k = (n − 1)k. We denote the upper α critical value by qα,k,υ . The Studentized range distribution table gives

10.5 Multiple Comparisons 537

values for selected values of k, υ, and α = 0.01, 0.05, and 0.10. The following theorem, due to Tukey, deﬁnes the test procedure. Theorem 10.5.1 Let Ti , i = 1, 2, . . . , k be the k sample means in a completely randomized design. Let μi , i = 1, 2, . . . , k be the true means and let ni = n be the common sample size. Then the probability that all 2k differences μi − μj will simultaneously satisfy the inequalities

s s Ti − Tj − qα,k,υ √ ≤ μi − μj ≤ Ti − Tj + qα,k,υ √ , n n

is (1 − α), where qα,k,υ is the upper α critical value of the Studentized range distribution. If, for a given i and j, zero is not contained in the preceding inequality, H0 : μi = μj can be rejected in favor of Ha : μi = μj , at the signiﬁcance level of α. Now we give a step-by-step approach to implementing Tukey’s method discussed earlier. PROCEDURE TO FIND (1–α)100% CONFIDENCE INTERVALS FOR DIFFERENCE OF MEANS WITH COMMON SAMPLE SIZE N: TUKEY’S METHOD 1. There are k2 comparisons of μi versus μj . 2. Compute the following quantities: ni

Ti =

yij

j=1

ni

, i = 1, 2, . . . , k,

and s2 =

k ni =n 2 1 yij − Ti , where N = kn. N −k i=1 j=1

3. From the Studentized range distribution table, ﬁnd the upper α critical value, qα, k, υ , where υ = N − k = (n − 1)k . 4. For each of k2 pair (i, j), i = j, compute the Tukey’s interval Ti − Tj − qα, k, υ √s , Ti − Tj + qα, k, υ √s . n n 5. Let NR denote insufﬁcient evidence for rejecting H0 . Create the following table for each of k2 pairwise difference μi − μj , i = j, and do not reject if the Tukey interval contains the number 0. Otherwise reject.

Table 10.8 is used to summarize the ﬁnal calculations of the Tukey method. In practice, there are now numerous statistical packages available for Tukey’s purpose. The following example is solved using Minitab. The necessary Minitab commands are given in Example 10.7.3.

538 CHAPTER 10 Analysis of Variance

Table 10.8 μi − μj

Ti − Tj

Tukey interval

Observation

Conclusion

μ1 − μ2

T1 − T2

...

Doesn’t contain 0

Reject

μ1 − μ3

T1 − T3

...

Contains 0

Do not reject

. . .

. . .

. . .

. . .

. . .

Example 10.5.1 Table 10.9 shows the 1-year percentage total return of the top ﬁve stock funds for ﬁve different categories (source: Money, July 2000). Which categories have similar top returns and which are different? Use 95% Tukey’s conﬁdence intervals.

Table 10.9 Large-cap

Mid-cap

Small-cap

Hybrid

Specialty

110.1

299.8

153.8

68.3

181.6

102.9

139.0

139.8

67.1

159.3

93.1

131.2

138.3

42.5

138.3

83.0

110.5

121.4

40.0

132.6

83.3

129.2

135.9

41.0

135.7

Solution For simplicity of computation, we will use SPSS (Minitab steps are given in Example 10.7.2). The following is the output. One-way ANOVA RETURN

Between Groups Within Groups Total

Sum of Squares

df

Mean Square

F

Sig.

41243.698 27877.580 69121.278

4 20 24

10310.925 1393.879

7.397

.001

10.5 Multiple Comparisons 539

Post Hoc Tests Multiple Comparisons Dependent Variable: RETURN Tukey HSD (I) FUND

(J) FUND

Mean

Std. Error

Sig.

Difference

95% Confidence Interval Lower Bound

Upper Bound

(I-J) 1.00

2.00

3.00

4.00

5.00

2.00

−67.4600

23.61253

.066

−138.1175

3.1975

3.00

−43.3600

23.61253

.382

−114.0175

27.2975

4.00

42.7000

23.61253

.396

−27.9575

113.3575

5.00

−55.0200

23.61253

.177

−125.6775

15.6375

1.00

67.4600

23.61253

.066

−3.1975

138.1175

3.00

24.1000

23.61253

.843

−46.5575

94.7575

4.00

110.1600*

23.61253

.001

39.5025

180.8175

5.00

12.4400

23.61253

.984

−58.2175

83.0975

1.00

43.3600

23.61253

.382

−27.2975

114.0175

2.00

−24.1000

23.61253

.843

−94.7575

46.5575

4.00

86.0600*

23.61253

.012

15.4025

156.7175

5.00

−11.6600

23.61253

.987

−82.3175

58.9975

1.00

−42.7000

23.61253

.396

−113.3575

27.9575

2.00

−110.1600*

23.61253

.001

−180.8175

−39.5025

3.00

−86.0600*

23.61253

.012

−156.7175

−15.4025

5.00

−97.7200*

23.61253

.004

−168.3775

−27.0625

1.00

55.0200

23.61253

.177

−15.6375

125.6775

2.00

−12.4400

23.61253

.984

−83.0975

58.2175

3.00

11.6600

23.61253

.987

−58.9975

82.3175

4.00

97.7200*

23.61253

.004

27.0625

168.3775

* The mean difference is significant at the .05 level.

540 CHAPTER 10 Analysis of Variance

Homogeneous Subsets RETURN Tukey HSDa FUND 4.00 1.00 3.00 5.00 2.00 Sig.

N 5 5 5 5 5

Subset for alpha = .05 2

1 51.7800 94.4800

94.4800 137.8400 149.5000 161.9400 .066

.396

Means for groups in homogeneous subsets are displayed. a Uses Harmonic Mean Sample Size = 5.000. The Tukey intervals for pairwise differences (μi − μj ) are in the foregoing computer printout. For example, the Tukey interval for (μ1 − μ2 ) is (−138.1, 3.2) and for (μ2 − μ4 ) is (39.5, 180.8). Also, sample mean and standard deviation are given in the output. For example, 94.48 is the sample mean of the five data points of large-cap funds, and 11.97 is the sample standard deviation of the five data points of large-cap funds. If the Tukey interval for a particular difference (μj − μi ) contains the number zero, we do not reject the H0 : μi = μj . Otherwise, we reject the H0 : μi = μj . For example the interval for (μ4 − μ2 ) is (39.5 − 180.8) and does not contain zero. Hence we reject H0 : μ4 = μ2 . The complete table corresponding to step 5 is produced in Table 10.10, where N.R. represents ‘‘not reject.’’

Table 10.10 μi − μ j

Ti − Tj

Tukey interval

Reject or N.R.

μ1 − μ2

161.94 − 94.48

(−138.1, 3.2)

N.R.

μ1 = μ2

μ1 − μ3

137.84 − 94.48

(−114.0, 27.3)

N.R.

μ1 = μ3

μ2 − μ3

137.84 − 161.94

(−46.6, 94.8)

N.R.

μ3 = μ2

μ1 − μ4

51.78 − 94.48

(−27.9, 113.3)

N.R.

μ4 = μ1

μ2 − μ4

51.78 − 161.94

(39.5, 180.8)

R

μ4 = μ2

μ3 − μ4

51.78 − 137.84

(15.4, 156.7)

R

μ4 = μ3

μ1 − μ5

149.50 − 94.98

(−125.6, 15.6)

N.R.

μ5 = μ1

μ2 − μ5

149.50 − 161.94

(−58.2, 83.1)

N.R.

μ5 = μ2

μ3 − μ5

149.50 − 137.84

(−82.3, 59.0)

N.R.

μ5 = μ3

μ4 − μ5

149.50 − 51.78

(−168.3, −27.1)

R

μ5 = μ4

Conclusion

10.5 Multiple Comparisons 541

Based on the 95% Tukey intervals, the average top return of hybrid funds is different from those for mid-cap, small-cap, and specialty funds. All other returns are similar.

In Tukey’s method, the conﬁdence coefﬁcient for the set of all pairwise comparisons {μi − μj } is exactly equal to 1 − α when all sample sizes are equal. For unequal sample sizes, the conﬁdence coefﬁcient is greater than 1 − α. In this sense, Tukey’s procedure is conservative when the sample sizes are not equal. In the case of unequal sample sizes, one has to estimate the standard deviation for each pairwise comparison. Tukey’s procedure for unequal sample sizes is sometimes referred to as the Tukey–Kramer method.

EXERCISES 10.5 10.5.1.

A large insurance company wants to determine whether there is a difference in the average time to process claim forms among its four different processing facilities. The data in Table 10.5.1 represent weekly average number of days to process a form over a period of 4 weeks.

Table 10.5.1 Facility 1

Facility 2

Facility 3

Facility 4

1.50

2.25

1.30

2.0

0.9

1.85

2.75

1.5

1.12

1.45

2.15

2.85

1.95

2.15

1.55

1.15

(a) Test whether there is a difference in the average processing times at the 0.05 level. (b) Test whether there is a difference, using Tukey’s method to ﬁnd which facilities are different. (c) Interpret your results and state any assumptions you have made in solving the problem. 10.5.2.

Table 10.5.2 gives the rental vacancy rates by U.S. region (source: U.S. Census Bureau) for 5 years.

Table 10.5.2 Rental units

1995

1996

1997

1998

Northeast

7.2

7.4

6.7

6.7

1999 6.3

Midwest

7.2

7.9

8.0

7.9

8.6

South

8.3

8.6

9.1

9.6

10.3

West

7.5

7.2

6.6

6.7

6.2

542 CHAPTER 10 Analysis of Variance

(a) Test at the 0.01 level whether the true rental vacancy rates by region are the same for all 5 years. (b) If there is a difference, use Tukey’s method to ﬁnd which regions are different. 10.5.3.

Table 10.5.3 gives lower limits of income (approximated to nearest $1000 and calculated as of March of the following year) by race for the top 5% of U.S. households from 1994 to 1998. (Source: U.S. Census Bureau.)

Table 10.5.3 Race

1994

1995

1996

1997

All Races

110

113

120

127

1998 132

White

113

117

123

130

136

Black

81

80

85

87

94

Hispanic

82

80

86

93

98

(a) Test at the 0.05 level whether the true lower limits of income for the top 5% of U.S. households for each race are the same for all 5 years. (b) If there is a difference, use Tukey’s method to ﬁnd which is different. (c) Interpret your results and state any assumptions you have made in solving the problem. 10.5.4.

The data in Table 10.5.4 represent the mean serum cholesterol levels (given in milligrams per deciliter) by race and age in the United States from 1978 to 1980 (source: “Report of the National Cholesterol Education Program Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults,” Arch. Intern. Med. 148, Jan. 1988).

Table 10.5.4 Race

Age 20–24

25–34

35–44

45–54

55–64

65–74

All races

180

199

217

227

229

221

White

180

199

217

227

230

222

Black

171

199

218

229

223

217

(a) Test at the 0.01 level whether the true mean cholesterol levels for all races in the United States during 1978–1980 are the same. (b) If there is a difference, use Tukey’s method to ﬁnd which of the races are different with respect to the mean cholesterol levels.

10.7 Computer Examples 543

10.6 CHAPTER SUMMARY In this chapter, we have introduced the basic idea of analyzing various experimental designs. In Section 10.3, we explained the one-way analysis of variance for the hypothesis testing problem for more than two means (different treatments being applied, or different populations being sampled). The two-way analysis of variance, having b blocks and k treatments consisting of b blocks of k experimental units each, is discussed in Section 10.5. We also describe one popular procedure called Tukey’s method for completely randomized, one-factor design for multiple comparisons. We saw in Chapter 9 that there are other possible designs, such as the Latin square design or Taguchi methods. We refer to specialized books on experimental design (Hicks and Turner) for more details on how to conduct ANOVA on such designs. In the ﬁnal section, we give some computational examples. We now list some of the key deﬁnitions introduced in this chapter: ■ ■ ■ ■

Completely randomized experimental design Randomized block design Studentized range distribution Tukey–Kramer method

In this chapter, we also learned the following important concepts and procedures: ■ ■ ■ ■

■

Analysis of variance procedure for two treatments One-way analysis of variance for k ≥ 2 populations One-way analysis of variance procedure for k ≥ 2 populations Procedure to ﬁnd (1 − α)100% conﬁdence intervals for difference of means with common sample size n; Tukey’s method Computational procedure for randomized block design

10.7 COMPUTER EXAMPLES Minitab, SPSS, SAS, and other statistical programming packages are especially useful when we perform an analysis of variance. As we have experienced in earlier sections, an ANOVA computation is very tedious to complete by hand.

10.7.1 Minitab Examples Example 10.7.1 (One-way ANOVA): The three random samples in Table 10.11 are independently obtained from three different normal populations with equal variances. At the α = 0.05 level of signiﬁcance, test for equality of means.

544 CHAPTER 10 Analysis of Variance

Table 10.11 Sample 1

Sample 2

Sample 3

64

56

81

84

74

92

75

69

84

77 80

Solution Enter sample 1 data in C1, sample 2 in C2, and sample 3 in C3. Stat > ANOVA > One-way (unstacked). . . > in Responses (in separate columns): type C1 C2 C3 and click OK We get the following output: One-Way Analysis of Variance Analysis of Variance Source DF SS Factor 2 560.7 Error 8 463.3 Total 10 1024.0

Level C1 C2 C3

N 5 3 3

Pooled StDev =

MS 280.3 57.9

F 4.84

P 0.042

Individual 95% CIs For Mean Based on Pooled StDev Mean StDev ----+---------+---------+---------+-76.000 7.517 (-----*------) 66.333 9.292 (-------*--------) 85.667 5.686 (-------*--------) -----+---------+--------+---------+-7.610 60 72 84 96

We can see that the output contains, SS, MS, individual column means, and standard deviation values. Also, the F -value gives the value of the test statistic, and the p-value is obtained as 0.042. Comparing this p-value of 0.042 with α = 0.05, we will reject the null hypothesis. If we want to create side-by-side box plots to graphically test homogeneity of variances, we can do the following.

10.7 Computer Examples 545

Enter all the data (from all three samples) in C1, and enter the sample identifier number in C2 (that is, 1 if the data belong to sample 1, 2 for sample 2, and 3 for sample 3). Graph > Boxplot > in Y column, type C1 and in X column, type C2 > click OK Then as in Example 10.3.2, interpret the resulting box plots.

Example 10.7.2 Give Minitab steps for randomized block design for the data of Example 10.4.1.

Solution To put the data into the format for Minitab, place all the data values in one column (say, C2). Let numbers 1, 2, 3, 4 represent the chemicals and numbers 1, 2, 3 represent the fabric material. In one column (say, C1) place numbers 1 through 4 with respect to the data values identifying the factor (chemical) used. In another column (say, C3) place corresponding numbers 1 through 3 to identify the second factor (material) used. See Table 10.12.

Table 10.12 C1 chemical

C2 response

C3 material

1

3

1

2

9

1

3

2

1

4

7

1

1

7

2

2

11

2

3

5

2

4

9

2

1

6

3

2

8

3

3

7

3

4

8

3

546 CHAPTER 10 Analysis of Variance

Then do the following: Stat > ANOVA > Two-way. . . > in Response: type C2, in Row Factor: type C1, and in Column factor: type C3 > OK We will get the following output. Two-Way Analysis of Variance Analysis of Variance for Response Source DF SS MS F Chemical 3 43.67 14.56 7.38 Material 2 16.17 8.08 4.10 Error 6 11.83 1.97 Total 11 71.67

P 0.019 0.075

Note that the output contains p-values for the effect both of the chemicals and of the materials. Because the p-value of 0.019 is less than α = 0.05, we reject the null hypothesis and conclude that there is a difference in mean resistance among the four chemicals. For the materials, the p-value of 0.075 is greater than α = 0.05, so we cannot reject the null hypothesis and conclude that there is no difference in the mean resistance among the materials.

Example 10.7.3 Give the Minitab steps for using Tukey’s method for the data of Example 10.5.1.

Solution In order to use Tukey’s method, it is necessary to enter the data in a particular way. Enter all the data points in column C1; first five from large-cap, next five from mid-cap, and so on, with the last five from specialty. In column C2, enter the number identifying the data points; the first four numbers are 1 (identifying 1 as the data belonging to large-cap), next five numbers are 2, and so on; the last five numbers are 5. Then: Stat > ANOVA > One-way. . . > Comparisons. . . > click Tukey’s, family error rate: and type 5 (to represent 100α% error) > OK > in Response: type C1, and in Factor: type C2 > OK We will get the output similar to that given in the solution part of Example 10.5.1. For discussion of the output, refer to Example 10.5.1.

10.7.2 SPSS Examples Example 10.7.4 Conduct a one-way ANOVA for the data of Example 10.7.1. Use α = 0.05 level of signiﬁcance, and test for equality of means.

10.7 Computer Examples 547

Solution In SPSS, we need to enter the data in a special way. First name column C1 as Sample, and column C2 as Values. In the Sample column, enter the numbers to identify from which group the data comes. In this case, enter 1 in the first five rows, 2 in the next three rows, and 3 in the last three rows. In the Values column, enter sample 1 data in the first five rows, sample 2 data in the next five rows, and sample 3 data in the last three rows. Then: Analyze > Compare Means > One-way ANOVA. . . > Bring Values to Dependent List: and Sample to Factor: > OK We will get the following output. ANOVA VALUES Between Groups Within Groups Total

Sum of Squares 560.667 463.333 1024.000

df 2 8 10

Mean Square 280.333 57.917

F 4.840

Sig. .042

Because Sig. Value 0.042 is less than α = 0.05, we reject the null hypothesis.

Example 10.7.5 Give the SPSS steps for using Tukey’s method for the data of Example 10.5.1.

Solution First name column C1 as Fund and column C2 as Return. In the Fund column, enter the numbers to identify from which group the data comes. In this case, the first four numbers are 1 (identifying 1 as the data belonging to large-cap), the next four numbers are 2, and so on, until the last four numbers are 5. In the Return column, enter large-cap return data in the first four rows, mid-cap data in the next four rows, and so on; the last four from speciality. Then: Analyze > Compare Means > One-way ANOVA. . . > Bring Return to Dependent List: and Fund to Factor: > Click Post-Hoc. . . > click Tukey > click Continue > OK

We will get the output as in Example 10.5.1. Interpretation of output is given in Example 10.5.1. When the treatment effects are significant, as in this example where the p-value is 0.001, the means must then be further examined to determine the nature of the effects. There are procedures called post hoc tests to assist the researcher in this task. For example, looking at the output column Sig., we could observe that there are significant differences in the mean returns between funds 2 and 4, and funds 4 and 5.

548 CHAPTER 10 Analysis of Variance

10.7.3 SAS Examples Example 10.7.6 Using SAS, conduct a one-way ANOVA for the data of Example 10.7.1. Use α = 0.05 level of signiﬁcance, and test for equality of means.

Solution We could use the following code. Options nodate nonumber; options ls=80 ps=50; DATA Scores; INPUT Sample Value @@; DATALINES; 1 64 1 84 1 75 1 77 1 80 2 56 2 74 2 69 3 81 3 92 3 84 ; PROC ANOVA DATA=Scores; TITLE ’ANOVA for Scores’; CLASS Sample; MODEL Value=Sample; MEANS Sample; RUN; We will get the following output: ANOVA for Scores The ANOVA Procedure Class Level Information Class

Levels

Sample

Values

3

Number of observations

1 2 3 11

The ANOVA Procedure Dependent Variable: Value

Source

DF

Model

2

Sum of Squares 560.666667

Mean Square 280.333333

F Value

Pr > F

4.84

0.0419

10.7 Computer Examples 549

Error Corrected Total

8

463.333333

10

1024.000000

57.916667

R-Square

Coeff Var

Root MSE

Value Mean

0.547526

10.01355

7.610300

76.00000

Source

DF

Sample

2

Anova SS 560.6666667

Mean Square

F Value

280.3333333

4.84

Pr > F 0.0419

The ANOVA Procedure Level of Sample

------------Value-----------Mean Std Dev

N

1 2 3

5 3 3

76.0000000 66.3333333 85.6666667

7.51664819 9.29157324 5.68624070

Because the p-value 0.0419 is less than α = 0.05, we reject the null hypothesis. We could have used PROC GLM instead of PROC ANOVA to perform the ANOVA procedure. Usually, PROC ANOVA is used when the sizes of the samples are equal; otherwise PROC GLM is more desirable. The next example will show how to do the multiple comparison using Tukey’s procedure.

Example 10.7.7 Give the SAS commands for using Tukey’s method for the data of Example 10.5.1.

Solution We could use the following code. Options nodate nonumber; options ls=80 ps=50; DATA Mfundrtn; INPUT Fund Return @@; DATALINES; 1 110.1 2 299.8 1 102.9 2 139.0 1 93.1 2 131.2 1 83.3 2 129.2 1 83.0 2 110.5 ;

3 3 3 3 3

153.8 139.8 138.3 135.9 121.4

4 4 4 4 4

68.3 67.1 42.5 41.0 40.0

5 5 5 5 5

181.6 159.3 138.3 135.7 132.6

550 CHAPTER 10 Analysis of Variance

PROC GLM DATA=Mfundrtn; TITLE ’ANOVA for Mutual fund returns’; CLASS Fund; MODEL Return=Fund; MEANS Fund / tukey; RUN; ANOVA for Mutual fund returns The GLM Procedure Class Level Information Class

Levels

Fund

5

Values 1 2 3 4 5

Number of observations

25

ANOVA for Mutual fund returns The GLM Procedure Dependent Variable: Return Source

Sum of Squares

DF

Mean Square

Model

4

Error

20

27877.58000

Corrected Total

24

69121.27840

Source Fund Source Fund

41243.69840

10310.92460

Coeff Var

Root MSE

0.596686

31.34524

37.33469

Type I SS

4

41243.69840

DF

Type III SS

4

Mean Square 10310.92460 Mean Square

41243.69840

Pr > F

7.40

0.0008

1393.87900

R-Square

DF

F Value

10310.92460

Return Mean 119.1080 F Value 7.40

Pr > F 0.0008

F Value

Pr > F

7.40

0.0008

ANOVA for Mutual fund returns The GLM Procedure Tukey’s Studentized Range (HSD) Test for Return

10.7 Computer Examples 551

NOTE: This test controls the Type I experiment wise error rate, but it generally has a higher Type II error rate than REGWQ.

Alpha 0.05 Error Degrees of Freedom 20 Error Mean Square 1393.879 Critical Value of Studentized Range 4.23186 Minimum Significant Difference 70.658 Means with the same letter are not significantly different. Tukey Grouping

Mean

N

A A A A A A

161.94

5

2

149.50

5

5

137.84

5

3

B B B

A

Fund

94.48

5

1

51.78

5

4

The GLM Procedure Tukey’s Studentized Range (HSD) Test for Value NOTE: This test controls the Type I experiment wise error rate, but it generally has a higher Type II error rate than REGWQ. Alpha 0.05 Error Degrees of Freedom 20 Error Mean Square 1393.879 Critical Value of Studentized Range 4.23186 Minimum Significant Difference 70.658 Means with the same letter are not significantly different. Tukey Grouping

B B B

A A A A A A A

Mean

N

Sample

161.94

5

2

149.50

5

5

137.84

5

3

94.48

5

1

51.78

5

4

552 CHAPTER 10 Analysis of Variance

Looking at the p-value of 0.008, which is less than α = 0.05, we conclude that there is a difference in mutual fund returns. In the previous example, we used the post hoc test Tukey. We could have used other options such as DUNCAN, SNK, LSD, and SCHEFFE. The test is performed at the default value of α = 0.05. If we want to specify, say, α = 0.01, or 0.1, we could have done so by using the command MEANS Fund / Tuckey ALPHA=0.01;. If we need all the confidence intervals in the Tukey method, in the code just given, we have to modify ‘MEANS Fund / Tukey;’ to ‘MEANS Fund / LSD TUKEY CLDIFF;’ which will result in the following output. ANOVA for Mutual fund returns The GLM Procedure Class Level Information Class

levels

Fund

5

Values 1 2 3 4 5

Number of observations

25

ANOVA for Mutual fund returns The GLM Procedure Dependent Variable: Return

Source

DF

Model

4

Error

20

Sum of Squares 41243.69840

Fund Source Fund

7.40

Pr > F 0.0008

1393.87900

69121.27840

R-Square

Coeff Var

0.596686

31.34524

DF

F Value

10310.92460

27877.58000

Corrected Total 24

Source

Mean Square

Root MSE 37.33469

Return Mean

Type I SS

Mean Square

4

41243.69840

10310.92460

DF

Type III SS

Mean Square

4

41243.69840

10310.92460

119.1080 F Value 7.40 F Value 7.40

Pr > F 0.0008 Pr > F 0.0008

10.7 Computer Examples 553

ANOVA for Mutual fund returns The GLM Procedure t-tests (LSD) for Return NOTE: This test controls the Type I comparisonwise error rate, not the experiment wise error rate.

Alpha Error Degrees of Freedom Error Mean Square Critical Value of t Least Significant Difference

0.05 20 1393.879 2.08596 49.255

Comparisons significant at the 0.05 level are indicated by ***. Difference Fund Comparison

Between Means

95% Confidence Limits

2 2 2 2 5 5 5 5 3 3 3 3 1 1 1 1 4 4 4 4

12.44 24.10 67.46 110.16 –12.44 11.66 55.02 97.72 –24.10 –11.66 43.36 86.06 –67.46 –55.02 –43.36 42.70 –110.16 –97.72 –86.06 –42.70

–36.81 –25.15 18.21 60.91 –61.69 –37.59 5.77 48.47 –73.35 –60.91 –5.89 36.81 –116.71 –104.27 –92.61 –6.55 –159.41 –146.97 –135.31 –91.95

– – – – – – – – – – – – – – – – – – – –

5 3 1 4 2 3 1 4 2 5 1 4 2 5 3 4 2 5 3 1

61.69 73.35 116.71 159.41 36.81 60.91 104.27 146.97 25.15 37.59 92.61 135.31 –18.21 –5.77 5.89 91.95 –60.91 –48.47 –36.81 6.55

*** *** *** ***

*** *** *** *** *** ***

ANOVA for Mutual fund returns The GLM Procedure Tukey’s Studentized Range (HSD) Test for Return NOTE: This test controls the Type I experiment wise error rate.

554 CHAPTER 10 Analysis of Variance

Alpha Error Degrees of Freedom Error Mean Square Critical Value of Studentized Range Minimum Significant Difference

0.05 20 1393.879 4.23186 70.658

Comparisons significant at the 0.05 level are indicated by ***. Difference Fund Comparison 2 2 2 2 5 5 5 5 3 3 3 3 1 1 1 1 4 4 4 4

– – – – – – – – – – – – – – – – – – – –

5 3 1 4 2 3 1 4 2 5 1 4 2 5 3 4 2 5 3 1

Between Means 12.44 24.10 67.46 110.16 –12.44 11.66 55.02 97.72 –24.10 –11.66 43.36 86.06 –67.46 –55.02 –43.36 42.70 –110.16 –97.72 –86.06 –42.70

Simultaneous 95% Confidence Limits –58.22 –46.56 –3.20 39.50 –83.10 –59.00 –15.64 27.06 –94.76 –82.32 –27.30 15.40 –138.12 –125.68 –114.02 –27.96 –180.82 –168.38 –156.72 –113.36

83.10 94.76 138.12 180.82 58.22 82.32 125.68 168.38 46.56 59.00 114.02 156.72 3.20 15.64 27.30 113.36 –39.50 –27.06 –15.40 27.96

***

***

***

*** *** ***

EXERCISES 10.7 10.7.1.

For the data of Exercise 10.5.4, perform a one-way analysis of variance using any of the software (Minitab, SPSS, or SAS).

10.7.2.

For the data of Exercise 10.5.2, perform Tukey’s test using any of the software (Minitab, SPSS, or SAS).

10.7.3.

For the data of Exercise 10.5.4, perform Tukey’s test using any of the software (Minitab, SPSS, or SAS).

PROJECTS FOR CHAPTER 10 10A. Transformations The basic model for the analysis of variance requires that the independent observations come from normal populations with equal variances. These requirements are rarely met in practice, and the extent to which they are violated affects the validity of the subsequent inference. Therefore, it is important

Projects for Chapter 10 555

for the investigator to decide whether the assumptions are at least approximately satisﬁed and, if not, what can be done to rectify the situation. Hence it is necessary to (a) examine the data for marked departures from the model and, if necessary, (b) apply an appropriate transformation to the data to bring it more in line with the basic assumptions. A simple way to check for the equality of the population variances is to calculate the sample variances and plot against mean as in Figure 10.3. If the graph suggests a relation between sample mean and variance, then the relation very likely exists between population mean and variance, and hence the population from which the samples are taken may very well be nonnormal. If a study of sample means and variances reveals a marked departure from the model, the observations may be transformed into a new set to which the methods of ANOVA are better suited. Three commonly used transformations are the following: (a) The logarithmic transformation: If the graph of sample means against sample variance suggests a relation of the form 2 s2 = C X ,

replace each observation X by its logarithm to the base 10, Y = log 10 X;

or, if some X-values are zero, by Y = log 10 (X + 1). (b) The square root transformation: If the relation is of the form s2 = CX

replace X by its square root, Y=

√ X

or, if the values of X are very close to zero, by the square root of X + 1/2 . This relation is found in data from Poisson populations, where the variance is equal to the mean. (c) The angular transformation: If the observations are counts of a binomial nature, and pˆ is the observed proportion, replace pˆ by θ = arcsin

p, ˆ

which is the principal angle (in degrees or radians) whose sine is the square root of p. ˆ (i) To check for the equality of the population variances, calculate the sample variances for each of the data sets given in the exercises of Section 10.3 and plot against the corresponding mean. (ii) If there is assumptional violation, perform one of the transformations described earlier and do the analysis of variance procedure for the transformed data.

556 CHAPTER 10 Analysis of Variance

10B. Anova with Missing Observations In the two-way analysis of variance, we assumed that each block cell has one treatment value. However, it is possible that some observations in some block cells may be missing for various reasons, such as that the investigator failed to record the observations, the subject discontinued participation in the experiment, or the subject moved to a different place or died prior to completion of the experiment. In those cases, this project gives a method of inserting estimates of the missing values. Let y.. denote the total of all kb observations. If the observation corresponding to the ith row and the jth column, which is denoted by yij ., is missing, then all the sums of squares are calculated as before, except that the yij term is replaced by yˆ ij =

bBj + kTi − y .. (k − 1) (b − 1)

,

where Ti denotes the total of b−1 observations in the ith row, Bj denotes the total of k−1 observations in the jth column, and y .. denotes the sum of all kb − 1 observations. Using calculus, one can show that yˆ ij minimizes the error sum of squares. One should not include these estimates when computing relevant degrees of freedom. With these changes, proceed to perform the analysis as in Section 10.4. For more details on the method, refer to Sahai and Ageel (2000), p. 145. Perform the test of Example 10.4.1, now with a missing value for material III and chemical C4 . Does the conclusion change?

10C. ANOVA in Linear Models In order to determine whether the multiple regression model introduced in Section 8.5 is adequate for predicting values of dependent variable y, one can use the analysis of variance F -test. The model is Y = β0 + β1 x1 + β2 x2 + · · · + βk xk + ε,

where ε = (ε1 , ε2 , . . . , εn ) ∼ N 0, σ 2 and εi and εj are uncorrected if i = j. Deﬁne the multiple coefﬁcient of determination, R2 , by 2 yi − yˆ i 2 R =1− . (yi − y)2

The Analysis of Variance F-Test H0 : β1 = β2 = . . . = βk = 0 versus Ha : At least one of the parameters, β1 , β2 , . . . , βk , differs from 0.

Projects for Chapter 10 557

Test statistic: Mean square for model Mean square for error SS model /k = SSE/[n − (k + 1)] > R2 k = , > 1 − R2 [n − (k + 1)]

F=

where n = number of observations k = number of parameters in the model excluding β0 .

From the F -table, determine the value of Fα with k numerator d.f. and n − (k + 1) denominator d.f. Then the rejection region is {F > Fα }. If we reject the null hypothesis, then the model can be taken as useful in predicting values of y. For the data of Example 8.5.1, test the overall utility of the ﬁtted model y = 66.12 − 0.3794X1 + 21.4365X2

using the F -test described earlier.

This page intentionally left blank

Chapter

11

Bayesian Estimation and Inference Objective: To study Bayesian analysis methods and procedures that are becoming very popular in building statistical models for real-world problems. 11.1 Introduction 560 11.2 Bayesian Point Estimation 562 11.3 Bayesian Conﬁdence Interval or Credible Intervals 11.4 Bayesian Hypothesis Testing 584 11.5 Bayesian Decision Theory 588 11.6 Chapter Summary 596 11.7 Computer Examples 596 Projects for Chapter 11 596

579

The Reverend Thomas Bayes (Source: http:en.wikipedia.org/wiki/Thomas_Bayes)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

559

560 CHAPTER 11 Bayesian Estimation and Inference

The Reverend Thomas Bayes (1702–1761) was a Nonconformist minister. In the 1720s Bayes started working on the theory of probability. Even though he did not publish any of his works on mathematics during his lifetime, Bayes was elected a Fellow of the Royal Society in 1742. His famous work titled “Essay toward solving a problem in the doctrine of chances” was published in the Philosophical Transactions of the Royal Society of London in 1764, after his death. The paper was sent to the Royal Society by Richard Price, a friend of Bayes. Another mathematical publication on asymptotic series also appeared after his death.

11.1 INTRODUCTION Bayesian procedures are becoming increasingly popular in building statistical models for real-world problems. In recent years, the Bayesian statistical methods have been increasingly used in scientiﬁc ﬁelds ranging from archaeology to computing. Bayesian inference is a method of analysis that combines information collected from experimental data with the knowledge one has prior to performing the experiment. Bayesian and classical (frequentist) methods take basically different outlooks toward statistical inference. In this approach to statistics, the uncertainties are expressed in terms of probabilities. In the Bayesian approach, we combine any new information that is available with the prior information we have, to form the basis for the statistical procedure. The classical approach to statistical inference that we have studied so far is based on the random sample alone. That is, if a probability distribution depends on a set of parameters θ, the classical approach makes inferences about θ solely on the basis of a sample X1 , . . . , Xn . This approach to inference is based on the concept of a sampling distribution. To correctly interpret traditional inferential procedures, it is necessary to fully understand the notion of a sampling distribution. In this approach, we analyze only one set of sample values. However, we have to imagine what could happen if we drew a large number of random samples from the population. For example, consider a normal sample with known variance. We have seen that a 95% conﬁdence interval for the population mean μ is given by the random √ √ interval X − 1.96σ/ n, X + 1.96σ/ n . This means that when samples are repeatedly taken from the population, at least 95% of the random intervals contain the true mean μ. The classical inferential approach does not use any of the prior information we might have as a result of, say, our familiarity with the problem, or information from earlier studies. Scientists and engineers are faced with the problem that there is typically only a single data set, and they need to determine the value of the parameter at the time the data are taken. The basic question then is, “What is the best estimate of a parameter one can make from the data using one’s prior information?” Statistical approaches that use prior knowledge, possibly subjective, in addition to the sample evidence to estimate the population parameters are known as Bayesian methods. Bayesian statistics provides a natural method for updating uncertainty in the light of evidence. Data are still assumed to come from a distribution belonging to a known parametric family. However, the Bayesian outlook toward inference is founded on the subjective interpretation of probability. Subjective probability is a way of stating our belief in the validity of a random event. The following example will illustrate the idea. Suppose we are interested in the proportion of all undergraduate students at a particular university who take on out-of-campus jobs for at least 20 hours a week. Suppose we randomly select, say, 50 students from this university and obtain the proportion of

11.1 Introduction 561

students who have out-of-campus jobs for at least 20 hours a week. Let us assume that the sample proportion is 30/50 = 0.6. In a frequentist approach, all of the inferential procedures, such as point estimation, interval estimation, or hypothesis testing, are based on the sampling distribution. That is, even though we are analyzing only one data set, it is necessary to have the knowledge of the mean, standard deviation, and shape of this sampling distribution of the proportion for the correct interpretation in classical inferential procedures. In the subjective interpretation of probability, the proportion of undergraduates who work on an out-of-campus job for at least 20 hours a week is assumed to be unknown and random. A probability distribution, called the prior, that represents our knowledge or belief about the location of this proportion before any data collected is used. For instance, the college placement ofﬁce already may have an opinion on this proportion based on its earlier experience. The classical approach ignores this prior knowledge, whereas the Bayesian approach incorporates this knowledge with the current observed data to update the value of this proportion. That is, after the data are collected our opinion about the proportion may change. Using Bayes’ rule, we will compute the posterior probability distribution for the proportion, based on our prior belief and evidence from the data. All of our inferences about the proportion are made by computing appropriate statistics of the posterior distribution. The Bayesian approach seeks to optimally merge information from two sources: (1) knowledge that is known from theory or opinion formed at the beginning of the research in the form of a prior, and (2) information contained in the data in the form of likelihood functions. Basically, the prior distribution represents our initial belief, whereas the information in the data is expressed by the likelihood function. Combining prior distribution and likelihood function, we can obtain the posterior distribution. This expresses our revised uncertainty in light of the data. The main difference between the Bayesian approach and the classical approach is that in the Bayesian setting, the parameter is viewed as random variables, whereas the classical approach considers the parameter to be ﬁxed but unknown. The parameter is random in the sense that we can assign to it a subjective probability distribution that describes our conﬁdence about the actual value of the parameter. Some of the reasons for Bayesian approaches are as follows: (1) Most Bayesian inferential conclusions are made conditional on the observed data. Unlike the traditional approach, one need not be concerned with data sets other than the one that is observed. There is no need to discuss sampling distributions using the Bayesian approach. Also, (2) from a Bayesian viewpoint, it is legitimate to talk about the probability that the proportion falls in a speciﬁc interval, say (0.2, 0.6), or the probability that a hypothesis is true. Too often, traditional inferential conclusions are misstated; for example, if a conﬁdence interval computed from a sample for a parameter is (0.2, 0.6), it is common for the student to incorrectly state that the population parameter falls in the interval (0.2, 0.6) with probability at least 0.90. The Bayesian viewpoint provides a convenient model for implementing the scientiﬁc method. The prior probability distribution can be used to state initial beliefs about the population of interest, relevant sample data are collected, and the posterior probability distribution reﬂects one’s new updated beliefs about the population parameter in light of the new data that were collected. All inferences about the parameter are made by computing appropriate summaries of the posterior probability distribution. Because of formidable theoretical and computational challenges, the Bayesian approach has found relatively limited use. Recent advances in Bayesian analysis combined with the

562 CHAPTER 11 Bayesian Estimation and Inference

growing power of computers are making Bayesian methods practical and increasingly popular. The Markov chain Monte Carlo (MCMC) method described in Section 13.5 is one of the computationally intensive methods that is often useful in Bayesian estimation.

11.2 BAYESIAN POINT ESTIMATION The cornerstone of Bayesian methodology is the Bayes theorem. It helps us to update our beliefs in the form of probability statements about the parameters after the sample has been taken. The conditional distribution of the parameters after observing the data is called the posterior distribution that integrates the prior and the sample information. Suppose we have two discrete random variables, X and Y . Then the joint probability function (pmf ) can be written as p(x, y) = p(x |y)pY (y) , and the marginal probability density function of X is pX (x) = y p(x, y) = y p(x |y)pY (y) . Then Bayes’ rule for the conditional p(y |x ) is p (y |x) =

p (x, y) p (x |y) pY (y) p (x |y) pY (y) = = . pX (x) pX (x) p (x |y) pY (y) y

The denominator in this expression is a ﬁxed normalizing factor that ensures that the If Y is continuous, the Bayes theorem can be stated as p (y |x ) =

y

p (y |x ) = 1.

p (x |y) pY (y) , p (x |y) pY (y) dy

where the integral is over the range of values of y. These two equations are the Bayes formulas for random variables. In Bayesian terminology, pY (y) represents the probability statement of our prior belief, p(x|y) is the probability of the data x given our prior beliefs, which is called the likelihood, and the updated probability p(y|x) is the posterior. Because pX (x) (which is the likelihood accumulated over all possible prior values) is independent of y, we can express the posterior distribution as proportional (∝) to [(likelihood) × (prior distribution)], that is, p(y|x) ∝ p(x|y)p(y).

We use the notation f (x|θ) to represent a probability distribution whose population parameter is considered to be a random variable. Now one of the problems is of ﬁnding a point estimate of the parameter θ (possibly a vector) for the population with distribution f (x|θ), given θ. Assume that π(θ) is the prior distribution of θ, which reﬂect the experimenter’s prior belief about θ. We will not distinguish between the scalars and vectors, which will be clear based on the speciﬁc situation. Suppose that we have a random sample X = (X1 , . . . , Xn ) of size n from f (x|θ). Then the posterior distribution can be written as f (θ|X1 , . . . , Xn ) =

L(X1 , . . . , Xn |θ)π(θ) f (θ, X1 , . . . , Xn ) = , f (X1 , . . . , Xn ) f (X1 , . . . , Xn )

11.2 Bayesian Point Estimation 563

where L(X1 , . . . , Xn |θ) is the likelihood function. Letting C represent all terms that do not involve θ (in this case, C = 1/f (X1 , . . . , Xn )), we have f (θ |X1 , . . . , Xn ) = CL(X1 , . . . , Xn |θ )π(θ),

For speciﬁc sample values X1 = x1 , X2 = x2 , . . . , Xn = xn , the foregoing equation can be written in a compact form as f (θ |x ) ∝ f (x |θ )π(θ),

where

x = (x1 , x2 , . . . , xn ).

This can be expressed as

posterior distribution ∝ prior distribution × likelihood .

The full result including the normalization can be written as (posterior distribution) = [(prior distribution) × (likelihood)] /

(prior × likelihood)

where the denominator is a ﬁxed normalizing factor obtained by the likelihood accumulated over all possible prior values. We can now give a formal deﬁnition. Deﬁnition 11.2.1 The distribution of θ, given data x1 , x2 , . . . , xn , is called the posterior distribution, which is given by π(θ |x ) =

f (x |θ )π(θ) , g(x)

(11.1)

where g (x) is the marginal distribution of X. The Bayes estimate of the parameter θ is the posterior mean. The marginal distribution g(x) can be calculated using the formula

g(x) =

⎧ ⎪ f (x|θ)π(θ), ⎪ ⎪ ⎨θ

in discrete case

∞ ⎪ ⎪ ⎪ f (x|θ)π(θ)dθ, ⎩

in continuous case

−∞

where π(θ) is the prior distribution of θ. Here, the marginal distribution g (x) is also called the predictive distribution of X, because it represents our current predictions of the values of X taking into account both the uncertainty about the value of θ and the residual uncertainty about the random variable X when θ is known. In a Bayesian setting, all the information about θ from the observed data and from the prior knowledge is contained in the posterior distribution, π(θ|x). In almost all practical cases, because we are combining our prior information with the information contained in the data, the posterior distribution provides a more reﬁned estimation of θ than the prior. All inferences from Bayesian methods are based on the posterior probability distribution of the parameter θ. Using the explanation given later, we will take the Bayes estimate of a parameter as the posterior mean.

564 CHAPTER 11 Bayesian Estimation and Inference

Furthermore, consider a Bayesian statistical inference problem where the parameter is a population proportion. In the Bernoulli trials, the population contains two types called “successes” and “failures.” The proportion of successes in the population is denoted by θ. We take a random sample of size n from the population and observe s successes and f failures. The goal is to learn about the unknown proportion θ on the basis of these data. In this situation, a model is represented by the population proportion θ. We do not know its value. In Chapter 5, we have seen that we could use the maximum likelihood estimator (MLE) for estimating θ, which did not use any prior knowledge we may have about θ. Note that the maximum likelihood estimate is broadly equivalent to ﬁnding the mode of the likelihood. In a Bayesian setting, we represent our beliefs about location of θ in terms of a prior probability distribution. We introduce proportion inference by using a discrete prior distribution for θ. We can construct a prior by specifying a list of possible values for the proportion θ, and then assigning probabilities to these values that reﬂect our knowledge about θ. Then the posterior probabilities can be computed using the Bayes theorem. The following example illustrates this concept.

Example 11.2.1 It is believed that cross-fertilized plants produce taller offspring than the self-fertilized plants. In order to obtain an estimate on the proportion θ of cross-fertilized plants that are taller, an experimenter observes a random sample of 15 pairs of plants that are exactly the same age. Each pair is grown in the same conditions with some cross-fertilized and the others self-fertilized. Based on previous experience, the experimenter believes that the following are possible values of θ and that the prior probability for each value of θ (prior weight) is π(θ). θ: π(θ):

0.80 0.13

0.82 0.15

0.84 0.22

0.86 0.25

0.88 0.15

0.90 0.10

From the experiment, it is observed that in 13 of 15 pairs, cross-fertilized is taller. Create a table with columns of the prior π(θ), likelihood of L(X1 , X2 , . . . , Xn |θ) for different values of θ and for the given sample, prior times likelihood, and posterior probability of θ. Based on the posterior probabilities, what value of θ has the highest support? Also, ﬁnd E(θ) based on the posterior probabilities.

Solution The likelihood of obtaining 13 of 15 taller plants to the different prior values of π are given using the binomial 15 pdf θ 13 (1 − θ)2 . For example, if the prior value of θ is 0.80, then the likelihood of θ given the 13 sample is

15 f (x|θ) = (0.8)13 (0.2)2 = 0.2309. 13

11.2 Bayesian Point Estimation 565

Table 11.1 Prior values Prior Likelihood of θ Prior times Posterior of θ π(θ) given sample likelihood probability of θ 0.80

0.13

0.2309

3.0017×10−2

0.11029

0.82

0.15

0.2578

0.03867

0.14208

0.2787

6.1314×10−2

0.22528 0.2661

0.84

0.22

0.86

0.25

0.2897

7.2425×10−2

0.88

0.15

0.2870

0.4305

0.15817

0.90

0.10

0.2669

0.02669

0.098064

0.27217

0.9998 ≈ 1.0

Total

From Table 11.1 we obtain (prior × likelihood) = 0.27217. Hence, the normalized value corresponding to θ = 0.80 is the posterior probability f (θ|x), which is equal to (0.030017/0.27217) = 0.11029. Now, we can obtain the table of posterior distribution ofa proportion π using the discrete prior given in Table 11.1. 15 When we substitute in Bayes’ rule, the factor would be canceled. Hence, in the calculation of the 13 15 13 13 2 θ (1 − θ)2 . likelihood function, we could have just used θ (1 − θ) instead of the full expression 13 Thus, the Bayesian estimate of θ is E(θ) = (0.8)(0.11029) + (0.82)(0.14028) + (0.84)(0.22528) + (0.86)(0.2661) + (0.88)(0.15817) + (0.9)(0.098065) = 0.84879 ≈ 0.85. It may be noted that the MLE of θ is 13/15 = 0.867.

In Example 11.2.1, the priors are called informative priors, because it favored certain values of θ; for example for the value θ = 0.86, the prior value of π (θ) is 0.25, which is higher than all the rest of the values. If there was no information or no strong prior opinions, then we could select a noninformative prior, which would have assigned equal prior probability of 1/6 to each of the possible values of θ. A noninformative prior (also called a ﬂat or uniform prior) provides little or no information. Based on the situation, noninformative priors may be quite disperse, may avoid only impossible values of the parameter, and oftentimes give results similar to those obtained by classical frequentist methods.

566 CHAPTER 11 Bayesian Estimation and Inference

Example 11.2.2 Repeat the Example 11.2.1 using a noninformative prior, π(θ) = 1/6, for each given value of θ.

Solution Here π(θ) = 16 for each value of θ. See Table 11.2.

Table 11.2 Prior values of θ

Prior π(θ)

Likelihood of θ given sample

Prior times likelihood

Posterior probability of θ

0.80

1/6

0.2309

3.8483×10−2

0.14333

0.82

1/6

0.2578

4.2967×10−2

0.16003

0.84

1/6

0.2787

0.04645

0.86

1/6

0.2897

4.8283×10−2

0.17982

0.88

1/6

0.2870

4.7833×10−2

0.17815

0.90

1/6

0.2669

4.4483×10−2

0.16567

0.2685

1.0

Total

0.173

The Bayesian estimate for the noninformative prior is E(θ) = (0.8)(0.14333) + (0.82)(0.16003) + (0.84)(0.173) + (0.86)(0.17982) + (0.88)(0.17815) + (0.9)(0.16567) = 0.85173.

It should be noted that because the choice of priors in Example 11.2.1 is only mildly informative, we do not see much difference in the values of Bayesian estimates. In general, it is difﬁcult to construct an acceptable prior, because most often it has to be based on subjective experiences. Therefore, it is relatively easy to use a “noninformative” prior. For example, if we have no information on the values of proportion θ, then one type of standard “noninformative” prior is to take the proportion θ as one of the equally spaced values 0, 0.1, 0.2, . . . , 0.9, 1. We can assign for each value of θ the same probability, π(θ) = 1/11. This prior is convenient and may work reasonably well when we do not have many data. It is fairly easy to construct a prior when there exists considerable prior information about the proportion of interest. The posterior distribution gives us information regarding the likelihood of values of θ given sample data. Then the question is how to use this information to estimate θ. Instead of having explicit probabilities, the prior may be given through an assumed probability distribution. We illustrate the calculations involved to ﬁnd the posterior distribution in the following example.

11.2 Bayesian Point Estimation 567

Example 11.2.3 Let X be a binomial random variable with parameters n and p. Assume that the prior distribution of p is uniform on [0,1]. Find the posterior distribution, f (p|x).

Solution Because X is binomial, the likelihood function is given by n x f (x|p) = p (1 − p)n−x . x Because p is uniform on [0,1], π(p) = 1, 0 ≤ p ≤ 1. Then the posterior distribution is given by n x f (p|x) ∝ f (x|p)π(p) = p (1 − p)n−x , x = 0, 1, . . . , n x which is the same as the likelihood.

This example illustrates that if the prior is noninformative (uniform), then the posterior is essentially the likelihood function. In the case where the prior and posterior are of the same functional form, we call it a conjugate prior. Bayesian inference becomes simpler when the prior density has the same functional form as the likelihood (which is the case for the conjugate prior) or when data are an independent sample from an exponential family (such as normal, Poisson, or binomial). The following example demonstrates the method of ﬁnding posterior distribution for a continuous random variable.

Example 11.2.4 Suppose that X is a normal random variable with mean μ and variance σ 2 , where σ 2 is known and μ is unknown. Suppose that μ behaves as a random variable whose probability distribution (prior) is π(μ) and is also normally distributed with mean μp and variance σp2 , both assumed to be known or estimated. Find the posterior distribution f (μ|x).

Solution Using the Bayes theorem, we have f (μ|x) =

f (x|μ)π(μ) f (x|μ)π(μ)dμ

2 2 2 2 √ 1 e−(x−μ) /2σ √ 1 e−(μ−μp ) /2σp 2πσ 2πσp = 2 2 2 2 √ 1 e−(x−μ) /2σ √ 1 e−(μ−μp ) /2σp dμ 2πσ 2πσp ! " 2 (μ−μp )2 − (x−μ) 1 2 + 2

=

2πσσp

e

2σ

2σp

.

(11.2)

568 CHAPTER 11 Bayesian Estimation and Inference

2 (μ−μ )2 Consider the exponential term in (11.2), namely, (x−μ) + 2σ 2p . 2σ 2 p

(μ − μp )2 1 (x − μ)2 + = 2 2 2 2σ 2σp 1 = 2 1 = 2

%

1 1 + 2 2 σ σp

%

(μ − μp )2 (x − μ)2 + 2 σ σp2

μ2 − 2

μp x + 2 2 σp σ

&

μ+

μ2p x2 + σ2 σp2

&

& % 2 σp + σ 2 2 μ2p μp x2 x μ −2 + 2 μ+ + 2 σ 2 σp2 σp2 σ σ2 σp

% 2 2 σ 2 σp2 μp x 1 σp + σ 2 + 2 μ μ −2 2 = 2 σ 2 σp2 σp + σ 2 σp2 σ

& μ2p x2 + 2 + 2 σp + σ 2 σ 2 σp % 2 2 σp2 σ2 1 σp + σ 2 μp + 2 x μ μ −2 = 2 σ 2 σp2 σp2 + σ 2 σp + σ 2 2 ⎤ σp2 σ2 μp + 2 x ⎦ + σp2 + σ 2 σp + σ 2 σ 2 σp2

⎡ 2 2 2 μ2p σp2 σ2 1 σp + σ ⎣ x2 + 2 − x+ 2 μp + 2 σ 2 σp2 σ2 σp σp2 + σ 2 σp + σ 2 % &2 2 2 σp2 σ2 1 σp + σ ˜ μ− = μp + 2 x + K, 2 σ 2 σp2 σp2 + σ 2 σp + σ 2 where ⎡ 2 ⎤ 2 2 μ2p σp2 1 σp + σ ⎣ x2 σ2 ˜ = K + 2 − μp + 2 x ⎦. 2 σ 2 σp2 σ2 σp σ 2 + σp2 σ + σp2 From the foregoing derivation, we obtain − 12

f (μ| x) = Ke

2 +σ 2 σp 2 σ 2 σp

!

μ−

2 σp σ2 2 +σ 2 μp + σ 2 +σ 2 x σp p

"2

,

where K does not contain μ. This implies that the posterior density f (μ |x ) is the pdf of normal random variable with mean σp2 σ2 μp + 2 x σp2 + σ 2 σp + σ 2

11.2 Bayesian Point Estimation 569

and variance σ 2 σp2 σp2 + σ 2

.

If we let τp = σ12 and τ = σ12 , then the posterior density can be rewritten as the pdf of normal random p variable with mean τp1+τ τp μp + τx and variance τp1+τ . As an example, suppose that μp = 100, σp = 15, and σ = 10, x = 115. Then f (μ |x ) is the pdf of a normal random variable with Mean =

225 100 (100) + (115) = 110.4 100 + 225 100 + 225

and Variance =

(100)(225) = 69.2. 100 + 225

11.2.1 Criteria for Finding the Bayesian Estimate In the Bayesian approach to parameter estimation, we use both the prior and observations. This leads to an estimation strategy based on the posterior distribution. How do we know that the estimate thus obtained is “good”? To assess the quality of likely estimators, we use a loss function L (θ, a) that measures the loss incurred by using a as an estimate of θ. Here θ is the parameter being estimated (in real-world problems it is not known), and a is the estimate of θ. Then the “optimal” or “best” estimate a = θˆ is chosen so as to minimize the expected loss E[L(θ, θˆ )], where the expectation is taken over θ with respect to the posterior distribution f (θ |x ). Here we mention two types of commonly used loss functions: quadratic and absolute error loss functions and the resulting estimates. (1) A quadratic (or squared error) loss function is of the form L(θ, a) = (a − θ)2 . In this case, E [L(θ, a)] =

L(θ, a)f (θ|x1 , . . . , xn )dθ

=

(a − θ)2 f (θ|x1 , . . . , xn )dθ.

Differentiating with respect to a and equating to zero, we obtain 2

(a − θ) f (θ |x1 , . . . , xn ) dθ = 0

This implies

a=

θf (θ |x1 , . . . , xn ) dθ.

This is the posterior mean (expected value) of θ, E (θ |x1 , . . . , xn ). Hence the quadratic loss function is minimized by taking the estimate of θ, that is, θˆ , to be the posterior mean. In previous examples in this section, we used this value as the estimate θˆ . Note that what the quadratic loss function displays

570 CHAPTER 11 Bayesian Estimation and Inference

is that if the estimate θˆ and the true parameter θ are close to each other, the loss we expect is very small. Likewise, if the difference is larger, the expected loss in estimating θ with θˆ is going to be large. (2) An absolute error loss function is of the form L (θ, a) = |a − θ|. In this case, L(θ, a)f (θ |x1 , . . . , xn )dθ

E [L(θ, a)] =

a (a − θ) f (θ |x1 , . . . , xn )dθ

= θ=−∞

+

∞ (θ − a)f (θ |x1 , . . . , xn )dθ

θ=a

Differentiating with respect to a and equating to zero, we obtain a f (θ |x1 , . . . , xn ) dθ − θ=−∞

∞ f (θ |x1 , . . . , xn ) dθ = 0

θ=a

The minimum loss is attained when the values of both integrals are equal to 12 . This can be achieved by taking θˆ to be the posterior median. The following can be considered as a general Bayesian procedure for point parameter estimation.

BAYESIAN PARAMETER ESTIMATION PROCEDURE 1. Consider the unknown parameter θ as a random variable. 2. Use a probability distribution(prior) to describe the uncertainty about the unknown parameter. 3. Update the parameter distribution using the Bayes theorem: P(θ|Data) ∝ P(θ)P(Data|θ), that is, (posterior of θ) ∝ (prior of θ).(likelihood). 4. The Bayes estimator of θ is set to be the expected value of the posterior distribution P(θ |Data) under quadratic loss function. 5. The Bayes estimator of θ is set to be posterior median under absolute error loss function.

From the procedure of Bayesian estimation, it is clear that a bad choice of prior may result in a bad estimate. Generally, if the priors are based on a previous and trustworthy sample, Bayesian estimation methods are desirable. A schematic ﬁgure of steps involved in the Bayesian estimate is given in Figure 11.1.

11.2 Bayesian Point Estimation 571

Prior info, P()

Posterior P(| Data)

Loss function

Updated

Likelihood P(Data |) ■ FIGURE 11.1 Bayesian estimation procedure.

In this chapter, we use only the quadratic loss function unless it is explicitly stated otherwise. We also mention that this loss function is very popular because of its analytic tractability. We now derive Bayesian point estimates for some speciﬁc distributions. Whereas uniform priors are useful in the noninformative situations, the beta family of distributions is one of the commonly taken informative priors. Distributions in the beta family take values in the interval (0, 1). Recall that if X ∼ beta(α, β), then the pdf of X is given by f (x) =

(α+β) α−1 (1 − x)β−1 , (α) (β) x

0≤x 0, β > 0.

The beta pdf can be written as f (x) = Cxα−1 (1 − x)β−1 ∝ xα−1 (1 − x)β−1 ,

where C =

(α+β) (α) (β) .

We also know that E (X) =

α , α+β

and

Var (X) =

αβ . (α + β)2 ((α + β + 1)

Example 11.2.5 Let X1 , . . . , Xn be a sample from geometric distribution with parameter p, 0 ≤ p ≤ 1. Assume that the prior distribution of p is beta with α = 4, and β = 4. (a) Find the posterior distribution of p. (b) Find the Bayes estimate under quadratic loss function.

Solution (a) Because p is Beta(4, 4), the prior density is (8) p3 (1 − p)3 = 140p3 (1 − p)3 . (4) (4)

572 CHAPTER 11 Bayesian Estimation and Inference

Because the r.v.’s Xi s have geometric distribution with parameter p, the likelihood is given by L(X1 , . . . , Xn |θ ) =

n 7

n xi −n x −1 n i i=1 p (1 − p) = p (1 − p) .

i=1

The product of the likelihood function and the prior is given by n n xi −n xi −n+3 n 3 3 n+3 (1 − p)i=1 140p (1 − p) = 140p . p (1 − p)i=1

Because, (posterior of p) ∝ (prior ofp) . (likelihood), rewriting the normalizing constant in the denominator of Equation (11.1) as C, and letting C1 = 140C, the posterior distribution (because n xi − n + 4 . α − 1 = n + 3, and β − 1 = ni=1 xi − n + 3) is Beta n + 4, i=1

(b) Recall that for a Beta(α, β) random variable, the mean is [α/(α + β)]. Because the Bayes estimate n xi − n + 4 is is the posterior mean, the mean of Beta n + 4, i=1

%

n

n+4 = n xi + 8 xi − n + 4 + (n + 4) n+4 &

i=1

i=1

Note that for large n, the Bayes estimate is approximately n/ ni=1 xi , which is the MLE of p. In general, for a Bernoulli random variable with unknown probability of success p in [0,1], the usual conjugate prior is the beta distribution, where the parameters of the beta distribution are chosen to reflect any prior information that we have. We will follow the idea of the previous example in a binomial experiment of tossing a coin.

Example 11.2.6 Suppose we are ﬂipping a biased coin, where the probability of heads p could be any value between 0 and 1. Given a sequence of toss samples x1 , x2 , . . . , xn , we want to estimate P (H) = p. We may have two sources of information: our prior belief, which we will express as a beta distribution, and the data, which could come from counts of heads x in n = 20 independent ﬂips of the coin, say x = 13. Suppose that in six prior tosses, we observed three heads and three tails, which lead us to believe that the value of p is near 0.5. Obtain the posterior distribution of p.

Solution Here our prior belief or assumption can be written in terms of beta distribution as π (p) =

(α + β) α−1 (1 − p)β−1 p (α) (β)

where α = 4 and β = 4. That is (noting (n) = (n − 1)!) π(p) =

7! p3 (1 − p)3 . (3!)(3!)

11.2 Bayesian Point Estimation 573

Hence, π(p) ∝ p3(1 − p)3 . Because the mean of a beta distribution is α/(α + β) and the variance is αβ/ (α + β)2 (α + β + 1) , for the prior, Mean(p) =

4 = 0.5, 4+4

and Var(p) =

(4)(4) = 0.028. (4 + 4)2 (4 + 4 + 1)

Let X denote the number of heads in 20 flips of this coin. Then X has a binomial distribution, and the pmf is given by 20 x f (x|p) = p (1 − p)20−x , x = 0, 1, . . . , 20. x This we can write as f (x|p) ∝ px (1 − p)20−x . In the 20 flips we have observed 13 heads. Then fix x = 13, and we are interested in the likelihood, which is the relative value of the function at different values of p: f (13|p, 20) ∝ p13 (1 − p)7 . The posterior probability of p, given x = 13, is π(p|x = 13) ∝ f (x|p)π(p) = p13 (1 − p)20−13 p3 (1 − p)3 = p16 (1 − p)10 . Thus, the posterior is a beta distribution with α = 17 and β = 11. Consequently, we can now obtain the mean and variance of p as Mean(p) =

17 = 0.607 17 + 11

and Var(p) =

(17)(11) = 0.008. (17 + 11)2 (17 + 11 + 1)

Note that the prior was beta distribution with mean 0.5 and variance 0.028. Figure 11.2 gives the prior and posterior densities. Note that if we had ignored the prior and just took the point estimation, then the MLE of p is MLE(p) = pˆ = 13 = 0.65. Compare this with the Bayesian estimate of p = 0.607. Because Beta(1, 1) is the Uniform [0, 1], 20

574 CHAPTER 11 Bayesian Estimation and Inference

4.5 4 3.5 Posterior

(x)

3 2.5 2

Prior

1.5 1 0.5 0

0

0.2

0.4

0.6

0.8

1

p ■ FIGURE 11.2 Prior and posterior distributions for the proportions.

the method of the previous example can be used for noninformative priors. The method could also be used in many applications. For example, suppose p represents the proportion of infected individuals in a population, and x is the number of infected individuals in a sample of size n. Then with a noninformative prior, we can show that the posterior of p is Beta(x + 1, n − x + 1). This type of setting can be used for estimating the true proportion of infected individuals in the population.

Example 11.2.7 Suppose for the past million days we have been predicting whether the sun will rise the next morning or ˆ and we were right (R) all these days. not. Each evening we say that the sun will rise the next morning (R), Suppose on the 106 evenings we predicted that the sun will rise on the next day. What is the probability that the sun will rise the next day?

Solution The problem can be cast in the following table form. 1 ˆ R R

2 ˆ R R

... ... ...

106 ˆ R R

106 + 1 ˆ R ?

ˆ = 1 if we use the frequency method of estimation (for example the MLE). Let us now consider the P(R|R) Bayes method. Suppose the prior is uniform on [0,1]. That is, ⎧ ⎨1, if 0 ≤ p ≤ 1 π(p) = ⎩0, otherwise.

11.2 Bayesian Point Estimation 575

Suppose we predict n times and we succeed x times. Then n x f (x|p) = p (1 − p)n−x . x The joint pdf is given by f (x, p) = f (x|p)π(p) n x = p (1 − p)n−x , x

x = 0, 1, . . . , n;

0 ≤ p ≤ 1.

By the Bayes theorem, the posterior pdf π(p|x) is π(p|x) =

f (x|p)π(p) 1

f (x|p)π(p)dp

0

= K(n, x)px (1 − p)n−x ,

0 ≤ p ≤ 1,

0 ≤ x ≤ n,

which is a beta probability distribution. Recall that the beta density is given by f (y) =

1 yα−1 (1 − y)β−1 B(α, β)

α . Thus, and E(Y ) = α+β

E [π (p |x )] =

x+1 x+1 . = (x + 1) + (n − x) + 1 n+2

In our example, x = 106 , n = 106 , which implies that the posterior mean is given by pˆ β =

106 + 1 ≈ 1. 106 + 2

Example 11.2.8

Let X1 , X2 , . . . , Xn be N μ, σ 2 random variables with prior π (μ) having N μ0 , σ02 distribution with known σ 2 . (a) Obtain the posterior distribution of μ. (b) Suppose it is known from past experience that the weight loss for a particular combination of diet and exercise program (if followed for a month) is normally distributed with mean 10 lb and standard deviation of 2 lb. A random sample of ﬁve persons who went through this program for a month produced the following weight loss in pounds: 14

8

11

7

11

What is the point estimate of the mean, μ? Assume σ 2 = 4.

576 CHAPTER 11 Bayesian Estimation and Inference

Solution

# $ (a) Because π (μ) ∼ N μ0 , σ02 , π (μ) ∝ exp (μ − μ0 )2 /σ02 and we omit the terms that do not depend on μ. We have from the data x = (x1 , . . . , xn ), the likelihood function, 4 n 7 (xi − μ)2 L (x1 , . . . , xn |μ ) = f (x |μ ) ∝ exp − 2σ 2 i=1 4 n 2 2 (xi − μ) /2σ = exp − , i=1

where μ is determined by the posterior distribution. The product of the likelihood function and the prior gives the posterior, which is obtained (after some algebra) as follows: f (μ|x) ∝ π(μ)f (x|μ) ∝ exp − (μ − μ1 )2 /2σ12 where μ1 =

n x+ 1 μ σ2 σ02 0 n + 1 σ2 σ02

and 1 σ12 = n . 1 + 2 σ2 σ0 2 Thus, the posterior distribution of μ is N μ1 , σ1 . (b) Note that the sample mean x = 10.2 lb, and sample standard deviation s = 2.77 lb. Now from part (a), the posterior distribution of μ is normal with mean n x+ 1 μ 5 (10.2) + 1 (10) σ2 σ02 0 22 22 μ1 = = = 10.167 n + 1 5 + 1 σ2 22 22 σ02

and variance 1 1 σ12 = n = 5 = 0.66667. 1 + + 212 σ2 22 σ02

Thus, the point estimate of μ is the posterior mean, 10.167. Figure 11.3 represents the prior and posterior densities of μ. Sometimes, the inverse of variance in the normal distribution is called the precision of the normal distribution and denoted by τ = 1/σ 2 . Also note that in part (a) of the previous example, if the prior variance σ02 → ∞, then the prior ﬂattens out, π(μ) ∝ c, a constant. This basically amounts to saying that prior information on μ decreases, that is, all μ are equally probable. This corresponds 2 to a noninformative prior. Also, in this case as σ02 → ∞, σ12 → σn and μ1 → x. Hence, in the limit

11.2 Bayesian Point Estimation 577

0.5 0.45 0.4 0.35 ()

0.3

Posterior

0.25 0.2 Prior

0.15 0.1 0.05 0

4

6

8

10

12

14

16

■ FIGURE 11.3 Prior and posterior densities of μ.

(i.e., for noninformative priors), the posterior f (μ|x) will have an N(x, σ 2 /n) distribution, which is exactly the same inference as in classical statistics. In Bayesian inference problems, one of the questions is, which will have relatively more inﬂuence, prior or likelihood? As we observe a large amount of data, it can be shown that the posterior distribution is almost exclusively determined by the data. That is, asymptotically, observed data will have a larger inﬂuence compared to the choice of prior, and thus the prior will be irrelevant. Hence, we can make the following general observations. If the prior is noninformative and we have a large data set, then we can expect that the likelihood will have greater inﬂuence. Whereas, if we have a small data set and an informative prior, then the prior will have a larger inﬂuence on the updated posterior distribution. Bayesian estimators are more complicated to compute than calculating the maximum likelihood estimates in simple cases. However, in complex settings Bayesian statistics are often relatively easier to compute. One of the problems in using Bayesian analysis is choosing an appropriate prior. There are no speciﬁc rules available for this purpose. For instance, the following priors are commonly used in the literature. If data are in [0,1], we could use uniform or beta distribution. If the data are in [0, ∞), normal (with nonnegative and relatively large μ), gamma, or log-normal distributions are used. If the data are in (−∞, ∞), normal or t-distributions are commonly used.

EXERCISES 11.2 11.2.1.

Suppose in a casino, two kinds of dice are used, one kind of which 98% are fair, and 2% are loaded such that ﬁve comes up 60% of the time and the rest of the numbers are equally probable. We pick a die at random and roll it three times. We get three consecutive ﬁves. What is the probability that the die is loaded?

578 CHAPTER 11 Bayesian Estimation and Inference

11.2.2.

It is believed that cross-fertilized plants produce taller offspring than self-fertilized plants. In order to obtain an estimate on the proportion θ of cross-fertilized plants that are taller, an experimenter observes a random sample of 15 pairs of plants exactly the same age, with each pair grown in the same conditions with one cross-fertilized and the other self-fertilized. Based on previous experience, the experimenter believes that the following are possible values of π and prior probabilities for each value (prior weight), π(θ): θ: π (θ):

0.80 0.03

0.82 0.40

0.84 0.22

0.86 0.15

0.88 0.15

0.90 0.05

From the experiment, it is observed that in 13 of 15 pairs, the cross-fertilized is taller. (a) Create a table with columns for prior, likelihood of θ given sample, prior times likelihood, and posterior probability of θ. Based on the posterior probabilities, what value of θ has the highest support? Also, ﬁnd E(θ) based on the posterior probabilities. (b) Redo part (a) with a completely noninformative prior, that is, take the prior for the proportion θ as one of the equally spaced values 0, 0.1, 0.2, . . . , 0.9, 1. Also assign for each value of θ the same probability, π(θ) = 1/11. (c) Calculate the MLE of θ and compare it with the Bayesian estimate. 11.2.3.

Consider the problem of estimating p in a binomial distribution. Let X be number of successes in a sample of size n. (a) Let the prior distribution of p be given by Beta(3,1), that is π (p) =

3p2 , 0,

0 1.

586 CHAPTER 11 Bayesian Estimation and Inference

This method of hypothesis testing is called Jeffreys’ hypothesis testing criterion. It basically says that if the posterior odds ratio is greater than 1, we accept the null hypothesis; otherwise, we reject the null in favor of the alternative hypothesis. Because we cannot determine the probability of a single value in the continuous variable case, it should be noted that for a simple null hypothesis of the form θ equals some speciﬁed value cannot be dealt with easily in the Bayesian framework. Hence, unlike the classical framework, here we mostly deal with the composite hypotheses for both null and alternative.

Example 11.4.1 A student taking a standardized test is classiﬁed as gifted if he or she scores at least 100 out of a possible score of 150. Otherwise the student is classiﬁed as not gifted. Suppose the prior distribution of the scores of all students is a normal with mean 100 and standard deviation 15. It is believed that scores will vary each time the student takes the test and that these scores can be modeled as a normal distribution with mean μ and variance 100. Suppose the student takes the test and scores 115. Test the hypothesis that the student can be classiﬁed as a gifted student.

Solution The hypothesis testing problem can be phrased as H0 : θ < 100 vs. Ha : θ ≥ 100. Referring to the Example 11.2.8, we know that the posterior distribution f (θ|x) is a normal with mean 110.4 and variance 69.2. Because the prior is an N(100, 225), we have π0 = P(θ < 100) = 1/2 and π1 = P(θ ≥ 100) = 1/2. We can now compute α0 = P (θ < 100 |x = 115 )

100 − 110.4 θ − 110.4 < =P √ √ 69.2 69.2

10.4 = 0.106 = P z ≤ −√ 69.2 and α1 = P (θ ≥ 100 |x = 115 ) = 1 − P (θ < 100 |x = 115 ) = 1 − 0.106 = 0.894. Thus, α0 /α1 = (0.106/0.894) = 0.119 < 1, and we reject H0 .

11.4 Bayesian Hypothesis Testing 587

BAYESIAN HYPOTHESIS TESTING PROCEDURE To test H0 : θ ∈ 0 vs. H1 : θ ∈ 1 , where 0 and 1 are given sets: 1. Consider θ as a random variable with prior distribution π(θ). 2. Compute the posterior distribution f (θ |x1 , . . . , xn ) of θ given x1 , . . . , xn , using Bayes’ theorem. 3. Compute α0 and α1 using the following formulas: α0 = P (θ ∈ 0 |x1 , . . . , xn ) ⎧ ⎪ f (θ |x1 , . . . , xn ) dθ, ⎪ ⎨ 0 = ⎪ ⎪ f (θ |x1 , . . . , xn ) , ⎩

if continuous if discrete

θ∈0

and α1 = P (θ ∈ 1 |x1 , . . . , xn ) ⎧ ⎪ f (θ |x1 , . . . , xn ) dθ, ⎪ ⎨ 1 = ⎪ ⎪ f (θ |x1 , . . . , xn ) , ⎩

if continuous if discrete.

θ∈1

4. Reject H0 if the posterior odds ratio,

α0 < 1. Otherwise accept. α1

In the foregoing procedure, we assume that P (θ ∈ 0 ) and P (θ ∈ 1 ) are both greater than zero.

EXERCISES 11.4 11.4.1.

The following is random data from a normal distribution with variance 9. 0.92 1.05 7.42 1.76

5.53 3.64 0.01 2.69

−4.47 1.54

−2.60 3.97

0.71 1.34

−3.66 −1.63

1.38 −1.24

3.87 −4.78

(a) Test the hypothesis, H0 : μ ≤ 0 vs. Ha : μ > 0. Assume that the prior is N(0, 4), so that μ ≤ 0 and μ > 0 are equally probable. (b) Compare your decision with classical hypothesis testing, with α = 0.05. 11.4.2.

(a) For the data of Exercise 11.3.2, using the Bayesian method, test the hypothesis H0 : μ ≤ 170 vs. Ha : μ > 170. (b) Compare your decision with classical hypothesis testing, with α = 0.05.

11.4.3.

It is known that a certain disease affects 10% of a population. Of a random sample of 50 patients in the disease group who are exposed to a new treatment, we observe that 12 patients were hospitalized in a year. Let μ be the population rate that needs hospitalization in a year. Assume μ has a Gamma(0.1, 2) prior. Let μ ∼ Gamma(0.1, 2) and f (x|μ) ∼

588 CHAPTER 11 Bayesian Estimation and Inference

Poi(50μ). Given that x = 0.24 is an observation of X, test the hypothesis H0 : p ≤ 0.10 vs. Ha : p > 0.10. (If X is the number of patients admitted in a year, assume X ∼ Poi (50μ), the Poisson approximation of the binomial.) 11.4.4.

For an upcoming congressional election, suppose we want to estimate the amount of support for a particular candidate in a district. By previous experience and voter registration data, we can assume that the prior distribution, the proportion of support, p, is a beta distribution with α = 10, and β = 8 (i.e., π (p) ∼ Beta (10, 8)). We conducted a survey of 1000 randomly selected voters, of whom 600 support the candidate. Test the hypothesis H0 : p ≥ 0.60 vs. Ha : p < 0.60.

11.4.5.

For the data of Exercise 11.3.5, test the hypothesis H0 : μ ≤ 2400 mg vs. Ha : μ > 2400 mg for this ethnic group.

11.4.6.

Suppose we have a coin (not necessarily balanced) with p being the probability of heads. Assume a uniform prior for p. Suppose in 20 tosses of this coin, we obtained 12 heads. Test the hypothesis H0 : p ≥ 0.50 vs. Ha : p > 0.50.

11.5 BAYESIAN DECISION THEORY Bayesian methods in general are more concerned with problems of decision making than with problems of inference. Decision theory, as the name implies, is concerned with the problem of making decisions. Statistical decision theory is concerned with optimal decision making under uncertainty or when statistical knowledge is available only on some of the uncertainties involved in the decision problem. Uncertainty could be about the true value related to the decision, or, uncertainty could be about the actual state of the nature. Abraham Wald (1902–1950) laid the foundation for statistical decision theory. Original works on the decision theory emerged out of game theory considerations. Many books and articles have been written on the various aspects of decision theory. The Bayesian approach to the decision theory was introduced by Leonard Jimmie Savage in 1954. In this section, we introduce the general idea of decision theory. We basically deal with analytical procedures for the decision-making process. This will involve selection of an optimum decision from a choice of courses of action among two or more alternatives. The Bayesian decision theory quantiﬁes the trade-offs between different decisions using costs and probabilities that accompany such decisions. Consider, as an example, a company deciding whether or not to market a new brand of toothpaste with a whitening agent. Clearly many factors will affect the decision (for example, the proportion of people who are likely to switch to the new brand, and the likelihood of other competing companies introducing similar toothpastes). These factors are generally unknown, but estimates can be obtained from statistical investigations. The classical statistical approach relies exclusively on the data obtained from these statistical investigations, ignoring other relevant information such as the company’s past experiences in marketing similar products. Statistical decision theory tries to combine other relevant information with the sample information to arrive at the optimal decision. Therefore, a Bayesian setting seems to be more appropriate for decision theory.

11.5 Bayesian Decision Theory 589

One piece of relevant information that decision theory considers is the possible consequences of the decisions. Often these consequences can be quantiﬁed. That is, the loss or gain of each decision can be expressed as a number (called the loss or utility). A loss or utility to a decision maker is the effect of the interaction of two factors: (1) the decision or action selected by the decision maker; and (2) the event or state of the world that actually occurs. Classical statistics does not explicitly use a loss function or a utility (payoff ) function. A second source of information that decision theory utilizes is the prior information. Prior information could be based on past experiences of similar situations or on expert opinion. We can follow the procedure explained next as a guideline for decision making.

GENERAL DECISION THEORY PROCEDURE 1. Identify the objectives of the decision-making process. 2. Identify the set of actions and set of possible events (states of nature). 3. Assign probabilities to the occurrence of each possible state of nature (prior). If more observations are available, calculate the posterior probabilities to the occurrence of each possible state of nature. 4. For each possible event, assign a numerical value to the anticipated payoff (or loss) of each course of action. 5. Compute the expected value of the payoffs (utility or loss function). This could be done by either using the prior probabilities if there are no observations, or using the posterior probabilities. 6. Select the optimum decision among the available alternative courses of action that maximizes the expected value of the payoffs.

We now consider an example to illustrate the idea of statistical decision making.

Example 11.5.1 Suppose you own a small stall at a ﬂea market that is open only on weekends. If the weather is good, you make a proﬁt of $200, and if it is bad, you close your stall and you make no (zero) proﬁt. However, you have the option of buying, from an insurance company, weather insurance that costs $75. The company pays you $210 if the weather is bad. Suppose you believe that the probability of good weather on a particular weekend is p. Compute the expected gain if you insure and if you do not. What is the best course of action? Arrive at a decision.

Solution From the information in the problem, we can obtain the utility gain or profit table shown in Table 11.4, based on our decision to insure or not insure. Suppose that we model the state of weather as good or bad by means of a random variable defined as follows. θ=

1,

if the weather is good

0,

if the weather is bad.

590 CHAPTER 11 Bayesian Estimation and Inference

Table 11.4 Weather Parameter Space → Decision Space ↓D Insurance (I)(d1)

Good (θ1 )

Bad (θ2 )

$125 (200–75)

$135 (210–75)

$200

$0

No Insurance (NI)(d2)

Suppose for our example we believe that during a particular weekend P(θ = 1) = p, and P(θ = 0) = 1 − p. This can be considered as prior information. The different values of θ are called states of nature. We assign (perhaps subjectively) a probability structure for the states of nature defined by a prior distribution π(θ). Now we can compute the expected gain when we insure and when we do not. Using the values in the table, Expected gain given we insure = (125) p + (135) (1 − p) = 135 − 10p Expected gain when do not insure = (200) p + (0) (1 − p) = 200p Hence, insurance is preferable if 135 − 10p > 200p or p

M).

We consider the problem of testing the null hypothesis H0 : M = m0

versus

Ha : M > m0 .

Assume that the underlying population distribution is continuous so that P (X ≤ M) = 0.5. Let Xi be the ith observation and let N + be the number of observations that are greater than m0 . N + will be our test statistic. We will reject H0 if, n+ the observed value of N + , is too large. This test is called the sign test. A test at signiﬁcance level α will reject H0 if n+ ≥ k, where k is chosen such that P(N + ≥ k when M = m0 ) = α.

Similarly, if the alternative is of the form Ha : M = m0 , the critical region is of the form N + ≤ k or N + ≥ k1 , where P(N + ≤ k) + P(N + ≥ k1 ) = α. In order to determine such a k and k1 , we need to determine the distribution of N + . The test works on the principle that if the sample were to come from a population with a continuous distribution, then each of the observations falls above the median or below the median with probability 12 . Hence, the number of sample values falling below the median follows a binomial distribution with parameters n and p = 12 , n being the sample size. If a sample value equals the hypothesized median m0 , that observation will be discarded and the sample size will be adjusted accordingly (we remark that such values should be very few). Thus, when H0 is true, N + will have a binomial distribution with parameters n and p = 12 . For this reason, some authors call this test the binomial test. The following box summarizes the test procedure and the corresponding critical regions.

608 CHAPTER 12 Nonparametric Tests

SIGN TEST H0 : M = m0 Alternative Hypothesis Ha : M > m0

Critical Region

1 n n N + ≥ k, where =α 2 i=k i n

1 n n =α 2 i

Ha : M < m0

k N + ≤ k, where i=0

Ha : M = m0

n N + ≥ k1 , where i=k1

or k N + ≤ k, where i=0

1 n α n = 2 2 i

α 1 n n = 2 2 i

If α or α/2 cannot be achieved exactly, choose k (or k and k1 ) so that the probability comes as close to α (or α/2) as possible.

We now summarize the procedure of the sign test in the case of an upper tail alternative. The other two cases are similar. HYPOTHESIS TESTING PROCEDURE BY SIGN TEST We test H0 : M = m0 vs. H1 : M > m0 . 1. Replace each value of the observation that is greater than m0 by a plus sign and each sample value less than m0 by a minus sign. If the sample value is equal to m0 , discard the observation and adjust the sample size n accordingly. 2. Let n+ be the number of +’s in the sample. For n and p = 12 , from the binomial table, ﬁnd γ = P (N + ≥ n+ ). 3. Decision: If γ is less than α, H0 must be rejected. Based on the sample, we will conclude that the median of the population is greater than m0 at the signiﬁcance level α. Otherwise do not reject H0 . Assumptions: The population distribution is continuous. The number of ties is small (less than 10% of the sample).

12.3 Nonparametric Hypothesis Tests for One Sample 609

Note that the approach described in the foregoing procedure is nothing but the p-value method for hypothesis testing regarding a median using the sign test. Recall that the p-value is the probability of observing a test statistic as extreme or more extreme than what was really observed, under the assumption that the null hypothesis is true. In the sign test, we had assumed that the median is M = m0 , so 50% of the data should be less than m0 and 50% of the data greater than m0 . Thus, we expect half of the data to result in plus signs and half to result in minus signs. Hence, we can think of the data as following a binomial distribution with p = 1/2 under the null hypothesis. The p-value is computed from its deﬁnition given by the formula p-value = P (N + ≥ n+ ) =

n n 1 n i=k

i

2

= γ.

The p-value method is to reject the null hypothesis if the computed p-value is greater than α. These binomial probabilities can be obtained from the binomial tables, or statistical software packages. The following example illustrates how we apply the three-step procedure.

Example 12.3.1 For the given data from an experiment 1.51

1.35

1.69

1.48

1.29

1.27

1.54

1.39

1.45

test the hypothesis that H0 : M = 1.4 versus Ha : M > 1.4 at α = 0.05.

Solution We test H0 : M = 1.4 versus Ha : M > 1.4. Replacing each value greater than 1.4 with a plus sign and each value less than 1.4 with a minus sign, we have + − + + − − + − +. Thus, n+ = 5. From the binomial table with n = 9 and p = 12 , we have P(N + ≥ 5) = 0.50. Thus, the p-value is 0.5. Because α = 0.05 < 0.50, the null hypothesis is not rejected. We conclude that the median does not exceed 1.4.

When the sample size n is large, we can apply the normal approximation to the binomial distribution. That is, the test statistic N + is approximately normally distributed. Thus, under H0 , N + will

610 CHAPTER 12 Nonparametric Tests

have approximate normal distribution with mean np = z-transform, we have Z=

n 2

and variance of np (1 − p) =

n 4.

By the

N + − n/2 2N + − n ∼ N(0, 1). = √ √ n n/4

We could utilize this test if n is large, that is, if np ≥ 5 and n(1 − p) ≥ 5. Hence, under H0 , because p = 1/2, if n ≥ 10, we could use the large sample test. The following table summarizes the large sample sign test. A SIGN TEST FOR A LARGE RANDOM SAMPLE When the sample size is large (n ≥ 10), we can use the normal approximation to a binomial. This leads to the large sample sign test: H0 : M = m0 versus Alternative Hypothesis Ha : M > m0

Rejection Region z ≥ zα

Ha : M < m0 Ha : M = m0

z ≤ −zα |z| ≥ zα/2

The test statistic is Z=

2N + − n . √ n

Decision: Reject H0 , if the test statistic falls in the rejection region, and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for a given α, and more experiments are needed. Assumptions: (i) Population distribution is continuous. (ii) Sample size greater than or equal to 10 (after the removal of ties). (iii) The number of ties is small (less than 10% of the sample size).

We illustrate this procedure with the following example.

Example 12.3.2 In order to measure the effectiveness of a new procedure for pruning grapes, 15 workers are assigned to prune an acre of grapes. The effectiveness is measured in worker-hours/acre for each person. 5.2 4.2

5.0 5.3

4.8 4.9

3.9 4.7

6.1 4.9

4.2

4.4

5.5

5.8

4.5

Test the null hypothesis that the median time to prune an acre of grapes with this method is 4.5 hours against the alternative that it is larger. Use α = 0.05.

12.3 Nonparametric Hypothesis Tests for One Sample 611

Solution We test H0 : M = 4.5 versus H0 : M > 4.5. Replacing each value greater than 4.5 with a plus sign and each value less than 4.5 with a minus sign, we have +++−+−−++−++++. Because there is one observation that is equal to 4.5, we must discard it and take n = 14. Thus N + = 10, using the large sample approximation, the test statistic is Z=

20 − 14 2N + − n = 1.6. = √ √ n 14

For α = 0.05, from the standard normal table, the value of z0.05 = 1.645. Hence, the rejection region is z > 1.645. Because the observed value of the test statistic does not fall in the rejection region, we do not reject the null hypothesis at α = 0.05 and conclude that the median time to prune an acre of grapes is 4.5 hours.

12.3.2 Wilcoxon Signed Rank Test In the sign test, we have considered only whether each observation is greater than m0 or less than m0 without giving any importance to the magnitude of the difference from m0 . An improved version of the sign test is the Wilcoxon signed rank test, in which one replaces the observations by their ranks of the ordered magnitudes of differences, |xi − m0 |. The smallest observation is ranked as 1, the next smallest will be 2, and so on. However, the Wilcoxon signed rank test requires an additional assumption that the continuous population distribution is symmetric with respect to its center. Thus, if the data are ordinal, the Wilcoxon test cannot be used.

HYPOTHESIS TESTING PROCEDURE BY WILCOXON SIGNED RANK TEST We test H0 : M = m0 versus H1 : M = m0 . 1. Compute the absolute differences zi = |xi − m0 | for each observation. Replace each value of the observation that is greater than m0 by a plus sign and each sample value that is less than m0 by a minus sign. If the sample value is equal to m0 , discard the observation and adjust the sample size n accordingly. 2. Assign each zi a value equal to its rank. If two values of zi are equal, assign each zi a rank equal to the average of ranks each should receive if there were not a tie. 3. Let W + be the sum of the ranks associated with plus signs and W − be the sums of ranks with negative signs.

612 CHAPTER 12 Nonparametric Tests

4. Decision: If m0 is the true median, then the observations should be evenly distributed about m0 . For a size α critical region, reject H0 if W + ≤ c1 , where P (W + ≤ c1 ) =

α , 2

or α . 2 Assumptions: The population distribution is continuous and symmetrical. The number of ties is small, less than 10% of the sample size. W + ≥ c2 , where P (W + ≥ c2 ) =

The exact distribution of W + is considerably complicated and we will not derive it. However, for certain values of n, the distribution is given in the Wilcoxon signed rank test table. For the Wilcoxon signed rank test, the rejections region based on the alternative hypothesis is given next. For Ha : M > m0 , rejection region is W + ≥ c, where P (W + ≥ c) = α,

and for Ha : M < m0 , rejection region is W + ≤ c, where P (W + ≤ c) = α.

We illustrate the Wilcoxon signed rank test with the following examples.

Example 12.3.3 For the given data that resulted from an experiment 1.51

1.35

1.69

1.48

1.29

1.27

1.54

1.39

1.45

test the hypothesis that H0 : M = 1.4 versus Ha : M = 1.4. Use α = 0.05.

Solution We test H0 : M = 1.4 versus Ha : M = 1.4. Here, α = 0.05, and m0 = 1.4. The results of steps 1 to 3 are given in Table 12.1. Thus, we have W + = 29 and n = 9. From the Wilcoxon signed-rank test table in the appendix, we should reject H0 if W + ≤ 6 or W + ≥ 38 with actual size of α = 0.054. Because W + = 29 does not fall in the rejection region, we do not reject the null hypothesis that M = 1.4.

12.3 Nonparametric Hypothesis Tests for One Sample 613

Table 12.1 xi

zi = |xi − 1.4|

Sign

Rank

1.51

0.11

+

5.5

1.35

0.05

−

3

1.69

0.29

+

9

1.48

0.08

+

4

1.29

0.11

−

5.5

1.27

0.13

−

7

1.54

0.14

+

8

1.39

0.01

−

1.5

1.45

0.01

+

1.5

Example 12.3.4 Air pollution in large U.S. cities is monitored to see whether it conforms to requirements set by the Environmental Protection Agency. The following data, expressed as an air pollution index, give the air quality of a city for 10 randomly selected days. 57.3

58.1

58.7

66.7

58.6

61.9

59.0

64.4

62.6

64.9

Test the hypothesis that H0 : M = 65 versus Ha : M < 65. Use α = 0.05.

Solution We test H0 : M = 65 versus Ha : M < 65. Here, α = 0.05, and m0 = 65. The results of steps 1 to 3 are given in Table 12.2. Thus, W + = 3, and n = 10. Using the Wilcoxon signed rank test table, we should reject H0 if W + ≤ 10 with actual size of α = 0.042. Because the observed value of W + falls in the rejection region, we reject H0 and conclude that the sample evidence suggests that we conclude the median air pollution index is less than 65.

The Wilcoxon signed rank test is a nonparametric alternative to the one-sample t-test. The question then is, how do we decide which one to choose? Choose the one-sample t-test if it is reasonable to assume that the population follows a normal distribution. Otherwise, choose the Wilcoxon nonparametric test. However, the Wilcoxon test will have less power. For example, a normal probability plot of the data of Example 12.3.4 is given in Figure 12.4. Looking at this ﬁgure, we can see that the normality assumption is a suspect. It may make more sense to use the nonparametric method.

614 CHAPTER 12 Nonparametric Tests

Table 12.2 xi

zi = |xi − 65|

Sign

Rank

57.3

7.7

−

10

58.1

6.9

−

9

58.7

6.3

−

8

66.7

1.7

+

3

58.8

6.2

−

7

61.9

4.1

−

5

59.0

6.0

−

6

64.4

0.6

−

2

62.6

2.4

−

4

64.9

0.1

−

1

Normal probability plot 0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 57

58

59

60

61

Average: 61.22 Std Dev: 3.32158 N: 10

62 63 Index

64

65

66

67

Kolmogorov-Smirnov Normality Test D⫹: 0.248 D⫺: 0.131 D: 0.248 Approximate P-Value: 0.081

■ FIGURE 12.4 Normal probability for air pollution index.

When sample size n is sufﬁciently large, under the assumption of H0 being true, the distribution of W + is approximately normal with mean E(W + ) =

1 n(n + 1) 4

12.3 Nonparametric Hypothesis Tests for One Sample 615

and variance Var(W + ) =

n(n + 1)(2n + 1) . 24

Hence, the test statistic is given by Z= √

W + − 14 n(n + 1)

n(n + 1)(2n + 1)/24

which is approximately the standard normal distribution. This approximation can be used when n > 20.

SUMMARY OF THE WILCOXON SIGNED RANK TEST FOR LARGE SAMPLES (N > 20) We test H0 : M = m0 versus M > m0 , upper tailed test Ha : M < m0 , lower tailed test M = m0 , two-tailed test. The test statistic: 1 n(n + 1) 4 Z= √ . n(n + 1)(2n + 1)/24 W+ −

Rejection region: ⎧ ⎪ ⎨ z > zα , z < −zα , ⎪ ⎩ |z| > z , α/2

upper tail RR lower tail RR two tail RR.

Decision: Reject H0 , if the test statistic falls in the RR, and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 , because there is not enough evidence to conclude that Ha is true for a given α and more experiments are needed. Assumptions: (i) The population distribution is continuous and symmetric about 0. (ii) Sample size is greater than or equal to 20. (iii) The number of ties is small, < 10% of the sample size.

We illustrate the Wilcoxon signed rank test with the following example.

616 CHAPTER 12 Nonparametric Tests

Example 12.3.5 The following data give the monthly rents (in dollars) paid by a random sample of 25 households selected from a large city. 425 960 1450 655 1025 750 670 975 660 880 1250 780 870 930 550 575 425 900 525 1800 545 840 765 950 1080 Using the large sample Wilcoxon signed rank test, test the hypotheses that the median rent in this city is $750 against the alternative that it is higher with α = 0.05.

Solution We test H0 : M = 750 versus Ha : M > 750. Here α = 0.05, and m0 = 750. The results of steps 1 to 3 are given in Table 12.3 (where the asterisk indicates zi = 0).

Table 12.3 xi

zi = |xi − 750|

Sign

425

325

−

19.5

960

210

+

15

1450

700

+

23

655

95

−

6

1025

302

+

18

750

0

∗

ignore

670

80

−

3

975

225

+

16.5

660

90

−

4.5

880

130

+

8

1250

500

+

22

780

30

+

2

870

120

+

7

930

180

+

11

550

200

−

12.5

Rank

(continued)

12.3 Nonparametric Hypothesis Tests for One Sample 617

Table 12.3 (continued) xi

zi = |xi − 750|

Sign

Rank

575

175

−

10

425

325

−

19.5

900

150

+

9

525

225

−

16.5

1800

1050

+

24

545

205

−

14

840

90

+

4.5

765

15

+

1

950

200

+

12.5

1080

330

+

21

Here, for n = 24, W + = 172.5, and the test statistic is 1 n(n + 1) 4 Z= √ n(n + 1)(2n + 1)/24 W+ −

1 172.5 − (24)(25) 4 = = 0.64286. 8 (24)(25)(49) 24 For α = 0.05, the rejection region is z > 1.645. Because the observed value of the test statistic does not fall in the rejection region, we do not reject the null hypothesis. There is not enough evidence to conclude that the median rent in this city is more than $750.

The rank tests are useful for situations when you suspect that the data do not follow the normal population. It is important to note that ignoring the tied observations reduces the effective sample size, which in turn reduces the power of the test (see Example 7.1.4 for the effect of n on the value of β). This loss is not signiﬁcant if there are only a few ties. However, if the ties are 10% or more, hypothesis testing using rank tests becomes considerably conservative. That is, they yield error probabilities that are signiﬁcantly high.

12.3.3 Dependent Samples: Paired Comparison Tests The sign test and the Wilcoxon signed rank test can also be used for paired comparisons. The experimental procedure typically consists of taking “before” and “after” type or otherwise matched as in

618 CHAPTER 12 Nonparametric Tests

the paired t-test case readings for each unit. Suppose there are n pairs of before and after observations and we are interested in testing the equality of the two medians. One way to test such observations is to consider the difference between the two observations for a unit to be a single observation on that unit. Thus, we can treat the sample as being n observations on a population of differences. For this new sample of differences, the testing problem becomes H0 : M = 0 versus Ha : M > 0(or M < 0, or M = 0).

Hence, the basic procedure could be summarized to ﬁrst ﬁnd the difference between the two units for each of the observations, and then follow the testing procedures explained earlier for the sign test or the Wilcoxon signed rank test. Both small sample and large sample cases can be handled as before. In the following example, we illustrate this concept for a large sample sign test.

Example 12.3.6 A dietary program claims that 3 months of its diet will reduce weight. In order to test this claim, a random sample of eight individuals who went through this program for 3 months is taken. The following table gives weight in pounds. Before After

180 172

199 191

175 172

226 230

189 178

205 199

169 171

211 201

Using a 5% signiﬁcance level, is there evidence to conclude that the program really reduces the population median weight?

Solution Let M denote the median of the population of difference of weights. We will use the difference as ‘‘after’’−’’before.’’ Then we will test H0 : M = 0

versus

Ha : M < 0.

We will use the large sample sign test. Replacing each value of the difference that is greater than zero by a + sign and less than zero by a − sign, we have Difference Sign

−8 −

−8 −

−3 −

4 +

−11 −

−6 −

2 +

−10 −

For n = 8 and N + = 2, the test statistic is given by Z=

2N + − n 4−8 = √ = −1.414. √ n 8

For α = 0.05, z0.05 = 1.645, and the rejection region is z < − 1.645. Because the observed value of the test statistic does not fall in the rejection region, we do not reject the null hypothesis. Thus, there is not enough evidence to conclude that the new program reduces the weight.

12.3 Nonparametric Hypothesis Tests for One Sample 619

EXERCISES 12.3 12.3.1.

It was reported that the median interest rate on 30-year ﬁxed mortgages in a certain large city is 7.75% on a particular day, with zero points. A random sample of nine lenders produced the following data of interest rates in percentage. 7.625 7.375 8.00 7.50 7.875 8.00 7.625 7.75 7.25 Test the hypothesis that the median interest rate in this city is different from 7.75%, using (a) the sign test, and (b) the Wilcoxon signed rank test. Use α = 0.01. Compare the two results.

12.3.2.

It is believed that a typical family spends 35% of its income on food and groceries. A sample of eight randomly selected families yielded the following data. 30

29

39

49

36

33

37

35

Test the hypothesis that the median percentage of family income spent for food and groceries is 35 against the alternative that it is less than 35. Use α = 0.05. 12.3.3.

The SAT scores (out of a maximum possible score of 1600) for a random sample of 10 students who took this test recently are: 1355 765 890 1089 986 1128 1157 1065 1224 567 Test the hypothesis that the median SAT score is 1000 against the alternative that it is greater using α = 0.05. Use both the sign test and the Wilcoxon signed rank test. Explain if the conclusions are different.

12.3.4.

The regulatory board of health in a particular state speciﬁes that the ﬂuoride levels in water must not exceed 1.5 parts per million (ppm). The 20 measurements given here represent the randomly selected daily early morning readings on ﬂuoride levels in water at a certain city. 0.88 0.82 0.71 0.92

0.97 1.11

0.95 0.84 0.90 0.81 0.97 0.85

0.87 0.97

0.78 0.75 0.83 0.91 0.78 0.87

Test the hypothesis that the median ﬂuoride level for this city is 0.90 against the alternative that the median is different from 0.9 at α = 0.01, using (a) the large sample sign test, and (b) the Wilcoxon signed rank test. Interpret the results. 12.3.5.

The following data give the weights (in pounds) for a random sample of 20 NFL players. 285 269

178 285

311 276 192 232 259 189 298 296 193 288 254 246 234 274

211 229

Test the hypothesis that the median weight of NFL players is 250 pounds against the alternative that it is greater at α = 0.05, using (a) the large sample sign test and (b) the Wilcoxon signed rank test.

620 CHAPTER 12 Nonparametric Tests

12.3.6.

The following data give the amount of money (in dollars) spent on textbooks by 18 students for the last academic year at a large university. 510 490

425 188

190 115

298 230

157 610

260 320 220 155

615 315

455 110

Test the hypothesis that the median amount spent on books at this university is $325 against the alternative that it is different using the large-sample sign test. Use α = 0.05. 12.3.7.

It is desired to study the effect of a special diet on systolic blood pressure. The following sample data are obtained for eight adults over 40 years of age before and after 6 months of this diet. Before After

185 188

222 235 217 229

198 190

224 226

197 185

228 225

234 231

At 95% conﬁdence level, is there evidence to conclude that the new diet reduces the systolic blood pressure in individuals of over 40 years old? Test (a) using the sign test, and (b) using the Wilcoxon signed rank test. Interpret the results. 12.3.8.

In an effort to study the effect on absenteeism of having a day-care facility at the workplace for women with newborn babies (less than 1 year old), a large company compared the number of absent days for a year for seven women with newborn children before and after instituting a day-care facility. Before After

20 16

18 9

35 22 17 24 15 22 28 19 13 10

At 99% conﬁdence level, is there evidence to conclude that having a day-care facility at the workplace reduces absenteeism for women with newborn children?

12.4 NONPARAMETRIC HYPOTHESIS TESTS FOR TWO INDEPENDENT SAMPLES In this section we learn how to test the equality of the medians of two independent samples from two populations. This is especially useful when one studies the treatment effects, such as the effect of a certain drug to treat a given medical condition when we have two groups—an experimental group and a control group—or the effect of a particular type of teaching method. We will describe the median test, which corresponds to the sign test, and the Wilcoxon rank sum test.

12.4.1 Median Test Let m1 and m2 be the medians of two populations 1 and 2, respectively, both with continuous distributions. Assume that we have a random sample of size n1 from population 1 and a random sample of size n2 from population 2. The median test can be summarized as follows.

12.4 Nonparametric Hypothesis Tests for Two Independent Samples 621

HYPOTHESIS TESTING PROCEDURE USING MEDIAN TEST We test m1 > m2 , upper tailed test H0 : m1 = m2

versus

Ha : m1 < m2 , lower tailed test m1 = m2 , two-tailed test.

1. Combine the two samples into a single sample of size n1 + n2 , keeping track of each observation’s original population. Arrange the n1 + n2 observations in increasing order and ﬁnd the median of this combined sample. If the median is one of the sample values, discard those observations and adjust the sample size accordingly. 2. Deﬁne N1b to be the number of observations of a sample from population 1 (under H0 we would expect this number to be around n1 /2). 3. Decision: If H0 is true, then we would expect N1b to be equal to some number around n1 /2. For Ha : m1 > m2 , rejection region is N1b ≤ c, where P(N1b ≤ c ) = α, for Ha : m1 < m2 , rejection region is N1b ≥ c, where P(N1b ≥ c ) = α, and for Ha : m1 = m2 , rejection region is N1b ≥ c1 , or N1b ≤ c2 , where

science & ELSEVIER technology books

Companion Web Site:

http://www.elsevierdirect.com/companions/9780123748485

Mathematical Statistics with Applications by Kandethody M. Ramachandran and Chris P. Tsokos

Resources for Professors: • • • •

All figures from the book available as PowerPoint slides and as jpegs. Links to Web sites carefully chosen to supplement the content of the textbook. Online Student Solutions Manual is now available through separate purchase. Also available with purchase of Mathematical Statistics with Applications, password protected and activated upon registration, online Instructors’ Solutions Manual.

TOOLS

FOR

TEACHING NEEDS ALL textbooks.elsevier.com YOUR

ACADEMIC PRESS To adopt this book for course use, visit http://textbooks.elsevier.com

Mathematical Statistics with Applications

Kandethody M.Ramachandran Department of Mathematics and Statistics University of South Florida Tampa,FL

Chris P.Tsokos Department of Mathematics and Statistics University of South Florida Tampa,FL

AMSTERDAM • BOSTON • HEIDELBERG • LONDON NEW YORK • OXFORD • PARIS • SAN DIEGO SAN FRANCISCO • SINGAPORE • SYDNEY • TOKYO

Academic Press is an imprint of Elsevier

Elsevier Academic Press 30 Corporate Drive, Suite 400, Burlington, MA 01803, USA 525 B Street, Suite 1900, San Diego, California 92101-4495, USA 84 Theobald’s Road, London WC1X 8RR, UK ∞ This book is printed on acid-free paper.

Copyright © 2009, Elsevier Inc. All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopy, recording, or any information storage and retrieval system, without permission in writing from the publisher. Permissions may be sought directly from Elsevier’s Science & Technology Rights Department in Oxford, UK: phone: (+44) 1865 843830, fax: (+44) 1865 853333, E-mail: [email protected] You may also complete your request on-line via the Elsevier homepage (http://elsevier.com), by selecting “Customer Support” and then “Obtaining Permissions.” Library of Congress Cataloging-in-Publication Data Ramachandran, K. M. Mathematical statistics with applications / Kandethody M. Ramachandran, Chris P. Tsokos. p. cm. ISBN 978-0-12-374848-5 (hardcover : alk. paper) 1. Mathematical statistics. 2. Mathematical statistics—Data processing. I. Tsokos, Chris P. II. Title. QA276.R328 2009 519.5–dc22 2008044556 British Library Cataloguing in Publication Data A catalogue record for this book is available from the British Library. ISBN 13: 978-0-12-374848-5 For all information on all Elsevier Academic Press publications visit our Web site at www.elsevierdirect.com

Printed in the United States of America 09 10 9 8 7 6 5 4 3 2 1

Dedicated to our families: Usha, Vikas, Vilas, and Varsha Ramachandran and Debbie, Matthew, Jonathan, and Maria Tsokos

This page intentionally left blank

Contents Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix About the Authors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Flow Chart . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxiii

CHAPTER 1 Descriptive Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.1.1 Data Collection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2 Basic Concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.2.1 Types of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3 Sampling Schemes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.1 Errors in Sample Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.3.2 Sample Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.4 Graphical Representation of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5 Numerical Description of Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.1 Numerical Measures for Grouped Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.5.2 Box Plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.6 Computers and Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1.8.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1 2 3 3 5 8 11 12 13 26 30 33 39 40 41 41 46 47 51

CHAPTER 2 Basic Concepts from Probability Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 2.1 2.2 2.3 2.4 2.5 2.6

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Random Events and Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Counting Techniques and Calculation of Probabilities . . . . . . . . . . . . . . . . . . . . . . . . The Conditional Probability, Independence, and Bayes’ Rule . . . . . . . . . . . . . . . . Random Variables and Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Moments and Moment-Generating Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.6.1 Skewness and Kurtosis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8 Computer Examples (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.1 Minitab Computations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2.8.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

54 55 63 71 83 92 98 107 108 109 110 110 112

vii

viii Contents

CHAPTER 3 Additional Topics in Probability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Special Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.1 The Binomial Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.2 Poisson Probability Distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.3 Uniform Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.4 Normal Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2.5 Gamma Probability Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3 Joint Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Covariance and Correlation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4 Functions of Random Variables. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.1 Method of Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.2 The pdf of Y = g(X), Where g Is Differentiable and Monotone Increasing or Decreasing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.3 Probability Integral Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.4 Functions of Several Random Variables: Method of Distribution Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.4.5 Transformation Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.5 Limit Theorems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7 Computer Examples (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 3 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

114 114 114 119 122 125 131 141 148 154 154 156 157 158 159 163 173 175 175 177 178 180

CHAPTER 4 Sampling Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.1.1 Finite Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 Sampling Distributions Associated with Normal Populations. . . . . . . . . . . . . . . . . 4.2.1 Chi-Square Distribution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.2 Student t-Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2.3 F-Distribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3 Order Statistics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4 Large Sample Approximations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.4.1 The Normal Approximation to the Binomial Distribution . . . . . . . . . . . 4.5 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.6.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

184 187 191 192 198 202 207 212 213 218 219 219 219 219 221

Contents ix

CHAPTER 5 Point Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225 5.1 5.2 5.3 5.4

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Method of Moments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . The Method of Maximum Likelihood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Some Desirable Properties of Point Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.1 Unbiased Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.4.2 Sufﬁciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5 Other Desirable Properties of a Point Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.1 Consistency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.2 Efﬁciency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.5.3 Minimal Sufﬁciency and Minimum-Variance Unbiased Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

226 227 235 246 247 252 266 266 270 277 282 283 285

CHAPTER 6 Interval Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291 6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.1.1 A Method of Finding the Conﬁdence Interval: Pivotal Method . . . . . . 6.2 Large Sample Conﬁdence Intervals: One Sample Case . . . . . . . . . . . . . . . . . . . . . . . 6.2.1 Conﬁdence Interval for Proportion, p . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2.2 Margin of Error and Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.3 Small Sample Conﬁdence Intervals for μ . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.4 A Conﬁdence Interval for the Population Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.5 Conﬁdence Interval Concerning Two Population Parameters . . . . . . . . . . . . . . . . . 6.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

292 293 300 302 303 310 315 321 330 330 330 332 333 334

CHAPTER 7 Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337 7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.1.1 Sample Size. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.2 The Neyman–Pearson Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.3 Likelihood Ratio Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4 Hypotheses for a Single Parameter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.1 The p-Value . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.4.2 Hypothesis Testing for a Single Parameter. . . . . . . . . . . . . . . . . . . . . . . . . . . .

338 346 349 355 361 361 363

x Contents

7.5 Testing of Hypotheses for Two Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.1 Independent Samples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.5.2 Dependent Samples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6 Chi-Square Tests for Count Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.1 Testing the Parameters of Multinomial Distribution: Goodness-of-Fit Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.2 Contingency Table: Test for Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.6.3 Testing to Identify the Probability Distribution: Goodness-of-Fit Chi-Square Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7.8.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 7 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

372 373 382 388 390 392 395 399 399 400 403 405 408

CHAPTER 8 Linear Regression Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 411 8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2 The Simple Linear Regression Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.1 The Method of Least Squares. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.2 Derivation of βˆ 0 and βˆ 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.3 Quality of the Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.4 Properties of the Least-Squares Estimators for the Model Y = β0 + β1 x + ε. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.2.5 Estimation of Error Variance σ 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3 Inferences on the Least Squares Estimators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.3.1 Analysis of Variance (ANOVA) Approach to Regression . . . . . . . . . . . . 8.4 Predicting a Particular Value of Y . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.5 Correlation Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6 Matrix Notation for Linear Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.6.1 ANOVA for Multiple Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.7 Regression Diagnostics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.8 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8.9.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 8 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

412 413 415 416 421 422 425 428 434 437 440 445 449 451 454 455 455 457 458 461

CHAPTER 9 Design of Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 465 9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 466 9.2 Concepts from Experimental Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467 9.2.1 Basic Terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 467

Contents xi

9.2.2

Fundamental Principles: Replication, Randomization, and Blocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.2.3 Some Speciﬁc Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3 Factorial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.1 One-Factor-at-a-Time Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.2 Full Factorial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.3.3 Fractional Factorial Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4 Optimal Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.4.1 Choice of Optimal Sample Size . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.5 The Taguchi Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9.7.2 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

471 474 483 483 485 486 487 487 489 493 494 494 494 497

CHAPTER 10 Analysis of Variance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 499 10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.2 Analysis of Variance Method for Two Treatments (Optional) . . . . . . . . . . . . . . . . . 10.3 Analysis of Variance for Completely Randomized Design . . . . . . . . . . . . . . . . . . . . 10.3.1 The p-Value Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.3.2 Testing the Assumptions for One-Way ANOVA . . . . . . . . . . . . . . . . . . . . . . 10.3.3 Model for One-Way ANOVA (Optional) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.4 Two-Way Analysis of Variance, Randomized Complete Block Design. . . . . . . 10.5 Multiple Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

500 501 510 515 517 522 526 536 543 543 543 546 548 554

CHAPTER 11 Bayesian Estimation and Inference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 559 11.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2 Bayesian Point Estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.2.1 Criteria for Finding the Bayesian Estimate . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.3 Bayesian Conﬁdence Interval or Credible Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.4 Bayesian Hypothesis Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.5 Bayesian Decision Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 11 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

560 562 569 579 584 588 596 596 596

xii Contents

CHAPTER 12 Nonparametric Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 599 12.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.2 Nonparametric Conﬁdence Interval . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3 Nonparametric Hypothesis Tests for One Sample . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.1 The Sign Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.2 Wilcoxon Signed Rank Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.3.3 Dependent Samples: Paired Comparison Tests . . . . . . . . . . . . . . . . . . . . . . . 12.4 Nonparametric Hypothesis Tests for Two Independent Samples. . . . . . . . . . . . . . 12.4.1 Median Test. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.4.2 The Wilcoxon Rank Sum Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5 Nonparametric Hypothesis Tests for k ≥ 2 Samples . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.1 The Kruskal–Wallis Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.5.2 The Friedman Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7.1 Minitab Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7.2 SPSS Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12.7.3 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 12. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

600 601 606 607 611 617 620 620 625 630 631 634 640 642 642 646 648 652

CHAPTER 13 Empirical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 657 13.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.2 The Jackknife Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3 An Introduction to Bootstrap Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.3.1 Bootstrap Conﬁdence Intervals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.4 The Expectation Maximization Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5 Introduction to Markov Chain Monte Carlo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.1 Metropolis Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.2 The Metropolis–Hastings Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.3 Gibbs Algorithm. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.5.4 MCMC Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7 Computer Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13.7.1 SAS Examples. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Projects for Chapter 13. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

658 658 663 667 669 681 685 688 692 695 697 698 699 699

CHAPTER 14 Some Issues in Statistical Applications: An Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 701 14.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.2 Graphical Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.3 Outliers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4 Checking Assumptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.1 Checking the Assumption of Normality. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.2 Data Transformation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

702 702 708 713 714 716

Contents xiii

14.4.3 Test for Equality of Variances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.4.4 Test of Independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5 Modeling Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.1 A Simple Model for Univariate Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.5.2 Modeling Bivariate Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.6 Parametric versus Nonparametric Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.7 Tying It All Together . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

719 724 727 727 730 733 735 746

Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 747 A.I A.II A.III A.IV

Set Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Review of Markov Chains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Common Probability Distributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Probability Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

747 751 757 759

References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 799 Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 803

This page intentionally left blank

Preface This textbook is of an interdisciplinary nature and is designed for a two- or one-semester course in probability and statistics, with basic calculus as a prerequisite. The book is primarily written to give a sound theoretical introduction to statistics while emphasizing applications. If teaching statistics is the main purpose of a two-semester course in probability and statistics, this textbook covers all the probability concepts necessary for the theoretical development of statistics in two chapters, and goes on to cover all major aspects of statistical theory in two semesters, instead of only a portion of statistical concepts. What is more, using the optional section on computer examples at the end of each chapter, the student can also simultaneously learn to utilize statistical software packages for data analysis. It is our aim, without sacriﬁcing any rigor, to encourage students to apply the theoretical concepts they have learned. There are many examples and exercises concerning diverse application areas that will show the pertinence of statistical methodology to solving real-world problems. The examples with statistical software and projects at the end of the chapters will provide good perspective on the usefulness of statistical methods. To introduce the students to modern and increasingly popular statistical methods, we have introduced separate chapters on Bayesian analysis and empirical methods. One of the main aims of this book is to prepare advanced undergraduates and beginning graduate students in the theory of statistics with emphasis on interdisciplinary applications. The audience for this course is regular full-time students from mathematics, statistics, engineering, physical sciences, business, social sciences, materials science, and so forth. Also, this textbook is suitable for people who work in industry and in education as a reference book on introductory statistics for a good theoretical foundation with clear indication of how to use statistical methods. Traditionally, one of the main prerequisites for this course is a semester of the introduction to probability theory. A working knowledge of elementary (descriptive) statistics is also a must. In schools where there is no statistics major, imposing such a background, in addition to calculus sequence, is very difﬁcult. Most of the present books available on this subject contain full one-semester material for probability and then, based on those results, continue on to the topics in statistics. Also, some of these books include in their subject matter only the theory of statistics, whereas others take the cookbook approach of covering the mechanics. Thus, even with two full semesters of work, many basic and important concepts in statistics are never covered. This book has been written to remedy this problem. We fuse together both concepts in order for students to gain knowledge of the theory and at the same time develop the expertise to use their knowledge in real-world situations. Although statistics is a very applied subject, there is no denying that it is also a very abstract subject. The purpose of this book is to present the subject matter in such a way that anyone with exposure to basic calculus can study statistics without spending two semesters of background preparation. To prepare students, we present an optional review of the elementary (descriptive) statistics in Chapter 1. All the probability material required to learn statistics is covered in two chapters. Students with a probability background can either review or skip the ﬁrst three chapters. It is also our belief that any statistics course is not complete without exposure to computational techniques. At

xv

xvi Preface

the end of each chapter, we give some examples of how to use Minitab, SPSS, and SAS to statistically analyze data. Also, at the end of each chapter, there are projects that will enhance the knowledge and understanding of the materials covered in that chapter. In the chapter on the empirical methods, we present some of the modern computational and simulation techniques, such as bootstrap, jackknife, and Markov chain Monte Carlo methods. The last chapter summarizes some of the steps necessary to apply the material covered in the book to real-world problems. The ﬁrst eight chapters have been class tested as a one-semester course for more than 3 years with ﬁve different professors teaching. The audience was junior- and senior-level undergraduate students from many disciplines who had had two semesters of calculus, most of them with no probability or statistics background. The feedback from the students and instructors was very positive. Recommendations from the instructors and students were very useful in improving the style and content of the book.

AIM AND OBJECTIVE OF THE TEXTBOOK This textbook provides a calculus-based coverage of statistics and introduces students to methods of theoretical statistics and their applications. It assumes no prior knowledge of statistics or probability theory, but does require calculus. Most books at this level are written with elaborate coverage of probability. This requires teaching one semester of probability and then continuing with one or two semesters of statistics. This creates a particular problem for non-statistics majors from various disciplines who want to obtain a sound background in mathematical statistics and applications. It is our aim to introduce basic concepts of statistics with sound theoretical explanations. Because statistics is basically an interdisciplinary applied subject, we offer many applied examples and relevant exercises from different areas. Knowledge of using computers for data analysis is desirable. We present examples of solving statistical problems using Minitab, SPSS, and SAS.

FEATURES ■

■ ■

■

■

During years of teaching, we observed that many students who do well in mathematics courses ﬁnd it difﬁcult to understand the concept of statistics. To remedy this, we present most of the material covered in the textbook with well-deﬁned step-by-step procedures to solve real problems. This clearly helps the students to approach problem solving in statistics more logically. The usefulness of each statistical method introduced is illustrated by several relevant examples. At the end of each section, we provide ample exercises that are a good mix of theory and applications. In each chapter, we give various projects for students to work on. These projects are designed in such a way that students will start thinking about how to apply the results they learned in the chapter as well as other issues they will need to know for practical situations. At the end of the chapters, we include an optional section on computer methods with Minitab, SPSS, and SAS examples with clear and simple commands that the student can use to analyze

Preface xvii

■

■

■

■

■

■ ■

data. This will help students to learn how to utilize the standard methods they have learned in the chapter to study real data. We introduce many of the modern statistical computational and simulation concepts, such as the jackknife and bootstrap methods, the EM algorithms, and the Markov chain Monte Carlo methods such as the Metropolis algorithm, the Metropolis–Hastings algorithm, and the Gibbs sampler. The Metropolis algorithm was mentioned in Computing in Science and Engineering as being among the top 10 algorithms having the “greatest inﬂuence on the development and practice of science and engineering in the 20th century.” We have introduced the increasingly popular concept of Bayesian statistics and decision theory with applications. A separate chapter on design of experiments, including a discussion on the Taguchi approach, is included. The coverage of the book spans most of the important concepts in statistics. Learning the material along with computational examples will prepare students to understand and utilize software procedures to perform statistical analysis. Every chapter contains discussion on how to apply the concepts and what the issues are related to applying the theory. A student’s solution manual, instructor’s manual, and data disk are provided. In the last chapter, we discuss some issues in applications to clearly demonstrate in a uniﬁed way how to check for many assumptions in data analysis and what steps one needs to follow to avoid possible pitfalls in applying the methods explained in the rest of this textbook.

This page intentionally left blank

Acknowledgments We express our sincere appreciation to our late colleague, co-worker, and dear friend, Professor A. N. V. Rao, for his helpful suggestions and ideas for the initial version of the subject textbook. In addition, we thank Bong-jin Choi and Yong Xu for their kind assistance in the preparation of the manuscript. Finally, we acknowledge our students at the University of South Florida for their useful comments and suggestions during the class testing of our book. To all of them, we are very thankful. K. M. Ramachandran Chris P. Tsokos Tampa, Florida

xix

This page intentionally left blank

About the Authors Kandethody M. Ramachandran is Professor of Mathematics and Statistics at the University of South Florida. He received his B.S. and M.S. degrees in Mathematics from the Calicut University, India. Later, he worked as a researcher at the Tata Institute of Fundamental Research, Bangalore center, at its Applied Mathematics Division. Dr. Ramachandran got his Ph.D. in Applied Mathematics from Brown University. His research interests are concentrated in the areas of applied probability and statistics. His research publications span a variety of areas such as control of heavy trafﬁc queues, stochastic delay equations and control problems, stochastic differential games and applications, reinforcement learning methods applied to game theory and other areas, software reliability problems, applications of statistical methods to microarray data analysis, and mathematical ﬁnance. Professor Ramachandran is extensively involved in activities to improve statistics and mathematics education. He is a recipient of the Teaching Incentive Program award at the University of South Florida. He is a member of the MEME Collaborative, which is a partnership among mathematics education, mathematics, and engineering faculty to address issues related to mathematics and mathematics education. He was also involved in the calculus reform efforts at the University of South Florida. Chris P. Tsokos is Distinguished University Professor of Mathematics and Statistics at the University of South Florida. Dr. Tsokos received his B.S. in Engineering Sciences/Mathematics, his M.A. in Mathematics from the University of Rhode Island, and his Ph.D. in Statistics and Probability from the University of Connecticut. Professor Tsokos has also served on the faculties at Virginia Polytechnic Institute and State University and the University of Rhode Island. Dr. Tsokos’s research has extended into a variety of areas, including stochastic systems, statistical models, reliability analysis, ecological systems, operations research, time series, Bayesian analysis, and mathematical and statistical modeling of global warming, among others. He is the author of more than 250 research publications in these areas. Professor Tsokos is the author of several research monographs and books, including Random Integral Equations with Applications to Life Sciences and Engineering, Probability Distribution: An Introduction to Probability Theory with Applications, Mainstreams of Finite Mathematics with Applications, Probability with the Essential Analysis, and Applied Probability Bayesian Statistical Methods with Applications to Reliability, among others. Dr. Tsokos is the recipient of many distinguished awards and honors, including Fellow of the American Statistical Association, USF Distinguished Scholar Award, Sigma Xi Outstanding Research Award, USF Outstanding Undergraduate Teaching Award, USF Professional Excellence Award, URI Alumni Excellence Award in Science and Technology, Pi Mu Epsilon, and election to the International Statistical Institute, among others.

xxi

This page intentionally left blank

Flow Chart This ﬂow chart gives some options on how to use the book in a one-semester or two-semester course. For a two-semester course, we recommend coverage of the complete textbook. However, Chapters 1, 9, and 14 are optional for both one- and two-semester courses and can be given as reading exercises. For a one-semester course, we suggest the following options: A, B, C, D.

One semester

Without probability background

With probability background

Ch. 2 A

B

C

D Ch. 3

Ch. 5

Ch. 5

Ch. 5

Ch. 6

Ch. 6

Ch. 6

Ch. 7

Ch. 7

Ch. 7

Ch. 8

Ch. 8

Ch. 8

Ch. 10

Ch.12

Ch. 11

Ch. 5 Ch. 4 Ch. 6 Ch. 5 Ch. 7 Ch. 6

Ch. 8 Ch. 11

Ch. 7 Ch. 12

Ch. 13

Ch. 13

Optional chapters

Ch. 12

Ch. 8

Ch. 10

Ch. 11

Ch. 12

xxiii

This page intentionally left blank

Chapter

1

Descriptive Statistics Objective: Review the basic concepts of elementary statistics. 1.1 Introduction 2 1.2 Basic Concepts 3 1.3 Sampling Schemes 8 1.4 Graphical Representation of Data 13 1.5 Numerical Description of Data 26 1.6 Computers and Statistics 39 1.7 Chapter Summary 40 1.8 Computer Examples 41 Projects for Chapter 1 51

Sir Ronald Aylmer Fisher (Source: http://www.stetson.edu/∼efriedma/periodictable/jpg/Fisher.jpg)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

1

2 CHAPTER 1 Descriptive Statistics

Sir Ronald Fisher F.R.S. (1890–1962) was one of the leading scientists of the 20th century who laid the foundations for modern statistics. As a statistician working at the Rothamsted Agricultural Experiment Station, the oldest agricultural research institute in the United Kingdom, he also made major contributions to Evolutionary Biology and Genetics. The concept of randomization and the analysis of variance procedures that he introduced are now used throughout the world. In 1922 he gave a new deﬁnition of statistics. Fisher identiﬁed three fundamental problems in statistics: (1) speciﬁcation of the type of population that the data came from; (2) estimation; and (3) distribution. His book Statistical Methods for Research Workers (1925) was used as a handbook for the methods for the design and analysis of experiments. Fisher also published the books titled The Design of Experiments (1935) and Statistical Tables (1947). While at the Agricultural Experiment Station he had conducted breeding experiments with mice, snails, and poultry, and the results he obtained led to theories about gene dominance and ﬁtness that he published in The Genetical Theory of Natural Selection (1930).

1.1 INTRODUCTION In today’s society, decisions are made on the basis of data. Most scientiﬁc or industrial studies and experiments produce data, and the analysis of these data and drawing useful conclusions from them become one of the central issues. The ﬁeld of statistics is concerned with the scientiﬁc study of collecting, organizing, analyzing, and drawing conclusions from data. Statistical methods help us to transform data to knowledge. Statistical concepts enable us to solve problems in a diversity of contexts, add substance to decisions, and reduce guesswork. The discipline of statistics stemmed from the need to place knowledge management on a systematic evidence base. Earlier works on statistics dealt only with the collection, organization, and presentation of data in the form of tables and charts. In order to place statistical knowledge on a systematic evidence base, we require a study of the laws of probability. In mathematical statistics we create a probabilistic model and view the data as a set of random outcomes from that model. Advances in probability theory enable us to draw valid conclusions and to make reasonable decisions on the basis of data. Statistical methods are used in almost every discipline, including agriculture, astronomy, biology, business, communications, economics, education, electronics, geology, health sciences, and many other ﬁelds of science and engineering, and can aid us in several ways. Modern applications of statistical techniques include statistical communication theory and signal processing, information theory, network security and denial of service problems, clinical trials, artiﬁcial and biological intelligence, quality control of manufactured items, software reliability, and survival analysis. The ﬁrst of these is to assist us in designing experiments and surveys. We desire our experiment to yield adequate answers to the questions that prompted the experiment or survey. We would like the answers to have good precision without involving a lot of expenditure. Statistically designed experiments facilitate development of robust products that are insensitive to changes in the environment and internal component variation. Another way that statistics assists us is in organizing, describing, summarizing, and displaying experimental data. This is termed descriptive statistics. A third use of statistics is in drawing inferences and making decisions based on data. For example, scientists may collect experimental data to prove or disprove an intuitive conjecture or hypothesis. Through the proper use of statistics we can conclude whether the hypothesis is valid or not. In the process of solving a real-life problem using statistics, the following three basic steps may be identiﬁed. First, consistent with the objective of the problem,

1.2 Basic Concepts 3

we identify the model—the appropriate statistical method. Then, we justify the applicability of the selected model to fulﬁll the aim of our problem. Last, we properly apply the related model to analyze the data and make the necessary decisions, which results in answering the question of our problem with minimum risk. Starting with Chapter 2, we will study the necessary background material to proceed with the development of statistical methods for solving real-world problems. In the present chapter we brieﬂy review some of the basic concepts of descriptive statistics. Such concepts will give us a visual and descriptive presentation of the problem under investigation. Now, we proceed with some basic deﬁnitions.

1.1.1 Data Collection One of the ﬁrst problems that a statistician faces is obtaining data. The inferences that we make depend critically on the data that we collect and use. Data collection involves the following important steps.

GENERAL PROCEDURE FOR DATA COLLECTION 1. Deﬁne the objectives of the problem and proceed to develop the experiment or survey. 2. Deﬁne the variables or parameters of interest. 3. Deﬁne the procedures of data-collection and measuring techniques. This includes sampling procedures, sample size, and data-measuring devices (questionnaires, telephone interviews, etc.).

Example 1.1.1 We may be interested in estimating the average household income in a certain community. In this case, the parameter of interest is the average income of a typical household in the community. To acquire the data, we may send out a questionnaire or conduct a telephone interview. Once we have the data, we may ﬁrst want to represent the data in graphical or tabular form to better understand its distributional behavior. Then we will use appropriate analytical techniques to estimate the parameter(s) of interest, in this case the average household income.

Very often a statistician is conﬁned to data that have already been collected, possibly even collected for other purposes. This makes it very difﬁcult to determine the quality of data. Planned collection of data, using proper techniques, is much preferred.

1.2 BASIC CONCEPTS Statistics is the science of data. This involves collecting, classifying, summarizing, organizing, analyzing, and interpreting data. It also involves model building. Suppose we wish to study household incomes in a certain neighborhood. We may decide to randomly select, say, 50 families and examine their household incomes. As another example, suppose we wish to determine the diameter of a rod, and we take 10 measurements of the diameter. When we consider these two examples, we note that in the ﬁrst case the population (the household incomes of all families in the neighborhood) really exists, whereas in the second, the population (set of all possible measurements of the diameter) is

4 CHAPTER 1 Descriptive Statistics

only conceptual. In either case we can visualize the totality of the population values, of which our sample data are only a small part. Thus we deﬁne a population to be the set of all measurements or objects that are of interest and a sample to be a subset of that population. The population acts as the sampling frame from which a sample is selected. Now we introduce some basic notions commonly used in statistics. Deﬁnition 1.2.1 A population is the collection or set of all objects or measurements that are of interest to the collector.

Example 1.2.1 Suppose we wish to study the heights of all female students at a certain university. The population will be the set of the measured heights of all female students in the university. The population is not the set of all female students in the university.

In real-world problems it is usually not possible to obtain information on the entire population. The primary objective of statistics is to collect and study a subset of the population, called a sample, to acquire information on some speciﬁc characteristics of the population that are of interest. Deﬁnition 1.2.2 The sample is a subset of data selected from a population. The size of a sample is the number of elements in it.

Example 1.2.2 We wish to estimate the percentage of defective parts produced in a factory during a given week (ﬁve days) by examining 20 parts produced per day. The parts will be examined each day at randomly chosen times. In this case “all parts produced during the week” is the population and the (100) selected parts for ﬁve days constitutes a sample.

Other common examples of sample and population are: Political polls: The population will be all voters, whereas the sample will be the subset of voters we poll. Laboratory experiment: The population will be all the data we could have collected if we were to repeat the experiment a large number of times (inﬁnite number of times) under the same conditions, whereas the sample will be the data actually collected by the one experiment. Quality control: The population will be the entire batch of items produced, say, by a machine or by a plant, whereas the sample will be the subset of items we tested. Clinical studies: The population will be all the patients with the same disease, whereas the sample will be the subset of patients used in the study. Finance: All common stock listed in stock exchanges such as the New York Stock Exchange, the American Stock Exchanges, and over-the-counter is the population. A collection of 20 randomly picked individual stocks from these exchanges will be a sample.

1.2 Basic Concepts 5

The methods consisting mainly of organizing, summarizing, and presenting data in the form of tables, graphs, and charts are called descriptive statistics. The methods of drawing inferences and making decisions about the population using the sample are called inferential statistics. Inferential statistics uses probability theory. Deﬁnition 1.2.3 A statistical inference is an estimate, a prediction, a decision, or a generalization about the population based on information contained in a sample. For example, we may be interested in the average indoor radiation level in homes built on reclaimed phosphate mine lands (many of the homes in west-central Florida are built on such lands). In this case, we can collect indoor radiation levels for a random sample of homes selected from this area, and use the data to infer the average indoor radiation level for the entire region. In the Florida Keys, one of the concerns is that the coral reefs are declining because of the prevailing ecosystems. In order to test this, one can randomly select certain reef sites for study and, based on these data, infer whether there is a net increase or decrease in coral reefs in the region. Here the inferential problem could be ﬁnding an estimate, such as in the radiation problem, or making a decision, such as in the coral reef problem. We will see many other examples as we progress through the book.

1.2.1 Types of Data Data can be classiﬁed in several ways. We will give two different classiﬁcations, one based on whether the data are measured on a numerical scale or not, and the other on whether the data are collected in the same time period or collected at different time periods. Deﬁnition 1.2.4 Quantitative data are observations measured on a numerical scale. Nonnumerical data that can only be classiﬁed into one of the groups of categories are said to be qualitative or categorical data.

Example 1.2.3 Data on response to a particular therapy could be classiﬁed as no improvement, partial improvement, or complete improvement. These are qualitative data. The number of minority-owned businesses in Florida is quantitative data. The marital status of each person in a statistics class as married or not married is qualitative or categorical data. The number of car accidents in different U.S. cities is quantitative data. The blood group of each person in a community as O, A, B, AB is qualitative data.

Categorical data could be further classiﬁed as nominal data and ordinal data. Data characterized as nominal have data groups that do not have a speciﬁc order. An example of this could be state names, or names of the individuals, or courses by name. These do not need to be placed in any order. Data characterized as ordinal have groups that should be listed in a speciﬁc order. The order may be either increasing or decreasing. One example would be income levels. The data could have numeric values such as 1, 2, 3, or values such as high, medium, or low. Deﬁnition 1.2.5 Cross-sectional data are data collected on different elements or variables at the same point in time or for the same period of time.

6 CHAPTER 1 Descriptive Statistics

Example 1.2.4 The data in Table 1.1 represent U.S. federal support for the mathematical sciences in 1996, in millions of dollars (source: AMS Notices). This is an example of cross-sectional data, as the data are collected in one time period, namely in 1996.

Table 1.1 Federal Support for the Mathematical Sciences, 1996 Federal agency

Amount

National Science Foundation

91.70

DMS

85.29

Other MPS

4.00

Department of Defense

77.30

AFOSR

16.70

ARO

15.00

DARPA

22.90

NSA

2.50

ONR

20.20

Department of Energy

16.00

University Support National Laboratories Total, All Agencies

5.50 10.50 185.00

Deﬁnition 1.2.6 Time series data are data collected on the same element or the same variable at different points in time or for different periods of time.

Example 1.2.5 The data in Table 1.2 represent U.S. federal support for the mathematical sciences during the years 1995–1997, in millions of dollars (source: AMS Notices). This is an example of time series data, because they have been collected at different time periods, 1995 through 1997.

For an extensive collection of statistical terms and deﬁnitions, we can refer to many sources such as http://www.stats.gla.ac.uk/steps/glossary/index.html. We will give some other helpful Internet sources that may be useful for various aspects of statistics: http://www.amstat.org/ (American

1.2 Basic Concepts 7

Table 1.2 United States Federal Support for the Mathematical Sciences in Different Years Agency

1995

1996

1997

National Science Foundation

87.69

91.70

98.22

DMS

85.29

87.70

93.22

2.40

4.00

5.00

Department of Defense

77.40

77.30

67.80

AFOSR

17.40

16.70

17.10

ARO

15.00

15.00

13.00

DARPA

21.00

22.90

19.50

NSA

2.50

2.50

2.10

ONR

21.40

20.20

16.10

Department of Energy

15.70

16.00

16.00

University Support

6.20

5.50

5.00

National Laboratories

9.50

10.50

11.00

180.79

185.00

182.02

Other MPS

Total, All Agencies

Statistical Association), http://www.stat.uﬂ.edu (University of Florida statistics department), http://www.stats.gla.ac.uk/cti/ (collection of Web links to other useful statistics sites), http://www. statsoft.com/textbook/stathome.html (covers a wide range of topics, the emphasis is on techniques rather than concepts or mathematics), http://www.york.ac.uk/depts/maths/histstat/welcome.htm (some information about the history of statistics), http://www.isid.ac.in/ (Indian Statistical Institute), http://www.math.uio.no/nsf/web/index.htm (The Norwegian Statistical Society), http://www.rss.org.uk/ (The Royal Statistical Society), http://lib.stat.cmu.edu/ (an index of statistical software and routines). For energy-related statistics, refer to http://www.eia.doe.gov/. There are various other useful sites that you could explore based on your particular need.

EXERCISES 1.2 1.2.1.

Give your own examples for qualitative and quantitative data. Also, give examples for crosssectional and time series data.

1.2.2.

Discuss how you will collect different types of data. What inferences do you want to derive from each of these types of data?

1.2.3.

Refer to the data in Example 1.2.4. State a few questions that you can ask about the data. What inferences can you make by looking at these data?

8 CHAPTER 1 Descriptive Statistics

1.2.4.

Refer to the data in Example 1.2.5. Can you state a few questions that the data suggest? What inferences can you make by looking at these data?

1.3 SAMPLING SCHEMES In any statistical analysis, it is important that we clearly deﬁne the target population. The population should be deﬁned in keeping with the objectives of the study. When the entire population is included in the study, it is called a census study because data are gathered on every member of the population. In general, it is usually not possible to obtain information on the entire population because the population is too large to attempt a survey of all of its members, or it may not be cost effective. A small but carefully chosen sample can be used to represent the population. A sample is obtained by collecting information from only some members of the population. A good sample must reﬂect all the characteristics (of importance) of the population. Samples can reﬂect the important characteristics of the populations from which they are drawn with differing degrees of precision. A sample that accurately reﬂects its population characteristics is called a representative sample. A sample that is not representative of the population characteristics is called a biased sample. The reliability or accuracy of conclusions drawn concerning a population depends on whether or not the sample is properly chosen so as to represent the population sufﬁciently well. There are many sampling methods available. We mention a few commonly used simple sampling schemes. The choice between these sampling methods depends on (1) the nature of the problem or investigation, (2) the availability of good sampling frames (a list of all of the population members), (3) the budget or available ﬁnancial resources, (4) the desired level of accuracy, and (5) the method by which data will be collected, such as questionnaires or interviews. Deﬁnition 1.3.1 A sample selected in such a way that every element of the population has an equal chance of being chosen is called a simple random sample. Equivalently each possible sample of size n has an equal chance of being selected.

Example 1.3.1 For a state lottery, 52 identical Ping-Pong balls with a number from 1 to 52 painted on each ball are put in a clear plastic bin. A machine thoroughly mixes the balls and then six are selected. The six numbers on the chosen balls are the six lottery numbers that have been selected by a simple random sampling procedure.

SOME ADVANTAGES OF SIMPLE RANDOM SAMPLING 1. Selection of sampling observations at random ensures against possible investigator biases. 2. Analytic computations are relatively simple, and probabilistic bounds on errors can be computed in many cases. 3. It is frequently possible to estimate the sample size for a prescribed error level when designing the sampling procedure.

1.3 Sampling Schemes 9

Simple random sampling may not be effective in all situations. For example, in a U.S. presidential election, it may be more appropriate to conduct sampling polls by state, rather than a nationwide random poll. It is quite possible for a candidate to get a majority of the popular vote nationwide and yet lose the election. We now describe a few other sampling methods that may be more appropriate in a given situation. Deﬁnition 1.3.2 A systematic sample is a sample in which every Kth element in the sampling frame is selected after a suitable random start for the ﬁrst element. We list the population elements in some order (say alphabetical) and choose the desired sampling fraction. STEPS FOR SELECTING A SYSTEMATIC SAMPLE 1. Number the elements of the population from 1 to N. 2. Decide on the sample size, say n, that we need. 3. Choose K = N/n. 4. Randomly select an integer between 1 to K . 5. Then take every K th element.

Example 1.3.2 If the population has 1000 elements arranged in some order and we decide to sample 10% (i.e., N = 1000 and n = 100), then K = 1000/100 = 10. Pick a number at random between 1 and K = 10 inclusive, say 3. Then select elements numbered 3, 13, 23, . . . , 993.

Systematic sampling is widely used because it is easy to implement. If the list of population elements is in random order to begin with, then the method is similar to simple random sampling. If, however, there is a correlation or association between successive elements, or if there is some periodic structure, then this sampling method may introduce biases. Systematic sampling is often used to select a speciﬁed number of records from a computer ﬁle. Deﬁnition 1.3.3 A stratiﬁed sample is a modiﬁcation of simple random sampling and systematic sampling and is designed to obtain a more representative sample, but at the cost of a more complicated procedure. Compared to random sampling, stratiﬁed sampling reduces sampling error. A sample obtained by stratifying (dividing into nonoverlapping groups) the sampling frame based on some factor or factors and then selecting some elements from each of the strata is called a stratiﬁed sample. Here, a population with N elements is divided into s subpopulations. A sample is drawn from each subpopulation independently. The size of each subpopulation and sample sizes in each subpopulation may vary. STEPS FOR SELECTING A STRATIFIED SAMPLE 1. Decide on the relevant stratiﬁcation factors (sex, age, income, etc.). 2. Divide the entire population into strata (subpopulations) based on the stratiﬁcation criteria. Sizes of strata may vary.

10 CHAPTER 1 Descriptive Statistics

3. Select the requisite number of units using simple random sampling or systematic sampling from each subpopulation. The requisite number may depend on the subpopulation sizes.

Examples of strata might be males and females, undergraduate students and graduate students, managers and nonmanagers, or populations of clients in different racial groups such as African Americans, Asians, whites, and Hispanics. Stratiﬁed sampling is often used when one or more of the strata in the population have a low incidence relative to the other strata.

Example 1.3.3 In a population of 1000 children from an area school, there are 600 boys and 400 girls. We divide them into strata based on their parents’ income as shown in Table 1.3.

Table 1.3 Classiﬁcation of School Children Boys

Girls

Poor

120

240

Middle Class

150

100

Rich

330

60

This is stratiﬁed data.

Example 1.3.4 Refer to Example 1.3.3. Suppose we decide to sample 100 children from the population of 1000 (that is, 10% of the population). We also choose to sample 10% from each of the categories. For example, we would choose 12 (10% of 120) poor boys; 6 (10% of 60 rich girls) and so forth. This yields Table 1.4. This particular sampling method is called a proportional stratiﬁed sampling.

Table 1.4 Proportional Stratiﬁcation of School Children Boys Girls Poor

12

24

Middle Class

15

10

Rich

33

6

1.3 Sampling Schemes 11

SOME USES OF STRATIFIED SAMPLING 1. In addition to providing information about the whole population, this sampling scheme provides information about the subpopulations, the study of which may be of interest. For example, in a U.S. presidential election, opinion polls by state may be more important in deciding on the electoral college advantage than a national opinion poll. 2. Stratiﬁed sampling can be considerably more precise than a simple random sample, because the population is fairly homogeneous within each stratum but there is a sizable variation between the strata.

Deﬁnition 1.3.4 In cluster sampling, the sampling unit contains groups of elements called clusters instead of individual elements of the population. A cluster is an intact group naturally available in the ﬁeld. Unlike the stratiﬁed sample where the strata are created by the researcher based on stratiﬁcation variables, the clusters naturally exist and are not formed by the researcher for data collection. Cluster sampling is also called area sampling. To obtain a cluster sample, ﬁrst take a simple random sample of groups and then sample all elements within the selected clusters (groups). Cluster sampling is convenient to implement. However, because it is likely that units in a cluster will be relatively homogeneous, this method may be less precise than simple random sampling.

Example 1.3.5 Suppose we wish to select a sample of about 10% from all ﬁfth-grade children of a county. We randomly select 10% of the elementary schools assumed to have approximately the same number of ﬁfth-grade students and select all ﬁfth-grade children from these schools. This is an example of cluster sampling, each cluster being an elementary school that was selected.

Deﬁnition 1.3.5 Multiphase sampling involves collection of some information from the whole sample and additional information either at the same time or later from subsamples of the whole sample. The multiphase or multistage sampling is basically a combination of the techniques presented earlier.

Example 1.3.6 An investigator in a population census may ask basic questions such as sex, age, or marital status for the whole population, but only 10% of the population may be asked about their level of education or about how many years of mathematics and science education they had.

1.3.1 Errors in Sample Data Irrespective of which sampling scheme is used, the sample observations are prone to various sources of error that may seriously affect the inferences about the population. Some sources of error can be controlled. However, others may be unavoidable because they are inherent in the nature of the sampling process. Consequently, it is necessary to understand the different types of errors for a proper

12 CHAPTER 1 Descriptive Statistics

interpretation and analysis of the sample data. The errors can be classiﬁed as sampling errors and nonsampling errors. Nonsampling errors occur in the collection, recording and processing of sample data. For example, such errors could occur as a result of bias in selection of elements of the sample, poorly designed survey questions, measurement and recording errors, incorrect responses, or no responses from individuals selected from the population. Sampling errors occur because the sample is not an exact representative of the population. Sampling error is due to the differences between the characteristics of the population and those of a sample from the population. For example, we are interested in the average test score in a large statistics class of size, say, 80. A sample of size 10 grades from this resulted in an average test score of 75. If the average test for the entire 80 students (the population) is 72, then the sampling error is 75 − 72 = 3.

1.3.2 Sample Size In almost any sampling scheme designed by statisticians, one of the major issues is the determination of the sample size. In principle, this should depend on the variation in the population as well as on the population size, and on the required reliability of the results, that is, the amount of error that can be tolerated. For example, if we are taking a sample of school children from a neighborhood with a relatively homogeneous income level to study the effect of parents’ afﬂuence on the academic performance of the children, it is not necessary to have a large sample size. However, if the income level varies a great deal in the feeding area of the school, then we will need a larger sample size to achieve the same level of reliability. In practice, another inﬂuencing factor is the available resources such as money and time. In later chapters, we present some methods of determining sample size in statistical estimation problems. The literature on sample survey methods is constantly changing with new insights that demand dramatic revisions in the conventional thinking. We know that representative sampling methods are essential to permit conﬁdent generalizations of results to populations. However, there are many practical issues that can arise in real-life sampling methods. For example, in sampling related to social issues, whatever the sampling method we employ, a high response rate must be obtained. It has been observed that most telephone surveys have difﬁculty in achieving response rates higher than 60%, and most face-to-face surveys have difﬁculty in achieving response rates higher than 70%. Even a well-designed survey may stop short of the goal of a perfect response rate. This might induce bias in the conclusions based on the sample we obtained. A low response rate can be devastating to the reliability of a study. We can obtain series of publications on surveys, including guidelines on avoiding pitfalls from the American Statistical Association (www.amstat.org). In this book, we deal mainly with samples obtained using simple random sampling.

EXERCISES 1.3 1.3.1.

Give your own examples for each of the sampling methods described in this section. Discuss the merits and limitations of each of these methods.

1.3.2.

Using the information obtained from the publications of the American Statistical Association (www.amstat.org), write a short report on how to collect survey data, and what the potential sources of error are.

1.4 Graphical Representation of Data 13

1.4 GRAPHICAL REPRESENTATION OF DATA The source of our statistical knowledge lies in the data. Once we obtain the sample data values, one way to become acquainted with them is to display them in tables or graphically. Charts and graphs are very important tools in statistics because they communicate information visually. These visual displays may reveal the patterns of behavior of the variables being studied. In this chapter, we will consider one-variable data. The most common graphical displays are the frequency table, pie chart, bar graph, Pareto chart, and histogram. For example, in the business world, graphical representations of data are used as statistical tools for everyday process management and improvements by decision makers (such as managers, and frontline staff) to understand processes, problems, and solutions. The purpose of this section is to introduce several tabular and graphical procedures commonly used to summarize both qualitative and quantitative data. Tabular and graphical summaries of data can be found in reports, newspaper articles, Web sites, and research studies, among others. Now we shall introduce some ways of graphically representing both qualitative and quantitative data. Bar graphs and Pareto charts are useful displays for qualitative data. Deﬁnition 1.4.1 A graph of bars whose heights represent the frequencies (or relative frequencies) of respective categories is called a bar graph.

Example 1.4.1 The data in Table 1.5 represent the percentages of price increases of some consumer goods and services for the period December 1990 to December 2000 in a certain city. Construct a bar chart for these data.

Table 1.5 Percentages of Price Increases of Some Consumer Goods and Services Medical Care

83.3%

Electricity

22.1%

Residential Rent

43.5%

Food

41.1%

Consumer Price Index

35.8%

Apparel & Upkeep

21.2%

Solution In the bar graph of Figure 1.1, we use the notations MC for medical care, El for electricity, RR for residential rent, Fd for food, CPI for consumer price index, and A & U for apparel and upkeep.

14 CHAPTER 1 Descriptive Statistics

100

Percentage

80 60 40 20 0

MC

EI

RR Fd Category

CPI

A&U

■ FIGURE 1.1 Percentage price increase of consumer goods.

Looking at Figure 1.1, we can identify where the maximum and minimum responses are located, so that we can descriptively discuss the phenomenon whose behavior we want to understand. For a graphical representation of the relative importance of different factors under study, one can use the Pareto chart. It is a bar graph with the height of the bars proportional to the contribution of each factor. The bars are displayed from the most numerous category to the least numerous category, as illustrated by the following example. A Pareto chart helps in separating signiﬁcantly few factors that have larger inﬂuence from the trivial many.

Example 1.4.2 For the data of Example 1.4.1, construct a Pareto chart.

Solution First, rewrite the data in decreasing order. Then create a Pareto chart by displaying the bars from the most numerous category to the least numerous category.

Looking at Figure 1.2, we can identify the relative importance of each category such as the maximum, the minimum, and the general behavior of the subject data. Vilfredo Pareto (1848–1923), an Italian economist and sociologist, studied the distributions of wealth in different countries. He concluded that about 20% of people controlled about 80% of a society’s wealth. This same distribution has been observed in other areas such as quality improvement: 80% of problems usually stem from 20% of the causes. This phenomenon has been termed the Pareto effect or 80/20 rule. Pareto charts are used to display the Pareto principle, arranging data so that the few vital factors that are causing most of the problems reveal themselves. Focusing improvement efforts on these few causes will have a larger impact and be more cost-effective than undirected efforts. Pareto charts are used in business decision making as a problem-solving and statistical tool

1.4 Graphical Representation of Data 15

Percentage increase

100 80 60 40 20 0

MC

RR

Fd CPI Category

EI

A&U

■ FIGURE 1.2 Pareto chart.

that ranks problem areas, or sources of variation, according to their contribution to cost or to total variation. Deﬁnition 1.4.2 A circle divided into sectors that represent the percentages of a population or a sample that belongs to different categories is called a pie chart. Pie charts are especially useful for presenting categorical data. The pie “slices” are drawn such that they have an area proportional to the frequency. The entire pie represents all the data, whereas each slice represents a different class or group within the whole. Thus, we can look at a pie chart and identify the various percentages of interest and how they compare among themselves. Most statistical software can create 3D charts. Such charts are attractive; however, they can make pieces at the front look larger than they really are. In general, a two-dimensional view of the pie is preferable.

Example 1.4.3 The combined percentages of carbon monoxide (CO) and ozone (O3 ) emissions from different sources are listed in Table 1.6.

Table 1.6 Combined Percentages of CO and O3 Emissions Transportation (T) 63%

Industrial process (I)

Fuel combustion (F)

Solid waste (S)

10%

14%

5%

Construct a pie chart.

Solution The pie chart is given in Figure 1.3.

Miscellaneous (M) 8%

16 CHAPTER 1 Descriptive Statistics

T(63.0%)

M(8.0%) S(5.0%) I(10.0%)

F(14.0%)

■ FIGURE 1.3 Pie chart for CO and O3 .

Deﬁnition 1.4.3 A stem-and-leaf plot is a simple way of summarizing quantitative data and is well suited to computer applications. When data sets are relatively small, stem-and-leaf plots are particularly useful. In a stem-and-leaf plot, each data value is split into a “stem” and a “leaf.” The “leaf” is usually the last digit of the number and the other digits to the left of the “leaf” form the “stem.” Usually there is no need to sort the leaves, although computer packages typically do. For more details, we refer the student to elementary statistics books. We illustrate this technique by an example.

Example 1.4.4 Construct a stem-and-leaf plot for the 20 test scores given below. 78 91

74 74

82 82

66 75

94 96

71 78

64 84

88 79

55 71

80 83

Solution At a glance, we see that the scores are distributed from the 50s through the 90s. We use the first digit of the score as the stem and the second digit as the leaf. The plot in Table 1.7 is constructed with stems in the vertical position.

Table 1.7 Stem-and-Leaf Display of 20 Exam Scores Stem

Leaves

5

5

6

6

4

7

8

4

1

4

5

8

8

2

8

0

2

4

3

9

4

1

6

9

1

1.4 Graphical Representation of Data 17

The stem-and-leaf plot condenses the data values into a useful display from which we can identify the shape and distribution of data such as the symmetry, where the maximum and minimum are located with respect to the frequencies, and whether they are bell shaped. This fact that the frequencies are bell shaped will be of paramount importance as we proceed to study inferential statistics. Also, note that the stem-and-leaf plot retains the entire data set and can be used only with quantitative data. Examples 1.8.1 and 1.8.6 explain how to obtain a stem-and-leaf plot using Minitab and SPSS, respectively. Refer to Section 1.8.3 for SAS commands to generate graphical representations of the data. A frequency table is a table that divides a data set into a suitable number of categories (classes). Rather than retaining the entire set of data in a display, a frequency table essentially provides only a count of those observations that are associated with each class. Once the data are summarized in the form of a frequency table, a graphical representation can be given through bar graphs, pie charts, and histograms. Data presented in the form of a frequency table are called grouped data. A frequency table is created by choosing a speciﬁc number of classes in which the data will be placed. Generally the classes will be intervals of equal length. The center of each class is called a class mark. The end points of each class interval are called class boundaries. Usually, there are two ways of choosing class boundaries. One way is to choose nonoverlapping class boundaries so that none of the data points will simultaneously fall in two classes. Another way is that for each class, except the last, the upper boundary is equal to the lower boundary of the subsequent class. When forming a frequency table this way, one or more data values may fall on a class boundary. One way to handle such a problem is to arbitrarily assign it one of the classes or to ﬂip a coin to determine the class into which to place the observation at hand. Deﬁnition 1.4.4 Let fi denote the frequency of the class i and let n be sum of all frequencies. Then the relative frequency for the class i is deﬁned as the ratio fi /n. The cumulative relative frequency for the class i is deﬁned by ik=1 fk /n. The following example illustrates the foregoing discussion.

Example 1.4.5 The following data give the lifetime of 30 incandescent light bulbs (rounded to the nearest hour) of a particular type. 872 1150 868

931 987 996

1146 958 1102

1079 1149 1130

915 1057 1002

879 1082 990

863 1053 1052

1112 1048 1116

979 1118 1119

1120 1088 1028

Construct a frequency, relative frequency, and cumulative relative frequency table.

Solution Note that there are n = 30 observations and that the largest observation is 1150 and the smallest one is 865 with a range of 285. We will choose six classes each with a length of 50.

18 CHAPTER 1 Descriptive Statistics

Class

Frequency

Relative frequency

fi

f i fi

Cumulative relative frequency i f k k=1 n

50−900

4

4/30

4/30

900−950

2

2/30

6/30

950−1000

5

5/30

11/30

1000−1050

3

3/30

14/30

1050−1100

6

6/30

20/30

1100−1150

10

10/30

30/30

When data are quantitative in nature and the number of observations is relatively large, and there are no natural separate categories or classes, we can use a histogram to simplify and organize the data. Deﬁnition 1.4.5 A histogram is a graph in which classes are marked on the horizontal axis and either the frequencies, relative frequencies, or percentages are represented by the heights on the vertical axis. In a histogram, the bars are drawn adjacent to each other without any gaps. Histograms can be used only for quantitative data. A histogram compresses a data set into a compact picture that shows the location of the mean and modes of the data and the variation in the data, especially the range. It identiﬁes patterns in the data. This is a good aggregate graph of one variable. In order to obtain the variability in the data, it is always a good practice to start with a histogram of the data. The following steps can be used as a general guideline to construct a frequency table and produce a histogram. GUIDELINE FOR THE CONSTRUCTION OF A FREQUENCY TABLE AND HISTOGRAM 1. Determine the maximum and minimum values of the observations. The range, R = maximum value − minimum value. 2. Select from ﬁve to 20 classes that in general are nonoverlapping intervals of equal length, so as to cover the entire range of data. The goal is to use enough classes to show the variation in the data, but not so many that there are only a few data points in many of the classes. The class width should be slightly larger than the ratio Largest value − Smallest value . Number of classes 3. The ﬁrst interval should begin a little below the minimum value, and the last interval should end a little above the maximum value. The intervals are called class intervals and the boundaries are called class boundaries. The class limits are the smallest and the largest data values in the class. The class mark is the midpoint of a class.

1.4 Graphical Representation of Data 19

4. None of the data values should fall on the boundaries of the classes. 5. Construct a table (frequency table) that lists the class intervals, a tabulation of the number of measurements in each class (tally), the frequency fi of each class, and, if needed, a column with relative frequency, fi /n, where n is the total number of observations. 6. Draw bars over each interval with heights being the frequencies (or relative frequencies).

Let us illustrate implementing these steps in the development of a histogram for the data given in the following example.

Example 1.4.6 The following data refer to a certain type of chemical impurity measured in parts per million in 25 drinkingwater samples randomly collected from different areas of a county. 11 24 35

19 31 18

24 16 24

30 23 18

12 25 27

20 26

25 32

29 17

15 22

21 26

(a) Make a frequency table displaying class intervals, frequencies, relative frequencies, and percentages. (b) Construct a frequency histogram.

Solution (a) We will use five classes. The maximum and minimum values in the data set are 35 and 11. Hence the class width is (35 − 11)/5 = 4.8 5. Hence, we shall take the class width to be 5. The lower boundary of the first class interval will be chosen to be 10.5. With five classes, each of width 5, the upper boundary of the fifth class becomes 35.5. We can now construct the frequency table for the data. Class

Class interval

fi = frequency

Relative frequency

Percentage

1

10.5 − 15.5

3

3/25 = 0.12

12

2

15.5 − 20.5

6

6/25 = 0.24

24

3

20.5 − 25.5

8

8/25 = 0.32

32

4

25.5 − 30.5

5

5/25 = 0.20

20

5

30.5 − 35.5

3

3/25 = 0.12

12

(b) We can generate a histogram as in Figure 1.4.

From the histogram we should be able to identify the center (i.e., the location) of the data, spread of the data, skewness of the data, presence of outliers, presence of multiple modes in the data, and whether the data can be capped with a bell-shaped curve. These properties provide indications of the

Frequency

20 CHAPTER 1 Descriptive Statistics

9 8 7 6 5 4 3 2 1 0

10.5

15.5

20.5 25.5 Data interval

30.5

35.5

■ FIGURE 1.4 Frequency histogram of impurity data.

proper distributional model for the data. Examples 1.8.2 and 1.8.7 explain how to obtain histograms using Minitab and SPSS, respectively.

EXERCISES 1.4 1.4.1.

According to the recent U.S. Federal Highway Administration Highway Statistics, the percentages of freeways and expressways in various road mileage–related highway pavement conditions are as follows: Poor 10%, Mediocre 32%, Fair 22%, Good 21%, and Very good 15%. (a) Construct a bar graph. (b) Construct a pie chart.

1.4.2.

More than 75% of all species that have been described by biologists are insects. Of the approximately 2 million known species, only about 30,000 are aquatic in any life stage. The data in Table 1.4.1 give proportion of total species by insect order that can survive exposure to salt (source: http://entomology.unl.edu/marine_insects/marineinsects.htm).

Table 1.4.1 Species

Percentage

Species

Percentage

Coleoptera

26%

Odonata

3%

Diptera

35%

Thysanoptera

3%

Hemiptera

15%

Lepidoptera

1%

Orthoptera

6%

Other

6%

Collembola

5%

1.4 Graphical Representation of Data 21

(a) Construct a bar graph. (b) Construct a Pareto chart. (c) Construct a pie chart. 1.4.3.

The data in Table 1.4.2 are presented to illustrate the role of renewable energy consumption in the U.S. energy supply in 2007 (source: http://www.eia.doe.gov/fuelrenewable.html). Renewable energy consists of biomass, geothermal energy, hydroelectric energy, solar energy, and wind energy.

Table 1.4.2 Source

Percentage

Coal

22%

Natural Gas

23%

Nuclear Electric Power

8%

Petroleum

40%

Renewable Energy

7%

(a) Construct a bar graph. (b) Construct a Pareto chart. (c) Construct a pie chart. 1.4.4.

A litter is a group of babies born from the same mother at the same time. Table 1.4.3 gives some examples of different mammals and their average litter size (source: http:// www.saburchill.com/chapters/chap0032.html).

Table 1.4.3 Species

Litter size

Bat

1

Dolphin

1

Chimpanzee

1

Lion

3

Hedgehog

5

Red Fox

6

Rabbit

6

Black Rat

11

22 CHAPTER 1 Descriptive Statistics

(a) Construct a bar graph. (b) Construct a Pareto chart. 1.4.5.

The following data give the letter grades of 20 students enrolled in a statistics course. A C

B D

F B

A A

C B

C A

D F

A B

B C

F A

(a) Construct a bar graph. (b) Construct a pie chart. 1.4.6.

According to the U.S. Bureau of Labor Statistics (BLS), the median weekly earnings of fulltime wage and salary workers by age for the third quarter of 1998 is given in Table 1.4.4.

Table 1.4.4 16 to 19 years

$260

20 to 24 years

$334

25 to 34 years

$498

35 to 44 years

$600

45 to 54 years

$628

55 to 64 years

$605

65 years and over

$393

Construct a pie chart and bar graph for these data and interpret. Also, construct a Pareto chart. 1.4.7.

The data in Table 1.4.5 are a breakdown of 18,930 workers in a town according to the type of work. Construct a pie chart and bar graph for these data and interpret.

1.4.8.

The data in Table 1.4.6 represent the number (in millions) of adults and children living with HIV/AIDS by the end of 2000 according to the region of the world (source: http://w3.whosea.org/hivaids/factsheet.htm). Construct a bar graph for these data. Also, construct a Pareto chart and interpret.

1.4.9.

The data in Table 1.4.7 give the life expectancy at birth, in years, from 1900 through 2000 (source: National Center for Health Statistics). Construct a bar graph for these data.

1.4.10.

Dolphins are usually identiﬁed by the shape and pattern of notches and nicks on their dorsal ﬁn. Individual dolphins are cataloged by classifying the ﬁn based on location of distinguishing marks. When a dolphin is sighted its picture can then be compared to the catalog of

1.4 Graphical Representation of Data 23

Table 1.4.5 Mining

58

Construction

1161

Manufacturing

2188

Transportation and Public Utilities

821

Wholesale Trade

657

Retail Trade

7377

Finance, Insurance, and Real Estate Services Total

890 5778 18,930

Table 1.4.6 Country Sub-Saharan Africa

Adults and children living with HIV/AIDS (in millions) 25.30

North Africa and Middle East

0.40

South and Southeast Asia

5.80

East Asia and Paciﬁc

0.64

Latin America

1.40

Caribbean

0.39

Eastern Europe and Central Asia

0.70

Western Europe

0.54

North America

0.92

Australia and New Zealand

0.15

dolphins in the area, and if a match is found, the dolphin can be recorded as resighted. These methods of mark-resight are for developing databases regarding the life history of individual dolphins. From these databases we can calculate the levels of association between dolphins, population estimates, and general life history parameters such as birth and survival rates.

24 CHAPTER 1 Descriptive Statistics

Table 1.4.7 Year

Life expectancy

1900

47.3

1960

69.7

1980

73.7

1990

75.4

2000

77.0

The data in Table 1.4.8 represent frequently resighted individuals (as of January 2000) at a particular location (source: http://www.eckerd.edu/dolphinproject/biologypr.html).

Table 1.4.8 Hammer (adult female)

59

Mid Button Flag (adult female)

41

Luseal (adult female)

31

84 Lookalike (adult female)

20

Construct a bar graph for these data. 1.4.11.

The data in Table 1.4.9 give death rates (per 100,000 population) for 10 leading causes in 1998 (source: National Center for Health Statistics, U.S. Deptartment of Health and Human Services). (a) Construct a bar graph. (b) Construct a Pareto chart.

1.4.12.

In a ﬁscal year, a city collected $32.3 million in revenues. City spending for that year is expected to be nearly the same, with no tax increase projected. Expenditure: Reserves 0.7%, capital outlay 29.7%, operating expenses 28.9%, debt service 3.2%, transfers 5.1%, personal services 32.4%. Revenues: Property taxes 10.2%, utility and franchise taxes 11.3%, licenses and permits 1%, inter governmental revenue 10.1%, charges for services 28.2%, ﬁnes and forfeits 0.5%, interest and miscellaneous 2.7%, transfers and cash carryovers 36%. (a) Construct bar graphs for expenditure and revenues and interpret. (b) Construct pie charts for expenditure and revenues and interpret.

1.4 Graphical Representation of Data 25

Table 1.4.9 Cause

Death rate

Accidents and Adverse Effects

34.5

Chronic Liver Disease and Cirrhosis

9.7

Chronic Obstructive Lung Diseases and Allied Conditions

42.3

Cancer

199.4

Diabetes Mellitus

23.9

Heart Disease

268.0

Kidney Disease

1.4.13.

Pneumonia and Inﬂuenza

35.1

Stroke

58.5

Suicide

10.8

Construct a histogram for the 24 examination scores given next. 78 91

1.4.14.

9.7

74 74

82 82

66 75

94 96

71 78

64 84

88 79

55 71

80 83

73 78

86 79

The following table gives radon concentration in pCi/liter obtained from 40 houses in a certain area. 2.9 7.9 15.9 6.2

0.6 13.5 17.1 2.8 3.8 16.0 2.1 6.4 17.2 0.5 13.7 11.5 2.9 3.6 6.1 8.8 2.2 9.4 8.8 9.8 11.5 12.3 3.7 8.9 13.0 7.9 11.7 6.9 12.8 13.7 2.7 3.5 8.3 15.9 5.1 6.0

(a) Construct a stem-and-leaf display. (b) Construct a frequency histogram and interpret. (c) Construct a pie chart and interpret. 1.4.15.

The following data give the mean of SAT Mathematics scores by state for 1999 for a randomly selected 20 states (source: The World Almanac and Book of Facts 2000). 558 568

503 553

565 510

572 525

546 595

517 502

542 526

(a) Construct a stem-and-leaf display and interpret. (b) Construct a frequency histogram and interpret. (c) Construct a pie chart and interpret.

605 475

493 506

499 568

26 CHAPTER 1 Descriptive Statistics

1.4.16.

A sample of 25 measurements is given here: 9 31 26

28 23 20

14 16 16

29 26 14

21 22 21

27 17

15 19

23 24

23 21

10 20

(a) Make a frequency table displaying class intervals, frequencies, relative frequencies, and percentages. (b) Construct a frequency histogram and interpret.

1.5 NUMERICAL DESCRIPTION OF DATA In the previous section we looked at some graphical and tabular techniques for describing a data set. We shall now consider some numerical characteristics of a set of measurements. Suppose that we have a sample with values x1 , x2 , . . . , xn . There are many characteristics associated with this data set, for example, the central tendency and variability. A measure of the central tendency is given by the sample mean, median, or mode, and the measure of dispersion or variability is usually given by the sample variance or sample standard deviation or interquartile range. Deﬁnition 1.5.1 Let x1 , x2 , . . . , xn be a set of sample values. Then the sample mean (or empirical mean) x is deﬁned by 1 xi . n n

x=

i=1

The sample variance is deﬁned by s2 =

n 1 (xi − x)2 . (n − 1) i=1

The sample standard deviation is s=

s2 .

The sample variance s2 and the sample standard deviation s both are measures of the variability or “scatteredness” of data values around the sample mean x. Larger the variance, more is the spread. We note that s2 and s are both nonnegative. One question we may ask is “why not just take the sum of the differences (xi − x) as a measure of variation?” The answer lies in the following result which shows that if we add up all deviations about the sample mean, we always get a zero value. Theorem 1.5.1 For a given set of measurements x1 , x2 , . . . , xn , let x be the sample mean. Then n i=1

(xi − x) = 0.

1.5 Numerical Description of Data 27

Proof. Since x = (1/n)

n

i=1 xi ,

n

we have n

i=1 xi

= nx. Now

(xi − x ) =

i=1

n

xi −

i=1

n

x

i=1

= nx − nx = 0.

Thus although there may be a large variation in the data values, ni=1 (xi − x) as a measure of spread would always be zero, implying no variability. So it is not useful as a measure of variability.

Sometimes we can simplify the calculation of the sample variance s2 by using the following computational formula: ⎡

2 ⎤ n n 1 2 ⎣ xi − n xi ⎦ s2 =

i=1

i=1

(n − 1)

.

If the data set has a large variation with some extreme values (called outliers), the mean may not be a very good measure of the center. For example, average salary may not be a good indicator of the ﬁnancial well-being of the employees of a company if there is a huge difference in pay between support personnel and management personnel. In that case, one could use the median as a measure of the center, roughly 50% of data fall below and 50% above. The median is less sensitive to extreme data values. Deﬁnition 1.5.2 For a data set, the median is the middle number of the ordered data set. If the data set has an even number of elements, then the median is the average of the middle two numbers. The lower quartile is the middle number of the half of the data below the median, and the upper quartile is the middle number of the half of the data above the median. We will denote Q1 = lower quartile Q2 = M = middle quartile (median) Q3 = upper quartile

The difference between the quartiles is called interquartile range (IQR). IQR = Q3 − Q1 .

A possible outlier (mild outlier) will be any data point that lies below Q1 − 1.5(IQR) or above Q3 + 1.5(IQR).

Note that the IQR is unaffected by the positions of those observations in the smallest 25% or the largest 25% of the data. Mode is another commonly used measure of central tendency. A mode indicates where the data tend to concentrate most.

28 CHAPTER 1 Descriptive Statistics

Deﬁnition 1.5.3 Mode is the most frequently occurring member of the data set. If all the data values are different, then by deﬁnition, the data set has no mode.

Example 1.5.1 The following data give the time in months from hire to promotion to manager for a random sample of 25 software engineers from all software engineers employed by a large telecommunications ﬁrm. 5 22 47

7 21 67

229 25 69

453 23 192

12 24 125

14 34

18 37

14 34

14 49

483 64

Calculate the mean, median, mode, variance, and standard deviation for this sample.

Solution The sample mean is 1 xi = 83.28 months. n n

x=

i=1

To obtain the median, first arrange the data in ascending order: 5 24 125

7 25 192

12 34 229

14 34 453

14 37 483

14 47

18 49

21 64

22 67

23 69

Now the median is the thirteenth number which is 34 months. Since 14 occurs most often (thrice), the mode is 14 months. The sample variance is s2 =

n 1 (xi − x)2 n−1 i=1

1 (5 − 83.28)2 + · · · + (125 − 83.28)2 = 24 = 16,478.

√ and the sample standard deviation is, s = s2 = 128.36 months. Thus, we have sample mean x = 83.28 months, median = 34 months, and mode = 14 months. Note that the mean is very much different from the other two measures of center because of a few large data values. Also, the sample variance s2 = 16,478 months, and the sample standard deviation s = 128.36 months.

Example 1.5.2 For the data of Example 1.5.1, ﬁnd lower and upper quartiles, median, and interquartile range (IQR). Check for any outliers.

1.5 Numerical Description of Data 29

Solution Arrange the data in an ascending order. 5 24 125

7 25 192

12 34 229

14 34 453

14 37 483

14 47

18 49

21 64

22 67

23 69

Then the median M is the middle (13th) data value, M = Q2 = 34. The lower quartile is the middle number below the median, Q1 = [(14 + 18)/2] = 16. The upper quartile, Q3 = [(67 + 69)/2] = 68. The interquartile range, (IQR) = Q3 − Q1 = 68 − 16 = 52. To test for outliers, compute Q1 − 1.5(IQR) = 16 − 1.5(52) = −62 and Q3 + 1.5(IQR) = 68 + 1.5(52) = 146. Then all the data that fall above 146 are possible outliers. None is below −62. Therefore the outliers are 192, 229, 453, and 483.

We have remarked earlier that the mean as a measure of central location is greatly affected by the extreme values or outliers. A robust measure of central location (a measure that is relatively unaffected by outliers) is the trimmed mean. For 0 ≤ α ≤ 1, a 100α% trimmed mean is found as follows: Order the data, and then discard the lowest 100α% and the highest 100α% of the data values. Find the mean of the rest of the data values. We denote the 100α% trimmed mean by xα . We illustrate the trimmed mean concept in the following example.

Example 1.5.3 For the data set representing the number of children in a random sample of 10 families in a neighborhood, ﬁnd the 10% trimmed mean (α = 0.1). 1

2

2

3

2

3

9

1

6

2

1

2

2

2

2

3

3

6

9

Solution Arrange the data in ascending order. 1

The data set has 10 elements. Discarding the lowest 10% (10% of 10 is 1) and discarding the highest 10% of the data values, we obtain the trimmed data set as 1 2 2 2 2 3 3 6 The 10% trimmed mean is 1+2+2+2+2+3+3+6 = 2.6. x0.1 = 8 Note that the mean for the data in the previous example without removing any observations is 3.1, which is different from the trimmed mean.

30 CHAPTER 1 Descriptive Statistics

Examples 1.8.2 and 1.8.7 explain how to obtain a histogram using Minitab and SPSS, respectively. Example 1.8.9 demonstrates the SAS commands to obtain the descriptive statistics. Although standard deviation is a more popular method, there are other measures of dispersion such as average deviation or interquartile range. We have already seen the deﬁnition of interquartile range. The average deviation for a sample x1 , . . . , xn is deﬁned by n

Average deviation =

|xi − x|

i=1

.

n

Calculation of average deviation is simple and straightforward.

1.5.1 Numerical Measures for Grouped Data When we encounter situations where the data are grouped in the form of a frequency table (see Section 1.4), we no longer have individual data values. Hence, we cannot use the formulas in Deﬁnition 1.5.1. The following formulas will give approximate values for x and s2 . Let the grouped data have l classes, with mi being the midpoint and fi being the frequency of class i, i = 1, 2, . . . , l. Let n = li=1 fi . Deﬁnition 1.5.4 The mean for a sample of size n, 1 fi m i , n l

x=

i=1

where mi is the midpoint of the class i and fi is the frequency of the class i. Similarly the sample variance,

s2 =

n 1 fi (mi − x)2 = n−1 i=1

2 fi mi

2 mi fi − i n n−1

.

The following example illustrates how we calculate the sample mean for a grouped data.

Example 1.5.4 The grouped data in Table 1.8 represent the number of children from birth through the end of the teenage years in a large apartment complex. Find the mean, variance, and standard deviation for these data:

Table 1.8 Number of Children and Their Age Group Class Frequency

0–3

4–7

8–11

12–15

16–19

7

4

19

12

8

1.5 Numerical Description of Data 31

Solution For simplicity of calculation we create Table 1.9.

Table 1.9 Class

fi

mi

mi f i

m2i f i

0−3

7

1.5

10.5

15.75

4−7

4

5.5

22

121

8−11

19

9.5

180.5

1714.75

12−15

12

13.5

162

2187

16−19

8

17.5

140

n = 50

mi fi = 515

2450

m2i fi = 6488.5

The sample mean is x=

1 515 = 10.30. fi m i = n 50 i

The sample variance is

2

fi mi 2 2 i 6488.5 − (515) f − m i n 2 50 i = = 24.16. s = n−1 49 √ √ The sample standard deviation is s = s2 = 24.16 = 4.92.

Using the following calculations, we can also ﬁnd the median for grouped data. We only know that the median occurs in a particular class interval, but we do not know the exact location of the median. We will assume that the measures are spread evenly throughout this interval. Let L = lower class limit of the interval that contains the median n = total frequency Fb = cumulative frequencies for all classes before the median class fm = frequency of the class interval containing the median w = interval width of the interval that contains the median

Then the median for the grouped data is given by M =L+

We proceed to illustrate with an example.

w (0.5n − Fb ). fm

32 CHAPTER 1 Descriptive Statistics

Example 1.5.5 For the data of Example 1.5.4, ﬁnd the median.

Solution First develop Table 1.10.

Table 1.10 fi

Cumulative f i

0−3

7

7

0.14

4−7

4

11

0.22

8−11

19

30

0.6

12−15

12

42

0.84

16−19

8

50

1.00

Class

Cumulative f i /n

The ﬁrst interval for which the cumulative relative frequency exceeds 0.5 is the interval that contains the median. Hence the interval 8 to 11 contains the median. Therefore, L = 8, fm = 19, n = 50, w = 3, and Fb = 11. Then, the median is M =L+

3 w (0.5n − Fb ) = 8 + ((0.5)(50) − 11) = 10.211. fm 19

It is important to note that all the numerical measures we calculate for grouped data are only approximations to the actual values of the ungrouped data if they are available. One of the uses of the sample standard deviation will be clear from the following result, which is based on data following a bell-shaped curve. Such an indication can be obtained from the histogram or stem-and-leaf display. EMPIRICAL RULE When the histogram of a data set is “bell shaped” or “mound shaped,” and symmetric, the empirical rule states: 1. Approximately 68% of the data are in the interval (x − s, x + s). 2. Approximately 95% of the data are in the interval (x − 2s, x + 2s). 3. Approximately 99.7% of the data are in the interval (x − 3s, x + 3s).

The bell-shaped curve is called a normal curve and is discussed later in Chapter 3. A typical symmetric bell-shaped curve is given in Figure 1.5.

1.5 Numerical Description of Data 33

Normal distribution 0.4

1 sd

0.3

0.2 2 sd 0.1 3 sd 0.0 ⫺3

⫺2

⫺1

0 x

1

2

3

■ FIGURE 1.5 Bell-shaped curve.

1.5.2 Box Plots The sample mean or the sample standard deviation focuses on a single aspect of the data set, whereas histograms and stem-and-leaf displays express rather general ideas about data. A pictorial summary called a box plot (also called box-and-whisker plots) can be used to describe several prominent features of a data set such as the center, the spread, the extent and nature of any departure from symmetry, and identiﬁcation of outliers. Box plots are a simple diagrammatic representation of the ﬁve number summary: minimum, lower quartile, median, upper quartile, maximum. Example 1.8.4 illustrates the method of obtaining box plots using Minitab.

PROCEDURE TO CONSTRUCT A BOX PLOT 1. Draw a vertical measurement axis and mark Q1 , Q2 (median), and Q3 on this axis as shown in Figure 1.6. 2. Construct a rectangular box whose bottom edge lies at the lower quartile, Q1 and whose upper edge lies at the upper quartile, Q3 . 3. Draw a horizontal line segment inside the box through the median. 4. Extend the lines from each end of the box out to the farthest observation that is still within 1.5(IQR) of the corresponding edge. These lines are called whiskers. 5. Draw an open circle (or asterisks *) to identify each observation that falls between 1.5(IQR) and 3(IQR) from the edge to which it is closest; these are called mild outliers. 6. Draw a solid circle to identify each observation that falls more than 3(IQR) from the closest edge; these are called extreme outliers.

34 CHAPTER 1 Descriptive Statistics

Extreme outliers

3(IQR ) Mild outliers 1.5(IQR ) Whisker

Q3

Q2

Q1 Whisker 1.5(IQR ) Mild outliers 3(IQR )

Extreme outliers ■ FIGURE 1.6 A typical box-and-whiskers plot.

We illustrate the procedure with the following example.

Example 1.5.6 The following data identify the time in months from hire to promotion to chief pharmacist for a random sample of 25 employees from a certain group of employees in a large corporation of drugstores. 5 22 47

7 21 67

229 25 69

453 23 192

12 24 125

14 34

18 37

14 34

14 49

483 64

Construct a box plot. Do the data appear to be symmetrically distributed along the measurement axis?

Solution Referring to Example 1.5.2, we find that the median, Q2 = 34. The lower quartile is Q1 = 14+18 = 16. 2 = 68. The upper quartile is Q3 = 67+69 2 The interquartile range is IQR = 68 − 16 = 52.

1.5 Numerical Description of Data 35

To find the outliers, compute Q1 − 1.5(IQR) = 16 − 1.5(52) = −62 and Q3 + 1.5(IQR) = 68 + 1.5(52) = 146. Using these numbers, we follow the procedure outlined earlier to construct the box plot in Figure 1.7. The * in the box plot represents an outlier. The first horizontal line is the first quartile, the second is the median, and the third is the third quartile.

500 Months

400 300 200 100 0 ■ FIGURE 1.7 Box plot for months to promotion.

By examining the relative position of the median line (the middle line in Figure 1.7), we can test the symmetry of the data. For example, in Figure 1.7, the median line is closer to the lower quartile than the upper line, which suggests that the distribution is slightly nonsymmetric. Also, a look at this box plot shows the presence of two mild outliers and two extreme outliers.

EXERCISES 1.5 1.5.1.

The prices of 12 randomly chosen homes in dollars (approximated to nearest thousand) in a growing region of Tampa in the summer of 2002 are given below. 176

105

133

140

305

215

207

210

173

150

78

96

Find the mean and standard deviation of the sampled home prices from this area. 1.5.2.

The following is a sample of nine mortgage companies’ interest rates for 30-year home mortgages, assuming 5% down. 7.625

7.500

6.625

7.625

6.625

6.875

7.375

5.375

7.500

(a) Find the mean and standard deviation and interpret. (b) Find lower and upper quartiles, median, and interquartile range. Check for any outliers and interpret. 1.5.3.

For four observations, it is given that mean is 6, median is 4, and mode is 3. Find the standard deviation of this sample.

36 CHAPTER 1 Descriptive Statistics

1.5.4.

The data given below pertain to a random sample of disbursements of state highway funds (in millions of dollars), to different states. 1188 537

1050 519

2882 2523

2802 316

780 1117

1171 1578

685 261

(a) Find the mean, variance, and range for these data and interpret. (b) Find lower and upper quartiles, median and interquartile range. Check for any outliers and interpret. (c) Construct a box plot and interpret. 1.5.5.

Maximal static inspiratory pressure (PImax) is an index of respiratory muscle strength. The following data show the measure of PImax (cm H2 O) for 15 cystic ﬁbrosis patients. 105 135

80 105

115 45

95 115

100 40

85 115

90 95

70

(a) Find the lower and upper quartiles, median, and interquartile range. Check for any outliers and interpret. (b) Construct a box plot and interpret. (c) Are there any outliers? 1.5.6.

Compute the mean, variance, and standard deviation for the data in Table 1.5.1 (assume that the data belong to a sample).

Table 1.5.1 Class Frequency

1.5.7.

0–4

5–9

10–14

15–19

20–24

5

14

15

10

6

(a) For any grouped data with l classes with group frequencies fi , and class midpoints mi , show that l

fi (mi − x) = 0.

i=1

(b) Verify this result for the data given in Exercise 1.5.6. 1.5.8.

(a) Given the sample values x1 , x2 , . . . , xn , show that n i=1

(xi − x)2 =

n i=1

xi2 −

n

2 xi

i=1

n

(b) Verify the result of part (a) for the data of Exercise 1.5.5.

.

1.5 Numerical Description of Data 37

1.5.9.

The following are the closing prices of some securities that a mutual fund holds on a certain day: 10.25 5.31 11.25 13.13 43.25 45.00 40.06 28.56 32.00 25.44 22.50 30.00 53.50 29.87 32.00 28.87 (a) (b) (c) (d) (e)

1.5.10.

32.56 51.50 53.37 37.50

37.06 47.00 51.38 30.44

39.00 53.50 26.00 41.37

Find the mean, variance, and range for these data and interpret. Find lower and upper quartiles, median, and interquartile range. Check for any outliers. Construct a box plot and interpret. Construct a histogram. Locate on your histogram x, x ± s, x ± 2s, and x ± 3s. Count the data points in each of the intervals x ± s, x ± 2s, and x ± 3s and compare this with the empirical rule.

The radon concentration (in pCi/liter) data obtained from 40 houses in a certain area are given below. 2.9 7.9 15.9 6.2 (a) (b) (c) (d) (e)

1.5.11.

18.00 22.75 24.75 42.19

0.6 13.5 17.1 2.8 3.8 16.0 2.1 6.4 17.2 0.5 13.7 11.5 2.9 3.6 6.1 8.8 2.2 9.4 8.8 9.8 11.5 12.3 3.7 8.9 13.0 7.9 11.7 6.9 12.8 13.7 2.7 3.5 8.3 15.9 5.1 6.0

Find the mean, variance, and range for these data. Find lower and upper quartiles, median, and interquartile range. Check for any outliers. Construct a box plot. Construct a histogram and interpret. Locate on your histogram x ± s, x ± 2s, and x ± 3s. Count the data points in each of the intervals x, x ± s, x ± 2s, and x ± 3s. How do these counts compare with the empirical rule?

A random sample of 100 households’ weekly food expenditure represented by x from a particular city gave the following statistics:

xi = 11,000, and

xi 2 = 1,900,000.

(a) Find the mean and standard deviation for these data. (b) Assuming that the food expenditure of the households of an entire city of 400,000 will have a bell-shaped distribution, how many households of this city would you expect to fall in each of the intervals, x ± s, x ± 2s, and x ± 3s? 1.5.12.

The following numbers are the hours put in by 10 employees of company in a randomly selected week: 40

46

40

54

18

45

34

60

39

42

(a) Calculate the values of the three quartiles and the interquartile range. Also, calculate the mean and standard deviation and interpret.

38 CHAPTER 1 Descriptive Statistics (b) Verify for this data set that 10 i=1 (xi − x) = 0. (c) Construct a box plot. (d) Does this data set contain any outliers? 1.5.13.

For the following data: 6.3 7.0 4.5

2.9 2.8 4.5

4.5 4.3 5.7

1.1 5.3 0.5

1.8 2.9 6.2

4.0 8.3 3.7

1.2 4.4 0.9

3.1 2.8 2.4

2.0 3.1 3.0

4.0 5.6 3.5

(a) Find the mean, variance, and standard deviation. (b) Construct a frequency table with ﬁve classes. (c) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation for the frequency table constructed in part (b) and compare it to the results in part (a). 1.5.14.

In order to assess the protective immunizing activity of various whooping cough vaccines, suppose that 30 batches of different vaccines are tested on groups of children. Suppose that the following data give immunity percentage in home exposure values (IPHE values). 85 42 79

51 12 90

41 70 43

90 38 40

91 97 89

40 34 85

39 94 71

69 77 30

45 88 25

47 91 21

(a) Find the mean, variance, and standard deviation and interpret. (b) Construct a frequency table with ﬁve classes. (c) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation for the table in part (b) and compare it to the results in part (a). 1.5.15.

The grouped data in Table 1.5.2 give the number of births by age group of mothers between ages 10 and 39 in a certain state in 2000. Find the median for this grouped data and interpret.

1.5.16.

Table 1.5.3 gives the distribution of the masses (in grams) of 50 salmon from a single young cohort.

Table 1.5.2 Age of mother

Number of births

10–14

895

15–19

55,373

20–24

122,591

25–29

139,615

30–34

127,502

35–39

68,685

1.6 Computers and Statistics 39

Table 1.5.3 Weight

155–164

165–174

175–184

185–194

195–204

8

11

18

9

4

Frequency

(a) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation (b) Find the median for this grouped data. 1.5.17.

After a pollution accident, 180 dead ﬁsh were recovered from a stream. Table 1.5.4 gives their lengths measured to the nearest millimeter.

Table 1.5.4 Length of Fish (mm) Frequency

1–19

20–39

40–59

60–79

80–99

38

31

59

45

7

(a) Using the grouped data formula, ﬁnd the mean, variance, and standard deviation. (b) Find the median for this grouped data and interpret.

1.6 COMPUTERS AND STATISTICS With present-day technology, we can automate most statistical calculations. For small sets of data, many basic calculations such as ﬁnding means and standard deviations and creating simple charts, graphing calculators are sufﬁcient. Students should learn how to perform statistical analysis using their handheld calculators. For deeper analysis and for large data sets, statistical software is necessary. Software also provides easier data entry and editing and much better graphics in comparison to calculators. There are many statistical packages available. Many such analyses can be performed with spreadsheet application programs such as Microsoft Excel, but a more thorough data analysis requires the use of more sophisticated software such as Minitab and SPSS. For students with programming abilities, packages such as MATLAB may be more appealing. For very large data sets and for complicated data analysis, one could use SAS. SAS is one of the most frequently used statistical packages. Many other statistical packages (such as R, Splus, and StatXact) are available; the utilities and advantages of each are based on the speciﬁc application and personal taste. For example, R is free software that is being increasingly used by statisticians and can be downloaded from http://www.r-project.org/, and a statistical tutorial for R can be found at http://www.biometrics.mtu.edu/CRAN/. For a good introduction to doing statistics with R, refer to the book by Peter Dalgaard, Introductory Statistics, with R, Springer, 2002. In this book, we will give some representative Minitab, SPSS, and SAS commands at the end of each chapter just to get students started on the technology. These examples are by no means a tutorial for

40 CHAPTER 1 Descriptive Statistics

the respective software. For a more thorough understanding and use of technology, students should look at the users’ manual that comes with the software or at references given at the end of the book. The computer commands are designed to be illustrative, rather than completely efﬁcient. In dealing with data analysis for real-world problems, we need to know which statistical procedure to use, how to prepare the data sets suitable for use in the particular statistical package, and ﬁnally how to interpret the results obtained. A good knowledge of theory supplemented with a good working knowledge of statistical software will enable students to perform sophisticated statistical analysis, while understanding the underlying assumptions and the limitations of results obtained. This will prevent us from misleading conclusions when using computer-generated statistical outputs.

1.7 CHAPTER SUMMARY In this chapter, we dealt with some basic aspects of descriptive statistics. First we gave basic deﬁnitions of terms such as population and sample. Some sampling techniques were discussed. We learned about some graphical presentations in Section 1.4. In Section 1.5 we dealt with descriptive statistics, in which we learned how to ﬁnd mean, median, and variance and how to identify outliers. A brief discussion of the technology and statistics was given in Section 1.6. All the examples given in this chapter are for a univariate population, in which each measurement consists of a single value. Many populations are multivariate, where measurements consist of more than one value. For example, we may be interested in ﬁnding a relationship between blood sugar level and age, or between body height and weight. These types of problems will be discussed in Chapter 8. In practice, it is always better to run descriptive statistics as a check on one’s data. The graphical and numerical descriptive measures can be used to verify that the measurements are sound and that there are no obvious errors due to collection or coding. We now list some of the key deﬁnitions introduced in this chapter. ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Population Sample Statistical inference Quantitative data Qualitative or categorical data Cross-sectional data Time series data Simple random sample Systematic sample Stratiﬁed sample Proportional stratiﬁed sampling Cluster sampling Multiphase sampling Relative frequency Cumulative relative frequency

1.8 Computer Examples 41

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Bar graph Pie chart Histogram Sample mean Sample variance Sample standard deviation Median Interquartile range Mode Mean Empirical rule Box plots

In this chapter, we have also introduced the following important concepts and procedures: ■ ■ ■ ■

■

■ ■

General procedure for data collection Some advantages of simple random sampling Steps for selecting a stratiﬁed sample Procedures to construct frequency and relative frequency tables and graphical representations such as stem-and-leaf displays, bar graphs, pie charts, histograms, and box plots Procedures to calculate measures of central tendency, such as mean and median, as well as measures of dispersion such as the variance and standard deviation for both ungrouped and grouped data Guidelines for the construction of frequency tables and histograms Procedures to construct a box plot

1.8 COMPUTER EXAMPLES In this section, we give some examples of how to use Minitab, SPSS, and SAS for creating graphical representations of the data as well as methods for the computation of basic statistics. Sometimes, the outputs obtained using a particular software package may not be exactly as explained in the book; they vary from one package to another, and also depend on the particular software version. It is important to obtain the explanation of outputs from the help menu of the particular software package for complete understanding. The “Computer Examples” sections of this book are not designed as manuals for the software, nor are they written in the most efﬁcient way. The idea is only to introduce some basic procedures, so that the students can get started with applying the theoretical material they have seen in each of the chapters.

1.8.1 Minitab Examples A good place to get help on Minitab is http://www.minitab.com/resources/. There are many nice sites available on Minitab procedures; for example, Minitab student tutorials can be obtained from

42 CHAPTER 1 Descriptive Statistics

http://www.minitab.com/resources/tutorials/. Here we illustrate only some of the basic uses of Minitab. In Minitab, we can enter the data in the spreadsheet and use the Windows pull-down menus, or we can directly enter the data and commands. We will mostly give procedures for the pull-down menus only. It is up to the user’s taste to choose among these procedures. It should be noted that with different versions of Minitab, there will be some differences in the pull-down menu options. It is better to consult the Help menu for the actual procedure.

Example 1.8.1 (Stem-and-Leaf): For the following data, construct a stem-and-leaf display using Minitab: 78 91

74 74

82 82

66 75

94 96

71 78

64 84

88 79

55 71

80 83

Solution For the pull-down menu, first enter the data in column 1. Then follow the following sequence. The boldface represents the actions.

Graph > Character Graphs > Stem-and-Leaf In Variables: type C1 and click OK We will get the following output: Stem-and-Leaf of C1 Leaf Unit = 1.0 1 5 5 2 6 4 3 6 6 7 7 1 (4) 7 5 9 8 0 4 8 8 3 9 14 1

N = 20

1 8 2

4 8 2

4 9 3

4

The following are the explanations of each column in the stem-and-leaf display, as given in the Minitab Help menu. The display has three columns: Left: Cumulative count of values from the top of the figure down and from the bottom of the figure up to the middle. Middle number in parentheses (stem): Count of values in the row containing the median. Parentheses around the median row are omitted if the median falls between two lines of the display. Right (leaves): Each value is a single digit to place after the stem digits, representing one data value. The leaf unit tells you where to put the decimal place in each number.

1.8 Computer Examples 43

Note that this display is a little different from the one we explained in Section 1.4. However, if we combine the stems and the corresponding leaves, we will get the representation as in Section 1.4.

Example 1.8.2 (Histogram): For the following data, construct a histogram: 25 38

37 16

20 40

31 32

31 33

21 24

12 39

25 26

36 27

27 19

Solution Enter the data in C1, then use the following sequence Graph > Histogram. . . > in Graph variables: type C1 > OK

We will get the histogram as shown in Figure 1.8.

6

Frequency

5 4 3 2 1 0 10

15

20

25

30

35

40

■ FIGURE 1.8 Histogram for data of Example 1.8.2.

If we want to change the number of intervals, after entering Graph variables, click Options. . . and click Number of intervals and enter the desired number, then OK.

Example 1.8.3 (Descriptive Statistics): In this example, we will describe how to obtain basic statistics such as mean, median, and standard deviation for the following data:

44 CHAPTER 1 Descriptive Statistics

5 22 47

7 21 67

229 25 69

453 23 192

12 24 125

14 34

18 37

14 34

14 49

483 64

Solution Enter the data in C1. Then use Stat > Basic Statistics > Display Descriptive Statistics. . . > in Variables: type C1 > click OK We will get the following output: Variable C1 TrMean 69.3

StDev 128.4 Q1 16.0

N 25 SE Mean 25.7

Mean 83.3 Minimum 5.0 Q3 68.0

Median 34.0 Maximum 483.0

Here, TrMean represents the trimmed mean. A 5% trimmed mean is calculated. Minitab removes the smallest 5% and the largest 5% of the values (rounded to the nearest integer) and then averages the remaining values. Also, SE Mean gives the standard error of the mean. It is calculated as StDev/SQRT (N), where StDev is the standard deviation.

Example 1.8.4 (Sorting and Box Plot): For the following data, ﬁrst sort in the increasing order and then construct a box plot to check for outliers. 870 1150 866

922 977 996

1146 958 1102

1120 1088 1028

1079 1139 1130

905 1055 1002

888 1082 990

865 1053 1052

1112 1048 1116

966 1118 1109

Solution After entering the data in C1, we can sort the data in increasing order as follows: Manip > Sort. . . > in Sort column(s): type C1 > in Store sorted column(s) in: type C2 > in Sorted by column: type C1 > OK In column C2, we will get the following sorted data: C2 865 866 870 888 905 922 958 966 977 990 996 1002 1028 1048 1052 1053 1055 1079 1082 1088 1102 1109 1112 1116 1118 1120 1130 1139 1146 1150 If we want to draw a box plot for the data, do the following:

1.8 Computer Examples 45

Graph > Box plot. . . > in Graph variables: under Y, type C1 > OK We will get the box plot as shown in Figure 1.9.

1150 1100 1050 1000 950 900 850 ■ FIGURE 1.9 Box plot data of Example 1.8.4.

Example 1.8.5 (Test of Randomness): Almost all of the analyses in this book assume that the sample is random. How can we verify whether the sample is really random? Project 12B explains a procedure called run test. Without going into details, this test is simple with Minitab. All we have to do is enter the data in C1. Then click Stat > Nonparametric > Runs Test. . . > in variables: enter C1 > OK For instance, if we have the following data: 24 38

31 49

28 51

43 49

28 62

we will get following output: Run Test C1 K = 44.0500 The observed number of runs = 14 The expected number of runs = 11.0000 10 Observations above K 10 below

56 33

48 41

39 58

52 63

32 56

46 CHAPTER 1 Descriptive Statistics

* N Small -- The following approximation may be invalid The test is significant at 0.1681 Cannot reject at alpha = 0.05 ‘‘Cannot reject’’ in the output means that it is reasonable to assume that the sample is random. For any data, it is always desirable to do a run test to determine the randomness.

1.8.2 SPSS Examples For SPSS, we will give only Windows commands. For all the pull-down menus, the sequence will be separated by the > symbol.

Example 1.8.6 Redo Example 1.8.1 with SPSS.

Solution After entering the data in C1,

Analyze > Descriptive Statistics > Explore. . . > At the Explore window select the variable and move to Dependent List; then click Plots. . ., select Stem-and-Leaf , click Continue, and click OK at the Explore Window We will get the output with a few other things, including box plots along with the stem-and-leaf display, which we will not show here.

Example 1.8.7 Redo Example 1.8.2 with SPSS.

Solution After entering the data:

Graphs > Histogram. . . > At the Histogram window select the variable and move to Variable, and click OK We will get the histogram, which we will not display here.

Example 1.8.8 Redo Example 1.8.3 with SPSS.

1.8 Computer Examples 47

Solution Enter the data. Then: Analyze > Descriptive Statistics > Frequencies. . . > At the Frequencies window select the variable(s); then open the Statistics window and check whichever boxes you desire under Percentile, Dispersion, Central Tendency, and Distribution > continue > OK For example, if you select Mean, Median, Mode, Standard Deviation, and Variance, we will get the following output and more:

N Mean Median Mode Std. Deviation Variance

Statistics VAR00001 Valid Missing

25 0 83.2800 34.0000 14.00 128.36488 16477.54333

1.8.3 SAS Examples We will now give some SAS procedures describing the numerical measures of a single variable. PROC UNIVARIATE will give mean, median, mode, standard deviation, skewness, kurtosis, etc. If we do not need median, mode, and so on, we could just as well use PROC MEANS in lieu of PROC UNIVARIATE. We can use the following general format in writing SAS programs with appropriate problem-speciﬁc modiﬁcations. There are many good online references as well as books available for SAS procedures. To get support on SAS, including many example codes, refer to the SAS support Web site: http://support.sas.com/. Another helpful site can be found at http://www.ats.ucla.edu/stat/sas/. There are many other sites that may suit your particular application. GENERAL FORMAT OF AN SAS PROGRAM DATA give a name to the data set; INPUT here we put variable names and column locations, if there are more than one variable; CARDS; (also we can use DATALINES;) Enter the data here; TITLE ‘here we include the title of our analysis’; PROC PRINT; PROC name of procedure (such as PROC UNIVARIATE) goes here; Options that we may want to include (such as the variables to be used) go here; RUN;

48 CHAPTER 1 Descriptive Statistics

After writing an SAS program, to execute it we can go to the menu bar and select run>submit, or click the “running man” icon. On execution, SAS will output the results to the Output window. All the steps used including time of execution and any error messages will be given in the Log window. In order to make the SAS outputs more manageable, we can use the following SAS command at the beginning of an SAS program: options ls=80 ps=50;

ls stands for line size, and this sets each line to be 80 characters wide. ps stands for page size and allows 50 lines on each page. This reduces the number of unnecessary page breaks. In order to avoid date and number, we can use the option commands: Options nodate nonumber;

Example 1.8.9 For the data of Example 1.8.3, use PROC UNIVARIATE to summarize the data.

Solution In the program editor window, type the following if you are entering the data directly. If you are using the data stored in a file, the comment line (with *) should be used instead of the input and data lines.

Options nodate nonumber; DATA ex9; INPUT ex9 @@; DATALINES; 5 7 229 453 12 14 18 14 14 483 22 21 25 23 24

34 37 34 49 64

47 67 69 192 125; PROC UNIVARIATE; TITLE; RUN; In this case we will get the following output:

N Mean Std Deviation

The UNIVARIATE Procedure Variable: ex9 Moments 25 Sum Weights 25 83.28 Sum Observations 2082 128.364884 Variance 16477.5433

1.8 Computer Examples 49

Skewness 2.45719194 Kurtosis 5.47138396 Uncorrected SS 568850 Corrected SS 395461.04 Coeff Variation 154.136508 Std Error Mean 25.6729767 Basic Statistical Measures Location Variability Mean 83.28000 Std Deviation 128.36488 Median 34.00000 Variance 16478 Mode 14.00000 Range 478.00000 Interquartile Range 49.00000 Tests for Location: Mu0=0 Test -Statistic-p ValueStudent’s t t 3.243878 Pr > |t| 0.0035 Sign M 12.5 Pr >= |M| = |S| 0 otherwise.

(a) For what value of λ is f a pdf? (b) Find F (x).

Solution

∞ (a) First note that f (x) ≥ 0. Now, for f (x) to be a pdf, we need −∞ f (x)dx = 1. Because f (x) = 0 for x ≤ 0, Therefore λ = 1. See Figure 2.6. 1=

∞ −∞

∞ f (x)dx = λxe−x dx 0

⎡ ⎤ ∞ ∞ ∞ = λ xe−x dx = λ⎣ −xe−x 0 + e−x dx⎦(using integration by parts) 0 0 ∞ = λ 0 − e−x 0 = λ. 0.5 0.4

P (a # X # b)

f (x )

0.3 0.2 0.1 0.0 a

b X Data

■ FIGURE 2.6 Probability as an area under a curve.

88 CHAPTER 2 Basic Concepts from Probability Theory

0.4

0.3

0.2

0.1

0.0 0

2

4

6

8

10

12

8

10

12

■ FIGURE 2.7 Graph of f (x) = xe−x .

1.2 1.0 0.8 0.6 0.4 0.2 0.0 0

2

4

6

■ FIGURE 2.8 Graph of F (x), x ≥ 0.

(b) The cumulative distribution function is x f (t)dt =

F (x) = −∞

⎧ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎩

0, x

te−t dt = 1 − (x + 1) e−x ,

x 750).

0≤y≤4 elsewhere.

98 CHAPTER 2 Basic Concepts from Probability Theory

Solution (a) ∞ E(Y ) =

yf (y)dy −∞

3 = 64

4

yy2 (4 − y) dy

0

= 2.4 and 4 Var(Y ) =

(y − 2.4)2

3 2 y (4 − y) dy 64

0

= 0.64. (b) Using the fact that Var(aY + b) = a2 Var(Y ), we have Var(X) = (300)2 Var(Y ) = 90,000(0.64) = 57,600. (c) P (X > 750) = P(300Y + 50 > 750)

7 =P Y > 3 3 = 64

4

y2 (4 − y) dy = 0.55339.

7/3

2.6.1 Skewness and Kurtosis Even though the mean μ and the standard deviation σ are signiﬁcant descriptive measures that locate the center and describe the spread or dispersion of probability density function f (x), they do not provide a unique characterization of the distribution. Two distributions may have the same mean and variance and yet could be very different, as in Figure 2.12. To better approximate the probability distribution of a random variable, we may need higher moments.

2.6 Moments and Moment-Generating Functions 99

0.5 0.4 0.3 0.2 0.1 0.0 Mean 5 1 Variance 5 1

Mean 5 1 Variance 5 1 ■ FIGURE 2.12 Same mean and variance.

Deﬁnition 2.6.4 The kth moment about the origin of a random variable X is deﬁned as EXk and denoted by μk , whenever it exists. The kth moment about its mean (also called central kth moment) $ # of a random variable X is deﬁned as E (X − μ)k and denoted by μk , k = 2, 3, 4, . . . , whenever it exists. In particular, we have E(X) = μ1 = μ, and σ 2 = μ2 . We have seen earlier that the second moment about mean (variance, σ 2 ) is used as a measure of dispersion about the mean. Deﬁnition 2.6.5 The standardized third moment about mean α3 =

E(X − μ)3 μ3 = 3/2 σ3 μ2

is called the skewness of the distribution of X. The standardized fourth moment about mean α4 =

E(X − μ)4 σ4

is called the kurtosis of the distribution. Skewness is used as a measure of the asymmetry (lack of symmetry) of a density function about its mean. Recall that a distribution, or data set, is symmetric if it looks the same to the left and right of the center point. If α3 = 0, then the distribution is symmetric about the mean, if α3 > 0, the distribution has a longer right tail, and if α3 < 0, the distribution has a longer left tail. Thus, the skewness of a normal distribution is zero. Kurtosis is a measure of whether the distribution is peaked or ﬂat relative to a normal distribution. Kurtosis is based on the size of a distribution’s tails. Positive kurtosis indicates too few observations in the tails, whereas negative kurtosis indicates too many observations in the tail of the distribution. Distributions with relatively large tails are called leptokurtic, and those with small tails are called platokurtic. A distribution which has the same kurtosis as a normal distribution is known as mesokurtic. It is known that the kurtosis for a standard normal distribution α4 = 3. An important expectation is the moment-generating function for a random variable, in a sense, this packages all the moments for a random variable in one expression.

100 CHAPTER 2 Basic Concepts from Probability Theory

Deﬁnition 2.6.6 For a random variable X, suppose that there is a positive number h such that for −h < t < h the mathematical expectation E etX exists. The moment-generating function (mgf) of the random variable X is deﬁned by ⎧ ⎨ etx p(x), MX (t) = E etX = ⎩ etx f (x)dx,

if discrete . if continuous

An advantage of the moment generating function is its ability to give the moments. Recall that the Maclaurin series of the function etx is etx = 1 + tx +

(tx)3 (tx)n (tx)2 + + ··· + + ···· 2! 3! n!

By using the fact that the expected value of the sum equals the sum of the expected values, the moment-generating function can be written as

%

(tX)3 (tX)n (tX)2 MX (t) = E etX = E 1 + tX + + + ··· + + ··· 2!

= 1 + tE[X] +

3!

&

n!

tn # $ t2 2 t3 3 E X + E X + · · · + E Xn + · · · 2! 3! n!

Taking the derivative of MX (t) with respect to t, we obtain 2 dMX (t) (t) = E[X] + tE[X] + t E X2 = MX dt 2! +

t (n−1) # n $ t3 3 E X + ··· E X + ··· + (n − 1)! 3!

Evaluating this derivative at t = 0, all terms except E[X] become zero. We have (0) = E[X]. MX

Similarly, taking the second derivative of MX (t), we obtain (0) = E X2 . MX (n)

Continuing in this manner, from the nth derivative MX (t) with respect to t, we obtain all the moments to be # $ (n) MX (0) = E Xn ,

n = 1, 2, 3, . . . .

We summarize these calculations in the following theorem.

2.6 Moments and Moment-Generating Functions 101

Theorem 2.6.3 If MX (t) exists, then for any positive integer k, d k MX (t) dt k

= MX (0) = μk . (k)

t=0

The usefulness of the foregoing theorem lies in the fact that, if the mgf can be found, the often difﬁcult process of integration or summation involved in calculating different moments can be replaced by the much easier process of differentiation. The following examples illustrate this fact.

Example 2.6.8 Let X be a random variable with pf p(x) =

n x p (1 − p)n−x , x

x = 0, 1, 2, . . . , n.

(This random variable is called a binomial random variable, and the pf is called a binomial distribution.) # $n Show that MX (t) = (1 − p) + pet , for all real values of t. Also obtain mean and variance of the random variable X.

Solution The moment-generating function of X is

MX (t) = E etX

=

n

n x p (1 − p)n−x etx

x=0

x

n n (pet )x (1 − p)n−x . = x x=0

Using the binomial formula, we have # $n MX (t) = pet + (1 − p) ,

−∞ < t < ∞.

The first two derivatives of MX (t) are (t) = n#(1 − p) + pet $(n−1) pet MX

and (t) = n(n − 1)#(1 − p) + pet $(n−2) pet 2 + n#(1 − p) + pet $(n−1) pet . MX

Thus, (0) = np μ = E(X) = MX

102 CHAPTER 2 Basic Concepts from Probability Theory

and σ 2 = E X2 − μ2 = M (0) − (np)2 = n(n − 1) p2 + np − (np)2 = np(1 − p) .

Example 2.6.9 Let X be a random variable with pmf f (x) = e−λ λx /(x!), x = 0, 1, 2, . . . . (Such a random variable is called a Poisson r.v. and the distribution is called a Poisson distribution with parameter λ.) Find the mgf of X.

Solution By definition MX (t) = EetX =

∞

etx f (x)

x=0

=

∞

t x ∞ eλ e−λ λx tx −λ e e = x!

x=0

= e−λ

∞

x=0

% t eλe

x=0 t = eλ(e −1)

x!

x & t e−(λe ) λet x!

% x & t ∞ e−(λe ) λet x!

x=0

∞ −(λet ) t x x t e (λe ) = 1. Thus We observe that e−(λe ) λet /x! is a Poisson pf with parameter λet . Hence x! x=0

from (1), MX (t) = eλ(e −1) . t

Example 2.6.10 Let X be a random variable with pdf given by

f (x) = Find mgf MX (t).

1 −x/β , βe

0,

x>0 otherwise.

2.6 Moments and Moment-Generating Functions 103

Solution By definition of mgf, −∞

etx f (x)dx

Mx (t) = ∞

=

∞ 1 etx e−x/β dx β 0

1 = β

∞ 1 1 − −t x e β dx, t < β 0

%

= =

1 1 − − e β ((1/β) − t)

1 β −t

& ∞ x

x=0

1 1 β = , β 1 − βt 1 − βt

t

0, otherwise.

Can we obtain the probability density of the variable X with the foregoing information?

2.7 CHAPTER SUMMARY In this chapter, we have introduced the concepts of random events and probability, how to compute the probabilities of events using counting techniques. We have studied the concept of conditional probability, independence, and Bayes’ rule. Random variables and distribution functions, moments, and moment-generating functions of random variables have also been introduced. The following lists some of the key deﬁnitions introduced in this chapter. ■ ■ ■ ■ ■

Sample space Mutually exclusive events Informal deﬁnition of probability Classical deﬁnition of probability Frequency interpretation of probability

108 CHAPTER 2 Basic Concepts from Probability Theory

■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Axiomatic deﬁnition of probability Multinomial coefﬁcients Conditional probability Mutually independent events Pairwise independent events Random variable (r.v.) Discrete random variable Discrete probability mass function Cumulative distribution function Continuous random variable Expected value kth moment about the origin kth moment about its mean Skewness and kurtosis Moment-generating function

The following important concepts and procedures have been discussed in this chapter: ■ ■ ■ ■

■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Method of computing probability by the classical approach Some basic properties of probability Computation of probability using counting techniques Four sampling methods: ❏ Sampling with replacement and the objects are ordered ❏ Sampling without replacement and the objects are ordered ❏ Sampling without replacement and the objects are not ordered ❏ Sampling with replacement and the objects are not ordered Permutation of n objects taken m at a time Combinations of n objects taken m at a time Number of combinations of n objects into m classes Some properties of conditional probability Law of total probability Steps to apply Bayes’ rule Some properties of distribution function Some properties of expected value Expectation of function of a random variable Properties of moment-generating functions

2.8 COMPUTER EXAMPLES (OPTIONAL) The three softwares packages, Minitab, SPSS, and SAS, that we are using in this book are not speciﬁcally designed for probability computations. However, the following examples are given to demonstrate that we will be able to use the software for some basic probability computations. We do not recommend using any of these three software packages for probability calculations; they are basically

2.8 Computer Examples (Optional) 109

designed for statistical computations. There are many other software packages such as Maple or MATLAB, that can be used efﬁciently for probability computations.

2.8.1 Minitab Computations In order to ﬁnd the cdf of a random variable, we can use the following commands in Example 2.8.1. We can use the mathematical expressions to ﬁnd the expected value of a discrete random variable.

Example 2.8.1 A random variable X has the following distribution: x p(x)

1 0.2

4 0.2

5 0.1

8 0.15

11 0.35

Find P(X ≤ 4).

Solution Enter x values in C1 and p(x) values in C2.

Calc > Probability Distributions > Discrete. . . > click Cumulative probability, and in Values in: enter C1, Probabilities in: enter C2, click input column: enter C1, in Optional storage: enter C3 > OK We will get the following output in column C3. 0.20

0.40

0.50

0.65

1.00

Example 2.8.2 For the random variable X in Example 2.8.1, ﬁnd E(X).

Solution Enter x values in column C1 (i.e., 1 4 5 8 11), and enter p(x) values in column C2. Use the following procedure.

Calc > Calculator. . . > Store results in variable: type C3 > in Expression: type (C1)*(C2) > click OK Then to find the sum of values in column C3 > Calc > Column Statistics. . . > click Sum and in Input variable: type C3 > click OK

We will get the output as Column Sum Sum of C3 = 6.5500

110 CHAPTER 2 Basic Concepts from Probability Theory

Note that this Sum gives the E(X). In the previous procedure, if we store the expression (C1)*(C1)*(C2) in column C4 and ﬁnd the sum of terms in C4, we will get E X2 . Using this, we will be able to compute Var(X). Using a similar procedure, we can obtain E(Xn ) for any n ≥ 1.

2.8.2 SPSS Examples Example 2.8.3 For the random variable X in Example 2.8.1, ﬁnd E(X).

Solution In column 1, enter the x values and column 2 enter the p(x) values. Then Transform > compute. . . > in target variable: type a name, say, product. Move var00001 and var00002 to Numeric Expression: field and put ‘‘*’’ in between them as (var00001)*(var00002). Then use the SUM(. , .) command to find the value of E(X)

2.8.3 SAS Examples Example 2.8.4 A random variable X has the following distribution: x P(X)

2 0.1

5 0.2

6 0.3

8 0.1

9 0.3

Using SAS, ﬁnd E(X).

Solution For discrete distributions where the random variable takes finite values, we can adapt the following procedure: data evalue; input x y n; z=x*y*n; cards; 2 .1 5 5 .2 5 6 .3 5 8 .1 5 9 .3 5 ; run;

2.8 Computer Examples (Optional) 111

proc means; run;

xρ(x); hence, multiplying by n, We know that if proc means is used just for x∗ y, that will give us 1n the number of values X takes will give us E(X) = xp(x). We will get the following output: The MEANS Procedure Variable N

Mean

Std Dev

Minimum

Maximum

=================================================== x

5 6.0000000 2.7386128 2.0000000

9.0000000

y

5 0.2000000 0.1000000 0.1000000

0.3000000

n

5 5.0000000

5.0000000

z

5 6.5000000 4.8476799 1.0000000

0

5.0000000

13.5000000

From this, we can see that E(X) = 6.5. A direct way to ﬁnd the expected value is by using “PROC IML.” options nodate nonumber; /* Finding expected value of a random variable */ proc iml; /* deﬁning all the variables */ x={2 5 6 8 9}; /* a row vector */ y={.1 .2 .3 .1 .3}; /* probabilities */ /* calculations */ z=x*y‘; /* print statements */ print “Display the vector x and probability y and the expected value”; print x y, z; quit;

We will get the following output: X 2

5

6

8

9

Y 0.1 0.2 0.3 0.1 0.3 Z 6.5

112 CHAPTER 2 Basic Concepts from Probability Theory

PROJECTS FOR CHAPTER 2 2A. The Birthday Problem The famous birthday problem is to ﬁnd the smallest number of people one must ask to get an even chance that at least two people have the same birthday. To solve this you can use the following steps. Find the probability that in a group of k people no two have the same probability. Let q be this probability. Then p = 1 − q is the probability that at least two people have the same birthday. Ignoring leap years, take the sample space S as all sequences of length k with each element one of the 365 days in the year. Thus there are 365k elements in S. (a) Find the total number of sequences with no common birthdays. (b) Assuming that each sequence is equally likely, show that q=

(365)(364) . . . (365 − k + 1) . 365k

(c) Write a computer program for calculating q for k = 2 to 50, and ﬁnd the ﬁrst k for which p > 0.5. This will give the least number of people we should ask to make it an even chance that at least two people will have the same birthday.

2B. The Hardy--Weinberg Law Hereditary traits in offspring depend on a pair of genes, one each contributed by the father and the mother. A gene is either a dominant allele, denoted by A, or a recessive allele, denoted by a. If the genotype is AA, Aa, or aA, then the hereditary trait is A, and if the genotype is aa, then the hereditary trait is a. Suppose that the probabilities of the mother carrying the genotypes aa, aA (same as Aa), and AA are p, q, and r, respectively. Here p + q + r = 1. The same probabilities are true for the father. (a) Assuming that the genetic contributions of the mother and father are independent and the matings are random, show that the respective probabilities for the ﬁrst-generation offspring are p1 = (p + q/2)2 , q1 = 2 (r + q/2) (p + q/2) , r1 = (r + q/2)2 .

Also ﬁnd P(A) and P(a). (b) The Englishman G. H. Hardy and the German W. Weinberg could show that the foregoing probabilities in a population stay constant for generations if certain conditions are fulﬁlled. This is known as the Hardy–Weinberg law. Under the conditions of part (a), using the induction argument, show that the Hardy–Weinberg law is satisﬁed, i.e., pn = p1 , qn = q1 , and rn = r1 for all n ≥ 1. The consequences of the Hardy–Weinberg law are that (i) no evolutionary change occurs through the process of sexual reproduction itself, and (ii) changes in allele and genotype frequencies can result only from additional forces on the gene pool of a species.

Chapter

3

Additional Topics in Probability Objective: In this chapter we present some special distributions, joint distributions of several random variables, functions of random variables, and some important limit theorems. 3.1 Introduction 114 3.2 Special Distribution Functions 114 3.3 Joint Probability Distributions 141 3.4 Functions of Random Variables 154 3.5 Limit Theorems 163 3.6 Chapter Summary 173 3.7 Computer Examples (Optional) 175 Projects for Chapter 3 180

Johann Carl Friedrich Gauss (Source: http://tobiasamuel.ﬁles.wordpress.com/2008/06/carl_friedrich_gauss.jpg)

German mathematician and physicist Carl Friedrich Gauss (1777–1855) is sometimes called the “prince of mathematics.” He was a child prodigy. At the age of 7, Gauss started elementary school,

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

113

114 CHAPTER 3 Additional Topics in Probability

and his potential was noticed almost immediately. His teachers were amazed when Gauss summed the integers from 1 to 100 instantly. At age 24, Gauss published one of the most brilliant achievements in mathematics, Disquisitiones Arithmeticae (1801). In it, Gauss systematized the study of number theory. Gauss applied many of his mathematical insights in the ﬁeld of astronomy, and by using the method of least squares he successfully predicted the location of the asteroid Ceres in 1801. In 1820 Gauss made important inventions and discoveries in geodesy, the study of the shape and size of the earth. In statistics, he developed the idea of the normal distribution. In the 1830s he developed theories of non-Euclidean geometry and mathematical techniques for studying the physics of ﬂuids. Although Gauss made many contributions to applied science, especially electricity and magnetism, pure mathematics was his ﬁrst love. It was Gauss who ﬁrst called mathematics “the queen of the sciences.”

3.1 INTRODUCTION In the previous chapter, we looked at the basic concepts of probability calculations, random variables, and their distributions. There are many special distributions that have useful applications in statistics. It is worth knowing the type of distribution that we can expect under different circumstances, because a better knowledge of the population will result in better inferential results. In the next section, we discuss some of these distributions with some additional distributions presented in Appendix A3. We also brieﬂy deal with joint distributions of random variables and functions of random variables. Limit theorems play an important role in statistics. We will present two limit theorems: the law of large numbers and the Central Limit Theorem.

3.2 SPECIAL DISTRIBUTION FUNCTIONS Random variables are often classiﬁed according to their probability distribution functions. In any analysis of quantitative data, it is a major step to know the form of the underlying probability distributions. There are certain basic probability distributions that are applicable in many diverse contexts and thus repeatedly arise in practice. A great variety of special distributions have been studied over the years. Also, new ones are frequently being added to the literature. It is impossible to give a comprehensive list of distribution functions in this book. There are many books and Web sites that deal with a range of distribution functions. A good list of distributions can be obtained from http://www.causascientia.org/math_stat/Dists/Compendium.pdf. In this section, we will describe some of the commonly used probability distributions. In Appendix A3, we list some more distributions with their mean, variance, and moment-generating functions. First we discuss some discrete probability distributions.

3.2.1 The Binomial Probability Distribution The simplest distribution is the one with only two possible outcomes. For example, when a coin (not necessarily fair) is tossed, the outcomes are heads or tails, with each outcome occurring with some positive probability. These two possible outcomes may be referred to as “success” if heads occurs and “failure” if tails occurs. Assume that the probability of heads appearing in a single toss is p; then the probability of tails is 1 − p = q. We deﬁne a random variable X associated with this experiment

3.2 Special Distribution Functions 115

as taking value 1 with probability p if heads occurs and value 0 if tails occurs with probability q. Such a random variable X is said to have a Bernoulli probability distribution. That is, X is a Bernoulli random variable if for some p, 0 ≤ p ≤ 1, the probability P(X = 1) = p and P(X = 0) = 1 − p. The probability function of a Bernoulli random variable X can be expressed as p(x) = P(X = x) =

px (1 − p)1−x ,

x = 0, 1

0,

otherwise.

Note that this distribution is characterized by the single parameter p. It can be easily veriﬁed that the mean and variance of X are E[X] = p, var(X) = pq, respectively, and the moment-generating function is MX (t) = pet + (1 − p). Even when the experimental values are not dichotomous, reclassifying the variable as a Bernoulli variable can be helpful. For example, consider blood pressure measurements. Instead of representing the numerical values of blood pressure, if we reclassify the blood pressure as “high blood pressure” and “low blood pressure,” we may be able to avoid dealing with a possible misclassiﬁcation due to diurnal variation, stress, and so forth, and concentrate on the main issue, which would be: Is the average blood pressure unusually high? In a succession of Bernoulli trials, one is more interested in the total number of successes (whenever a 1 occurs in a Bernoulli trial, we term it a “success”). The probability of observing exactly k successes in n independent Bernoulli trials yields the binomial probability distribution. In practice, the binomial probability distribution is used when we are concerned with the occurrence of an event, not its magnitude. For example, in a clinical trial, we may be more interested in the number of survivors after a treatment. Deﬁnition 3.2.1 A binomial experiment is one that has the following properties: (1) The experiment consists of n identical trials. (2) Each trial results in one of the two outcomes, called a success S and failure F. (3) The probability of success on a single trial is equal to p and remains the same from trial to trial. The probability of failure is 1 − p = q. (4) The outcomes of the trials are independent. (5) The random variable X is the number of successes in n trials. Earlier we have seen that the number of ways of obtaining x successes in n trials is given by

n! n = . x!(n − x)! x

Deﬁnition 3.2.2 A random variable X is said to have binomial probability distribution with parameters (n, p) if and only if

n x n−x P(X = x) = p(x) = p q x ⎧ n! ⎨ x!(n−x)! px qn−x , x = 0, 1, 2, . . . , n, 0 ≤ p ≤ 1, and q = 1 − p = ⎩ 0, otherwise.

116 CHAPTER 3 Additional Topics in Probability

To show the dependence on n and p, denote p(x) by b(x, n, p) and the cumulative probabilities by B(x, n, p) =

x

b(i, n, p)

i=0

Binomial probabilities are tabulated in the binomial table. By the binomial theorem, we have (p + q)n =

n n x=0

x

px qn−x .

n x n−x p q = 1n = 1, for all n ≥ 1 x=0 x and 0 ≤ p ≤ 1. Hence, p(x) is indeed a probability function. The binomial probability distribution is characterized by two parameters, the number of independent trials n and the probability of success p.

Because (p + q) = 1, we conclude that

x

i=0 b(i, n, p) =

n

Example 3.2.1 It is known that screws produced by a certain machine will be defective with probability 0.01 independently of each other. If we randomly pick 10 screws produced by this machine, what is the probability that at least two screws will be defective?

Solution Let X be the number of defective screws out of 10. Then X can be considered as a binomial r.v. with parameters (10, 0.01). Hence, using the binomial pf p(x), given in Definition 3.2.2, we obtain P(X ≥ 2) =

10 10 x=2

x

(0.01)x (0.99)10−x

= 1 − [P(X = 0) + P(X = 1)] = 0.004.

In Chapter 2, we saw Mendel’s law. In biology, the result “gene frequencies and genotype ratios in a randomly breeding population remain constant from generation to generation” is known as the Hardy–Weinberg law.

Example 3.2.2 Suppose we know that the frequency of a dominant gene, A, in a population is equal to 0.2. If we randomly select eight members of this population, what is the probability that at least six of them will display the dominant phenotype? Assume that the population is sufﬁciently large that removing eight individuals will not affect the frequency and that the population is in Hardy–Weinberg equilibrium.

3.2 Special Distribution Functions 117

Solution First of all, note that an individual can have the dominant gene, A, if the person has traits AA, aA, or Aa. Hence, if the gene frequency is 0.2, the probability that an individual is of genotype A is P(A) = P(AA ∪ Aa ∪ aA) = P(AA) + 2P(Aa) = (0.2)2 + 2(0.2)(0.8) = 0.36.

Let X denote the number of individuals out of eight that display the dominant phenotype. Then X is binomial with n = 8, and p = 0.36. Thus, the probability that at least six of them will display the dominant phenotype is P(X ≥ 6) = P(X = 6) + P(X = 7) + P(X = 8) =

8 10 i=6

i

(0.36)i (0.64)10−i = 0.029259.

For large n, calculation of binomial probabilities is tedious. Many statistical software packages have binomial probability distribution commands. For the purpose of this book, we will use the binomial table that gives the cumulative probabilities B(x, n, p) for n = 2 through n = 20 and p = 0.05, 0.10, 0.15, . . . , 0.90, 0.95. If we need the probability of a single term, we can use the relation P(X = x) = b(x, n, p) = B(x, n, p) − B(x − 1, n, p).

Example 3.2.3 A manufacturer of inkjet printers claim that only 5% of their printers require repairs within the ﬁrst year. If of a random sample of 18 of the printers, four required repairs within the ﬁrst year, does this tend to refute or support the manufacturer’s claim?

Solution Let us assume that the manufacturer’s claim is correct; that is, the probability that a printer will require repairs within the first year is 0.05. Suppose 18 printers are chosen at random. Let p be the probability that any one of the printers will require repairs within the first year. We now find the probability that at least four of these out of the 18 will require repairs during the first year. Let X represent the number of printers that require repair within the first year. Then X follows the binomial pmf with p = 0.05, n = 18. The probability that four or more of the 18 will require repair within the first year is given by P(X ≥ 4) =

18 18 x=4

x

(0.05)x (0.95)18−x

118 CHAPTER 3 Additional Topics in Probability

or, using the binomial table, 18

b(x, 18, 0.05) = 1 − B(3, 18, 0.05)

x=4

= 1 − 0.9891 = 0.0109. This value (approximately 1.1%) is very small. We have shown that if the manufacturer’s claim is correct, then the chances of observing four or more bad printers out of 18 are very small. But we did observe exactly four bad ones. Therefore we must conclude that the manufacturer’s claim cannot be substantiated.

MEAN, VARIANCE, AND MGF OF A BINOMIAL RANDOM VARIABLE Theorem 3.2.1 If X is a binomial random variable with parameters n and p, then E(X) = μ = np Var(X) = σ 2 = np(1 − p). Also the moment-generating function # $n MX (t) = pet + (1 − p) .

Proof. We derive the mean and the variance. The derivation for mgf is given in Example 2.6.5. Using the binomial pmf, p(x) = (n!/(x!(n − x)!))px qn−x , and the deﬁnition of expectation, we have μ = E(X) =

n

xp(x) =

x=0

=

n x=1

n x=0

x

n! px (1 − p)n−x x!(n − x)!

n! px (1 − p)n−x , (x − 1)!(n − x)!

since the ﬁrst term in the sum is zero, as x = 0. Let i = x − 1. When x varies from 1 through n, i = (x − 1) varies from zero through (n − 1). Hence, μ=

n−1

n! pi+1 (1 − p)n−i−1 i!(n − i − 1)!

i=0 n−1

= np

i=0

= np,

(n − 1)! pi (1 − p)n−1−i i!(n − 1 − i)!

3.2 Special Distribution Functions 119

because the last summand is that of a binomial pmf with parameter (n − 1) and p, hence, equals 1. To ﬁnd the variance, we ﬁrst calculate E [X(X − 1)]. E [X(X − 1)] =

n

x(x − 1)

x=0

=

n x=2

n! px (1 − p)n−x x!(n − x)!

n! px (1 − p)n−x , (x − 2)!(n − x)!

because the ﬁrst two terms are zero. Let i = x − 2. Then, E [X(X − 1)] =

n−2 i=0

n! pi+2 (1 − p)n−i−2 i!(n − i − 2)!

= n(n − 1)p2

n−2 i=0

(n − 2)! pi (1 − p)n i!(n − 2 − i)!

= n(n − 1)p2 ,

because the last summand is that of a binomial pf with parameter (n − 2) and p thus equals 1. Note that E(X(X − 1)) = EX2 − E(X), and so we obtain σ 2 = Var(X) = E(X2 ) − [E(X)]2 = E [X(X − 1)] + E(X) − [E(X)]2 = n(n − 1)p2 + np − (np)2 = −np2 + np = np(1 − p).

3.2.2 Poisson Probability Distribution The Poisson probability distribution was introduced by the French mathematician Siméon-Denis Poisson in his book published in 1837, which was entitled Recherches sur la probabilité des jugements en matières criminelles et matière civile and dealt with the applications of probability theory to lawsuits, criminal trials, and the like. Consider a statistical experiment of which A is an event of interest. A random variable that counts the number of occurrences of A is called a counting random variable. The Poisson random variable is an example of a counting random variable. Here we assume that the numbers of occurrences in disjoint intervals are independent and the mean of the number occurrences is constant.

120 CHAPTER 3 Additional Topics in Probability

Deﬁnition 3.2.3 A discrete random variable X is said to follow the Poisson probability distribution with parameter λ > 0, denoted by Poisson(λ), if P(X = x) = f (x, λ) = f (x) =

e−λ λx , x!

x = 0, 1, 2, . . .

The Poisson probability distribution is characterized by the single parameter, λ, which represents the mean of a Poisson probability distribution. Thus, in order to specify the Poisson distribution, we only need to know the mean number of occurrences. This distribution is of fundamental theoretical and practical importance. Rare events are modeled by the Poisson distribution. For example, the Poisson probability distribution has been used in the study of telephone systems. The number of incoming calls into a telephone exchange during a unit time might be modeled by a Poisson variable assuming that the exchange services a large number of customers who call more or less independently. Some other problems where Poisson representation can be used are the number of misprints in a book, radioactivity counts per unit time, the number of plankton (microscopic plant or animal organisms that ﬂoat in bodies of water) per aliquot of seawater, or count of bacterial colonies per petri plate in a microbiological study. In stem cell research, the Poisson distribution is used to analyze the redundancy of clusters in the stem cell database. A Poisson probability distribution has the unique property that its mean equals its variance.

MEAN, VARIANCE, AND MOMENT–GENERATING FUNCTION OF A POISSON RANDOM VARIABLE Theorem 3.2.2 If X is a Poisson random variable with parameter λ, then E(X) = λ Var(X) = λ. Also the moment-generating function is MX (t) = eλ(e −1) . t

The proof of this result is similar to that we used in Theorem 3.2.1 in this section. One needs to use i the Maclaurin’s expansion, eλ = ∞ i=0 (λ /i!).

Example 3.2.4 Let X be a Poisson random variable with λ = 1/2. Find (a) P(X = 0) (b) P(X ≥ 3)

Solution (a) We have P(X = 0) = p(0) =

e−1/2 (1/2)0 = e−1/2 = 0.60653. 0!

3.2 Special Distribution Functions 121

(b) Here we will use complementary event to compute the required probability. That is, P(X ≥ 3) = 1 − P(X ≤ 2) = 1 − [p(0) + p(1) + p(2)] % & e−1/2 (1/2) e−1/2 (1/2)2 −1/2 =1− e + + 1! 2! = 1 − 0.98561 = 0.01439. When n is large and p small, binomial probabilities are often approximated by Poisson probabilities. In these situations, where performing the factorial and exponential operations required for direct calculation of binomial probabilities is a lengthy and tedious process and tables are not available, the Poisson approximation is more feasible. The following theorem states this result.

POISSON APPROXIMATION TO THE BINOMIAL PROBABILITY DISTRIBUTION Theorem 3.2.3 If X is a binomial r.v. with parameters n and p, then for each value x = 0, 1, 2, . . . and as p → 0, n → ∞ with np = λ constant, lim

n→∞

n x

px (1 − p)n−x =

e−λ λx . x!

The proof of this result is similar to that we used in Theorem 3.2.1. In the present context, the Poisson probability distribution is sometimes referred to as “the distribution of rare events” because of the fact that p is quite small when n is large. Usually, if p ≤ 0.1 and n ≥ 40 we could use the Poisson approximation in practice. In general, another rule of thumb is to use Poisson approximation to binomial in the case of np < 5.

Example 3.2.5 If the probability that an individual suffers an adverse reaction from a particular drug is known to be 0.001, determine the probability that out of 2000 individuals, (a) exactly three and (b) more than two individuals will suffer an adverse reaction.

Solution Let Y be the number of individuals who suffer an adverse reaction. Then Y is binomial with n = 2000 and p = 0.001. Because n is large and p is small, we can use the Poisson approximation with λ = np = 2. (a) The probability that exactly three individuals will suffer an adverse reaction is P(Y = 3) =

23 e−2 = 0.18. 3!

That is, there is approximately an 18% chance that exactly three individuals of 2000 will suffer an adverse reaction.

122 CHAPTER 3 Additional Topics in Probability

(b) The probability that more than two individuals will suffer an adverse reaction is P(Y > 2) = 1 − P(Y = 0) − P(Y = 1) − P(Y = 2) = 1 − 5e−2 = 0.323. Similarly, there is approximately a 32.3% chance that more than two individuals will have an adverse reaction.

Now we will discuss some continuous distributions. As mentioned earlier, if X is a continuous random variable with pdf f (x), then b P(a ≤ X ≤ b) =

f (x)dx. a

3.2.3 Uniform Probability Distribution The uniform probability distribution is used to generate random numbers from other distributions and also is useful as a “ﬁrst guess” if no other information about a random variable X is known, other than that it is between a and b. Also, in real-world problems that have uniform behavior in a given interval, we can characterize the probabilistic behavior of such a phenomenon by the uniform distribution. (See Figure 3.1.) Deﬁnition 3.2.4 A random variable X is said to have a uniform probability distribution on (a, b), denoted by U(a, b), if the density function of X is given by ⎧ ⎨ 1 , f (x) = b − a ⎩ 0,

a ≤ x ≤ b, otherwise.

The cumulative distribution function is given by ⎧ 0, ⎪ ⎪ ⎪ ⎨ 1 x−a dx = F (x) = , ⎪ b−a ⎪b − a ⎪ −∞ ⎩ 1, x

f (x ) ⫽ 1/(b ⫺ a ) f (x ) ⫽ 0

f (x ) ⫽ 0 a

b

■ FIGURE 3.1 Uniform probability density.

x 6) =

4 1 dx = . 10 10

6

(c) 8 P(3 < X < 8) =

1 1 dx = . 10 2

3

MEAN, VARIANCE, AND MOMENT–GENERATING FUNCTION OF A UNIFORM RANDOM VARIABLE Theorem 3.2.4 If X is a uniformly distributed random variable on (a, b), then E(X) =

a+b . 2

and Var(X) =

(b − a)2 . 12

Also, the moment-generating function is ⎧ tb ta ⎪ ⎨e − e , MX (t) = t(b − a) ⎪ ⎩ 1,

t = 0 t = 0.

124 CHAPTER 3 Additional Topics in Probability

Proof. We will obtain the mean and the variance and leave the derivation of the moment-generating function as an exercise. By deﬁnition we have ∞ E(X) =

x −∞

b = a

1 dx b−a

⎛ ⎞ b 1 ⎝ x2 ⎠ 1 dx = x b−a b−a 2 a

a+b . = 2

Also E(X2 ) =

b a

= =

⎛ ⎞ 3 b 1 1 ⎝ x ⎠ dx = x2 b−a b−a 3 a

1 b3 − a3 3 b−a 1 2 (b + ab + a2 ) as b3 − a3 = (b − a)(b2 + ab + a2 ). 3

Thus, Var(X) = E(X2 ) − (E(X))2 1 2 (a + b)2 (b + ab + a2 ) − 3 4 1 (b − a)2 . = 12 =

Example 3.2.7 The melting point, X, of a certain solid may be assumed to be a continuous random variable that is uniformly distributed between the temperatures 100◦ C and 120◦ C. Find the probability that such a solid will melt between 112◦ C and 115◦ C.

Solution The probability density function is given by ⎧ ⎨ 1 , f (x) = 20 ⎩ 0

100 ≤ x ≤ 120 otherwise.

3.2 Special Distribution Functions 125

Hence, 115 P(112 ≤ X ≤ 115) =

3 1 dx = = 0.15. 20 20

112

Thus, there is a 15% chance of this solid melting between 112◦ C and 115◦ C.

3.2.4 Normal Probability Distribution The single most important distribution in probability and statistics is the normal probability distribution. The density function of a normal probability distribution is bell shaped and symmetric about the mean. The normal probability distribution was introduced by the French mathematician Abraham de Moivre in 1733. He used it to approximate probabilities associated with binomial random variables when n is large. This was later extended by Laplace to the so-called Central Limit Theorem, which is one of the most important results in probability. Carl Friedrich Gauss in 1809 used the normal distribution to solve the important statistical problem of combining observations. Because Gauss played such a prominent role in determining the usefulness of the normal probability distribution, the normal probability distribution is often called the Gaussian distribution. Gauss and Laplace noticed that measurement errors tend to follow a bell-shaped curve, a normal probability distribution. Today, the normal probability distribution arises repeatedly in diverse areas of applications. For example, in biology, it has been observed that the normal probability distribution ﬁts data on the heights and weights of human and animal populations, among others. We should also mention here that almost all basic statistical inference is based on the normal probability distribution. The question that often arises is, when do we know that our data follow the normal distribution? To answer this question we have speciﬁc statistical procedures that we study in later chapters, but at this point we can obtain some constructive indications of whether the data follows the normal distribution by using descriptive statistics. That is, if the histogram of our data can be capped with a bell-shaped curve (Figure 3.2), if the stem-and-leaf diagram is fairly symmetrical with respect to its center, and/or by invoking the empirical rule “backwards,” we can obtain a good indication whether our data follow the normal probability distribution. Deﬁnition 3.2.5 A random variable X is said to have a normal probability distribution with parameters μ and σ 2 , if it has a probability density function given by f (x) = √

1 2πσ

2 2 e−(x−μ) /2σ , −∞ < x < ∞, −∞ < μ < ∞, σ > 0.

If μ = 0, and σ = 1, we call it standard normal random variable. For any normal random variable with mean μ and variance σ 2 , we use the notation X ∼ N(μ, σ 2 ). When a random variable X has a standard normal probability distribution, we will write X ∼ N(0, 1) (X is a normal with mean 0 and variance 1). Probabilities for a standard normal probability distribution are given in the normal table.

126 CHAPTER 3 Additional Topics in Probability

0.5

0.4

0.3

0.2

0.1

0.0 0 ■ FIGURE 3.2 Standard normal density function.

MEAN, VARIANCE, AND MGF OF A NORMAL RANDOM VARIABLE Theorem 3.2.5 If X ∼ N(μ, σ 2 ), then E(X) = μ and Var(X) = σ 2 . Also the moment-generating function is 1 2 2 MX (t) = etμ+ 2 t σ .

If X ∼ N(μ, σ 2 ), then the z-transform (or z-score) of X, Z = X−μ σ , is an N(0, 1) random variable. This fact will be used in calculating probabilities for normal random variables.

Example 3.2.8 (a) For X ∼ N(0, 1), calculate P(Z ≥ 1.13). (b) For X ∼ N(5, 4), calculate P(−2.5 < X < 10).

Solution (a) Using the normal table, P(Z ≥ 1.13) = 1 − 0.8708 = 0.1292. The shaded part in the graph represents the P(Z ≥ 1.13). 0.5

0.5

0.4

0.4

0.3

0.3

0.2

0.2

0.1

0.1 0.0

0.0 1.13

22.5 5

10

3.2 Special Distribution Functions 127

(b) Using the z-transform, we have

P(−2.5 < X < 10) = P

−2.5 − 5 10 − 5 z0 ) = 0.25. (b) P(Z < z0 ) = 0.95. (c) P(Z < z0 ) = 0.12. (d) P(Z > z0 ) = 0.68.

Solution (a) From the normal table, and using the fact that the shaded area in the figure is 0.25, we obtain z0 ≈ 0.675. (b) Because P(Z < z0 ) = 1 − P(Z ≥ z0 ) = 0.95 = 0.5 + 0.45. This implies, P(Z > z0 ) = 0.05. From the normal table, z0 = 1.645.

0.5 0.4 0.3 0.2 0.1 0.0 Z0

(c) From the normal table, z0 = −1.175. (d) Using the normal table, we have P(Z > z0 ) = 0.5 + P(0 < Z < z0 ) = 0.68. This implies, P(Z ≤ z0 ) = 0.32. From the normal table, z0 = − 0.465.

Example 3.2.10 The scores of an examination are assumed to be normally distributed with μ = 75 and σ 2 = 64. What is the probability that a score chosen at random will be greater than 85?

128 CHAPTER 3 Additional Topics in Probability

Solution Let X be a randomly chosen score from the exam scores. Then, X ∼ N(75, 64).

X − 75 85 − 75 P(X > 85) = P > = 1.25 8 8 = P(Z > 1.25) = 0.1056. 0.5 0.4 0.3 0.2 0.1 0.0 1.25

Thus, there is about a 10.56% chance that the score will be greater than 85.

In practice, whenever a large number of small effects are present and acting additively, it is reasonable to assume that observations will be normal. When the number of data is small, it is risky to assume a normal distribution without a proper testing. Apart from histogram, box-plot, and stem-and-leafdisplays, one of the most useful tools for assessing normality is a quantile quantile or QQ plot. This is a scatterplot with the quantiles of the scores on the horizontal axis and the expected normal scores on the vertical axis. The expected normal scores are calculated by taking the z-scores of (ri −0.5)/n, where ri is the rank ith observation in increasing order. The steps in constructing a QQ plot are as follows: First, we sort the data in an ascending order. If the plot of these scores against the expected normal scores is a straight line, then the data can be considered normal. Any curvature of the points indicates departures from normality. This procedure obtaining a normal plot (QQ plot is similar to normal plot for a normal distribution) is described in Project 4C. Figure 3.3 shows a normal probability plot generated by Minitab. If plotted points do not ﬁt the line well, but bend away from it in places, the distribution may be nonnormal. The shapes in Figure 3.4 will give some indication of the distribution of the data. 0.999 0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 21.5

21.0

■ FIGURE 3.3 Normal probability plot.

20.5

0.0

0.5

3.2 Special Distribution Functions 129

If the layout of points appears to bend up and to the left of the normal line that indicates a long tail to the right, or right skew.

If the layout of points bends down and to the right of the normal line that indicates a long tail to the left, or left skew.

An S-shaped layout of points indicates shorter than normal tails, thus, a smaller variance is expected.

If the layout of points starts below the normal line, bends to follow it, and ends above it, this will indicate long tails. That is, there is more variance than we would expect in a normal distribution.

■ FIGURE 3.4 Shapes indicating distribution of the data.

Almost all of the statistical software packages include a procedure for obtaining the graph of a normal probability plot that can be used to test the normality of a data. A discussion of how to do this is given in Section 14.4. Errors in the measurements can also act in a multiplicative (rather than additive) manner. In that case, the assumption of normality is not justiﬁed. A closely related distribution to normal distribution is the log-normal distribution. A variable might be modeled as log-normal if it can be thought of as the multiplicative effect of many small independent factors. This distribution arises in physical problems when the domain of the variate, X, is greater than zero and its histogram is markedly skewed. If a random variable Y is normally distributed, then exp(Y ) has a log-normal distribution. Thus, the natural logarithm of a log-normally distributed variable is normally distributed. That is, if X is a random variable with log-normal distribution, then ln(X) is normally distributed. Most biological evidence suggests that the growth processes of living tissue proceed by multiplicative, not additive, increments. Thus, the measures of body size should at most follow a log-normal rather than normal distribution. Also, the sizes of plants and animals is approximately log-normal. The log-normal distribution is also useful in modeling of claim sizes in the insurance industry. The probability density function of a log-normal random variable, X, is given as f (x) =

⎧ ⎨ ⎩

2 2 1 √ e−(ln x−μy ) /2σy , xσy 2π

x > 0, σy > 0,

0,

otherwise.

−∞ < μy < ∞

130 CHAPTER 3 Additional Topics in Probability

where μy and σy are the mean and standard deviation of Y = ln(X). These parameters are related to the parameters of the random variable X as follows: + μy = ln

+

μ4x , μ2x + σx2

σy = ln

μ2x + σx2 . μ2x

We can verify that the expected value X is E(X) = eμy +(σy /2) 2

and the variance is Var(X) = (eσy − 1)e2μy +σy . 2

2

The question of when the log-normal distribution is applicable in a given physical problem after a certain amount of data has been obtained can be answered by creating a normal probability plot of ln(X) and testing for normality. Thus, if the natural logarithms of the data show normality, log-normal distribution may be more appropriate. If X is log-normally distributed with parameters μy and σy , and 0 < a < b, then with Y = ln(X) P(a ≤ X ≤ b) = P(ln a ≤ Y ≤ ln b)

Y − μy ln b − μy ln a − μy ≤ ≤ =P σy σy σy = P(a ≤ Z ≤ b ),

where Z ∼ N(0, 1). This probability can be obtained from the standard normal table.

Example 3.2.11 In an effort to establish a suitable height for the controls of a moving vehicle, information was gathered about X, the amounts by which the heights of the operators vary from 60 inches, which is the minimum height. It was veriﬁed that the data that were collected followed the log-normal distribution by normal probability plot of Y = ln X. Assume that μx = 6 in. and σx = 2 in. (a) What percentage of operators would have a height less than 65.5 in.? (b) If an operator is chosen at random, what is the probability that his or her height will be between 64 and 66 in.?

Solution (a) Here, X = 65.5 − 60 = 5.5. Also, + μy = ln

μ4x 2 μx + σx2

+

= ln

64 62 + 2 2

= 1.74,

3.2 Special Distribution Functions 131 + σy = ln

μ2x + σx2 μ2x

+

= ln

Thus,

62 + 22 = 0.053. 62

(ln 5.5) − 1.74 P(X ≤ 5.5) = P(Y ≤ ln 5.5) = P Z ≤ 0.053

= P(Z ≤ −0.67) = 0.2514. Hence, about 25.14% of the heights of the operators vary from 60 inches. (b) Similar to part (a), we get P(4 ≤ X ≤ 6) = P(ln 4 ≤ Y ≤ ln 6)

(ln 6) − 1.74 (ln 4) − 1.74 ≤Z≤ =P 0.053 0.053 = P(−6.67 ≤ Z ≤ 0.98) = 0.8365.

3.2.5 Gamma Probability Distribution The gamma probability distribution has found applications in various ﬁelds. For example, in engineering, the gamma probability distribution has been employed in the study of system reliability. We describe the gamma function before we introduce the gamma probability distribution. The gamma function, denoted by (a), is deﬁned as (a) =

∞ e−x xa−1 dx, a > 0. 0

It can be shown using the integration by parts that for a > 1, (a) = (a − 1) (a − 1). In particular, if n is a positive integer, (n) = (n − 1)!. Deﬁnition 3.2.6 A random variable X is said to possess a gamma probability distribution with parameters α > 0 and β > 0 if it has the pdf given by f (x) =

⎧ ⎨

1 xα−1 e−x/β , βα (α) ⎩ 0,

if x > 0 otherwise.

The gamma density has two parameters, α and β. We denote this by Gamma(α, β). The parameter α is called a shape parameter, and β is called a scale parameter. Changing α changes the shape of the density, whereas varying β corresponds to changing the units of measurement (such as changing from seconds to minutes). Varying these two parameters will generate different members of the gamma family. If we take α to be a positive integer, we get a special case of gamma probability distribution, known as the Erlang distribution. This is used extensively in queuing theory to model waiting times. Figure 3.5 gives an indication of how α and β inﬂuence the shape and scale of f (x).

132 CHAPTER 3 Additional Topics in Probability

Gamma pdfs for (2, 3), (3, 1), (4, 3), and (2, 4) 0.3 Gam(3, 1)

0.25 0.2 0.15

Gam(2, 3)

0.1

Gam(2, 4) Gam(4, 3)

0.05 0

0

5

10

15

20

25

■ FIGURE 3.5 Gamma pdfs for different degrees of freedom.

MEAN, VARIANCE, AND MGF OF A GAMMA RANDOM VARIABLE Theorem 3.2.6 If X is a gamma random variable with parameters α > 0 and β > 0, then E(X) = αβ

and

Var(X) = αβ2 .

Also, the moment-generating function is MX (t) =

1 , (1 − βt)α

t

0.

Hence, using integration by parts, we obtain 1 P(X < 1) = 2

1 0

x2 e−x dx = 1 −

5 = 0.08025. 2e

3.2 Special Distribution Functions 133

0.30 0.25 0.20 0.15 0.10 0.05 0.00

1

Thus, there is about an 8% chance that on a given day the fuel consumption will be less than 1 million gallons. (b) Because the airport can store only 2 million gallons, the fuel supply will be inadequate if the fuel consumption X is greater than 2. Thus, 1 P(X > 2) = 2

∞ x2 e−x dx = 0.677. 2

0.30 0.25 0.20 0.15 0.10 0.05 0.00

2

We can conclude that there is about a 67.7% chance that the fuel supply of 2 million gallons will be inadequate on a given day. So, if the model is right, the airport needs to store more than 2 million gallons of fuel.

We now describe two special cases of gamma probability distribution. In the pdf of the gamma, we let α = 1, we get the pdf of an exponential random variable. Deﬁnition 3.2.7 A random variable X is said to have an exponential probability distribution with parameter β if the pdf of X is given by ⎧ ⎨ 1 e−x/β , f (x) = β ⎩ 0,

β > 0; 0 ≤ x < ∞ otherwise.

Exponential random variables are often used to model the lifetimes of electronic components such as fuses, for survival analysis, and for reliability analysis, among others. The exponential distribution (Figure 3.6) is also used in developing models of insurance risks.

134 CHAPTER 3 Additional Topics in Probability

Exponential (3) 0.35 0.3 0.25 0.2 0.15 0.1 0.05 0

0

5

10

15

20

■ FIGURE 3.6 Probability density function for exponential r.v.

MEAN, VARIANCE, AND MGF OF AN EXPONENTIAL RANDOM VARIABLE Theorem 3.2.7 If X is an exponential random variable with parameters β > 0, then E(X) = β

and

Var(X) = β2 .

Also the moment-generating function is MX (t) =

1 , (1 − βt)

t

200) = 200 dx = 0.2865. The chance that the generator will last more than 160 e 200 hours is about 28.65%.

Another special case of gamma probability distribution that is useful in statistical inference problems is the chi-square distribution. Deﬁnition 3.2.8 Let n be a positive integer. A random variable, X, is said to have a chi-square (χ2 ) distribution with n degrees of freedom if and only if X is a gamma random variable with parameters α = n/2 and β = 2. We denote this by X ∼ χ2 (n). Hence, the probability density function of a chi-square distribution with n degrees of freedom is given by ⎧ ⎨ 1 x(n/2)−1 e−x/2 , f (x) = 2n 2n/2 ⎩ 0,

0≤x 0) (d) E(X) and Var(X)

3.2.2.

Let X be a Poisson random variable with λ = 1/3. Find (a) P(X = 0) (b) P(X ≥ 4).

3.2.3.

For a standard normal random variable Z, ﬁnd the value of z0 such that (a) P(Z > z0 ) = 0.05 (b) P(Z < z0 ) = 0.88 (c) P(Z < z0 ) = 0.10 (d) P(Z > z0 ) = 0.95.

3.2 Special Distribution Functions 137

3.2.4.

Let X ∼ N(12, 5). Find the value of x0 such that (a) P(X > x0 ) = 0.05 (b) P(X < x0 ) = 0.98 (c) P(X < x0 ) = 0.20 (d) P(X > x0 ) = 0.90.

3.2.5.

Let X ∼ N(10, 25). Compute (a) P(X ≤ 20) (b) P(X > 5) (c) P(12 ≤ X ≤ 15) (d) P(|X − 12| ≤ 15).

3.2.6.

A quarterback on a football team has a pass completion rate of 0.62. If, in a given game, he attempts 16 passes, what is the probability that he will complete (a) 12 passes? (b) More than half of his passes? (c) Interpret your result. (d) Out of the 16 passes, what is the expected number of completions?

3.2.7.

A consulting group believes that 70% of the people in a certain county are satisﬁed with their health coverage. Assuming that this is true, ﬁnd the probability that in a random sample of 15 people from the county: (a) Exactly 10 are satisﬁed with their health coverage, and interpret. (b) Not more than 10 are satisﬁed with their health coverage, and interpret. (c) What is the expected number of people out of 15 that are satisﬁed with their health coverage?

3.2.8.

A man ﬁres at a target six times; the probability of his hitting it each time is independent of other tries and is 0.40. (a) What is the probability that he will hit at least once? (b) How many times must he ﬁre at the target so that the probability of hitting it at least once is greater than 0.77? (c) Interpret your ﬁndings.

3.2.9.

A certain electronics company produces a particular type of vacuum tube. It has been observed that, on the average, three tubes of 100 are defective. The company packs the tubes in boxes of 400. What is the probability that a certain box of 400 tubes will contain (a) r defective tubes? (b) At least k defective tubes? (c) At most one defective tube? (d) Interpret your answers to (a), (b), and (c).

3.2.10.

Suppose that, on average, in every two pages of a book there is one typographical error, and that the number of typographical errors on a single page of the book is a Poisson r.v. with λ = 1/2. What is the probability of at least one error on a certain page of the book? Interpret your result.

138 CHAPTER 3 Additional Topics in Probability

3.2.11.

Show that the probabilities assigned by Poisson probability distribution satisfy the requirements that 0 ≤ p(x) ≤ 1 for all x and x p(x) = 1.

3.2.12.

In determining the range of an acoustic source using the triangulation method, the time at which the spherical wave front arrives at a receiving sensor must be measured accurately. Measurement errors in these times can be modeled as possessing uniform probability distribution from −0.05 to 0.05 microseconds. What is the probability that a particular arrival time measurement will be in error by less than 0.01 microsecond? What does your answer mean?

3.2.13.

The hardness of a piece of ceramic is proportional to the ﬁring time. Assume that a rating system has been devised to rate the hardness of a ceramic piece and that this measure of hardness is a random variable that is distributed uniformly between 0 and 10. If a hardness in [5,9] is desirable for kitchenware, what is the probability that a piece chosen at random will be suitable for kitchen use?

3.2.14.

A receiver receives a string of 0s and 1s transmitted from a certain source. The receiver used a majority rule. That is, if the receiver acquires ﬁve symbols, of which three or more are 1s, it decides that a 1 was transmitted. The receiver is correct only 85% of the time. What is P(W ), the probability of a wrong decision if the probabilities of receiving 0s and 1s are equally likely? What can you conclude from your result?

3.2.15.

The efﬁciency X of a certain electrical component may be assumed to be a random variable that is distributed uniformly between 0 and 100 units. What is the probability that X is: (a) Between 60 and 80 units? (b) Greater than 90 units? (c) Interpret (a) and (b).

3.2.16.

The reliability function of a system or a piece of equipment at time t is deﬁned by R(t) = P(T ≥ t) = 1 − F (t)

where T , the failure time, is a random variable with a known distribution. A certain vacuum tube has been observed to fail uniformly over the interval [t1 , t2 ]. (a) Determine the reliability of such a tube at time t, t1 ≤ t ≤ t2 . (b) If 180 ≤ t ≤ 220, what is the reliability of such a tube at 200 hours? (c) The failure or hazard rate function ρ(t) is deﬁned by ρ(t) =

− dR(t) f (t) f (t) dt . = = 1 − F (t) R(t) R(t)

Calculate the failure rate of this vacuum tube. Interpret your result. 3.2.17.

An electrical component was studied in the laboratory, and it was determined that its failure rate was approximately equal to β1 = 0.05. What is the reliability of such a component at 10 hours?

3.2 Special Distribution Functions 139

3.2.18.

Suppose that the life length of a mechanical component is normally distributed. (a) If σ = 3 and μ = 100, ﬁnd the reliability of such a system at 105 hours. (b) What should be the expected life of the component if it has reliability of 0.90 for 120 hours?

3.2.19.

A geologist deﬁnes granite as a rock containing quartz, feldspar, and small amounts of other minerals, provided that it contains not more than 75% quartz. If all the percentages are equally likely, what proportion of granite samples that the geologist collects during his lifetime will contain from 50% to 65% quartz?

3.2.20.

For a normal random variable with pdf, 2 2 1 e−(x−μ) /2σ , f (x) = √ 2πσ

show that

∞

−∞ f (x)dx

∞<x 1, (a) = (a − 1) (a − 1).

3.2.29.

(a) Find the moment-generating function for a gamma probability distribution with parameter α > 0 and β > 0. [Hint: In the integral representation of E(etX ), change the variable t to u = (1 − βt)x/β, with (1 − βt) > 0.] (b) Using the mgf of a gamma probability distribution, ﬁnd E(X) and Var(X).

3.2.30.

Let X be an exponential random variable. Show that, for numbers a > 0 and b > 0, P(X > a + b |X > a ) = P(X > b).

(This property of the exponential distribution is called the memoryless property of the distribution.) 3.2.31.

A random variable X is said to have a beta distribution with parameters α and β if and only if the density function of X is f (x) =

where B(α, β) =

1 0

⎧ α−1 β−1 ⎨ x (1−x) , ⎩

B(α,β)

α, β > 0; 0 ≤ x ≤ 1

0,

otherwise

xα−1 (1 − x)β−1 dx. (α) (β) (α+β) . α α+β and Var(X)

(a) Show that B(α, β) = (b) Show that E(X) = 3.2.32.

=

αβ . (α+β)2 (α+β+1)

The daily proportion of major automobile accidents across the United States can be treated as a random variable having a beta distribution with α = 6 and β = 4. Find the probability that, on a certain day, the percentage of major accidents is less than 80% but greater than 60%. Interpret your answer.

3.3 Joint Probability Distributions 141

3.2.33.

Suppose that network breakdowns occur randomly and independently of each other on an average rate of three per month. (a) What is the probability that there will be just one network breakdown during December? Interpret. (b) What is the probability that there will be at least four network breakdowns during December? Interpret. (c) What is the probability that there will be at most seven network breakdowns during December? Interpret.

3.2.34.

Let X be a random variable denoting the number of events occurring in the time interval (0, t]. Show that X has a gamma probability distribution with parameters n and λ.

3.2.35.

In order to etch an aluminum tray successfully, the pH of the acid solution used must be between 1 and 4. This acid solution is made by mixing a ﬁxed quantity of etching compound in powder form with a given volume of water. The actual pH of the solution obtained by this method is affected by the potency of the etching compound, by slight variations in the volume of water used, and perhaps by the pH of the water. Thus, the pH of the solution varies. Assume that the random variable that describes the random phenomenon is gamma distributed with α = 2 and β = 1. (a) What is the probability that an acid solution made by the foregoing procedure will satisfactorily etch a tray? (b) What would the answer to part (a) be if α = 1 and β = 2?

3.3 JOINT PROBABILITY DISTRIBUTIONS We have thus far conﬁned ourselves to studying one-dimensional or univariate random variables and their properties. In many practical situations, we are required to deal with several, not necessarily independent random variables. For example, we might be interested in a study involving the weights and heights (W, H) of a certain group of persons. In this situation, we need the two random variables (W, H), and it is likely that these two are related. Then it becomes important to study the joint effect of these random variables, which will lead to ﬁnding the joint probability distributions. In this section, we conﬁne our studies to two random variables and their joint distributions, which are called bivariate distributions. We consider the random variables to be either both discrete or both continuous. We now deﬁne joint distribution of two random variables. Deﬁnition 3.3.1 (a) Let X and Y be random variables. If both X and Y are discrete, then f (x, y) = P(X = x, Y = y)

is called the joint probability function (joint pmf ) of X and Y . (b) If both X and Y are continuous then f (x, y) is called the joint probability density function (joint pdf ) of X and Y if and only if b d P(a ≤ X ≤ b, c ≤ Y ≤ d) =

f (x, y)dxdy. a c

142 CHAPTER 3 Additional Topics in Probability

Example 3.3.1 A probability class contains 10 African American, 8 Hispanic American, and 15 white students. If 12 students are randomly selected from this class, and if X = number of black students, and Y = number of white students, ﬁnd the joint probability function of the bivariate random variable (X, Y ).

Solution There are a total of 33 students. The number of ways in which x African American, and y white students can be picked (which means, the remaining 12 − (x + y) students are Hispanic American) can be obtained using the multiplication principle as

10

15

8

x

x

12 − x − y

The number of ways to pick 12 students from 33 students is

.

33

. Hence, the joint probability function is

12 P(X = x, Y = y) =

10

15

x

y

8

12 − x − y 33

12 where 0 ≤ x ≤ 10, 0 ≤ y ≤ 12, and 4 ≤ x + y ≤ 12. The last constraint is needed because there are only eight Hispanic Americans, so the combined minimum number of whites and African Americans should be at least 4.

We follow the notation: x,y to denote x y . The joint distribution of two random variables has to satisfy the following conditions. Theorem 3.3.1 If X and Y are two random variables with joint probability function f (x, y), then 1. f (x, y) ≥ 0 for all x and y. 2. If X and Y are discrete, then x,y f (x, y) = 1, where the sum is over all values (x, y) that are assigned nonzero probabilities. If X and Y are continuous, then ∞ ∞ f (x, y) = 1. −∞ −∞

Given the joint probability distribution (pdf or pmf ), the probability distribution function of a component random variable can be obtained through the marginals.

3.3 Joint Probability Distributions 143

Deﬁnition 3.3.2 The marginal pmf of X denoted by fX (x) (or f (x), when there is no confusion) is deﬁned by ⎧ ∞ ⎪ ⎪ f (x, y)dy, ⎨

if X and Y are continuous,

fX (x) = −∞ ⎪ ⎪ f (x, y), ⎩

if X and Y are discrete.

all y

Similarly, the marginal pdf of Y is deﬁned by ⎧ ∞ ⎪ ⎪ f (x, y)dx, ⎨

if X and Y are continuous,

fY (y) = −∞ ⎪ ⎪ f (x, y), ⎩

if X and Y are discrete.

all x

Note that

⎧ b ⎪ ⎨ f (x)dx, X P(a ≤ X ≤ b) = a ⎪ ⎩ fX (x),

if X and Y are continuous, if X and Y are discrete,

where summation is over all values of X from a to b.

Example 3.3.2 Find the marginal probability density function of the random variables X and Y , if their joint probability function is given by Table 3.1.

Table 3.1 y x

−2

0

1

4

Sum

−1

0.2

0.1

0.0

0.2

0.5

3 5

0.1 0.1

0.2 0.0

0.1 0.0

0.0 0.0

0.4 0.1

Sum

0.4

0.3

0.1

0.2

1.0

Find the marginal densities of X and Y .

Solution By definition, the marginal pdfs of X are given by the column sums (summands over y for fixed x), and the marginal pdfs of Y are obtained by the row sums. Hence, xi −1 3 5 otherwise fX (xi ) 0.5 0.4 0.1 0

yj −2 0 1 4 otherwise fY (yi ) 0.4 0.3 0.1 0.2 0

144 CHAPTER 3 Additional Topics in Probability

Using the joint probability distribution and the marginals, we can now introduce the conditional probability distribution function. Deﬁnition 3.3.3 The conditional probability distribution of the random variable X given Y is given by f (x |y ) = f (x |Y = y ) ⎧ f (x, y) ⎪ ⎪ , ⎨ fY (y) = ⎪ P(X = x, Y = y) ⎪ ⎩ , fY (y)

if X and Y are continuous, fY (y) = 0, if X and Y are discrete.

We note that both the marginal probability densities of X and Y as well as the conditional pdf must satisfy the two important conditions of a pdf. We know that two events A and B are independent if P(A ∩ B) = P(A)P(B). It is usually more convenient to establish independence through the probability functions. Hence, we deﬁne independence for bivariate probability distribution as follows. Deﬁnition 3.3.4 Let X and Y have a joint pmf or pdf f (x, y). Then X and Y are independent if and only if f (x, y) = fX (x)fY (y),

for all x and y.

That is, for independent random variables, the joint pdf is the product of the marginals.

Example 3.3.3 Let f (x, y) =

3x,

0 ≤ y ≤ x ≤ 1,

0,

otherwise.

(a) Find P X ≤ 12 , 14 < Y < 34 . (b) Find the marginals fX (x) and fY (y). (c) Find the conditional f (x |y )(0 < y < 1). Also compute f x|Y = 12 . (d) Are X and Y independent?

Solution (a) The domain of the function f(x,y) is given in Figure 3.8. The required probability P X ≤ 12 , 14 < Y < 34 is the volume over the area of the shaded region as shown by Figure 3.9. That is,

P X ≤ 12 , 14 < Y < 34

1/2 x =

3xdydx 1/4 1/4

3.3 Joint Probability Distributions 145

1 dx 3x x − 4 1/4 1/2 3x3 3x2 = − 3 8

=

1/2

1/4

5 . = 128 y

1

f (x, y )⫽ 3x in this region x

1

■ FIGURE 3.8 Domain of f (x, y).

y y⫽x

1

The region 0 ⬍ x ⬍ 1/2 and 1/4 ⬍ y ⬍ 3/4

0.0

0.2

0.4

0.6

0.8

1.0

x 1.2

■ FIGURE 3.9 Region of integration.

(b) To find the marginals, we note that for each x, y varies from 0 to x(0 < y < x). Therefore x fX (x) =

3xdy = 3x y|x0 = 3x2 ,

0 < x < 1.

0

Similarly, for each y, x varies from y to 1. 1 fY (y) = y

1 3x2 3 3y2 3xdx = = − 2 2 2 y

3 = (1 − y2 ), 2

0 < y < 1.

146 CHAPTER 3 Additional Topics in Probability

(c) Using the definition of conditional density 2x f (x, y) 3x = = 3 , 2) fY (y) 1 − y2 (1 − y 2

f (x |y ) =

y ≤ x ≤ 1.

From this we have f x |y = 12 =

8 2x 2 = 3 x, 1 − 12

1 ≤ x ≤ 1. 2

(d) To check for independence of X and Y 1 fX (1)fY 12 = (3) 98 = 27 8 = 3 = f 1, 2 . Hence, X and Y are not independent.

Recall that in the case of a univariate random variable X, with probability function f (x), we have ⎧ ⎨ xf (x), EX = x ⎩ xf (x)dx,

|x|f (x) < ∞, for discrete r.v. if x if |x|f (x)dx < ∞, for continuous r.v.

Now we deﬁne similar concepts for bivariate distribution. Deﬁnition 3.3.5 Let f (x, y) be the joint probability function, and let g(x, y) be such that ∞ ∞ x,y |g(x, y)|f (x, y) < ∞ in the discrete case, or −∞ −∞ |g(x, y)|f (x, y)dxdy < ∞, in the continuous case. Then the expected value of g(X, Y ) is given by

Eg(X, Y ) =

⎧ ⎪ ⎪ ⎨

g(x, y)f (x, y),

if X, Y are discrete,

x,y

∞ ∞ ⎪ ⎪ g(x, y)f (x, y)dxdy, ⎩ −∞ −∞

if X, Y are continuous.

In particular

E(X, Y ) =

⎧ ⎪ ⎪ ⎨

xyf (x, y),

if X, Y are discrete,

x,y

∞ ∞ ⎪ ⎪ xyf (x, y)dxdy, ⎩ −∞ −∞

if X, Y are continuous.

The following properties of mathematical expectation are easy to verify.

PROPERTIES OF EXPECTED VALUE 1. E(aX + bY ) = aE(X ) + bE(Y ). 2. If X and Y are independent, then E(XY ) = E(X )E(Y ). However, the converse is not necessarily true.

3.3 Joint Probability Distributions 147

Example 3.3.4 Let f (x, y) = 3x, 0 ≤ y ≤ x ≤ 1. (a) Find E(4X − 3Y ), (b) Find E(XY ).

Solution

(a) E(X) = xfX (x)dx and E(Y ) = yfY (y)dy. Recall that earlier (Example 3.3.3) we have computed fX (x) = 3x2 (0 < x < 1) and fY (y) = 3 (1 − y2 ), 0 ≤ y ≤ 1. Using these results, we have 2 1 E(X) =

x3x2 dx =

3 , 4

0

1 E(Y ) =

3 3 y (1 − y2 )dy = . 2 8

0

Hence, E(4X − 3Y ) = 3 −

15 9 = . 8 8

(b) 1x xy(3x)dydx =

E(XY ) =

3 . 10

0 0

Conditional expectations are deﬁned in the same way as univariate expectations, except that the conditional density is utilized in place of the unconditional density function. Deﬁnition 3.3.6 Let X and Y be jointly distributed with pf or pdf f (x, y). Let g be a function of x. Then the conditional expectation of g(x) given, Y = y is E(g(X) |y ) = E(g(X) |Y = y ) ⎧ g(x)f (x |y ), ⎨ = all x ⎩ g(x)f (x |y )dx,

if X, Y are discrete, if X, Y are continuous.

Note that E(g(X) |y ) is a function of y. If we let Y range over all of its possible values, the conditional expectation E(g(X) |Y ) can be thought of as a function of the random variable Y . We will then be able to ﬁnd the mean and variance of E(g(X) |Y ), as given in the following result, the proof of which is left as an exercise. Theorem 3.3.2 Let X and Y be two random variables. Then (a) E(X) = E[E(X|Y )]. (b) Var(X) = E[Var(X|Y )] + Var[E(X|Y )].

148 CHAPTER 3 Additional Topics in Probability

Example 3.3.5 Let X and Y be two random variables with joint density function given by ⎧ ⎨x2 + xy , 0 ≤ x ≤ 1 and 0 ≤ y ≤ 2 3 f (x, y) = ⎩ 0, otherwise. Find the conditional expectation, E X|Y = 12 .

Solution First we will find the conditional density, f (x |y ). The marginal fY (y) =

1

x2 +

xy 1 1 dx = + y, 3 3 6

0 < y < 2.

0

Therefore, f (x|y) =

x2 + xy f (x, y) 3 , = 1 1 fY (y) y + 6 3

0 ≤ x ≤ 1.

Hence, x2 + 6x 12 2 x x + . = f x|Y = 12 = 1 1 5 6 12 + 3 Thus,

E X|Y = 12

1 xf (x |y ) dx

= 0

1 =

x

11 12 2 x x + dx = = 0.733. 5 6 15

0

3.3.1 Covariance and Correlation We will now deﬁne the covariance and correlation coefﬁcient of two random variables. Deﬁnition 3.3.7 (i) The covariance between two random variables X and Y is deﬁned by σXY = Cov(X, Y ) = E(X − μX )(Y − μY ) = E(XY ) − μX μY ,

where μX = E(X) and μY = E(Y ).

3.3 Joint Probability Distributions 149

(ii) The correlation coefﬁcient, ρ = ρ(x, y) is deﬁned by Cov(X, Y ) ρ= √ . Var(X)Var(Y )

Correlation is the measure of the linear relationship between the random variables X and Y . If Y = aX+b(a = 0), then ρ(x, y) = 1. If dependence on X and Y needs to be speciﬁed, we will use the notation, ρXY . From the deﬁnition of the covariance of X and Y , we note that if small values of X, for which (X − μX ) < 0, tend to be associated with small values of Y , for which (Y − μY ) < 0, and similarly large values of X with large values of Y, then Cov(X, Y ) ≡ E[(X − μX )(Y − μY )] can be expected to be positive. On the other hand, if small values of X tend to be associated with large values of Y and vice versa so that (X − μX ) and (Y − μY ) are of opposite signs, then Cov(X, Y ) < 0. Thus, covariance can be thought of as a signed measure of the variation of Y relative to X. If X and Y are independent, then it follows from the deﬁnition of covariance that Cov(X, Y ) = 0. The correlation coefﬁcient of X and Y , is a dimensionless quantity that measures the linear relationship between the random variables X and Y . PROPERTIES OF COVARIANCE AND CORRELATION COEFFICIENT (a) −1 ≤ ρ ≤ 1. (b) If X and Y are independent, then ρ = 0. The converse is not true. (c) If Y = aX + b, then Cov (X; Y ) =

⎧ ⎨ 1,

if a > 0,

⎩−1,

if a < 0.

Note that Cov (X , X ) = Var (X ). (d) If U = a1 X + b1 and V = a2 Y + b2 , then (i) Cov (U, V ) = a1 a2 Cov (X , Y ), and (ii) ρUV =

⎧ ⎨ ρXY ,

if a1 a2 > 0

⎩−ρ

otherwise.

XY ,

(e) Var (aX + bY ) = a2 Var (X ) + b 2 Var (Y ) + 2abCov (X , Y ).

Example 3.3.6 The joint probability density of the random variables X and Y is given by ⎧ ⎨ 1 e−y/8 , f (x, y) = 64 ⎩ 0,

0≤x≤y 1). 3.3.10.

The joint pdf of X and Y is f (x, y) =

⎧ ⎨ 1 (4x + 2y + 1), 28 ⎩

0 ≤ x ≤ 2, 0 ≤ y ≤ 2

0,

elsewhere.

Find (a) fX (x) and fY (y), and (b) f (y |x ). 3.3.11.

Find the joint mgf of the random variables (X, Y ) deﬁned in Problem 3.3.9.

3.3.12.

The joint density of a random variable (X, Y ) is given by f (x, y) =

⎧ 3 3 ⎨x y , ⎩

16

0 ≤ x ≤ 2, 0 ≤ y ≤ 2

0,

elsewhere.

(a) Find marginals of X and Y , and (b) ﬁnd f (y |x ). 3.3.13.

The joint probability function of a discrete random variable (X, Y ) is given by f (x, y) =

⎧ ⎨ ⎩

2 6xy n(n+1)(2n+1) ,

x, y = 1, 2, . . . , n,

0,

otherwise.

Find (a) f (x |y ), and (b) f (y |x ). [Hint: ni=1 i2 = (n(n + 1)(2n + 1))/6.] 3.3.14.

Consider bivariate random variables with the density n x+α−1 f (x, y) = y (1 − y)n−x+β−1 , x

for x = 0, 1, . . . , n and 0 < y ≤ 1.

3.3 Joint Probability Distributions 153

Verify that n x f (x |y ) ∝ y (1 − y)n−x x

and f (y|x) ∝ yx+α−1 (1 − y)n−x+β−1 .

3.3.15.

The joint density function of the discrete random variable (X, Y ) is given in Table 3.3.2.

Table 3.3.2 y x

1

2

3

1

1 6

1 6

1 6

2

1 6

1 12

1 12

3

1 12

1 12

0

(a) Find E(XY ). (b) Find Cov(X, Y ). (c) Find the correlation coefﬁcient ρX,Y . 3.3.16.

The joint probability function of the continuous random variable (X, Y ) is given by f (x, y) =

⎧ ⎨ 1 (4x + 2y + 1), 28 ⎩

0 ≤ x < 2, 0 ≤ y < 2,

0,

otherwise.

(a) Find E(XY ). (b) Find Cov(X, Y ). (c) Find the correlation coefﬁcient ρXY . 3.3.17.

Let X and Y be random variables and U = aX + b, V = cY + d, where a, b, c, d are constants. , if ac > 0 ρXY , Show that ρUV = −ρXY , otherwise.

3.3.18.

Let X and Y be two independent random variables, and let Y = aX + b, where a and b are constants. Show that (a) ρXY = 1 if a > 0, and (b) ρXY = −1 if a < 0.

3.3.19.

If |ρXY | = 1, then prove that P(Y = aX + b) = 1.

154 CHAPTER 3 Additional Topics in Probability

3.3.20.

Let X and Y be two random variables with joint density function f (x, y) =

⎧ ⎨8xy,

0≤x≤y≤1

⎩ 0,

otherwise.

(a) Find the conditional expectation, E X|Y = 34 . (b) Find Cov(X, Y ). 3.3.21.

Let X and Y be two random variables with joint density function f (x, y) =

⎧ ⎨e−y ,

0≤x≤y

⎩ 0,

otherwise.

(a) Find the conditional expectation, E(X|Y = y). (b) Find Cov(X, Y ). (c) Are X and Y independent? Why? 3.3.22.

Let f (x, y) =

c

(1 + x2 ) 1 − y2

,

−∞ < x < ∞,

−1 < y < 1.

Find the c that makes f (x, y) the probability density function of the random variable (X, Y ). Determine whether X and Y are independent. 3.3.23.

If the random variables X and Y are independent and have equal variances, what is the coefﬁcient of correlation between the random variables X and aX +Y , where a is a constant?

3.4 FUNCTIONS OF RANDOM VARIABLES In this section we discuss the methods of ﬁnding the probability distribution of a function of a random variable X. We are given the distribution of X, and we are required to ﬁnd the distribution of g(X). There are many physical problems that call for the derivation of the distribution of a function of a random variable. The following is one of the classical examples. The velocity V of a gas molecule (Maxwell–Boltzmann law) behaves as a gamma-distributed random variable. We would like to derive the distribution of E = mV 2 , the kinetic energy of the gas molecule. Because the value of the velocity is the outcome of a random experiment, so is the value of E. This is a problem of ﬁnding the distribution of a function of a random variable E = g(V ). We now illustrate various techniques for ﬁnding the distribution of g(X) by means of examples.

3.4.1 Method of Distribution Functions Basically the method of distribution functions is as follows. If X is a random variable with pdf fX (x) and if Y is some function of X, then we can ﬁnd the cdf FY (y) = P(Y ≤ y) directly by integrating fX (x) over the region for which {Y ≤ y}. Now, by differentiating FY (y), we get the probability density function fY (y) of Y . In general, if Y is a function of random variables X1 , . . . , Xn , say g(X1 , . . . , Xn ), then we can summarize the method of distribution function as follows.

3.4 Functions of Random Variables 155

PROCEDURE TO FIND CDF OF A FUNCTION OF R.V. USING THE METHOD OF DISTRIBUTION FUNCTIONS 1. Find the region {Y ≤ y } in the (x1 , x2 , . . . , xn ) space, that is ﬁnd the set of (x1 , x2 , . . . , xn ) for which g(x1 , . . . , xn ) ≤ y . 2. Find FY (y ) = P(Y ≤ y ) by integrating f (x1 , x2 , . . . , xn ) over the region {Y ≤ y }. 3. Find the density function fY (y ) by differentiating FY (y ).

Example 3.4.1 Let X ∼ N(0, 1). Using the cdf of X, ﬁnd the pdf of X2 .

Solution Let Y = X2 . Note that the pdf of X is 2 1 f (x) = √ e−x /2 , 2π

−∞ < x < ∞.

Then the cumulative distribution function of Y for a given y ≥ 0 is F (y) = P(Y ≤ y) = P(X2 ≤ y) √ √ = P(− y ≤ X ≤ y) √

y =

2 1 √ e−x /2 dx 2π √

− y

√

y =2 0

2 1 √ e−x /2 dx, 2π

2 (by the symmetry of e−x /2 ).

Hence, by differentiating F (y), we obtain the probability density function as 2 1 fY (y) = √ e−y/2 √ 2 y 2π

=

⎧ ⎨ √1 y−1/2 e−y/2 ,

0 y, . . . , Xn > y) = P(X1 > y)P(X2 > y) . . . P(Xn > y) (because of independence) = (1 − F (y))n . This implies FY1 (y) = 1 − (1 − F (y))n and fY1 (y) = n(1 − F (y))n−1 f (y). Consider Yn . Its cdf is given by FYn (y) = P(Yn ≤ y) = (F (y))n . This implies that fYn (y) = n(F (y))n−1 f (y).

3.4 Functions of Random Variables 159

3.4.5 Transformation Method A simple generalization of the method of distribution functions to functions of more than one variable is the transformation method. We illustrate the method for bivariate distributions. The method is similar for the multivariate case. Let the joint pdf of (X, Y ) be f (x, y). Let U = g1 (X, Y ); V = g2 (X, Y ). The mapping from (X, Y ) to (U, V ) is assumed to be one-to-one and onto. Hence, there are functions, h1 and h2 such that x = h−1 1 (u, v),

and y = h−1 2 (u, v).

Deﬁne the Jacobian of the transformation J by ∂x ∂u J = ∂y ∂u

∂x ∂v . ∂y ∂u

Then the joint pdf of U and V is given by −1 f (u, v) = f (h−1 1 (u, v), h2 (u, v)) |J| .

Example 3.4.6 Let X and Y be independent random variables with common pdf f (x) = e−x , (x > 0). Find the joint pdf of U = X/(X + Y ), V = X + Y .

Solution We have U = X/(X + Y ) = X/V . Hence, X = UV and Y = V − X = V − UV = V (1 − U). Thus, the Jacobian v u J = . −v 1 − u Then |J| = v(1 − u) + uv = v(> 0). Note that 0 ≤ u ≤ 1, 0 < v < ∞. −1 f (u, v) = f h−1 1 (u, v), h2 (u, v) |J| = e−uv e−v(1−u) v = ve−v ,

0 ≤ u ≤ 1, 0 < v < ∞.

160 CHAPTER 3 Additional Topics in Probability

Suppose we want the marginal fV (v) and fU (v), that is, 1 fV (v) =

ve−v du = ve−v ,

0 0.

162 CHAPTER 3 Additional Topics in Probability

3.4.3.

Let f (x, y) be the probability density function of the continuous random variable (X, Y ). If U = XY , show that the probability density function of U is given by ∞ fU (u) =

f

1 , v dv. v v

u

−∞

3.4.4.

The joint pdf of X and Y is f (x, y) = θe−(x+θy) ,

θ > 0, x > 0.

Find the pdf of XY . 3.4.5.

If the joint pdf of (X, Y ) is

− 21 2 x2 +y2 1 f (x, y) = e 4σ1 σ2 , 2πσ1 σ2

− ∞ < x < ∞, − ∞ < y < ∞; σ1 , σ2 > 0

ﬁnd the pdf of X2 + Y 2 . 3.4.6.

Let X1 , . . . , Xn be independent and identically distributed random variables with pdf f (x) = (1/θ)e−x/θ , x > 0, θ > 0. Find the pdf of ni=1 Xi .

3.4.7.

Let f (x, y) be the pdf of the continuous random variable (X, Y ). If U = X + Y , then show that the probability density function of U is given by ∞ fU (u) =

f (u − v, v)dv. −∞

3.4.8.

Let X be uniformly distributed over (−2, 2) and Y = X2 . Find the Cov(X, Y ). Are X and Y independent?

3.4.9.

Let X ∼ N(μ, σ 2 ). Show that is N(0, 1). (a) Z = (X−μ) σ (b) U =

(X−μ)2 σ2

is χ2 (1).

3.4.10.

Let X ∼ N(μ, σ 2 ). Find the pdf of Y = eX.

3.4.11.

The probability density of the velocity, V , of a gas molecule, according to the Maxwell– Boltzmann law, is given by f (v, β) =

⎧ 2 ⎨cv2 e−βv , ⎩

0,

v > 0, elsewhere

where c is an appropriate constant and β depends on the mass of the molecule and the absolute temperature. Find the density function of the kinetic energy E, which is given by E = g(V ) = 12 mV 2 .

3.5 Limit Theorems 163

3.4.12.

Let X and Y be two independent random variables, each normally distributed, with parameters (μ1 , σ12 ), and (μ2 , σ22 ), respectively. Show that the probability density function of U = X/Y is given by fU (u) =

3.4.13.

σ1 σ2 2 , π σ1 + σ22 u2

−∞ < u < ∞.

Let f (x, y) =

1 −1/2σ 2 x2 +y2 e , 2πσ 2

−∞ < x, y < ∞

be the joint pdf of (X, Y ). Let U=

X2 + Y 2

V = tan−1

and

Y , X

0 ≤ V ≤ 2π.

Find the joint pdf of (U, V ). 3.4.14.

Let the joint pdf of (X, Y ) be given by f (x, y) =

⎧ ⎨β−2 e−{(x+y)/β} , ⎩

0,

x, y > 0, β > 0, elsewhere.

X−Y and V = Y . Find the joint pdf of (U, V ). 2 Let X and Y be independent and identically distributed random variables with pdf

Let U = 3.4.15.

f (x) =

⎧ ⎨ 1 e−x/2 , 2

⎩

0,

x ≥ 0, otherwise.

Find the distribution of (X − Y )/2. 3.4.16.

If X and Y are independent and chi-square distributed random variables with n1 and n2 degrees of freedom, respectively. Obtain the joint distribution of (U, V ), where U = X + Y and V = X/Y .

3.5 LIMIT THEOREMS Limit theorems play a very important role in the study of probability theory and in its applications. In Chapter 2, we saw that the frequency interpretation of probability depends on the long-run proportion of times the outcome (event) would occur in repeated experiments. Also, in Section 3.2, we learned that some binomial probabilities can be computed using either the Poisson probability distribution or the normal probability distribution using the limiting arguments. Many random variables that we encounter in nature have distributions close to the normal probability distribution. These modeling

164 CHAPTER 3 Additional Topics in Probability

simpliﬁcations are possible because of various limit theorems. In this section, we discuss the law of large numbers and the Central Limit Theorem. First we give Chebyshev’s theorem, which is a useful result for proving limit theorems. It gives a lower bound for the area under a curve between two points that are on opposite sides of the mean and are equidistant from the mean. The strength of this result lies in the fact that we need not know the distribution of the underlying population, other than its mean and variance. This result was developed by the Russian mathematician Pafnuty Chebyshev (1821–1894).

CHEBYSHEV’S THEOREM Theorem 3.5.1 Let the random variable X have a mean μ and standard deviation σ. Then for K > 0, a constant, P(|X − μ| < Kσ) ≥ 1 −

1 . K2

Proof. We will work with the continuous case. By deﬁnition of the variance of X, σ 2 = E(X − μ)2 =

∞

(x − μ)2 f (x)dx

−∞ μ−Kσ

(x − μ)2 f (x)dx +

= −∞

μ+Kσ

μ−Kσ

μ−Kσ

(x − μ)2 f (x)dx +

≥

(x − μ)2 f (x)dx +

−∞

∞

∞

(x − μ)2 f (x)dx

μ+Kσ

(x − μ)2 f (x)dx.

μ+Kσ

Note that (x − μ)2 ≥ K2 σ 2 for x ≤ μ − Kσ or x ≥ μ + Kσ. The equation above can be rewritten as ⎡

⎢ σ 2 ≥ K2 σ 2 ⎣

⎤

∞

μ−Kσ

⎥ f (x)dx⎦

f (x)dx + −∞

μ+Kσ

= K2 σ 2 [P{X ≤ μ − Kσ} + P{X ≥ μ + Kσ}] = K2 σ 2 P{|X − μ| ≥ Kσ}.

This implies that P{|X − μ| ≥ Kσ} ≤

1 K2

3.5 Limit Theorems 165

or P (|X − μ| < Kσ) ≥ 1 −

1 . K2

We can also write Chebyshev’s theorem as P{|X − μ| ≥ ε} ≤

# $ E (X − μ)2 ε2

=

Var(X) , ε2

for some ε > 0.

Equivalently, P{|X − μ| ≥ Kσ} ≤

1 . K2

In other words, Chebyshev’s inequality states that the probability that a random variable X differs from its mean by at least K standard deviations is less than or equal to 1/K2 (K ≥ 2). In statistics, if we do not have any idea of the population distribution, Chebyshev’s theorem is used in the following manner. For any data set (regardless of the shape of the distribution), at least (1−(1/k2 ))100% of observations will lie within k(≥ 1) standard deviations of the mean. For example, at least (1−(1/22 ))100% = 75% of the data will fall in the interval (x−2s, x+2s) and at least 88.9% of the observations will lie within three standard deviations of the mean. If the population distribution is bell shaped, we have a better result than Chebyshev’s theorem, namely, the empirical rule that states the following: (i) approximately 68% of the observations lie within one standard deviation of the mean; (ii) approximately 95% of the observations lie within two standard deviations of the mean; and (iii) approximately 99.7% of the observations lie within three standard deviations of the mean.

Example 3.5.1 A random variable X has mean 24 and variance 9. Obtain a bound on the probability that the random variable X assumes values between 16.5 to 31.5.

Solution From Chebyshev’s theorem. P {μ − Kσ < X < μ + Kσ} ≥ 1 − Equating μ + Kσ to 31.5 and μ − Kσ to 16.5 with μ = 24 and σ = Hence, P {16.5 < X < 31.5} ≥ 1 −

1 . K2 √ 9 = 3, we obtain K = 2.5.

1 = 0.84. (2.5)2

166 CHAPTER 3 Additional Topics in Probability

Example 3.5.2 Let X be a random variable that represents the systolic blood pressure of the population of 18- to 74-year-old men in the United States. Suppose that X has mean 129 mm Hg and standard deviation 19.8 mm Hg. (a) Obtain a bound on the probability that the systolic blood pressure of this population will assume values between 89.4 and 168.6 mm Hg. (b) In addition, assume that the distribution of X is approximately normal. Using the normal table, ﬁnd P(89.4 ≤ X ≤ 168.6). Compare this with the empirical rule.

Solution (a) Because we are given only the mean and standard deviation, and no distribution is specified, we use Chebyshev’s theorem. We have P {μ − Kσ < X < μ + Kσ} ≥ 1 −

1 . K2

Equating μ + Kσ to 168.6 and μ − Kσ to 89.4 with μ = 129 and σ = 19.8, we obtain K = 2. Hence, P {89.4 ≤ X ≤ 168.6} ≥ 1 −

1 = 0.75. (2)2

(b) Because X is normally distributed with mean 129 and standard deviation 19.8, using the z-score, we get

168.6 − 129 89.4 − 129 ≤Z≤ P(89.4 ≤ X ≤ 168.6) = P 19.8 19.8 = P(−2 ≤ Z ≤ 2) = 0.9544. Hence, approximately 95.44% of this population will have systolic blood pressure values between 89.4 and 168.6 mm Hg. This compares well with the 95% value from the empirical rule.

We could use Chebyshev’s inequality to prove the following result, which is called the weak law of large numbers. The law of large numbers states that if the sample size n is large, the sample mean rarely deviates from the mean of the distribution of X, which in statistics is called the population mean.

LAW OF LARGE NUMBERS Theorem 3.5.2 Let X1 , . . . , Xn be a set of pairwise independent random variables with E(Xi ) = μ, and var(Xi ) = σ 2 . Then for any c > 0, 0 / σ2 P μ−c ≤X ≤μ+c ≥1− 2 nc

3.5 Limit Theorems 167

and as n → ∞, the probability approaches 1. Equivalently,

Sn − μ < ε → 1 P n as n → ∞.

Proof. Because X1 , . . . , Xn are iid random variables, we know that Var(Sn ) = nσ 2 , and Var(Sn /n) = σ 2 /n. Also, E(Sn /n) = μ. By Chebyshev’s theorem, for any ε > 0,

Sn σ2 − μ ≥ ε ≤ 2 . P n nε

Thus, for any ﬁxed ε,

as n → ∞. Equivalently,

Sn − μ ≥ ε → 0 P n

Sn − μ < ε → 1 P n

as n → ∞. Thus, without any knowledge of the probability distribution function of Sn , the (weak) law of large numbers states that the sample mean, X = Sn /n, will differ from the population mean by less than an arbitrary constant, ε > 0, with probability that tends to 1 as n tends to ∞. Because of this, the law of large numbers is also called the “law of averages.” This result basically states that we can start with a random experiment whose outcome cannot be predicted with certainty, and by taking averages, we can obtain an experiment in which the outcome can be predicted with a high degree of accuracy. The law of large numbers in its simplest form for the Bernoulli random variables was introduced by Jacob Bernoulli toward the end of the 16th century. This result in generality was ﬁrst proved by the Russian mathematician A. Khintchine in 1929. This result is widely used in its applications to insurance, statistics, and the study of heredity.

Example 3.5.3 Let X1 , . . . , Xn be iid Bernoulli random variables with parameter p. Verify the law of large numbers.

Solution For Bernoulli random variables we know that EXi = p, and Var(Xi ) = p(1 − p). Thus, by Chebyshev’s theorem, 1 , Sn 0 / σ2 − p ≤ c ≥ 1 − 2 P p − c ≤ X ≤ p + c = P n nc p(1 − p) =1− → 1, as n → ∞. nc2 This verifies the weak law of large numbers.

168 CHAPTER 3 Additional Topics in Probability

Example 3.5.4 Consider n rolls of a balanced die. Let Xi be the outcome of the ith roll, and let Sn = for any ε > 0,

Sn 7 − ≥ ε → 0 P n 2 as n → ∞.

n

i=1Xi . Show that,

Solution Because the die is balanced, EXi = 7/2. By the law of large numbers, for any ε > 0,

Sn 7 − ≥ε →0 P n 2 as n → ∞, or equivalently,

Sn 7 P − < ε → 1 n 2

as n → ∞.

One of the most important results in probability theory is the Central Limit Theorem. This basically states that the z-transform of the sample mean is asymptotically standard normal. The amazing thing about the Central Limit Theorem is that no matter what the shape of the original distribution is, the (sampling) distribution of the mean approaches a normal probability distribution. We state one version of the Central Limit Theorem. In a restricted case, the proof uses the idea that the momentgenerating functions of Zn converge to the moment-generating function of the standard normal random variable. The general proof is a little bit more involved. Because the proof of the Central Limit Theorem is available in most probability books, we will not give the proof here. CENTRAL LIMIT THEOREM (CLT) Theorem 3.5.3 If X1 , . . . , Xn is a random sample from an inﬁnite population with mean μ, variance σ 2 , √ and the moment-generating function MX (t), then the limiting distribution of Zn = (X − μ)/(σ/ n) as n → ∞ is the standard normal probability distribution. That is, 1 lim P(Zn ≤ z) = √ n→∞ 2π

If Sn =

n

i=1 Xi ,

z

2 e−t /2 dt.

−∞

then we can rewrite Zn as

n X−μ X−μ Zn = √ , √ = nσ/ n σ/ n =

Sn − nμ √ , σ n

since nX =

n i=1

Xi .

3.5 Limit Theorems 169 √ Then the CLT states that Zn = (Sn − nμ) /σ n is approximately N(0, 1) for large n. The Central Limit Theorem basically says that when we repeat an experiment a large number of times, the average (almost always) follows a Gaussian distribution.

Example 3.5.5 X1 , X2 , . . . are iid random variables such that Xi =

1,

with probability p,

0,

with probability 1 − p.

√

Show that Zn = (Sn − np)/ npq is approximately normal for large n, where Sn =

n

i=1 Xi , and q = 1 − p.

Solution We know that E(X) = p; E(X2 ) = p; Var(X) = p − p2 = pq. √ Hence, by the CLT, the limiting distribution of Zn = (Sn − np)/ npq as n → ∞ is the standard normal probability distribution.

Example 3.5.6 A soft-drink vending machine is set so that the amount of drink dispensed is a random variable with a mean of 8 ounces and a standard deviation of 0.4 ounces. What is the approximate probability that the average of 36 randomly chosen ﬁlls exceed 8.1 ounces?

Solution

√ From the CLT, ((X − 8)/(0.4/ 36)) ∼ N(0, 1). Hence, from the normal table, ⎧ ⎫ ⎨ 0 / 8.1 − 8.0 ⎬ P X > 8.1 = P Z > 0.4 ⎩ ⎭ √ 36

= p {Z > 1.5} = 0.0668.

Example 3.5.7 Numbers in decimal form are often approximated by the closest integers. Suppose n numbers X1 , . . . , Xn are approximated by their closest integers J1 , J2 , . . . , Jn . Let Ui = Xi − Ji . Assume that Ui are uniform on (−0.5, 0.5) and that Ui s are independent. n Ui (a) Show that √i=1 ∼ N(0, 1) as n → ∞. n/12 n 1 , Ui 5 −5 i=1 ≤ √ ≤ √ . (b) Find P √ 300/12 300/12 300/12

170 CHAPTER 3 Additional Topics in Probability / 0 (c) Find the value of a such that P −a ≤ Ui ≤ a = 0.95 2 3 (d) For n = 106 , ﬁnd a such that P −a ≤ 106 i=1 Ui ≤ a = 0.99.

Solution (a) Because Ui s are uniform in (−0.5, 0, 5), Kn = ni=1 Ji . Then

Ui = 0, Var(Ui ) = 1/12. Let, Sn =

n

i=1 Xi , and

2 3 (Xi − Ji ) ≤ a P{|Sn − Kn | ≤ a} = P −a ≤ 2 3 = P −a ≤ Ui ≤ a . n Ui − 0 i=1 ∼ N(0, 1) as n → ∞. √ n/12 (b) For n = 300; a = 5. Using the normal table, n , 1 Ui −5 5 P √ ≤ √ i=1 ≤ √ = 0.68. 300/12 300/12 300/12 By the CLT,

(c) Now, 2 3 0.95 = P −a ≤ Ui ≤ a , =P From the normal table, we get √

1 −a a ≤Z≤ √ . √ 300/12 300/12

a = 1.96. This implies, a = 9.8. 300/12

(d) We have 0.99 = P

⎧ ⎨

6

−a ≤

⎩

10 i=1

−a

Ui ≤ a

⎫ ⎬ ⎭ a

4

≤Z≤ . 106 /12 106 /12 Now, using the normal table, we have a/ 106 /12 = 2.58. Hence, a = 745. =P

Example 3.5.8 A casino has a coin, suspected to be biased. Estimate p (probability of heads) such that they can be conﬁdent that their estimate (say, p) ˆ is within 0.01 of p (unknown). What is the minimum number of times we need to toss this coin?

3.5 Limit Theorems 171

Solution Set Xj =

Suppose we decided to use pˆ =

Xi n , that is,

1,

if H as j’th toss,

0,

if T as j’th toss.

#Heads . n

We want P{|X − p| < 0.01} = 0.99. √ Because Y = ni=1 Xi ∼ Bin(n, p), we have EY = np, Var(Y ) = npq. By the CLT, (X − p)/ pq/n ∼ N(0, 1). Now, X−p 0.01 −0.01 < √ < √ 0.99 = P √ pq/n pq/n pq/n 1 , 0.01 −0.01 2), where X100 = (1/100) 3.5.12.

100

i=1 Xi .

Let X1 , . . . , Xn be a sequence of independent Poisson-distributed random variables, with √ parameter λ. Let Sn = ni=1 Xi . Show that Zn = ((Sn − nλ)/ nλ) ∼ N(0, 1).

3.6 Chapter Summary 173

3.5.13.

Let X1 , . . . , Xn be a sequence of independent uniformly-distributed over [0,1) random √ variables. Let Sn = ni=1 Xi . Show that Zn = ((Sn − nλ)/ nλ) ∼ N(0, 1).

3.5.14.

Suppose that 2500 customers subscribe to a telephone exchange. There are 80 trunk lines available. Any one customer has the probability of 0.03 of needing a trunk line on a given call. Consider the situation as 2500 trials with probability of “success” p = 0.03. What is the approximate probability that the 2500 customers will “tie up” the 80 trunk lines at any given time?

3.5.15.

Suppose a group of people have an average IQ of 122 with standard deviation 2. Obtain a bound on the probability that IQ values of this group will be between 104 and 120.

3.5.16.

Let X be a random variable that represents the diastolic blood pressure (DBP) of the population of 18- to 74-year-old men in the United States who are not taking any corrective medication. Suppose that X has mean 80.7 mm Hg and standard deviation 9.2. (a) Obtain a bound on the probability that the DBP of this population will assumes values between 53.1 and 108.3 mm Hg. (b) In addition, assume that the distribution of X is approximately normal. Using the normal table, ﬁnd P(53.1 ≤ X ≤ 108.3). Compare this with the empirical rule.

3.5.17.

Color blindness appears in 2% of the people in a certain population. How large must a random sample be in order to be 99% certain that a color-blind person is included in the sample?

3.5.18.

A shirt manufacturer knows that, on the average, 2% of his product will not meet quality speciﬁcations. Find the greatest number of shirts constituting a lot that will have, with probability 0.95, fewer than ﬁve defectives.

3.5.19.

A random sample of size 100 is taken from a population with mean 1 and variance 0.04. Find the probability that the sample mean is between 0.99 and 1.

3.5.20.

The lifetime X (in hours) of a certain electrical component has the pdf f (x) = (1/3)e−(1/3)x , x > 0. If a random sample of 36 is taken from these components, ﬁnd P(X < 2).

3.5.21.

A drug manufacturer receives a shipment of 10,000 calibrated “eyedroppers” for administering the Sabin poliovirus vaccine. If the calibration mark is missing on 500 droppers, which are scattered randomly throughout the shipment, what is the probability that, at most, two defective droppers will be detected in a random sample of 125?

3.6 CHAPTER SUMMARY In this chapter we looked at some special distribution functions that arise in practice. It should be noted that we discussed only a few of the important probability distributions. There many other discrete and continuous distributions that will be useful and appropriate in particular applications. Some of them are given in Appendix A3. A larger list of probability distributions can be found at http://www.causascientia.org/math_stat/Dists/Compendium.pdf, among many other

174 CHAPTER 3 Additional Topics in Probability

places. For more than one random variable, we learned the joint distributions. We also saw how to ﬁnd the density and cumulative distribution for the functions of a random variable. Limit theorems are a crucial part of probability theory. We have introduced the Chebyshev’s inequality, the law of large numbers, and the Central Limit Theorem for the random variables. We now list some of the key deﬁnitions introduced in this chapter: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Bernoulli probability distribution Binomial experiment Poisson probability distribution Probability distribution Normal (or Gaussian) probability distribution Standard normal random variable Gamma probability distribution Exponential probability distribution Chi-square (χ2 ) distribution Joint probability density function Bivariate probability distributions Marginal pdf Conditional probability distribution Independence of two r.v.s Expected value of a function of bivariate r.v.s Conditional expectation Covariance Correlation coefﬁcient

In this chapter, we have also learned the following important concepts and procedures: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Mean, variance, and moment-generating function (mgf ) of a binomial random variable Mean, variance, and mgf of a Poisson random variable Poisson approximation to the binomial probability distribution Mean, variance, and mgf of a uniform random variable Mean, variance, and mgf of a normal random variable Mean, variance, and mgf of a gamma random variable Mean, variance, and mgf of an exponential random variable Mean, variance, and mgf of a chi-square random variable Properties of expected value Properties of the covariance and correlation coefﬁcient Procedure to ﬁnd the cdf of a function of r.v. using the method of distribution functions The pdf of Y = g(X), where g is differentiable and monotone increasing or decreasing The pdf of Y = g(X), using the probability integral transformation The transformation method to ﬁnd the pdf of Y = g(X1 , . . . , Xn ) Chebyshev’s theorem Law of large numbers Central Limit Theorem (CLT)

3.7 Computer Examples (Optional) 175

3.7 COMPUTER EXAMPLES (OPTIONAL) 3.7.1 Minitab Examples Minitab contains subroutines that can do pdf and cdf computations. For example, for binomial random variables, the pdf and cdf can be respectively computed using the following comments.

MTB > pdf k; SUBC > binomial n p.

and

MTB > cdf; SUBC > binomial n p.

Practice: Try the following and see what you get.

MTB > pdf 3; SUBC > binomial 5 0.40.

will give K 3.00 and

MTB > cdf; SUBC > binomial 5 0.40.

will give BINOMIAL WITH N = 5 P = 0.400000 K P(X LESS OR = K) 0 0.0778 1 0.3370 2 0.6826 3 0.9130 4 0.9898 5 1.0000

P(X = K) 0.2304

176 CHAPTER 3 Additional Topics in Probability

Similarly, if we want to calculate the cdf for a normal probability distribution with mean k and standard deviation s, use the following comments.

MTB > cdf x; SUBC > normal k s.

will give P(X ≤ x). Practice: Try the following.

MTB > cdf 4.20; SUBC > normal 4 2.

We can use the invcdf command to ﬁnd the inverse cdf. For a given probability p, P(X ≤ x) = F (x) = p, we can ﬁnd x for a given distribution. For example, for a normal probability distribution with mean k and standard deviation s, use the following.

MTB > invcdf p; SUBC > normal k s.

We can also use the pull-down menus to compute the probabilities. The following example illustrates this for a binomial probability distribution.

Example 3.7.1 A manufacturer of a color printer claims that only 5% of their printers require repairs within the ﬁrst year. If out of a random sample of 18 of their printers, four required repairs within the ﬁrst year, does this tend to refute or support the manufacturer’s claim? Use Minitab.

Solution Type the numbers 1 through 18 in C1. Then

Calc > Probability Distributions > Binomial. . . > choose Cumulative probability > in Number of trials, enter 18 and in Probability of success, enter 0.05 > in Input column: type C1 > Click OK We will get the following output.

3.7 Computer Examples (Optional) 177

Cumulative Distribution Function Binomial with n=18 and p=0.0500000 x P(X Compute > type in the Target Variable: y > Use the scroll bar beside the Functions box to find CDF.BINOM(q, n, p) > Highlight it and use the up button to load it into the Numeric Expression: box. Set q to 3 (success, the x-value), n to 18 (total trials) and p to 0.05 (probability of success) > OK In the second column, we will get the y-values as 0.99. Hence, P(X ≤ 3) = 0.99.

We can use this procedure for many other distributions.

178 CHAPTER 3 Additional Topics in Probability

3.7.3 SAS Examples Sometimes, we can use computer calculations to ﬁnd out the exact probability of a certain event in lieu of approximations. For example, when n is large in a binomial experiment, we can use normal approximation to calculate the probabilities. The following example shows how to calculate binomial probabilities using SAS codes.

Example 3.7.3 Suppose that a certain drug to treat a disease has a success rate of p = 0.65. This drug is given to n = 500 patients with the disease. (a) What is the probability that 335 or fewer show improvement? (b) What is the probability that more than 320 show improvement? (c) What is the probability that exactly 300 show improvement? (d) What is the probability that the number of improvements lies in the interval (300,350)?

Solution Let X = number of patients showing improvement. Then X is a binomial random variable with parameters n = 500 and p = 0.65. (a) First three lines in the following code are comment lines. In general, it is always helpful to include the comment lines to explain about the program. /*This program can be used to compute probability*/ /* that a Binomial variable with parameters p*/ /*and n is less than or equal to x*/ data binomial; p=0.65; n=500; x=335; y=probbnml(p,n,x); cards; proc print; run; The following is the SAS output from running the foregoing program. Obs 1

p 0.65

n 500

x 335

y 0.83753

Here y = 0.83753 is the P (X ≤ 335). (b) To calculate P(X > 320), we can use the following. data binomial; p=0.65; n=500;

3.7 Computer Examples (Optional) 179

x=320; y=probbnml(p,n,x); z=1–y; cards; proc print; run; The following is the SAS output from running the foregoing program, where the value of z is the probability we are looking for. Obs 1

p 0.65

n 500

x 320

y 0.33516

z 0.66484

Hence, P(X > 320) = 0.66484. (c) To find P(X = 300), we can use the following. data binomial; p=0.65; n= 500; x1=300; y1=probbnml(p,n,x1); x2=299; y2=probbnml(p,n,x2); z=y1−y2; cards; proc print; run; The following is the SAS output from running the foregoing program, where the value of z is the probability we are looking for. Obs p n x1 y1 1 0.65 500 300 0.011327

x2 y2 z 299 .008864418 .002462253

(d) To find P(300 < X < 350), use the following. data binomial; p=0.65; n=500; x1=300; y1=probbnml(p,n,x1); x2=349; y2=probbnml(p,n,x2);

180 CHAPTER 3 Additional Topics in Probability

z=y2−y1; cards; proc print; run; We will get the following output. Obs p n x1 1 0.65 500 300

y1 x2 y2 z 0.011327 349 0.98982 0.97849

Hence, P(300 < X < 350) = 0.97849.

Similar procedures could be used to calculate probabilities for other distributions. In order to test for normality of a given data set using a normal probability plot, we can use PROC UNIVARIATE (see Chapter 1 for explanation) in the following manner. Normal plot is called qqplot in SAS.

proc univariate data=K noprint; /*Specify the name of data set as K*/ qqplot standard; run; quit;

Note that this avoids printing of all the standard output due to the univariate command, and we get only the QQ plot. If we need a straight line in the plot, we can modify the commands as follows.

proc univariate data=K noprint; /*Specify the name of data set as B*/ qqplot standard/ normal (mu=m, sigma=s); run; quit;

PROJECTS FOR CHAPTER 3 3A. Mixture Distribution In statistical modeling, if the data are contaminated by outliers or if the samples are drawn from a population formed by a mixture of two populations, one could use mixture distributions. Mixture distributions are used frequently in medical applications, such as micro array analysis. Suppose a random variable X has pdf f1 (x) with probability p1 and pdf f2 (x) with probability p2 , where p1 + p2 = 1. Then we say that the r.v. X has a mixture distribution. This can be thought of as observing

Projects for Chapter 3 181

a Bernoulli random variable Z that is equal to 1 with probability p1 and 2 with probability p2 . Thus, X=

X1 ∼ f1 (x), X2 ∼ f2 (x),

if Y = 1, if Y = 2.

(a) Show that the pdf of X is given by f (x) = p1 f1 (x) + p2 f2 (x). (b) If (μ1 , σ12 ) and (μ2 , σ22 ) are means and variances of f1 (x) and f2 (x), respectively, show that μ = E(X) = p1 μ1 + p2 μ2 ,

and σ 2 = Var(X) = p1 σ12 + p2 σ12 + p1 μ21 + p2 μ22 − (p1 μ1 + p2 μ2 )2 .

3B. Generating Samples from Exponential and Poisson Probability Distribution (a) Generate a sample from 1θ e−x/θ (θ is chosen). Let Y1 , Y2 , . . . , Yn be a sample from a U(0, 1) distribution. Let F (x) = 1−e−x/θ (cdf of exponential). Then Y = F (x) is uniform. yj = 1−e−x/θ implies xj = − θ ln(1 − yi ) = − θ ln ui , where u1 , u2 , . . . ., un is a sample from U(0, 1). Then X1 , . . . , Xn is a sample from an exponential distribution with parameter θ. (b) Suppose we want to generate a sample from a Poisson probability distribution with parameter λ. X1 , . . . , Xn is a sample from an exponential distribution with parameter 1/λ till ni=1 Xi just exceeds 1. Then yn (n − 1) is a sample values form a Poisson probability distribution with parameter λ.

EXERCISE 3B Let u1 , u2 , . . . , un be a sample from U(0, 1). Show that n 2 , (i) X = −2 ln(ui ) ∼ χ2n (ii) X = −β

i=1 α

ln(ui ) ∼ gamma(α, β), and

i=1 α

(iii) X =

i=1 α+β

ln(ui )

∼ Beta(α, β).

ln(ui )

i=1

3C. Coupon Collector’s Problem Suppose there are n distinct colors of coupons. Each color of coupon is equally likely to occur. When a complete set of coupons with each color represented is assembled, you win a prize. Let X = # coupons for a complete set. Find (a) Distribution of X, (b) E(X), and (c) Var(X).

182 CHAPTER 3 Additional Topics in Probability

3D. Recursive Calculation of Binomial and Poisson Probabilities A simple way to calculate binomial probabilities is as follows: For a given n and p, evaluate b(0, n, p) and then apply the recursive relationship b(x + 1, n, p) = b(x, n, p)

p(n − x) (1 − p)(x + 1)

to obtain other binomial probabilities. (a) Derive this recursion formula. (b) For n = 15, p = 0.4, using the recursive formula, compute all other probabilities starting from x = 0. The following recursive formulas are very useful in calculating successive Poisson probabilities: f (x − 1, λ) = f (x, λ)

x λ

and f (x + 1, λ) =

λ e−λ λx+1 = f (x, λ) . (x + 1)! x+1

For example, if λ = 2.5, we know that f (0, 2.5) = e−2.5 = 0.08208. Using this, calculate (c) f (1, 2.5) and f (2, 2.5).

Chapter

4

Sampling Distributions Objective: In this chapter we study the probability distributions of various sample statistics such as the sample mean and the sample variance and illustrate their usefulness. 4.1 Introduction 184 4.2 Sampling Distributions Associated with Normal Populations 4.3 Order Statistics 207 4.4 Large Sample Approximations 212 4.5 Chapter Summary 218 4.6 Computer Examples 219 Projects for Chapter 4 221

191

Abraham de Moivre (Source: http://en.wikipedia.org/wiki/File:Abraham_de_Moivre.jpg)

Abraham de Moivre (1667–1754) was a French mathematician known for his work on the normal distribution and probability theory. He is famous for de Moivre’s formula, which links complex

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

183

184 CHAPTER 4 Sampling Distributions

numbers and trigonometry. He ﬂed France and went to England to escape the persecution of Protestants. In England he wrote a book on probability theory, titled The Doctrine of Chances. This book was very popular among gamblers. The normal distribution was ﬁrst introduced by de Moivre in an article in 1733 in the context of approximating certain binomial distributions for large n, and is now called the theorem of de Moivre–Laplace.

4.1 INTRODUCTION Sampling distributions play a very important role in statistical analysis and decision making. We begin with studying the distribution of a statistic computed from a random sample. Based on the probabilistic foundation of Chapters 2 and 3, the present study marks the beginning of our learning of statistics beyond the descriptive phase. Because a sample is a set of random variables X1 , . . . , Xn , it follows that a sample statistic that is a function of the sample is also random. We call the probability distribution of a sample statistic its sampling distribution. Sampling distributions provide the link between probability theory and statistical inference. The ability to determine the distribution of a statistic is a critical part in the construction and evaluation of statistical procedures. It is important to observe that there is a difference between the distribution of population from which the sample was taken and the distribution of the sample statistic. In general, a population has a distribution called a population distribution, which is usually unknown, whereas a statistic has a sampling distribution, which is usually different from the population distribution. The sampling distribution of a statistic provides a theoretical model of the relative frequency histogram for the likely values of the statistic that one would observe through repeated sampling. Even though some of the terms in this section have already been deﬁned in Chapter 1, we now present these deﬁnitions in terms of random variables. These abstractions are introduced to develop scientiﬁcally based methods of analyzing the data, and one should always keep in mind the underlying population. Deﬁnition 4.1.1 A sample is a set of observable random variables X1 , . . . , Xn . The number n is called the sample size. In most of the inferential procedures that we study in this book, we are dealing with random samples. We call the random variables X1 , . . . , Xn identically distributed if every Xi has the same probability distribution. Deﬁnition 4.1.2 A random sample of size n from a population is a set of n independent and identically distributed (iid) observable random variables X1 , . . . , Xn . Note that in a sample (not a random sample), Xi s need not be independent or identically distributed. For the results of this book to be applicable, it is important to ensure that the selection of a sample is at least approximately random. The signiﬁcance of random sampling is that the probability distribution of a statistic can be easily derived. Random sampling helps us to control systematic basis. For a ﬁnite population, one can serially number the elements of the population and then select a random sample with the help of a table of random digits. One of the simplest ways to select a random sample of ﬁnite size is to use a table of random numbers. When the population size is very large, such a method can become very taxing and sometimes practically impossible. However, there are excellent computer

4.1 Introduction 185

programs for generating random samples from large populations, and these programs can be used. Now we deﬁne a statistic. Deﬁnition 4.1.3 A function T of observable random variables X1 , . . . , Xn that does not depend on any unknown parameters is called a statistic. The sample mean X = (1/n) ni=1 Xi is a function of X1 , . . . , Xn . The sample median and sample variance S 2 are also examples of statistics. It is important to observe that even with random sampling, there is sampling variability or error. That is, if we select different samples from the same population, a statistic will take different values in different samples. Thus, a sample statistic is a random variable, and hence it has a probability distribution. In order for us to study the behavior of the phenomenon a sample statistic represents, we must identify its probability distribution. Deﬁnition 4.1.4 The probability distribution of a sample statistic is called the sampling distribution. We can illustrate these deﬁnitions with the following example with a ﬁnite population and a ﬁnite sample size. In this case, we take all possible samples of size n from a population of size N.

Example 4.1.1 Let the population consist of the numbers {1, 2, 3, 4, 5}. Consider all possible samples consisting of three numbers randomly chosen without replacement from this population. Obtain the distribution of the sample mean.

Solution Disregarding the order, it is clear that there are

5 = 10 equally likely possible samples of size 3. They are 3

(1,2,3), (1,2,4), (1,2,5), (1,3,4), (1,3,5), (1,4,5), (2,3,4), (2,3,5), (2,4,5), and (3,4,5). Calculating the mean, X, for each of the samples, we will get the sampling distribution of X as x

2 1

7 3

8 3

3 1

10 3

11 3

4 1

1 1 2 2 2 1 1 10 10 10 10 10 10 10 For example, in the table, P X = 8/3 = 2/10 because the two samples (1,2,5) and (1,3,4) both give an x = 8/3, which is an estimate of the population mean, μ. p (x)

In general, sampling distributions are theoretical distributions that consist of possibly an infinite number of sample statistics taken from an infinite number of randomly selected samples of a fixed sample size. For example, if a sample of size n = 30 were taken from a large population an infinite number of times, the combined means taken from all the samples would make up the sampling distribution of the mean. Every sample statistic has a sampling distribution. The next result states that if one selects a random sample from a population with mean μ and variance σ 2 , then regardless of the form of the population distribution, one can obtain the mean and standard deviation of the statistic X in terms of the mean and standard deviation of the population. This is explained in the following result.

186 CHAPTER 4 Sampling Distributions

Theorem 4.1.1 Let X1 , . . . , Xn be a random sample of size n from a population with mean μ and variance σ 2 . Then E(X) = μ and Var(X) = σ 2 /n. Proof. The mean and variance of X is given by, E X =E

1 Xi n n

i=1

=

and

1 n

n i=1

μ=

1 E(Xi ) n n

=

i=1

1 nμ = μ. n

n 1 Var X = Var Xi n i=1

n 1 Var(Xi ) (because Xi s are independent and = 2 n i=1

Var(aXi ) = a2 Var (Xi )) σ2 1 . = 2 nσ 2 = n n

2 . Note that from the previous theorem, μ = μ and We denote E X = μX and Var X = σX X √ σX = σ/ n. Here, σX is called the standard error of the mean. It is important to notice that the variance of each of the random variables X1 , X2 , . . . , Xn is σ 2 , whereas the variance of the sample mean X is σ 2 /n, which is smaller than the population variance σ 2 for n ≥ 2. The implication of Theorem 4.1.1 is that the sample means become more and more reliable as an estimate of μ as the sample size is increased, as we would expect. From Chebyshev’s inequality, 1 P X − μX < kσ X ≥ 1 − 2 . k

√ √ Let ε = (kσ/ n). Then k = (ε n)/σ. Since μX = μ, the above inequality can be written as σ2 P X − μ < ε ≥ 1 − 2 . nε

Thus, for any ε > 0, the probability that the difference between X and μ less than ε can be made arbitrarily close to 1 by choosing the sample size n is sufﬁciently large. We illustrate this result in the following example.

Example 4.1.2 A particular brand of drink has an average of 12 ounces per can. As a result of randomness, there will be small variations in how much liquid each bottle really contains. It has been observed that the amount of liquid in these bottles is normally distributed with σ = 0.8 ounce. A sample of 10 bottles of this brand of

4.1 Introduction 187

soda is randomly selected from a large lot of bottles, and the amount of liquid, in ounces, is measured in each. Find the probability that the sample mean will be within 0.5 ounce of 12 ounces.

Solution Let X1 , X2 , . . . , X10 denote the ounces of liquid measured for each of the bottles. We know that Xi s are normally distributed with mean μ = 12 and variance σ 2 = 0.64. From Theorem 4.1.1, X possesses a normal distribution (actually, for the normality part, we use Corollary 4.2.2) with a mean 12 and variance σ 2 /n = 0.64/10 = 0.064. We find P X −12| ≤ 0.5) = P −0.5 ≤ X − 12 ≤ 0.5 X − 12 0.5 0.5 =P − √ ≤ √ ≤ √ σ/ n σ/ n σ/ n

0.5 0.5 ≤Z ≤ =P − 0.253 0.253 = P(−1.97 ≤Z ≤ 1.97) = 0.9512. using standard normal table . Hence, the chance is about 0.95% that the mean amount of drink in any 10 bottles randomly chosen will be between 11.5 to 12.5 ounces.

4.1.1 Finite Population

Let {c1 , c2 , . . . , cN } be a ﬁnite population. Then the population mean μ = (1/N) N i=1 ci and the 2 population variance σ 2 = (1/N) N i=1 (ci − μ) . The following theorem for the sample mean and variance is stated without proof. Theorem 4.1.2 If X1 , . . . , Xn is a sample of size n (chosen without replacement) from a population {c1 , c2 , . . . , cN }, then E X =μ σ2 Var X = n

N −n . N −1

We remark here that the sample in the theorem is not a random sample and Xi s are not iid random variables. The factor (N − n)/(N − 1) in the foregoing theorem is often called the ﬁnite population correction factor. It is close to 1 unless the sample amounts to a signiﬁcant portion of the population. Note that the sampling without replacement causes dependence among the Xi s. However, if the sample size n is small relative to the population size N, the population correction factor is approximately 1. Hence, we will not use the ﬁnite population correlation factor in the derivation of sampling distribution, unless it is absolutely necessary.

188 CHAPTER 4 Sampling Distributions

Example 4.1.3 Obtain the mean and variance of X in Example 4.1.1.

Solution

First note that for the population in Example 4.1.1, the population mean is μ = (1/N) N i=1 ci = 3 and the 2 population variance is σ 2 = (1/N) N i=1 (ci − μ) = 2. Applying the probability distribution of X given in Example 4.3.1, we obtain

7 1 8 2 2 10 2 1 E X =2 + + +3 + 10 3 10 3 10 10 3 10

1 11 1 +4 + 3 10 10 = 3, and 2 2 Var X = E X − EX = 22

+ 32 =

2 10

+

1 10

+

2 2 1 2 7 8 + 3 10 3 10

2

10 2 2 11 1 1 + + 42 − 32 3 10 3 10 10

2 1 × = 0.3333. 3 2

This is the same as (σ 2 /n). [(N − n)/(N − 1)]. In this case we observe that the variance of X is precisely one sixth of the original variance.

Example 4.1.4 Let X1 , . . . , Xn be a random sample from a population with mean μ and variance σ 2 . Consider the sample variance n 2 1 Xi − X . n−1

S2 =

i=1

Show that E(S 2 ) = σ 2 .

Solution It can be shown that (see Exercise 1.5.8) n

1 n−1

n

Xi − X

i=1

2

=

i=1

Xi2 − nX n−1

2

.

4.1 Introduction 189

Hence,

⎛ n ⎞ 2 2 X − nX n ⎜ ⎟ i n 1 2 2 ⎜ i=1 ⎟ E Xi − E X . E S2 = E ⎜ ⎟= ⎝ ⎠ n−1 n−1 n−1 i=1

Using the fact that E X2 = Var (X) + μ2 and Theorem 4.1.1, we have

E S2

σ2 1 n 2 2 2 n σ +μ − +μ = n−1 n−1 n

1 n n n 2 − σ + − μ2 = n−1 n−1 n−1 n−1 = σ2.

This shows that the expected value of the sample variance is the same as the variance of the population under consideration.

EXERCISES 4.1 4.1.1.

Let the population be given by the numbers {−2, −1, 0, 1, 2}. Take all random samples of size 3. (a) Without replacement, obtain the following in each case. (i) The sampling distribution of the sample mean. (ii) The sampling distribution of the sample median. (iii) The sampling distribution of the sample standard deviation. (iv) The mean and variance of the sample mean. (b) How many samples of size 3 can we get, if we sample with replacement?

4.1.2.

(a) How many different samples of size n = 2 can be chosen from a ﬁnite population of size 12 if the sampling is without replacement? (b) What is the probability of each sample in part (a), if each sample of size 2 is equally likely? (c) Find the value of the ﬁnite population correction factor.

4.1.3.

Let the population be given by {1, 2, 3}. Let p(x) = 1/3 for x = 1, 2, 3. Take samples of size 3 with replacement. (a) Calculate μ and σ 2 . (b) Obtain the sampling distribution of the sample mean. (c) Obtain the mean and variance of the sample mean.

4.1.4.

Find the value of the ﬁnite population correlation factor for (a) n = 8 and N = 60. (b) n = 8 and N = 1000. (c) n = 15 and N = 60.

190 CHAPTER 4 Sampling Distributions n

2 2 Xi − X . Find E[ S ]. Compare

4.1.5.

For a random sample X1 , . . . , Xn , let (S )2 = (1/n) this with E S 2 .

4.1.6.

For a random sample X1 , . . . , Xn with mean μ and variance σ 2 , let Tn = total. Show that E (Tn ) = nμ and Var (Tn ) = nσ 2 .

i=1

n

Xi , the sample

i=1

4.1.7.

A particular brand of sugar is sold in 5-lb packages. The weight of sugar in these packages can be assumed to be normally distributed with mean μ = 5 lb and standard deviation σ = 2 lb. What is the probability that the mean weight of sugar in 15 randomly selected packages will be within 0.2 lb of 5 lb?

4.1.8.

A random sample of size 150 is taken from an inﬁnite population having the mean μ = 15 and standard deviation σ = 2.5. What is the probability that X will be between 10.5 and 18.5?

4.1.9.

The distribution of heights of all students in a large university has a normal distribution with a mean of 66 inches and a standard deviation of 2 inches. What is the probability that the mean height of 26 randomly selected students from this university will be more than 70 inches?

4.1.10.

An image-encoding algorithm, when used to encode images of a certain size, uses a mean of 110 milliseconds with a standard deviation of 15 milliseconds. What is the probability that the mean time (in milliseconds) for encoding 50 randomly selected images of this size will be between 90 milliseconds and 135 milliseconds? What assumptions do we need to make?

4.1.11.

In order to evaluate a new release of a database management system, a database administrator runs a benchmark program several times and measures the time to completion in seconds. Assuming that the distribution of times is normal with mean 95 seconds and with standard deviation of 10 seconds, what proportion of measurement times will fall below 85 seconds?

4.1.12.

A population of disk drives manufactured by a certain company runs with mean seek time of 10 milliseconds with standard deviation of 0.1 milliseconds. What proportion of samples of size 250 would you expect to result in a mean less than 9 milliseconds? What assumptions do we need to make?

4.1.13.

Suppose that the national norm of a science test for 12th graders on a particular year has a mean of 215 and a standard deviation of 35. (a) A random sample of 55 12th graders is selected. What is the probability that this group will average more than 230? (b) A random sample of 200 12th graders is selected. What is the probability that this group will average over 230? (c) A random sample of 35 12th graders is selected. What is the probability that this group will average over 230? (d) How does the sample size inﬂuence the probability?

4.2 Sampling Distributions Associated with Normal Populations 191

4.1.14.

Scores on the Wechsler Adult Intelligence Scale for the 20 to 34 age group are approximately normally distributed with mean equal to 110 and standard deviation equal to 25. If we select 100 people at random, what is the probability that this group will have an average score of 125 or above?

4.1.15.

It is known that a healthy human body has an average temperature of 98.6◦ F, with a standard deviation of 0.95◦ F. Sixty healthy humans are selected at random. What is the probability that their temperatures average at least 99.1◦ F?

4.2 SAMPLING DISTRIBUTIONS ASSOCIATED WITH NORMAL POPULATIONS The sampling distribution of a statistic will depend upon the population distribution from which the samples are taken. In this section we discuss the sampling distributions of some statistics that are based on a random sample drawn from a normal distribution. These statistics are used in many statistical procedures that are very important in solving real-world problems. The following result establishes the distribution of a linear combination of independent normal random variables. Theorem 4.2.1 Let X1 , . . . , Xn be independent random variables with the distribution of Xi being normal with mean μi and variance σi2 . Let a1 , a2 , . . . , an be real constants. Then the distribution of Y = ni=1 ai Xi is normal with mean μY = ni=1 ai μi and variance σY2 = ni=1 ai2 σi2 . Proof. The moment-generating function of Y is given by n

MY (t) = Ee( i=1 ai Xi )t 7 = Ee(ai Xi )t [by independence ofXi s] i

=

7

Ee(ai t)Xi

i

= =

7 i

7

= e[(

MXi (ai t)

[using the deﬁnition of mgf)

2 2 2 e(ai μi t+(1/2)ai σi t )

i

i ai μi )t+(1/2)

[using mgf of a normal] 2

2 i ai σi

t2 ]

which is the mgf of a normal random variable with mean

i a i μi

and variance

2 2 i ai σi .

In Theorem 4.2.1 let ai = 1/n, μi = μ, and σ12 = σ 2 , we obtain the following result, which provides the distribution of the sample mean. Corollary 4.2.2 Let X1 , . . . , Xn be a random sample of size n from a normal population with mean μ and variance σ 2 . Then X = (1/n)

n i=1

Xi

2 = σ 2 /n. is normally distributed with mean μX = μ and variance σX

192 CHAPTER 4 Sampling Distributions

Recall that we have used the notation X ∼ N(μ, σ 2 ) to mean that the random variable X is normally distributed with mean μ and variance σ 2 . From Corollary 4.2.2, X ∼ N(μ, σ 2 /n) and hence by the √ z-transformation we obtain the standard normal random variable, Z = X − μ / σ/ n ∼ N(0, 1).

Example 4.2.1 A company that manufactures cars claims that the gas mileage for its new line of hybrid cars, on the average, is 60 miles per gallon with a standard deviation of 4 miles per gallon. A random sample of 16 cars yielded a mean of 57 miles per gallon. If the company’s claim is correct, what is the probability that the sample mean is less than or equal to 57 miles per gallon? Comment on the company’s claim about the mean gas mileage per gallon of its cars. What assumptions did you make?

Solution Let X represent the gas mileage for the new car (in miles per gallon). If the company’s claim is true, then from Corollary 4.2.2, X is normally distributed with mean μ = 60 and variance σ 2 /n = 16/16 = 1. Hence, 57 − 60 X − 60 P X ≤ 57 = P ≤ 1 1 = P(Z ≤ −3) ≈ 1 − 0.999 = 0.001. Therefore, if the company’s claim is correct, it is very unlikely that the mean value of the random sample of 16 cars will be 57 miles per gallon. Because the mean is indeed 57 miles per gallon, we conclude that the company’s claim is very likely not true. Here we have assumed that the sample of 16 measurements comes from a normal population, so that we could apply the results of Corollary 4.2.2.

Now we introduce some distributions that can be derived from a normal distribution. These distributions play a very important role in inferential problems.

4.2.1 Chi-Square Distribution A chi-square distribution is used in many inferential problems, for example, in inferential problems dealing with the variance. Recall that the chi-square distribution is a special case of a gamma distribution with α = n/2 and β = 2. If n is a positive integer, then the parameter n is called the degrees of freedom. However, if n is not an integer, but β = 2, we still refer to this distribution as a chi-square. The mgf of a χ2 − random variable is M(t) = (1 − 2t)−n/2 . The mean and variance of a chi-square distribution are μ = n and σ 2 = 2n, respectively. That is, the mean of a χ2 (n) random variable is equal to its degree of freedom and the variance is twice the degree of freedom. We now give some useful results for χ2 − random variables. Theorem 4.2.3 Let X1 , . . . , Xk be independent χ2 − random variables with n1 , . . . , nk degrees of freedom, k respectively. Then the sum V = i=1 Xi is chi-square distributed with n1 + n2 + · · · + nk degrees of freedom.

4.2 Sampling Distributions Associated with Normal Populations 193

Proof. The mgf of V is

MV (t) =

k 7

(1 − 2t)−ni /2 = (1 − 2t)

−

k

ni /2

i=1

.

i=1

This implies that V ∼ χ2

k i=1 ni

.

Our next result states that the difference of two chi-square random variables is a chi-square random variable, given by the following theorem. The proof is left as an exercise. Theorem 4.2.4 Let X1 and X2 be independent random variables. Suppose that X1 is χ2 with n1 degrees of freedom, whereas Y = X1 + X2 is chi-square with n degrees of freedom, where n > n1 . Then X2 = Y − X1 is a chi-square random variable with n − n1 degrees of freedom. The following result shows that we can generate a chi-square random variable from a gamma random variable. Theorem 4.2.5 If a random variable X has a gamma distribution with parameters α and β, then 2X ∼χ2 (2α). β

Y=

Proof. Recall that the mgf of the gamma random variable X is (1 − βt)−α .

2X t MY (t) = M 2X (t) = E e β β

X( β2 t)

=E e

= MX

2 t β

= (1 − 2t)−α = (1 − 2t)−

2α 2 .

Hence, Y ∼ χ2 (2α). The following result states that by squaring a standard normal random variable, we can generate a chi-square random variable, with one degree of freedom. Theorem 4.2.6 If X is a standard normal random variable, then X2 is chi-square random variable with 1 d.f. Proof. Because X ∼ N(0, 1) the moment-generating function of X2 is ∞ MX2 (t) = −∞

2 1 2 etx √ e−x /2 dx = (1 − 2t)−1/2 . 2π

This implies that X2 ∼ χ2 (1). Figure 4.1 gives the probability densities of the random variables X and X2 .

194 CHAPTER 4 Sampling Distributions

Densities of Standard normal r.v. and its square 4 3.5 3 2.5 pdf of X 2

2 1.5

pdf of X

1 0.5 0 23

22

21

0

1

2

3

■ FIGURE 4.1 pdf of standard normal r.v. and the pdf of its square.

The following result is a direct consequence of Theorems 4.2.3 and 4.2.6. This result illustrates how to obtain a random sample from chi-square distribution if we have a random sample of n measurements from a normal population. Theorem 4.2.7 Let the random sample X1 , . . . , Xn be from a N(μ, σ 2 ) distributed. Then Zi = (Xi − μ)/ σ, i = 1, . . . , n are independent standard normal random variables and n i=1

Zi2 =

n Xi − μ 2 i=1

σ

has a χ2 -distribution with n degrees of freedom. In particular, if X1 , . . . , Xn are independent standard normal random variables, then Y 2 = ni=1 Xi2 is chi-square distributed with n degrees of freedom. If X ∼ χ2 (n), then from the chi-square table, we can compute the values of χα2 (n) such that P X > χα2 (n) = α,

as shown by Figure 4.2. 2 (15) look in the chi-square table with the row labeled 15 d.f. For example, if X ∼ χ2 (15), to ﬁnd χ0.95 2 and the column headed χ0.950 and obtain the value as 7.26094. Thus, with 15 degrees of freedom, P (X > 7.26094) = 0.95. Also, if X is a chi-square random variable with 11 degrees of freedom, from 2 (11) = 19.675. Therefore, P (X > 19.675) = 0.05. the chi-square table we have χ0.05

4.2 Sampling Distributions Associated with Normal Populations 195

X 2(n ) ■ FIGURE 4.2 Chi-square probability density.

Example 4.2.2 Let the random variables X1 , X2 , . . . , X5 be from an N (5, 1) distribution. Find a number a such that ⎞ ⎛ 5 2 (Xi − 5) ≤ a⎠ = 0.90. P⎝ i=1

Solution By Theorem 4.2.7,

5 i=1

Zi2 =

5 Xi −5 2 i=1

1

=

5

(Xi − 5)2 has a chi-square distribution with 5 degrees of

i=1

freedom. Because the upper tail area is 0.10, looking at the chi-square table with 5 d.f. and the column 2 , we obtain a = 9.23635. Thus, corresponding to χ0.10 ⎞ ⎛ 5 2 (Xi − 5) ≤ 9.23635⎠ = 0.90. P⎝ i=1

Example 4.2.3 Suppose that X is χ2 − random variable with 20 degrees of freedom. Use the chi-square table to obtain the following: (a) Find x0 such that P (X > x0 ) = 0.95. (b) Find P (X ≤ 12.443).

Solution (a) For 20 degrees of freedom, using the chi-square table, we have P (X > 10.851) = 0.95. Hence, x0 = 10.851.

196 CHAPTER 4 Sampling Distributions

(b) From the chi-square table, P (X ≤ 12.443) = 0.10. The following result gives the probability distribution for a function of the sample variance S 2 .

Theorem 4.2.8 If X1 , . . . , Xn is a random sample from a normal population with the mean μ and variance σ 2 , then (a) the random variable n

Xi − X

(b)

i=1

σ2

2 =

(n − 1) S 2 . σ2

has a chi-square distribution with (n − 1) degrees of freedom. (c) X and S 2 are independent. Proof. We will only prove part (a). For part (b), we will give some comments on the proof. n 2 (a) We know from Theorem 4.2.7 that 1/σ 2 i=1 (Xi − μ) has a chi-square distribution with n degrees of freedom. Thus, n n 2 1 2= 1 (X Xi − X + X − μ − μ) i σ2 σ2 i=1 i=1 % n & n 2 2 1 Xi − X + = 2 X−μ σ i=1 i=1 n Since 2 Xi − X X − μ = 0 i=1

(n − 1) S 2 + = σ2

X−μ √ σ/ n

2 .

The left-hand side of this equation has a chi-square distribution with n degrees of freedom. √ # √ $2 ∼ χ2 (1). Also, since X − μ / σ/ n ∼N (0, 1) by Theorem 4.2.6 we have X − μ / σ/ n Now from Theorem 4.2.4, (n − 1) S 2 /σ 2 ∼ χ2 (n − 1). (b) We will accept the result of part (b) without proof here. A rigorous proof depends on geometric properties of the multivariate normal distribution, which is beyond the scope of this book. A proof based on moment-generating functions is relatively straightforward, where essentially we can ﬁrst show that the random variable X and the vector of ran dom variables X1 − X, . . . , Xn − X are independent. Because S 2 is a function of the vector X1 − X, . . . , Xn − X , it is then independent of X.

4.2 Sampling Distributions Associated with Normal Populations 197

Example 4.2.4 Let X1 , X2 , . . . , X10 be a random sample from a normal distribution with σ 2 = 0.8. Find two positive numbers a and b such that the sample variance S 2 satisﬁes P a ≤ S 2 ≤ b = 0.90.

Solution

2 Because (n−1)S ∼ χ2 (n − 1), we have σ2

P a ≤ S2 ≤ b = P

(n − 1) a (n − 1) S 2 (n − 1) b . ≤ ≤ σ2 σ2 σ2

The desired values can be found by setting the upper tail area and lower tail area each equal to 0.05. Using the chi-square table with n − 1 = 9 degrees of freedom, we have (n − 1) b 9b 2 = 16.919 = χ0.05,9 = , 0.8 σ2 which implies b = ((16.919) × (0.8) /9) = 1.50. Similarly, (n − 1) a 9a 2 = 3.325 = χ0.95,9 = . 2 0.8 σ So we have a = ((3.325) × (0.8) /9) = 0.295. Hence, P 0.295 ≤ S 2 ≤ 1.50 = 0.90. It is important to note that this is not the only interval that would satisfy P a ≤ S 2 ≤ b = 0.90 but it is a convenient one.

Example 4.2.5 A fruit-drink company wants to know the variation, as measured by the standard deviation, of the amount of juice in 16-ounce cans. From past experience, it is known that σ 2 = 2. The company statistician decides to take a sample of 25 cans from the production line and compute the sample variance. Assuming that the sample values may be viewed as a random sample from a normal population, ﬁnd a value of b such that P S 2 > b = 0.05.

198 CHAPTER 4 Sampling Distributions

Solution To find the necessary probability, use the fact that (n − 1) S 2 /σ 2 ∼ χ2 (n − 1), with n = 25, 24b 24S 2 2 0.05 = P(S > b) = P > 2 2 = P(χ2 > c). 2 c = 2 (36.4151) = 3.03 and From the chi-square table we obtain, c = 36.4151. Hence, b = 24 24

P S 2 > 3.03 = 0.05.

SUMMARY OF CHI-SQUARE DISTRIBUTION Let X1 , . . . , Xn be iid N μ, σ 2 random variables. Then 1. X has N μ, σ 2 /n distribution, 2. (n − 1)S 2 /σ 2 has a chi-square distribution with (n − 1) degrees of freedom, and 3. X and S 2 are independent. 4. A χ2 − random variable has a mean equal to its degrees of freedom and a variance equal to twice its degrees of freedom.

4.2.2 Student t-Distribution Let the random variables X1 , . . . , Xn follow a normal distribution with mean μ and variance σ 2 . √ If σ is known, then we know that n X − μ /σ is N (0, 1). However, if σ is not known (as is usually the case), then it is routinely replaced by the sample standard deviation s. If the sample size is large, one could suppose that s ≈ σ and apply the Central Limit Theorem and obtain that √ n X − μ /S is approximately an N (0, 1). However, if the random sample is small, then the dis √ tribution of n X − μ /S is given by the so-called Student t-distribution (or simply t-distribution). This was originally developed by W. S. Gosset in 1908. Because his employers, the Guinness brewery, would not permit him to publish this important work in his own name, he used the pseudonym “Student.” Thus, the distribution is known as the Student t-distribution. Deﬁnition 4.2.2 If Y and Z are independent random variables, Y has a chi-square distribution with n degrees of freedom, and Z ∼ N (0, 1), then T = √

Z Y /n

is said to have a (Student) t-distribution with n degrees of freedom. We denote this by T ∼ Tn . The probability density of the random variable T with n degrees of freedom is given by − n+1 2 n+1 t2 2 n 1 + , −∞ < t < ∞. f (t) = √ n πn 2

4.2 Sampling Distributions Associated with Normal Populations 199

T density for n 5 2, n 5 10, n 5 20, n 5 30 0.4

n52

0.35 0.3 0.25

n 5 10 n 5 20

0.2 0.15

n 530

0.1 0.05 24 23

22

21

0

1

2

3

4

■ FIGURE 4.3 The Student t-distribution.

Figure 4.3 illustrates the behavior of the t-distributions for n = 2, 10, 20, and 30. It is clear from Figure 4.3 that as n becomes larger and larger, it is almost impossible to distinguish the graphs. It can be shown that the t-distribution tends to a standard normal distribution as the degrees of freedom (equivalently, the sample size n) tend to inﬁnity. In fact, the standard normal distribution provides a good approximation to the t-distribution for sample sizes of 30 or more. We will use this approximation in the statistical inference problems for n ≥ 30. The t-density is symmetric about zero, and then we have E (T ) = 0. If n > 2, it can be shown that Var (T ) = n/ (n − 2). The value of tα,n is such that P t > tα,n = α (the shaded area in Figure 4.4) is obtained from the t-table. For example, if a random variable X has a t-distribution with 9 degrees of freedom and α = 0.01, then t0.01,9 = 2.821. If we have a random sample from a normal population, the following result involving a t-distribution is useful in applications. Theorem 4.2.9 If X and S 2 are the mean and the variance of a random sample of size n from a normal population with the mean μ and variance σ 2 , then T =

X−μ √ S/ n

has a t-distribution with (n−1) degrees of freedom. Proof. By Corollary 4.2.2, Z=

X−μ √ ∼ N (0, 1) . σ/ n

200 CHAPTER 4 Sampling Distributions

f (t ) 0.4 0.35 0.3 0.25 0.2 0.15 0.1 0.05 24 23

22

21

0

1

2

3

4

t

■ FIGURE 4.4 Probability of t-distribution.

By Theorem 4.2.8, we have Y=

n 2 (n − 1) S 2 1 Xi − X ∼ χ2 (n − 1) . = 2 2 σ σ i=1

Hence, X−μ √ σ/ n

T = 8

(n−1)S 2 σ 2 (n−1)

∼ 9

Z χ2 (n−1) n−1

.

Also, X and S 2 are independent. Thus, Y and Z are independent, and by Deﬁnition 4.2.2, T follows a t-distribution with (n − 1) degrees of freedom. How can we distinguish between given degrees of freedom and the degrees of freedom from a sample? For the t-distribution, if n is given as the degrees of freedom, we will just use n. However, if a random sample of size n is given, then the corresponding degrees of freedom will be (n − 1), as given in Theorem 4.2.9. The assumption that the sample comes from a normal population is not that onerous. In practice, it is necessary to check that the sampled population is approximately bell shaped and not too much skewed. Construction of the normal-scores plot or histogram is a way to check for approximate normality. See Project 4C.

4.2 Sampling Distributions Associated with Normal Populations 201

Example 4.2.6 A manufacturer of fuses claims that with 20% overload, the fuses will blow in less than 10 minutes on the average. To test this claim, a random sample of 20 of these fuses was subjected to a 20% overload, and the times it took them to blow had the mean of 10.4 minutes and a sample standard deviation of 1.6 minutes. It can be assumed that the data constitute a random sample from a normal population. Do they tend to support or refute the manufacturer’s claim?

Solution Given y = 10.4, s = 1.6, n = 20, and μ = 10. Hence t=

y−μ 10.4 − 10 = 1.118. √ √ = s/ n 1.6/ 20

The degree of freedom is n − 1 = 19. From the t-table, the probability that t exceeds 1.328 is 0.10, and because the observed value of t = 1.118 is less than t0.10 (19) = 1.328 and 0.10 is a pretty large probability, we conclude that the data tend to agree with the manufacturer’s claim.

We will study the problems of the foregoing type in Chapter 7, where we will be learning about hypothesis testing. Prior to Student’s work on the t-distribution, a very large number of observations were necessary for design and analysis of experiments. Today, the use of the t-distribution often makes it possible to draw reliable conclusions from samples as small as 15 to 30 experimental units, provided that the samples are representative of their populations and that normality could reasonably be assumed or justiﬁed for the population.

Example 4.2.7 The human gestation period—the period of time between conception and labor—is approximately 40 weeks (280 days), measured from the ﬁrst day of the mother’s last menstrual period. For a newborn fullterm infant, the length appropriate for gestational age is assumed to be normally distributed with μ = 50 centimeters and σ = 1.25 centimeters. Compute the probability that a random sample of 20 infants born at full term results in a sample mean greater than 52.5 centimeters.

Solution Let X be length (measured in centimeters) of a newborn full-term infant. Then X ∼ N (50, 1.56/20). Hence

52.5 − 50 = 8.94 ≈ 0. P X > 52.5 = P t > √ 1.25/ 20 Thus, the probability of such an occurrence is negligible.

In the previous example, it should be noted that P X > 52.5 ≈ 0 does not imply that the probability of observing a newborn full-term infant with length greater than 52.5 centimeters is zero. In fact, with 19 degrees of freedom, P (X > 52.5) = P (t > 2) ≈ 0.025.

202 CHAPTER 4 Sampling Distributions

4.2.3 F-Distribution The F -distribution was developed by Fisher to study the behavior of two variances from random samples taken from two independent normal populations. In applied problems we may be interested in knowing whether the population variances are equal or not, based on the response of the random samples. Knowing the answer to such a question is also important in selecting the appropriate statistical methods to study their true means. Deﬁnition 4.2.3 Let U and V be chi-square random variables with n1 and n2 degrees of freedom, respectively. Then if U and V are independent, F=

U/n1 V /n2

is said to have an F-distribution with n1 numerator degrees of freedom and n2 denominator degrees of freedom. We denote this by F ∼ F (n1 , n2 ). The pdf for a random variable X ∼ F (n1 , n2 ) is given by ⎧ ⎨ ((n1 + n2 )/2) n1 n1 /2 n21 −1 n1 −(n1+n2 )/2 1 + x x , x>0 n n (n /2) (n /2) 2 2 1 2 f (x) = ⎩ 0, elsewhere.

A graph of f (x) for various values of n is given in Figure 4.5.

F – density with n 1⫽ 3, n 2⫽ 2, and n 1⫽ 12, n 2⫽ 6

0.7 0.6

F (3, 2)

0.5 0.4

F (12, 6)

0.3 0.2 0.1 0

1

2

3

4

■ FIGURE 4.5 pdfs of F -distribution.

5

6

7

4.2 Sampling Distributions Associated with Normal Populations 203

0.7 0.6 0.5 0.4 0.3

F (n 1,n 2)

0.2 0.1 0

1

2

3

4

5

6

7

■ FIGURE 4.6 F -distribution probability.

To ﬁnd Fα (n1 , n2 ) such that P (F > Fα (n1 , n2 )) = α (shaded area in Figure 4.6), we use the F -table. For example, if F has 3 numerator and 6 denominator degrees of freedom, then F0.01 (3, 6) = 9.78. If we know Fα (n1 , n2 ), it is possible to ﬁnd F1−α (n2 , n1 ) by using the identity F1−α (n2 , n1 ) = 1/Fα (n1 , n2 ) .

Using this identity we can obtain F0.99 (6, 3) = 1/F0.01 (3, 6) = 1/9.78 = 0.10225. When we need to compare the variances of two normal populations, we will use the following result. Theorem 4.2.10 Let two independent random samples of size n1 and n2 be drawn from two normal populations with variances σ12 , σ22 , respectively. If the variances of the random samples are given by S12 , S22 , respectively, then the statistic S 2 /σ 2 σ2S2 F = 12 12 = 22 12 S2 /σ2 σ1 S2

has the F-distribution with (n1 − 1) numerator and (n2 − 1) denominator degrees of freedom. Proof. From Theorem 4.2.9, we know that U=

(n1 − 1) S12 σ12

∼ χ2 (n1 − 1)

and V =

(n2 − 1) S22 σ22

∼ χ2 (n2 − 1) .

Also, U and V are independent. From Deﬁnition 4.2.3, F ∼ F (n1 − 1, n2 − 1).

204 CHAPTER 4 Sampling Distributions

Corollary 4.2.11 If σ12 = σ22 , then S2 F = 12 ∼F (n1 − 1, n2 − 1). S2

When σ12 = σ22 , we refer to them as two populations that are homogeneous with respect to their variances.

Example 4.2.8 Let S12 denote the sample variance for a random sample of size 10 from Population I and let S22 denote the sample variance for a random sample of size 8 from Population II. The variance of Population I is assumed to be three times the variance of Population II. Find two numbers a and b such that P a ≤ S12 /S22 ≤ b = 0.90 assuming S12 to be independent of S22 .

Solution From the problem, we can assume that σ12 = 3σ22 with n1 = 10 and n2 = 8. Thus, we can write S12 /σ12

S12 /3σ22 S12 = = , S22 /σ22 S22 /σ22 3S22 this has F -distribution with n1 − 1 = 9 numerator and n2 − 1 = 7 denominator degrees of freedom. Using the F -table, F0.05 (9, 7) = 3.68. Now to find F0.95 such that S12 < F0.95 = 0.05. P 3S22 We proceed as follows:

P

S12 3S22

< F0.95

=P

3S22 S12

>

1 F0.95

= 0.05.

Indexing ν1 = 7 and ν2 = 9 in the F -table, we have 1/F0.95 (7, 9) = 3.29 or F0.95 = 1/3.29 = 0.304. Hence, the entire probability statement is S12 S12 ≤ 3.68 = P 0.912 ≤ 2 ≤ 11.04 = 0.90. P 0.304 ≤ 3S22 S2 Thus, a = 0.912 and b = 11.04.

EXERCISES 4.2 4.2.1.

Let Y have a chi-square distribution with 15 degrees of freedom. Find the following probabilities. (a) P (Y ≤ y0 ) = 0.025 (b) P (a < Y < b) = 0.95 (c) P (Y ≥ 22.307).

4.2 Sampling Distributions Associated with Normal Populations 205

4.2.2.

Let Y have a chi-square distribution with 7 degrees of freedom. Find the following probabilities. (a) P (Y > y0 ) = 0.025 (b) P (a < Y < b) = 0.90 (c) P (Y > 1.239).

4.2.3.

The time to failure T of a microwave oven has an exponential distribution with pdf f (t) =

1 −t/2 e , 2

t > 0.

If three such microwave ovens are chosen and t is the mean of their failure times, ﬁnd the following: (a) Distribution of T . (b) P T > 2 . 4.2.4.

Let X1 , X2 , . . . , X10 be a random sample from a standard normal distribution. Find the numbers a and b such that 10 2 Xi ≤ b = 0.95. P a≤

4.2.5.

Let X1 , X2 , . . . , X5 be a random sample from the normal distribution with mean 55 and variance 223. Let

i=1

Y=

5

(Xi − 55)2 /223

i=1

and Z=

5

Xi − X

2

/223.

i=1

(a) Find the distribution of the random variables Y and Z. (b) Are Y and Z independent? (c) Find (i)P(0.62 ≤ Y ≤ 0.76), and (ii)P(0.77 ≤ Z ≤ 0.95). 4.2.6.

Let X and Y be independent chi-square random variables with 14 and 5 degrees of freedom, respectively. Find (a) P (|X − Y | ≤ 11.15), (b) P (|X − Y | ≥ 3.8).

4.2.7.

A particular type of vacuum-packed coffee packet contains an average of 16 ounces. It has been observed that the number of ounces of coffee in these packets is normally distributed with σ = 1.41 ounce. A random sample of 15 of these coffee packets is selected, and the observations are used to calculate s. Find the numbers a and b such that P a ≤ S 2 ≤ b = 0.90.

4.2.8.

An optical ﬁrm buys glass slabs to be ground into lenses, and it is known that the variance of the refractive index of the glass slabs is to be no more than 1.04 × 10−3 . The ﬁrm rejects a shipment of glass slabs if the sample variance of 16 pieces selected at random exceeds

206 CHAPTER 4 Sampling Distributions

1.15 × 10−3 . Assuming that the sample values may be looked on as a random sample from a normal population, what is the probability that a shipment will be rejected even though σ 2 = 1.04 × 10−3 ? 4.2.9.

Assume that T has a t-distribution with 8 degrees of freedom. Find the following probabilities. (a) P (T ≤ 2.896) (b) P (T ≤ −1.860) (c) The value of a such that P (−a < T < a) = 0.99

4.2.10.

Assume that T has a t-distribution with 15 degrees of freedom. Find the following probabilities. (a) P (T ≤ 1.341) (b) P (T ≥ −2.131) (c) The value of a such that P (−a < T < a) = 0.95

4.2.11.

A psychologist claims that the mean age at which female children start walking is 11.4 months. If 20 randomly selected female children are found to have started walking at a mean age of 11.5 months with standard deviation of 2 months, would you agree with the psychologist’s claim? Assume that the sample came from a normal population.

4.2.12.

Let U1 and U2 be independent random variables. Suppose that U1 is χ2 with ν1 degrees of freedom while U = U1 + U2 is chi-square with ν degrees of freedom, where ν > ν1 . Then prove that U2 is chi-square random variable with ν − ν1 degrees of freedom.

4.2.13.

Show that if X ∼ χ2 (ν), then EX = ν and Var (X) = 2ν.

4.2.14.

Let X1 , . . . , Xn be a random sample with Xi ∼ χ2 (1), for i = 1, . . . , n. Show that the distribution of X−1 Z= √ 2/n

as n → ∞ is standard normal. 4.2.15. 4.2.16.

Find the variance of S 2 , assuming the sample X1 , X2 , . . . , Xn is from N μ, σ 2 .

Let X1 , X2 , . . . , Xn be a random sample an exponential distribution with parameter

from n −1 θ. Show that the random variable 2θ Xi ∼ χ2 (2n). i=1

4.2.17.

Let X and Y be independent random variables from an exponential distribution with common parameter θ = 1. Show that X/Y has an F -distribution. What is the number for degrees of freedom?

4.2.18.

Prove that if X has a t-distribution with n degrees of freedom, then X2 ∼ F (1, n).

4.2.19.

Let X be F distributed with 9 numerator and 12 denominator degrees of freedom. Find (a) P (X ≤ 3.87), (b) P (X ≤ 0.196), (c) The value of a and b such that P (a < Y < b) = 0.95.

4.3 Order Statistics 207

4.2.20.

Prove that if X ∼ F (n1 , n2 ), then 1/X ∼ F (n2 , n1 ).

4.2.21.

Find the mean and variance of F (n1 , n2 ) random variable.

4.2.22.

Let X11 , X12 , . . . , X1n1 be a random sample with sample mean X1 from a normal population with mean μ1 and variance σ12 , and let X21 , X22 , . . . , X2n2 be a random sample with sample mean X2 from a normal population with mean μ2 and variance σ22 . Assume the two samples are independent. Show that the sampling distribution of X1 − X2 is normal with mean μ1 − μ2 and variance σ12 /n1 + σ22 /n2 .

4.2.23.

Let X1 , X2 , . . . , Xn1 be a random sample from a normal population with mean μ1 and variance σ 2 , and Y1 , Y2 , . . . , Yn2 be a random sample from an independent normal population with mean μ2 and variance σ 2 . Show that X − Y − (μ1 − μ2 ) ∼ T(n1 +n2 −2) T = 8 (n1 −1)S12 +(n2 −1)S22 1 1 + n1 n2 n1 +n2 −2

4.2.24.

Show that a t-distribution tends to a standard normal distribution as the degrees of freedom tend to inﬁnity.

4.2.25.

Show that the mgf of a χ2 random variable is M (t) = (1 − 2t)−ν/2 . Using the mgf, show that the mean and variance of a chi-square distribution are ν and 2ν, respectively.

4.2.26.

Let the random variables X1 , X2 , . . . , X10 be normally distributed with mean 8 and variance 4. Find a number a such that ⎛

P⎝

10 Xi − 8 2 i=1

4.2.27.

2

⎞

≤ a⎠ = 0.95

Let X2 ∼ F (1, n). Show that X ∼ t (n).

4.3 ORDER STATISTICS In practice, the random variables of interest may depend on the relative magnitudes of the observed variable. For example, we may be interested in the maximum mileage per gallon of a particular class of cars. In this section, we study the behavior of ordering a random sample from a continuous distribution. Deﬁnition 4.3.1 Let X1 , . . . , Xn be a random sample from a continuous distribution with pdf f (x). Let Y1 , . . . , Yn be a permutation of X1 , . . . , Xn such that Y1 ≤ Y2 ≤ · · · ≤ Yn .

Then the ordered random variables Y1 , . . . , Yn are called the order statistics of the random sample X1 , . . . , Xn . Here Yk is called the kth order statistic. Because of continuity, the equality sign could be ignored.

208 CHAPTER 4 Sampling Distributions

Remark. Although Xi ’s are iid random variables, the random variables Yi ’s are neither independent nor identically distributed. Thus, the minimum of Xi ’s is Y1 = min (X1 , . . . , Xn )

and the maximum is Yn = max (X1 , . . . , Xn ).

The order statistics of the sample X1 , X2 , . . . , Xn can also be denoted by X(1) , X(2) , . . . , X(n) where X(1) < X(2) < · · · < X(n) .

Here X(k) is the kth order statistic and is equal to Yk in Deﬁnition 4.3.1. One of the most commonly used order statistics is the median, the value in the middle position in the sorted order of the values.

Example 4.3.1 (i) The range R = Yn − Y1 is a function of order statistics. (ii) The sample median M equals Ym+1 if n = 2m + 1. Hence, the sample median M is an order statistic, when n is odd. If n is even then the sample median can $ # be obtained using the order statistic, M = (1/2) Yn/2 + Y(n/2)+1 .

The following result is useful in determining the distribution of functions of more than one order statistics. Theorem 4.3.1 Let X1 , . . . , Xn be a random sample from a population with pdf f (x). Then the joint pdf of order statistics Y1 , . . . , Yn is f (y1 , . . . , yn ) =

⎧ ⎨ n!f (y1 )f (y2 ) . . . f (yn ), ⎩

0,

for y1 < · · · < yn otherwise.

The pdf of the kth order statistic is given by the following theorem. Theorem 4.3.2 The pdf of Yk is fk (y) = fYk (y) =

n! f (y) (F (y))k−1 (1 − F (y))n−k , (k − 1)! (n − k)!

for −∞ < y < ∞, where F (y) = P(Xi ≤ y) is the cdf of Xi . In particular, the pdf of Y1 is f1 (y) = nf (y) [1 − F (y)]n−1 and the pdf of Yn is fn (y) = nf (y) [F (y)]n−1 . In the following example, we will derive pdf for Yn .

4.3 Order Statistics 209

Example 4.3.2 Let X1 , . . . , Xn be a random sample from U [0, 1]. Find the pdf of the kth order statistic Yk .

Solution Since the pdf of Xi is f (x) = 1, 0 ≤ x ≤ 1, the cdf is F (x) = x, 0 ≤ x ≤ 1. Using Theorem 4.3.2, the pdf of the kth order statistic Yk reduces to fk (y) =

n! yk−1 (1 − y)n−k , 0 ≤ y ≤ 1 (k − 1)! (n − k)!

which is a beta distribution with α = k and β = n − k + 1.

The next example gives the so-called extreme (i.e., largest) value distribution, which is the distribution of the order statistic Yn .

Example 4.3.3 Find the distribution of the nth order statistic Yn of the sample X1 , . . . , Xn from a population with pdf f (x).

Solution Let the cdf of Yn be denoted by Fn (y). Then

Fn (y) = P(Yn ≤ y) = P

max Xi ≤ y

1≤i≤n

= P(X1 ≤ y, . . . , Xn ≤ y) = [F (y)]n (by independence). Hence, the pdf fn (y) of Yn is fn (y) =

d d [F (y)]n = n[F (y)]n−1 F (y) dy dy

= n[F (y)]n−1 f (y). In particular, if X1 , . . . , Xn is a random sample from U [0, 1], then the cumulative extreme value distribution is given by ⎧ ⎪ 0, ⎪ ⎨ Fn (y) = yn , ⎪ ⎪ ⎩1,

y 1.

210 CHAPTER 4 Sampling Distributions

Example 4.3.4 A string of 10 light bulbs is connected in series, which means that the entire string will not light up if any one of the light bulbs fails. Assume that the lifetimes of the bulbs, τ1 , . . . , τ10 , are independent random variables that are exponentially distributed with mean 2. Find the distribution of the life length of this string of light bulbs.

Solution Note that the pdf of τi is f(t) = 2e−2t , 0 < t < ∞, and the cumulative distribution of τi is Fτi (t) = 1−e−2t . Let T represent the lifetime of this string of light bulbs. Then, T = min(τ1 , . . . , τ10 ). Thus, FT (t) = 1 − [1 − Fτi (t)]10 . Hence, the density of T is obtained by differentiating FT (t) with respect to t, that is, fT (t) = 10fτi (t)[1 − Fτi (t)]9 2(10)e−2t (e−2t )9 = 20e−20t , 0,

=

0 0

0,

otherwise,

212 CHAPTER 4 Sampling Distributions

and fn (yn ) =

⎧ ⎨ n e−yn /θ 1 − e−yn /θ n−1 , ⎩

θ

0,

if yn > 0 otherwise.

(b) Let n = 2l + 1. Show that the sampling distribution of the median, M, is given by ⎧ ⎨ n! e−m(l+1)/θ 1 − e−m/θ l , 2 f (m) = (l!) θ ⎩ 0,

for m > 0 otherwise.

4.3.11.

Let X1 , . . . , Xn be a random sample from a beta distribution with α = 2 and β = 3. Find the joint pdf of Y1 and Yn .

4.3.12.

Let X1 , . . . , Xn be a random sample from a geometric distribution with pmf pi = P (X = i) = pqi−1 , i = 1, 2, . . . , 0 < p < 1, q = 1 − p.

Show that P(Yk = y) =

n i=k

n (y−1)(n−i) n−i q {q [1 − qy ]i − [1 − qy−1 ]i }, i

y = 1, 2, . . . .

4.4 LARGE SAMPLE APPROXIMATIONS If the sample size is large, the normality assumption on the underlying population can be relaxed. A useful generalization of Corollary 4.2.2 follows. Theorem 4.4.1 Suppose that the population (not necessarily normal) from which samples are taken has a probability distribution with mean μ and variance σ 2 . Then the standardized variable (or z-transform) associated with X, given by Z=

X−μ √ σ/ n

is asymptotically standard normal. That is, 1 lim P (Z ≤ z) = √ n→∞ 2π

z

2 e−u /2 du.

−∞

Theorem 4.4.1 follows directly from the Central Limit Theorem. The consequence of this for statistics is that, regardless of the form of the population distribution, the distribution of the z-transform of a sample mean X will be approximately a standard normal random variable whenever n is large. This fact will be used in almost all large sample inference problems. It is important to note that, by

4.4 Large Sample Approximations 213

Theorem 4.2.2, if the random sample came from a normal population, then sampling distribution of the mean is normally distributed regardless of the size of the sample. We could use the foregoing results if the population variance σ 2 is known or when the sample size is large. Even though the required sample size to apply Theorem 4.4.1 will depend on the particular distribution of the population, for practical purposes we will consider the sample size to be large enough if n ≥ 30.

Example 4.4.1 The average SAT score for freshmen entering a particular university is 1100 with a standard deviation of 95. What is the probability that the mean SAT score for a random sample of 50 of these freshmen will be anywhere from 1075 to 1110?

Solution

√ The distribution of X has the mean μX = 1100 and σX = 95/ 50. By Theorem 4.4.3, √ X ∼ N 1100, σX = 95/ 90 . The z-series corresponding to 1075 and 1110 are z = [(1075 − 1100)/ √ √ 95/ 50 = −1.8608 and z = (1110 − 1100)/95/ 50 = 0.74432. Hence P 1075 ≤ X ≤ 1110 = P (−1.8608 ≤ Z ≤ 0.74432) = 0.739 means that we are 73.9% certain based on the given data that the mean SAT score is between 1075 and 1110, inclusive.

4.4.1 The Normal Approximation to the Binomial Distribution We know that a binomial random variable Y , with parameters n and p = P (success), can be viewed as the number of successes in n trials and can be written as Y=

n

Xi

i=1

where, Xi =

1

with probability p

0

with probability (1 − p).

The fraction of successes in n trials is 1 Y = Xi = X. n n n

i=1

Hence, Y /n is a sample mean. Since E (Xi ) = p and Var (Xi ) = p (1 − p), we have n

Y 1 1 E =E Xi = np = p n n n i=1

214 CHAPTER 4 Sampling Distributions

■ FIGURE 4.7 Probability function of discrete r.v.

and

n p (1 − p) 1 Y = 2 . Var Var (Xi ) = n n n i=1

Because Y = nX, by the Central Limit Theorem, Y has an approximate normal distribution with mean μ = np and variance σ 2 = np(1 − p). Because the calculation of the binomial probabilities is cumbersome for large sample sizes n, the normal approximation to the binomial distribution is widely used. A useful rule of thumb for use of the normal approximation to the binomial distribution is to make sure n is large enough if np ≥ 5 and n(1 − p) ≥ 5. Otherwise, the binomial distribution may be so asymmetric that the normal distribution may not provide a good approximation. Other rules, such as np ≥ 10 and n(1 − p) ≥ 10, or np(1 − p) ≥ 10, are also used in the literature. Because all of these rules are only approximations, for consistency’s sake we will use np ≥ 5 and n(1 − p) ≥ 5 to test for largeness of sample size in the normal approximation to the binomial distribution. If need arises, we could use the more stringent condition np(1 − p) ≥ 10. Recall that discrete random variables take no values between integers, and their probabilities are concentrated at the integers as shown in Figure 4.7. However, the normal random variables have zero probability at these integers; they have nonzero probability only over intervals. Because we are approximating a discrete distribution with a continuous distribution, we need to introduce a correction factor for continuity which is explained below.

CORRECTION FOR CONTINUITY FOR THE NORMAL APPROXIMATION TO THE BINOMIAL DISTRIBUTION (a) To approximate P(X ≤ a) or P(X > a), the correction for continuity is (a + 0.5), that is,

(a + 0.5) − np P(X ≤ a) = P Z < √ np(1 − p) and

(a + 0.5) − np . P(X > a) = P Z > √ np(1 − p)

(b) To approximate P(X ≥ a) or P(X < a), the correction for continuity is (a − 0.5), that is,

(a − 0.5) − np P(X ≥ a) = P Z > √ np(1 − p)

4.4 Large Sample Approximations 215

f (x)

x i ⫺1/2 i ⫹1/2 ■ FIGURE 4.8 Continuity correction for P(X = i).

and

(a − 0.5) − np . P(X < a) = P Z < √ np(1 − p)

(c) To approximate P(a ≤ X ≤ b), treat ends of the intervals separately, calculating two distinct z-values according to steps (a) and (b), that is,

(a + 0.5) − np (a − 0.5) − np 225) = P Z > 7.8804 more than 225 drivers will use less than one third of the acceleration lane length before merging.

EXERCISES 4.4 4.4.1.

A random sample size of 150 is taken from an inﬁnite population having mean μ = 8 and variance σ 2 = 4. What is the probability that X will be between 7.5 and 10?

4.4.2.

A machine that is used to ﬁll bottles with soda has been observed to have a true standard deviation in the amounts of ﬁll of approximately σ = 1.25 ounces. However, the mean ounces of ﬁll μ may change from day to day, because of change of operator or adjustments in the machine. If n = 55 observations on ounces of ﬁll are taken on a given day, ﬁnd the probability that the sample mean will be within 0.5 ounce of the true population mean. State any assumptions.

4.4.3.

The times spent by customers coming to a certain gas station to ﬁll up can be viewed as independent random variables with a mean of 3 minutes and a variance of 1.5 minutes. Approximate the probability that a random sample of 75 customers in this gas station will spend a total time less than 3 hours. Interpret your results and state any assumptions.

4.4.4.

Refer to Exercise 4.4.3. Find the number of customers, m, such that the probability that all the m customers can ﬁll up in less than 3 hours is approximately 0.2.

4.4.5.

In the mathematics department of a certain university, in a particular semester, 1250 students took the elementary algebra ﬁnal examination. The mean was 69% with a standard deviation of 5.4%. If a random sample of 60 students is selected from this population, what is the

4.4 Large Sample Approximations 217

probability that the average score of this sample will be at most 75.08? Interpret your results and state any assumptions. 4.4.6.

For a newborn full-term infant, the weight appropriate for gestational age is assumed to be normally distributed with μ = 3025 grams and σ = 165 grams. Compute the probability that a random sample of 50 infants born at full term results in a sample mean of less than 3500 grams.

4.4.7.

Let X1 , . . . , Xn be a random sample, each with mean μ1 and standard deviation σ1 . Also, let Y1 , Y2 , . . . , Ym be a random sample, each with mean μ2 and a standard deviation σ2 . Assume that both the samples are from normal populations. Verify that 1 2 X − Y ∼ N μ1 − μ2 , 1n σ12 + m σ2 .

4.4.8.

Let X1 , . . . , Xn be a random sample, each with mean μ1 and standard deviation σ1 . Also, let Y1 , Y2 , . . . , Yn be a random sample independent of X1 , . . . , Xn , each with mean μ2 and a standard deviation σ2 . Prove that the random variable Vn =

X − Y − (μ1 − μ2 ) 9 2 2 σ1 +σ2 n

satisﬁes the conditions of Theorem 4.4.1 and hence Vn is asymptotically normal. 4.4.9.

Suppose X is a binomial random variable with n = 20 and p = 0.2. Find the probability that X ≤ 10 using binomial tables and compare this to the corresponding value found from normal approximation.

4.4.10.

Using normal approximation, ﬁnd the probability of obtaining 90 heads in 150 tosses of a fair coin. Is the normal approximation valid? Why?

4.4.11.

A car rental company ﬁnds that each day 6% of the persons making reservations will not show up. If the rental company reserves for 215 persons with only 200 automobiles, what is the probability that an automobile will be available for every person who shows up holding a reservation? (Use the normal approximation.)

4.4.12.

The president of the United States is thought to have a positive approval rating of 58% of the people at a certain time. In a random sample of 1200 people, what is the approximate probability that the number of positive approvals will be at least 750? Interpret your results and state any assumptions.

4.4.13.

In the United States, sudden infant death syndrome (SIDS) is one of the leading causes of postneonatal deaths (those occurring between the ages of 28 days and 1 year). Thus far, the most signiﬁcant risk factor discovered for SIDS is placing babies to sleep in a prone position (on their stomachs). Suppose the rate of death due to SIDS is 0.00103 per year. In a random sample of 5000 infants between the ages of 28 days and 1 year, what is the approximate probability that the number of SIDS-related deaths will be at least 10? Interpret your results and state any assumptions.

218 CHAPTER 4 Sampling Distributions

4.4.14.

Let X and Y be independent binomial random variables with parameters (n, p1 ) and (m, p2 ), respectively. X Y (a) Find E − . n n

X Y (b) Find Var − . n n

X Y X Y X Y (c) Show that − ∼N E − , Var − , for large n. n n n n n n

4.5 CHAPTER SUMMARY In this chapter, we learned about sampling distributions. In sampling distributions associated with normal populations, we have seen that we can generate chi-square, t-, and F -distributions. In Section 4.3 we dealt with order statistics. Then in Section 4.4 we looked at large sample approximations such as the normal approximation to the binomial distribution. In the following section, we will give Minitab examples to show how the idea of sampling distribution can be explored using statistical software. We will now list some of the key deﬁnitions introduced in this chapter. ■

Sampling distribution

■

Sample and sample size

■

Random sample

■

Statistic

■

Standard error

■

Finite population correction factor

■

Degrees of freedom

■

t-distribution

■

F -distribution

■

Order statistics

In this chapter, we have also presented the following important concepts and procedures: ■

Sampling distribution associated with normal distribution

■

Results on chi-square distribution

■

Results on Student t-distribution

■

Results on F -distribution

■

Derivation of probability density functions for order statistics

■

Large sample approximations

■

Normal approximation to the binomial

■

Correction for continuity for the normal approximation to the binomial distribution

4.6 Computer Examples 219

4.6 COMPUTER EXAMPLES 4.6.1 Minitab Examples Example 4.6.1 Create three samples of size 30 from standard normal distribution using Minitab, and draw histograms for each sample.

Solution We can use the following procedure: 1. Open a new worksheet. 2. Choose Calc > Random Data > Normal. 3. Generate 30 rows of data. 4. Store results in C1-C3. 5. Enter a mean of 0 and a standard deviation of 1 and click OK. 6. Choose Graph > Character Graphs > Histogram and enter C1-C3 in the variable box and click OK. We will not give the data or any of the three histograms that we will get. These histograms are just lines containing *’s. If we need actual histograms, in step 6 use Graph > Histogram and enter C1 in the graph variable box and click OK If we wish to generate descriptive statistics, then 7. Choose Stat > Basic Statistics > Display Descriptive statistics. . . , enter C1-C3 in the variable box, and click OK. If we would like to see the mean for the three samples, 8. Choose Calc > Row Statistics, then click Mean and in the Input variables type C1-C3. In Store Result in: C4 and Click OK. To see the histogram of these averages, follow step 6 with C4 in the graph variable box. Using a similar procedure, one could generate samples from normal distributions with different means and standard deviations, as well as from other distributions.

4.6.2 SPSS Examples If we have the full version of SPSS, we can write code that can be used to simulate a sampling distribution with different values of p. However, with the student version, it is not easy to simulate. Therefore, we will not give SPSS examples in this chapter.

4.6.3 SAS Examples Example 4.6.2 Generate 50,000 observations from a normal distribution with mean 30 and standard deviation 8. Obtain summary statistics for these data and draw a graph.

220 CHAPTER 4 Sampling Distributions

Solution We could use the following program. title ’50000 Obs Sample from a Normal Distribution’; title2 ’with Mean=30 and Standard Deviation=8’; data normaldat; do n=1 to 50000; X=8*rannor(55)+ 30; output; end; run; proc univariate data=normaldat; var x; run; proc chart; vbar x / midpoints=6 to 54 by 2; format x msd.; run; In the foregoing program, rannor(55), the number 55 is just a seed number to obtain the same series of random numbers each time we run the program. If we use ‘0’, each time we run the program we will get a different set of random numbers. We will not give the output.

Example 4.6.3 From an exponential distribution, draw 10,000 samples, each sample of size 15. Compute the mean of each sample and draw a chart for the means. This will be an approximate sampling distribution of X for a ﬁxed sample of size 15.

Solution Use the following program. title ’10000 Sample Means with 15 Obs per Sample’; title2 ’Drawn from an Exponential Distribution’; data sample15; do Sample=1 to 10000; do n=1 to 15; X=ranexp(3); output; end; end;

Projects for Chapter 4 221

proc means data=sample 15 noprint; output out=mean 15 mean=Mean; var x; by sample; run; proc chart data=mean 15; vbar mean/axis=1800 midpoints=0.10 to 2.05 by .1; run; proc univariate data=mean4 noextrobs=0 normal mu0=1; var mean; run; This will produce an approximate sampling distribution of X. We will not give the output.

PROJECTS FOR CHAPTER 4 4A. A Method to Obtain Random Samples from Different Distributions Most of the statistical software packages contain a random number generator that produces approximations to random numbers from the uniform distribution U [0, 1]. To simulate the observation of any other continuous random variables, we can start with uniform random numbers and associate these to the distribution we want to simulate. For example, suppose we wish to simulate an observation from the exponential distribution F (x) = 1 − e−0.5x ,

0 < x < ∞.

First produce the value of y from the uniform distribution. Then solve for x from the equation y = F (x) = 1 − e−0.5x .

So x = [− ln (1 − y)] /0.5 is the corresponding value of the exponential random variable. For instance, if y = 0.67, then x = [− ln (1 − y)] /0.5 = 2.2173. If we wish to simulate a sample from the distribution F from the different values of y obtained from the uniform distribution, the procedure is repeated for each new observation x. (a) Simulate 10 observations of a random variable having exponential distribution with mean and standard deviation both equal to 2. (b) Select 1500 random samples of size n = 10 measurements from a population with an exponential distribution with mean and standard deviation both equal to 2. Calculate sample mean for each of these 1500 samples and draw a relative frequency histogram. Based on Theorems 4.1.1 and 4.4.1, what can you conclude? It should be noted that in general, if Y ∼ U (0, 1) random variable, then we can show that X = − lnY λ will give an exponential random variable with parameter λ. Uniform random variables could also

222 CHAPTER 4 Sampling Distributions

be used to generate random variables from other distributions. For example, let Ui s be iid U [0, 1] random variables. Then, X = −2

ν

2 , ln (Ui ) ∼ χ2ν

i=1

and Y = −β

α

ln (Ui ) ∼ Gamma (α, β) .

i=1

Of course, these transformations are useful only when ν and α are integers. More efﬁcient methods, such as MCMC methods, are discussed in Chapter 13.

4B. Simulation Experiments When the derivation via probability rules is too difﬁcult or complicated to be carried out, one can use simulation experiments to obtain information about a statistic’s sampling distribution. The following characteristics of the experiment must be speciﬁed: (i) The population distribution (normal with μ = 10 and σ = 2, exponential with λ = 5, etc.) (ii) The sample size n and the statistic of interest (X, S, etc.) (iii) The number of replications k (such as k = 300) Then, using a computer program, obtain k different random samples, each of size n, from the designated population distribution. Calculate the value of the statistic for each of the k replications. Construct a histogram for this k statistic. This histogram gives the approximate sampling distribution of the statistic. The larger the value of k, the better will be the approximation. (a) For your simulation study, use the population distribution as normal with μ = 3.4 and σ = 1.2. For n = 8 perform k = 500 replications and draw a histogram for values of the sample means. Repeat the experiment with n = 15, n = 25, and n = 35 and draw the histograms. Based on this exercise, you will be able to intuitively verify the result that X based on a large n tends to be closer to μ than does X based on a small n. (b) Repeat the experiment of part (a) with different values of k, such as k = 200, k = 750, and k = 1000. (c) Repeat the simulation study with different distributions such as exponential distribution.

4C. A Test for Normality Many statistical procedures require that the population be at least approximately normal. Therefore, a procedure is needed for checking that the sampled data could have come from a normal distribution. There are many procedures, such as the normal-score plot, or Lilliefors test for normality, available in statistics for this purpose. We will describe the normal-score plot, which is an effective way to detect deviations from normality. The normal scores consist of values of z that divide the axes into equal probability intervals. For a sample of size 4, the normal scores are −z0.20 = −0.84, −z0.40 = −0.25, z0.40 = −0.25, and z0.20 = 0.84.

Projects for Chapter 4 223

STEPS TO CONSTRUCT A NORMAL PLOT 1. Rearrange the n data points in ascending order. 2. Obtain the n normal scores. 3. Plot the kth largest observation, versus the k th normal score, for all k . 4. If the data were from a standard normal distribution, the plot would resemble a 45 degree line through the origin. 5. If the observations were from normal (but not from standard normal), the pattern should still be a straight line. However, the line need not pass through the origin or have a slope 1.

In applications, a minimum of 15 to 20 observations is needed to reach a more accurate conclusion.

EXERCISES 1.

For different observations, construct normal plots and check for normality of the corresponding populations.

2.

Using software (such as Minitab), generate 15 observations each from the following distributions: (a) Normal (2, 4), (b) Uniform (0, 1), (c) Gamma (2, 4), and (d) Exponential (2). For each of these data sets, draw a probability plot and note the geometry of the plots.

This page intentionally left blank

Chapter

5

Point Estimation Objective: In this chapter we study some statistical methods to ﬁnd point estimators of population parameters and study their properties. 5.1 Introduction 226 5.2 The Method of Moments 227 5.3 The Method of Maximum Likelihood 235 5.4 Some Desirable Properties of Point Estimators 246 5.5 Other Desirable Properties of a Point Estimator 266 5.6 Chapter Summary 282 5.7 Computer Examples 283 Projects for Chapter 5 285

C. R. Rao (Source: http:www.science.psu.edu/alert/Rao6-2007.htm)

Calyampudi Radhakrishna (C. R.) Rao (1920–) is a contemporary statistician whose work has inﬂuenced not just statistics, but such diverse ﬁelds as anthropology, biometry, demography, economics, Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

225

226 CHAPTER 5 Point Estimation

genetics, geology, and medicine. Several statistical terms and equations are named after Rao. He has worked with many other famous statisticians such as Blackwell, Fisher, and Neyman and has had dozens of theorems named after him. Rao earned an M.A. in mathematics and another M.A. in statistics, both in India, and earned his Ph.D. and Sc.D. at Cambridge University. The following was stated in the Preface to the 1991 special issue of the Journal of Quantitative Economics in Rao’s honor: “Dr. Rao is a very distinguished scientist and a highly eminent statistician of our time. His contributions to statistical theory and applications are well known, and many of his results, which bear his name, are included in the curriculum of courses in statistics at bachelor’s and master’s level all over the world. He is an inspiring teacher and has guided the research work of numerous students in all areas of statistics. His early work had greatly inﬂuenced the course of statistical research during the last four decades. One of the purposes of this special issue is to recognize Dr. Rao’s own contributions to econometrics and acknowledge his major role in the development of econometric research in India.” The importance of statistics can be summarized in Rao’s own words: “If there is a problem to be solved, seek statistical advice instead of appointing a committee of experts. Statistics can throw more light than the collective wisdom of the articulate few.”

5.1 INTRODUCTION In statistical analysis, point estimation of population parameters plays a very signiﬁcant role. In studying a real-world phenomenon we begin with a random sample of size n taken from the totality of a population. The initial step in statistically analyzing these data is to be able to identify the probability distribution that characterizes this information. Because the parameters of a distribution are its deﬁning characteristics, it becomes necessary to know the parameters. In the present chapter, we assume that the form of the population distribution is known (binomial, normal, etc.) but the parameters of the distribution (p for a binomial; μ and σ 2 for a normal, etc.) are unknown. We shall estimate these parameters using the data from our random sample. It is extremely important to have the best possible estimate of the population parameter(s). Having such estimates will lead to a better and more accurate statistical analysis. For example, in the area of phosphate mining in Florida, we may be interested in estimating the average radioactivity from both uranium and radium in a clay settling area of a mining site. Suppose that a random sample of 10 such sites resulted in a sample average of 40 pCi/g (picocuries/gram) of radioactivity. We may use this value as an estimate of the average radioactivity for all of the settling areas of mining sites in Florida. Because many Florida crops are grown on clay settling areas, this type of estimate is important for accessing the radioactivity-associated risks that are due to eating food from the crops grown on these clay settling areas. We will now introduce some of the more useful statistical point estimation methods, discuss their properties, and illustrate their usefulness with a number of applications. The importance of point estimates lies in the fact that many statistical formulas are based on them. For example, the point estimates of mean and standard deviation are used in the calculation of conﬁdence intervals and in many formulas for hypothesis testing. These topics are covered in subsequent chapters. Also, in most applied problems, a certain numerical characteristic of the physical phenomenon may be of interest; however, its value may not be observable directly. Instead, suppose it is possible to observe one or more random variables, the distribution of which depends on the characteristic of interest. Our

5.2 The Method of Moments 227

objective will be to develop methods that use the observed values of random variables (sample data) in order to gain information about the unknown and unobservable characteristic of the population. Let X1 , . . . , Xn be independent and identically distributed (iid) random variables (in statistical language, a random sample) with a pdf or pf f (x, θ1 , . . . θl ), where θ1 , . . . , θl are the unknown population parameters (characteristics of interest). For example, a normal pdf has parameters μ (the mean) and σ 2 (the variance). The actual values of these parameters are not known. The problem in point estimation is to determine statistics gi (X1 , . . . , Xn ), i = 1, . . . , l, which can be used to estimate the value of each of the parameters—that is, to assign an appropriate value for the parameters θ = (θ1 , . . . , θl ) based on observed sample data from the population. These statistics are called estimators for the parameters, and the values calculated from these statistics using particular sample data values are called estimates of the parameters. Estimators of θi are denoted by θˆ i , where θˆ i = gi (X1 , . . . , Xn ), i = 1, . . . , l. Observe that the estimators are random variables. As a result, an estimator has a distribution (which we called the sampling distribution in Chapter 4). When we actually run the experiment and observe the data, let the observed values of the random variables be X1 , . . . , Xn be x1 , . . . , xn ; then, θˆ (X1 , . . . , Xn ) is an estimator, and its value θˆ (x1 , . . . , xn ) is an estimate. For example, in case of the normal distribution, the parameters of interest are θ1 = μ, and θ2 = σ 2 , that is, θ = (μ, σ 2 ). If the estimators of μ and σ 2 are X = (1/n) ni=1 Xi and S 2 = (1/n − 1) ni=1 (Xi − X)2 respectively, then, the corresponding n estimates are x = (1/n) i=1 xi and s2 = (1/n − 1) ni=1 (xi − x)2 , the mean and variance corresponding to the particular observed sample values. In this book, we use capital letters such as X and S 2 to represent the estimators, and lowercase letters such as x and s2 to represent the estimates. There are many methods available for estimating the true value(s) of the parameter(s) of interest. Three of the more popular methods of estimation are the method of moments, the method of maximum likelihood, and Bayes’ method. A very popular procedure among econometricians to ﬁnd a point estimator is the generalized method of moments. In this chapter we study only the method of moments and the method of maximum likelihood for obtaining point estimators and some of their desirable properties. In Chapter 11, we shall discuss Bayes’ method of estimation. There are many criteria for choosing a desired point estimator. Heuristically, some of them can be explained as follows (detailed coverage is given in Sections 5.2 through 5.5). An estimator, θˆ , is unbiased if the mean of its sampling distribution is the parameter θ. The bias of θˆ is given by B = E(θˆ ) − θ. The estimator satisﬁes the consistency property if the sample estimator has a high probability of being close to the population value θ for a large sample size. The concept of efﬁciency is based on comparing variances of the different unbiased estimators. If there are two unbiased estimators, it is desirable to have the one with the smaller variance. The estimator has the sufﬁciency property if it fully uses all the sample information. Minimal sufﬁcient statistics are those that are sufﬁcient for the parameter and are functions of every other set of sufﬁcient statistics for those same parameters. A method due to Lehmann and Scheffé can be used to ﬁnd a minimal sufﬁcient statistic.

5.2 THE METHOD OF MOMENTS How do we ﬁnd a good estimator with desirable properties? One of the oldest methods for ﬁnding point estimators is the method of moments. This is a very simple procedure for ﬁnding an estimator for one or more population parameters. Let μk = E[Xk ] be the kth moment about the origin of a

228 CHAPTER 5 Point Estimation random variable X, whenever it exists. Let mk = (1/n) ni=1 Xik be the corresponding kth sample moment. Then, the estimator of μk by the method of moments is mk . The method of moments is based on matching the sample moments with the corresponding population (distribution) moments and is founded on the assumption that sample moments should provide good estimates of the corresponding population moments. Because the population moments μk = hk (θ1 , θ2 , . . . , θl ) are often functions of the population parameters, we can equate corresponding population and sample moments and solve for these parameters in terms of the moments.

METHOD OF MOMENTS Choose as estimates those values of the population parameters that are solutions of the equations μk = mk , k = 1, 2, . . . , l. Here μk is a function of the population parameters.

For example, the ﬁrst population moment is μ1 = E(X), and the ﬁrst sample moment is X = n X. If k = 2, then the second population and i=1 Xi /n. Hence, the moment estimator of μ1 is 2 sample moments are μ2 = E(X ) and m2 = (1/n) ni=1 Xi2 , respectively. Basically, we can use the following procedure in ﬁnding point estimators of the population parameters using the method of moments.

THE METHOD OF MOMENTS PROCEDURE Suppose there are l parameters to be estimated, say θ = (θ1 , . . . , θl ). 1. Find l population moments, μk , k = 1, 2, . . . , l. μk will contain one or more parameters θ1 , . . . , θl . 2. Find the corresponding l sample moments, mk , k = 1, 2, . . . , l. The number of sample moments should equal the number of parameters to be estimated. 3. From the system of equations, μk = mk , k = 1, 2, . . . , l, solve for the parameter θ = (θ1 , . . . , θl ); this will be a moment estimator of θˆ .

The following examples illustrate the method of moments for population parameter estimation.

Example 5.2.1 Let X1 , . . . , Xn be a random sample from a Bernoulli population with parameter p. (a) Find the moment estimator for p. (b) Tossing a coin 10 times and equating heads to value 1 and tails to value 0, we obtained the following values: 0

1

1

0

1

0

1

Obtain a moment estimate for p, the probability of success (head).

1

1

0

5.2 The Method of Moments 229

Solution

(a) For the Bernoulli random variable, μk = E[X] = p, so we can use m1 to estimate p. Thus, m1 = pˆ =

1 Xi . n n

i=1

Let Y=

n

Xi .

i=1

Then, the method of moments estimator for p is pˆ = Y /n. That is, the ratio of the total number of heads to the total number of tosses will be an estimate of the probability of success. (b) Note that this experiment results in Bernoulli random variables. Thus, using part (a) with Y = 6, we 6 = 0.6. get the moment estimate of p is pˆ = 10 We would use this value pˆ = 0.6, to answer any probabilistic questions for the given problem. For example, what is the probability of exactly obtaining 8 heads out of 10 tosses of this coin? This can be 10 obtained by using the binomial formula, with pˆ = 0.6, that is, P(X = 8) = (0.6)8 (0.4)10−8 . 8

In Example 5.2.1, we used the method of moments to ﬁnd a single parameter. We demonstrate in Example 5.2.2 how this method is used for estimating more than one parameter.

Example 5.2.2 Let X1 , . . . , Xn be a random sample from a gamma probability distribution with parameters α and β. Find moment estimators for the unknown parameters α and β.

Solution For the gamma distribution (see Section 3.2.5), E[X] = αβ

and

E X2 = αβ2 + α2 β2 .

Because there are two parameters, we need to find the first two moment estimators. Equating sample moments to distribution (theoretical) moments, we have 1 Xi = X = αβ, n n

i=1

1 2 Xi = αβ2 + α2 β2 . n n

and

i=1

# $ Solving for α and β we obtain the estimates as α = (x/β) and β = {(1/n) ni=1 xi2 − x2 }/x .

230 CHAPTER 5 Point Estimation

Therefore, the method of moments estimators for α and β are

αˆ =

X βˆ

and

βˆ =

n 1 X2 − X2 n i i=1

X

n

Xi − X

=

2

i=1

,

nX

which implies that 2

αˆ =

2

X X X = = n . n 2 2 βˆ 1 Xi − X Xi2 − X n i=1

i=1

Thus, we can use these values in the gamma pdf to answer questions concerning the probabilistic behavior of the r.v. X.

Example 5.2.3 Let the distribution of X be N(μ, σ 2 ). (a) For a given sample of size n, use the method of moments to estimate μ and σ 2 . (b) The following data (rounded to the third decimal digit) were generated using Minitab from a normal distribution with mean 2 and a standard deviation of 1.5. 3.163 1.883 3.252 3.716 −0.049 −0.653 0.057 2.987 4.098 1.670 1.396 2.332 1.838 3.024 2.706 0.231 3.830 3.349 −0.230 1.496 Obtain the method of moments estimates of the true mean and the true variance.

Solution (a) For the normal distribution, E(X) = μ, and because Var(X) = EX2 − μ2 , we have the second moment as E(X2 ) = σ 2 + μ2 . Equating sample moments to distribution moments we have n 1 Xi = μ1 = μ n i=1

and μ2 =

n 1 2 X i = σ 2 + μ2 . n i=1

5.2 The Method of Moments 231

Solving for μ and σ 2 , we obtain the moment estimators as μ ˆ =X and σˆ 2 =

n n 2 1 2 1 2 Xi − X . Xi − X = n n i=1

i=1

(b) Because we know that the estimator of the mean is μ ˆ = X and the estimator of the variance is σˆ 2 = n 2 ˆ = 2.005, and σˆ 2 = 6.12−(2.005)2 = 2.1. (1/n) i=1 Xi2 −X , from the data the estimates are μ Notice that the true mean is 2 and the true variance is 2.25, which we used to simulate the data.

In general, using the population pdf we evaluate the lower order moments, ﬁnding expressions for the moments in terms of the corresponding parameters. Once we have population (theoretical) moments, we equate them to the corresponding sample moments to obtain the moment estimators.

Example 5.2.4 Let X1 , . . . , Xn be a random sample from a uniform distribution on the interval [a, b]. Obtain method of moment estimators for a and b.

Solution Here, a and b are treated as parameters. That is, we only know that the sample comes from a uniform distribution on some interval, but we do not know from which interval. Our interest is to estimate this interval. The pdf of a uniform distribution is ⎧ ⎨ 1 , a≤x≤b f (x) = b − a ⎩ 0, otherwise. Hence, the first two population moments are b μ1 = E(X) = a

x a+b dx = b−a 2

and

μ2 = E(X2 ) =

b a

x2 a2 + ab + b2 dx = . b−a 3

The corresponding sample moments are μ ˆ1 = X

and

μ ˆ2 =

n 1 2 Xi . n i=1

Equating the first two sample moments to the corresponding population moments, we have μ ˆ1 =

a+b 2

and

μ ˆ2 =

a2 + ab + b2 3

232 CHAPTER 5 Point Estimation

which, solving for a and b, results in the moment estimators of a and b, aˆ = μ ˆ1 −

9 3 μ ˆ2 −μ ˆ 21

bˆ = μ ˆ1 +

and

9 3 μ ˆ2 −μ ˆ 21 .

In Example 5.2.4, if a = −b, that is, X1 , . . . , Xn is a random sample from a uniform distribution on the interval (−b, b), the problem reduces to a one-parameter estimation problem. However, in this case E(Xi ) = 0, so the ﬁrst moment cannot be used to estimate b. It becomes necessary to use the second moment. For the derivation, see Exercise 5.2.3. It is important to observe that the method of moments estimators need not be unique. The following is an example of the nonuniqueness of moment estimators.

Example 5.2.5 Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ > 0. Show that both 2 (1/n) ni=1 Xi and (1/n) ni=1 Xi2 − (1/n) ni=1 Xi are moment estimators of λ.

Solution We know that E(X) = λ, from which we have a moment estimator of λ as (1/n) we have Var(X) = λ, equating the second moments, we can see that

n

i=1 Xi . Also, because

λ = E(X2 ) − (EX)2 , so that n 1 2 Xi − λˆ = n

i=1

2 n 1 Xi . n i=1

Thus, λˆ =

n 1 Xi n i=1

and λˆ =

n 1 2 Xi − n i=1

2 n 1 Xi . n i=1

Both are moment estimators of λ. Thus, the moment estimators may not be unique. We generally choose X as an estimator of λ, for its simplicity.

5.2 The Method of Moments 233

It is important to note that, in general, we have as many moment conditions as the parameters. In Example 5.2.5, we have more moment conditions than parameters, because both the mean and variance of Poisson random variables are the same. Given a sample, this results in two different estimates of a single parameter. One of the questions could be, can these two estimators be combined in some optimal way? This is done by the so-called generalized method of moments (GMM). We will not deal with this topic. As we have seen, the method of moments ﬁnds estimators of unknown parameters by equating the corresponding sample and population moments. This method often provides estimators when other methods fail to do so or when estimators are harder to obtain, as in the case of a gamma distribution. Compared to other methods, method of moments estimators are easy to compute and have some desirable properties that we will discuss in ensuing sections. The drawback is that they are usually not the “best estimators” (to be deﬁned later) available and sometimes may even be meaningless.

EXERCISES 5.2 5.2.1.

Let X1 , . . . , Xn be a random sample of size n from the geometric distribution for which p is the probability of success. (a) Use the method of moments to ﬁnd a point estimator for p. (b) Use the following data (simulated from geometric distribution) to ﬁnd the moment estimator for p: 2 4

5 34

7 19

43 21

18 23

19 6

16 21

11 7

22 12

How will you use this information? [The pdf of a geometric distribution is f (x) = p(1 − p)x−1 , for x = 1, 2, . . . . Also μ = 1/p.] 5.2.2.

Let X1 , . . . , Xn be a random sample of size n from the exponential distribution whose pdf (by taking θ = 1/β in Deﬁnition 2.3.7) is f (x, θ) =

⎧ ⎨θe−θx ,

x≥0

⎩ 0,

x < 0.

(a) Use the method of moments to ﬁnd a point estimator for θ. (b) The following data represent the time intervals between the emissions of beta particles. 0.9 0.1 0.1 0.5 0.4

0.1 0.1 0.5 3.0 0.5

0.1 0.1 0.4 1.0 0.8

0.8 2.3 0.6 0.5 0.1

0.9 0.8 0.2 0.2 0.1

0.1 0.3 0.4 2.0 1.7

0.1 0.2 0.2 1.7 0.1

0.7 0.1 0.1 0.1 0.2

1.0 1.0 0.8 0.3 0.3

0.2 0.9 0.2 0.1 0.1

234 CHAPTER 5 Point Estimation

Assuming the data follow an exponential distribution, obtain a moment estimate for the parameter θ. Interpret. 5.2.3.

Let X1 , . . . , Xn be a random sample from a uniform distribution on the interval (θ − 1, θ + 1). (a) Find a moment estimator for θ. (b) Use the following data to obtain a moment estimate for θ: 11.72

5.2.4.

12.81

12.09

13.47

12.37

The probability density of a one-parameter Weibull distribution is given by 2αxe−αx , 0, 2

f(x) =

x>0 otherwise.

(a) Using a random sample of size n, obtain a moment estimator for α. (b) Assuming that the following data are from a one-parameter Weibull population, 1.87 1.83

1.60 0.64

2.36 1.53

1.12 0.73

0.15 2.26

obtain a moment estimate of α. 5.2.5.

Let X1 , . . . , Xn be a random sample from the truncated exponential distribution with pdf f(x) =

e−(x−θ) , 0,

x≥θ otherwise.

Find the method of moments estimate of θ. 5.2.6.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f(x, α) =

1 + αx , 2

−1 ≤ x ≤ 1, and − 1 ≤ α ≤ 1.

Find the moment estimators for α. 5.2.7.

Let X1 , . . . , Xn be a random sample from a population with pdf f (x) =

⎧ ⎨ 2α2 ,

x≥α

⎩ 0,

otherwise.

x3

Find a method of moments estimator for α. 5.2.8.

Let X1 , . . . , Xn be a random sample from a negative binomial distribution with pmf p(x, r, p) =

x+r−1 x p (1 − p)x , 0 ≤ p ≤ 1, x = 0, 1, 2, . . . . r−1

5.3 The Method of Maximum Likelihood 235 # $ Find method of moments estimators for r and p. [Here E[X] = r(1 − p)/p and E X2 = r(1 − p)(r − rp + 1)/p2 .] 5.2.9.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f (x) =

(θ + 1) xθ ,

0 ≤ x ≤ 1; θ > −1

0,

otherwise.

Use the method of moments to obtain an estimator of θ. 5.2.10.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f (x) =

2β−2x , β2

0<x 0.

5.3 The Method of Maximum Likelihood 239

Hence, the likelihood function is n

xi

n 7 λxi e−λ λi=1 e−nλ L (λ) = . = n : xi ! i=1 xi ! i=1

Then, taking the natural logarithm, we have ln L(λ) =

n

xi ln λ − nλ −

i=1

n

ln (xi !)

i=1

and differentiating with respect to λ results in n

d ln L(λ) i=1 = dλ λ

xi −n

and n

d ln L(λ) i=1 = 0, implies dλ λ

xi − n = 0.

That is, n

λ=

xi

i=1

= x.

n

Hence, the MLE of λ is λˆ = X.

It can be veriﬁed that the second derivative is negative and, hence, we really have a maximum. Sometimes the method of derivatives cannot be used for ﬁnding the MLEs. For example, the likelihood is not differentiable in the range space. In this case, we need to make use of the special structures available in the speciﬁc situation to solve the problem. The following is one such case.

Example 5.3.4 Let X1 , . . . , Xn be a random sample from U(0, θ), θ > 0. Find the MLE of θ.

Solution Note that the pdf of the uniform distribution is ⎧ ⎨1 , f (x) = θ ⎩ 0,

0≤x≤θ otherwise.

240 CHAPTER 5 Point Estimation

L ()

X(n )

■ FIGURE 5.1 Likelihood function for uniform probability distribution.

Hence, the likelihood function is given by ⎧ ⎨ 1 , L (θ, x1 , x2 , . . . , xn ) = θ n ⎩ 0,

0 ≤ x1 , x2 , . . . , xn ≤ θ otherwise.

When θ ≥ max(xi ), the likelihood is (1/θ n ), which is positive and decreasing as a function of θ (for fixed n). However, for θ < max(xi ) the likelihood drops to 0, creating a discontinuity at the point max(xi ) (this is the minimum value of θ that can be chosen which still satisfies the condition 0 ≤ xi ≤ θ), and Figure 5.1 shows that the maximum occurs at this point. Hence, we will not be able to find the derivative. Thus, the MLE is the largest order statistic, θˆ = max (Xi ) = X(n) .

In the previous example, because E(X) = (θ/2), we can see that θ = 2E(X). Hence, the method of moments estimator for θ is θˆ = 2X. Sometimes the method of moments estimator can give meaningless results. To see this, suppose we observe values 3, 5, 6, and 18 from a U(0, θ) distribution. Clearly, the maximum likelihood estimate of θ is 18, whereas the method of moments estimate is 16, which is not quite acceptable, because we have already observed a value of 18. As mentioned earlier, if the unknown parameter θ represents a vector of parameters, say θ = (θ1 , . . . , θl ), then the MLEs can be obtained from solutions of the system of equations ∂ ln L (θ1 , . . . , θn ) = 0, ∂θ

for

i = 1, . . . , l.

These are called the maximum likelihood equations and the solutions are denoted by (θˆ 1 , . . . , θˆ l ).

Example 5.3.5

Let X1 , . . . , Xn be N μ, σ 2 . (a) If μ is unknown and σ 2 = σ02 is known, ﬁnd the MLE for μ. (b) If μ = μ0 is known and σ 2 is unknown, ﬁnd the MLE for σ 2 . (c) If μ and σ 2 are both unknown, ﬁnd the MLE for θ = μ, σ 2 .

5.3 The Method of Maximum Likelihood 241

Solution In order to avoid notational confusion when taking the derivative, let θ = σ 2 . Then, the likelihood function is ⎞ ⎛ n 2 (x − μ) i ⎟ ⎜ ⎟ ⎜ i=1 L (μ, θ) = (2πθ)−n/2 exp⎜− ⎟ ⎠ ⎝ 2θ or n

(xi − μ)2 n n i=1 ln L (μ, θ) = − ln (2π) − ln θ − . 2 2 2θ (a) When θ = θ0 = σ02 is known, the problem reduces to estimating the only one parameter, μ. Differentiating the log-likelihood function with respect to μ,

∂ ln L (μ, θ0 ) = ∂μ

2

n

(xi − μ)

i=1

.

2θ0

Setting the derivative equal to zero and solving for μ, n

(xi − μ) = 0.

i=1

From this, n

xi = nμ

μ = x.

or

i=1

Thus, we get μ ˆ = X. (b) When μ = μ0 is known, the problem reduces to estimating the only one parameter, σ 2 = θ. Differentiating the log-likelihood function with respect to θ, n

(xi − μ)2 −n i=1 ∂ ln L (μ, θ) = + . ∂θ 2θ 2θ 2 Setting the derivative equal to zero and solving for θ, we get n i=1 θˆ = σˆ 2 =

(Xi − μ0 )2 n

.

242 CHAPTER 5 Point Estimation

(c) When both μ and θ are unknown, we need to differentiate with respect to both μ and θ individually:

∂ ln L (μ, θ) = ∂μ

2

n

(xi − μ)

i=1

2θ

and n

(xi − μ)2 −n i=1 ∂ ln L (μ, θ) = + . ∂θ 2θ 2θ 2 Setting the derivatives equal to zero and solving simultaneously, we obtain μ ˆ = X, n

Xi − X

σˆ 2 = θˆ =

i=1

n

2 = S 2 .

Note that in (a) and (c), the estimates for μ are the same; however, in (b) and (c), the estimates for σ 2 are different.

At times, the maximum likelihood estimators may be hard to calculate. It may be necessary to use numerical methods to approximate values of the estimate. The following example gives one such case.

Example 5.3.6 Let X1 , . . . , Xn be a random sample from a population with gamma distribution and parameters α and β. Find MLEs for the unknown parameters α and β.

Solution The pdf for the gamma distribution is given by ⎧ ⎨ xα−1 e−x/β (α)βα , f (x) = ⎩ 0,

x > 0,

α > 0,

β>0

otherwise.

The likelihood function is given by n

n − xi /β 7 1 xiα−1 e i=1 . L = L(α, β) = ( (α) βα )n i=1

Taking the logarithms gives ln L = −n ln (α) − nα ln β + (α − 1)

n i=1

ln xi −

n x . β i=1

5.3 The Method of Maximum Likelihood 243

Now taking the partial derivatives with respect to α and β and setting both equal to zero, we have ∂ (α) ln L = −n − n ln β + ln xi = 0 ∂α (α) n

i=1

∂ α xi ln L = −n + = 0. ∂β β β2 n

i=1

Solving the second one to get β in terms of α, we have β=

x . α

Substituting this β in the first equation, we have to solve (α) x − n ln + ln xi = 0 (α) α n

−n

i=1

for α > 0. There is no closed-form solution for α and β. In this case, one can use numerical methods such as the Newton--Raphson method to solve for α, and then use this value to find β.

There are many references available on the Web. Explaining the Newton–Raphson method, for instance, http://web.as.uky.edu/statistics/users/viele/sta601s08/nummax.pdf gives the algorithm for the gamma distribution. In only a few cases are we able to obtain a simple form for the maximum likelihood equation that can be solved by setting the ﬁrst derivative to zero. Often we cannot write an equation that can be differentiated to ﬁnd the MLE parameter estimates. This is especially true in the situation where the model is complex and involves many parameters. Evaluating the likelihood exhaustively for all values of the parameters becomes almost impossible, even with modern computers. This is why so-called optimization algorithms have become indispensable to statisticians. The purpose of an optimization algorithm is to ﬁnd as fast as possible the set of parameter values that make the observed data most likely. There are many such algorithms available. We describe the Newton–Raphson method in Project 5F, and another powerful algorithm, known as the EM algorithm, is given in Section 13.4. Sometimes, it may be necessary to estimate a function of a parameter. The following invariance property of maximum likelihood estimators is very useful in those cases. Theorem 5.3.1 Let h(θ) be a one-to-one function of θ. If θˆ = (θˆ 1 , . . . , θˆ l ) is the MLE of θ = (θ1 , . . . , θl ), then the MLE of a function h(θ) = (h1 (θ), . . . , hk (θ)) of these parameters is h(θˆ ) = (h1 (θˆ ), . . . , hk (θˆ )) for 1 ≤ k ≤ l. As a consequence of the invariance property, in Example 5.3.5, we can obtain the estimator of the 9 √ 2 true standard deviation as σˆ = σˆ 2 = (1/n) ni=1 Xi − X .

244 CHAPTER 5 Point Estimation

It is also known that, under very general conditions on the joint distribution of the sample and for a large sample size n, the MLE θˆ is approximately the minimum variance unbiased estimator (this concept is introduced in the next section) of θ.

EXERCISES 5.3 5.3.1.

Let X1 , . . . , Xn be a random sample recorded as heads or tails resulting from tossing a coin n times with unknown probability p of heads. Find the MLE pˆ of p. Also using the invariance property, obtain an MLE for q = 1 − p. How would you use the results you have obtained?

5.3.2.

Suppose X1 , . . . , Xn are a random sample from an exponential distribution with parameter θ. Find the MLE of θˆ . Also using the invariance property, obtain an MLE for the variance.

5.3.3.

Let X be a random variable representing the time between successive arrivals at a checkout counter in a supermarket. The values of X in minutes (rounded to the nearest minute) are 1 12

2 7

3 3

7 2

11 11

4 7

13 2

Assume that the pdf of X is f (x) = (1/θ)e−(x/θ) . Use these data to ﬁnd MLE θˆ . How can you use this estimate you have just derived? 5.3.4.

Let X1 , . . . , Xn be a random sample from the truncated exponential distribution with pdf f (x) =

⎧ ⎨e−(x−θ) , ⎩

0,

x≥θ otherwise.

Show that the MLE of θ is min(Xi ). 5.3.5.

The pdf of a random variable X is given by f (x) =

⎧ ⎨ 2x2 e−x2 /α2 , ⎩

x>0

α

0,

otherwise.

Using a random sample of size n, obtain MLE αˆ for α. 5.3.6.

The pdf of a random variable X is given by 1 exp αn − eα , P (X = n) = n!

n = 0, 1, 2, . . . .

Using a random sample of size n, obtain MLE αˆ for α. 5.3.7.

Let X1 , . . . , Xn be a random sample from a two-parameter Weibull distribution with pdf f (x) =

Find the MLEs of α and β.

⎧ α ⎨ αα xα−1 e−(x/β) , β ⎩

0,

x≥0 otherwise.

5.3 The Method of Maximum Likelihood 245

5.3.8.

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

⎧ 2 ⎨ αx e−x /2α , ⎩

x>0

0,

otherwise.

Find the MLEs of α. 5.3.9.

Let X1 , . . . , Xn be a random sample from a two-parameter exponential population with density 1 − (x−υ) θ , e θ

f (x, θ, υ) =

for

x ≥ υ,

θ > 0.

Find MLEs for θ and υ when both are unknown. 5.3.10.

Let X1 , . . . , Xn be a random sample from the shifted exponential distribution with pdf f (x) =

⎧ ⎨λe−λ(x−θ) , ⎩

x≥θ

0,

otherwise.

Obtain the maximum likelihood estimators of θ and λ. 5.3.11.

Let X1 , . . . , Xn be a random sample on [0, 1] with pdf f (x) =

(2θ) [x(1 − x)]θ−1 , (θ)2

θ > 0.

What equation does the maximum likelihood estimate of θ satisfy? 5.3.12.

Let X1 , . . . , Xn be a random sample with pdf f (x) =

⎧ ⎨(α + 1)xα ,

0≤x≤1

⎩

otherwise.

0,

Find the MLE of α. 5.3.13.

Let X1 , . . . , Xn be a random sample from a uniform distribution with pdf f (x) =

⎧ 1 ⎨ 3θ+2 , ⎩

0,

0 ≤ x ≤ 3θ + 2 otherwise.

Obtain the MLE of θ. 5.3.14.

Let X1 , . . . , Xn be a random sample from a Cauchy distribution with pdf f (x) =

Find the MLE for β.

1 $, # π 1 + (x − β)2

−∞ < x < ∞.

246 CHAPTER 5 Point Estimation

5.3.15.

The following data represent the amount of leakage of a ﬂuorescent dye from the bloodstream into the eye in patients with abnormal retinas: 1.6 1.8

1.4 6.3

1.2 2.4

2.2 2.3

1.8 18.9

1.7 22.8

Assuming that these data come from a normal distribution, ﬁnd the maximum likelihood estimate of (μ, σ). 5.3.16.

Let X1 , . . . , Xn be a random sample from a population with gamma distribution and parameters α and β. Show that the MLE of μ = αβ is the sample mean μ ˆ = X.

5.3.17.

The lifetimes X of a certain brand of component used in a machine can be modeled as a random variable with pdf f (x) = (1/θ) e−(x/θ) . The reliability R(x) of the component is deﬁned as R(x) = 1 − F (x). Suppose X1 , X2 , . . . , Xn are the lifetimes of n components randomly selected and tested. Find the MLE of R(x).

5.3.18.

Using the method explained in Project 4A, generate 20 observations of a random variable having an exponential distribution with mean and standard deviation both equal to 2. What is the maximum likelihood estimate of the population mean? How much is the observed error?

5.3.19.

Let X1 , . . . , Xn be a random sample from a Pareto distribution (named after the economist Vilfredo Pareto) with shape parameter a. The density function is given by ⎧ a ⎨ , f (x) = xa+1 ⎩ 0,

x≥1 otherwise.

(The Pareto distribution is a skewed, heavy-tailed distribution. Sometimes it is used to model the distribution of incomes.) Show that the maximum likelihood estimator of a is aˆ = n

n

.

ln (Xi )

i=1

5.3.20.

Let X1 , . . . , Xn be a random sample from N (θ, θ), 0 < θ < ∞. Find the maximum likelihood estimate of θ.

5.4 SOME DESIRABLE PROPERTIES OF POINT ESTIMATORS Two different methods of ﬁnding estimators for population parameters have been introduced in the preceding sections. We have seen that it is possible to have several estimators for the same parameter. For a practitioner of statistics, an important question is going to be which of many available sample statistics, such as mean, median, smallest observation, or largest observation, should be chosen to represent all of the sample? Should we use the method of moments estimator, the maximum

5.4 Some Desirable Properties of Point Estimators 247

likelihood estimator, or an estimator obtained through some other method of least squares (we will see this method in Chapter 8)? Now we introduce some common ways to distinguish between them by looking at some desirable properties of these estimators.

5.4.1 Unbiased Estimators It is desirable to have the property that the expected value of an estimator of a parameter is equal to the true value of the parameter. Such estimators are called unbiased estimators. Deﬁnition 5.4.1 A point estimator θˆ is called an unbiased estimator of the parameter θ if E(θˆ ) = θ for all possible values of θ. Otherwise θˆ is said to be biased. Furthermore, the bias of θˆ is given by B = E(θˆ ) − θ.

Note that the bias is nothing but the expected value of the (random) error, E(θˆ − θ). Thus, the estimator is unbiased if the bias is 0 for all values of θ. The bias occurs when a sample does not accurately represent the population from which the sample is taken. It is important to observe that in order to check whether θˆ is unbiased, it is not necessary to know the value of the true parameter. Instead, one can use the sampling distribution of θˆ . We demonstrate the basic procedure through the following example.

Example 5.4.1 Let X1 , . . . , Xn be a random sample from a Bernoulli population with parameter p. Show that the method of moments estimator is also an unbiased estimator.

Solution We can verify that the moment estimator of p is n

pˆ =

Xi

i=1

n

=

Y . n

Because for binomial random variables, E (Y ) = np, it follows that

1 1 Y = E (Y ) = · np = p. E pˆ = E n n n Hence, pˆ = Y /n is an unbiased estimator for p.

In fact, we have the following result, which states that the sample mean is always an unbiased estimator of the population mean. Theorem 5.4.1 The mean of a random sample X is an unbiased estimator of the population mean μ.

248 CHAPTER 5 Point Estimation

Proof. Let X1 , . . . , Xn be random variables with mean μ. Then, the sample mean is X = (1/n)

n

i=1 Xi .

1 1 EXi = · nμ = μ. n n n

EX =

i=1

Hence, X is an unbiased estimator of μ. How is this interpreted in practice? Suppose that a data set is collected with n numerical observations x1 , . . . , xn . The resulting sample mean may be either less than or greater than the true population mean, μ (remember, we do not know this value). If the sampling experiment was repeated many times, then the average of the estimates calculated over these repetitions of the sampling experiment will equal the true population mean. If we have to choose among several different estimators of a parameter θ, it is desirable to select one 2 that is unbiased. The following result states that the sample variance S 2 = (1/n − 1) ni=1 Xi − X is an unbiased estimator of the population variance σ 2 . This is one of the reasons why in the deﬁnition of the sample variance, instead of dividing by n, we divide by (n − 1). Theorem 5.4.2 If S 2 is the variance of a random sample from an inﬁnite population with ﬁnite variance σ 2 , then S 2 is an unbiased estimator for σ 2 . Proof. Let X1 , . . . , Xn be iid random variables with variance σ 2 < ∞. We have

& % n n / 02 1 1 ¯ 2= (Xi − μ) − X − μ E E Xi − X n−1 n−1 i=1 & % i=1 n / 02 1 2 . E {Xi − μ} − nE X − μ = n−1

E S2 =

i=1

2 Because E{(Xi − μ)2 } = σ 2 and E{ X − μ } = σ 2 /n, it follows that E S2 =

% n & 1 σ2 2 = σ2. σ −n n−1 n i=1

Hence, S 2 is an unbiased estimator of σ 2 .

It is important to observe the following: 1. S 2 is not an unbiased estimator of the variance of a ﬁnite population. 2. Unbiasedness may not be retained under functional transformations, that is; if θˆ is an unbiased estimator of θ, it does not follow that f (θˆ ) is an unbiased estimator of f (θ). 3. Maximum likelihood estimators or moment estimators are not, in general, unbiased. 4. In many cases it is possible to alter a biased estimator by multiplying by an appropriate constant to obtain an unbiased estimator. The following example will show that unbiased estimators need not be unique.

5.4 Some Desirable Properties of Point Estimators 249

Example 5.4.2 Let X1 , . . . , Xn be a random sample from a population with ﬁnite mean μ. Show that the sample mean X and 13 X + 23 X1 are both unbiased estimators of μ.

Solution By Theorem 1, X is unbiased. Now E 13 X + 23 X1 = 13 μ + 23 μ = μ. Hence, 13 X + 23 X1 is also an unbiased estimator of μ.

How many unbiased estimators can we ﬁnd? In fact, the following example shows that if we have two unbiased estimators, there are inﬁnitely many unbiased estimators.

Example 5.4.3 Let θˆ 1 and θˆ 2 be two unbiased estimators of θ. Show that θˆ 3 = aθˆ 1 + (1 − a) θˆ 2 , 0 ≤ a ≤ 1 is an unbiased estimator of θ. Note that θˆ 3 is a convex combination of θˆ 1 and θˆ 2 . In addition, assume that θˆ 1 and θˆ 2 are independent, and Var(θˆ 1 ) = σ12 and Var(θˆ 2 ) = σ22 . How should the constant a be chosen in order to minimize the variance of θˆ 3 ?

Solution We are given that E(θˆ 1 ) = θ and E(θˆ 2 ) = θ. Therefore, E θˆ 3 = E aθˆ 1 + (1 − a) θˆ 2 = aEθˆ 1 + (1 − a) Eθˆ 2 = aθ + (1 − a) θ = θ. Hence θˆ 3 is unbiased. By independence, Var θˆ 3 = Var aθˆ 1 + (1 − a) θˆ 2 = a2 Var θˆ 1 + (1 − a)2 Var θˆ 2 = a2 σ12 + (1 − a)2 σ22 . To find the minimum, d Var θˆ 3 = 2aσ12 − 2(1 − a)σ22 = 0, da

250 CHAPTER 5 Point Estimation

gives us a=

σ22 σ12 + σ22

.

d 2 V (θˆ ) = 2σ 2 + 2σ 2 > 0, V (θˆ ) has a minimum at this value of a . Thus, if σ 2 = σ 2 , then Because da 3 3 2 1 2 1 2 a = 1/2.

Example 5.4.4 Let X1 , . . . , Xn be a random sample from a population with pdf ⎧ ⎨ 1 e−x/β , x>0 β f (x) = ⎩ 0, otherwise. Show that the method of moments estimator for the population parameter β is unbiased.

Solution From Section 5.2, we have seen that the method of moments estimator for β is the sample mean X, and the population mean is β. Because E(X) = μ = β, the method of moments estimator for the population parameter β is unbiased.

As we have seen, there can be many unbiased estimators of a parameter θ. Which one of these estimators can we choose? If we have to choose an unbiased estimator, it will be desirable to choose the one with the least variance. If an estimator is biased, then we should prefer the one with low bias as well as low variance. Generally, it is better to have an estimator that has low bias as well as low variance. This leads us to the following deﬁnition. Deﬁnition 5.4.2 The mean square error of the estimator θˆ , denoted by MSE(θˆ ), is deﬁned as 2 MSE θˆ = E θˆ − θ .

Through the following calculations, we will now show that the MSE is a measure that combines both bias and variance. 2 2 MSE θˆ = E θˆ − θ = E θˆ − E θˆ + E θˆ − θ ! 2 2 " = E θˆ − E θˆ + E θˆ − θ + 2 θˆ − E θˆ E θˆ − θ 2 2 + E E θˆ − θ + 2E θˆ − E θˆ E θˆ − θ = E θˆ − E θˆ 2 = Var θˆ + E θˆ − θ ,

5.4 Some Desirable Properties of Point Estimators 251 because letting B = E(θˆ ) − θ, we get MSE θˆ = Var θˆ + B2 .

B is called the bias of the estimator. Also, E(θˆ − E(θˆ ))(E(θˆ ) − θ) = 0. Because the bias is zero for unbiased estimators, it is clear that MSE(θˆ ) = Var(θˆ ). Mean square error measures, on average, how close an estimator comes to the true value of the parameter. Hence, this could be used as a criterion for determining when one estimator is “better” than another. However, in general, it is difﬁcult to ﬁnd θˆ to minimize MSE(θˆ ). For this reason, most of the time, we look only at unbiased estimators in order to minimize Var(θˆ ). This leads to the following deﬁnition. Deﬁnition 5.4.3 The unbiased estimator θˆ that minimizes the mean square error is called the minimum variance unbiased estimator (MVUE) of θ.

Example 5.4.5 Let X1 , X2 , X3 be a sample of size n = 3 from a distribution with unknown mean μ, −∞ < μ < ∞, where the variance σ 2 is a known positive number. Show that both θˆ 1 = X and θˆ 2 = [(2X1 + X2 + 5X3 ) /8] are unbiased estimators for μ. Compare the variances of θˆ 1 and θˆ 2 .

Solution We have 1 E θˆ 1 = E X = · 3μ = μ, 3 and 1 E θˆ 2 = [2EX1 + EX2 + 5EX3 ] 8 1 = [2μ + μ + 5μ] = μ. 8 Hence, both θˆ 1 and θˆ 2 are unbiased estimators. However, σ2 , Var θˆ 1 = 3 whereas Var θˆ 2 = Var =

2X1 + X2 + 5X3 8

1 2 25 2 30 2 4 2 σ + σ + σ = σ . 64 64 64 64

Because Var(θˆ 1 ) < Var(θˆ 2 ), we see that X is a better unbiased estimator in the sense that the variance of X is smaller.

252 CHAPTER 5 Point Estimation

It is important to observe that the maximum likelihood estimators are not always unbiased, but it can be shown that for such estimators the bias goes to zero as the sample size increases.

5.4.2 Sufficiency In the statistical inference problems on a parameter, one of the major questions is: Can a speciﬁc statistic replace the entire data without losing pertinent information? Suppose X1 , . . . , Xn is a random sample from a probability distribution with unknown parameter θ. In general, statisticians look for ways of reducing a set of data so that these data can be more easily understood without losing the meaning associated with the entire collection of observations. Intuitively, a statistic U is a sufﬁcient statistic for a parameter θ if U contains all the information available in the data about the value of θ. For example, the sample mean may contain all the relevant information about the parameter μ, and in that case U = X is called a sufﬁcient statistic for μ. An estimator that is a function of a sufﬁcient statistic can be deemed to be a “good” estimator, because it depends on fewer data values. When we have a sufﬁcient statistic U for θ, we need to concentrate only on U because it exhausts all the information that the sample has about θ. That is, knowledge of the actual n observations does not contribute anything more to the inference about θ. Deﬁnition 5.4.4 Let X1 , . . . , Xn be a random sample from a probability distribution with unknown parameter θ. Then, the statistic U = g(X1 , . . . , Xn ) is said to be sufﬁcient for θ if the conditional pdf or pf of X1 , . . . , Xn given U = u does not depend on θ for any value of u. An estimator of θ that is a function of a sufﬁcient statistic for θ is said to be a sufﬁcient estimator of θ.

Example 5.4.6 Let X1 , . . . , Xn be iid Bernoulli random variables with parameter θ. Show that U = for θ.

n

i=1 Xi is sufﬁcient

Solution The joint probability mass function of X1 , . . . , Xn is n

f (X1 , . . . , Xn ; θ) = Because U =

θ i=1

Xi

(1 − θ)

n−

n

i=1

Xi

0 ≤ θ ≤ 1.

,

n

i=1 Xi we have

f (X1 , . . . , Xn ; θ) = θ U (1 − θ)n−U ,

0 ≤ U ≤ n.

Also, because U ∼ B(n, θ), we have n U f (u; θ) = θ (1 − θ)n−U . u Also, f (x1 , . . . , xn |U = u ) =

f (x1 , . . . , xn , u) = fU (u)

f (x1 ,...,xn ) fU (u) ,

0,

u=

xi

otherwise.

5.4 Some Desirable Properties of Point Estimators 253

Therefore, ⎧ θ u (1−θ)n−u ⎪ = 1 ⎪ ⎨ n n u n−u θ (1−θ) f (x1 , . . . , xn |U = u) = u u ⎪ ⎪ ⎩ 0,

if u =

Example 5.4.7 Let X1 , . . . , Xn be a random sample from U(0, θ). That is, 1, θ

if 0 < x < θ

0,

otherwise.

Show that U = max X is sufﬁcient for θ. 1≤i≤n

Solution The joint density or the likelihood function is given by

f (x1 , . . . , xn ; θ) =

1 θn ,

if 0 < x1 , . . . , xn < θ

0,

otherwise.

The joint pdf f (x1 , . . . , xn ; θ) can be equivalently written as f (x1 , . . . , xn ; θ) =

1 θn ,

if xmin > 0, xmax < θ

0,

otherwise.

Now, we can compute the pdf of U. F (u) = P (U ≤ u) = P (X1 , . . . , Xn ≤ u) =

n 7 i=1

=

n 7 i=1

P (Xi ≤ u) (because of independence) ⎛ u ⎞ 1 un ⎝ dx⎠ = n , 0 < u < θ. θ θ 0

The pdf of U may now be obtained as f (u) =

nun−1 d F (u) = , du θn

0 0 otherwise

f (X1 , . . . , Xn |U ) is a function of u and xmin which is independent of θ. Hence, U = max Xi is sufficient 1≤i≤n

for θ.

The outcome X1 , . . . , Xn is always sufﬁcient, but we will exclude this trivial statistic from consideration. In the previous two examples, we were given a statistic and asked to check whether it was sufﬁcient. It can often be tedious to check whether a statistic is sufﬁcient for a given parameter based directly on the foregoing deﬁnition. If the form of the statistic is not given, how do we guess what is the sufﬁcient statistic? Now think of working out the conditional probability by hand for each of our guesses! In general, this will be a tedious way to go about ﬁnding sufﬁcient statistics. Fortunately, the Neyman–Fisher factorization theorem makes it easier to spot a sufﬁcient statistic. The following result will give us a convenient way of verifying sufﬁciency of a statistic through the likelihood function. NEYMAN–FISHER FACTORIZATION CRITERIA Theorem 5.4.3 Let U be a statistic based on the random sample X1 , . . . , Xn . Then, U is a sufﬁcient statistic for θ if and only if the joint pdf (or pf ) f (x1 , . . . , xn ; θ) (which depends on the parameter θ) can be factored into two nonnegative functions. f (x1 , . . . , xn ; θ) = g (u, θ) h (x1 , . . . , xn ) ,

for all x1 , . . . , xn ,

where g (u, θ) is a function only of u and θ and h (x1 , . . . , xn ) is a function of only x1 , . . . , xn and not of θ.

Proof. (Discrete case.) We will only give the proof in the discrete case, even though the result is also true for the continuous case. First suppose that U (X1 , . . . , Xn ) is sufﬁcient for θ. Then, X1 = x1 , X2 = x2 , . . . , Xn = xn if and only if X1 = x1 , X2 = x2 , . . . , Xn = xn and U (X1 , . . . , Xn ) = U (x1 , . . . , xn ) = u(say). Therefore f (x1 , . . . , xn ; θ) = Pθ X1 = x1 , X2 = x2 , . . . , Xn = xn and U = u = Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u ) Pθ (U = u) .

5.4 Some Desirable Properties of Point Estimators 255

Because U is assumed to be sufﬁcient for θ, the conditional probability Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u ) does not depend on θ. Let us denote this conditional probability by h(x1 , . . . , xn ). Clearly Pθ (U = u) is a function of u and θ. Let us denote this by g(u, θ). It now follows from the equation above that f (x1 , . . . , xn ; θ) = g (u, θ) h (x1 , . . . , xn )

as was to be shown. To prove the converse, assume that f (x1 , . . . , xn ; θ) = g (u, θ) h (x1 , . . . , xn ) .

Deﬁne the set Au by Au = {(x1 , . . . , xn ) : U (x1 , . . . , xn ) = u} .

That is, Au is the set of all (x1 , . . . , xn ) such that U maps it into u. We note that Au does not depend on θ. Now Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u) Pθ X1 = x1 , X2 = x2 , . . . , Xn = xn and U = u = Pθ (U = u) ⎧ ⎨ Pθ (X1 =x1 ,X2 =x2 ,...,Xn =xn and U=u) , if (x1 , . . . , xn ) ∈ Au Pθ (U=u) = ⎩ 0, if (x1 , . . . , xn ) ∈ / Au .

/ Au , then, clearly, If (x1 , . . . , xn ) ∈ f (x1 , . . . , xn ; θ) = Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u )

which is independent of θ. If (x1 , . . . , xn ) ∈ Au , then, using the factorization criterion, we obtain Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn |U = u ) =

Pθ (X1 = x1 , X2 = x2 , . . . , Xn = xn ) Pθ (U = u)

=

f (x1 , . . . , xn ; θ) = Pθ (U = u)

g (u, θ) h (x1 , . . . , xn ) g (u, θ) h (x1 , . . . , xn ) (x1 ,...,xn )∈Au

=

g (u, θ) h (x1 , . . . , xn ) = g (u, θ) h (x1 , . . . , xn ) (x1 ,...,xn )∈Au

h (x , . . . , xn ) 1 h (x1 , . . . , xn ) (x1 ,...,xn )∈Au

256 CHAPTER 5 Point Estimation

Therefore, the conditional distribution of X1 , . . . , Xn given U does not depend on θ, proving that U is sufﬁcient. One can use the following procedure to verify that a given statistic is sufﬁcient. This procedure is based on factorization criteria rather than using the deﬁnition of sufﬁciency directly.

PROCEDURE TO VERIFY SUFFICIENCY 1. Obtain the joint pdf or pf fθ (x1 , . . . , xn ). 2. If necessary, rewrite the joint pdf or pf in terms of the given statistic and parameter so that one can use the factorization theorem. 3. Deﬁne the functions g and h, in such a way that g is a function of the statistic and parameter only and h is a function of the observations only. 4. If step 3 is possible, then the statistic is sufﬁcient. Otherwise, it is not sufﬁcient.

In general, it is not easy to use the factorization criterion to show that a statistic U is not sufﬁcient. We now give some examples using the factorization theorem.

Example 5.4.8 Let X1 , . . . , Xn denote a random sample from a geometric population with parameter p. Show that X is sufﬁcient for p.

Solution For the geometric distribution, the pf is given by ⎧ ⎨ p (1 − p)x−1 , f (x, p) = ⎩ 0,

x≥1 otherwise.

Hence, the joint pf is f (x1 , . . . , xn ; p) = pn (1 − p) =

−n+

n

xi

i=1

⎧ ⎨pn (1 − p)nx−n , ⎩

0,

if x1 , . . . , xn ≥ 1 otherwise.

Take, g(x, p) = pn (1 − p)nx−n

Thus, X is sufficient for p.

and

h(x1 , . . . , xn ) =

⎧ ⎨1,

if

⎩0,

otherwise.

xi ≥ 1

5.4 Some Desirable Properties of Point Estimators 257

Example 5.4.9 Let X1 , . . . , Xn denote a random sample from a U (0, θ) with pdf ⎧ 1 ⎪ ⎨ , fθ (x) = θ ⎪ ⎩ 0,

0 < x < θ,

θ>0

otherwise.

Show that X(n)= max Xi is sufﬁcient for θ, using the factorization theorem. 1≤i≤n

Solution The likelihood function of the sample is ⎧ 1 ⎪ ⎨ , θn fθ (x1 , . . . , xn ) = ⎪ ⎩ 0,

if 0 < x1 , . . . , xn < θ, otherwise.

We can now write fθ (x1 , . . . ., xn ) as fθ (x1 , . . . , xn ) = h (x1 , . . . , xn ) g θ, x(n) , for all x1 , . . . , xn where h (x1 , . . . , xn ) =

⎧ ⎨ 1, ⎩

0,

if x1 , . . . , xn > 0 otherwise

and ⎧ ⎪ 1 ⎨ n, θ g θ; x(n) = ⎪ ⎩ 0,

if 0 < x(n) < θ, otherwise.

From the factorization theorem, we now conclude that X(n) is sufficient for θ. In the next definition, we introduce the concept of joint sufficiency.

Deﬁnition 5.4.5 Two statistics U1 and U2 are said to be jointly sufﬁcient for the parameters θ1 and θ2 if the conditional distribution of X1 , . . . , Xn given U1 and U2 does not depend on θ1 or θ2 . In general, the statistic U = (U1 , . . . , Un ) is jointly sufﬁcient for θ = (θ1 , . . . , θn ) if the conditional distribution of X1 , . . . , Xn given U is free of θ. Now we state the factorization criteria for joint sufﬁciency analogous to the single population parameter case.

258 CHAPTER 5 Point Estimation

THE FACTORIZATION CRITERIA FOR JOINT SUFFICIENCY Theorem 5.4.4 The two statistics U1 and U2 are jointly sufﬁcient for θ1 and θ2 if and only if the likelihood function can be factored into two non-negative functions, f (x1 , . . . , xn ; θ1 , θ2 ) = g(u1 , u2 ; θ1 , θ2 ) h(x1 , . . . , xn ) where g (u1 , u2 ; θ1 , θ2 ) is only a function of u1 , u2 ; θ1 and θ2 , and h(x1 , xn ) is free of θ1 or θ2 .

Example 5.4.10 Let X1 , . . . , Xn be a random sample from N(μ, σ 2 ). (a) If μ is unknown and σ 2 = σ02 is known, show that X is a sufﬁcient statistic for μ. (b) If μ = μ0 is known and σ 2 is unknown, show that ni=1 (Xi − μ0 )2 is sufﬁcient for σ 2 . (c) If μ and σ 2 are both unknown, show that ni=1 Xi and ni=1 Xi2 are jointly sufﬁcient for μ and σ 2 .

Solution The likelihood function of the sample is ⎡

⎤ 2 (X − μ) i ⎢ ⎥ 1 ⎢ i=1 ⎥ L= exp − ⎢ ⎥ ⎣ ⎦ 2σ 2 (2π)n/2 σ n n

n & n 1 2 2 exp xi − 2μ xi + nμ = 2σ 2 (2π)n/2 σ n i=1 i=1 ⎞ ⎛ n 2 x

⎜ i ⎟ 2μnx nμ2 ⎜ i=1 ⎟ −n/2 −n exp − 2 . σ exp ⎜− = (2π) ⎟ exp ⎝ 2σ 2 ⎠ 2σ 2 2σ 1

%

(a) When σ 2 = σ02 is known, use the factorization criteria, with 2nμx − nμ2 g(x, μ) = exp 2σ02 and

⎛

⎞ n 2 x ⎜ i ⎟ ⎜ i=1 ⎟ h(x1 , . . . , xn ) = (2π)−n/2 σ −n exp ⎜− ⎟. ⎝ 2σ 2 ⎠

Therefore, X is sufficient for μ.

5.4 Some Desirable Properties of Point Estimators 259

(b) When μ = μ0 is known, let

g

n

(Xi − μ)2 , σ 2

i=1

n 2 (xi − μ) i=1 = σ −n exp − 2σ 2

and h(x1 , . . . , xn ) = Thus, ni=1 (Xi − μ)2 is sufficient for σ 2 . (c) When both μ and σ 2 are unknown, use

g

n

xi ,

i=1

n

xi2 , μ, σ 2

i=1

1 . (2π)n/2

n n 2 2 x − 2μ x + nμ i i i=1 i=1 = σ −n exp − 2 2σ

and h(x1 , . . . , xn ) = Hence,

n

i=1 Xi and

1 . (2π)n/2

n

2 2 i=1 Xi are jointly sufficient for μ and σ .

Example 5.4.11 Suppose that we have a random sample X1 , . . . , Xn from a discrete distribution given by fθ (x) = C (θ) 2−x/θ ,

x = θ, θ + 1, θ + 2, . . . ;

θ>0

where C (θ) > 0 is a normalizing constant. Using the factorization theorem, ﬁnd a sufﬁcient statistic for θ.

Solution The joint density function f (x1 , . . . , xn ; θ) of the sample X1 , . . . , Xn is ⎧ n ⎪ ⎪ − (xi /θ) ⎨ i=1 , x1 , x2 , . . . , xn are integers ≥ θ f (x1 , . . . , xn ; θ) = C (θ) 2 ⎪ ⎪ ⎩ 0, otherwise . The function f (x1 , . . . , xn ; θ) can be written as −

f (x1 , . . . , xn ; θ) = h(x1 , . . . , xn ) C (θ) 2

n

(xi /θ)

i=1

g1 θ, x(1)

260 CHAPTER 5 Point Estimation

where x(1) = min (x1 , . . . , xn ), and i

h(x1 , x2 , . . . , xn ) =

⎧ ⎨1,

if xj − x(1) ≥ 0 is an integer for j = 1, 2, . . . , n

⎩0,

otherwise

and

⎧ ⎨1,

g1 θ, x(1) = ⎩0,

if x(1) ≥ θ otherwise.

Thus, xi , x(1) f (x1 , . . . , xn ; θ) = h(x1 , . . . , xn ) g θ, n

− (xi / θ) g1 θ, x(1) . Using the factorization theorem, we conclude that where g θ, xi , x(1) = C(θ)2 i=1 xi , x(1) is jointly sufﬁcient for θ. This result shows that even for a single parameter, we may need more than one statistic for sufﬁciency.

When using the factorization criterion, one has to be careful in cases where the range space depends on the parameter. Using the factorization criterion, we can prove the following result, which says that if we have a unique maximum likelihood estimator, then that estimator will be a function of the sufﬁcient statistic. Theorem 5.4.5 If U is a sufﬁcient statistic for θ, the maximum likelihood estimator of θ, if unique, is a function of U. Proof. Because U is sufﬁcient, by Theorem 5.4.1, the joint pdf can be factored as f (x1 , . . . , xn ; θ) = g(u, θ) h(x1 , . . . , xn ).

This depends on θ only through the statistic U. To maximize L we need to maximize g(U, θ). Many common distributions such as Poisson, normal, gamma, and Bernoulli are members of the exponential family of probability distributions. The exponential family of distributions has density functions of the form f (x; θ) =

⎧ ⎨exp [k(x)c(θ) + S(x) + d(θ)] , ⎩

where B does not depend on the parameter θ.

0,

if x ∈ B x∈ /B

5.4 Some Desirable Properties of Point Estimators 261

Example 5.4.12 Write the following in exponential form. e−λ λx x! (b) px (1 − p)1−x 1 2 (c) √ e−(x−μ) /2 2π (a)

Solution (a) We have e−λ λx = exp [x ln λ − ln x! − λ] . x! Here k(x) = x, c (λ) = ln λ, S(x) = − ln (x!), and d (λ) = −λ. (b) Similarly, !

px (1 − p)1−x = exp x ln

p 1−p

" + ln (1 − p) ,

x = 0 or 1.

(c) This is the standard normal density. 2 2 1 2 √ e−(x−μ) /2 = exp xμ − x2 − μ2 − 12 ln (2π) , 2π

−∞ < x < ∞.

Note that in the previous example, for each of the cases, ni=1 Xi is a sufﬁcient statistic for the parameter. In the next result, we give a generalization of this fact. Theorem 5.4.6 Let X1 , . . . , Xn be a random sample from a population with pdf or pmf of the exponential form f (x; θ) =

exp [k(x)c (θ) + S(x) + d (θ)] , 0,

where B does not depend on the parameter θ. The statistic

n

i=1 k (Xi )

if x ∈ B x∈ /B

is sufﬁcient for θ.

Proof. The joint density % f (x1 , . . . , xn ; θ) = exp c (θ)

n

k (xi ) +

i=1

% = exp c (θ)

n

& S (xi ) + nd (θ)

i=1

n

&4 k (xi ) + nd (θ)

exp

i=1

Using the factorization theorem, the statistic

n

i=1 k (Xi )

% n i=1

is sufﬁcient.

&4 S (xi )

.

262 CHAPTER 5 Point Estimation

It does not follow that every function of a sufﬁcient statistic is sufﬁcient. However, any one-to-one function of a sufﬁcient statistic is also sufﬁcient. Every statistic need not be sufﬁcient. When they do exist, sufﬁcient estimators are very important, because if one can ﬁnd a sufﬁcient estimator it is ordinarily possible to ﬁnd an unbiased estimator based on the sufﬁcient statistic. Actually, the following theorem shows that if one is searching for an unbiased estimator with minimal variance, it has to be restricted to functions of a sufﬁcient statistics. RAO–BLACKWELL THEOREM Theorem 5.4.7 Let X1 , . . . , Xn be a random sample with joint pf or pdf f (x1 , . . . , xn ; θ) and let U = (U1 , . . . , Un ) be jointly sufﬁcient for θ = (θ1 , . . . , θn ). If T is any unbiased estimator of k (θ), and if T ∗ = E (T |U ), then: (a) T ∗ is an unbiased estimator of k(θ). (b) T ∗ is a function of U, and does not depend on θ. (c) Var T ∗ ≤ Var(T ) for every θ, and Var T ∗ < Var(T ) for some θ unless T ∗ = T with probability 1.

Proof. (a) By the property of conditional expectation and by the fact that T is an unbiased estimator of k(θ), E T ∗ = E(E(T |U)) = E(T ) = k(θ).

Hence, T ∗ is an unbiased estimator of k(θ). (b) Because U is sufﬁcient for θ, the conditional distribution of any statistic (hence, for T ), given U, does not depend on θ. Thus, T ∗ = E(T |U) is a function of U. (c) From the property of conditional probability, we have the following: Var (T ) = E (Var (T |U )) + Var (E (T |U )) = E (Var (T |U )) + Var T ∗ .

Because Var (T |U ) ≥ 0 for all u, it follows that E (Var (T |U )) ≥ 0. Hence, Var T ∗ ≤ Var(T ). We ∗ note that Var T = Var(T ) if and only if Var (T |U) = 0 or T is a function of U, in which case T ∗ = T (from the deﬁnition of T ∗ = E (T |U ) = T ). In particular, if k (θ) = θ, and T is an unbiased estimator of θ, then T ∗ = E (T |U ) will typically give the MVUE of θ. If T is the sufﬁcient statistic that best summarizes the data from a given distribution with parameter θ, and we can ﬁnd some function g of T such that E (g (T )) = θ, it follows from the Rao–Blackwell theorem that g(T ) is the UMVUE for θ.

EXERCISES 5.4 5.4.1.

Let X1 , . . . , Xn be a random sample from a population with density f (x) =

e−(x−θ) , 0,

for x > θ otherwise.

5.4 Some Desirable Properties of Point Estimators 263

(a) Show that X is a biased estimator of θ. (b) Show that X is an unbiased estimator of μ = 1 + θ. 5.4.2.

The mean and variance of a ﬁnite population {a1 , . . . , aN } are deﬁned by μ=

N N 1 1 (ai − μ)2 . ai and σ 2 = N N i=1

i=1

For a ﬁnite population, show that the sample variance S 2 is a biased estimator of σ 2 . 5.4.3.

For an inﬁnite population with ﬁnite variance σ 2 , show that the sample standard deviation S is a biased estimator for σ. Find an unbiased estimator of σ. [We have seen that S 2 is an unbiased estimator of σ 2 . From this exercise, we see that a function of an unbiased estimator need not be an unbiased estimator.]

5.4.4.

Let X1 , . . . , Xn be a random sample from an inﬁnite population with ﬁnite variance σ 2 . Deﬁne S 2 =

n 2 1 Xi − X . n i=1

2

Show that S 2 is a biased estimator for σ 2 , and that the bias of S 2 is − σn . Thus, S 2 is negatively biased, and so on average underestimates the variance. Note that S 2 is the MLE of σ 2 . 5.4.5.

Let X1 , . . . , Xn be a random sample from a population with the mean μ. What condition must be imposed on the constants c1 , c2 , . . . , cn so that c1 X1 + c2 X2 + · · · + cn Xn

is an unbiased estimator of μ? 5.4.6.

Let X1 , . . . , Xn be a random sample from a geometric distribution with parameter θ. Find an unbiased estimate of θ.

5.4.7.

Let X1 , . . . , Xn be a random sample from U (0, θ) distribution. Let Yn = max{X1 , . . . , Xn }. We know (from Example 5.3.4) that θˆ 1 = Yn is a maximum likelihood estimator of θ. (a) Show that θˆ 2 = 2X is a method of moments estimator. (b) Show that θˆ 1 is a biased estimator, and θˆ 2 is an unbiased estimator of θ. (c) Show that θˆ 3 =

5.4.8. 5.4.9.

n+1 ˆ n θ1

is an unbiased estimator of θ.

Let X1 , . . . , Xn be a random sample from a population with mean μ and variance 1. Show 2 that μ ˆ 2 = X is a biased estimator of μ2 , and compute the bias. Let X1 , . . . , Xn be a random sample from an N μ, σ 2 distribution. Show that the estimator μ ˆ = X is the MVUE for μ.

264 CHAPTER 5 Point Estimation

5.4.10.

Let X1 , . . . , Xn1 be a random sample from an N μ1 , σ 2 distribution and let Y1 , . . . , Yn2 be a random sample from a N μ2 , σ 2 distribution. Show that the pooled estimator σˆ 2 =

5.4.11.

(n1 − 1) S12 + (n2 − 1) S22 n1 + n 2 − 2

is unbiased for σ 2 , where S12 and S22 are the respective sample variances. Let X1 , . . . , Xn be a random sample from an N μ, σ 2 distribution. Show that the sample median, M, is an unbiased estimator of the population mean μ. Compare the variances of X and M. [Note: For the normal distribution, the mean, median, and mode all occur at the same location. Even though both X and M are unbiased, the reason we usually use the mean instead of the median as the estimator of μ is that X has a smaller variance than M.]

5.4.12.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. Show that the sample mean X is sufﬁcient for λ.

5.4.13.

Let X1 , . . . , Xn be a random sample from a population with density function fσ (x) =

|x| 1 exp − , 2σ σ

−∞ < X < ∞,

σ > 0.

Find a sufﬁcient statistic for the parameter σ. 5.4.14.

Show that if θˆ is a sufﬁcient statistic for the parameter θ and if the maximum likelihood estimator of θ is unique, then the maximum likelihood estimator is a function of this sufﬁcient statistic θˆ .

5.4.15.

Let X1 , . . . , Xn be a random sample from an exponential population with parameter θ. (a) Show that ni=1 Xi is sufﬁcient for θ. Also show that X is sufﬁcient for θ. (b) The following is a random sample from exponential distribution. 1.5 0.3 5.7

3.0 2.0 0.1

2.6 1.8 0.2

6.8 1.0 0.5

0.7 0.7 0.4

2.2 0.7

1.3 1.6

1.6 3.0

1.1 2.0

6.5 2.5

(i) What is an unbiased estimate of the mean? (ii) Using part (a) and these data, ﬁnd two sufﬁcient statistics for the parameter θ. 5.4.16.

Let X1 , . . . , Xn be a random sample from a one-parameter Weibull distribution with pdf f (x) =

⎧ ⎨2αxe−αx2 , ⎩

(a) Find a sufﬁcient statistic for α. (b) Using part (a), ﬁnd an UMVUE for α.

0,

x>0 otherwise.

5.4 Some Desirable Properties of Point Estimators 265

5.4.17.

Let X1 , . . . , Xn be a random sample from a population with density function ⎧ 1 ⎪ ⎨ , f (x) = θ ⎪ ⎩ 0,

Show that 5.4.18.

−

θ θ ≤x≤ ,θ>0 2 2 otherwise.

min Xi , max Xi is sufﬁcient for θ.

1≤i≤n

1≤i≤n

Let X1 , . . . , Xn be a random sample from a G(1, β) distribution. (a) Show that U = ni=1 Xi is a sufﬁcient statistic for β. (b) The following is a random sample from a G(1, β) distribution. 0.3 0.3

3.4 3.7

0.4 0.1

1.8 1.3

0.7 1.2

1.0 3.3

0.1 0.2

2.3 1.3

3.7 0.6

2.0 0.4

Find a sufﬁcient statistic for β. 5.4.19.

Show that X1 is not sufﬁcient for μ, if X1 , . . . , Xn is a sample from N(μ, 1).

5.4.20.

Let X1 , . . . , Xn be a random sample from the truncated exponential distribution with pdf ⎧ ⎨eθ−x ,

f (x) =

⎩ 0,

x>θ otherwise.

Show that X(1) = min(Xi ) is sufﬁcient for θ. 5.4.21.

Let X1 , . . . , Xn be a random sample from a distribution with pdf f (x) =

⎧ ⎨θxθ−1 , ⎩

0 < x < 1,

0,

θ>0

otherwise.

Show that U = X1 , . . . , Xn is a sufﬁcient statistic for θ. 5.4.22.

Let X1 , . . . , Xn be a random sample of size n from a Bernoulli population with parameter p. Show that pˆ = X is the UMVUE for p.

5.4.23.

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

Show that

n

2 i=1 Xi

⎧ ⎨ 2x e−x2 /α ,

x>0

α

⎩

0,

otherwise.

is sufﬁcient for the parameter α.

266 CHAPTER 5 Point Estimation

5.5 OTHER DESIRABLE PROPERTIES OF A POINT ESTIMATOR In this section, we discuss a few more properties of point estimators that can be used in choosing a particular estimator.

5.5.1 Consistency It is a desirable property that the values of an estimator be closer to the value of the true parameter being estimated as the sample size becomes larger. To this end, we now introduce the notion of consistent estimators. Consistency is a large-sample, or asymptotic, property. That is, it describes the behavior of estimators as the sample size n becomes inﬁnitely large. In this section, we use the notation θˆ n for θˆ to show the dependence of the estimator on the sample size n. Deﬁnition 5.5.1 The estimator θˆ n is said to be a consistent estimator of θ if, for any ε > 0, lim P ˆθn − θ ≤ ε = 1

n→∞

or equivalently, lim P ˆθn − θ > ε = 0. n→∞

The statement “θˆ n is a consistent estimator of θ” is equivalent to “θˆ n converges in probability to θ.” That is, the sample estimator should have a high probability of being close to the population value θ for large sample size n. The idea of consistency can be observed in Figure 5.2, where θˆ n converges to θ. If it did not, θˆ n would not be a consistent estimator of θ. If the estimator is unbiased, we have the following result, which gives a sufﬁcient condition for the consistency of an estimator. However, it is important to note that a consistent estimator need not be unbiased, and hence this result is not a necessary condition.

A SUFFICIENT CONDITION FOR CONSISTENCY OF AN UNBIASED ESTIMATOR Theorem 5.5.1 An unbiased estimator θˆ n of θ is a consistent estimator for θ if lim Var θˆ n = 0. n→∞

x n

■ FIGURE 5.2 Consistency of an estimator.

5.5 Other Desirable Properties of a Point Estimator 267

The proof of this theorem follows directly from Chebyshev’s inequality. A general version of this result is proved in Theorem 5.5.3.

Example 5.5.1 Let X1 , . . . , Xn be a random sample with true mean μ and ﬁnite variance. Then, the sample mean X is a consistent estimator of the population mean μ.

Solution We show this result in two ways. , we obtain (i) Using Chebyshev’s inequality, P{|X − μ| ≥ ε} ≤ Var(x) ε2 σ2 # $ P X − μ ≤ k ≥ 1 − X k2 σ2 = 1 − 2 → 1as n → ∞. k n Hence, X is a consistent estimator of μ. (ii) First note that X is an unbiased estimator of μ. Because Var X = σ 2 /n , we have σ2 = 0. n→∞ n lim

Thus, from the previous theorem, X is a consistent estimator of μ.

We can generalize Theorem 5.5.1 even when the estimator is biased. The following result states that the mean square error of θˆ n decreases to zero as more and more observations are incorporated into its computation. TEST FOR CONSISTENCY Theorem 5.5.2 Let θˆ n be an estimator of θ and let Var θˆ n be ﬁnite. If lim E

n→∞

2 θˆ n − θ =0

then θˆ n is a consistent estimator of θ.

Proof. Using Chebyshev’s inequality, we obtain E P ˆθn − θ ≥ ε ≤

Because lim E

n→∞

2 θˆ n − θ

s2

.

2 θˆ n − θ = 0, [by hypothesis]

268 CHAPTER 5 Point Estimation

the right-hand side converges to zero. Thus, lim P ˆθn − θ ≥ ε = 0. n→∞

Consequently θˆ n is a consistent estimator of θ.

Furthermore, we know that E

2 2 = Var θˆ n + B θˆ n θˆ n − θ ,

and for unbiased estimators, the bias B θˆ n is zero. As a result, Theorem 5.5.1 is a particular case of Theorem 5.5.3. We now summarize the procedure for testing for consistency of an estimator as follows:

PROCEDURE TO TEST FOR CONSISTENCY 1. Check whether the estimator θˆ n is unbiased or not. 2. Calculate Var θˆ n and B θˆ n , the bias of θˆ n . 3. An unbiased estimator is consistent if Var θˆ n → 0 as n → ∞. 4. A biased estimator is consistent if both Var θˆ n → 0 and B θˆ n → 0 as n → ∞.

Example 5.5.2

Let X1 , . . . , Xn be a random sample from N μ, σ 2 population. (a) Show that the sample variance S 2 is a consistent estimator for σ 2 . (b) Show that the maximum likelihood estimators for μ and σ 2 are consistent estimators for μ and σ 2 .

Solution (a) We have already seen that ES 2 = σ 2 , and hence, S 2 is an unbiased estimator of σ 2 . Because $ # the sample is drawn from a normal distribution, we know that (n − 1) S 2 /σ 2 has a chi-square distribution with (n − 1) d.f. and (n − 1) S 2 Var = 2 (n − 1). σ2 Thus,

2 (n − 1) = Var

(n − 1) S 2 σ2

=

(n − 1)2 Var S 2 . σ4

5.5 Other Desirable Properties of a Point Estimator 269

This implies that 2σ 4 Var S 2 = → 0 as n → ∞. n−1 Hence, S 2 is a consistent estimator of the variance of a normal population. 2 ˆ is (b) We have seen that the MLE of μ is μ ˆ = X, and that of σ 2 is σˆ n2 = (1/n) ni=1 Xi − X . Now μ an unbiased estimator of μ, and Var( X ) = (σ 2 /n) → 0 as n → ∞. Therefore, from Theorem 5.5.1, X is a consistent estimator for μ. Now we will use the identity 2 2 = Var θˆ n + B θˆ n E θˆ n − θ to show that the MLE for σ 2 is biased with n−1 2 σ E σˆ n2 = n and n−1 2 1 σ − σ2 = − σ2. B σˆ n2 = n n Thus, σˆ n2 = (1/n)

n

i=1 Xi − X

2

= ((n − 1) /n) S 2 . Using part (a), we get (n − 1)2 Var S 2 Var σˆ n2 = 2 n 2 2 4 2(n − 1) σ 2 (n − 1) 2σ = . = n2 (n − 1) n2

Therefore, −σ 2 = 0, and lim Var σˆ n2 lim B σˆ n2 = lim n→∞ n→∞ n n→∞ 2 2(n − 1) σ 2 = lim = 0. n→∞ n2 By Theorem 5.5.3, σˆ n2 =

n 2 1 Xi − X n i=1

is a consistent estimator of σ 2 .

From the foregoing example we can see that consistent estimators need not be unique. It turns out that most of the MLEs and method of moments estimators derived for important probability distributions are consistent.

270 CHAPTER 5 Point Estimation

5.5.2 Efficiency We have seen that there can be more than one unbiased estimator for a parameter θ. We have also mentioned that the one with the least variance is desirable. Here, we introduce the concept of efﬁciency, which is based on comparing variances of the different unbiased estimators. If there are two unbiased estimators, it is desirable to have the one with a smaller variance. Deﬁnition 5.5.2 If θˆ 1 and θˆ 2 are two unbiased estimators for θ, the efﬁciency of θˆ 1 relative to θˆ 2 is the ratio Var θˆ 2 e θˆ 1 , θˆ 2 = . Var θˆ 1

If Var θˆ 2 > Var θˆ 1 , or equivalently, e θˆ 1 , θˆ 2 > 1, then, θˆ 1 is relatively more efficient than θˆ 2 . That is θˆ 1 has a smaller variance as compared to the variance of θˆ 2 . We summarize the following procedure to compare the efﬁciencies of the different unbiased estimators.

PROCEDURE TO TEST RELATIVE EFFICIENCY 1. Check for unbiasedness of θˆ 1 and θˆ 2 . 2. Calculate the variances of θˆ 1 and θˆ 2 . 3. Calculate the relative efﬁciency as

Var θˆ 2 e θˆ 1 ; θˆ 2 = . Var θˆ 1

4. Conclusion: If e θˆ 1, θˆ 2 < 1, θˆ 2 is more efﬁcient than θˆ 1 , and if e θˆ 1 , θˆ 2 > 1, then, θˆ 1 is more efﬁcient than θˆ 2 . Among the unbiased estimators, the more efﬁcient estimator is preferable.

Example 5.5.3 Let X1 , . . . , Xn , n > 3, be a random sample from a population with a true mean μ and variance σ 2 . Consider the following three estimators of μ: θˆ 1 =

1 (X1 + X2 + X3 ) , 3

θˆ 2 =

1 1 3 X1 + X2 + · · · + Xn−1 + Xn , 8 4 (n − 2) 8

and θˆ 3 = X. (a) Show that each of the three estimators is unbiased. (b) Find e θˆ 2 , θˆ 1 , e θˆ 3 , θˆ 1 , and e θˆ 3 , θˆ 2 . Which of the three estimators is more efﬁcient?

5.5 Other Desirable Properties of a Point Estimator 271

Solution (a) Given E(Xi ) = μ, i = 1, 2, . . . , n. Then, 1 3μ =μ E θˆ 1 = [E(X1 ) + E(X2 ) + E(X3 )] = 3 3 1 1 3 E(X2 ) + · · · + E Xn−1 + E(Xn ) E θˆ 2 = E(X1 ) + 8 4 (n − 2) 8 =

3 1 1 (n − 2) μ + μ = μ μ+ 8 4 (n − 2) 8

E θˆ 3 = E X = μ. Hence, θˆ 1 , θˆ 2 , and θˆ 3 are unbiased estimators of μ. (b) Computing the variances, we have 1 Var θˆ 1 = (Var(X1 ) + Var(X2 ) + Var(X3 )) 9 =

1 2 σ2 3σ = . 9 3

σ2 9 (n − 2) σ 2 σ2 Var θˆ 2 = + + 64 64 16 (n − 2)2 =

9σ 2 n + 16 2 2σ 2 + = σ . 64 16 (n − 2) 32 (n − 2)

σ2 Var θˆ 3 = . n The relative efficiencies are

Var θˆ 2 σ 2 (n + 16) /32 (n − 2) e θˆ 1 , θˆ 2 = = σ 2 /3 Var θˆ 1 =

3 (n + 16) < 1 for n > 3. 32 (n − 2)

Thus, for n ≥ 4, θˆ 2 is more efficient than θˆ 1 . Var θˆ 1 n σ 2 /3 ˆ ˆ e θ3 , θ1 = = 2 = > 1 for n ≥ 4. ˆ 3 σ /n Var θ3

272 CHAPTER 5 Point Estimation

Hence, for n > 3, θˆ 3 is more efficient than θˆ 1 . n+16 2 Var θˆ 2 32(n−2) σ e θˆ 3 , θˆ 2 = = σ 2 /n Var θˆ 3 =

n2 + 16n > 1 for n ≥ 4. 32 (n − 2)

Therefore, even though both θˆ 3 θˆ 2 are based on all the n observations, for n > 3, the sample mean θˆ 3 is more efficient than θˆ 2 .

It is reasonable to compare estimators on the basis of variance alone if they are both unbiased. To facilitate the cases where the estimators are biased, we use the mean square error (MSE) in the deﬁnition of relative efﬁciency. Deﬁnition 5.5.3 An estimator θˆ 1 is more efﬁcient than θˆ 2 if MSEθˆ 1 ≤ MSEθˆ 2

with strict inequality for some θ. Also, the relative efﬁciency of θˆ 1 with respect to θˆ 2 is 2 E θˆ 2 − θ MSE θˆ 2 ˆ ˆ e θ1 , θ2 = . 2 = MSE θˆ 1 E θˆ 1 − θ

Example 5.5.4 Let X1 , . . . , Xn , n ≥ 2 be a random sample from a normal population with a true mean μ and variance σ 2 . Consider the following two estimators of σ 2 : θˆ 1 = S 2 , and θˆ 2 = S 2 . Find e θˆ 1 , θˆ 2 .

Solution

2 2 (n − 1), E(S 2 ) = σ 2 , and MSE(S 2 ) = Var(S 2 ). Also, 2(n − 1) = Var (n−1)S 2 = Because (n−1)S ∼ χ 2 2 σ σ (n−1)2 Var(S 2 ). σ4

Thus, MSE θˆ 1 =

2 σ4. n−1

Also, it can be shown that (2n − 1) 2 MSE S 2 = σ . n2

5.5 Other Desirable Properties of a Point Estimator 273

Thus, the relative efficiency of θˆ 1 with respect to θˆ 2 is MSE θˆ 2 MSE S 2 ˆ ˆ e θ1 , θ2 = = MSE S 2 MSE θˆ 1

=

(2n−1) 2 σ (2n − 1) (n − 1) n2 = . 2 σ2 2n2 (n−1)

For n ≥ 2, it can be seen that e θˆ 1 , θˆ 2 < 1. Hence, S 2 is relatively more efficient than S 2 .

We have seen that it is possible that one unbiased estimator is more efﬁcient than another. This leads to the possibility of having one unbiased estimator more efﬁcient than all the other unbiased estimators. This directs us to the following deﬁnition. Deﬁnition 5.5.4 An unbiased estimator θˆ 0 , is said to be a uniformly minimum variance unbiased estimator (UMVUE) for the parameter θ if, for any other unbiased estimator θˆ Var θˆ 0 ≤ Var θˆ ,

for all possible values of θ. It is not always easy to ﬁnd an UMVUE for a parameter. However, the following result gives a lower bound for the variance of any unbiased estimator.

CRAMÉR–RAO INEQUALITY Theorem 5.5.3 Let X1 , . . . , Xn be a random sample from a population with pdf (or pf ) fθ (x) that depends on a parameter θ. If θˆ is an unbiased estimator of θ, then, under very general conditions, the following inequality is true: Var θˆ ≥

! nE

1

∂ ln fθ (x) 2 ∂θ

".

If θˆ is an unbiased estimator of ψ(θ), then Var θˆ ≥

∂ψ(θ) 2 ∂θ

. ∂ ln f (x) 2 nE ∂θ θ

274 CHAPTER 5 Point Estimation

If L(θ) is the likelihood function, we can rewrite the Cramér–Rao inequality in the form Var θˆ ≥

! E

1

∂ ln L(θ) 2 ∂θ

"

From the Cramér–Rao inequality, we can obtain the following result.

EFFICIENT ESTIMATOR Theorem 5.5.4 If θˆ is an unbiased estimator of θ and if Var θˆ =

! nE

1

∂ ln fθ (x) 2 ∂θ

",

then θˆ is a uniformly minimum variance unbiased estimator (UMVUE) of θ. Sometimes θˆ is also referred to as an efﬁcient estimator.

Note that if the function f (.) is sufﬁciently smooth, it can be shown that ∂ ln fθ (x) 2 ∂2 ln fθ (x) = Var[ln fθ (x)] . E = −E ∂θ ∂θ 2

Hence, the Cramér–Rao inequality in this case can be rewritten as Var θˆ ≥

1

2 −nE ∂ ln∂θf2θ (x)

=

1

.

∂ ln f (x) nVar ∂θ θ

Now, we will give a procedure to apply the Cramér–Rao inequality.

CRAMÉR–RAO PROCEDURE TO TEST FOR EFFICIENCY 2

1. For the pdf (or pf), ﬁnd ∂ lnf∂θ(x) and ∂ ln f2(x) . ∂θ ! ! 2 2 "" 2. Calculate (1/n) E − ∂ ln f2(x) if f (x) is smooth, or else calculate 1/nE ∂ ln∂θf (x) . ∂θ

3. Calculate Var (θˆ ). 4. If the result of step 2 is equal to the result of step 3, then, θˆ is efﬁcient for θ.

Example 5.5.5 Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population with density function f (x). Show that X is an efﬁcient estimator for μ.

5.5 Other Desirable Properties of a Point Estimator 275

Solution To calculate the Cramér–Rao lower bound, we have ln f (x) = c −

(x − μ)2 , 2σ 2

where c is a constant not involving μ. Then ∂ ln f (x) x−μ = ∂μ σ2 and ∂2 ln f (x) 1 =− 2 ∂μ2 σ or 1

% nE −

∂2 ln f (x) ∂θ 2

1

& = nE

1 σ2

=

σ2 = Var X . n

Therefore, X is an efficient estimator of μ. That is, X is an UMVUE of μ.

Example 5.5.6 Suppose p(x) is the Poisson distribution with parameter λ. Show that the sample mean Xn is an efﬁcient estimator for λ.

Solution

−λ

Here the density function is given by p(x) = λx ex! . Taking logarithms, ln p(x) = x ln λ − λ − ln(x!) ∂ ln p(x) x = − 1, ∂λ λ and x ∂2 ln p(x) =− 2 ∂λ2 λ Therefore, using the fact that the expected value of a Poisson r.v. is λ,

1

2 f (x) nE − ∂ ln ∂λ2

=

1 λ = = Var X . n nE X λ2

Hence, X is an efficient estimator of λ.

Example 5.5.7 Let X1 , . . . , Xn be a random sample from a Bernoulli trial with probability of success p. Show that the maximum likelihood estimator is also an efﬁcient estimator.

276 CHAPTER 5 Point Estimation

Solution

Note that the MLE of p is pˆ = (1/n) ni=1 Xi = X/n, the fraction of successes in the total number of trials, n. Because we can view n Bernoulli trials as being a single observation from a binomial distribution with parameters n and p, the likelihood function is n x L(p) = p (1 − p)x . x Then, n + x ln p + (n − x) ln(1 − p). x

ln L(p) = ln Now

∂ ln L(p) x n−x x − np = − = . ∂p p 1−p p(1 − p) Hence,

% E

% & & x − np 2 ∂ ln L(p) 2 =E ∂p p (1 − p) = =

Var(x) [p(1 − p)]2 np(1 − p)

n . [p(1 − p)]2 p(1 − p)

Therefore, the Cramér– Rao bound is ! E Now

1

∂ ln L(p) 2 ∂p

" =

p(1 − p) . n

X Var pˆ = Var n 1 = 2 Var(x) n 1 p(1 − p) = 2 np(1 − p) = . n n

Because the variance of the estimator is equal to the Cramér–Rao lower bound, we conclude that pˆ = X n is an efficient estimator of p.

It is important to note that an UMVUE may not exist for a given problem. Even when an UMVUE exists, it is not necessary that it have a variance equal to the Cramér–Rao lower bound. The term

5.5 Other Desirable Properties of a Point Estimator 277

I(θ) = E

!

∂ ln f (x) ∂θ

2 "

is called the Fisher information. In fact, for a random sample of size n with ! 2 " ∂ ln L(θ) likelihood function L(θ), the Fisher information is deﬁned as In (θ) = E . It can be ∂θ shown that the Fisher information in a sample of size n is n times the Fisher information in one observation. That is, In (θ) = nI(θ).

5.5.3 Minimal Sufficiency and Minimum-Variance Unbiased Estimation In the study of statistics, it is desirable to reduce the data contained in the sample as much as possible without losing relevant information. Our objective is to ﬁnd minimal sufﬁcient statistics and use them to develop uniformly minimum variance unbiased estimators (UMVUEs) for true parameters. Whenever sufﬁcient statistics exist, then a statistician with those summary measures is as well off as the statistician with the entire sample, for point estimation purposes. Minimal sufﬁcient statistics are those that are sufﬁcient for the parameters and are functions of every other set of sufﬁcient statistics for those same parameters. Deﬁnition 5.5.5 A sufﬁcient statistic T (X) is called a minimal sufﬁcient statistic if for any other statistic T (X), T (X) is a function of T (X). That is, T (X) = g T (X) .

Using this deﬁnition, it is difﬁcult to determine whether a set of statistics is, in fact, minimal sufﬁcient. Now we will present a method due to Lehmann and Scheffé that will be of great help in ﬁnding a minimal sufﬁcient statistic. We can summarize the Lehmann and Scheffé method to ﬁnd a minimal sufﬁcient statistic as follows. Let X1 , . . . , Xn be a random sample with pdf or pmf f (x) that depends on a parameter θ. Let (x1 , . . . , xn ) and (y1 , . . . , yn ) be two different sets of values of (X1 , . . . , Xn ). Let L (θ; x1 , . . . , xn ) L (θ; y1 , . . . , yn )

be the ratio of the likelihoods evaluated at these two points. Suppose it is possible to ﬁnd a function g(x1 , . . . , xn ) such that this ratio will be free of the unknown parameter θ if and only if g(x1 , . . . , xn ) = g(y1 , . . . , yn ). If such a function g can be found, then g(X1 , . . . , Xn ) is a minimal sufﬁcient statistic for θ.

Example 5.5.8 Let X1 , . . . , Xn be a random sample from the Bernoulli distribution where P (Xi = 1) = p and P (Xi = 0) = 1 − p, with p unknown. Find a minimal sufﬁcient statistic for p.

278 CHAPTER 5 Point Estimation

Solution The ratio of the likelihoods is

p(x1 , . . . , xn ) p xi (1 − p)n− xi L(x1 , . . . , xn ) = = L(y1 , . . . , yn ) p(y1 , . . . , yn ) p yi (1 − p)n− yi

xi − yi p = . 1−p This ratio is to be independent of p, if and only if n

xi −

i=1

n

yi = 0

i=1

which implies n i=1

xi =

n

yi .

i=1

Therefore, g (X1 , . . . , Xn ) =

n

Xi

i=1

is a minimal sufficient statistic for p.

Example 5.5.9 Let X1 , . . . , Xn be a random sample from a U(0, θ) distribution. Find a minimal sufﬁcient statistic for θ.

Solution The likelihood function is

⎧ ⎨ 1, θn L= ⎩ 0,

if max(x1 , . . . , xn ) ≤ θ otherwise .

Denote by xmax = max(x1 , . . . , xn ), and ymax = max(y1 , . . . , yn ). Then, the ratio of the likelihood functions is ⎧ ⎪ ⎪ ⎪ ⎨

1, L(x1 , . . . , xn ) = 0, L(y1 , . . . , yn ) ⎪ ⎪ ⎪ ⎩ undeﬁned,

if max (xmax , ymax ) ≤ θ, if ymax < xmax , and ymax ≤ θ ≤ xmax , elsewhere.

Thus, the ratio will not depend on θ if and only if xmax = ymax . Therefore, a minimal sufficient statistic for θ is X(n) , the largest order statistic.

5.5 Other Desirable Properties of a Point Estimator 279

It is important to note that although we often can ﬁnd a single statistic that is minimal sufﬁcient for one parameter, this need not be the case (see Exercise 5.5.1). For most of the density functions that we consider, any unbiased estimator that is a function of a minimal sufﬁcient statistic will be a uniformly minimum variance unbiased estimator (UMVUE), that is, it will posses the smallest variance possible among unbiased estimators.

Example 5.5.10 Let X1 , . . . , Xn be a random sample from the normal distribution with known mean μ = μ0 and unknown variance σ 2 . Show that ni=1 (Xi − μ0 )2 is the minimal sufﬁcient statistic for σ 2 . Use this statistic to ﬁnd an MVUE of σ 2 .

Solution The ratio of the likelihoods is # $ exp − (xi − μ0 )2 /2σ 2 L(x1 , . . . , xn ) # $ = L (y1 , . . . , yn ) exp − (yi − μ0 )2 /2σ 2 ! = exp

3" 1 2 2− 2 ) ) (y (x . − μ − μ 0 0 i i 2σ 2

In order for this ratio to be free of σ 2 , we need

(yi − μ0 )2 =

(xi − μ0 )2 .

(Xi − μ0 )2 is minimal sufficient for σ 2 . Hence, Because E(Xi − μ0 )2 = σ 2 , we can see that (1/n) (Xi − μ0 )2 is an unbiased estimator of σ 2 . Because this is a function of a minimal sufficient statistic, (1/n) ni=1 (Xi − μ0 )2 is an MVUE of σ 2 .

EXERCISES 5.5 5.5.1.

Show that the maximum likelihood estimator for p, Yn /n in a binomial distribution is consistent.

5.5.2.

Show that Yn , the nth-order statistic from a U(0, θ) distribution, is a consistent estimator for θ.

5.5.3.

Let X1 , . . . , Xn be a random sample with EXi = μi , EXi2 = μ2 , and EXi4 = μ4 , all ﬁnite. Show that S 2 = (1/n) ni=1 (Xi − X)2 is a consistent estimator of σ 2 = Var(Xi ).

5.5.4.

Let X1 , . . . , Xn be a random sample from a population with pdf f (x) =

αxα−1 , 0,

for 0 < x < 1; α > 0 otherwise.

280 CHAPTER 5 Point Estimation

Is the method of moments estimator for α consistent? 5.5.5.

Let X1 , . . . , Xn be a random sample from an exponential population with parameter θ. Show that X is a consistent estimator of θ.

5.5.6.

Let X1 , . . . , Xn and Y1 , . . . , Yn be independent random samples from populations with means μ1 and μ2 variances σ12 and σ22 , respectively. Show that the difference X − Y is a consistent estimator of μ1 − μ2 .

5.5.7.

Let X1 , . . . , Xn be a random sample from a population with pdf f (x) =

1 (1−α)/α , αx

0,

for 0 < x < 1; α > 0 otherwise.

(a) Show that the maximum likelihood estimator of α is αˆ = − (1/n) (b) Is αˆ of part (a) an unbiased estimator of α? (c) Is αˆ of part (a) a consistent estimator of α? 5.5.8.

n

i=1 ln Xi .

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

x −x2 /(2α) , αe

0,

for x > 0 otherwise.

(a) Determine the maximum likelihood estimator αˆ of α. (b) Is αˆ of part (a) an unbiased estimator of α? (c) Is αˆ of part (a) a consistent estimator of α? 5.5.9.

Let X1 , . . . , Xn be a random sample from the uniform distribution on the interval (θ, θ + 1). Let θˆ 1 = X −

1 n , θˆ 2 = X(n) − , 2 n+1

where X(n) is the nth order statistic. Find the efﬁciency of θˆ 2 relative to θˆ 1 . 5.5.10.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population. Let θˆ 1 be the sample mean and θˆ 2 be the sample median. It is known that Var(θˆ 2 ) = (1.2533)2 (σ 2 /n). Find the efﬁciency of θˆ 2 relative to θˆ 1 .

5.5.11.

Let X1 , . . . , Xn be a random sample from an exponential population with parameter θ. Show that X is efﬁcient for θ.

5.5.12.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population. Show that 2 (n − 1) 4 MSE S 2 = σ , n2

where S 2 = (1/n)

n i=1

2 Xi − X .

5.5 Other Desirable Properties of a Point Estimator 281

5.5.13.

Prove % E

% & & ∂2 ln f (x) ∂ ln f (x) 2 = −E , ∂θ ∂θ 2

making suitable assumptions. 5.5.14.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ) population. (a) Show that the sample variance S 2 is an UMVUE for σ 2 when the value of μ is not known. (b) Show that the variance of S 2 is greater than the Cramér–Rao lower bound.

5.5.15.

Let X1 , . . . , Xn be a random sample from a U(0, θ) distribution. Let X(n) be the nth order statistic. (a) Show that θˆ 1 = X(n) , θˆ 2=2X, and θˆ 3= n+1 n X(n) are unbiased estimators of θ. (b) Find the efﬁciency of θˆ 1 relative to θˆ 2 . (c) Find the efﬁciency of θˆ 2 relative to θˆ 3 .

5.5.16.

Let X1 , . . . , Xn , (n ≥ 2) be a random sample from a distribution with pdf 1

f (x) = π[1+(x−θ)2 ] ,

−∞ < x < ∞,

−∞ < θ < ∞.

Show that the Cramér–Rao lower bound for a UBE of θ is 2/n. 5.5.17.

Let X1 , . . . , Xn , n > 4, be a random sample from a population with a mean μ and variance σ 2 . Consider the following three estimators of μ: θˆ 1 =

1 (X1 + 2X2 + 5X3 + X4 ) , 9

θˆ 2 =

1 2 1 1 X1 + X2 + X3 + . . . + Xn−1 + Xn , 5 5 5 (n − 3) 5

and θˆ 3 = X.

(a) Show that each of the three estimators is unbiased. (b) Find e(θˆ 2 , θˆ 1 ), e(θˆ 3 , θˆ 1 ), and e(θˆ 3 , θˆ 2 ). 5.5.18.

Find the Cramér–Rao lower bound for the variance of an unbiased estimator of θ, based on a sample of size n for the following pdfs: (i) f (x, θ) = θ12 xe−x/θ , x > 0, θ > 0. (ii) f (x, θ) = θxθ−1 ,

5.5.19.

0 < x < 1, θ > 0.

Let Y1 , . . . , Yn be a random sample from the uniform distribution over the interval (θ − 1, θ + 1). Show that the order statistics X1 = min(Yi ) and Xn = max(Yi ) are jointly sufﬁcient for θ. Also, show that X1 and Xn are jointly minimal for θ.

282 CHAPTER 5 Point Estimation

5.5.20.

Let X1 , . . . , Xn be a random sample from a normal distribution with unknown mean μ and known variance σ 2 . Find the maximum likelihood estimator of μ and show that it is a function of a minimal sufﬁcient statistic.

5.5.21.

Let X1 , . . . , Xn be a random sample from a normal distribution with unknown mean μ and unknown variance σ 2 . Show that ni=1 Xi and ni=1 Xi2 are jointly minimal sufﬁcient for μ and σ 2 . Also show that X and S 2 are UMVUEs for μ and σ 2 .

5.5.22.

Let X1 , . . . , Xn be a random sample from the Weibull density ⎧ ⎨ 2x e−x2 /α, f (x) = α ⎩ 0,

x>0 otherwise.

Find an UMVUE for α. 5.5.23.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. Find a minimal sufﬁcient statistic for λ.

5.5.24.

Let X1 , . . . , Xn be a random sample from a gamma distribution with parameters α and β, both unknown. Find minimal sufﬁcient statistics for the parameters α and β.

5.5.25.

Let X1 , . . . , Xn be a random sample from a distribution with density function f (x) =

ex−β , 0,

x≥β otherwise.

Find an UMVUE for β. 5.5.26.

Let X1 , . . . , Xn be a random sample from the exponential distribution with pdf f (x) =

1 −x/β , βe

x>0

0,

otherwise.

Show that X is an UMVUE for β. Also show that 5.5.27.

n n+1

2

X is an MVUE for β2 .

Let X1 , . . . , Xn be a random sample from a Rayleigh distribution with pdf f (x) =

2x e−x2 /β , β

x>0

0,

otherwise.

Find an UMVUE for β.

5.6 CHAPTER SUMMARY In this chapter we have discussed the basic concepts of point estimation. Two methods of ﬁnding point estimators were described—the method of moments and the method of maximum likelihood. We have seen that the maximum likelihood estimators possess the invariance property, which states that if θˆ is a maximum likelihood estimator of the parameter θ, then h(θˆ ) is a maximum likelihood estimator

5.7 Computer Examples 283

for h(θ). Some desirable properties of the point estimators that we have discussed are unbiasedness, consistency, efﬁciency, and sufﬁciency. Unbiasedness means that the expected value of the sample statistic (the mean of its probability distribution) should be equal to the parameter. Unbiasedness guards against consistently producing under- or overestimates of the parameter in repeated sampling. If the estimator is consistent, then, as the sample size increases, the estimator can be expected to get closer and closer to the population parameter. Efﬁcient estimators have the lowest variance among all other estimators. A sufﬁcient estimator is a “good” estimator of the population parameter θ in the sense that it depends on fewer data values. We will now list some of the key deﬁnitions introduced in this chapter. ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Method of moments Likelihood function Maximum likelihood equations Unbiased estimator Mean square error Minimum variance unbiased estimator Consistent estimator Efﬁciency Uniformly minimum variance unbiased estimator Efﬁcient estimator Sufﬁcient estimator Jointly sufﬁcient Minimal sufﬁcient statistic

In this chapter, we have also learned the following important concepts and procedures. ■ ■ ■ ■ ■ ■

The method of moments procedure Procedure to ﬁnd MLE Procedure to test for consistency Procedure to test relative efﬁciency Cramér–Rao procedure to test for efﬁciency Procedure to verify sufﬁciency

5.7 COMPUTER EXAMPLES Because in the earlier chapters we have already given steps to obtain summary statistics such as the mean and variance using SPSS and SAS, we could use those commands to obtain point estimates as we will do with Minitab. Therefore, we will not give separate subsections for SPSS and SAS procedures. The following examples illustrate Minitab procedures.

Example 5.7.1 Generate 50 sample points from an N(4, 4) distribution and ﬁnd the descriptive statistics. Obtain an unbiased and sufﬁcient estimate of μ.

284 CHAPTER 5 Point Estimation

Solution Because we know that the sample mean x is an unbiased and sufficient estimate of the population mean μ, we only need to find the sample mean of the generated data. Calc > Random Data > Normal . . . > Type 50 in Generate __ rows of data > Store in column(s): type C1 > type in Mean: 4.0 and in Standard deviation: 2.0 > click OK . The following is one possible output. C1 4.76039 5.16925 3.25593 2.80955 2.43494 1.76723 2.86329 5.38503

5.07819 3.68845 2.66181 4.19032 2.01465 3.15460 5.97599

4.85263 6.40513 1.01352 4.65449 4.02358 4.81882 7.75170

4.08032 6.13801 5.82506 3.48680 8.22997 0.36250 7.10011

6.77772 7.20015 6.04212 6.39083 2.44516 0.85002 6.61681

4.21677 2.41415 5.22235 6.56357 0.39563 14.47052 0.97982

1.51811 3.50008 5.29924 1.32281 3.78948 0.79586 4.01400

Now follow the procedure to obtain the descriptive statistics from Example 1.8.3 to obtain Descriptive Statistics Variable N Mean Median TrMean StDev SE Mean C1 50 4.116 4.135 4.115 2.047 0.289 Variable Minimum Maximum Q1 Q3 C1 0.362 8.230 2.443 5.863

We can see that the unbiased and sufﬁcient estimate of the mean μ for these data is x = 4.116.

Example 5.7.2 Generate 35 samples from a U(0, 5) distribution and using the descriptive statistics command, ﬁnd the maximum likelihood estimate for this data.

Solution We know that for a random sample X1 , . . . , Xn from U(0, θ), the MLE, θˆ = max(Xi ) = X(n) , the nth order statistic. We can use the following steps to obtain the estimate.

Calc > Random Data > Uniform. . . > Type 35 in Generate __ rows of data > Store in column(s): type C1 > type in Lower end point: 0.0 and in Upper end point: 5.0 > click OK One possible output is given below.

Projects for Chapter 5 285

C1 4.32848 0.07934 2.92537 4.20844 3.25272

4.79402 3.12453 2.39721 3.75506 4.61083

0.34515 1.69073 4.84440 4.56626 3.06527

0.08428 3.44003 1.79129 3.50280 2.34003

1.93000 0.47447 4.38718 1.95689 0.40877

0.27878 2.28072 3.60697 0.56969 2.52708

3.12992 0.49205 0.94159 1.02543 1.44525

Now follow the procedure to obtain the descriptive statistics from Example 1.8.3 to obtain Descriptive Statistics Variable N Mean Median TrMean StDev SE Mean C1 35 2.417 2.397 2.413 1.541 0.260 Variable Minimum Maximum Q1 Q3 C1 0.079 4.844 0.942 3.607 Therefore the MLE θˆ = 4.884.

For the previous example, it should be noted that because we are generating random data, each time we follow this procedure, we will be getting different answers. When we have a particular data set, enter the data in C1 and just use the procedure to ﬁnd the descriptive statistics. For other distributions, click the appropriate distribution in Random Data.

PROJECTS FOR CHAPTER 5 5A. Asymptotic Properties In general, we do not have a single sample with one estimator of the unknown parameter θ. Rather, we will have a general formula that deﬁnes an estimator for any sample size. This gives a sequence of estimators of θ: θˆ = hn (X1 , . . . , Xn ) , n = 1, 2, . . . ..

In this case, we can deﬁne the following asymptotic properties: (i) The sequence of estimators θˆ n is said to be asymptotically unbiased for θ if bias θˆ n → 0 as n → ∞. (ii) Suppose θˆ n and yˆ n are two sequences of estimators that are asymptotically unbiased for θ. The asymptotic relative efﬁciency of θˆ n to yˆ n is deﬁned by Var θˆ n . lim n Var yˆ n

(a) Show that θˆ n is asymptotically unbiased if and only if E θˆ n → θ as n → ∞.

286 CHAPTER 5 Point Estimation

(b) Let X1 , . . . , Xn be a random sample from a distribution with unknown mean μ and variance σ 2 . It is known that the method of moments estimators for μ and σ 2 are, respectively, the 2 sample mean X and S 2n = (1/n) ni=1 Xi − X = ((n − 1)/n) Sn2 , where Sn2 is the sample variance. (i) Show that S 2n is an asymptotically unbiased estimator of σ 2 . (ii) Show that the asymptotic relative efﬁciency of S 2n to Sn2 is 1. (iii) Show that MSE S 2n < MSE Sn2 . Thus, Sn2 is unbiased but S 2n has a smaller mean square error. However, it should be noted that the difference is very small and approaches zero as n becomes large.

5B. Robust Estimation The estimators derived in this chapter are for particular parameters of a presumed underlying family of distributions. However, if the choice of the underlying family of distributions is based on past experience, there is a possibility that the true population will be slightly different from the model used to derive the estimators. Formally, a statistical procedure is robust if its behavior is relatively insensitive to deviations from the assumptions on which it is based. If the behavior of an estimator is taken as its variance, a given estimator may have minimum variance for the distribution used, but it may not be very good for the actual distribution. Hence, it is desirable for the derived estimators to have small variance over a range of distributions. We call such estimators robust estimators. The following illustrates how the variance of an estimator can be affected by deviations from the presumed underlying population model. Consider estimating the mean of a standard normal distribution. Let X1 , . . . , Xn be a random sample from a standard normal distribution. Suppose the population actually follows a contaminated normal distribution. That is, for 0 ≤ δ ≤ 1, 100 (1 − δ) % of the observations come from an N(0, 1) distribution and the remaining 100δ% of observations come from an N(0, 5) distribution. We already know that the minimum variance unbiased estimator of the mean μ of an uncontaminated normal distribution is the sample mean. A less effective alternative would be the sample median. (a) Conduct a simulation study with sample size n that takes, say, 5000 random samples of 100 observations each. Find the mean and median. Also ﬁnd the sample variance of each. For various values of δ, say 0.0, 0.01, 0.05, 0.1, 0.2, 0.3, and 0.4, create a table of variances of sample mean and sample variance. Compare the variances as the value of δ increases. (b) The aim of robust estimation is to derive estimators with variance near that of the sample mean when the distribution is standard normal while having the variance remain relatively stable as δ increases. One such estimator is the α − trimmedmean. Let 0 ≤ α ≤ 0.5, and deﬁne k = [nα], where [x] is the greatest integer that is less than or equal to x. For the ordered sample, discard the k highest and lowest observations and ﬁnd the mean of the remaining n − k observations. That is, let X(1) ≤ X(2) ≤ . . . ≤ X(n) be the ordered sample, and deﬁne Xα =

X(1+k) ≤ X(2+k) ≤ . . . ≤ X(n+k) . n − 2k

For the values of δ and the samples in part (a), compute the mean and the 0.05-, 0.1-, 0.25-, and 0.5-trimmed means. Discuss the robustness.

Projects for Chapter 5 287

5C. Numerical Unbiasedness and Consistency (a) Run the simulation of a normal experiment with increasing sample size. Numerically show the unbiased and consistent properties of the sample mean. Run the experiment at least up until n = 1000. (b) Repeat the experiment of part (a), now with an exponential distribution.

5D. Averaged Squared Errors (ASEs) Generate 25 samples of size 40 from a normal population with μ = 10, and σ 2 = 4. For each of the 25 samples: 40

(a) Compute: x, s2 =

40

(xi −x)2

i=1

,

39

=

s12

40

(xi −x)2

i=1

, and

40

s22

=

(xi −x)2

i=1

41

.

(b) Compute the average squared error (ASE) for each of the estimates s2 , s12 , s22 as follows. !! K " " 2 2 (xi − x)2 /39 for K = 1, 2, . . . , 25; and Ks be the sample variance for the Let Ks = i=1

Kth sample. Then, the average squared error is 2 25 2 Ks − σ 2 ASE =

i=1

.

25

Repeat this procedure for the other two estimators. Compare the three ASEs and check which has the least ASE. (c) Repeat (a) and (b) with a sample size of 15.

5E. Alternate Method of Estimating the Mean and Variance (a) Consider the following alternative method of estimating μ and σ 2 . We sample sequentially, and at each stage we compute the estimates of μ and σ 2 as follows. Let X1 , . . . , Xn , Xn+1 be the sample values. Compute n

Xn =

n+1

Xi

i=1

n

, Xn+1 =

i=1

n+1

n+1

Xi − Xn

2 Sn+1 =

i=1

n

n

Xi − Xn

Xi , Sn2 =

i=1

2 .

The sequential procedure is stopped when 2 2 ≤ 0.01. Sn − Sn+1

This will also determine the sample size. (b) Compare the sample sizes and estimates in 5D and 5E.

n−1

2 , and

288 CHAPTER 5 Point Estimation

5F. Newton–Raphson in One Dimension For a given function g(x), suppose we need to solve g(θ) = 0. Using the ﬁrst-order Taylor expansion, dg , and setting g(θ) = 0, we get θ ≈ x − gg(x) g(θ) ≈ g(x) + (θ − x)g (x), where g (x) = dx (x) . Thus, starting with an initial guess solution x, the guess is updated by θ using the previous formula. This derivation is the basis for the Newton–Raphson iterative method for obtaining the solution of g(θ) = 0. This is given by g (θn ) , θ(n+1) = θn − g (θn )

n ≥ 0,

where θn is the value of θ at the nth iteration, starting with the initial guess, θ0 . For a good approximation of the solution, the choice of θ0 is important. The convergence of this algorithm cannot be guaranteed. For the MLE, we want to ﬁnd a solution of g(θ) =

dL = 0, dθ

where L = L (θ) is the likelihood function of the random sample X1 , . . . , Xn . An iterative algorithm for ﬁnding the MLE can be given by dL (θn ) dθ , θ(n+1) = θn − 2 d L (θn ) dθ 2

n ≥ 0.

Write a computer program to ﬁnd the MLE of α for a gamma distribution with parameters α and β.

5G. The Empirical Distribution Function The estimators in this chapter yield a single real value (point estimate) for each parameter. In Chapter 6, we will learn about so-called interval estimates. In this project, we use an estimation procedure that estimates the whole distribution function, F , of a random variable X. We now deﬁne the empirical distribution. The empirical distribution function for a random sample X1 , . . . , Xn from a distribution F is the function deﬁned by Fn (x) =

1 #{i, 1 ≤ i ≤ n : Xi ≤ x}. n

It can be shown that nFn (x) is a binomial random variable with E [Fn (x)] = F (x) and Var [Fn (x)] =

1 F (x) [1 − F (x)] . n

Also, by the strong law of large numbers, for each real number x, lim Fn (x) = F (x) with probability 1.

n→∞

Projects for Chapter 5 289

One of the tests to determine whether a random sample comes from a speciﬁc distribution is the Kolmogorov–Smirnov (K-S) test. The K-S test is based on the maximum distance between the empirical distribution function and the actual cumulative distribution function of this speciﬁc distribution (such as, say, the normal distribution). Using the method of Project 4A (or using any statistical software), generate 100 sample points from a normal distribution with mean 2 and variance 9. Graph the empirical distribution function for this sample. Compare this graph with the graph of the N(2, 9) distribution.

This page intentionally left blank

Chapter

6

Interval Estimation Objective: To learn some statistical methods that are commonly used to obtain interval estimation or conﬁdence limits of the unknown population parameters. 6.1 Introduction 292 6.2 Large Sample Conﬁdence Intervals: One Sample Case 300 6.3 Small Sample Conﬁdence Intervals for μ 310 6.4 A Conﬁdence Interval for the Population Variance 315 6.5 Conﬁdence Interval Concerning Two Population Parameters 6.6 Chapter Summary 330 6.7 Computer Examples 330 Projects for Chapter 6 334

321

Karl Pearson (Source: http://www-history.mcs.st-and.ac.uk/∼ history/PictDisplay/Pearson.html)

Karl Pearson (1857–1936) is considered the founder of the 20th-century science of statistics. Pearson has contributed in several different ﬁelds such as anthropology, biometry, eugenics, scientiﬁc method, and statistical theory. He applied statistics to biological problems of heredity and evolution.

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

291

292 CHAPTER 6 Interval Estimation

He is the author of The Grammar of Science, the three volumes of The Life, Letters and Labors of Francis Galton, and The Ethic of Free Thought. Pearson was the founder of the statistical journal Biometrika. In 1900, he published a paper on the chi-square goodness of ﬁt test. This is one of Pearson’s most signiﬁcant contributions to statistics. In 1893, Pearson coined the term “standard deviation.”

6.1 INTRODUCTION In the previous chapter, we studied methods for ﬁnding point estimators for the population parameters. In general the estimates will differ from the true parameter values by varying amounts depending on the sample values obtained. In addition, the point estimates do not convey any measure of reliability. In this chapter, we discuss another type of estimation, called an interval estimation. Although point estimators are useful, interval estimators convey more information about the data that are used to obtain the point estimate. The purpose of using an interval estimator is to have some degree of conﬁdence of securing the true parameter. For an interval estimator of a single parameter θ, we will use the random sample to ﬁnd two quantities L and U such that L < θ < U with some probability. Because L and U depend on the sample values, they will be random. This interval (L, U) should have two properties: (1) P(L < θ < U) is high, that is, the true parameter θ is in (L, U) with high probability, and (2) the length of the interval (L, U) should be relatively narrow on the average. In summary, interval estimation goes a step beyond point estimation by providing, in addition to the estimating interval (L, U), a measure of one’s conﬁdence in the accuracy of the estimate. Interval estimators are called conﬁdence intervals and the limits are called U and L, the upper and lower conﬁdence limits, respectively. The associated levels of conﬁdence are determined by speciﬁed probabilities. The width of the conﬁdence interval reﬂects the amount of variability inherent in the point estimate. Thus, our objective is to ﬁnd a narrow interval with high probability of enclosing the true parameter, θ. We will restrict our attention to single parameter estimation. The probability that a conﬁdence interval will contain the true parameter θ is called the conﬁdence coefﬁcient. The conﬁdence coefﬁcient gives the fraction of the time that the constructed interval will contain the true parameter, under repeated sampling. Let L and U be the lower and upper conﬁdence limits for a parameter θ based on a random sample X1 , . . . , Xn . Both L and U are functions of the sample. We can write the interval estimate of θ as P (L ≤ θ ≤ U) = 1 − α

and we read it as we are (1 − α)100% conﬁdent that the true parameter θ is located in the interval (L, U ). The number 1 − α is the conﬁdence coefﬁcient, and the interval (L, U ) is referred to as a (1 − α)100% conﬁdence interval ((1 − α)100% CI) for θ. Thus, if we want a 95% conﬁdence interval for, say, population mean μ, then α = 0.05. Note that for the discrete random variables, we may not be able to ﬁnd a lower bound L and an upper bound U such that the probability, P (L ≤ θ ≤ U ), is exactly (1 − α). In such a case we can choose L and U such that P (L ≤ θ ≤ U ) ≥ 1 − α. How do we ﬁnd the conﬁdence interval? For this, we use the error structure of the point estimator to obtain this interval. For instance, we know that the sample mean, X, is a point estimate (MLE or

6.1 Introduction 293

unbiased estimator) of the population mean μ. In this case, we know that the standard error of X is √ σ/ n. If the sample came from a normal population, then for a 95% conﬁdence interval for the mean, multiply the standard error by 1.96 and then add and subtract this product from the sample mean. From this we can also observe that, if everything else remains the same, the size of the conﬁdence interval reduces as the sample size increases.

Example 6.1.1 As part of a promotion, the management of a large health club wants to estimate average weight loss for its members within the ﬁrst 3 months after joining the club. They took a random sample of 45 members of this health club and found that they lost an average of 13.8 pounds within the ﬁrst 3 months of membership with a sample standard deviation of 4.2 pounds. Find a 95% conﬁdence interval for the true mean. What if a random sample of 200 members of this health club also resulted in the same sample mean and sample standard deviation?

Solution Here a point estimate of the true mean μ is the sample mean x = 13.8 pounds. Because n = 45 is large enough, we can use the Central Limit Theorem and use approximate normality for the distribution of X √ with mean μ and the approximate standard error (4.2/ 45) = 0.626. Thus a 95% confidence interval is 13.8 ± (1.96)(0.626), resulting in the interval (12.57, 15.03). Thus, on average, with 95% confidence, one can expect the true mean to lie in this interval. √ For n = 200, the standard error is (4.2/ 200) ≈ 0.297. Thus a 95% confidence interval is 13.8 ± (1.96)(0.297) resulting in the interval (13.22, 14.38). Thus the more sample values (that is, the more information) we have, the tighter (smaller width) the interval. The previous example was built on our knowledge of the sampling distribution of the sample mean. What if the sampling distribution of the statistic we are interested in is not readily available? More generally, our success in building confidence intervals for an estimate of a parameter depends on identifying a quantity known as the pivot. We now describe this method.

6.1.1 A Method of Finding the Confidence Interval: Pivotal Method The pivotal method is a general method of constructing a conﬁdence interval using a pivotal quantity. This relies on our knowledge of sampling distributions. Here we have to ﬁnd a pivotal quantity with the following two characteristics: (i) It is a function of the random sample (a statistic or an estimator θˆ ) and the unknown parameter θ, where θ is the only unknown quantity, and (ii) It has a probability distribution that does not depend on the parameter θ. From (i) and (ii), it is important to note that the pivotal quantity depends on the parameter, but its distribution is independent of the parameter. Let X1 , . . . , Xn be a random sample and let θˆ be a reasonable point estimate of θ. For instance, θˆ could be the maximum likelihood (or some other) estimator of θ. In general, ﬁnding a pivotal quantity may not be easy. However, if θˆ is the sample mean X or sample variance S 2 , we could ﬁnd a pivotal quantity with known sampling distributions. Suppose p (θˆ , θ) is a pivotal quantity with known probability distribution that is independent of θ.

294 CHAPTER 6 Interval Estimation

(Usually, the probability distribution of the pivotal quantity will be standard normal, t, χ2 , or F -distribution.) The following are some of the standard pivotal quantities: If the sample X1 , . . . , Xn is from N(μ, σ 2 ) √ (i) With μ unknown and σ known, let X be the sample mean. Then the pivot is (X−μ)/(σ/ n), which has an N(0, 1) distribution (see comments after Corollary 4.2.2). √ (ii) With μ unknown and σ unknown, then the pivot is (X − μ)/(S/ n), which has a tdistribution with (n − 1) degrees of freedom (see Theorem 4.2.9). If n is large, using CLT, the distribution of the pivot is approximately N(0, 1). (iii) If σ 2 is unknown, then the pivot is (n − 1)S 2 /σ 2 , which has a χ2 -distribution with (n − 1) degrees of freedom (see Theorem 4.2.8). For a given value of α, (0 < α < 1), and constants a and b, with (a < b), let P (a ≤ p (θˆ , θ) ≤ b) = 1 − α.

Hence, given θˆ , the inequality is solved for θ to obtain a region of θ values, usually an interval corresponding to the observed θˆ -value. The following examples illustrate the pivotal method.

Example 6.1.2 Suppose we have a random sample X1 , . . . , Xn from N(μ, 1). Construct a 95% conﬁdence interval for μ.

Solution Here the confidence coefficient is 0.95. We know that the maximum likelihood estimator of μ is X, which has an N(μ, 1/n) distribution. Note that this distribution depends on the unknown value of μ, and hence X cannot be a pivot. However, taking the z-transform of X, we obtain the pivotal quantity as Z=

X−μ X−μ √ = √ σ/ n 1/ n

which has an N(0, 1) distribution that is a function of the sample measurements and does not depend on μ. Hence, this Z can be taken as a pivot p (θˆ , θ). Now to find a and b such that P (a ≤ Z = p (θˆ , θ) ≤ b) = 0.95. One such choice is to find the value of a such that p (−a ≤ Z ≤ a) = 0.95. From the normal table, P (−zα/2 ≤ Z ≤ zα/2 ) = 0.95, where zα/2 represents the value of z with tail area α/2. This implies a = zα/2 = 1.96. Hence, P (−1.96 ≤ Z ≤ 1.96) = 0.95 or, using the definition of Z and solving for μ, we obtain

1.96 1.96 = 0.95. P X− √ ≤μ≤X+ √ n n

6.1 Introduction 295 √ √ Hence, a 95% confidence interval for μ is (X − (1.96/ n), X + (1.96/ n)). Thus, the lower confidence √ √ limit L is X − (1.96/ n) and the upper confidence limit U is X + (1.96/ n).

From the derivation of Example 6.1.1, it follows that

σ P X − μ < zα/2 √ = 1 − α. n

Thus, for a normal population with known variance σ 2 , if X is used as an estimator of the true mean √ μ, the probability that the error will be less than zα/2 σ/ n is 1 − α. It is important to note that there is some arbitrariness in choosing a conﬁdence interval for a given problem. There may be several pivotals for θˆ that could be used. Also, it is not necessary to allocate equal probability to the two tails of the distribution; however, doing so may result in the shortest length conﬁdence interval for a given conﬁdence coefﬁcient. When we make the statement of the form

1.96 1.96 P X− √ ≤μ≤X+ √ = 0.95, n n

we mean that, in an inﬁnite series of trials in which repeated samples of size n are drawn from the same population and 95% conﬁdence intervals for μ are calculated by the same method for each of the samples, the proportion of intervals that actually include μ will be 0.95. Figure 6.1 illustrates this idea, where the vertical line represents the position of true mean μ and each of the horizontal lines represents a 95% conﬁdence interval of the sample, 20 samples of size n are taken. √ √ A statement of the type P (x − (1.96/ n) ≤ μ ≤ x + (1.96/ n)) = 0.95, where x is the observed sample mean, is misleading. Once we calculate this interval using a particular sample, then either this interval contains the true mean μ or not, and hence the probability will be either 0 or 1. Thus, the correct interpretation of conﬁdence interval for the population mean is that if samples of the same size, n, are drawn repeatedly from a population, and a conﬁdence interval is calculated from each sample, then 95% of these intervals should contain the population mean. This is often stated as √ √ “We are 95% conﬁdent that the true mean is in the interval (X − zα/2 (σ/ n), X + zα/2 (σ/ n)).” Thus,

■ FIGURE 6.1 95% confidence intervals for μ.

296 CHAPTER 6 Interval Estimation

PDF of P

■ FIGURE 6.2 Probability density of the pivot.

the correct interpretation requires the conﬁdence limits to be variables. This concept of conﬁdence interval is attributed to Neyman. We can follow the accompanying procedure to ﬁnd a conﬁdence interval for the parameter θ.

PROCEDURE TO FIND A CONFIDENCE INTERVAL FOR θ USING THE PIVOT 1. Find an estimator θˆ of θ: usually MLE of θ works. 2. Find a function of θ and θˆ , p(θ, θˆ ) (pivot), such that the probability distribution of p(. , . ) does not depend on θ. 3. Find a and b such that P(a ≤ p(θ, θˆ ) ≤ b) = 1 − α. Choose a and b such that P(p(θ, θˆ )≤a) = α/2 and P(p(θ, θˆ ) ≥ b) = α/2 (see Figure 6.2 where the shaded area in each side is α/2). 4. Now, transform the pivot conﬁdence interval to a conﬁdence interval for the parameter θ. That is, work with the inequality in step 3 and rewrite it as P(L ≤ θ ≤ U) = 1 − α, where L is the lower conﬁdence limit and U is the upper conﬁdence limit.

The following example is given to show that the success of ﬁnding a pivotal quantity depends on our ability to ﬁnd the right transformation of the statistic and its distribution so that the transformed variable is a pivot.

Example 6.1.3 Suppose the random sample X1 , . . . , Xn has U(0, θ) distribution. Construct a 90% conﬁdence interval for θ and interpret. Identify the upper and lower conﬁdence limits.

Solution From Example 5.3.4, we know that U = max Xi 1≤i≤n

6.1 Introduction 297

is the MLE of θ. The random variable U has the pdf fU (u) = nu n−1 /θ n ,

0 ≤ u ≤ θ.

This is not independent of the parameter θ. Let Y = U/θ, then (using the Jacobians described in Chapter 3) the pdf of Y is given by fY (y) = ny n−1 ,

0 ≤ y ≤ 1.

Hence, Y satisfies the two characteristics of the pivotal quantity. Thus, Y = U/θ is a pivot. Now, we have to find a and b such that p (a ≤

U ≤ b) = 0.90. θ

pdf of Y

0.05

0.05

y 1

0

To find a and b we use the cdf of Y, FY (y) = yn , 0 ≤ y ≤ 1, as follows. F (y )

0.95

0.05

0

a

y

b1

FY (a) = 0.05

and

FY (b) = 0.95

an = 0.05

and

b n = 0.95

which implies that

298 CHAPTER 6 Interval Estimation

resulting in a=

√ n

0.05

and

b=

√ n 0.95.

Write

P

√ n

0.05

0, x > 0. Construct a 95% conﬁdence interval for θ and interpret. [Hint: Recall that ni=1Xi has a gamma distribution with α = n, β = θ.]

6.1.8.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ. (a) Construct a 90% conﬁdence interval for λ. (b) Suppose that the number of raisins in a bowl of a particular brand of cereal is observed to be 25. Assuming that the number of raisins in a bowl is Poisson distributed, estimate the expected number of raisins per bowl with a 90% conﬁdence interval. (c) How many bowls of cereal need to be sampled in order to estimate the expected number of raisins per bowl with a standard error of less than 0.2?

6.1.9.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). (a) Construct a (1 − α)100% conﬁdence interval for μ when the value of σ 2 is known. (b) Construct a (1 − α)100% conﬁdence interval for μ when the value of σ 2 is unknown.

6.1.10.

Let X1 , . . . , Xn be a random sample from an N(μ1 , σ 2 ) population and Y1 , . . . , Yn be an independent random sample from an N(μ2 , σ 2 ) distribution where σ 2 is assumed to be known. Construct a (1 − α)100% interval for (μ1 − μ2 ). Interpret its meaning.

300 CHAPTER 6 Interval Estimation

6.1.11.

Let X1 , . . . , Xn be a random sample from a uniform distribution on [θ, θ + 1]. Find a 99% conﬁdence interval for θ, using an appropriate pivot.

6.2 LARGE SAMPLE CONFIDENCE INTERVALS: ONE SAMPLE CASE If the sample size is large, then by the Central Limit Theorem, certain sampling distributions can be assumed to be approximately normal. That is, if θ is an unknown parameter (such as μ, p, (μ1 − μ2 ), (p1 − p2 )), then for large samples, by the Central Limit Theorem, the z-transform z=

θˆ − θ σθˆ

possesses an approximately standard normal distribution, where θˆ is the MLE of θ and σθˆ is its standard deviation. Then as in Example 6.1.1, the pivotal method can be used to obtain the conﬁdence interval for the parameter θ. For θ = μ, n ≥ 30 will be considered large; for the binomial parameter p, n is considered large if np, and n(1 − p) are both greater than 5. PROCEDURE TO CALCULATE LARGE SAMPLE CONFIDENCE INTERVAL FOR θ 1. Find an estimator (such as the MLE) of θ, say θˆ . 2. Obtain the standard error, σθˆ of θˆ . 3. Find the z-transform z = (θˆ − θ)/σθˆ . Then z has an approximately standard normal distribution. 4. Using the normal table, ﬁnd two tail values −zα/2 and zα/2 . 5. An approximate (1 − α)100% conﬁdence interval for θ is θˆ − zα/2 σθˆ , θˆ + zα/2 σθˆ , that is, P θˆ − zα/2 σθˆ ≤ θ ≤ θˆ + zα/2 σθˆ = 1 − α. 6. Conclusion: We are (1 −α)100% conﬁdent that the true parameter θ lies in the interval θˆ − zα/2 σθˆ , θˆ + zα/2 σθˆ .

Example 6.2.1 Let θˆ be a statistic that is normally distributed with mean θ and standard deviation σθˆ , where σ is assumed to be known. Find a conﬁdence interval for θ that possesses a conﬁdence coefﬁcient equal to 1 − α.

Solution The z-transform of θˆ is Z=

θˆ − θ σθˆ

and has a standard normal distribution. Select two tail values −zα/2 and zα/2 such that P (−zα/2 ≤ Z ≤ zα/2 ) = 1 − α. Because of symmetry, this is the shortest interval that contains the area 1 − α. Then, P (θˆ − zα/2 σθˆ ≤ θ ≤ θˆ + zα/2 σθˆ ) = 1 − α.

6.2 Large Sample Conﬁdence Intervals: One Sample Case 301

Therefore, the confidence limits of θ are θˆ − zα/2 σθˆ and θˆ + zα/2 σθˆ . Hence, (1 − α)100% confidence interval for θ is given by θˆ ± zα/2 σθˆ .

If in particular for a large sample of size n, let θˆ = X be the sample mean. Then the large sample (1 − α)100% conﬁdence interval for the population mean μ is σ S X ± zα/2 √ X ± zα/2 √ n n

where S is a point estimate of σ. That is,

S S P X − zα/2 √ ≤ μ ≤ X + zα/2 √ = 1 − α. n n

As we have seen in Section 6.1, the correct interpretation of this conﬁdence interval is that in a repeated √ sampling, approximately (1 − α)100% of all intervals of the form X ± zα/2 (S/ n) include μ, the true mean. Suppose x and s are the sample mean and the sample standard deviation, respectively, for a particular set of n observed sample values x1 , . . . , xn . Then we do not know whether the particular √ √ interval (x − zα/2 (s/ n), x − zα/2 (s/ n)) contains μ. However, the procedure that produced this interval does capture the true mean in approximately (1 − α)100% of cases. This interpretation will be assumed hereafter, when we make a statement such as, “We are 95% conﬁdent that the true mean will lie in the interval (74.1, 79.8).”

Example 6.2.2 Two statistics professors want to estimate average scores for an elementary statistics course that has two sections. Each professor teaches one section and each section has a large number of students. A random sample of 50 scores from each section produced the following results: (a) Section I: x1 = 77.01, s1 = 10.32 (b) Section II: x2 = 72.22, s2 = 11.02 Calculate 95% conﬁdence intervals for each of these three samples.

Solution Because n = 50 is large, we could use normal approximation. For α = 0.05, from the normal table: zα/2 = z0.025 = 1.96. The confidence intervals are: (a) We have

s1 10.32 x1 ± zα/2 √ = 77.01 ± 1.96 √ n 50 which gives a 95% confidence interval (74.149, 79.871). (b) We can compute

s2 11.02 x2 ± zα/2 √ = 72.22 ± 1.96 √ n 50 which gives the interval (69.165, 75.275).

302 CHAPTER 6 Interval Estimation √ It may be noted that if the population is normal with a known variance σ 2 , we can use X ± zα/2 (σ/ n) as the conﬁdence interval for the population mean μ, irrespective of the sample size. However, if σ 2 is √ unknown, in order to use X ± zα/2 (s/ n) as an approximate conﬁdence interval for μ, the sample size has to be large for the Central Limit Theorem to hold. However to use this approximate procedure, we do not need the condition that samples arise from a normal distribution. We will consider sample size to be large if n ≥ 30 (applicable to estimators of the mean). If not, we shall use the small sample procedure discussed in the next section.

Example 6.2.3 Fifteen vehicles were observed at random for their speeds (in mph) on a highway with speed limit posted as 70 mph, and it was found that their average speed was 73.3 mph. Suppose that from past experience we can assume that vehicle speeds are normally distributed with σ = 3.2. Construct a 90% conﬁdence interval for the true mean speed μ, of the vehicles on this highway. Interpret the result.

Solution Because the population is given to be normal with standard deviation σ = 3.2, sample size need not be large given x = 73.3 and σ = 3.2. Here, n = 15, and α = 0.10. Thus, zα/2 = z0.05 = 1.645. Hence, a 90% confidence interval for μ is given by 3.2 3.2 73.3 − 1.645 √ < μ < 73.3 + 1.645 √ 15 15 or 71.681 < μ < 74.919. Interpretation: We are 90% confident that the true mean speed μ of the vehicles on this highway is between 71.681 and 74.919.

6.2.1 Confidence Interval for Proportion, p Consider a binomial distribution with parameter p. Let X be the number of successes in n trials. Then the maximum likelihood estimator pˆ of p is pˆ = X/n. It can be shown, using the procedure outlined at the beginning of this section, that an approximate large sample (1 − α)100% conﬁdence interval for p is

8 pˆ − zα/2

p(1 ˆ − p) ˆ , pˆ + zα/2 n

8

p(1 ˆ − p) ˆ . n

That is, P pˆ − zα/2

8

p(1 ˆ − p) ˆ < p < pˆ + zα/2 n

8

p(1 ˆ − p) ˆ n

= 1 − α.

A natural question is: “How do we determine the sample size that we have is sufﬁcient for the normal approximation that is used in the foregoing formula?” There are various rules of thumb that are used to determine the adequacy of the sample size for normal approximation. Some of the popular rules

6.2 Large Sample Conﬁdence Intervals: One Sample Case 303 are that np and n(1 − p) should be greater than 10, or that pˆ ± 2 p(1 ˆ − p)/n ˆ should be contained in the interval (0, 1), or np (1 − p) ≥ 10, etc. All of these rules perform poorly when p is nearer to 0 or 1. Recently, there have been many works on coverage analysis for conﬁdence intervals. We refer to a survey article by Lee et al. for more details on this topic. For simplicity of calculations, we will use the rule that np and n(1 − p) are both greater than 5.

Example 6.2.4 An auto manufacturer gives a bumper-to-bumper warranty for 3 years or 36,000 miles for its new vehicles. In a random sample of 60 of its vehicles, 20 of them needed ﬁve or more major warranty repairs within the warranty period. Estimate the true proportion of vehicles from this manufacturer that need ﬁve or more major repairs during the warranty period, with conﬁdence coefﬁcient 0.95. Interpret.

Solution Here we need to find a 95% confidence interval for the true proportion, p. Here, pˆ = 20/60 = 1/3. For α = 0.05, zα/2 = z0.025 = 1.96. Hence, a 95% confidence interval for p is ; < 8 2 < 1 = 3 p(1 ˆ − p) ˆ 1 3 = ± 1.96 pˆ ± zα/2 n 3 60 which gives the confidence interval as (0.21405, 0.45262). That is, we are 95% confident that the true proportion of vehicles from this manufacturer that need five or more major repairs during the warranty period will lie in the interval (0.21405, 0.45262).

6.2.2 Margin of Error and Sample Size In real-world problems, the estimates of the proportion p are usually accompanied by a margin of error, rather than a conﬁdence interval. For example, in the news media, especially leading up to election time, we hear statements such as “The CNN/USA Today/Gallup poll of 818 registered voters taken on June 27–30 showed that if the election were held now, the president would beat his challenger 52% to 40%, with 8% undecided. The poll had a margin of error of plus or minus four percentage points.” What is this “margin of error”? According to the American Statistical Association, the margin of error is a common summary of sampling error that quantiﬁes uncertainty about a survey result. Thus, the margin of error is nothing but a conﬁdence interval. The number quoted in the foregoing statement is half the maximum width of a 95% conﬁdence interval, expressed as a percentage. Let b be the width of a 95% conﬁdence interval for the true proportion, p. Let pˆ = x/n be an estimate for p where x is the number of successes in n trials. Then, 8 8 x (x/n)(1 − (x/n)) (x/n)(1 − (x/n)) x b = + 1.96 − − 1.96 n n n n 8 8 (x/n)(1 − (x/n)) 1 = 3.92 ≤ 3.92 , n 4n

because (x/n)(1 − (x/n)) = p(1 ˆ − p) ˆ ≤ 14 .

304 CHAPTER 6 Interval Estimation

Thus, the margin of error associated with pˆ = (x/n) is 100d%, where 9 1 3.92 1.96 max b 4n = = √ . d= 2 2 2 n

From the foregoing derivation, it is clear that we can compute the margin of error for other values of α by replacing 1.96 by the corresponding value of zα/2 . A quick look at the formula for the conﬁdence interval for proportions reveals that a larger sample would yield a shorter interval (assuming other things being equal) and hence a more precise estimate of p. The larger sample is more costly in terms of time, resources, and money, whereas samples that are too small may result in inaccurate inferences. Then, it becomes beneﬁcial for ﬁnding out the minimum sample size required (thus less costly) to achieve a prescribed degree of precision (usually, the minimum degree of precision acceptable). We have seen that the large sample (1 − α)100% conﬁdence interval for p is 8 pˆ − zα/2

p(1 ˆ − p) ˆ < p < pˆ + zα/2 n

8

p(1 ˆ − p) ˆ . n

Rewriting it, we have pˆ − p ≤ zα/2

8

zα/2 p(1 ˆ − p) ˆ = √ p(1 ˆ − p) ˆ n n

which shows that, with probability (1 − α), the estimate pˆ is within zα/2 p(1 ˆ − p)/n ˆ units of p. Because p(1 ˆ − p) ˆ ≤ 1/4, for all values of p, ˆ we can write the foregoing inequality as zα/2 pˆ − p ≤ √ n

8

zα/2 1 = √ . 4 2 n

If we wish to estimate p at level (1 − α) to within d units of its true value, that is |pˆ − p| ≤ d, the √ sample size must satisfy the condition (zα/2 /(2 n)) ≤ d, or n≥

z2α/2 4d 2

.

Thus, to estimate p at level (1 − α) to within d units of its true value, take the minimal sample size as n = z2α/2 /4d 2 , and if this is not an integer, round up to the next integer. Sometimes, we may have an initial estimate p˜ of the parameter p from a similar process or from a pilot study or simulation. In this case, we can use the following formula to compute the minimum required size of the sample to estimate p, at level (1 − α), to within d units by using the formula n=

z2α/2 p(1 ˜ − p) ˜ d2

and, if this is not an integer, rounding up to the next integer.

6.2 Large Sample Conﬁdence Intervals: One Sample Case 305

A similar derivation for calculation of sample size for estimation of the population mean μ at level (1 − α) with margin of error E is given by n=

z2α/2 σ 2 E2

and, if this is not an integer, rounding up to the next integer. This formula can be used only if we know the population standard deviation, σ. Although it is unlikely to know σ when the population mean itself is not known, we may be able to determine σ from an earlier similar study or from a pilot study/simulation.

Example 6.2.5 A dendritic tree is a branched formation that originates from a nerve cell. In order to study brain development, researchers want to examine the brain tissues from adult guinea pigs. How many cells must the researchers select (randomly) so as to be 95% sure that the sample mean is within 3.4 cells of the population mean? Assume that a previous study has shown σ = 10 cells.

Solution A 95% confidence corresponds to α = 0.05. Thus, from the normal table, zα/2 = z0.025 = 1.96. Given that E = 3.4 and σ = 10, and using the sample size formula, the required sample size n is n=

z2α/2 σ 2 E2

=

(1.96)2 (10)2 = 33.232. (3.4)2

Thus, take n = 34.

Example 6.2.6 Suppose that a local TV station in a city wants to conduct a survey to estimate support for the president’s policies on economy within 3% error with 95% conﬁdence. (a) How many people should the station survey if they have no information on the support level? (b) Suppose they have an initial estimate that 70% of the people in the city support the economic policies of the president. How many people should the station survey?

Solution Here α = 0.05, and thus zα/2 = 1.96. Also, d = 0.03. (a) With no information on p, we use the sample size formula: n=

z2α/2 4d 2

=

(1.96)2 = 1067.1. 4(0.03)2

Hence, the TV station must survey 1068 people.

306 CHAPTER 6 Interval Estimation

(b) Because p˜ = 0.7, the required sample size is calculated from n= =

z2α/2 p(1 ˜ − p) ˜ d2 (1.96)2 (0.70)(0.30) = 896.37. (0.03)2

Thus, the TV station must survey at least 897 people.

In practice, we should realize that one of the key factors of a good design is not sample size by itself; it is getting representative samples. Even if we have a very large sample size, if the sample is not representative of our target population, then sample size means nothing. Therefore, whenever possible, we should use random sampling procedures (or other appropriate sampling procedures) to ensure that our target population is properly represented.

EXERCISES 6.2 6.2.1.

A survey indicates that it is important to pay attention to truth in political advertising. Based on a survey of 1200 people, 35% indicated that they found political advertisements to be untrue; 60% say that they will not vote for candidates whose advertisements are judged to be untrue; and of this latter group, only 15% ever complained to the media or to the candidate about their dissatisfaction. (a) Find a 95% conﬁdence interval for the percentage of people who ﬁnd political advertising to be untrue. (b) Find a 95% conﬁdence interval for the percentage of voters who will not vote for candidates whose advertisements are considered to be untrue. (c) Find a 95% conﬁdence interval for the percentage of those who avoid voting for candidates whose advertisements are considered untrue and who have complained to the media or to the candidate about the falsehood in commercials. (d) For each case above, interpret the results and state any assumptions you have made.

6.2.2.

Many mutual funds use an investment approach involving owning stocks whose price/earnings multiples (P/Es) are less than the P/E of the S&P 500. The following data give P/Es of 49 companies a randomly selected mutual fund owns in a particular year. 6.8 5.6 8.5 8.5 8.4 7.5 9.3 9.4 9.9 9.6 9.0 9.4 13.7 16.6 9.1 10.1 8.9 11.7 12.8 11.5 12.0 10.6 11.1 6.4 11.4 9.9 14.3 11.5 11.8 13.3 12.8 13.7 14.2 14.0 15.5 16.9 18.0 17.9 21.8 18.4

7.8 7.1 10.6 11.1 12.3 12.3 13.9 12.9 34.3

Find a 98% conﬁdence interval for the mean P/E multiples. Interpret the result and state any assumptions you have made.

6.2 Large Sample Conﬁdence Intervals: One Sample Case 307

6.2.3.

Let X1 , . . . , Xn be a random sample from N(μ, σ 2 ) distribution, σ 2 known. (a) Show that μ ˆ = X is a maximum likelihood estimator of the population mean μ. (b) Show that

2σ 2σ P X− √ Basic Statistics > 1-sample t. . . , in variables: enter C1, click Confidence interval, in Level default value is 95, if any other value, enter that value, and click OK

6.7 Computer Examples 331

We will obtain the following output. T Confidence Intervals Variable N C1 6

Mean 5.898

StDev 1.968

SE Mean 0.804

95.0% C.I. (3.832, 7.964)

Example 6.7.2 (Large Sample): For the data 6.8 5.6 8.5 8.5 8.4 7.5 9.3 9.4 7.8 7.1 9.9 9.6 9.0 13.7 9.4 16.6 9.1 10.1 10.6 11.1 8.9 11.7 12.8 11.5 10.6 12.0 11.1 6.4 12.3 12.3 11.4 9.9 15.5 14.3 11.5 13.3 11.8 12.8 13.7 13.9 12.9 14.2 14.0 obtain a 98% conﬁdence interval for μ.

Solution Enter the data in C1. Then click Stat > Basic Statistics > 1-Sample Z. . . > in Variables: type C1 > click Confidence interval, and enter 98 in Level: > enter 5 in Sigma: > OK We will obtain the following output. THE ASSUMED SIGMA = 5.00 Variable N MEAN STDEV SE MEAN 98.0 PERCENT C.I. C1 49 12.124 4.700 0.714 (10.462, 13.787)

Example 6.7.3 For the following data, ﬁnd a 90% conﬁdence interval for μ1 − μ2 Sample 1 Sample 2

1.2 4.2

3.1 2.7

1.7 3.6

2.8 3.9

3.0

Solution Enter sample 1 in C1 and sample 2 in C2. Then click

Stat > Basic Statistics > 2-Sample t. . . > click Sample in different columns > in First: enter C1 and in Second: enter C2 > enter 90 in Confidence Level: (if equality of variance can be assumed, click Assume equal variances) > OK

332 CHAPTER 6 Interval Estimation

We will obtain the following output: TWOSAMPLE T FOR C1 VS C2

C1 C2

N 5 4

MEAN 2.360 3.600

STDEV 0.856 0.648

SE MEAN 0.38 0.32

90 PCT CI FOR MU C1 − MU C2: (−2.22, −0.26) TTEST MU C1 = MU C2 (VS NE): T = −2.39 P = 0.048 DF = 7 POOLED STDEV = 0.774

6.7.2 SPSS Examples Example 6.7.4 Consider the data 66

74

79

80

77

78

65

79

81

69

Using SPSS, obtain a 99% conﬁdence interval for μ.

Solution One easy way to obtain the confidence interval in SPSS is to use the hypothesis testing procedure. The procedure is as follows: First enter the data in C1. Then click Analyze > Compare Means > One-sample t Test. . . , > Move var00001 to Test Variable(s), and Click Options . . . , and enter 99 in Confidence interval:, click Continue, and OK Note that the default value is 95%. We will obtain the following output: One-Sample Statistics N Mean Std. deviation VAR00001 10 74.8000 5.99630

Std. error mean 1.89620

One-Sample Test Test Value = 0 99% Confidence interval of the Mean difference t df Sig.(2-tailed) difference Lower Upper VAR00000 39.447 9 .000 74.8000 68.6377 80.9623 From this, we obtain the 99% confidence interval as (68.6377, 80.9623).

6.7 Computer Examples 333

6.7.3 SAS Examples Example 6.7.5 The following data give P/E for a particular year of 49 mutual fund companies owned by a randomly selected mutual fund. 6.8 9.9 12.8 11.8 34.3

5.6 9.6 11.5 13.3 13.7

8.5 9.0 12.0 13.9 12.3

8.5 16.6 10.6 12.9 18.0

8.4 9.1 11.1 14.2 9.4

7.5 10.1 6.4 14.0 12.3

9.3 10.6 11.4 15.5 16.9

9.4 11.1 9.9 17.9 12.8

7.8 8.9 14.3 21.8 13.7

7.1 11.7 11.5 18.4

Find a 98% conﬁdence interval for the mean P/E multiples. Use SAS procedures.

Solution We could use the following procedure. DATA peratio; INPUT patio @@; DATALINES; 6.8 5.6 8.5 8.5 8.4 7.5 9.3 9.4 7.8 7.1 9.9 9.6 9.0 9.4 13.7 16.6 9.1 10.1 11.1 8.9 11.7 12.8 11.5 12.0 10.6 11.1 6.4 12.3 11.4 9.9 14.3 11.5 11.8 13.3 12.8 13.7 14.2 14.0 15.5 16.9 18.0 17.9 21.8 18.4 34.3 ; PROC MEANS data = peratio lclm uclm alpha = 0.02; var peratio; RUN;

10.6 12.3 13.9

12.9

We will obtain the following output: The MEANS Procedure Analysis Variable : peratio Lower 98% Upper 98% CL for Mean CL for Mean -----------------------------------------------------------10.5084971 13.7404825 -----------------------------------------------------------Hence, we will obtain the 98% confidence interval for the P/E ratios as (10.50, 13.74).

EXERCISES 6.7 6.7.1.

Using any of the software packages (Minitab, SPSS, or SAS), obtain conﬁdence intervals for at least one data set taken from each section of this chapter.

334 CHAPTER 6 Interval Estimation

PROJECTS FOR CHAPTER 6 6A. Simulation of Coverage of the Small Confidence Intervals for μ (a) Generate 25 samples of size 15 from a normal population with μ = 10 and σ 2 = 4. Using a statistical package (such as Minitab), compute the 95% conﬁdence intervals for each of the samples using the small sample formula. From your output, determine the proportion of the 25 intervals that cover the true mean μ = 10. (b) What would you expect if the sample size is increased to 100? Would the width of the interval increase or decrease? Would you expect more or fewer of these intervals to contain the true mean 10? Check your answers with actual computation. (c) Repeat with 20 samples of size 10.

6B. Confidence Intervals Based on Sampling Distributions If we want to obtain a (1 − α)100% conﬁdence interval for θ, begin with an estimator θˆ of θ and determine its sampling distribution. Now select two probability levels, α1 and α2 , so that α = α1 +α2 . Generally we let α1 = α2 . Take a sample and calculate the value of θˆ , say θˆ = k. Now we need to determine the values of the upper and lower conﬁdence limits. Find a value θL such that p (θˆ ≥ k) = α1

and θU such that p (θˆ ≤ k) = α2 .

Then a (1 − α)100% conﬁdence interval for θ will be θL < θ < θU .

(a) Let X1 , . . . , Xn be a random sample from U(0, θ) distribution. Obtain a (1 − α)100% conﬁdence interval for θ, using the method of sampling distribution. (b) Let X have a binomial distribution with parameters n and p. First show that there is no quantity that satisﬁes the conditions of a pivotal quantity. Then using the method of sampling distributions, obtain a (1 − α)100% conﬁdence interval for p.

6C. Large Sample Confidence Intervals: General Case The method of ﬁnding a conﬁdence interval for a parameter θ that we described in this chapter depends on our ability to ﬁnd the pivotal quantity. We have seen that such a quantity may not exist. In those cases, the method of sampling distribution described in the previous project could be used. However, this method can involve some difﬁcult calculations. For large samples, we can utilize the following procedure, which is based on the asymptotic distribution of maximum likelihood estimators. Under fairly general conditions, the maximum likelihood estimators have a limiting distribution that is normal. Also, maximum likelihood estimators are asymptotically efﬁcient. Hence, for a large sample

Projects for Chapter 6 335

the maximum likelihood estimator θˆ of θ will have approximately normal distribution with mean θ. Also, if the Cramér–Rao lower bound exists, the limiting variance of θˆ will be σ 2ˆ = θ

! E

1

∂ ln L 2 ∂θ

".

Hence, Z=

θˆ − θ ∼ N(0, 1). σθˆ

Then a large sample (1 − α)100% conﬁdence interval is obtained from the probability statement

θˆ − θ P −zα/2 < < zα/2 σθˆ

≈ 1 − α.

We summarize the procedure to construct large sample conﬁdence intervals. 1. Determine the maximum likelihood estimator, θˆ , of θ. Also ﬁnd the maximum likelihood estimators of all other unknown parameters. 2. Obtain the variance σθˆ (if possible directly, otherwise by using the Cramér–Rao lower bound). 3. In the expression for σθˆ , substitute θˆ for θ. Replace all other unknown parameters by its maximum likelihood estimators. Let the resulting quantity be denoted by sθˆ . 4. Now construct a (1 − α)100% conﬁdence interval for θ from θˆ − zα/2 sθˆ < θ < θˆ + zα/2 sθˆ .

(a) Using the foregoing procedure, show that a large sample (1 − α)100% conﬁdence interval for the parameter p in a binomial distribution based on n trials is 8 pˆ − zα/2

p(1 ˆ − p) ˆ < p < pˆ + zα/2 n

8

p(1 ˆ − p) ˆ . n

(b) Let X1 , . . . , Xn be a random sample from a normal population with parameters μ and σ 2 . Derive a large sample conﬁdence interval for σ 2 using the above procedure. (c) Let X1 , . . . , Xn be a random sample from a population with a pdf ⎧ ⎪ ⎨ 1 e−x/θ , f (x) = θ ⎪ ⎩0,

Derive a large sample conﬁdence interval for θ.

x>0 otherwise.

336 CHAPTER 6 Interval Estimation

6D. Prediction Interval for an Observation from a Normal Population In many cases, we may be interested in predicting future observations from a population, rather than making an inference. A (1 − α)100% prediction interval for a future observation X is an interval of the form (XL , XU ) such that p (XL < X < XU ) = 1 − α. Similarly to conﬁdence intervals, we can also deﬁne one-sided prediction intervals. Assume that the population is normal with known variance σ 2 . Let X1 , . . . , Xn be a random sample from this population. Then the sampling distribution of 2 = the difference X − X (we use X to denote Xn ) is normal with mean zero and variance σ 2 + σX (1 + (1/n))σ 2 . Then a (1 − α)100% prediction interval for X is given by +

X − zα/2

+ 1 1 1+ 1+ σ 2 , X + zα/2 σ2 . n n

Thus, we are (1 − α)100% conﬁdent that the next observation, Xn+1 , will lie in this interval. As in conﬁdence intervals, if the sample size is large, replace σ by sample standard deviation s. In case, where both μ and σ are not known, and the sample size is small (so that the Central Limit $ # √ Theorem cannot be applied), it can be shown that (Xn+1 − Xn )/(Sn 1 + (1/n)) has a t-distribution with (n − 1) degrees of freedom. Thus, a (1 − α)100% prediction interval for Xn+1 is given by

9 9 X − tα/2,n−1 (1 + (1/n))S 2 , X + tα/2,n−1 (1 + (1/n))S 2 .

A standard measure of the capacity of lungs to expel air in breathing is called forced expiratory volume (FEV). The FEV1 is the volume exhaled during the ﬁrst second of a forced expiratory maneuver started from the level of total lung capacity. The following data (source: M. Bland, An Introduction to Medical Statistics, Oxford University Press, 1995) represents FEV measurements (in liters) from 57 male medical students. 4.47 4.47 3.48 5.00 3.42 3.78

3.10 3.57 4.20 4.50 3.60 3.75

4.50 4.90 3.50 2.85 5.10 5.20 3.70 5.30 4.71 4.20 4.16 3.70 3.20 4.56 4.78 4.05 3.54 4.14

4.14 4.32 4.80 5.10 4.10 4.30 3.83 3.90 3.60 3.96 2.98 3.54

Obtain a 95% prediction interval for a future observation Xn+1 .

4.80 3.10 4.30 4.70 3.39 3.69 4.47 3.30 3.19 2.85

4.68 4.08 4.44 5.43 3.04

Chapter

7

Hypothesis Testing Objective: In this chapter, various methods of testing hypotheses will be discussed. 7.1 Introduction 338 7.2 The Neyman–Pearson Lemma 349 7.3 Likelihood Ratio Tests 355 7.4 Hypotheses for a Single Parameter 361 7.5 Testing of Hypotheses for Two Samples 372 7.6 Chi-Square Tests for Count Data 388 7.7 Chapter Summary 399 7.8 Computer Examples 399 Projects for Chapter 7 408

Jerzy Neyman (Source: http://sciencematters.berkeley.edu/archives/volume2/issue12/legacy.php)

Jerzy Neyman (1894–1981) made far-reaching contributions in hypothesis testing, conﬁdence intervals, probability theory, and other areas of mathematical statistics. His work with Egon Pearson gave

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

337

338 CHAPTER 7 Hypothesis Testing

logical foundation and mathematical rigor to the theory of hypothesis testing. Their ideas made sure that samples were large enough to avoid false representation. Neyman made a broader impact in statistics throughout his lifetime.

7.1 INTRODUCTION Statistics plays an important role in decision making. In statistics, one utilizes random samples to make inferences about the population from which the samples were obtained. Statistical inference regarding population parameters takes two forms: estimation and hypothesis testing, although both hypothesis testing and estimation may be viewed as different aspects of the same general problem of arriving at decisions on the basis of observed data. We already saw several estimation procedures in earlier chapters. Hypothesis testing is the subject of this chapter. Hypothesis testing has an important role in the application of statistics to real-life problems. Here we utilize the sampled data to make decisions concerning the unknown distribution of a population or its parameters. Pioneering work on the explicit formulation as well as the fundamental concepts of the theory of hypothesis testing are due to J. Neyman and E. S. Pearson. A statistical hypothesis is a statement concerning the probability distribution of a random variable or population parameters that are inherent in a probability distribution. The following example illustrates the concept of hypothesis testing. An important industrial problem is that of accepting or rejecting lots of manufactured products. Before releasing each lot for the consumer, the manufacturer usually performs some tests to determine whether the lot conforms to acceptable standards. Let us say that both the manufacturer and the consumer agree that if the proportion of defectives in a lot is less than or equal to a certain number p, the lot will be released. Very often, instead of testing every item in the lot, we may test only a few items chosen at random from the lot and make decisions about the proportion of defectives in the lot; that is, we make the decisions about the population on the basis of sample information. Such decisions are called statistical decisions. In attempting to reach decisions, it is useful to make some initial conjectures about the population involved. Such conjectures are called statistical hypotheses. Sometimes the results from the sample may be markedly different from those expected under the hypothesis. Then we can say that the observed differences are signiﬁcant and we would be inclined to reject the initial hypothesis. These procedures that enable us to decide whether to accept or reject hypotheses or to determine whether observed samples differ signiﬁcantly from expected results are called tests of hypotheses, tests of signiﬁcance, or rules of decision. In any hypothesis testing problem, we formulate a null hypothesis and an alternative hypothesis such that if we reject the null, then we have to accept the alternative. The null hypothesis usually is a statement of either the “status quo” or “no effect.” A guideline for selecting a null hypothesis is that when the objective of an experiment is to establish a claim, the nulliﬁcation of the claim should be taken as the null hypothesis. The experiment is often performed to determine whether the null hypothesis is false. For example, suppose the prosecution wants to establish that a certain person is guilty. The null hypothesis would be that the person is innocent and the alternative would be that the person is guilty. Thus, the claim itself becomes the alternative hypothesis. Customarily, the alternative hypothesis is the statement that the experimenter believes to be true. For example, the alternative hypothesis is the reason a person is arrested (police suspect the person is not innocent). Once the hypotheses

7.1 Introduction 339

have been stated, appropriate statistical procedures are used to determine whether to reject the null hypothesis. For the testing procedure, one begins with the assumption that the null hypothesis is true. If the information furnished by the sampled data strongly contradicts (beyond a reasonable doubt) the null hypothesis, then we reject it in favor of the alternative hypothesis. If we do not reject the null, then we automatically reject the alternative. Note that we always make a decision with respect to the null hypothesis. Note that the failure to reject the null hypothesis does not necessarily mean that the null hypothesis is true. For example, a person being judged “not guilty” does not mean the person is innocent. This basically means that there is not enough evidence to reject the null hypothesis (presumption of innocence) beyond “a reasonable doubt.” We summarize the elements of a statistical hypothesis in the following. THE ELEMENTS OF A STATISTICAL HYPOTHESIS 1. The null hypothesis, denoted by H0 , is usually the nulliﬁcation of a claim. Unless evidence from the data indicates otherwise, the null hypothesis is assumed to be true. 2. The alternate hypothesis, denoted by Ha (or sometimes denoted by H1 ), is customarily the claim itself. 3. The test statistic, denoted by TS, is a function of the sample measurements upon which the statistical decision, to reject or not reject the null hypothesis, will be based. 4. A rejection region (or a critical region) is the region (denoted by RR) that speciﬁes the values of the observed test statistic for which the null hypothesis will be rejected. This is the range of values of the test statistic that corresponds to the rejection of H0 at some ﬁxed level of signiﬁcance, α, which will be explained later. 5. Conclusion: If the value of the observed test statistic falls in the rejection region, the null hypothesis is rejected and we will conclude that there is enough evidence to decide that the alternative hypothesis is true. If the TS does not fall in the rejection region, we conclude that we cannot reject the null hypothesis.

In practice one may have hypotheses such as H0 : μ = μ0 against one of the following alternatives: ⎧ ⎪ ⎪ ⎪ ⎨ or ⎪ or ⎪ ⎪ ⎩

Ha : μ = μ0 , Ha : μ < μ0 , Ha : μ > μ0 ,

called a two-tailed alternative called a lower (or left) tailed alternative called an upper (or right) tailed alternative

A test with a lower or upper tailed alternative is called a one-tailed test. In an applied hypothesis testing problem, we can use the following general steps. GENERAL METHOD FOR HYPOTHESIS TESTING 1. From the (word) problem, determine the appropriate null hypothesis, H0 , and the alternative, Ha . 2. Identify the appropriate test statistics and calculate the observed test statistic from the data. 3. Find the rejection region by looking up the critical value in the appropriate table. 4. Draw the conclusion: Reject or fail to reject the null hypothesis, H0 . 5. Interpret the results: State in words what the conclusion means to the problem we started with.

340 CHAPTER 7 Hypothesis Testing

It is always necessary to state a null and an alternate hypothesis for every statistical test performed. All possible outcomes should be accounted for by the two hypotheses.

Example 7.1.1 In a coin-tossing experiment, let p be the probability of heads. We start with the claim that the coin is fair, that is, H0 : p = 1/2. We test this against one of the following alternatives: (a) Ha : The coin is not fair (p = 1/2). This is a two-tailed alternative. (b) Ha : The coin is biased in favor of heads (p > 1/2). This is an upper tailed alternative. (c) Ha : The coin is biased in favor of tails (p < 1/2). This is a lower tailed alternative.

It is important to observe that the test statistic is a function of a random sample. Thus, the test statistic itself is a random variable whose distribution is known under the null hypothesis. The value of a test statistic when speciﬁc sample values are substituted is called the observed test statistic or simply test statistic. For example consider the hypothesis H0 : μ = μo versus Ha : μ = μo , where μo is known. Assume that the population is normal with a known variance σ 2 . Consider X, an unbiased estimator of μ √ based on the random sample X1 , . . . , Xn . Then Z = (X − μ0 )/(σ/ n) is a function of the random sample X1 , . . . , Xn , and has a known distribution, a standard normal, under H0 . If x1 , x2 , . . . , xn are √ speciﬁc sample values, then z = (x − μ0 )/(σ/ n) is called the observed sample statistic or simply sample statistic. Deﬁnition 7.1.1 A hypothesis is said to be a simple hypothesis if that hypothesis uniquely speciﬁes the distribution from which the sample is taken. Any hypothesis that is not simple is called a composite hypothesis.

Example 7.1.2 Refer to Example 7.1.1. The null hypothesis p =1/2 is simple, because the hypothesis completely speciﬁes the distribution, which in this case will be a binomial with p = 1/2 and with n being the number of tosses. The alternative hypothesis p = 1/2 is composite because the distribution now is not completely speciﬁed (we do not know the exact value of p).

Because the decision is based on the sample information, we are prone to commit errors. In a statistical test, it is impossible to establish the truth of a hypothesis with 100% certainty. There are two possible types of errors. On the one hand, one can make an error by rejecting H0 when in fact it is true. On the other hand, one can also make an error by failing to reject the null hypothesis when in fact it is false. Because the errors arise as a result of wrong decisions, and the decisions themselves are based on random samples, it follows that the errors have probabilities associated with them. We now have the following deﬁnitions.

7.1 Introduction 341

Table 7.1 Statistical Decision and Error Probabilities Statistical

True state of null hypothesis

decision

H 0 true

H 0 false

Do not reject H0

Correct decision

Type II error (β)

Reject H0

Type I error (α)

Correct decision

The decision and the errors are represented in Table 7.1. Deﬁnition 7.1.2 (a) A type I error is made if H0 is rejected when in fact H0 is true. The probability of type I error is denoted by α. That is, P (rejecting H0 |H0 is true) = α.

The probability of type I error, α, is called the level of signiﬁcance. (b) A type II error is made if H0 is accepted when in fact Ha is true. The probability of a type II error is denoted by β. That is, P (not rejecting H0 |H0 is false) = β.

It is desirable that a test should have a = β = 0 (this can be achieved only in trivial cases), or at least we prefer to use a test that minimizes both types of errors. Unfortunately, it so happens that for a ﬁxed sample size, as α decreases, β tends to increase and vice versa. There are no hard and fast rules that can be used to make the choice of α and β. This decision must be made for each problem based on quality and economic considerations. However, in many situations it is possible to determine which of the two errors is more serious. It should be noted that a type II error is only an error in the sense that a chance to correctly reject the null hypothesis was lost. It is not an error in the sense that an incorrect conclusion was drawn, because no conclusion is made when the null hypothesis is not rejected. In the case of type I error, a conclusion is drawn that the null hypothesis is false when, in fact, it is true. Therefore, type I errors are generally considered more serious than type II errors. For example, it is mostly agreed that ﬁnding an innocent person guilty is a more serious error than ﬁnding a guilty person innocent. Here, the null hypothesis is that the person is innocent, and the

Prob (TYPE II Error) 5 Beta Under H 0

Prob (TYPE I Error) 5 Alpha Under Ha

Critical value

342 CHAPTER 7 Hypothesis Testing

alternate hypothesis is that the person is guilty. “Not rejecting the null hypothesis” is equivalent to acquitting a defendant. It does not prove that the null hypothesis is true, or that the defendant is innocent. In statistical testing, the signiﬁcance level α is the probability of wrongly rejecting the null hypothesis when it is true (that is, the risk of ﬁnding an innocent person guilty). Here the type II risk is acquitting a guilty defendant. The usual approach to hypothesis testing is to ﬁnd a test procedure that limits α, the probability of type I error, to an acceptable level while trying to lower β as much as possible. The consequences of different types of errors are, in general, very different. For example, if a doctor tests for the presence of a certain illness, incorrectly diagnosing the presence of the disease (type I error) will cause a waste of resources, not to mention the mental agony to the patient. On the other hand, failure to determine the presence of the disease (type II error) can lead to a serious health risk. To formulate a hypothesis testing problem, consider the following situation. Suppose a toy store chain claims that at least 80% of girls under 8 years old prefer dolls over other types of toys. We feel that this claim is inﬂated. In an attempt to dispose of this claim, we observe the buying pattern of 20 randomly selected girls under 8 years old, and we observe X, the number of girls under 8 years old who buy stuffed toys or dolls. Now the question is, how can we use X to conﬁrm or reject the store’s claim? Let p be the probability that a girl under 8 chosen at random prefers stuffed toys or dolls. The question now can be reformulated as a hypothesis testing problem. Is p ≥ 0.8 or p < 0.8? Because we would like to reject the store’s claim only if we are highly certain of our decision, we should choose the null hypothesis to be H0 : p ≥ 0.8, the rejection of which is considered to be more serious. The null hypothesis should be H0 : p ≥ 0.8, and the alternative Ha : p < 0.8. In order to make the null hypothesis simple, we will use H0 : p = 0.8, which is the boundary value with the understanding that it really represents H0 : p ≥ 0.8. We note that X, the number of girls under 8 years old who prefer stuffed toys or dolls, is a binomial random variable. Clearly a large sample value of X would favor H0 . Suppose we arbitrarily choose to accept the null hypothesis if X > 12. Because our decision is based on only a sample of 20 girls under 8, there is always a possibility of making errors whether we accept or reject the store chain’s claim. In the following example, we will now formally state this problem and calculate the error probabilities based on our decision rule.

Example 7.1.3 A toy store chain claims that at least 80% of girls under 8 years old prefer dolls over other types of toys. After observing the buying pattern of many girls under 8 years old, we feel that this claim is inﬂated. In an attempt to dispose of this claim, we observe the buying pattern of 20 randomly selected girls under 8 years old, and we observe X, the number of girls who buy stuffed toys or dolls. We wish to test the hypothesis H0 : p = 0.8 against Ha : p < 0.8. Suppose we decide to accept the H0 if X > 12 (that is X ≥ 13). This means that if {X ≤ 12} (that is X < 13) we will reject H0 . (a) Find α. (b) Find β for p = 0.6. (c) Find β for p = 0.4. (d) Find the rejection region of the form {X ≤ K} so that (i) α = 0.01; (ii) α = 0.05. (e) For the alternative Ha :p = 0.6, ﬁnd β for the values of α in part (d).

7.1 Introduction 343

Solution The TS X is the number of girls under 8 years old who buy dolls. X follows the binomial distribution with n = 20 and p, the unknown population proportion of girls under 8 who prefer dolls. We now calculate α and β. (a) For p = 0.8, the probability of type I error is α = P{reject H0 |H0 is true} = P{X ≤ 12|p = 0.8} =

12 20 x=0

x

(0.8)x (0.2)20−x

= 0.0321. If we calculate α for any other value of p > 0.8, then we will find that it is smaller than 0.0321. Hence, there is at most a 3.21% chance of rejecting a true null hypothesis. That is, if the store’s claim is in fact true, then the chance that our test will erroneously reject that claim is at most 3.21%. (b) Here p = 0.6. The probability of type II error is β = P{accept H0 |H0 false} = P{X > 12|p = 0.6} = 1 − P{X ≤ 12|p = 0.6} = 1 − 0.584 = 0.416 so there is a 4.2% chance of accepting a false null hypothesis. Thus, in case the store’s claim is not true, and the truth is that only 60% of girls under 8 years old prefer dolls over other types of toys, then there is a 4.2% chance that our test will erroneously conclude that the store’s claim is true. (c) If p = 0.4, then β = P{accept H0 |H0 false} = P{X > 12|p = 0.4} = 1 − P{X ≤ 12|p = 0.4} = 1 − 0.979 = 0.021. That is, there is a 2.1% chance of accepting a false null hypothesis. (d) (i) To find K such that α = P{X ≤ K|p = 0.8} = 0.01 from the binomial table, K = 11. Hence, the rejection region is: Reject H0 if {X ≤ 11}. (ii) To find K such that α = P{X ≤ K|p = 0.8} = 0.05

344 CHAPTER 7 Hypothesis Testing

from the binomial table, α = 0.05 falls between K = 12 and K = 13. However, for K = 13, the value for α is 0.087, exceeding 0.05. If we want to limit α to be no more than 0.05, we will have to take K = 12. That is, we reject the null hypothesis if X ≤ 12, yielding an α = 0.0321 as shown in (a). (e) (i) When a = 0.01, from (d), the rejection region is of the form {X ≤ 11}. For p = 0.6, β = P{accept H0 |H0 false} = P{Y > 11|p = 0.6} = 1 − P{Y ≤ 11|p = 0.6} = 1 − 0.404 = 0.596. (ii) From (a) and (b) for testing the hypothesis H0 : p = 0.8 against Ha : p < 0.8 with n = 20. We see that when α is 0.0321, β is 0.416. From (d)(i) and (e)(i) for the same hypothesis, we see that when α is 0.01, β is 0.596. This holds in general. Thus, we observe that for fixed n as α decreases, β increases and vice versa.

In the next example, we explore what happens to β as the sample size increases, with α ﬁxed.

Example 7.1.4 Let X be a binomial random variable. We wish to test the hypothesis H0 : p = 0.8 against Ha : p = 0.6. Let α = 0.03 be ﬁxed. Find β for n = 10 and n = 20.

Solution

For n = 10, using the binomial tables, we obtain P{X ≤ 5|p = 0.8} ∼ = 0.03. Hence the rejection region for the hypothesis H0 : p = 0.8 vs. Ha : p = 0.6 is given by reject H0 if X ≤ 5. The probability of type II error is β = P{accept H0 |p = 0.6} β = P{X > 5|p = 0.6} = 1 − P{X ≤ 5|p = 0.6} = 0.733. For n = 20, as shown in Example 7.1.3, if we reject H0 for X ≤ 12, we obtain P(X ≤ 12|p = 0.8) ∼ = 0.03 and β = P(X > 12|p = 0.6) = 1 − P{X ≤ 12|p = 0.6} = 0.416. We see that for a fixed α, as n increases β decreases and vice versa. It can be shown that this result holds in general.

7.1 Introduction 345

In order for us to compute the value of β, it is necessary that the alternate hypothesis is simple. Now we will discuss a three-step procedure to calculate β.

STEPS TO CALCULATE β 1. Decide an appropriate test statistic (usually this is a sufﬁcient statistic or an estimator for the unknown parameter, whose distribution is known under H0 ). 2. Determine the rejection region using a given α, and the distribution of the test statistic (TS). 3. Find the probability that the observed test statistic does not fall in the rejection region assuming Ha is true. This gives β. That is, β = P(T .S. falls in the complement of the rejection region|Ha is true).

Example 7.1.5 A random sample of size 36 from a population with known variance, σ 2 = 9, yields a sample mean of x = 17. Find β, for testing the hypothesis H0 : μ = 15 versus Ha : μ = 16. Assume α = 0.05.

Solution Here n = 36, x = 17, and σ 2 = 9. In general, to test H0 : μ = μ0 versus Ha : μ > μ0 , we proceed as follows. An unbiased estimator of μ is X. Intuitively we would reject H0 if X is large, say X > c. Now using α = 0.05, we will determine the rejection region. By the definition of α, we have P (X > c |μ = μ0 ) = 0.05 or P

X − μ0 c − μ0 √ > √ |μ = μ0 σ/ n σ/ n

= 0.05

√ √0 > But if μ = μ0 , because the sample size n ≥ 30, [(X − μ0 )/(σ/ n)] ∼ N(0, 1). Therefore, P X−μ (σ/ n) c−μ √ 0 = 0.05 is equivalent to P Z > c−μ √ 0 = 0.05. From standard normal tables, we obtain P (Z > (σ/ n) (σ/ n) √ √ 0 = 1.645 or c = μ0 + 1.645(σ/ n). 1.645) = 0.05. Hence c−μ (σ/ n)

Therefore, the rejection region is the set of all sample means x such that

σ x > μ0 + 1.645 √ . n Substituting μ0 = 15, and σ = 3, we obtain √ μ0 + 1.645(σ/ n) = 15 + 1.645 The rejection region is the set of x such that x ≥ 15.8225.

3 36

= 15.8225.

346 CHAPTER 7 Hypothesis Testing

Then by definition, β = P (X ≤ 15.8225 when μ = 16). Consequently, for μ = 16,

β=P

X − 16 15.8225 − 16 √ √ ≤ σ/ n 3/ 36

= P (Z ≤ −0.36) = 0.3594. That is, under the given information, there is a 35.94% chance of not rejecting a false null hypothesis.

7.1.1 Sample Size It is clear from the preceding example that once we are given the sample size n, an α, a simple alternative Ha , and a test statistic, we have no control over β and it is exactly determined. Hence, for a given sample size and test statistic, any effort to lower β will lead to an increase in α and vice versa. This means that for a test with ﬁxed sample size it is not possible to simultaneously reduce both α and β. We also notice from Example 7.1.4 that by increasing the sample size n, we can decrease β (for the same α) to an acceptable level. The following discussion illustrates that it may be possible to determine the sample size for a given α and β. Suppose we want to test H0 : μ = μ0 versus Ha : μ > μ0 . Given α and β, we want to ﬁnd n, the sample size, and K, the point at which the rejection begins. We know that α = P (X > K when μ = μ0 ) X − μ0 K − μ0 =P √ > √ , when μ = μ0 σ/ n σ/ n

(7.1)

= P (Z > za )

and β = P (X ≤ K, when μ = μa ) X − μa K − μa =P √ ≤ √ , when μ = μa σ/ n σ/ n = P (z ≤ −zβ ).

From Equations (7.1) and (7.2), zα =

K − μ0 √ σ/ n

(7.2)

7.1 Introduction 347

and −zβ =

K − μa √ . σ/ n

This gives us two equations with two unknowns (K and n), and we can proceed to solve them. Eliminating K, we get

μ0 + zα

σ √ n

= μa − zβ

σ . √ n

From this we can derive √

n=

(zα + zβ )σ . μa − μ 0

Thus, the sample size for an upper tail alternative hypothesis is n=

(zα + zβ )2 σ 2 (μa − μ0 )2

.

The sample size increases with the square of the standard deviation and decreases with the square of the difference between mean value of the alternative hypothesis and the mean value under the null hypothesis. Note that in real-world problems, care should be taken in the choice of the value of μa for the alternative hypothesis. It may be tempting for a researcher to take a large value of μa in order to reduce the required sample size. This will seriously affect the accuracy (power) of the test. This alternative value must be realistic within the experiment under study. Care should also be taken in the choice of the standard deviation σ. Using an underestimated value of the standard deviation to reduce the sample size will result in inaccurate conclusions similar to overestimating the difference of means. Usually, the value of σ is estimated using a similar study conducted earlier. The problem could be that the previous study may be old and may not represent the new reality. When accuracy is important, it may be necessary to conduct a pilot study only to get some idea on the estimate of σ. Once we determine the necessary sample size, we must devise a procedure by which the appropriate data can be randomly obtained. This aspect of the design of experiments is discussed in Chapter 9.

Example 7.1.6 Let σ = 3.1 be the true standard deviation of the population from which a random sample is chosen. How large should the sample size be for testing H0 : μ = 5 versus Ha : μ = 5.5, in order that α = 0.01 and β = 0.05?

Solution We are given μ0 = 5 and μa = 5.5. Also, zα = z0.01 = 2.33 and zβ = z0.05 = 1.645. Hence, the sample size n=

(zα + zβ )2 σ 2 (μa − μ0 )2

=

(2.33 + 1.645)2 (3.1)2 = 607.37. (0.5)2

348 CHAPTER 7 Hypothesis Testing

So, n = 608 will provide the desired levels. That is, in order for us to test the foregoing hypothesis, we must randomly select 608 observations from the given population.

From a practical standpoint, the researcher typically chooses α, and the sample size β is ignored. Because a trade-off exists between α and β, choosing a very small value of α will tend to increase β in a serious way. A general rule of thumb is to pick reasonable values of α, possibly in the 0.05 to 0.10 range so that β will remain reasonably small.

EXERCISES 7.1 7.1.1.

An appliance manufacturer is considering the purchase of a new machine for stamping out sheet metal parts. If μ0 (given) is the true average of the number of good parts stamped out per hour by their old machine and μ is the corresponding true unknown average for the new machine, the manufacturer wants to test the null hypothesis μ = μ0 versus a suitable alternative. What should the alternative be if he does not want to buy the new machine unless it is (a) more productive than the old one? (b) At least 20% more productive than the old one?

7.1.2.

Formulate an alternative hypothesis for each of the following null hypotheses. (a) H0 : Support for a presidential candidate is unchanged after the start of the use of TV commercials. (b) H0 : The proportion of viewers watching a particular local news channel is less than 30%. (c) H0 : The median grade point average of undergraduate mathematics majors is 2.9.

7.1.3.

It is suspected that a coin is not balanced (not fair). Let p be the probability of tossing a head. To test H0 : p = 0.5 against the alternative hypothesis Ha : p > 0.5, a coin is tossed 15 times. Let Y equal the number of times a head is observed in the 15 tosses of this coin. Assume the rejection region to be {Y ≥ 10}. (a) Find α. (b) Find β for p = 0.7. (c) Find β for p = 0.6. (d) Find the rejection region for {Y ≥K} for α = 0.01, and α = 0.03. (e) For the alternative Ha : p = 0.7, ﬁnd β for the values of α given in (d).

7.1.4.

In Exercise 7.1.3: (a) Assume that the rejection region is {Y ≥ 8}. Calculate α and β if p = 0.6. Compare the results with the corresponding values obtained in Exercise 7.1.3. (This gives the effect of enlarging the rejection region on α and β.) (b) Assume that the rejection region is {Y ≥ 8}. Calculate α and β if p = 0.6 and (i) the coin is tossed 20 times, or (ii) the coin is tossed 25 times. (This shows the effect of increasing the sample size on α and β for a ﬁxed rejection region.)

7.1.5.

Suppose we have a random sample of size 25 from a normal population with an unknown mean μ and a standard deviation of 4. We wish to test the hypothesis H0 : μ = 10 vs.

7.2 The Neyman–Pearson Lemma 349

Ha : μ > 10. Let the rejection region be deﬁned by: reject H0 if the sample mean X > 11.2. (a) Find α. (b) Find β for Ha : μ = 11. (c) What should the sample size be if α = 0.01 and β = 0.8? 7.1.6.

A process for making steel pipe is under control if the diameter of the pipe has mean 3.0 in. with standard deviation of no more than 0.0250 in. To check whether the process is under control, a random sample of size n = 30 is taken each day and the null hypothesis μ = 3.0 is rejected if X is less than 2.9960 or greater than 3.0040. Find (a) the probability of type I error; (b) the probability of type II error when μ = 3.0050 in. Assume σ = 0.0250 in.

7.1.7.

A bowl contains 20 balls, of which x are green and the remain- der red. To test H0 : x = 10 versus Ha : x = 15, three balls are selected at random without replacement, and H0 is rejected if all three balls are green. Calculate α and β for this test.

7.1.8.

Suppose we have a sample of size 6 from a population with pdf f (x) = (1/θ)e−x/θ , x > 0, θ > 0. We wish to test H0 : θ = 1 vs. Ha : θ > 1. Let the rejection region be deﬁned by reject H0 if 6 i=1 Xi > 8. (a) Find α. (b) Find β for Ha : θ = 2.

7.1.9.

Let σ 2 = 16 be the variance of a normal population from which a random sample is chosen. How large should the sample size be for testing H0 : μ = 25 versus Ha : μ = 24, in order that α = 0.05 and β = 0.05?

7.2 THE NEYMAN–PEARSON LEMMA In practical hypothesis testing situations, there are typically many tests possible with signiﬁcance level α for a null hypothesis versus alternative hypothesis (see Project 7A). This leads to some important questions, such as (1) how to decide on the test statistic and (2) how to know that we selected the best rejection region. In this section, we study the answer to these questions using the Neyman–Pearson approach. Deﬁnition 7.2.1 Suppose that W is the test statistic and RR is the rejection region for a test of hypothesis concerning the value of a parameter θ. Then the power of the test is the probability that the test rejects H0 when the alternative is true. That is, π = Power(θ) = P(W in RR when the parameter value is an alternative θ).

If H0 : θ = θ0 and Ha : θ = θ0 , then the power of the test at some θ = θ1 = θ0 is Power(θ1 ) = P(reject H0 |θ = θ1 ).

But, β(θ1 ) = P(accept H0 |θ = θ1 ). Therefore, Power(θ1 ) = 1 − β(θ1 ).

A good test will have high power.

350 CHAPTER 7 Hypothesis Testing

Note that the power of a test H0 cannot be found until some true situation Ha is speciﬁed. That is, the sampling distribution of the test statistic when Ha is true must be known or assumed. Because β depends on the alternative hypothesis, which being composite most of the time does not specify the distribution of the test statistic, it is important to observe that the experimenter cannot control β. For example, the alternative Ha : μ < μ0 does not specify the value of μ, as in the case of the null hypothesis, H0 : μ = μ0 .

Example 7.2.1 Let X1 , . . . , Xn be a random sample from a Poisson distribution with parameter λ, that is, the pdf is given by f (x) = e−λ λx /(x!). Then the hypothesis H0 : λ = 1 uniquely speciﬁes the distribution, because f (x) = e−1 /(x!) and hence is a simple hypothesis. The hypothesis Ha : λ > 1 is composite, because f (x) is not uniquely determined.

Deﬁnition 7.2.2 A test at a given α of a simple hypothesis H0 versus the simple alternative Ha that has the largest power among tests with the probability of type I error no larger than the given α is called a most powerful test. Consider the test of hypothesis H0 : θ = θ0 versus Ha : θ = θ1 . If α is ﬁxed, then our interest is to make β as small as possible. Because β = 1 − Power(θ1 ), by minimizing β we would obtain a most powerful test. The following result says that among all tests with given probability of type I error, the likelihood ratio test given later minimizes the probability of a type II error, in other words, it is most powerful. Theorem 7.2.1 (Neyman–Pearson Lemma) Suppose that one wants to test a simple hypothesis H0 : θ = θ0 versus the simple alternative hypothesis Ha : θ = θ1 based on a random sample X1 , . . . , Xn from a distribution with parameter θ. Let L(θ) ≡ L(θ; X1 , . . . , Xn ) > 0 denote the likelihood of the sample when the value of the parameter is θ. If there exist a positive constant K and a subset C of the sample space Rn (the Euclidean n-space) such that 1.

L(θ0 ) ≤ K for (x1 , x2 , . . . , xn ) ∈ C L(θ1 )

2.

L(θ0 ) ≥ K for (x1 , x2 , . . . , xn ) ∈ C , where C is the complement of C, and L(θ1 )

3. P [(X1 , . . . , Xn ) ∈ C; θ0 ] = α. Then the test with critical region C will be the most powerful test for H0 versus Ha . We call α the size of the test and C the best critical region of size α. Proof. We prove this theorem for continuous random variables. For discrete random variables, the proof is identical with sums replacing the integral. Let S be some region in Rn , an n-dimensional Euclidean space. For simplicity we will use the following notation:

L(θ) = S

...

S

L(θ; x1 , x2 , . . . , xn )dx1 dx2 , . . . , dxn S

7.2 The Neyman–Pearson Lemma 351

Note that P((X1 , . . . , Xn ) ∈ C; θ0 ) =

f (x1 , . . . , xn ; θ0 )dx1 , . . . , dxn C

=

L(θ0 ; x1 , . . . , xn )dx1 , . . . , dxn . C

Suppose that there is another critical region, say B, of size less than or equal to α, that is B L(θ0 ) ≤ α. Then 0≤

L(θ0 ) −

B

C

L(θ0 ) = α by assumption 3.

L(θ0 ), because C

Therefore, 0≤

L(θ0 ) −

L(θ0 ) B

C

L(θ0 ) +

=

C∩B

C∩B

=

L(θ0 ) −

L(θ0 ) −

L(θ0 )

C ∩B

C∩B

L(θ0 ) −

C∩B

L(θ0 ).

C ∩B

Using assumption 1 of Theorem 7.2.1, KL(θ1 ) ≥ L(θ0 ) at each point in the region C and hence in C ∩ B . Thus

L(θ0 ) ≤ K C∩B

L(θ1 ).

C∩B

By assumption 2 of the theorem, KL(θ1 ) ≤ L(θ0 ) at each point in C , and hence in C ∩ B. Thus,

L(θ0 ) ≥ K

C ∩B

L(θ1 ).

C ∩B

Therefore,

0≤

L(θ0 ) − C∩B

≤K

⎧ ⎪ ⎨ ⎪ ⎩

C∩B

L(θ0 )

C ∩B

L(θ1 ) − C ∩B

⎫ ⎪ ⎬ L(θ1 ) . ⎪ ⎭

352 CHAPTER 7 Hypothesis Testing

That is, 0≤K

⎧ ⎪ ⎨ ⎪ ⎩

L(θ1 ) +

C∩B

=K

⎧ ⎨ ⎩

L(θ1 ) −

L(θ1 )−

C∩B

⎫ ⎬

L(θ1 ) −

L(θ1 )

C ∩B

C∩B

⎫ ⎪ ⎬

⎪ ⎭

L(θ1 ) . ⎭ B

C

As a result,

L(θ1 ) ≥

L(θ1 ). B

C

Because this is true for every critical region B of size ≤ α, C is the best critical region of size α, and the test with critical region C is the most powerful test of size α. When testing two simple hypotheses, the existence of a best critical region is guaranteed by the Neyman–Pearson lemma. In addition, the foregoing theorem provides a means for determining what the best critical region is. However, it is important to note that Theorem 7.2.1 gives only the form of the rejection region; the actual rejection region depends on the speciﬁc value of α. In real-world situations, we are seldom presented with the problem of testing two simple hypotheses. There is no general result in the form of Theorem 7.4.1 for composite hypotheses. However, for hypotheses of the form H0 : θ = θ0 versus Ha : θ > θ0 , we can take a particular value θ1 > θ0 and then ﬁnd a most powerful test for H0 : θ = θ0 versus Ha : θ > θ1 . If this test (that is, the rejection region of the test) does not depend on the particular value θ1 , then this test is said to be a uniformly most powerful test for H0 : θ = θ0 versus Ha : θ > θ0 . The following example illustrates the use of the Neyman–Pearson lemma.

Example 7.2.2 Let X1 , . . . , Xn denote an independent random sample from a population with a Poisson distribution with mean λ. Derive the most powerful test for testing H0 : λ = 2 versus Ha : λ = 1/2.

Solution Recall that the pdf of Poisson variable is p(x) =

e−λ λx , x!

λ > 0, x = 0, 1, 2, . . .

0,

otherwise.

Thus, the likelihood function is n " ! ( xi ) λ i=1 e−λn

L=

n

(xi !)

i=1

.

7.2 The Neyman–Pearson Lemma 353

For λ = 2,

!

2

n

xi

e−2n

i=1

L(θ0 ) = L(λ = 2) =

"

n

(xi !)

i=1

and for λ = 1/2,

⎡

⎣(1/2) L(θ1 ) = L(λ = 1/2) =

n

xi

i=1

⎤ e−(1/2)n ⎦

n

(xi !)

i=1

Thus, 2( xi ) e−n2 L(θ0 ) = σ02 ?

354 CHAPTER 7 Hypothesis Testing

Solution To test H0 : σ 2 = σ02 versus Ha : σ 2 > σ12 . We have

L(σ02 ) =

n 7

√

i=1

−

1 2πσ0n

(xi − μ)2 2σ02

e

(xi − μ)2 2σ0n .

−

1 = √ e ( 2π)n σ0n Similarly,

(xi − μ)2

−

2σ12

1 L(σ12 ) = √ e ( 2π)n σ1n

.

Therefore, the most powerful test is, reject H0 if, L(σ02 ) L(σ12 )

=

σ12

n ! (σ12 −σ02 )2

σ02

e

−

(xi − μ)2

2σ12 σ02

"

≤K

for some K. Taking the natural logarithms, we have

n ln

σ1 σ0

−

(σ12 − σ02 )

(xi − μ)2 ≤ ln K

2σ12 σ02

or

!

σ1 (xi − μ)2 ≥ n ln σ0

"

− ln K

2σ12 σ02 σ12 − σ02

= C.

To find the rejection region for a fixed value of α, write the region as (xi − μ)2 σ02

≥

C σ02

= C.

Note that (xi − μ)2 /σ02 has a χ2 -distribution with n degrees of freedom. Under the H0 because the same rejection region (does not depend upon the specific value of σ12 in the alternative) would be used for any σ12 > σ02 , the test is uniformly most powerful.

The foregoing example shows that, in order to test for variance using a sample from a normal distribution, we could use the chi-square table to obtain the critical value for the rejection region given α.

7.3 Likelihood Ratio Tests 355

EXERCISES 7.2 7.2.1.

Suppose X1 , . . . , Xn is a random sample from a normal distribution with a known variance of σ 2 and an unknown mean of μ. Find the most powerful α-level test of H0 : μ = μ0 versus Ha : μ = μa if (a) μ0 > μa , and (b) μa > μ0 .

7.2.2.

Show that the most powerful test obtained in Example 7.2.1 is uniformly most powerful for testing H0 : μ ≤ μ0 versus Ha : μ > μa , but not uniformly most powerful for testing H0 : μ = μ0 versus Ha : μ = μ0 .

7.2.3.

Suppose X1 , . . . , Xn is a random sample from a U(0, θ) distribution. Find the most powerful α-level test for testing H0 : θ = θ0 versus Ha : θ = θ1 , where θ0 < θ1 .

7.2.4.

Let X1 , . . . , Xn be a random sample from a geometric distribution with parameter p. Find the most powerful test of H0 : p = p0 versus Ha : p = pa (> p0 ). Is this uniformly most powerful test for H0 : p = p0 versus Ha : p > p0 ?

7.2.5.

Let X1 , . . . , Xn be a random sample from a distribution having a pdf of ⎧ 2 ⎨ 2y − y2 η , e f (y) = η2 ⎩ 0,

if x > 0 otherwise.

Find a uniformly most powerful test for testing H0 : η = η0 versus Ha : η < η0 . 7.2.6.

Let X be a single observation from the pdf f (x) =

θxθ−1 ,

0<x p0 .

7.2.8.

Let X1 , . . . , Xn be a random sample from a Poisson distribution with mean λ. Find a best critical region for testing H0 : λ = 3 against Ha : λ = 6.

7.3 LIKELIHOOD RATIO TESTS The Neyman–Pearson lemma provides a method for constructing most powerful tests for simple hypotheses. We also have seen that in some instances when a hypothesis is not simple, it is possible to ﬁnd uniformly most powerful tests. In general, uniformly most powerful (UMP) tests do not exist for composite hypotheses. As an example, consider the two-sided hypothesis, at level α, given by H0 : μ = μ0

vs.

Ha : μ = μ0

where μ is the mean of a normal population with known variance σ 2 . If X is the sample mean of a random sample of size n, then as shown earlier, we can use the test statistic

356 CHAPTER 7 Hypothesis Testing

Z=

X − μ0 >√ . σ n

For Ha : μ = μ1 > μ0 , the rejection region for the most powerful test would be Reject H0 if z > zα .

On the other hand for Ha : μ = μ2 < μ0 , the rejection region for the most powerful test would be Reject H0 if z < −zα .

Thus, the rejection region depends on the speciﬁc alternative. Consequently, the two-sided hypothesis just given has no UMP test. In this section, we shall study a general procedure that is applicable when one or both H0 and Ha are composite. In fact, this procedure works for simple hypotheses as well. This method is based on the maximum likelihood estimation and the ratio of likelihood functions used in the Neyman–Pearson lemma. We assume that the pdf or pmf of the random variable X is f (x, θ), where θ can be one or more unknown parameters. Let represent the total parameter space that is the set of all possible values of the parameter θ given by either H0 or H1 . Consider the hypotheses H0 : θ ∈ 0 vs. Ha : θ ∈ a = − 0 .

where θ is the unknown population parameter (or parameters) with values in , and 0 is a subset of . Let L(θ) be the likelihood function based on the sample X1 , . . . , Xn . Now we deﬁne the likelihood ratio corresponding to the hypotheses H0 and Ha . This ratio will be used as a test statistic for the testing procedure that we develop in this section. This is a natural generalization of the ratio test used in the Neyman–Pearson lemma when both hypotheses were simple. Deﬁnition 7.3.1 The likelihood ratio λ is the ratio max L(θ; x1 , . . . , xn )

λ=

θ∈0

max L(θ; x1 , . . . , xn ) θ∈

=

L∗0 . L∗

We note that 0 ≤ λ ≤ 1. Because λ is the ratio of nonnegative functions, λ ≥ 0. Because 0 is a subset of , we know that max L(θ) ≤ max L(θ). Hence, λ ≤ 1. θ∈0

θ∈

If the maximum of L in 0 is much smaller as compared with the maximum of L in , that is, if λ is small, it would appear that the data X1 , . . . , Xn do not support the null hypothesis θ ∈ 0 . On the other hand, if λ is close to 1, one could conclude that the data support the null hypothesis, H0 . Therefore, small values of λ would result in rejection of the null hypothesis, and large values nearer to 1 will result a decision in support of the null hypothesis.

7.3 Likelihood Ratio Tests 357 For the evaluation of λ, it is important to note that max θ ∈ L(θ) = L(θˆ ml. ), where θˆ ml. is the maximum likelihood estimator of θ ∈ , and max θ ∈ 0 L(θ) is the likelihood function with unknown parameters replaced by their maximum likelihood estimators subject to the condition that θ ∈ 0 . We can summarize the likelihood ratio test as follows. LIKELIHOOD RATIO TESTS (LRTs) To test H0 : θ ∈ 0 vs. Ha : θ ∈ a max L(θ; x1 , . . . , xn )

λ=

L∗ = 0 max L(θ; x1 , . . . , xn ) L∗

θ∈0 θ∈

will be used as the test statistic. The rejection region for the likelihood ratio test is given by Reject H0 if λ ≤ K . K is selected such that the test has the given signiﬁcance level α.

Example 7.3.1 Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). Assume that σ 2 is known. We wish to test, at level α, H0 : μ = μ0 vs. Ha : μ = μ0 . Find an appropriate likelihood ratio test.

Solution We have seen that to test H0 : μ = μ0

vs.

Ha : μ = μ0

there is no uniformly most powerful test for this case. The likelihood function is n

L(μ) =

√

1 2πσ

n − e

2σ 2

Here, 0 = {μ0 } and a = R − {μ0 }. Hence,

n

L∗0 = max

μ=μ0

n − 1 e √ 2πσ n

=

(xi − μ)2

i=1

n − 1 e √ 2πσ

.

(xi − μ)2

i=1

2σ 2

(xi − μ0 )2

i=1

2σ 2

.

358 CHAPTER 7 Hypothesis Testing

Similarly, n

L∗ =

max

−∞ K (note that L L1 = 2 ( 19 ) That is, reject H0 if

10−x 18 >K 2x 19 x 2 ⇔ 18 > K1

⇔

19

19 x > K1 . 9

Hence, reject H0 if X > C; P(X > C|H0 : θ = 0.05) ≤ 0.05. Using the binomial tables, we have P(X > 2|θ = 0.05) = 0.0116

and P(X ≥ 2|θ = 0.05) = 0.0862.

Reject H0 if X > 2. If we want α to be exactly 0.05, we have to use randomized test. Reject with probability 0.0384 0.0762 = 0.5039 if X = 2. The likelihood ratio tests do not always produce a test statistic with a known probability distribution such as the z-statistic of Example 7.3.1. If we have a large sample size, then we can obtain an approximation to the distribution of the statistic λ, which is beyond the level of this book.

EXERCISES 7.3 7.3.1.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). Assume that σ 2 is unknown. We wish to test, at level α, H0 : μ = μ0 vs Ha : μ < μ0 . Find an appropriate likelihood ratio test.

7.3.2.

Let X1 , . . . , Xn be a random sample from an N(μ, σ 2 ). Assume that both μ and σ 2 are unknown. We wish to test, at level α, H0 : σ 2 = σ02 vs. Ha : σ 2 > σ02 . Find an appropriate likelihood ratio test.

7.3.3.

Let X1 , . . . , Xn be a random sample from an N(μ1 , σ 2 ) and let Y1 , Y2 , . . . , Yn be an independent sample from an N(μ2 , σ 2 ), where σ 2 is unknown. We wish to test, at level α, H0 : μ1 = μ2 vs. Ha : μ1 = μ2 . Find an appropriate likelihood ratio test.

7.3.4.

Let X1 , . . . , Xn be a sample from a Poisson distribution with parameter λ. Show that a likelihood ratio test of H0 : λ = λ0 vs. Ha : λ = λ0 rejects the null hypothesis if X ≥ m1 or X ≤ m2 .

7.4 Hypotheses for a Single Parameter 361

7.3.5.

Let X1 , . . . , Xn be a sample from an exponential distribution with parameter θ. Show that a likelihood ratio test of H0 : θ = θ0 vs. Ha : θ = θ0 rejects the null hypothesis if ni=1 Xi ≥ m1 n or i=1 Xi ≤ m2 .

7.3.6.

A clinical oncology program developed a set of guidelines for their cancer patients to follow. It is believed that the proportion of patients who are still living after 24 months is greater for those who follow the guidelines. Of the 40 patients who followed the guidelines, 30 are still living after 24 months, whereas of 32 patients who did not follow the guidelines, 21 are living after 24 months. Find a likelihood ratio test at α = 0.01 to decide whether the program is effective.

7.4 HYPOTHESES FOR A SINGLE PARAMETER In this section, we ﬁrst introduce the concept of p-value. After that, we study hypothesis testing concerning a single parameter.

7.4.1 The p-Value In hypothesis testing, the choice of the value of α is somewhat arbitrary. For the same data, if the test is based on two different values of α, the conclusions could be different. Many statisticians prefer to compute the so-called p-value, which is calculated based on the observed test statistic. For computing the p-value, it is not necessary to specify a value of α. We can use the given data to obtain the p-value. Deﬁnition 7.4.1 Corresponding to an observed value of a test statistic, the p-value (or attained signiﬁcance level) is the lowest level of signiﬁcance at which the null hypothesis would have been rejected. For example, if we are testing a given hypothesis with α = 0.05 and we make a decision to reject H0 and we proceeded to calculate the p-value equal to 0.03, this means that we could have used an α as low as 0.03 and still maintain the same decision, rejecting H0 . Based on the alternative hypothesis, one can use the following steps to compute the p-value.

STEPS TO FIND THE p-VALUE 1. Let TS be the test statistic. 2. Compute the value of TS using the sample X1 , . . . , Xn . Say it is a. 3. The p-value is given by

p-value =

⎧ ⎪ P (T S < a|H0 ), ⎪ ⎪ ⎨ P (T S > a|H0 ), ⎪ ⎪ ⎪ ⎩ P (|T S| > |a||H0 ),

if lower tail test if upper tail test if two tail test.

362 CHAPTER 7 Hypothesis Testing

Example 7.4.1 To test H0 : μ = 0 vs. Ha : μ = 0, suppose that the test statistic Z results in a computed value of 1.58. Then, the p-value = P (|Z| > 1.58) = 2(0.0571) = 0.1142. That is, we must have a type I error of 0.1142 in order to reject H0 . Also, if Ha : μ > 0, then the p-value would be P (Z > 1.58) = 0.0582. In this case we must have an α of 0.0582 in order to reject H0 .

The p-value can be thought of as a measure of support for the null hypothesis: The lower its value, the lower the support. Typically one decides that the support for H0 is insufﬁcient when the p-value drops below a particular threshold, which is the signiﬁcance level of the test. REPORTING TEST RESULT AS p-VALUES 1. Choose the maximum value of α that you are willing to tolerate. 2. If the p-value of the test is less than the maximum value of α, reject H0 .

If the exact p-value cannot be found, one can give an interval in which the p-value can lie. For example, if the test is signiﬁcant at α = 0.05 but not signiﬁcant for α = 0.025, report that 0.025 ≤ p-value ≤ 0.05. So for α > 0.05, reject H0 , and for α < 0.025, do not reject H0 . In another interpretation, 1−(p-value) is considered as an index of the strength of the evidence against the null hypothesis provided by the data. It is clear that the value of this index lies in the interval [0, 1]. If the p-value is 0.02, the value of index is 0.98, supporting the rejection of the null hypothesis. Not only do p-values provide us with a yes or no answer, they provide a sense of the strength of the evidence against the null hypothesis. The lower the p-value, the stronger the evidence. Thus, in any test, reporting the p-value of the test is a good practice. Because most of the outputs from statistical software used for hypothesis testing include the p-value, the p-value approach to hypothesis testing is becoming more and more popular. In this approach, the decision of the test is made in the following way. If the value of α is given, and if the p-value of the test is less than the value of α, we will reject H0 . If the value of α is not given and the p-value associated with the test is small (usually set at p-value < 0.05), there is evidence to reject the null hypothesis in favor of the alternative. In other words, there is evidence that the value of the true parameter (such as the population mean) is signiﬁcantly different (greater, or lesser) than the hypothesized value. If the p-value associated with the test is not small (p > 0.05), we conclude that there is not enough evidence to reject the null hypothesis. In most of the examples in this chapter, we give both the rejection region and p-value approaches.

Example 7.4.2 The management of a local health club claims that its members lose on the average 15 pounds or more within the ﬁrst 3 months after joining the club. To check this claim, a consumer agency took a random sample of 45 members of this health club and found that they lost an average of 13.8 pounds within the ﬁrst 3 months of membership, with a sample standard deviation of 4.2 pounds.

7.4 Hypotheses for a Single Parameter 363

(a) Find the p-value for this test. (b) Based on the p-value in (a), would you reject the null hypothesis at α = 0.01?

Solution (a) Let μ be the true mean weight loss in pounds within the first 3 months of membership in this club. Then we have to test the hypothesis H0 : μ = 15 versus Ha : μ < 15 Here n = 45, x = 13.8, and s = 4.2. Because n = 45 > 30, we can use normal approximation. Hence, the test statistic is 13.8 − 15 = −1.9166 z= √ 4.2/ 45 and p-value = P (Z < −1.9166) P (Z < −1.92) = 0.0274. Thus, we can use an α as small as 0.0274 and still reject H0 . (b) No. Because the p-value = 0.0274 is greater than α = 0.01, one cannot reject H0 .

In any hypothesis testing, after an experimenter determines the objective of an experiment and decides on the type of data to be collected, we recommend the following step-by-step procedure for hypothesis testing.

STEPS IN ANY HYPOTHESIS TESTING PROBLEM 1. State the alternative hypothesis, Ha (what is believed to be true). 2. State the null hypothesis, H0 (what is doubted to be true). 3. Decide on a level of signiﬁcance α. 4. Choose an appropriate TS and compute the observed test statistic. 5. Using the distribution of TS and α, determine the rejection region(s) (RR). 6. Conclusion: If the observed test statistic falls in the RR, reject H0 and conclude that based on the sample information, we are (1 − α)100% conﬁdent that Ha is true. Otherwise, conclude that there is not sufﬁcient evidence to reject H0 . In all the applied problems, interpret the meaning of your decision. 7. State any assumptions you made in testing the given hypothesis. 8. Compute the p-value from the null distribution of the test statistic and interpret it.

7.4.2 Hypothesis Testing for a Single Parameter Now we study the testing of a hypothesis concerning a single parameter, θ, based on a random sample X1 , . . . , Xn . Let θˆ be the sample statistic. First, we deal with tests for the population mean μ for large and small samples. Next, we study procedures for testing the population variance σ 2 . We conclude the section by studying a test procedure for the true proportion p.

364 CHAPTER 7 Hypothesis Testing

To test the hypothesis H : μ = μ0 concerning the true population mean μ, when we have a large sample (n ≥ 30) we use the test statistic Z given by Z=

X − μ0 √ S/ n

where S is the sample standard deviation and μ0 is the claimed mean under H0 (if the population variance is known, we replace S with σ. For a small random sample (n < 30), the test statistic is T =

X − μ0 √ S/ n

where μ0 is the claimed value of the true mean, and X and S are the sample mean and standard deviation, respectively. Note that we are using the lowercase letters, such as z and t, to represent the observed values of the test statistics Z and T , respectively. In practice, with raw data, it is important to verify the assumptions. For example, in the small sample case, it is important to check for normality by using normal plots. If this assumption is not satisﬁed, the nonparametric methods described in Chapter 12 may be more appropriate. In addition, because the sample statistic such as X and S will be greatly affected by the presence of outliers, drawing a box plot to check for outliers is a basic practice we should incorporate in our analysis. We now summarize the typical test of hypothesis for tests concerning population (true) mean. In order to compute the observed test statistic, z in the large sample case and t in the small sample √ √ case, calculate the values of z = (x − μ0 )/(s/ n) and t = [(x − μ0 )/(s/ n)], respectively.

SUMMARY OF HYPOTHESIS TESTS FOR μ Large Sample (n ≥ 30) To test H0 : μ = μ0 versus μ > μ0 , upper tail test μ < μ0 , lower tail test Ha : μ = μ0 , two-tailed test

Small Sample (n < 30) To test H0 : μ = μ0 versus μ > μ0 , upper tail test Ha : μ < μ0 , lower tail test μ = μ0 , two-tailed test

X − μ0 √ σ/ n Replace σ by S, if σ is unknown. Test statistic: Z =

⎧ ⎪ ⎪ ⎨z > zα , Rejection region : z < −zα , ⎪ ⎪ ⎩|z| > z , α/2

Test statistic: T =

upper tail RR lower tail RR two tail RR

X − μ0 √ S/ n

⎧ ⎪ ⎪ ⎨t > tα,n−1 , RR : t < −tα,n−1 , ⎪ ⎪ ⎩|t| > t α/2,n−1 ,

upper tail RR lower tail RR two tail RR

7.4 Hypotheses for a Single Parameter 365 Assumption: n ≥ 30

Assumption: Random sample comes from a normal population

Decision: Reject H0 , if the observed test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, keep H0 so that there is not enough evidence to conclude that Ha is true for the given α and more experiments may be needed.

Example 7.4.3 It is claimed that sports-car owners drive on the average 18,000 miles per year. A consumer ﬁrm believes that the average mileage is probably lower. To check, the consumer ﬁrm obtained information from 40 randomly selected sports-car owners that resulted in a sample mean of 17,463 miles with a sample standard deviation of 1348 miles. What can we conclude about this claim? Use α = 0.01.

Solution Let μ be the true population mean. We can formulate the hypotheses as H0 : μ = 18,000 versus Ha : μ < 18,000. The observed test statistic (for n ≥ 30) is z=

x − μ0 ∼ 17,463 − 18,000 √ √ = σ/ n 1348/ 40

= −2.52. Rejection region is {z < −z0.01 } = {z < −2.33}. Decision: Because z = −2.52 is less than −2.33, the null hypothesis is rejected at α = 0.01. There is sufficient evidence to conclude that the mean mileage on sport cars is less than 18,000 miles per year.

Example 7.4.4 In a frequently traveled stretch of the I-75 highway, where the posted speed is 70 mph, it is thought that people travel on the average of at least 75 mph. To check this claim, the following radar measurements of the speeds (in mph) is obtained for 10 vehicles traveling on this stretch of the interstate highway. 66

74

79

80

69

77

78

65

79

81

Do the data provide sufﬁcient evidence to indicate that the mean speed at which people travel on this stretch of highway is at most 75 mph? Test the appropriate hypothesis using α = 0.01. Draw a box plot and normal plot for this data, and comment.

Solution We need to test H0 : μ = 75 vs. Ha : μ > 75

366 CHAPTER 7 Hypothesis Testing

Speed

80 75 70 65

■ FIGURE 7.1 Box plot of speed data.

For this sample, the sample mean is x = 74.8 mph and the standard deviation is σ = 5.9963 mph. Hence, the observed test statistic is t=

x − μ0 74.8 − 75 √ = √ σ/ n 5.9963/ 10

= −0.10547. From the t-table, t0.019 = 2.821. Hence, the rejection region is {t > 2.821}. Because, t = −0.10547 does not fall in the rejection region, we do not reject the null hypothesis at α = 0.01. Note that we assumed that the vehicles were randomly selected and that collected data follow the normal distribution, because of the small sample size, n < 30, we use the t-test. Figures 7.1 and 7.2 are the box plot and the normal plot of the data, respectively.

99

ML Estimates Mean : 74.8 Std Dev: 5.68858

Percent

95 90 80 70 60 50 40 30 20 10 5 1 55

65

75

85

95

Data ■ FIGURE 7.2 Normal probability plot for speed.

The box plot suggests that there are no outliers present. However, the normal plot indicates that the normality assumption for this data set is not justified. Hence, it may be more appropriate to do a nonparametric test.

7.4 Hypotheses for a Single Parameter 367

Example 7.4.5 In attempting to control the strength of the wastes discharged into a nearby river, an industrial ﬁrm has taken a number of restorative measures. The ﬁrm believes that they have lowered the oxygen consuming power of their wastes from a previous mean of 450 manganate in parts per million. To test this belief, readings are taken on n = 20 successive days. A sample mean of 312.5 and the sample standard deviation 106.23 are obtained. Assume that these 20 values can be treated as a random sample from a normal population. Test the appropriate hypothesis. Use α = 0.05.

Solution Here we need to test the following hypothesis: H0 : μ = 450 vs. Ha : μ < 450 Given n = 20, x = 312.5, and s = 106.23. The observed test statistic is t=

312.5 − 450 = −5.79. √ 106.23/ 20

The rejection region for α = 0.05 and with 19 degrees of freedom is the set of t-values such that {t < −t0.05,19 } = {t < −1.729}. Decision: Because t = −5.79 is less than −1.729, reject H0 . There is sufficient evidence to confirm the firm’s belief. For large random samples, the following procedure is used to perform tests of hypotheses about the population proportion, p.

Example 7.4.6 A machine is considered to be unsatisfactory if it produces more than 8% defectives. It is suspected that the machine is unsatisfactory. A random sample of 120 items produced by the machine contains 14 defectives. Does the sample evidence support the claim that the machine is unsatisfactory? Use α = 0.01.

Solution Let Y be the number of observed defectives. This follows a binomial distribution. However, because np0 and nq0 are greater than 5, we can use a normal approximation to the binomial to test the hypothesis. So we need to test H0 : p = 0.08 versus Ha : p > 0.08. Let the point estimate of p be pˆ = (Y /n) = 0.117, the sample proportion. Then the value of the TS is pˆ − p0 0.117 − 0.08 = 0.137. z= 8 = 8 p0 q0 (0.08)(0.92) n 120 For α = 0.01, z0.01 = 2.33. Hence, the rejection region is {z > 2.33}.

368 CHAPTER 7 Hypothesis Testing

Decision: Because 0.137 is not greater than 2.33, we do not reject H0 . We conclude that the evidence does not support the claim that the machine is unsatisfactory.

SUMMARY OF LARGE SAMPLE HYPOTHESIS TEST FOR p To test H0 : p = p0 versus p > p0 ,

upper tail test

Ha : p < p0 ,

lower tail test.

Test statistic: pˆ − p0 Z= , σpˆ

8 where

σpˆ =

p0 q0 , n

⎧ ⎪ ⎨ z > zα , Rejection region : z < −zα , ⎪ ⎩|z| > z , α/2

where

q0 = 1 − p 0 .

upper tail RR lower tail RR two tail RR,

where z is the observed test statistic. Assumption: n is large. A good rule of thumb is to use the normal approximation to the binomial distribution only when np0 and n(1 − p0 ) are both greater than 5. Decision: Reject H0 , if the observed test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α and more data are needed.

Note that this an approximate test, and the test can be improved by increasing the sample size. Now we give the procedure for testing the population variance when the samples come from a normal population. SUMMARY OF HYPOTHESIS TEST FOR THE VARIANCE σ 2 To test H0 : σ 2 = σ02 versus σ 2 > σ02 , Ha : σ 2 < σ02 , σ 2 = σ02 ,

upper tail test lower tail test two-tailed test.

7.4 Hypotheses for a Single Parameter 369

Test statistic: χ2 =

(n − 1)S 2 σ02

where S 2 is the sample variance. Observed value of test statistic: (n − 1)s 2 σ02 ⎧ 2 χ2 > χα,n−1 , ⎪ ⎪ ⎨ 2 2 χ < χ1−α,n−1 , Rejection region : ⎪ ⎪ 2 2 ⎩χ2 > χ2 α/2,n−1 or χ < χ1−α/2,n−1 ,

upper tail RR lower tail RR two tail RR

2 where χα,n−1 is such that the area under the chi-square distribution with (n − 1) degrees of freedom to its right is equal to α.

Assumption: Sample comes from a normal population. Decision: Reject H0 , if the observed test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α and more data are needed.

Because the chi-square distribution is not symmetric, the “equal tails” used for the two-sided alternative may not be the best procedure. However, in real-world problems we seldom use a two tail test for the population variance.

Example 7.4.7 A physician claims that the variance in cholesterol levels of adult men in a certain laboratory is at least 100. A random sample of 25 adult males from this laboratory produced a sample standard deviation of cholesterol levels as 12. Test the physician’s claim at 5% level of signiﬁcance.

Solution To test H0 : σ 2 = 100 versus Ha : σ 2 < 100 for α = 0.05, and 24 degrees of freedom, the rejection region is 2 RR = {χ2 < χ1−α,n−1 } = {χ2 < 13.484}.

The observed value of the TS is χ2 =

(n − 1)S 2 σ02

=

(24)(144) = 34.56. 100

370 CHAPTER 7 Hypothesis Testing

Because the value of the test statistic does not fall in the rejection region, we cannot reject H0 at 5% level of significance. Here, we assumed that the 25 cholesterol measurements follow the normal distribution.

EXERCISES 7.4 7.4.1.

A random sample of 50 measurements resulted in a sample mean of 62 with a sample standard deviation 8. It is claimed that the true population mean is at least 64. (a) Is there sufﬁcient evidence to refute the claim at the 2% level of signiﬁcance? (b) What is the p-value? (c) What is the smallest value of α for which the claim will be rejected?

7.4.2.

A machine in a certain factory must be repaired if it produces more than 12% defectives among the large lot of items it produces in a week. A random sample of 175 items from a week’s production contains 45 defectives, and it is decided that the machine must be repaired. (a) Does the sample evidence support this decision? Use α = 0.02. (b) Compute the p-value.

7.4.3.

A random sample of 78 observations produced the following sums: 78 i=1

xi = 22.8,

78

(xi − x)2 = 2.05.

i=1

(a) Test the null hypothesis that μ = 0.45 against the alternative hypothesis that μ < 0.45 using α = 0.01. Also ﬁnd the p-value. (b) Test the null hypothesis that μ = 0.45 against the alternative hypothesis that μ = 0.45 using α = 0.01. Also ﬁnd the p-value. (c) What assumptions did you make for solving (a) and (b)? 7.4.4.

Consider the test H0 : μ = 35 vs. Ha : μ > 35 for a population that is normally distributed. (a) A random sample of 18 observations taken from this population produced a sample mean of 40 and a sample standard deviation of 5. Using α = 0.025, would you reject the null hypothesis? (b) Another random sample of 18 observations produced a sample mean of 36.8 and a sample standard deviation of 6.9. Using α = 0.025, would you reject the null hypothesis? (c) Compare and discuss the decisions of parts (a) and (b).

7.4.5.

According to the information obtained from a large university, professors there earned an average annual salary of $55,648 in 1998. A recent random sample of 15 professors from this university showed that they earn an average annual salary of $58,800 with a sample standard deviation of $8300. Assume that the annual salaries of all the professors in this university are normally distributed.

7.4 Hypotheses for a Single Parameter 371

(a) Suppose the probability of making a type I error is chosen to be zero. Without performing all the steps of test of hypothesis, would you accept or reject the null hypothesis that the current mean annual salary of all professors at this university is $55,648? (b) Using the 1% signiﬁcance level, can you conclude that the current mean annual salary of professors at this university is more than $55,648? 7.4.6.

A check-cashing service company found that approximately 7% of all checks submitted to the service were without sufﬁcient funds. After instituting a random check veriﬁcation system to reduce its losses, the service company found that only 70 were rejected in a random sample of 1125 that were cashed. Is there sufﬁcient evidence that the check veriﬁcation system reduced the proportion of bad checks at α = 0.01? What is the p-value associated with the test? What would you conclude at the α = 0.05 level?

7.4.7.

A manufacturer of washers provides a particular model in one of three colors, white, black, or ivory. Of the ﬁrst 1500 washers sold, it is noticed that 550 were of ivory color. Would you conclude that customers have a preference for the ivory color? Justify your answer. Use α = 0.01.

7.4.8.

A test of the breaking strength of six ropes manufactured by a company showed a mean breaking strength of 6425 lb and a standard deviation of 120 lb. However, the manufacturer claimed a mean breaking strength of 7500 lb. (a) Can we support the manufacturer’s claim at a level of signiﬁcance of 0.10? (b) Compute the p-value. What assumptions did you make for this problem?

7.4.9.

A sample of 10 observations taken from a normally distributed population produced the following data: 44

31

52

48

46

39

43

36

41

49

(a) Test the hypothesis that H0 : μ = 44 vs. Ha : μ = 44 using α = 0.10. Draw a box plot and normal plot for this data, and comment. (b) Find a 90% conﬁdence interval for the population mean μ. (c) Discuss the meanings of (a) and (b). What can we conclude? 7.4.10.

The principal of a charter school in Tampa believes that the IQs of its students are above the national average of 100. From the past experience, IQ is normally distributed with a standard deviation of 10. A random sample of 20 students is selected from this school and their IQs are observed. The following are the observed values. 95 113

91 100

110 100

93 133 124 116

119 113 113 110

107 106

110 115

89 113

(a) Test for the normality of the data (b) Do the IQs of students at the school run above the national average at α = 0.01? 7.4.11.

In order to ﬁnd out whether children with chronic diarrhea have the same average hemoglobin level (Hb) that is normally seen in healthy children in the same area, a random

372 CHAPTER 7 Hypothesis Testing

sample of 10 children with chronic diarrhea are selected and their Hb levels (g/dL) are obtained as follows. 12.3 11.4 14.2 15.3 14.8 13.8 11.1 15.1 15.8 13.2

Do the data provide sufﬁcient evidence to indicate that the mean Hb level for children with chronic diarrhea is less than that of the normal value of 14.6 g/dL? Test the appropriate hypothesis using α = 0.01. Draw a box plot and normal plot for this data, and comment. 7.4.12.

A company that manufactures precision special-alloy steel shafts claims that the variance in the diameters of shafts is no more than 0.0003. A random sample of 10 shafts gave a sample variance of 0.00027. At the 5% level of signiﬁcance, test whether the company’s claim can be substantiated.

7.4.13.

It was claimed that the average annual expenditures per consumer unit had continued to rise, as measured by the Consumer Price Index annual averages (Bureau of Labor Statistics report, 1995). To test this claim, 100 consumer units were randomly selected in 1995 and found to have an average annual expenditure of $32,277 with a standard deviation of $1200. Assuming that the average annual expenditure of all consumer units was $30,692 in 1994, test at the 5% signiﬁcance level whether the annual expenditure per consumer unit had really increased from 1994 to 1995.

7.4.14.

It is claimed that two of three Americans say that the chances of world peace are seriously threatened by the nuclear capabilities of other countries. If in a random sample of 400 Americans, it is found that only 252 hold this view, do you think the claim is correct? Use α = 0.05. State any assumptions you make in solving this problem.

7.4.15.

According to the Bureau of Labor Statistics (1996), the average price of a gallon of gasoline in all U.S. cities in the United States in January 1996 was $1.129. A later random sample in 24 cities found the mean price to be $1.24 with a standard deviation of 0.01. Test at α = 0.05 to see whether the average price of a gallon of gas in the cities had recently changed.

7.4.16.

A manufacturer claims that the mean life of batteries manufactured by his company is at least 44 months. A random sample of 40 of these batteries was tested, resulting in a sample mean life of 41 months with a sample standard deviation of 16 months. Test at α = 0.01 whether the manufacturer’s claim is correct.

7.5 TESTING OF HYPOTHESES FOR TWO SAMPLES In this section we study the hypothesis testing procedures for comparing the means and variances of two populations. For example, suppose that we want to determine whether a particular drug is effective for a certain illness. The sample subjects will be randomly selected from a large pool of people with that particular illness and will be assigned randomly to the two groups. To one group we will administer a placebo; to the other we will administer the drug of interest. After a period of time, we measure a physical characteristic, say the blood pressure, of each subject that is an indicator of the severity of the illness. The question is whether the drug can be considered effective on the population from which our samples have been selected. We will consider the cases of independent and dependent samples.

7.5 Testing of Hypotheses for Two Samples 373

7.5.1 Independent Samples Two random samples are drawn independently of each other from two populations, and the sample information is obtained. We are interested in testing a hypothesis about the difference of the true means. Let X11 , . . . , X1n be a random sample from population 1 with mean μ1 and variance σ12 , and X21 , . . . , X2n be a random sample from population 2 with mean μ2 and variance σ22 . Let Xi , i = 1, 2, represent the respective sample means and Si2 , i = 1, 2, represent the sample variances. In this case, we shall consider following three cases in testing hypotheses about μ1 and μ2 : (i) when σ12 and σ22 are known, (ii) when σ12 and σ22 are unknown and n1 ≥ 30 and n2 ≥ 30, and (iii) when σ12 and σ22 are unknown and n1 < 30 and n2 < 30. In case (iii) we have the following two possibilities, (a) σ12 = σ22 , and (b) σ12 = σ22 . In the large sample case, knowledge of population variances σ12 and σ22 does not make much difference. If the population variances are unknown, we could replace them with sample variances as an approximation. If both n1 ≥ 30 and n2 ≥ 30 (large sample case), we can use normal approximation. The following box sums up a large sample hypothesis testing procedure for the difference of means for the large sample case. SUMMARY OF HYPOTHESIS TEST FOR μ1 − μ2 FOR LARGE SAMPLES (n1 & n2 ≥ 30) To test H0 : μ1 − μ2 = D0 versus

⎧ ⎪ ⎨μ1 − μ2 > D0 , Ha : μ1 − μ2 < D0 , ⎪ ⎩ μ1 − μ2 = D0 ,

upper tailed test lower tailed test two-tailed test.

The test statistic is Z=

X 1 − X 2 − D0 . + σ12 σ22 + n2 n1

Replace σi by Si , if σi ,i = 1,2 are not known. Rejection region is ⎧ ⎪ ⎪ ⎨z > zα , RR :

z < −zα , ⎪ ⎪ ⎩ |z| > zα/2 ,

upper tail RR lower tail RR two tail RR,

where z is the observed test statistic given by x 1 − x 2 − D0 . z= + σ22 σ12 + n2 n1

374 CHAPTER 7 Hypothesis Testing

Assumption: The samples are independent and n1 and n2 ≥ 30. Decision: Reject H0 , if test statistic falls in the RR and conclude that Ha is true with (1 − a)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α and more experiments are needed.

Example 7.5.1 In a salary equity study of faculty at a certain university, sample salaries of 50 male assistant professors and 50 female assistant professors yielded the following basic statistics.

Male assistant professor Female assistant professor

Sample mean salary $36,400 $34,200

Sample standard deviation 360 220

Test the hypothesis that the mean salary of male assistant professors is more than the mean salary of female assistant professors at this university. Use α = 0.05.

Solution Let μ1 be the true mean salary for male assistant professors and μ2 be the true mean salary for female assistant professors at this university. To test H0 : μ1 − μ2 = 0 vs. Ha : μ1 − μ2 > 0 the test statistic is 36,400 − 34,200 x 1 − x 2 − D0 = 36.872. = + z= + 2 2 s2 s1 (360)2 (220)2 + + n1 n2 50 50 The rejection region for α = 0.05 is {z > 1.645}. Because z = 36.872 > 1.645, we reject the null hypothesis at α = 0.05. We conclude that the salary of male assistant professors at this university is higher than that of female assistant professors for α = 0.05. Note that even though σ12 and σ22 are unknown, because n1 ≥ 30 and n2 ≥ 30, we could replace σ12 and σ22 by the respective sample variances. We are assuming that the salaries of male and female are sampled independently of each other.

Given next is the procedure we follow to compare the true means from two independent normal populations when n1 and n2 are small (n1 < 30 or n2 < 30) and we can assume homogeneity in the population variances, that is, σ12 = σ22 . In this case, we pool the sample variances to obtain a point estimate of the common variance.

7.5 Testing of Hypotheses for Two Samples 375

COMPARISON OF TWO POPULATION MEANS, SMALL SAMPLE CASE (POOLED t-TEST) To test H0 : μ1 − μ2 = D0 versus μ1 − μ2 > D0 ,

upper tailed test

Ha : μ1 − μ2 < D0 , μ1 − μ2 = D0 ,

lower tailed test two-tailed test.

The test statistic is T =

X 1 − X 2 − D0 9 Sp n11 + n12

Here the pooled sample variance is Sp2 =

(n1 − 1)S12 + (n2 − 1)S22 . n1 + n2 − 2

Then the rejection region is ⎧ ⎪ t > tα , ⎪ ⎨ RR : t < −tα , ⎪ ⎪ ⎩|t| > t

α/2 ,

upper tailed test lower tail test two-tailed test

where t is the observed test statistic and tα is based on (n1 + n2 − 2) degrees of freedom, and such that P(T > tα ) = α. Decision: Reject H0 , if test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for given α. Assumptions: The samples are independent and come from normal populations with means μ1 and μ2 , and with the (unknown) but equal variances, that is, σ12 = σ22 .

Now we shall consider the case where σ12 and σ22 are unknown and cannot be assumed to be equal. In such a case the following test is often used. For the hypothesis ⎧ ⎪ ⎨μ1 − μ2 > D0 H0 : μ1 − μ2 = D0 vs. H0 : μ1 − μ2 < D0 ⎪ ⎩μ − μ = D 1 2 0

376 CHAPTER 7 Hypothesis Testing

deﬁne the test statistic Tν as Tν =

X 1 − X 2 − D0 8 S12 S22 n1 + n2

where Tν has a t-distribution with ν degrees of freedom, and ν=

# 2 $2 (s1 /n1 ) + (s22 /n2 ) (s2 /n2 )2 (s12 /n1 )2 + 2 n1 − 1 n2 − 1

.

The value of ν will not necessarily be an integer. In that case, we will round it down to the nearest integer. This method of hypothesis testing with unequal variances is called the Smith–Satterthwaite procedure. Even though this procedure is not widely used, some simulation studies have shown that the Smith–Satterthwaite procedure perform well when variances are unequal and it gives results that are more or less equivalent to those obtained with the pooled t-test when the variances are equal. However, when the sample sizes are approximately equal, the pooled t-test may still be used. Note that in addressing the question which of the cases (iii)(a) or (iii)(b) to use in a given problem, we suggest that if the point estimates S12 of σ12 , and S22 of σ22 are approximately the same, then it is logical to assume homogeneity, σ12 = σ22 and use (iii)(a), whereas if S12 and S22 are signiﬁcantly different we use (iii)(b). More appropriately, we have tests that can be used to test hypotheses concerning σ12 = σ22 or σ12 = σ22 , known as the F -test, which we discuss at the end of this subsection.

Example 7.5.2 The intelligence quotients (IQs) of 17 students from one area of a city showed a sample mean of 106 with a sample standard deviation of 10, whereas the IQs of 14 students from another area chosen independently showed a sample mean of 109 with a sample standard deviation of 7. Is there a signiﬁcant difference between the IQs of the two groups at α = 0.02? Assume that the population variances are equal.

Solution We test H0 : μ1 − μ2 = 0 vs. Ha : μ1 − μ2 = 0 Here n1 = 17, x1 = 106, and s1 = 10. Also, n2 = 14, x2 = 109, and s2 = 7. We have 2 = sp

=

(n1 − 1)s12 + (n2 − 1)s22 n1 + n2 − 2 (16)(10)2 + (13)(7)2 = 77.138. 29

7.5 Testing of Hypotheses for Two Samples 377

The test statistic is T =

106 − 109 X 1 − X 2 − D0 = −0.94644. = 8 8 1 √ 1 1 1 sp + + 77.138 n1 n2 17 14

For α = 0.02, t0.01,29 = 2.462. Hence, the rejection region is t < − 2.462 or t > 2.462. Because the observed value of the test statistic, T = −0.94644, does not fall in the rejection region, there is not enough evidence to conclude that the mean IQs are different for the two groups. Here we assume that the two samples are independent and taken from normal populations.

Example 7.5.3 Assume that two populations are normally distributed with unknown and unequal variances. Two independent samples were drawn from these populations and the data obtained resulted in the following basic statistics: n1 = 18

x1 = 20.17

s1 = 4.3

n2 = 12

x2 = 19.23

s2 = 3.8

Test at the 5% signiﬁcance level whether the two population means are different.

Solution We need to test the hypothesis H0 : μ1 − μ2 = 0 versus Ha : μ1 − μ2 = 0. Here n1 = 18, x1 = 20.17, and s1 = 4.3. Also, n2 = 12, x2 = 19.23, and s2 = 3.8. The degrees of freedom for the t-distribution are given by ν=

2 2 s1 /n1 + s22 /n2 (s2 /n2 )2 (s12 /n1 )2 + 2 n1 − 1 n2 − 1 2 (4.3) (3.8)2 2 18 + 12

=

(4.3)2 18 17

2

+

(3.8)2 12 11

2 = 25.685.

Hence, we have ν = 25 degrees of freedom. For α = 0.05, t0.025,25 = 2.060. Thus, the rejection region is t < −2.060 or t > 2.060. The test statistic is given by Tν =

x 1 − x 2 − D0 8 S12 S22 n1 + n2

378 CHAPTER 7 Hypothesis Testing

= +

20.17 − 19.23 (3.8)2 (4.3)2 + 18 12

= 0.62939.

Because the observed value of the test statistic, Tν = 0.62939, does not fall in the rejection region, we do not reject the null hypothesis. At α = 0.05 there is not enough evidence to conclude that the population means are different. Note that the assumptions we made are that the samples are independent and came from two normal populations. No homogeneity assumption is made.

Example 7.5.4 Infrequent or suspended menstruation can be a symptom of serious metabolic disorders in women. In a study to compare the effect of jogging and running on the number of menses, two independent subgroups were chosen from a large group of women, who were similar in physical activity (aside from running), heights, occupations, distribution of ages, and type of birth control methods being used. The ﬁrst group consisted of a random sample of 26 women joggers who jogged “slow and easy” 5 to 30 miles per week, and the second group consisted of a random sample of 26 women runners who ran more than 30 miles per week and combined long, slow distance with speed work. The following summary statistics were obtained (E. Dale, D. H. Gerlach, and A. L. Wilhite, “Menstrual Dysfunction in Distance Runners,” Obstet. Gynecol. 54, 47–53, 1979). Joggers x1 = 10.1, s1 = 2.1 Runners x2 = 9.1, s2 = 2.4 Using α = 0.05, (a) test for differences in mean number of menses for each group assuming equality of population variances, and (b) test for differences in mean number of menses for each group assuming inequality of population variances.

Solution Here we need to test H0 : μ1 − μ2 = 0 versus Ha : μ1 − μ2 = 0. Here, n1 = 26, x1 = 10.1, and s1 = 2.1. Also, n2 = 26, x2 = 9.1, and s2 = 2.4. (a) Under the assumption σ12 = σ22 , we have 2 = sp

=

(n1 − 1)s12 + (n2 − 1)s22 n1 + n2 − 2 (25)(2.1)2 + (25)(2.4)2 = 5.085. 50

The test statistic is T =

X 1 − X 2 − D0 9 sp n11 + n12

7.5 Testing of Hypotheses for Two Samples 379 10.1 − 9.1 = √ = 1.5989. 9 1 + 1 5.085 26 26 For α = 0.05, t0.025,50 ≈ 1.96. Hence, the rejection region is t < −1.96 and t > 1.96. Because T = 1.589 does not fall in the rejection region, we do not reject the null hypothesis. At α = 0.05 there is not enough evidence to conclude that the population mean number of menses for joggers and runners are different. (b) Under the assumption σ12 = σ22 , we have ν=

2 2 s1 /n1 + s22 /n2 (s12 /n1 )2 (s22 /n2 )2 n1 −1 + n2 −1

(2.1)2 (2.4)2 2 26 + 26 =

2 2 = 49.134. (2.1)2 (2.4)2 26 26 + 25 25

Hence, we have ν = 49 degrees of freedom. Because this value is large, the rejection region is still approximately t < − 1.96 and t > 1.96. Hence, the conclusion is the same as that of part (a). In both parts (a) and (b), we assumed that the samples are independent and came from two normal populations.

Now we present the summary of the test procedure for testing the difference of two proportions, inherent in two binomial populations. Here, again we assume that the binomial distribution is approximated by the normal distribution and thus it is an approximate test.

SUMMARY OF HYPOTHESIS TEST FOR (p1 − p2 ) FOR LARGE SAMPLES (ni pi > 5 AND ni qi > 5, FOR i = 1, 2) To test H0 : p1 − p2 = D0 versus p1 − p2 < D0 , Ha : p1 − p2 > D0 , p1 − p2 = D0 ,

upper tailed test lower tailed test two-tailed test

at signiﬁcance level α, the test statistic is pˆ 1 − pˆ 2 − D0 Z= 9 pˆ 1 qˆ 1 pˆ 2 qˆ 2 n1 + n2 where z is the observed value of Z .

380 CHAPTER 7 Hypothesis Testing

The rejection region is ⎧ ⎪ ⎨ z > zα , RR : z < −zα , ⎪ ⎩|z| > z , α/2

upper tailed RR lower tailed RR two-tailed RR

Assumption: The samples are independent and ni pi > 5 and ni qi > 5, for i = 1,2. Decision: Reject H0 if the test statistic falls in the RR and conclude that Ha is true with (1 − a)100% conﬁdence. Otherwise, do not reject H0 , because there is not enough evidence to conclude that Ha is true for given α and more experiments are needed.

Example 7.5.5 Because of the impact of the global economy on a high-wage country such as the United States, it is claimed that the domestic content in manufacturing industries fell between 1977 and 1997. A survey of 36 randomly picked U.S. companies gave the proportion of domestic content total manufacturing in 1977 as 0.37 and in 1997 as 0.36. At the 1% level of signiﬁcance, test the claim that the domestic content really fell during the period 1977–1997.

Solution Let p1 be the domestic content in 1977 and p2 be the domestic content in 1997. Given n1 = n2 = 36, pˆ 1 = 0.37 and pˆ 2 = 0.36. We need to test H0 : p1 − p2 = 0 vs. Ha : p1 − p2 > 0. The test statistic is z= 9

= 9

pˆ 1 − pˆ 2 pˆ 1 qˆ 2 pˆ 1 qˆ 2 n1 + n2 0.37 − 0.36 (0.37)(0.63) + (0.36)(0.64) 36 36

= 0.08813.

For α = 0.01, z0.01 = 2.325. Hence, the rejection region is z > 2.325. Because the observed value of the test statistic does not fall in the rejection region, at α = 0.01, there is not enough evidence to conclude that the domestic content in manufacturing industries fell between 1977 and 1997.

Let X1 , . . . , Xn and Y1 , . . . , Yn be two independent random samples from two normal populations with sample variances s12 and s22 , respectively. The problem here is of testing for the equality of the

7.5 Testing of Hypotheses for Two Samples 381

variances, H0 : σ12 = σ22 . We have already seen in Chapter 4 that S 2 /σ 2 F = 12 12 S2 /σ2

follows the F -distribution with ν1 = n1 − 1 numerator and ν2 = n2 − 1 degrees of freedom. Under the assumption H0 : σ12 = σ22 , we have S2 F = 12 S2

which has an F -distribution with (ν1 , ν2 ) degrees of freedom. We summarize the test procedure for the equality of variances.

TESTING FOR THE EQUALITY OF VARIANCES To test H0 : σ12 = σ22 versus σ12 > σ22 ,

lower tailed test

Ha : σ12 < σ22 , σ12 = σ22 ,

upper tailed test two-tailed test

at signiﬁcance level α, the test statistic is S2 F = 12 . S2 The rejection region is ⎧ ⎪ ⎨ RR :

f > Fα (ν1 ,ν2 ), f < F1−α (ν1 ,ν2 ), ⎪ ⎩f > F (ν ,ν ) or f < F α/2 1 2 1−α/2 (ν1 ,ν2 ),

where f is the observed test statistic given by f =

upper tailed RR lower tailed RR two-tailed RR

s12 . s22

Decision: Reject H0 if the test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, keep H0 , because there is not enough evidence to conclude that Ha is true for a given α and more experiments are needed. Assumption: (i) The two random samples are independent. (ii) Both populations are normal.

Recall from Section 4.2 that in order to ﬁnd F1−α (ν1 , ν2 ), we use the identity F1−α (ν1 , ν2 ) = (1/Fα (ν2 , ν1 )).

382 CHAPTER 7 Hypothesis Testing

Example 7.5.6 Consider two independent random samples X1 , . . . , Xn from an N(μ1 , σ12 ) distribution and Y1 , . . . , Yn from an N(μ2 , σ22 ) distribution. Test H0 : σ12 = σ22 versus Ha : σ12 = σ22 for the following basic statistics: n1 = 25, x1 = 410, s12 = 95, and n2 = 16, x2 = 390, s22 = 300 Use α = 0.20.

Solution Test H0 : σ12 = σ22 versus Ha : σ12 = σ22 . This is a two-tailed test. Here the degrees of freedom are ν1 = 24 and ν2 = 15. The test statistic is s2 95 = 0.317. F = 12 = 300 s2 From the F -table, F0.10 (24, 15) = 1.90 and F0.90 (24, 15) =(1/F0.10 (15, 24)) = 0.50. Hence, the rejection region is F > 1.90 or F < 0.56. Because the observed value of the test statistic, 0.317, is less than 0.56, we reject the null hypothesis. There is evidence that the population variances are not equal.

7.5.2 Dependent Samples We now consider the case where the two random samples are not independent. When two samples are dependent (the samples are dependent if one sample is related to the other), then each data point in one sample can be coupled in some natural, nonrandom fashion with each data point in the second sample. This situation occurs when each individual data point within a sample is paired (matched) to an individual data point in the second sample. The pairing may be the result of the individual observations in the two samples: (1) representing before and after a program (such as weight before and after following a certain diet program), (2) sharing the same characteristic, (3) being matched by location, (4) being matched by time, (5) control and experimental, and so forth. Let (X1i , X2i ), for i = 1, 2, . . . , n, be a random sample. X1i , and X2j (i = j) are independent. To test the signiﬁcance of the difference between two population means when the samples are dependent, we ﬁrst calculate for each pair of scores the difference, Di = X1i − X2i , i = 1, 2, . . . , n, between the two scores. Let μD = E(Di ). Because pairs of observations form a random sample D1 , . . . , Dn are independent and identically distributed random variables, if d1 , . . . , dn are the observed values of D1 , . . . , Dn , then we deﬁne

d=

1 n

n i=1

1 di and sd2 = n−1

n i=1

(di − d)2 =

2 n n 1 di2 − di n i=1 i=1 n−1

.

Now the testing for these n observed differences will proceed as in the case of a single sample. If the number of differences is large (n ≥ 30), large sample inferential methods for one sample case can be used for the paired differences. We now summarize the hypothesis testing procedure for small samples.

7.5 Testing of Hypotheses for Two Samples 383

SUMMARY OF TESTING FOR MATCHED PAIRS EXPERIMENT To test μD > d0 , H0 : μD = d0 versus Ha : μD < d0 , μD = d0 ,

upper tail test lower tail test two-tailed test

√0 (this approximately follows a Student t-distribution with (n − 1) degrees of the test statistic: T = D−D SD / n

freedom). The rejection region is

⎧ ⎪ ⎨ t > tα,n−1 , t < −tα,n−1 , ⎪ ⎩|t| > t α/2,n−1 ,

upper tail RR lower tail RR two-tailed RR

where t is the observed test statistic. Assumptions: The differences are approximately normally distributed. Decision: Reject H0 if the test statistic falls in the RR and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 , because there is not enough evidence to conclude that Ha is true for a given α and more data are needed.

Example 7.5.7 A new diet and exercise program has been advertised as remarkable way to reduce blood glucose levels in diabetic patients. Ten randomly selected diabetic patients are put on the program, and the results after 1 month are given by the following table: Before 268 225 252 192 307 228 246 298 231 185 After 106 186 223 110 203 101 211 176 194 203 Do the data provide sufﬁcient evidence to support the claim that the new program reduces blood glucose level in diabetic patients? Use α = 0.05.

Solution We need to test the hypothesis H0 : μD = 0

vs.

Ha : μD < 0.

First we calculate the difference of each pair given in the following table. Before 268 225 252 192 307 228 246 298 231 185 After 106 186 223 110 203 101 211 176 194 203 Difference −162 −39 −29 −82 −104 −127 −35 −122 −37 18 (after−before)

384 CHAPTER 7 Hypothesis Testing

From the table, the mean of the differences is d = −71.9 and the standard deviation sd = 56.2. The test statistic is t=

d − d0 −71.9 √ = −4.0457 ≈ −4.05. √ = sd / n 56.2/ 10

From the t-table, t0,05,9 = 1.833. Because the observed value of t = − 4.05 < −t0,05,9 = −1.833, we reject the null hypothesis and conclude that the sample evidence suggests that the new diet and exercise program is effective.

We can also obtain a (1 − α)100% conﬁdence interval for μD using the formula

S S D − tα/2 √d , D + tα/2 √d n n

where tα/2 is obtained from the t-table with (n − 1) degrees of freedom. The interpretation of the conﬁdence interval is identical to the earlier interpretation.

Example 7.5.8 For the data in Example 7.5.7, obtain a 95% conﬁdence interval for μD and interpret its meaning.

Solution We have already calculated d = − 71.9 and sd = 56.2. From the t-table, t0.025,9 = 2.262. Hence, a 95% confidence interval for μD is (−112.1, −31.7). That is, P(−112.1 ≤ μD ≤ −31.7) = 0.95. Note that μD = μ1 − μ2 , and from the confidence limits we can conclude with 95% confidence that μ2 is always greater than μ1 , that is, μ2 > μ1 .

It is interesting to compare the matched pairs test with the corresponding two independent sample test. One of the natural questions is, why must we take paired differences and then calculate the mean and standard deviation for the differences—why can’t we just take the difference of means of each 2 need not be equal to sample, as we did for independent samples? The answer lies in the fact that σD 2 σ(X . Assume that −X ) 1

2

E(Xji ) = μj , Var(Xji ) = σj2 , for j = 1, 2,

and Cov(X1i , X2i ) = ρσ1 σ2

where ρ denotes the assumed common correlation coefﬁcient of the pair (X1i , X2i ) for i = 1, 2, . . . , n. Because the values of Di , i = 1, 2, . . . , n, are independent and identically distributed, μD = E(Di ) = E(X1i ) − E(X2i ) = μ1 − μ2

7.5 Testing of Hypotheses for Two Samples 385

and 2 = Var(D ) = Var(X ) + Var(X ) − 2Cov(X , X ) σD i 1i 2i 1i 2i

= σ12 + σ22 − 2ρσ1 σ2 .

From these calculations, E(D) = μD = μ1 − μ2

and σ2 1 σ 2 = Var(D) = D = (σ12 + σ22 − 2ρσ1 σ2 ). D n n

Now, if the samples were independent with n1 = n2 = n, E(X1 − X2 ) = μ1 − μ2

and σ2

(X1 −X2 )

=

1 2 (σ + σ22 ). n 1

2 < σ2 Hence, if ρ > 0, then σD . As a result, we can see that the matched pairs test reduces any (X1 −X2 ) variability introduced by differences in physical factors in comparison to the independent samples test when ρ > 0. It is also important to observe that normality assumption for the difference does not imply that the individual samples themselves are normal. Also, in a matched pairs experiment, there is no need to assume the equality of variances for the two populations. Matching also reduces degrees of freedom, because in case of two independent samples, the degrees of freedom is (n1 + n2 − 2), whereas for the case of two dependent samples it is only (n − 1).

EXERCISES 7.5 7.5.1.

Two sets of elementary school children were taught to read by different methods, 50 by each method. At the conclusion of the instructional period, a reading test gave results y1 = 74, y2 = 71, s1 = 9, and s2 = 10. What is the attained signiﬁcance level if you wish to see if there is evidence of a real difference between the two population means? What would you conclude if you desired an α-value of 0.05?

7.5.2.

The following information was obtained from two independent samples selected from two normally distributed populations with unknown but equal variances. Sample 1 14 15 11 14 10 8 13 10 12 16 15 Sample 2 17 16 21 12 20 18 16 14 21 20 13 20 13

Test at the 2% signiﬁcance level whether μ1 is lower than μ2 .

386 CHAPTER 7 Hypothesis Testing

7.5.3.

In the academic year 1997–1998, two random samples of 25 male professors and 23 female professors from a large university produced a mean salary for male professors of $58,550 with a standard deviation of $4000 and an average for female professors of $53,700 with a standard deviation of $3200. At the 5% signiﬁcance level, can you conclude that the mean salary of all male professors for 1997–1998 was higher than that of all female professors? Assume that the salaries of male and female professors are both normally distributed with equal standard deviations.

7.5.4.

It is believed that the effects of smoking differ depending on race. The following table gives the results of a statistical study for this question.

Whites African Americans

Number in the study 400 280

Average number of cigarettes per day 15 15

Number of lung cancer cases 78 70

Do the data indicate that African Americans are more likely to develop lung cancer due to smoking? Use α = 0.05. 7.5.5.

A supermarket chain is considering two sources A and B for the purchase of 50-pound bags of onions. The following table gives the results of a study.

Number of bags weighed Mean weight Sample variance

Source A 80 105.9 0.21

Source B 100 100.5 0.19

Test at α = 0.05 whether there is a difference in the mean weights. 7.5.6.

In order to compare the mean Hemoglobin (Hb) levels of well-nourished and undernourished groups of children, random samples from each of these groups yielded the following summary.

Well nourished Undernourished

Number of children 95 75

Sample mean 11.2 9.8

Sample standard deviation 0.9 1.2

Test at α = 0.01 whether the mean Hb levels of well-nourished children were higher than those of undernourished children. 7.5.7.

An aquaculture farm takes water from a stream and returns it after it has circulated through the ﬁsh tanks. In order to ﬁnd out how much organic matter is left in the waste water after the circulation, some samples of the water are taken at the intake and other samples are taken at the downstream outlet and tested for biochemical oxygen demand (BOD). BOD is a common environmental measure of the quantity of oxygen consumed by microorganisms during the decomposition of organic matter. If BOD increases, it can be said that the waste

7.5 Testing of Hypotheses for Two Samples 387

matter contains more organic matter than the stream can handle. The following table gives data for this problem. Upstream 9.0 6.8 6.5 8.0 7.7 8.6 6.8 8.9 7.2 7.0 Downstream 10.2 10.2 9.9 11.1 9.6 8.7 9.6 9.7 10.4 8.1

Assuming that the samples come from a normal distribution, (a) Test that the mean BOD for the downstream samples is less than for the samples upstream at α = 0.05. Assume that the variances are equal. (b) Test for the equality of the variances at α = 0.05. (c) In parts (a) and (b), we assumed samples are independent. Now, we feel this assumption is not reasonable. Assuming that the difference of each pair is approximately normal, test that the mean BOD for the downstream samples is less than for the upstream samples at α = 0.05. 7.5.8.

Suppose we want to know the effect on driving of a drug for cold and allergy, in a study in which the same people were tested twice, once after 1 hour of taking the drug and once when no drug is taken. Suppose we obtain the following data, which represent the number of cones (placed in a certain pattern) knocked down by each of the nine individuals before taking the drug and after an hour of taking the drug. No drug After drug

0 1

0 5

3 6

2 5

0 5

0 5

3 6

3 1

1 6

Assuming that the difference of each pair is coming from an approximately normal distribution, test if there is any difference in the individuals’ driving ability under the two conditions. Use α = 0.05. 7.5.9.

Suppose that we want to evaluate the role of intravenous pulse cyclophosphamide (IVCP) infusion in the management of nephrotic syndrome in children with steroid resistance. Children were given a monthly infusion of IVCP in a dose of 500 to 750 mg/m2 . The following data (source: S. Gulati and V. Kher, “Intravenous pulse cyclophosphamide—A new regime for steroid resistant focal segmental glomerulosclerosis,” Indian Pediatr. 37, 2000) represent levels of serum albumin (g/dL) before and after IVCP in 14 randomly selected children with nephrotic syndrome. Pre-IVCP 2.0 2.5 1.5 2.0 2.3 2.1 2.3 1.0 2.2 1.8 2.0 2.0 1.5 3.4 Post-IVCP 3.5 4.3 4.0 4.0 3.8 2.4 3.5 1.7 3.8 3.6 3.8 3.8 4.1 3.4

Assuming that the samples come from a normal distribution: (a) Test whether the mean Pre-IVCP is less than the mean Post-IVCP at α = 0.05. Assume that the variances are equal. (b) Test for the equality of the variances at α = 0.05. (c) In parts (a) and (b), we assumed that the samples are independent. Now, we feel this assumption is not reasonable. Assuming that the difference of each pair is approximately normal, test that the mean Pre-IVCP is less than the Post-IVCP at α = 0.05.

388 CHAPTER 7 Hypothesis Testing

7.5.10.

2 is an unbiased estimator of σ 2 . Show that SD D

7.5.11.

Test H0 : σ12 = σ22 versus Ha : σ12 = σ22 for the following data. n1 = 10, x1 = 71, s12 = 64

and

n2 = 25, x2 = 131, s22 = 96.

Use α = 0.10. 7.5.12.

The IQs of 17 students from one area of a city showed a mean of 106 with a standard deviation of 10, whereas the IQs of 14 students from another area showed a mean of 109 with a standard deviation of 7. Test for equality of variances between the IQs of the two groups at α = 0.02.

7.5.13.

The following data give SAT mean scores for math by state for 1989 and 1999 for 20 randomly selected states (source: The World Almanac and Book of Facts 2000). State Arizona Connecticut Alabama Indiana Kansas Oregon Nebraska New York Virginia Washington Illinois North Carolina Georgia Nevada Ohio New Hampshire

1989 523 498 539 487 561 509 560 496 507 515 539 469 475 512 520 510

1999 525 509 555 498 576 525 571 502 499 526 585 493 482 517 568 518

Assuming that the samples come from a normal distribution: (a) Test that the mean SAT score for math in 1999 is greater than that in 1989 at α = 0.05. Assume the variances are equal. (b) Test for the equality of the variances at α = 0.05.

7.6 CHI-SQUARE TESTS FOR COUNT DATA In this section, we study several commonly used tests for count data. These are basically large sample tests based on a χ2 -approximation. Suppose that we have outcomes of a multinomial experiment that consists of K mutually exclusive and exhaustive events A1 , . . . , Ak . Let P(Ai ) = pi , i = 1, 2, . . . , k. Then ni=1 pi = 1. Let the experiment be repeated n times, and let Xi (i = 1, 2, . . . , k) represent the number of times the event Ai occurs. Then (X1 , . . . , Xk ) have a multinomial distribution with parameters n, p1 , . . . , pk .

7.6 Chi-Square Tests for Count Data 389

Let Q2 =

k (Xi − npi )2 . (Xi − npi )2 i=1

It can be shown that for large n, the random variable Q2 is approximately χ2 -distributed with (k − 1) degrees of freedom. It is usual to demand npi ≥ 5 (i = 1, 2, . . . , k) for the approximation to be valid, although the approximation generally works well if for only a few values of i (about 20%), npi ≥ 1 and the rest (about 80%) satisfy the condition npi ≥ 5. This statistic was proposed by Karl Pearson in 1900. It should be noted that the χ2 -tests that we discuss in this section are approximate tests valid for large samples. Often Xi is called the observed frequency and is denoted by Oi (this is the observed value in class i), and npi is called the expected frequency and is denoted by Ei (this is the theoretical distribution frequency under the null hypothesis). Thus, with these notations, we get Q2 =

k (Oi − Ei )2 i=1

Ei

.

Example 7.6.1 A plant geneticist grows 200 progeny from a cross that is hypothesized to result in a 3 : 1 phenotypic ratio of red-ﬂowered to white-ﬂowered plants. Suppose the cross produces 170 red- to 30 white-ﬂowered plants. Calculate the value of Q2 for this experiment.

Solution There are two categories of data totaling n = 200. Hence, k = 2. Let i = 1 represent red-flowered and i = 2 represent white-flowered plants. Then O1 = 170, and O2 = 30. Here, H0 : The flower color population ratio is not different from 3 : 1, and the alternate is Ha : The flower color population sampled has a flower color ratio that is not 3 red : 1 white. Under the null hypothesis, the expected frequencies are E1 = (200)(3/4) = 150, and E2 = (200)(1/4) = 50. Hence, Q2 =

2 (Oi − Ei )2 Ei i=1

=

(30 − 50)2 (170 − 150)2 + = 10.667. 150 50

The type of calculation in Example 7.6.1 gives a measure of how close our observed frequencies come to the expected frequencies and is referred to as a measure of goodness of ﬁt. Smaller values of Q2 values indicate better ﬁt. One of the most frequent uses of the χ2 -test is in comparison of observed frequencies. Unless the sample size is exactly 100, percentages cannot be used. These are approximate tests. Let the random

390 CHAPTER 7 Hypothesis Testing

variables (X1 , . . . , Xk ) have a multinomial distribution with parameters n, p1 , . . . , pk . Let n be known. We will now present some important tests based on the chi-square statistic.

7.6.1 Testing the Parameters of Multinomial Distribution: Goodness-of-Fit Test Let an experiment have k mutually exclusive and exhaustive outcomes A1 , A2 , . . . , Ak . We would like to test the null hypothesis that all the pi = p(Ai ), i = 1, 2, . . . , k are equal to known numbers pi0 , i = 1, 2, . . . , k. We now summarize the test procedure.

TESTING THE PARAMETERS OF A MULTINOMIAL DISTRIBUTION (SUMMARY) To test H0 : p1 = p10 , . . . , pk = pk 0 versus Ha : At least one of the probabilities is different from the hypothesized value. The test is always a one-sided upper tail test. Let Oi be the observed frequency, Ei = npi0 be the expected frequency (frequency under the null hypothesis), and k be the number of classes. The test statistic is Q2 =

k (Oi − Ei )2 . Ei i=1

The test statistic Q 2 has an approximate chi-square distribution with k − 1 degrees of freedom. The rejection region is 2 Q 2 ≥ χα,k −1 .

Assumption: Ei ≥ 5: Exact methods are available. Computing the power of this test is difﬁcult.

This test is known as the goodness-of-ﬁt test. It implies that if the observed data are very close to the expected data, we have a very good ﬁt and we accept the null hypothesis. That is, for small Q2 values, we accept H0 .

Example 7.6.2 A TV station broadcasts a series of programs on the ill effects of smoking marijuana. After the series, the station wants to know whether people have changed their opinion about legalizing marijuana. Given in the following tables are the data based on a survey of 500 randomly chosen people:

7.6 Chi-Square Tests for Count Data 391

Before the Series Was Shown For legalization Decriminalization Existing law No opinion (ﬁne or imprisonment) 7% 18% 65% 10% After the Series Was Shown For legalization Decriminalization Existing law No opinion (ﬁne or imprisonment) 39% 9% 36% 16% Here, n = 4, and we wish to test H0 : p1 = 0.07; p2 = 0.18; p3 = 0.65; p4 = 0.1 versus Ha : At least one of the probabilities is different from the hypothesized value. The test is always an upper tail test. Test this hypothesis using α = 0.01.

Solution We have E1 = (500)(0.07) = 35; E2 = 90; E3 = 325; E4 = 50. The observed frequencies are O1 = (500)(0.39) = 195; O2 = 45; O3 = 180; O4 = 80. The test statistic is

Q2 =

4 (Oi − Ei )2 i=1

% =

Ei

(45 − 90)2 (180 − 325)2 (80 − 50)2 (195 − 35)2 + + + 35 90 325 50

&

= 836.62. 2 From the χ2 -table, χ0.01,3 = 11.3449. Because the test statistic Q2 = 836.62 > 11.3449, we reject H0 at α = 0.01. Hence, the data suggest that people have changed their opinion after the series on the ill effects of smoking marijuana was shown.

392 CHAPTER 7 Hypothesis Testing

Example 7.6.3 A die is rolled 60 times and the face values are recorded. The results are as follows. Up face Frequency

1 8

2 11

3 5

4 12

5 15

6 9

Is the die balanced? Test using α = 0.05.

Solution If the die is balanced, we must have p1 = p2 = . . . = p6 =

1 6

where pi = P(face value on the die is i), i = 1, 2, . . . , 6. This has the discrete uniform distribution. Hence, H0 : p1 = p2 = . . . = p6 =

1 6

versus Ha : At least one of the probabilities is different from the hypothesized value of 1/6 E1 = n1 p1 = (60)(1/6) = 10, . . . , E6 = 10. We summarize the calculations in the following table: Face value Frequency, Oi Expected value, Ei

1 8 10

2 11 10

3 5 10

4 12 10

5 15 10

6 9 10

The test statistic value is given by Q2 =

6 (Oi − Ei )2 i=1

Ei

= 6.

2 From the chi-square table with 5 d.f., χ0.05,5 = 11.070.

Because the value of the test statistic does not fall in the rejection region, we do not reject H0 . Therefore, we conclude that the die is balanced.

7.6.2 Contingency Table: Test for Independence One of the uses of the χ2 -statistic is in contingency (dependence) testing where n randomly selected items are classiﬁed according to two different criteria, such as when data are classiﬁed on the basis of two factors (row factor and column factor) where the row factor has r levels and the column factor has c levels. The obtained data are displayed as shown in the following table, where nij represents

7.6 Chi-Square Tests for Count Data 393

the number of data values under row i and column j. Our interest here is to test for independence of two methods of classiﬁcation of observed events. For example, we might classify a sample of students by sex and by their grade on a statistics course in order to test the hypothesis that the grades are dependent on sex. More generally the problem is to investigate a dependency (or contingency) between two classiﬁcation criteria. Levels of column factor 1 2 … c n11 n12 n1c n21 n21 n2c

Row total 1 n1 2 n2 . . r nr1 nr2 anrc nr Column total n.1 n.2 n.c N c c r r where N = n.j = ni. = nij is the grand total. Row levels

j=1

i=1

i=1 j=1

We wish to test the hypothesis that the two factors are independent. We summarize the procedure in the following table for testing that the factors represented by the rows are independent with that represented by the columns.

TESTING FOR THE INDEPENDENCE OF TWO FACTORS To test H0 : The factors are independent versus Ha : The factors are dependent the test statistic is, Q2 =

r c (Oij − Eij )2 Eij i=1 j=1

where Oij = nij and Eij =

ni nj N

.

Then under the null hypothesis the test statistic Q 2 has an approximate chi-square distribution with (r − 1)(c − 1) degrees of freedom. 2 Hence, the rejection region is Q 2 > χα,(r −1)(c−1) . Assumption: Eij ≥ 5.

394 CHAPTER 7 Hypothesis Testing

Example 7.6.4 The following table gives a classiﬁcation according to religious afﬁliation and marital status for 500 randomly selected individuals.

Marital status

Single With spouse Total

A 39 172 211

Religious afﬁliation B C D None 19 12 28 18 61 44 70 37 80 56 98 55

Total 116 384 500

For α = 0.01, test the null hypothesis that marital status and religious afﬁliation are independent.

Solution We need to test the hypothesis H0 : Marital status and religious affiliation are independent versus Ha : Marital status and religious affiliation are dependent. Here, c = 5, and r = 2. For α = 0.01, and for (c − 1)(r − 1) = 4 degrees of freedom, we have 2 χ0.01,4 = 13.2767

Hence, the rejection region is Q2 > 13.2767. ni nj . Thus, We have Eij = N E11 =

(116)(211) (116)(80) = 48.952; E12 = = 18.5; 500 500

E13 =

(116)(56) (116)(98) = 12.992, E14 = = 22.736; 500 500

E15 =

(116)(55) (384)(211) = 12.76, E21 = = 162.05; 500 500

E22 =

(384)(80) (384)(56) = 61.44; E23 = = 43.008; 500 500

and E24 =

(384)(98) = 75.264; 500

The value of the test statistic is Q2 =

r c (Oij −Eij )2 Eij i=1 j=1

E25 =

(384)(55) = 42.24. 500

7.6 Chi-Square Tests for Count Data 395 % =

(19 − 18.5)2 (12 − 12.992)2 (28 − 22.736)2 (39 − 48.952)2 + + + 48.952 18.5 12.992 22.736

&

+

(172 − 162.05)2 (61 − 61.44)2 (44 − 43.08)2 (18 − 12.76)2 + + + 12.76 162.05 61.44 43.08

+

(37 − 42.24)2 (70 − 75.264)2 + 75.264 42.24

= 7.1351. Because the observed value of Q2 does not fall in the rejection region, we do not reject the null hypothesis at α = 0.01. Therefore, based on the observed data, the marital status and religious affiliation are independent.

7.6.3 Testing to Identify the Probability Distribution: Goodness-of-Fit Chi-Square Test Another application of the chi-square statistic is using it for goodness-of-ﬁt tests in a different context. In hypothesis testing problems we often assume that the form of the population distribution is known. For example, in a χ2 -test for variance, we assume that the population is normal. The goodness-of-ﬁt tests examine the validity of such an assumption if we have a large enough sample. We now describe the goodness-of-ﬁt test procedure for such applications. GOODNESS-OF-FIT TEST PROCEDURES FOR PROBABILITY DISTRIBUTIONS Let X1 , . . . ,Xn be a sample from a population with cdf F (x), which may depend on the set of unknown parameters θ. We wish to test H0 : F (x) = F0 (x), where F0 (x ) is completely speciﬁed. 1. Divide the range of values of the random variables X1 into K nonoverlapping intervals I1 , I2 , . . . , IK . Let Oj be the number of sample values that fall in the interval Ij (j = 1, 2, . . . , K ). 2. Assuming the distribution of X to be F0 (x), ﬁnd P(X ∈ Ij ). Let P(X ∈ Ij ) = πi . Let ej = nπj be the expected frequency. 3. Compute the test statistic Q 2 given by Q2 =

K (Oi − Ei )2 . Ei i=1

The test statistic Q 2 has an approximate χ2 -distribution with (K − 1) degrees of freedom. 2 4. Reject the H0 if Q 2 ≥ χα, (K −1) . 5. Assumptions: ej ≥ 5, j = 1, 2, . . . , K .

If the null hypothesis does not specify F0 (x) completely, that is, if F0 (x) contains some unknown parameters θ1 , θ2 , . . . , θp , we estimate these parameters by the method of maximum likelihood. Using

396 CHAPTER 7 Hypothesis Testing

these estimated values we specify F0 (x) completely. Denote the estimated F0 (x) by Fˆ 0 (x). Let 2 3 πˆ i = P X ∈ Ii |Fˆ 0 (x)

and

ˆ i = nπˆ i . E

The test statistic is Q2 =

K (Oi − eˆ i )2 . eˆ i i=1

The statistic Q2 has an approximate chi-square distribution with (K − 1 − p) degrees of freedom. We 2 . reject H0 if Q2 ≥ χa,(K−1−p) We now illustrate the method of goodness-of-ﬁt with an example.

Example 7.6.5 The grades of students in a class of 200 are given in the following table. Test the hypothesis that the grades are normally distributed with a mean of 75 and a standard deviation of 8. Use α = 0.05. Range Number of students

0–59 12

60–69 36

70–79 90

80–89 44

90–100 18

Solution We have O1 = 12, O2 = 36, O3 = 90, O4 = 44, O5 = 18. We now compute πi (i = 1, 2, . . . , 5), using the continuity correction factor, π1 = P{X ≤ 59.5|H0 } = P{z ≤ 59.5−75 } = 0.0262, 8 π2 = 0.2189, π3 = 0.4722, π4 = 0.2476, π5 = 0.0351, and E1 = 5.24, E2 = 43.78, E3 = 94.44, E4 = 49.52, E5 = 7.02. The test statistic results in Q2 =

n (Oi − ei )2 i=1

=

ei

(36 − 43.78)2 (90 − 94.44)2 (44 − 49.52)2 (18 − 7.02)2 (12 − 5.74)2 + + + + 5.74 43.78 94.44 49.52 7.02

= 26.22.

7.6 Chi-Square Tests for Count Data 397

2 Q2 has a chi-square distribution with (5 − 1) = 4 degrees of freedom. The critical value is χ0.05,4 = 7.11. Hence, the rejection region is Q2 > 7.11. Because the observed value of Q2 = 26.22 > 7.11, we reject H0 at α = 0.05. Thus, we conclude that the population is not normal.

EXERCISES 7.6 7.6.1.

The following table gives the opinion on collective bargaining by a random sample of 200 employees of a school system, belonging to a teachers’ union. Opinion on Collective Bargaining by Teachers’ Union For Against Undecided Total Staff 30 15 15 60 Faculty 50 10 40 100 Administration 10 25 5 40 Column totals 90 50 60 200

Test the hypotheses H0 : Opinion on collective bargaining is independent of employee classiﬁcation versus Ha : Opinion on collective bargaining is dependent on employee classiﬁcation using α = 0.05. 7.6.2.

A random sample was taken of 300 undergraduate students from a university. The students in the sample were classiﬁed according to their gender and according to the choice of their major. The result is given in the following table.

Gender Male Female Total

Arts and sciences 75 45 120

College Engineering 40 12 52

Business 24 15 39

Other 66 23 89

Total 205 95 300

Test the hypothesis that the choice of the major by undergraduate students in this university is independent of their gender. Use α = 0.01. 7.6.3.

The speeds of vehicles (in mph) passing through a section of Highway 75 are recorded for a random sample of 150 vehicles and are given below. Test the hypothesis that the speeds are normally distributed with a mean of 70 and a standard deviation of 4. Use a = 0.01. Range Number

7.6.4.

40–55 12

56–65 14

66–75 78

76–85 40

> 85 6

Based on the sample data of 50 days contained in the following table, test the hypothesis that the daily mean temperatures in the city are normally distributed with mean 77 and variance 6. Use α = 0.05.

398 CHAPTER 7 Hypothesis Testing

Temperature Number of days

7.6.5.

46–55 4

56–65 6

66–75 13

76–85 23

86–95 4

A presidential candidate advertises on TV by comparing his positions on some important issues with those of his opponent. After a series of advertisements, a pollster wants to know whether people have changed their opinion about the candidate. The following are the data based on a survey of 950 randomly chosen people: Before the Advertisement Was Shown Support the Oppose the Need to know more Undecided candidate candidate about the candidate 40% 20% 5% 35% After the Advertisement Was Shown Support the Oppose the Need to know more Undecided candidate candidate about the candidate 45% 25% 2% 28%

Let pi , i = 1, 2, 3, 4, represent the respective true proportions. Test H0 : p1 = 0.35; p2 = 0.20; p3 = 0.15; p4 = 0.3

versus Ha : At least one of the probabilities is different from the hypothesized value. Test this hypothesis using α = 0.05. 7.6.6.

A survey of footwear preferences of a random sample of 100 undergraduate students (50 females and 50 males) from a large university resulted in the following data. Boots Female Male

12 10

Leather shoes 9 12

Sneakers

Sandals

Other

12 17

10 7

7 4

(a) Let pi , i = 1, 2, 3, 4, 5, represent the respective true proportions of students with a particular footwear preference, and let H0 : p1 = 0.20; p2 = 0.20; p3 = 0.30; p4 = 0.20; p5 = 0.10

versus Ha : At least one of the probabilities is different from the hypothesized value. Test this hypothesis using α = 0.05. (b) Test the hypothesis that the choice of footwear by undergraduate students in this university is independent of their gender, using α = 0.05.

7.8 Computer Examples 399

7.7 CHAPTER SUMMARY In this chapter, we have learned various aspects of hypothesis testing. First, we dealt with hypothesis testing for one sample where we used test procedures for testing hypotheses about true mean, true variance, and true proportion. Then we discussed the comparison of two populations through their true means, true variances, and true proportions. We also introduced the Neyman–Pearson lemma and discussed likelihood ratio tests and chi-square tests for categorical data. We now list some of the key deﬁnitions in this chapter. ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Statistical hypotheses Tests of hypotheses, tests of signiﬁcance, or rules of decision Simple hypothesis Composite hypothesis Type I error Type II error The level of signiﬁcance The p-value or attained signiﬁcance level The Smith–Satterthwaite procedure Power of the test Most powerful test Likelihood ratio

In this chapter, we also learned the following important concepts and procedures: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

General method for hypothesis testing Steps to calculate β Steps to ﬁnd the p-value Steps in any hypothesis testing problem Summary of hypothesis tests for μ Summary of large sample hypothesis tests for p Summary of hypothesis tests for the variance σ 2 Summary of hypothesis tests for μ1 − μ2 for large samples (n1 & n2 ≥ 30) Summary of hypothesis tests for p1 − p2 for large samples Testing for the equality of variances Summary of testing for a matched pairs experiment Procedure for applying the Neyman–Pearson lemma Procedure for the likelihood ratio test Testing the parameters of a multinomial distribution (summary) Testing the independence of two factors Goodness-of-ﬁt test procedures for probability distributions

7.8 COMPUTER EXAMPLES In the following examples, if the value of α is not speciﬁed, we will always take it as 0.05.

400 CHAPTER 7 Hypothesis Testing

7.8.1 Minitab Examples Example 7.8.1 (t-Test): Consider the data 66

74

79

80

69

77

78

65

79

81

Using Minitab, test H0 : μ = 75 vs. H1 : μ > 75.

Solution Enter the data in C1. Then Stat > Basic Statistics > 1-sample t. . . > In Variables: enter C1 > choose Test Mean > enter 75 > in Alternative: choose greater than and click OK

We obtain the following output. T-Test of the Mean Test of mu = 75.00 vs mu > 75.00 Variable C1

N 10

Mean 74.80

StDev 6.00

SE Mean 1.90

T −0.11

P 0.54

Example 7.8.2 For the following data: Sample 1: 16 18 21 13 19 16 18 15 20 19 14 21 14 Sample 2: 14 15 10 13 11 7 12 11 12 15 14 Test H0 : μ1 = μ2 vs. H1 : μ1 < μ2 . Use α = 0.02.

Solution Enter sample 1 data in C1 and sample 2 data in C2. Then Stat > Basic Statistics > 2-sample t. . . > Choose Samples in different columns > in Alternative: choose less than > in Confidence level: enter 98 > click Assumed equal variances and click OK

We obtain the following output. Two Sample T-test and Confidence Interval Two sample T for C1 vs C2

7.8 Computer Examples 401

N 13 11

C1 C2

Mean 17.23 12.18

StDev 2.74 2.40

SE Mean 0.76 0.72

98% CI for mu C1 − mu C2: (2.38, 7.71) T-Test mu C1 = mu C2 (vs 1-sample z. . . > in Variables: Type C1 > choose Test Mean and enter 12 > choose not equal in Alternative, and Type 4.7 for sigma > Click OK We obtain the following output. Z-Test Test of mu = 12.000 vs mu not = 12.000 The assumed sigma = 4.70 Variable C1

N 49

Mean 12.124

StDev 4.700

SE Mean 0.671

Z 0.19

P 0.85

Here the test statistic is 0.19 and the p-value is 0.85, which is larger than 0.05. Hence, we cannot reject the null hypothesis.

402 CHAPTER 7 Hypothesis Testing

Example 7.8.4 (Contingency Table): Consider the following data with ﬁve levels and two factors. Test for dependence of the factors. Factors 1 39 172

1 2

2 19 61

Levels 3 4 12 28 44 70

5 18 37

Solution In C1 enter the data in column 1 (39 and 172), and continue to C5. Then Stat > Tables > Chi-Square-Test. . . > in Columns containing the table: Type C1 C2 C3 C4 C5 > click OK

We will obtain the following output. Chi-Square Test Expected counts are printed below observed counts C1 C2 C3 C4 C5 Total 1 39 19 12 28 18 116 48.95 18.56 12.99 22.74 12.76 2 Total

172 162.05 211

61 61.44 80

44 43.01 56

70 75.26 98

37 42.24 55

384 500

Chi-Sq = 2.023 + 0.010 + 0.076 + 1.219 + 2.152 + 0.611 + 0.003 + 0.023 + 0.368 + 0.650 = 7.135 DF = 4, p-value = 0.129

Example 7.8.5 (Paired t-Test): Consider the data of Example 7.5.7. Using Minitab, perform a paired t-test.

Solution Enter sample 1 in column C1 and sample 2 in column C2. Then: Stat > Basic Statistics > Paired t. . . > in First Sample: Type C2, and in the Second sample: Type C1 > click options > and click less than (if α is other than 0.05, enter appropriate percentage in Confidence level: and enter appropriate number if it is not zero in Test mean:) > click OK > OK

7.8 Computer Examples 403

We obtain the following output. Paired T-test and Confidence Interval T for C2 − C1 N Mean StDev SE Mean C2 10 171.3 47.1 14.9 C1 10 243.2 40.1 12.7 Difference 10 −71.9 56.2 17.8 95% CI for mean difference: (−112.1, −31.7) T-Test of mean difference = 0 (vs < 0): T-Value = −4.05 p-value = 0.001 Paired

because the p-value 0.001 < 0.05 = α.

7.8.2 SPSS Examples Example 7.8.6 Consider the data 66

74

79

80

69

77

78

65

79

81

Using SPSS, test H0 : μ = 75 vs. H1 : μ > 75.

Solution Use the following procedure: 1. Enter the data in column 1. 2. Click Analyze > Compare Means > One-sample t Test. . . , Move var00001 to Test Variable(s), and change Test Value: 0 to 75. Click OK We obtain the following output. One-Sample Statistics

VAR00001

N 10

Mean 74.8000

Std. Deviation 5.99630

Std. Error Mean 1.89620

One-Sample Test Test Value = 75 95% Confidence Interval of the Sig. Mean Difference t df (2-tailed) Difference Lower Upper VAR00001 −.105 9 .918 −.2000 −4.4895 4.0895 For the one sample t-test H0 : μ = 75 vs. H1 : μ > 75, the t-statistic is −0.105 with 9 degrees of freedom. The p-value is 0.46 > 0.02. Hence, we will not reject the null hypothesis.

404 CHAPTER 7 Hypothesis Testing

If we want the computer to calculate the p-value in the previous example, use the following procedure. 1. Enter the test statistic (−0.105) in the data editor using ‘teststat’. 2. Click Transform > compute. . . 3. Type ‘p-value’ in the box called Tarobtain value. In the box called Functions: scroll and click on CDF.T(q,df) and move to Numeric Expressions. 4. The CDF(q,df) will appear as CDF(?,?) in the Numeric Expressions box. Replace teststat for q and 9 for df (the degree of freedom in this example is 9). Click OK

We obtain the p-value as 0.46.

Example 7.8.7 For the following data Sample 1: Sample 2:

16 14

18 15

21 10

13 13

19 11

16 7

18 12

15 11

20 12

19 15

14 14

21

14

Test H0 : μ1 = μ2 vs. H1 : μ1 < μ2 . Use α = 0.02.

Solution In column 1, under the title ‘‘group’’ enter 1s to identify the sample 1 data and 2s to identify sample 2 data. In column C2, under the title ‘‘data’’ enter the data corresponding to samples 1 and 2. Then: Analyze > Compare Means > Independent Samples t-test. . . > bring Data to Test Variable(s): and group to Grouping Variable:, click Define Groups. . . , and enter 1 for sample 1, 2 for sample 2 > click continue > click Options. . . . Enter 98 in Confidence interval: > click continue > OK We obtain the following output. Group Statistics GROUP N DATA 1.00 13 2.00 11

Mean 17.2308 12.1818

Independent Samples Test Levene’s Test t-test for for Equality Equality of Variances of Means F Sig. t

DATA Equal variances assumed Equal variances not assumed

.975

.334

df

4.753

22

4.808

21.963

Std. Deviation 2.74329 2.40076

Std. Error Mean .76085 .72386

Sig. Mean Std. Error 98% Confidence (2-tailed) Difference Difference Interval of the Difference Lower Upper .000 5.0490 1.06237 2.38419 7.71372 .000

5.0490

1.05017

2.41443

7.68347

7.8 Computer Examples 405

Looking at the statistical significance values, which are greater than 0.05, we do not reject the null hypothesis.

Example 7.8.8 (Paired t-Test) For the data of Example 7.5.7, use SPSS to test whether the data provide sufﬁcient evidence for the claim that the new program reduces blood glucose level in diabetic patients. Use α = 0.05.

Solution Enter after data in column C1 and before data in column C2. Then:

Analyze > Compare Means > Paired-Sample T-Test > bring after and before to Paired Variables: so that it will look after-before > click OK We obtain the following output. Paired Samples Statistics Mean N Std. Deviation Std. Error Mean Pair 1 AFTER 171.3000 10 47.11228 14.89821 BEFORE 243.2000 10 40.12979 12.69015 Paired Samples Correlations N Correlation Sig. Pair 1 AFTER & BEFORE 10 .179 .621 Paired Samples Test Paired Differences Mean

t

df Sig. (2-tailed)

Std. Deviation

Std. Error 95% Confidence Mean Interval of the Difference Upper

Pair 1 AFTER -- −71.9000 BEFORE

17.75791 56.15544

Lower −112.0712

−31.7288 −4.049 9

.003

Because the significance level for the test is 0.003, which is less than α = 0.05, we reject the null hypothesis.

7.8.3 SAS Examples To conduct a hypothesis test using SAS, we could use proc ttest, or proc means with option of computing the t-value and corresponding probability. However, to use this, we need a hypothesis of the form H0 : μ = 0. For testing nonzero values, H0 : μ = μ0 , we must create a new variable

406 CHAPTER 7 Hypothesis Testing

by subtracting μ0 from each observation, and then use the test procedure for this new variable. The following example illustrates this concept.

Example 7.8.9 (t-Test): The following radar measurements of speed (in miles per hour) are obtained for 10 vehicles traveling on a stretch of interstate highway. 66

74

79

80

69

77

78

65

79

81

Do the data provide sufﬁcient evidence to indicate that the mean speed at which people travel on this stretch of highway is at least 75 mph? Test using α = 0.01. Use an SAS procedure to do the analysis.

Solution In the SAS editor, type in the following commands. data speed; title ’Test on highway speed’; input X @@; Y=X-75; datalines; 66 74 79 80 69 77 78 65 79 81 ; PROC TTEST data=speed; run; We obtain the following output. Test on highway speed The TTEST Procedure Statistics Lower CL

Upper CL

Variable N

Mean

Mean

Mean

X Y

70.511 −4.489

74.8 −0.2

79.089 4.0895

10 10

Std Dev 4.1245 4.1245 Variable X Y

Lower CL Std Dev 5.9963 5.9963 T-Tests DF 9 9

Upper CL Std Dev 10.947 10.947 t Value 39.45 −0.11

Std Err 1.8962

Pr > |t| |t| −4.05 0.0029 Because the p-value 0.0029 is less than α = 0.05, we reject the null hypothesis.

PROJECTS FOR CHAPTER 7 7A. Testing on Computer-Generated Samples (a) Small sample test: Generate a sample of size 20 from a normal population with μ = 10, and σ 2 = 4. (i) Perform a t-test for the test H0 : μ = 10 versus Ha : μ = 10 at level α = 0.05. (ii) Perform the test H0 : σ 2 = 4 versus Ha : σ 2 = 4 at level α = 0.05. Repeat the procedure 10 times, and comment on the results. (b) Large sample test:

Projects for Chapter 7 409

Generate a sample of size 50 from a normal population with μ = 10, and σ 2 = 4. Perform a z-test for the test H0 : μ = 10 versus Ha : μ = 10 at level α = 0.05. Repeat the procedure 10 times and comment on the results.

7B. Conducting a Statistical Test with Confidence Interval Let θ be any population parameter. Consider the three tests of hypotheses H0 : θ = θ0 vs. Ha : θ > θ0

(1)

H0 : θ = θ0 vs. Ha : θ < θ0

(2)

H0 : θ = θ0 vs. Ha : θ = θ0

(3)

The following procedure can be exploited to test a statistical hypothesis utilizing the conﬁdence intervals.

Procedure to Use Confidence Interval for Hypothesis Testing Let θ be any population parameter. (a) For test (1), that is, H0 : θ = θ0 vs. Ha : θ > θ0

choose a value for α. From a random sample, compute a conﬁdence interval for θ using a conﬁdence coefﬁcient equal to 1 − 2α. Let L be the lower end point of this conﬁdence interval. Reject H0 if θ0 < L. That is, we will reject the null hypothesis if the conﬁdence interval is completely to the right of θ0 . (b) For test (2), that is, H0 : θ = θ0 vs. Ha : θ < θ0

choose a value for α. From a random sample, compute a conﬁdence interval for θ using a conﬁdence coefﬁcient equal to 1 − 2α. Let U be the upper end point of this conﬁdence interval. Reject H0 if U < θ0 . That is, we will reject the null hypothesis if the conﬁdence interval is completely to the left of θ0 . (c) For test (3), that is, H0 : θ = θ0 vs. Ha : θ = θ0

410 CHAPTER 7 Hypothesis Testing

choose a value for α. From a random sample, compute a conﬁdence interval for θ using a conﬁdence coefﬁcient equal to 1 − α. Let L be the lower end point and U be the upper end point of this conﬁdence interval. Reject H0 if θ0 < L or U < θ0 . That is, we will reject the null hypothesis if the conﬁdence interval does not contain θ0 . (i) For any large data set, conduct all three of these hypothesis tests using a conﬁdence interval for the population mean. (ii) For any small data set, conduct all three of these hypothesis tests using a conﬁdence interval for the population mean.

Chapter

8

Linear Regression Models Objective: In this chapter we will study linear relationships in sample data and use the method of least squares to estimate the necessary parameters. 8.1 Introduction 412 8.2 The Simple Linear Regression Model 413 8.3 Inferences on the Least-Squares Estimators 428 8.4 Predicting a Particular Value of Y 437 8.5 Correlation Analysis 440 8.6 Matrix Notation for Linear Regression 445 8.7 Regression Diagnostics 451 8.8 Chapter Summary 454 8.9 Computer Examples 455 Projects for Chapter 8 461

Sir Francis Galton (Source: http://en.wikipedia.org/wiki/Francis_Galton)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

411

412 CHAPTER 8 Linear Regression Models

English scientist Sir Francis Galton (1822–1911), a cousin of Charles Darwin, made signiﬁcant contributions to both genetics and psychology. He is the inventor of regression and a pioneer in applying statistics to biology. One of the data sets that he considered consisted of the heights of fathers and ﬁrst sons. He was interested in predicting the height of son based on the height of father. Looking at the scatterplots of these heights, Galton saw that the trend was linear and increasing. After ﬁtting a line to these data (using the techniques described in this chapter), he observed that for fathers whose heights were taller than the average, the regression line predicted that taller fathers tended to have shorter sons and shorter fathers tended to have taller sons. There is a regression toward the mean. That is how the method of this chapter got its name: regression.

8.1 INTRODUCTION In earlier chapters, we were primarily concerned about inferences on population parameters. In this chapter, we examine the relationship between one or more variables and create a model that can be used for predictive purposes. For example, consider the question “Is there statistical evidence to conclude that the countries with the highest average blood-cholesterol levels have the greatest incidence of heart disease?” It is important to answer this if we want to make appropriate lifestyle and medical choices. We will study the relationship between variables using regression analysis. Our aim is to create a model and study inferential procedures when one dependent and several independent variables are present. We denote by Y the random variable to be predicted, also called the dependent variable (or response variable) and by xi the independent (or predictor) variables used to model (or predict) Y . For example, let (x, y) denote the height and weight of an adult male. Our interest may be to ﬁnd the relationship between height and weight from a sample measurements of n individuals. The process of ﬁnding a mathematical equation that best ﬁts the noisy data is known as regression analysis. In his book Natural Inheritance, Sir Francis Galton introduced the word regression in 1889 to describe certain genetic relationships. The technique of regression is one of the most popular statistical tools to study the dependence of one variable with respect to another. There are different forms of regression: simple linear, nonlinear, multiple, and others. The primary use of a regression model is prediction. When using a model to predict Y for a particular set of values of x1 , . . . , xk , one may want to know how large the error of prediction might be. Regression analysis, in general after collecting the sample data, involves the following steps.

PROCEDURE FOR REGRESSION MODELING 1. Hypothesize the form of the model as Y = f (x1 , . . . , xk ; β0 , β1 , . . . , βk ) + ε. Here ε represents the random error term. We assume that E(ε) = 0 but Var (ε) = σ 2 is unknown. From this we can obtain E(Y ) = f (x1 , . . . , xk ; β0 , β1 , . . . , βk ). 2. Use the sample data to estimate unknown parameters in the model. 3. Check for goodness of ﬁt of the proposed model. 4. Use the model for prediction.

8.2 The Simple Linear Regression Model 413

The function f (x1 , . . . , xk ; β0 , β1 , . . . , βk )(k ≥ 1) contains the independent or predictor variables x1 , . . . , xn (assumed to be nonrandom) and unknown parameters or weights β0 , β1 , . . . , βk and ε representing the random or error variable. We now proceed to introduce the simplest form of a regression model, called simple linear regression.

8.2 THE SIMPLE LINEAR REGRESSION MODEL Consider a random sample of n observations of the form (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ), where X is the independent variable and Y is the dependent variable, both being scalars. A preliminary descriptive technique for determining the form of relationship between X and Y is the scatter diagram. A scatter diagram is drawn by plotting the sample observations in Cartesian coordinates. The pattern of the points gives an indication of a linear or nonlinear relationship between the variables. In Figure 8.1a, the relationship between x and y is fairly linear, whereas the relationship is somewhat like a parabola in Figure 8.1b, and in Figure 8.1c there is no obvious relationship between the variables. Once the scatter diagram reveals a linear relationship, the problem then is to ﬁnd the linear model that best ﬁts the given data. To this end, we will ﬁrst give a general deﬁnition of a linear statistical model, called a multiple linear regression model.

y

y

x

x (a) Linear relationship

(b) Quadratic relationship

y

x (c) No relationship ■ FIGURE 8.1 Scatter diagram.

414 CHAPTER 8 Linear Regression Models

Deﬁnition 8.2.1 A multiple linear regression model relating a random response Y to a set of predictor variables x1 , . . . , xk is an equation of the form Y = β0 + β1 x1 + β2 x2 + · · · + βk xk + ε

where β0 , . . . , βk are unknown parameters, x1 , . . . , xk are the independent nonrandom variables, and ε is a random variable representing an error term. We assume that E(ε) = 0, or equivalently, E(Y ) = β0 + β1 x1 + β2 x2 + · · · + βk xk .

To understand the basic concepts of regression analysis we shall consider a single dependent variable Y and a single independent nonrandom variable x. We assume that there are no measurement errors in xi . The possible measurement errors in y and the uncertainties in the assumed model are expressed through the random error ε. Our inability to provide an exact model for a natural phenomenon is expressed through the random term ε, which will have a speciﬁed probability distribution (such as a normal) with mean zero. Thus, one can think of Y as having a deterministic component, E(Y ), and a random component, ε. If we take k = 1 in the multiple linear regression model, we have a simple linear regression model. Deﬁnition 8.2.2 If Y = β0 + β1 x + ε, this is called a simple linear regression model. Here, β0 is the y-intercept of the line and β1 is the slope of the line. The term ε is the error component. This basic linear model assumes the existence of a linear relationship between the variables x and y that is disturbed by a random error ε. The known data points are the pairs (x1 , y2 ), (x2 , y2 ), . . . , (xn , yn ); the problem of simple linear regression is to ﬁt a straight line optimal in some sense to the set of data, as shown in Figure 8.2.

25 20 15 10 5 0 ⫺5 ⫺10 ⫺5

0

5

10

■ FIGURE 8.2 Scatterplot and least-squares regression line.

15

8.2 The Simple Linear Regression Model 415

Now, the problem becomes one of ﬁnding estimators for β0 and β1 . Once we obtain the “good” estimators βˆ 0 and βˆ 1 , we can ﬁt a line to the data given by the prediction equation Yˆ = βˆ 0 + βˆ 1 x. The question then becomes whether this predicted line gives the “best” (in some sense) description of the data. We now describe the most widely used technique, called the method of least squares, to obtain the estimators or weights of the parameters.

8.2.1 The Method of Least Squares As stated (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) are the n observed data points, with corresponding errors εi , i = 1, . . . , n. That is, Yi = β0 + β1 xi + εi ,

i = 1, 2, . . . , n.

We assume that the errors εi , i = 1, . . . , n are independent and identically distributed with E(εi ) = 0, i = 1, . . . , n, and Var(εi ) = σ 2 , i = 1, . . . , n. One of the ways to decide on how well a straight line ﬁts the set of data is to determine the extent to which the data points deviate from the line. The straight line model for the response Y for a given x is Y = β0 + β1 x + ε.

Because we assumed that E(ε) = 0, the expected value of Y is given by E(Y ) = β0 + β1 x.

The estimator of the E(Y ), denoted by Yˆ , can be obtained by using the estimators βˆ 0 and βˆ 1 of the parameters β0 and β1 , respectively. Then, the ﬁtted regression line we are looking for is given by Yˆ = βˆ 0 + βˆ 1 x.

For observed values (xi , yi ), we obtain the estimated value of yi as yˆ i = βˆ 0 + βˆ 1 xi .

The deviation of observed yi from its predicted value yˆ i , called the ith residual, is deﬁned by ei = yi − yˆ i = yi − βˆ 0 + βˆ 1 xi .

The residuals, or errors ei , are the vertical distances between observed and predicted values of yi s (Figure 8.3). y .

ei

. . .

. . .

x ■ FIGURE 8.3 Illustration of ei .

416 CHAPTER 8 Linear Regression Models

Deﬁnition 8.2.3 The sum of squares for errors (SSE) or sum of squares of the residuals for all of the n data points is SSE =

n

e21 =

i=1

n

2 yi − βˆ 0 + βˆ 1 xi

i=1

The least-squares approach to estimation is to ﬁnd βˆ 0 and βˆ 1 that minimize the sum of squared residuals, SSE. Thus, in the method of least squares, we choose β0 and β1 so that SSE is a minimum. The quantities βˆ 0 and βˆ 1 that make the SSE a minimum are called the least-squares estimates of the parameters β0 and β1 , and the corresponding line yˆ = βˆ 0 + βˆ 1 x is called the least-squares line. Deﬁnition 8.2.4 The least-squares line yˆ = βˆ 0 + βˆ 1 x is one that satisﬁes the following property: SSE =

n

yi − yˆ i

2

i=1

is a minimum for any other straight line model with SE =

n

yi − yˆ i = 0

i=1

Thus, the least-squares line is a line of the form y = b0 + b1 x for which the error sum of squares n 2 i=1 (yi − b0 − b1 x) is a minimum. The minimum is taken over all values of b0 and b1 , and (x1 , y1 ), (x2 , y2 ), . . . , (xn , yn ) are observed data pairs. The problem of ﬁtting a least-squares line now reduces to ﬁnding the quantities βˆ 0 and βˆ 1 that minimize the error sum of squares.

8.2.2 Derivation of βˆ 0 and βˆ 1

Now we derive expressions for βˆ 0 and βˆ 1 . If SSE attains a minimum, then the partial derivatives of SSE with respect to β0 and β1 are zeros. That is, n

∂ ∂SSE = ∂β0

i=1

∂β0

=− =2

4

[yi − (β0 + β1 xi )]2

n

2 [yi − (β0 + β1 xi )]

i=1n

yi − nβ0 − β1

i=1

i=1

and ∂ ∂SSE = ∂β1

n

n

4 [yi − (β0 + β1 xi )]2

i=1

∂β1

(8.1)

xi

=0

8.2 The Simple Linear Regression Model 417

=−

n

2 [yi − (β0 + β1 xi )]xi

(8.2)

i=1

= −2

n

xi yi − β0

i=1

n

n

xi − β1

i=1

x12

= 0.

i=1

Equations (8.1) and (8.2) are called the least squares equations for estimating the parameters of a line. From (8.1) and (8.2) we obtain a set of linear equations called the normal equations, n

yi = nβ0 + β1

i=1

n

(8.3)

xi

i=1

and n

xi yi = β0

i=1

n

xi + β1

i=1

n

xi2 .

(8.4)

i=1

Solving for β0 and β1 from Equations (8.3) and (8.4), we obtain n

(xi − x) (yi − y)

i=1 βˆ 1 = n

n =

(xi − x)2

i=1

n

xi yi −

i=1 n n x12 − i=1

n

xi

i=1

n

n

yi

i=1 2

n

n

=

xi yi −

i=1

xi

i=1

i=1

x12 −

n

yi

i=1

n

n

xi

i=1 n

2

(8.5)

xi

i=1

n

and βˆ 0 = y − βˆ 1 x.

(8.6)

To simplify the formula for βˆ 1 , set Sxx =

n i=1

xi2 −

n

2

xi

i=1

n

, Sxy =

n i=1

xi yi −

n

xi

i=1

n

yi

i=1

n

we can rewrite (8.5) as Sxy βˆ 1 = . Sxx

It can be shown (by using the second derivatives) that (8.5) and (8.6) do indeed minimize SSE. Now we will summarize the procedure for ﬁtting a least-squares line.

418 CHAPTER 8 Linear Regression Models

PROCEDURE FOR FITTING A LEAST-SQUARES LINE 1. Form the n data points (x1 , y1 ),(x2 , y2 ), . . . ,(xn , yn ), and compute the following quantities: n n n n 2 n 2 i=1 xi , i=1 xi , i=1 yi , i=1 yi , and i=1 xi yi . Also compute the sample means, x = (1/n) ni=1 xi and y = (1/n) ni=1 yi . 2. Compute Sxx =

n

x12 −

n

2 xi

i=1

=

n

i=1

n

xi − x

2

i=1

and Sxy =

n i=1

xi yi −

n

xi

i=1

n

yi

i=1

n

=

n

xi − x

yi − y .

i=1

3. Compute βˆ 0 and βˆ 1 by substituting the computed quantities from step 1 into the equations Sxy βˆ 1 = Sxx and βˆ 0 = y − βˆ 1 x . 4. The ﬁtted least-squares line is yˆ = βˆ 0 + βˆ 1 x . 5. For a graphical representation, in the xy -plane, plot all the data points and draw the least-squares line obtained in step 4.

Once we have accomplished the best-ﬁt combination of the two parameters β0 and β1 , any deviation of either parameter away from its optimum value will cause the sum of squares error to increase. Thus, the optimum combination of the pairs (βˆ 0 , βˆ 1 ) forms a global minimum point of the error sum of squares among all possible values of β0 and β1 for the given data set.

8.2 The Simple Linear Regression Model 419

Example 8.2.1 Use the method of least squares to ﬁt a straight line to the accompanying data points. Give the estimates of β0 and β1 . Plot the points and sketch the ﬁtted least-squares line. The observed data values are given in the following table. −1 −5

x y

0 −4

−2 −7

2 2

5 6

6 9

8 13

11 21

12 20

−3 −9

Solution Form a table to compute various terms

xi −1 0 2 −2 5 6 8 11 12 −3

xi = 38

yi −5 −4 2 −7 6 9 13 21 20 −9

Sxx =

n

x12 −

n

Sxy =

n i=1

xi yi −

n

= 408 −

xi

n

2 xi = 408

(38)2 = 263.6 10

yi

i=1

n x = 3.8

xi yi = 709

xi

n

i=1

1 0 4 4 25 36 64 121 144 9

2

i=1

i=1

yi = 46

xi2

xi yi 5 0 4 14 30 54 104 231 240 27

and

= 709 −

(38)(46) = 534.2 10

y = 4.6.

Therefore, Sxy 534.2 = 2.0266 βˆ 1 = = Sxx 263.6

420 CHAPTER 8 Linear Regression Models

25 20 15 10 5 0 ⫺5 ⫺10 ⫺5

0

5

10

15

■ FIGURE 8.4 Simple regression line.

and βˆ 0 = y − βˆ 1 x = 4.6 − (2.0266)(3.8) = −3.1011.

Hence, the least-squares line for these data is yˆ = βˆ 0 + βˆ 1 x = −3.1011 + 2.0266x

and its plot is shown in Figure 8.4. Recall that for the regression line yˆ = βˆ 0 + βˆ 1 x. we have deﬁned SSE to be SSE =

n

yi − yˆ i

2

=

i=1

n

yi − βˆ 0 − βˆ 1 xi

SSE = Syy − βˆ 1 Sxy , where Syy =

n i=1

We know that n

yi − βˆ 0 − βˆ 1 xi

2

i=1

=

n

yi − y + βˆ 1 x − βˆ 1 xi

i=1

.

i=1

We now show that

SSE =

2

2

y12 −

n

2 yi

i=1

n

=

n i=1

(yi − y)2 .

8.2 The Simple Linear Regression Model 421

=

n

(yi − y) − βˆ 1 (xi − x)

2

i=1

=

n

(yi − y)2 + βˆ 12

i=1

n

(xi − x)2 − 2βˆ 1

i=1

n

(xi − x) (yi − y)

i=1

= Syy + βˆ 12 Sxx − 2βˆ 1 Sxy .

Recall that βˆ 1 =

Sxy Sxx .

Substituting for βˆ 1 , we obtain

SSE = Syy − = Syy −

Sxy 2 Sxy Sxx − 2 Sxy Sxx Sxx

Sxy Sxy Sxx

= Syy − βˆ 1 Sxy .

8.2.3 Quality of the Regression Once we obtain the linear model, the question is, How well does this line ﬁt the data? We could make use of the residuals eˆ i = yi − βˆ 0 − βˆ 1 xi

to answer the question and to assess the quality of the ﬁt. If our model is good, then the residual eˆ i should be close to the random error ε with mean zero. Furthermore, the residuals should contain little or no information about the model, and there should be no recognizable pattern. If we plot the residuals versus the independent variables on the x-axis, ideally, the plot should look like a horizontal blur, the residuals showing no relationship to the x-values, as shown by Figure 8.5. Otherwise, these plots reveal a not very good ﬁt of the given data, as shown by Figure 8.6, and we need to improve our model speciﬁcations. Thus, a symmetric trend in the plot of residuals ei versus xi or yˆ i (i = 1, . . . , n) indicates that the assumed regression model is not correct.

e

y

■ FIGURE 8.5 Good fit.

422 CHAPTER 8 Linear Regression Models

e

y

■ FIGURE 8.6 Not a good fit.

Whereas the residual plots give us a visual representation of the quality of ﬁt, a numerical measure of how well the regression explains the data is obtained by calculating the coefﬁcient of determination, also called the R2 of the regression. This is discussed in Project 8B. Regression analysis with any of the standard statistical software packages will contain an output value of the R2 . This value will be between 0 and 1; closer to 1 means a better ﬁt. For example, if the value of R2 is 0.85, the regression captures 85% of the variation in the dependent variable. This is generally considered good regression.

8.2.4 Properties of the Least-Squares Estimators for the Model Y = β0 + β1 x + ε We discussed in Chapter 4 the concept of sampling distribution of sample statistics such as that of X. Similarly, knowledge of the distributional properties of the least-squares estimators βˆ 0 and βˆ 1 is necessary to allow any statistical inferences to be made about them. The following result gives the sampling distribution of the least-squares estimators. Theorem 8.2.1 Let Y = β0 + β1 x + ε be a simple linear regression model with ε ∼ N(0, σ 2 ), and let the errors εi associated with different observations yi (i = 1, . . . , N) be independent. Then (a) βˆ 0 and βˆ 1 have normal distributions. (b) The mean and variance are given by E βˆ 0 = β0 ,

Var βˆ 0 =

1 x2 + n Sxx

σ2,

and E βˆ 1 = β1 ,

σ2 Var βˆ 1 = , Sxx

n 2 1 xi . In particular, the least-squares estimators βˆ 0 and βˆ 1 are unbiased n i=1 i=1 estimators of β0 and β1 , respectively.

where Sxx =

n

xi2 −

8.2 The Simple Linear Regression Model 423

Proof. We know that Sxy βˆ 1 = Sxx =

n 1 (xi − x) Yi − Y Sxx i=1

% n & n 1 (xi − x) Yi − Y (xi − x) = Sxx i=1

i=1

n 1 (xi − x) Yi = Sxx i=1

where the last equality follows from the fact that

n

(xi − x) =

i=1

distributed, the sum

n

xi − nx = 0. Because Yi is normally

i=1

n 1 (xi − x)Yi is also normal. Furthermore, Sxx i=1

E[βˆ 1 ] =

n 1 (xi − x)E[Yi ] Sxx i=1

=

n 1 (xi − x)(β0 + β1 xi ) Sxx i=1

=

n n β1 β0 (xi − x) + (xi − x)xi Sxx Sxx i=1

= β1

1 Sxx

i=1

n

(xi − x)xi

i=1

% n & n 1 2 = β1 x1 − x xi Sxx i=1

⎡ = β1

n ⎢

1 ⎢ ⎢ Sxx ⎣

i=1

i=1

⎛ n ⎞⎤ n x i ⎜ ⎟⎥ ⎜ i=1 ⎟⎥ x12 − xi ⎜ ⎟⎥ ⎝ n ⎠⎦

⎡ ⎢ n ⎢ ⎢ x12 − ⎢ ⎢ ⎣i=1

= β1

1 Sxx

= β1

1 Sxx = β1 . Sxx

i=1

n

2 ⎤ xi

i=1

n

⎥ ⎥ ⎥ ⎥ ⎥ ⎦

424 CHAPTER 8 Linear Regression Models

For the variance we have, Var βˆ 1 = Var

%

n 1 (xi − x) Yi Sxx

&

i=1

n 1 (xi − x)2 Var [Yi ] = 2 Sxx

(since the Yi ’s are independent)

i=1

n 1 (xi − x)2 Var (Yi ) = Var (β0 + β1 + εi ) = Var (εi ) = σ 2 = σ2 2 Sxx i=1

=

σ2 Sxx

.

Note that both Y and βˆ 1 are normal random variables. It can be shown that they are also independent (see Exercise 8.3.3). Because βˆ 0 = y − βˆ 1 x is a linear combination of Y and βˆ 1 , it is also normal. Now, # $ E βˆ 0 = E Y − βˆ 1 x = E Y − xE βˆ 1 & % n n 1 1 (β0 + β1 x) − xβ1 Yi − xβ1 = =E n n i=1

i=1

= β0 + xβ1 − xβ1 = β0 .

The variance of βˆ 0 is given by Var βˆ 0 = Var Y − βˆ 1 x # $ = Var Y + x2 Var βˆ 1 (since Y and βˆ 1 are independent) 1 x2 σ 2 x2 σ2 = σ2. + + = n Sxx n Sxx

If an estimator θˆ is a linear combination of the sample observations and has a variance that is less than or equal to that of any other estimator that is also a linear combination of the sample observations, then θˆ is said to be a best linear unbiased estimator (BLUE) for θ. The following result states that among all unbiased estimators for β0 and β1 which are linear in Yi , the least-square estimators have the smallest variance.

GAUSS–MARKOV THEOREM Theorem 8.2.2 Let Y = β0 + β1 x + ε be the simple regression model such that for each xi ﬁxed, each Yi is an observable random variable and each ε = εi , i = 1, 2, . . . , n is an unobservable random variable. Also, let the random variable εi be such that E[εi ] = 0, Var(εi ) = σ 2 and Cov(εi , εj ) = 0, if i = j. Then the least-squares estimators for β0 and β1 are best linear unbiased estimators.

8.2 The Simple Linear Regression Model 425

It is important to note that even when the error variances are not constant, there still can exist unbiased least-square estimators, but the least-squares estimators do not have minimum variance.

8.2.5 Estimation of Error Variance σ 2 The greater the variance, σ 2 , of the random error ε, the larger will be the errors in the estimation of model parameters β0 and β1 . We can use already-calculated quantities to estimate this variability of errors. It can be shown that (see Exercise 8.2.1(b)) that E(SSE) = (n − 2)σ 2 .

Thus, an unbiased estimator of the error variance, σ 2 , is σˆ 2 = (SSE)/(n − 2). We will denote (SSE)/ (n − 2) by MSE (Mean Square Error).

EXERCISES 8.2 8.2.1.

For a random sample of size n, (a) Show that the error sum of squares can be expressed by SSE = Syy − βˆ 1 Sxy .

(b) Show that E[SSE] = (n − 2)σ 2 . 8.2.2.

The following are midterm and ﬁnal examination test scores for 10 students from a calculus class, where x denotes the midterm score and y denotes the ﬁnal score for each student. x y

68 74

87 79

75 80

91 93

82 88

77 79

86 97

82 95

75 89

79 92

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.3.

The following data give the annual incomes (in thousands of dollars) and amounts (in thousands of dollars) of life insurance policies for eight persons. Annual income 42 58 27 36 70 24 53 37 Life insurance 150 175 25 75 250 50 250 100 (a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph.

8.2.4.

Consider a simple linear model Y = β0 + β1 x + ε, with ε ∼ N(0, σ 2 ). Show that n

−σ 2 cov(βˆ 0 , βˆ 1 ) =

xi

i=1 n n xi2 − i=1

n i=1

2 . xi

426 CHAPTER 8 Linear Regression Models

8.2.5.

(a) Show that the least-squares estimates of β0 and β1 of a line can be expressed as βˆ 0 = y − βˆ 1 x

and

n

(xi − x) (yi − y) i=1 . βˆ 1 = n (xi − x)2 i=1

(b) Using part (a), show that the line ﬁtted by the method of least squares passes through the point (x, y). 8.2.6.

Crickets make their chirping sounds by rapidly sliding one wing over the other. The faster they move their wings, the higher the number of chirping sounds that are produced. Scientists have noticed that crickets move their wings faster in warm temperatures than in cold temperatures (they also do this when they are threatened). Therefore, by listening to the pitch of the chirp of crickets, it is possible to tell the temperature of the air. The following table gives the number of cricket chirps per 13 seconds recorded at 10 different temperatures. Assume that the crickets are not threatened. Temperature Number of chirps

60 20

66 25

70 31

73 33

78 36

80 39

82 42

87 48

90 49

92 52

Calculate the least-squares regression line for these data and discuss its usefulness. 8.2.7.

Consider the regression model y = β1 x + ε

where ε ∼ N(0, σ 2 ). Show that n

xi yi i=1 . βˆ 1 = n 2 xi i=1

8.2.8.

A farmer collected the following data, which show crop yields for various amounts of fertilizer used. Fertilizer (pounds/100 sq. ft) Yield (bushels)

0 6

4 7

8 10

10 13

15 17

18 18

20 22

25 23

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.9.

An economist desires to estimate a line that relates personal disposable income (DI) to consumption expenditures (CE). Both DI and CE are in thousands of dollars. The following gives the data for a random sample of nine households of size four.

8.2 The Simple Linear Regression Model 427

DI CE

25 21

22 20

19 17

36 28

40 34

47 41

28 25

52 45

60 51

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.10.

The following data represent systolic blood pressure readings on 10 randomly selected females between ages 40 and 82. Age (x) Systolic (y)

63 151

70 149

74 164

82 157

60 144

44 130

80 157

71 160

71 121

41 125

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.11.

Itisbelievedthatexposuretosolarradiationincreasesthepathogenesisofmelanoma.Suppose that the following data give sunspot relative number and age-adjusted total incidence (incidence is the number of cases per 100,000 population) for 8 different years in a certain region. Sunspot relative number Incidence total

104 4.7

12 1.9

40 3.8

75 2.9

110 0.9

180 2.7

175 3.9

30 1.6

(a) Calculate the least-squares regression line for these data. (b) Plot the points and the least-squares regression line on the same graph. 8.2.12.

It is believed that the average size of a mammal species is a major factor in the period of gestation (the period of development in the uterus from conception until birth). In general, it is observed that the bigger the mammal is, the longer the gestation period. Table 8.2.1 gives adult mass in kilograms and gestation period in weeks of some species (source: http://www.saburchill.com/chapters/chap0037.html).

Table 8.2.1 Species

Adult mass (kg)

Gestation period (weeks)

African elephant

6000

88

Horse

400

48

Grizzly bear

400

30

Lion

200

17

Wolf

34

9

Badger

12

8

Rabbit

2

4.5

Squirrel

0.5

3.5

428 CHAPTER 8 Linear Regression Models

Table 8.2.2 Species

Gestation period (weeks)

Indian elephant

89.0

Camel

57.0

Sea lion

51.4

Dog

8.7

Rat

3.0

Hamster

2.3

(a) Calculate the least-squares regression line for these data with adult mass as the independent variable. (b) Plot the points and the least-squares regression line on the same graph. (c) Calculate the least-squares regression line for these data with gestation period as the independent variable. (d) Assuming that the regression model of part (c) holds for all mammals, estimate the adult mass in kilograms for the mammals given in Table 8.2.2.

8.3 INFERENCES ON THE LEAST-SQUARES ESTIMATORS Once we obtain the estimators of the slope β1 and intercept β0 of the model regression line, we are in a position to use Theorem 8.2.1 to make inferences regarding these model parameters. Using the properties of βˆ 0 and βˆ 1 , in this section we study the conﬁdence intervals and hypothesis tests concerning these parameters. From Theorem 8.2.1, we can write Z1 =

βˆ 1 − β1 √σ Sxx

∼ N(0, 1).

Also, it can be shown that SSE/σ 2 is independent of βˆ 1 and has a chi-square distribution with n − 2 degrees of freedom. Let the mean square error be deﬁned by MSE =

n 1 SSE = [yi − (βˆ 0 + βˆ 1 xi )]2 . n−2 n−2 i=1

Then using Deﬁnition 4.2.2, we have Z βˆ 1 − β1 tβ1 = 8 = 9 SSE σ2

n−2

MSE Sxx

8.3 Inferences on the Least-Squares Estimators 429

which follows the t-distribution with n − 2 degrees of freedom. Similarly, let Z0 =

βˆ − β0 0 ∼ N(0, 1). 2 σ 1n + Sx yy

Also, it can be shown that βˆ 0 and SSE are independent. Hence, tβ0 = 8

z0 SSE σ2

βˆ 0 − β0 1/2 2 MSE 1n + Sx

=

xx

n−2

follows the t-distribution with n − 2 degrees of freedom. From these derivations, we can obtain the following procedure about the conﬁdence intervals for the slopes β1 and for the intercept β0 .

PROCEDURE FOR OBTAINING CONFIDENCE INTERVALS FOR β0 AND β1 1. Compute Sxx , Sxy , Sxy , y , and x as in the procedure for ﬁtting a least-squares line. 2. Compute βˆ 1 , βˆ 0 using equations βˆ 1 = (Sxy )/(Sxx ) and βˆ 0 = y − βˆ 1 x , respectively. 3. Compute SSE by SSE = Syy − βˆ 1 Sxy . 4. Deﬁne MSE (mean square error) to be MSE =

SSE , n−2

where n = Number of pairs of observations x1 , y1 , . . . , xn , yn . 5. A (1 − α)100% conﬁdence interval for β1 is given by +

βˆ 1 − tα/2,n−2

+ MSE , βˆ 1 + tα/2,n−2 Sxx

MSE Sxx

where ta/2 is the upper tail α/2-point based on a t-distribution with (n − 2) degrees of freedom. 6. A (1 − α)100% conﬁdence interval for β0 is given by ⎛

%

⎝βˆ 0 − tα/2, n−2 MSE

x2 1 + n Sxx

& 2

% , βˆ 0 + tα/2,n−2 MSE

x2 1 + n Sxx

We illustrate this procedure for obtaining conﬁdence limits with an example.

& 1/2 ⎞ ⎠.

430 CHAPTER 8 Linear Regression Models

Example 8.3.1 For the data of Example 8.2.1: (a) Construct a 95% conﬁdence interval for β0 and interpret. (b) Construct a 95% conﬁdence interval for β1 and interpret.

Solution The following calculations were obtained in Example 8.2.1: Sxx = 263.6, Sxy = 534.2, y = 4.6 and x = 3.8. Also, βˆ 1 = 2.0266, βˆ 0 = −3.1011. In addition to those calculations, we can compute n

y12 = 1302 and Syy =

i=1

n

y12 −

n

2 yi

i=1

n

i=1

= 1302 −

(46)2 = 1090.4. 10

Now, SSE = Syy − βˆ 1 Sxy = 1090.4 − (2.0266)(534.2) = 7.79028. Hence, MSE =

SSE 7.79028 = = 0.973785. n−2 8

Now from the t-table, we have t0.025,8 = 2.306. (a) A 95% confidence interval for β0 is given by ⎛

%

2 ⎝βˆ 0 − tα/2,n−2 MSE 1 + x n Sxx

&1/2

%

x2 1 + , βˆ 0 + tα/2,n−2 MSE n Sxx

&1/2 ⎞ ⎠

⎛

&1/2 % (3.8)2 1 ⎝ + = −3.1011 − (2.306) (0.973785) 10 263.6 %

1 (3.8)2 −3.1011 + (2.306) (0.973785) + 10 263.6

&1/2 ⎞ ⎠

From which we obtain a 95% confidence interval for β0 as (−3.9846, −2.2176). Thus, we can conclude with 95% confidence that the true value of the intercept, β0 , is between −3.9846 and −2.2176.

8.3 Inferences on the Least-Squares Estimators 431

(b) A 95% confidence interval for β1 is given by 8 8

MSE MSE ˆ ˆβ1 − tα/2,n−2 , β1 + tα/2,n−2 Sxx 8 Sxx 8 0.973785 0.973785 , 2.0266 + (2.306) = 2.0266 − (2.306) 236.6 236.6 from which we obtain a 95% confidence interval for β1 as (1.8864, 2.1668). Thus, we can conclude with 95% confidence that the true value of the slope of the linear regression model is between 1.8864 and 2.1663.

One of the assumptions for linear regression model that we have made is that the variance of the errors is a constant and independent of x. Errors with this property are called homoscedastic. If the variance of the errors is not constant, the errors are called heteroscedastic. In the heteroscedastic case, standard errors and conﬁdence intervals based on the assumption that s2 is an estimate of σ 2 may be somewhat deceptive. Now we introduce hypothesis testing concerning the slope and intercept of the ﬁtted least-squares line. We use tβ0 and tβ1 deﬁned earlier as the test statistic for testing hypotheses concerning β0 and β1 , respectively. The usual one- and two-sided alternatives apply. We proceed to summarize these test procedures. HYPOTHESIS TEST FOR β0 One-sided test

Two-sided test

H0 : β0 = β00 (β00 is a speciﬁc value of β0 )

H0 : β0 = β00

Ha : β0 > β00 or β0 < β00

Ha : β0 = β00

Test statistic:

Test statistic:

βˆ 0 − β00

" 1/2 x 1 MSE + n Sxx

βˆ 0 − β00

" 1/2 x 1 MSE + n Sxx

tβ0 = !

tβ0 = !

Rejection region:

Rejection region:

t > tα, (n−2) (upper tail region) t < −tα, (n−2) (lower tail region)

|t| > tα/2,(n−2)

Decision: If tβ0 falls in the rejection region, reject the null hypothesis at level of signiﬁcance α. Assumptions: Assume that the errors εi , i = 1, . . . , n are independent and normally distributed with E (εi ) = 0, i = 1, . . . , n, and Var (εi ) = σ 2 , i = 1, . . . , n.

We now illustrate this procedure with the following example.

432 CHAPTER 8 Linear Regression Models

Example 8.3.2 Using the data given in Example 8.2.1, test the hypothesis H0 : β0 = −3 versus Ha : β0 = −3 using the 0.05 level of signiﬁcance.

Solution We test H0 : β0 = −3 versus Ha : β0 = −3. Here β00 = −3. The rejection region is t < −2.306 or t > 2.306. From the calculations of the previous example, we have βˆ 0 − β00 1/ 2 2 MSE 1n + Sx

tβ0 =

xx

−3.1011 − (−3) 1 + (3.8)2 1/2 (0.973785) 10 263.6

=

= −0.26041. Because the test statistic does not fall in the rejection region, at α = 0.05, we do not reject H0 .

HYPOTHESIS TEST FOR β1 One-sided test H0 : β1 = β10 β10 is a speciﬁc value of β1

Two-sided test H0 : β1 = β10

Ha : β1 > β10 or β1 < β10

Ha : β1 = β10

Test statistic:

Test statistic:

βˆ 1 − β10 tβ1 = 8 MSE Sxx

βˆ 1 − β10 tβ1 = 8 MSE Sxx

Rejection region:

Rejection region:

t > tα,(n−2) (upper tail region) t < −tα,(n−2) (lower tail region)

|t| > tα/2,(n−2)

Decision: If tβ1 falls in the rejection region, reject the null hypothesis at conﬁdence level α. Assumptions: Assume that the errors εi , i = 1, . . . , n are independent and normally distributed with E (εi ) = 0, i = 1, . . . , n, and Var (εi ) = σ 2 , i = 1, . . . , n.

The test of hypothesis H0 : β1 = 0 answers the question, Is the regression signiﬁcant? If β1 = 0, we conclude that there is no signiﬁcant linear relationship between X and Y , and hence, the independent

8.3 Inferences on the Least-Squares Estimators 433

variable X is not important in predicting the values of Y if the relationship of Y and X is not linear. Note that if β1 = 0, then the model becomes y = β0 + ε. Thus, the question of the importance of the independent variable in the regression model translates into a narrower question of the test of hypothesis H0 : β1 = 0. That is, the regression line is actually a horizontal line through the intercept, β0 .

Example 8.3.3 Using the data given in Example 8.2.1, test the hypothesis H0 : β1 = 2 versus Ha : β1 = 2 using the 0.05 level of signiﬁcance.

Solution We test H0 : β1 = 2 vs. Ha : β1 = 2. We know that βˆ 1 = 2.0266. For α = 0.05 and n = 10, the rejection region is t < −2.306 or t > 2.306. The test statistic is βˆ 1 − β10 tβ1 = 8 MSE Sxx 2.0266 − 2 = 0.4376. = 8 2.0266 − 2 263.6 Because the test statistic does not fall in the rejection region, at α = 0.05, we do not reject H0 . Thus, for α = 0.05, the given data support the null hypothesis that the true value of the slope, β1 , of the regression line is equal to 2.

Another problem closely related to the problem of estimating the regression coefﬁcients β0 and β1 is that of estimating the mean of the distribution of Y for a given value of x, that is, estimating β0 + β1 x. For a ﬁxed value of x, say x0 , we have the following conﬁdence limits. A (1 − α)100% conﬁdence interval for β0 + β1 x is given by + 2 x0 − x 1 ˆβ0 + βˆ 1 x ± tα 2 se + / n Sxx where

+ se =

2 Syy − Sxy . (n − 2)Sxx

We could use the data from the previous example to easily calculate a conﬁdence interval for β0 +β1 x.

434 CHAPTER 8 Linear Regression Models

8.3.1 Analysis of Variance (ANOVA) Approach to Regression Another approach to hypothesis testing is based on ANOVA. A detailed explanation of this approach is given in Chapter 10. Here we present necessary steps for regression. The main reason for this presentation is the fact that most of the major statistical software outputs for regression analysis (see Section 8.9) are given in the form of ANOVA tables. It can be veriﬁed that (see Exercise 8.3.7) n

(yi − y)2 =

i=1

n

yi − yˆ i

2

+

i=1

n

yˆ i − y

2

.

i=1

Denoting SST =

n

(yi − y)2 , SSE =

i=1

n

yi − yˆ i

2

, and SSR =

i=1

n

yˆ i − y

2

,

i=1

the foregoing equation can be written as SST = SSR + SSE.

Note that the total sum of squares (SST ) is a measure of the variation of yi ’s around the mean y, and SSE is the residual or error sum of squares that measures the lack of ﬁt of the regression model. Hence, SSR (sum of squares of regression or model) measures the variation that can be explained by the regression model. We saw that to test the hypothesis H0 : β1 = 0 vs. Ha : β1 = 0, the statistic tβ1 = 8

βˆ 1 MSE Sxx

was used, where tβ1 follows a t-distribution with (n − 2) degrees of freedom. From Exercise 4.2.18, we know that tβ21 =

βˆ 12 MSE Sxx

follows an F -distribution with numerator degrees of freedom 1 and denominator degrees of freedom (n − 2). We can also verify that tβ21 =

MSR . MSE

8.3 Inferences on the Least-Squares Estimators 435

Table 8.1 ANOVA Table for Simple Regression Source of variation

Degrees of freedom

Sum of squares

Regression (model)

1

SSR

Error (residuals)

n−2

SSE

Total

n−1

SST

Mean sum of squares MSR =

F-ratio MSR MSE

SSR d.f.

SSE d.f.

Thus, to test H0 : β1 = 0 vs. Ha : β1 = 0, we could use the statistic MSR ∼ F (1, n − 2) MSE

and reject H0 if MSR ≥ Fα (1, n − 2). MSE

These can be summarized by Table 8.1, known as the ANOVA table. The last column in the ANOVA table gives the statistic (MSR)/(MSE). It is also customary to give another column with the p-value of the test.

Example 8.3.4 In a study of baseline characteristics of 20 patients with foot ulcers, we want to see the relationship between the stage of ulcer (determined using the Yarkony-Kirk scale, a higher number indicating a more severe stage, with range 1 to 6), and duration of ulcer (in days). Suppose we have the data shown in Table 8.2. (a) Give an ANOVA table to test H0 : β1 = 0 vs. Ha : β1 = 0. What is the conclusion of the test based on α = 0.05? (b) Write down the expression for the least-squares line.

Table 8.2 Stage of Ulcer (x)

4

3

5

4

4

3

3

4

6

3

Duration (d)

18

6

20

15

16

15

10

18

26

15

Stage of Ulcer (x)

3

4

3

2

3

2

2

3

5

6

Duration (d)

8

16

17

6

7

7

8

11

21

24

436 CHAPTER 8 Linear Regression Models

Table 8.3 Source of variation

Sum of squares

Mean sum of squares

F-ratio

p-Value

1

570.04

570.04

77.05

0.000

Error (residuals)

18

133.16

7.40

Total

19

703.20

Regression (model)

Degrees of freedom

Solution (a) We test H0 : β1 = 0 vs. Ha : β1 = 0. We will use Minitab to generate the ANOVA table (Table 8.3). Because the p-value is less than 0.001, for α = 0.05, we reject the null hypothesis that β1 = 0 and conclude that there is a relationship between the stage of ulcer and its duration. (b) Again, using the Minitab output, we get the least-squares line as d = 4.61x − 2.40.

EXERCISES 8.3 8.3.1.

An experiment was conducted to observe the effect of an increase in temperature on the potency of an antibiotic. Three one ounce portions of the antibiotic were stored for equal lengths of time at each of the following Fahrenheit temperatures: 40◦ , 55◦ , 70◦ , and 90◦ . The potency readings observed at the end of the experimental period were Potency reading, y 49 38 27 24 38 33 19 28 16 18 23 Temperature, x 40◦ 55◦ 70◦ 90◦ (a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line as a check on your calculations. (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.3.2.

Consider the data x 38 26 48 22 40 15 30 33 y 10 11 16 8 12 5 10 11 (a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line as a check on your calculations. (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.3.3.

Show that Y and βˆ 1 are independent, under the usual assumptions of a simple linear regression model.

8.3.4.

Using the data of Exercise 8.2.10, calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.4 Predicting a Particular Value of Y 437

8.3.5.

The following data represent survival time in days after a heart transplant and patient age in years at the time of transplant for 10 randomly selected patients. Age at transplant 28 41 46 53 39 36 47 29 48 44 Survival time, in days 7 278 44 48 406 382 1995 176 323 1846

(a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line. (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively.

8.3.6.

The following data represent weights of cigarettes (g) from different manufacturers and their nicotine contents (mg). Weight 15.8 14.9 9.0 4.5 15.0 17.0 8.6 12.0 4.1 16.0 Nicotine 0.957 0.886 0.852 0.911 0.889 0.919 0.969 1.118 0.946 1.094

(a) Find the least-squares line appropriate for these data. (b) Plot the points and graph the line. Do you think the linear regression is appropriate? (c) Calculate the 95% conﬁdence intervals for β0 and β1 , respectively. 8.3.7.

Show that n n n 2 2 2 yi − y = yi − yˆ i + yˆ i − y . i=1

i=1

i=1

8.4 PREDICTING A PARTICULAR VALUE OF Y In the earlier sections, we have seen how to ﬁt a least-squares line for a given set of data. Also using this line, we could ﬁnd E(Y ), for any given value of x. Instead of obtaining this mean value, we may be interested in predicting the particular value of Y for a given x. In fact, one of the primary uses of the estimated regression line is to predict the response value of Y for a given value of x. Prediction problems are very important in several real-world problems; for example, in economics one may be interested in a particular gain associated with an investment. Let Yˆ 0 denote a predictor of a particular value of Y = Y0 and let the corresponding values of x be x0 . We shall choose Yˆ 0 to be E(Yˆ |x0 ). Let Yˆ denote a predictor of a particular value of Y . Then the error η of the predictor in comparison to a particular value of Y is η = Y − Yˆ 0 .

438 CHAPTER 8 Linear Regression Models

Both Y and Yˆ are normal random variables, and the error is a linear function of Y and Yˆ . This means that η itself is normally distributed. Also, because E(Yˆ ) = E(Y ), we have E(η) = E(Y |x0 ) − E Yˆ = 0.

Furthermore, Var(η) = Var Y − Yˆ = Var(Y ) + Var Yˆ − 2Cov Y, Yˆ .

We can consider Y and Yˆ as independent, because we are predicting a different value of Y , not used in the calculation of Yˆ . Therefore, Cov(Y, Yˆ ) = 0. In that case Var(η) = Var(Y0 ) + Var Yˆ 0 % & 1 (x − x)2 2 2 + =σ +σ n Sxx & % 1 (x − x)2 σ2. = 1+ + n Sxx

Hence, the error of predicting a particular value of Y , given x, is normally distributed with mean zero (x−x)2 1 and variance 1 + n + Sxx σ 2 . That is,

& 1 (x − x)2 σ2 , η ∼ N 0, 1 + + n Sxx

%

and Z=

Y − Yˆ

8 σ

2 1 + 1n + (x−x) Sxx

∼ N(0, 1).

If we substitute the sample standard deviation S for σ, then we can show that Y − Yˆ T = 8 2 S 1 + 1n + (x−x) S xx

follows the t-distribution with [n − (k + 1)] degrees of freedom. Using this fact, we now give a prediction interval for the random variable Y , the response of a given situation. We know that P −tα/2 < T < tα/2 = 1 − α.

8.4 Predicting a Particular Value of Y 439

Substituting for T , we have ⎞

⎛ ⎜ P⎜ ⎝−tα/2

0, there is a positive relation between X and Y (increasing slope); and when ρ < 0, we have a negative relationship (decreasing slope). Thus, the correlation coefﬁcient can be used to measure how well the linear regression model ﬁts the data. Let (X1 , Y1 ), (X2 , Y2 ), . . . , (Xn , Yn ) be a random sample from a bivariate normal distribution. The maximum likelihood estimator of ρ is the sample correlation coefﬁcient deﬁned by ρˆ or r, n

Xi − X Yi − Y

r= +

i=1 n

Xi − X

n 2

i=1

Yi − Y

(8.7)

2

i=1

Sxy . = Sxx Syy

Equivalently, we can rewrite (8.7) by n

n

Xi Yi −

i=1

n i=1

Xi

n

Yi

i=1

r = ;⎡ . < 2 ⎤ ⎡ 2 ⎤ < n n n n 2 0, values of y increase as the values of x increase, and the data set is said to be positively correlated. When r < 0,

442 CHAPTER 8 Linear Regression Models

values of y decrease as the values of x increase, and the data set is said to be negatively correlated. In this book, we use the term correlation only when referring to linear relationships. In actual practice we can use the value of r to decide whether it is appropriate to develop linear regression models in a given situation. As a rule of thumb, if r > 0.30 or r < −0.30, we proceed with developing a linear regression model. However, a much higher or lower value is desirable. For example, if in a given problem where r = 0.77, it conveys to us that approximately 77% of the data we have are linearly related. The probability distribution for r is difﬁcult to obtain. For large samples, this difﬁculty could be overcome by using the fact that the Fisher z-transform, given by z = (1/2) ln[(1 + r)/(1 − r)]

is approximately normally distributed with mean μz = (1/2) ln[(1 + ρ)(1 − ρ)] and variance σz = 1/(n − 3). Thus, for large random samples, we can test hypotheses about ρ using the approximate test statistic: Z=

=

z − μz σz

1+ρ (1/2) ln 1+r 1−r − (1/2) 1−ρ √1 n−3

.

For example, suppose we are interested in testing the hypothesis that the true value of ρ is a speciﬁc number, say, ρ0 , with a certain value of α. We can proceed to make a decision by following the procedure given next.

HYPOTHESIS TEST FOR ρ One-sided test

Two-sided test

H0 : ρ = ρ0

H0 : ρ = ρ0

Ha : ρ > ρ0 or

Ha : ρ = ρ0

Ha : ρ < ρ0 Test statistic:

Z=

(1/2) ln

1+r 1−r

√

1+ρ −(1/2) 1−ρ0

1

0

Test statistic:

Z=

(1/2) ln

n−3

1+r 1−r

√

−(1/2)

1

1+ρ0 1−ρ0

n−3

Rejection region:

Rejection region:

z > za (upper tail region) z < −za (lower tail region)

|z| > za/2

Decision: If Z falls in the rejection region, reject the null hypothesis at conﬁdence level α. Assumption: (X ,Y ) follow the bivariate normal, and this test procedure is approximate.

8.5 Correlation Analysis 443

Example 8.5.1 For the data given in Example 8.2.1, would you say that the variables X and Y are independent? Use α = 0.05.

Solution We test H0 : ρ = 0 vs. Ha : ρ = 0. From Example 8.2.1, we have the following summary: n

xi = 38;

i=1

n

yi = 46;

i=1

n

xi yi = 709

i=1

and n

xi2 = 408;

i=1

n

yi2 = 1302; n = 10.

i=1

Hence, n

n

Xi Yi −

i=1

n i=1

Xi

n

Yi

i=1

r = ;⎡ < 2 ⎤ ⎡ 2 ⎤ < n n n n 1.96. Because the observed value of the test statistic falls in the rejection region, we reject the null hypothesis and conclude that at α = 0.05, the variables X and Y are dependent.

444 CHAPTER 8 Linear Regression Models

EXERCISES 8.5 8.5.1.

The table shows the midterm and ﬁnal examination test scores for 10 students from a differential equations class, where x denotes the midterm scores and y denotes the ﬁnal scores. x y

68 74

87 89

75 80

91 93

82 88

77 79

86 97

82 95

75 89

79 92

(a) At 95% conﬁdence level, test whether X and Y are independent. (b) Find the p-value. (c) State any assumptions you have made in solving the problem. 8.5.2.

The following table gives the annual incomes (in thousands of dollars) and amounts (in thousands of dollars) of life insurance policies for eight persons. Annual income 42 58 27 36 70 24 53 37 Life insurance 150 175 25 75 250 50 250 100 (a) At the 98% conﬁdence level, test whether annual income and the amount of life insurance policies are independent. (b) Find the attained signiﬁcance level. (c) State any assumptions you have made in solving the problem.

8.5.3.

Show that n

n

n

Xi Yi −

i=1

i=1

Xi

n

Yi

i=1

r = ;⎡ < 2 ⎤ ⎡ 2 ⎤ < n n n n 1) independent variables are used to predict the dependent variable. The model to be studied is of the form Y = β0 + β1 x1 + β1 x2 + · · · + βk xk + ε.

Here, ε ∼ N 0, σ 2 . This model is called a multiple regression model. Let y1 , y2 , . . . , yn be n independent observations on Y . Then each observation yi can be written as yi = β0 + β1 xi1 + β2 xi2 + · · · + βk xik + ε

where xij is the jth independent variable for the ith observation, i = 1, 2, . . . , n, and εi s are independent as in the simple linear regression case. It is sometimes advantageous to introduce matrices to study the linear equations. Let x0 = 1. Deﬁne the following matrices: ⎡ ⎢ ⎢ ⎢ ⎢ X=⎢ ⎢ ⎢ ⎣

x0 x0 . . . x0

⎢ ⎢ ⎢ ⎢ Y =⎢ ⎢ ⎢ ⎣

⎤

⎡

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

⎢ ⎢ ⎢ ⎢ ε=⎢ ⎢ ⎢ ⎢ ⎣

and

x1k x2k . . . xnk

⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

⎡ ⎢ ⎢ ⎢ ⎢ β=⎢ ⎢ ⎢ ⎣

. . . . . .

⎡

x12 x22 . . . xn2

β0 β1 . . . βk

. . . . . .

⎤

x11 x21 . . . xn1

ε1 ε2 . . . εn

y1 y2 . . . yn

⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

⎤ ⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎥ ⎦

Thus the n equations representing the linear equations can be rewritten in the matrix form as Y = Xβ + ε.

In particular, for the n observations from the simple linear model of the form Y = β0 + β1 x + ε

we can write Y = Xβ + ε,

(8.8)

446 CHAPTER 8 Linear Regression Models

where ⎡ ⎢ ⎢ ⎢ ⎢ Y =⎢ ⎢ ⎢ ⎣

⎤

y1 y2 . . . yn

⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

⎡

1 1 1 1 1 1

⎢ ⎢ ⎢ ⎢ X=⎢ ⎢ ⎢ ⎣

x1 x2 . . . xn

⎤

⎡

⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

We can see that

⎢ ⎢ ⎢ ⎢ ε=⎢ ⎢ ⎢ ⎣

⎡ % X X =

1 x1

1 x2

. .

. .

. .

ε1 ε2 . . . εn

⎢ &⎢ ⎢ 1 ⎢ ⎢ xn ⎢ ⎢ ⎣

1 1 . . . 1

x1 x2 . . . xn

⎤ ⎥ ⎥ ⎥ ⎥ ⎥, ⎥ ⎥ ⎦

%

and

& β0 β= . β1

⎤

⎡ ⎥ ⎥ n ⎥ ⎢ ⎥ ⎢ ⎥=⎢ n ⎥ ⎣ ⎥ xi ⎦ i=1

n i=1 n i=1

⎤ xi

⎥ ⎥ ⎥, ⎦ 2 x1

where denotes the transpose of a matrix. Also, ⎡

n

⎤ yi

⎢ i=1 ⎢ X Y = ⎢ n ⎣ xi yi

⎥ ⎥ ⎥. ⎦

i=1

Let us now go back to the multiple regression model Y = β0 + β1 x1 + β1 x2 + · · · + βk xk + ε.

The least-squares estimators βˆ i of βi for i = 0, 1, 2, . . . , k are the ones that minimize the sum of squares SSE =

n

e2i =

i=1

n

2 yi − βˆ 0 + βˆ 1 x1 + βˆ 2 x2 + · · · + βˆ k xk

i=1

y − Xβˆ ˆ ˆ = y y − y Xβˆ − Xβˆ y + βX Xβ.

= y − Xβˆ

To minimize SSE with respect to β, we differentiate SSE with respect to β and equate it to zero. Thus, ∂ y y − y X β − β X y + X β Xβ = 0 ∂β

yielding (X X)βˆ = X Y.

8.6 Matrix Notation for Linear Regression 447

Assuming the matrix (X X) is invertible, we obtain βˆ = (X X−1 )X Y.

Now we summarize the procedure to obtain a multiple linear regression equation. PROCEDURE TO OBTAIN A MULTIPLE LINEAR REGRESSION EQUATION 1. Rewrite the n observations Yi = β0 + β1 x1i + β1 x2i + · · · + βk xki , i = 1, 2, . . . , n in the matrix notation as Y = Xβ + ε where X , Y , and β are deﬁned in (1). 2. Compute (X X )−1 and obtain the estimators of β as βˆ = (X X )−1 X Y . 3. Then the regression equation is ˆ Yˆ = X β.

Example 8.6.1 Using the data given in Example 8.2.1, use the matrix approach to solve the problem of operations.

Solution From the data of Example 8.2.1 we have ⎡ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ Y =⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎢ ⎣

−9 −7 −5 −4 2 6 9 13 21 20

⎤

⎡

⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎥ ⎢ ⎥ ⎢ and X = ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎥ ⎢ ⎢ ⎥ ⎢ ⎥ ⎥ ⎢ ⎣ ⎦

1 1 1 1 1 1 1 1 1 1

−3 −2 −1 0 2 5 6 8 11 12

⎤ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎥. ⎥ ⎥ ⎥ ⎥ ⎥ ⎥ ⎦

Thus, we can write % X X =

10 38

& % & % 38 46 0.1548 X Y = (X X)−1 = 408 709 −0.0144

& −0.0144 . 0.0038

448 CHAPTER 8 Linear Regression Models

Hence, %

0.1548 −0.0144 % & % & −3.1009 βˆ = = 0 . βˆ 1 2.0266

βˆ = (X X)−1 (X Y )

−0.0144 0.0038

&%

46 709

&

Thus, the least-squares line is given by yˆ = −3.1009 + 2.0266X, which is identical to the regression line we obtained in Example 8.2.1.

Example 8.6.2 The following data relate to the prices (Y ) of ﬁve randomly chosen houses in a certain neighborhood, the corresponding ages of the houses (x1 ), and square footage (x2 ). Price y in thousands Age x1 in Square footage x2 in thousands of dollars years of square feet 100 1 1 80 5 1 104 5 2 94 10 2 130 20 3 Fit a multiple linear regression model Y = β0 + β1 x1 + β2 x2 + ε to the foregoing data.

Solution We have ⎡

⎤ ⎡ 100 1 1 ⎢ 80 ⎥ ⎢1 5 ⎢ ⎥ ⎢ ⎢ ⎥ ⎢ Y = ⎢ 104 ⎥ ; X = ⎢1 5 ⎢ ⎥ ⎢ ⎣ 94 ⎦ ⎣1 0 130 1 20 ⎡ ⎤ 508 ⎢ ⎥ X Y = ⎣4560⎦ 966

⎤ 1 ⎡ 1⎥ 5 ⎥ ⎥ ⎢ 2⎥ ; X X = ⎣41 ⎥ 2⎦ 9 3

41 551 96

⎤ 9 ⎥ 96⎦ ; 19

8.6 Matrix Notation for Linear Regression 449

and

⎡

2.3076 ⎢ (X X)−1 = ⎣ 0.1565 −1.8840 Hence,

0.1565 0.0258 −0.2044

⎤ −1.8840 ⎥ −0.2044⎦ . 1.9779

⎡

⎤ 66.1252 ⎢ ⎥ (X X)−1 (X Y ) = ⎣−0.3794⎦ . 21.4365

Thus, the regression model is y = 66.12 − 0.3794x1 + 21.4365x2 .

8.6.1 ANOVA for Multiple Regression As in Section 8.3, we can obtain an ANOVA table for multilinear regression (with k independent or explanatory variables) to test the hypothesis H0 : β1 = β2 = · · · = βk = 0

versus Ha : At least one of the parameters βj = 0, j = 1, . . . , k.

The calculations for multiple regression are almost identical to those for simple linear regression, except that the test statistic (MSR)/(MSE) has an F (k, n − k − 1) distribution. Note that the F -test does not indicate which of the parameters βj = 0, except to say that at least one of them is not zero. The ANOVA table for multiple regression is given by Table 8.4.

Table 8.4 ANOVA Table for Multiple Regression Source of variation

Degrees of freedom

Sum of squares

Regression (Model)

k

SSR

Error (Residuals)

n−k−1

SSE

n−1

SST

Total

Mean sum of squares MSR =

SSR d.f.

SSE d.f.

F-ratio MSR MSE

450 CHAPTER 8 Linear Regression Models

Example 8.6.3 For the data of Example 8.6.2, obtain an ANOVA table and test the hypothesis H0 : β1 = β2 = 0 vs. Ha : at least one of the βi = 0, i = 1, 2. Use α = 0.05.

Solution We test H0 : β1 = β2 = 0 vs. Ha : At least one of the βi = 0, i = 1, 2. Here n = 5, k = 2. Using Minitab, we obtain the ANOVA table (Table 8.5). Based on the p-value, we cannot reject the null hypothesis at α = 0.05.

Table 8.5 Source of Degrees of Sum of Mean sum of F-ratio p-Value variation freedom squares squares Regression (Model)

2

956.5

478.2

Error (Residuals)

2

382.7

191.4

Total

4

1339.2

2.50

0.286

EXERCISES 8.6 8.6.1.

Given the data X1 3 2 3 1

(a) (b) (c) (d) 8.6.2.

X2 1 5 3 2

y 4 3 6 5

Write the multiple regression model in matrix form. Find X X, (X X)−1 , and X y. Estimate β. Estimate the error variance.

A study is conducted to estimate the demand for housing (y) based on current interest rate X1 and the rate of unemployment. The data in Table 8.6.1 are obtained. (a) Fit the multiple regression model y = β0 + β1 x1 + β1 x2 + ε.

8.7 Regression Diagnostics 451

Table 8.6.1 Units sold

Interest rate (%)

Unemployment rate (%)

65

9.0

10.0

59

9.3

8.0

80

8.9

8.2

90

9.1

7.7

100

9.0

7.1

105

8.7

7.2

(b) Test whether the model is signiﬁcant. 8.6.3.

The following data give the annual incomes (in thousands of dollars) and amounts (in thousands of dollars) of life insurance policies for eight persons. Annual income 42 58 27 36 70 24 53 37 Life insurance 150 175 25 75 250 50 250 100 Calculate the least-squares regression line for these data using matrix operations.

8.6.4.

The following is a random sample of height (in inches) and weight (in pounds) of seven basketball players. Height Weight

73 186

83 234

77 208

80 237

85 265

71 190

80 220

Calculate the least-squares regression line for these data using matrix operations.

8.7 REGRESSION DIAGNOSTICS In the previous sections, we derived least-squares estimators for the parameters in the linear regression model. These estimators are useful as long as we can determine (1) how well the model ﬁts the data and (2) how good our estimates are in providing possible relationships between variables of interest. Some of these problems are discussed in Chapter 14 in a uniﬁed manner. We now brieﬂy discuss some aspects of the adequacy of the simple linear regression model. In multiple regression, in addition to the problems discussed here, there are other problems, such as collinearity and model speciﬁcation (inclusion of all relevant variables, as well as exclusion of irrelevant variables), that need to be examined. They are beyond the level of this text. Many graphical methods and numerical tests dealing with these problems are available in the literature and are often called regression diagnostics. Most of the major statistical software packages incorporate these tests, making it easier to perform regression diagnostics so as to detect potential problems. We have seen that the (ordinary) least-squares regression model must meet the following assumptions.

452 CHAPTER 8 Linear Regression Models

1. Linearity. The existence of a linear relationship between x and y is the basis of the simple linear regression model. A simple method to test for linearity is to draw a scatterplot of data points. As we explained in Section 8.2, we could also plot residual ei versus xi or Yˆ i . A symmetric trend in the plot of the residuals versus the explanatory variable or the ﬁtted values indicates there is a problem with the obtained regression model. For a correct model, the residuals should center around zero across the explanatory variables and the ﬁtted values. The degree of linear relationship can be ascertained by the correlation coefﬁcient, r, given in Section 8.5 or by using the value of the coefﬁcient of determination r 2 , explained in Project 8B. Most statistical software packages give the value of r 2 (refer to outputs given in Section 8.9). The closer the value of r 2 is to 1, the better the least-squares equation yˆ = βˆ 1 x + βˆ 0 performs as a predictor of y. 2. Homoscedasticity (homogeneity of variance). This assumption says that the variance of the error term remains constant across all values of x. In this case we know by the Gauss–Markov theorem that the least-squares estimators βˆ 0 and βˆ 1 are the best linear unbiased estimators of β0 and β1 . A frequently used graphical method is to draw the residuals versus a ﬁtted plot. This can be easily done using statistical software packages. The graph of residuals ei versus ﬁtted values Yˆ i or explanatory variable xi indicates a change in the spread of residuals as Yˆ or x changes. It may look like Figure 8.7. If the variances of yi values are not constant, the inferences we made, such as conﬁdence intervals on means, prediction, and so forth, are off. The severity of this discrepancy depends on the degree of the assumption violation. If we see that the pattern of data points only changes slightly, that will indicate a mild heteroscedasticity. Two numerical tests for heteroscedasticity are explained in Section 14.4.3.

Residuals versus the fitted values (response is C2)

20

Residual

10

0 ⫺10 ⫺20 0

10

20

30

40 50 Fitted value

■ FIGURE 8.7 Scatterplot of fitted values versus residuals.

60

70

80

90

8.7 Regression Diagnostics 453

3. Independence of εi and εj , for i = j. This assumption speciﬁes that the errors associated with one observation should not be correlated with the errors of any other observation. In general, whether the two samples are independent of each other is decided by the structure of the experiment from which they arise. Violation of the independence assumption can occur in a variety of situations. For example, if we take a survey on a certain issue on children’s education from one particular school, these observations may reﬂect some pattern, thus violating the independence assumption. If data are collected on the same variable over time, then the assumption of independence will be violated. Project 12B explains a run test for check of this assumption. Also, see Section 14.4.4. 4. Normality of the errors. This assumption speciﬁes that the distribution of the εi values should be normal. This assumption is crucial when sample size is small if the p-value for the test is to be valid. For large samples, by the Central Limit Theorem this assumption becomes less important unless the prediction of a single value of y is involved. Thus a test of normality is necessary mainly when the t-test is used. Section 14.4.1 explains some of the tests for normality. A simple way is to draw a probability plot for the errors to conform to the assumption of normality. If we observe nonnormality, one of the ways to overcome the problem is to use data transformation such as logarithmic transformation, as explained in Section 14.4.2, and perform the regression analysis on the transformed data. Sometimes nonparametric methods may be more appropriate, but we will not deal with this topic in this book. Another important issue is the existence of inﬂuential observations, individual observations that have a strong inﬂuence on estimated coefﬁcients. If a single observation substantially changes our results, we need to do further investigation. The ordinary least-squares method is quite sensitive for outlying observations, both for independent variables and for dependent variables, and can have an adverse effect on the estimate. In higher dimensional data, these outlying observations can remain unnoticed. This aspect in one explanatory variable case is discussed in Project 8C. One of the simple ways to identify such observations is to draw a scatterplot. In the scatterplot, if we see a data point that is farther away from the rest of the data points, that is an indication of possible inﬂuential points. The natural question is, if we ﬁnd that the data violate one or more of the assumptions, what can we do about it? We have already explained that violation of the normality assumption in large samples is not an issue unless prediction is involved, because prediction depends on normality of an individual observation. Thus, if the inferences are based on the t- or F -tests or prediction is involved, we may be able to transform Y to Y to achieve normality. If we have predicted Y , then back-transform to predict Y . If we observe nonlinearity of data, we may be able to transform x to x = h(x) such that Y is linear in x , or consider a polynomial model in x, in which case the ideas of multiple linear regression may be utilized. Robust estimates of variances of β0 and β1 or the method of weighted least squares may be used to deal with the case of nonconstant variance. Often careful experimental design could be done to remove possible correlation in errors. There are also robust methods available for correlation analysis. We refer to specialized books on regression methods for further details on these issues. If we detect inﬂuential observations, there are statistical techniques available, such as least trimmed squares estimators, to deal with outlying observations.

454 CHAPTER 8 Linear Regression Models

8.8 CHAPTER SUMMARY In this chapter, we ﬁrst derived the least-squares line and its properties. Then we learned about the conﬁdence intervals for the coefﬁcients in the regression model and did hypothesis tests on the values of the coefﬁcients. We introduced the matrix notation for linear regression as well as for multiple regression. We discussed how to predict a particular value of Y for a given value of X. In order to study the dependence of X and Y , we presented correlation analysis. The following are some of the key deﬁnitions we have used in this chapter. ■

Predictors

■

Response variable

■

Regression analysis

■

Multiple linear regression model

■

Simple linear regression model

■

Sum of squares for errors (SSE)

■

Sum of squares of the residuals

■

Least-squares line

■

Least-squares equations

■

Normal equations

■

Best linear unbiased estimator (BLUE)

■

Correlation analysis

The following important concepts and procedures were discussed in this chapter: ■

Procedure for regression modeling

■

Procedure for ﬁtting a least-squares line

■

Properties of the least-squares estimators for the model Y = β0 + β1 x + ε

■

The Gauss–Markov theorem

■

Procedure for obtaining conﬁdence intervals of β0 and β1

■

Procedure to obtain a multiple linear regression equation

■

Prediction interval for the response variable Y

■

Hypothesis testing for correlation, ρ

■

Linearity

■

Homoscedasticity

■

Independence of εi and εj , for i = j

■

Normality of the errors

■

Inﬂuential observations

8.9 Computer Examples 455

8.9 COMPUTER EXAMPLES 8.9.1 Minitab Examples Example 8.9.1 For the data in Example 8.2.1, use the method of least squares to ﬁt a straight line to the accompanying data points. Give the estimates of β0 and β1 . Plot the points and sketch the ﬁtted least-squares line.

Solution Enter independent variable, x, in C1 and the response variable, y, in C2. Then: Stat > Regression > Regression. . . > in Response: type C2, and in Predictors: type C1 > click OK We obtain the following output.

Regression Analysis The regression equation is

C2 = –3.10 + 2.03 C1 Predictor Constant C1

Coef –3.1009 2.02656

StDev 0.3888 0.06087

S = 0.9883 R-Sq = 99.3% Analysis of Variance Source DF SS Regression 1 1082.6 Residual Error 8 7.8 Total 9 1090.4 Unusual Observations Obs C1 C2 8 11.0 21.000 Residual St Resid 1.809 2.18R

T –7.98 33.29

P 0.000 0.000

R-Sq(adj) = 99.2% MS 1082.6 1.0

Fit 19.191

F 1108.34

P 0.000

StDev Fit 0.538

R denotes an observation with a large standardized residual

From this the estimate of β0 is −3.1009, and the estimate of β1 is 2.02656. Hence, the regression line is yˆ = −3.1009 + 2.02656x. Now to obtain the ﬁtted regression line, use the following procedure: Stat > Regression > Fitted Line Plot. . . > in Response(Y): type C2, and in Predictors(X): type C1 > click Linear OK

456 CHAPTER 8 Linear Regression Models

We obtain the following graph. Regression plot Y ⫽ ⫺3.1009 ⫹ 2.02656x R⫺Sq ⫽ 99.3%

20

C2

10

0

⫺20

0

5 C1

10

If in addition, we need, say, 95% conﬁdence and predictor bands, then use Stat > Regression > Fitted Line Plot. . . > in Response(Y): type C2, and in Predictor(X): type C1 > click Linear > click options. . . > click Display confidence bands and Display predictor bands > in Title: type a title for the graph and OK > OK

We obtain the following graph. Regression line with 95% confidence and predictor bands Y ⫽ ⫺3.1009 ⫹ 2.02656 x R⫺Sq ⫽ 99.3%

20

C2

10

0 Regression 95% C 95% R

⫺20

0

5 C1

10

8.9 Computer Examples 457

8.9.2 SPSS Examples A detailed explanation of regression methods including diagnostics using SPSS can be obtained at the site: http://www.ats.ucla.edu/stat/spss/webbooks/reg/. We will just demonstrate a simple case with an example.

Example 8.9.2 The following is a random sample of height (in inches) and weight (in pounds) of seven basketball players. Height

73

83

77

80

85

71

80

Weight

186

234

208

237

265

190

220

Calculate the least-squares regression line for these data using SPSS.

Solution Enter height in column 1 and weight in column 2. Then Analyze > Regression > Linear. . . > move var00002 to dependent:, and var00001 to Independent(s): > click OK

We obtain the following output: Regression: Variables Entered/Removed Model 1

Variables Entered VAR00001

Variables Removed .

Method Enter

a All requested variables entered. b Dependent Variable: VAR00002 Model Summary: Model R R Square Adjusted R Square Std. Error of the Estimate 1 .947 .897 .876 9.86006 a Predictors: (Constant), VAR00001 ANOVA: Model Sum of Squares df Mean Square F Sig. 1 Regression 4223.896 1 4223.896 43.446 .001 Residual 486.104 5 97.221 Total 4710.000 6 a Predictors: (Constant), VAR00001 b Dependent Variable: VAR00002

458 CHAPTER 8 Linear Regression Models

Coefficients: Unstandardized Coefficients Model B Std. Error 1 (Constant) −188.476 62.083 VAR00001 5.208 .790

Standardized Coefficients Beta .947

t

Sig.

−3.036 .029 6.591 .001

a Dependent Variable: VAR00002 Looking at the coefficients, we see that βˆ 0 = −188.476 and βˆ 1 = 5.208. Hence, the regression line is given by yˆ = −188.476 + 5.208x. Because the coefficient of determination r 2 is 0.897, and the p-value is small, the model fit looks pretty good.

8.9.3 SAS Examples For regression analysis, we can use the SAS procs called GLM, which stands for General Linear Model, and REG, which stands for regression. In the following example we will give a simpliﬁed version of the foregoing procedure. A good explanation of regression methods including diagnostics using SAS can be obtained at http://www.ats.ucla.edu/stat/sas/webbooks/reg/.

Example 8.9.3 Using the SAS commands, redo Example 8.9.1.

Solution We can use the following commands. options nodate nonumber; data exreg; INPUT x y @@; datalines; –1 –5 0 –4 2 2 –2 –7 5 6 6 9 8 13 11 21 12 20 –3 –9 ; proc reg data=exreg; title ‘Regression of Y on X’; model y=x / p clm; run;

8.9 Computer Examples 459

We obtain the following output.

Regression of Y on X The REG Procedure Model: MODEL1 Dependent Variable: y Analysis of Variance

Source

DF

Sum of Squares

Mean Square

Model 1 1082.58589 Error 8 7.81411 Corrected Total 9 1090.40000

Root MSE Dependent Mean Coeff Var

F Value

1082.58589 0.97676

1108.34

0.98831 R-Square 4.60000 Adj R-Sq 21.48508

Pr > F

1, otherwise a2 < 0; both are not admissible because a is a fraction. Hence, a=

σ1 σ1 + σ2

and

1−a=

σ2 . σ1 + σ2

Using the second derivative test, we can verify that this indeed is a minimum for var(X1 − X2 ). From this analysis we can see that the sample sizes that maximize the information in the data relevant to the parameter μ1 − μ2 subject to the constraint n1 + n2 = n are n1 =

σ1 n σ1 + σ2

and

n2 =

σ2 n. σ1 + σ2

9.5 The Taguchi Methods 489

As a special case, we can see that when σ12 = σ22 , the optimal design is to take n1 = n2 .

EXERCISES 9.4 9.4.1.

A total of 100 sample points were taken from two populations with variances σ12 = 4 and σ22 = 9. Find n1 and n2 that will result in the maximum amount of information about (μ1 − μ2 ).

9.4.2.

Suppose in Exercise 9.4.1 we want to take n = n1 = n2 . How large should n be to obtain the same information as that implied by the solution of Exercise 9.4.1?

9.5 THE TAGUCHI METHODS Taguchi methods were developed by Genichi Taguchi to improve the implementation of total quality control in Japan. These methods are claimed to have provided as much as 80% of Japanese quality gains. They are based on the design of experiments to provide near-optimal quality characteristics for a speciﬁc objective. A special feature of Taguchi methods is that they integrate the methods of statistical design of experiments into a powerful engineering process. The Taguchi methods are in general simpler to implement. Taguchi methods are often applied on the Japanese manufacturing ﬂoor by technicians to improve their processes and their product. The goal is not just to optimize an arbitrary objective function, but also to reduce the sensitivity of engineering designs to uncontrollable factors or noise. The objective function used is the signal-to-noise ratio, which is then maximized. This moves design targets toward the middle of the design space so that external variation affects the behavior of the design as little as possible. This permits large reductions in both part and assembly tolerances, which are major drivers of manufacturing cost. Linking quality characteristics to cost through the Taguchi loss function (Taguchi and Yokoyama, 1994) was a major advance in quality engineering, as well as in the ability to design for cost. Taguchi methods are also called robust design. In 1982, the American Supplier Institute introduced Dr. Taguchi and his methods to the U.S. market. Using a well-planned experimental design, such as a fractional factorial design, it is possible to efﬁciently obtain information about the model and the underlying process. Clearly, the purpose of these methods is to control and ensure the quality of the end product. In the conventional approach, this is achieved by further testing a few end products that are randomly chosen or using control charts and making decisions based on certain preset criteria, such as acceptable or unacceptable. Thus, “quality” of the product is thought of as inside or outside of speciﬁcations. Instead, Taguchi suggested that we should specify a target value, and the quality should be thought of as the variation from the target. As an example, suppose we make n observations of the output x1 , . . . , xn of a process at times 1, 2, . . . , n, as shown in Figure 9.3. The control chart consists of a plot of observed output values (xi ’s) on the y-axis and the times of observation, 1, 2, . . . , n on the x-axis, as shown in the ﬁgure. The letter T represents the target value. If

490 CHAPTER 9 Design of Experiments

. TU x

.

T ⫽ Target value

.

1

TL

n

2 Time

■ FIGURE 9.3 Control plot of processing times and outputs.

L

L TL

T

x1

x2

TU

x3

■ FIGURE 9.4 Loss function.

the output value is between TL and TU , the process is deemed to be operating satisfactorily; otherwise the process is said to be out of control and the output value is considered unsatisfactory. Some other examples are (1) deﬁning speciﬁcation limits for acceptance, such as stating that the diameter of bolts must be between 9.8 mm and 10.2 mm with mean 10 mm, and (2) that the waiting time in a line should be less than 30 minutes for at least 90% of customers. In all these situations, the speciﬁcations partition the state of the process as acceptable or unacceptable, that is, it classiﬁes the state as a dichotomy. This is often called the “goal post mentality.” The basic idea of the Taguchi approach is a shift in mindset from demarking the quality as acceptable or unacceptable to a more ﬂexible and realistic classiﬁcation. The traditional approach to quality control does not take into account the size of departure from the target value. To accommodate the size of such departure as a signiﬁcant factor in quality control, let us introduce the concept of loss function (see Chapter 11). If an output value x differs from the target value T , let L(T, x) denote the loss incurred, say in dollars. Other possible losses could also be reputation or customer satisfaction. For the control chart example, we can assign the loss function L(T, x) =

0, L,

if TU < x < TL if x > TL or x < TU

where L is a constant and x is the measured value. This is schematically shown in Figure 9.4. From Figure 9.4, it is seen that we view outputs x1 and x2 as having equal quality, whereas x2 and x3 are considered to have vastly differing quality (x2 is acceptable and x3 is not acceptable). A more

9.5 The Taguchi Methods 491

L(T, x )

L TL

x

T

TU

■ FIGURE 9.5 Quadratic loss function.

reasonable conclusion would be that x1 has excellent quality, whereas x2 and x3 are similar, both being poor. In Taguchi’s approach, the loss function takes into account the size of departure from the target value. For example, a popular choice for the loss function is L(T, X) = k(X − T )2 ,

where L = loss incurred, k = constant, X = actual value of the measured output, and T = target value.

We can schematically represent the behavior as shown by Figure 9.5. This form of loss function is called the quadratic loss function. The choice of k depends on the particular problem. For example, the scaling factor k can be used to convert loss into monetary units to accommodate comparisons of systems with different capital loss. Or, in product manufacturing, let D denote the allowed deviation from the target, and let A denote the loss due to a defective product. Then a choice of k can be k = (A/D)2 . As shown earlier, the average loss is E(L) and is given by E(L) = k[(E(X) − T )2 + σ 2 ] = k[(bias)2 + variance]

where σ 2 is the variance of X (measured quality, which is assumed to be random). In Taguchi, the variation from the target can be broken into components containing bias and product variation. Thus, if our aim is to minimize the expected loss, E(L), we should not only require E(X) = μ to be close to T but also should reduce the variance. It turns out that often these requirements are contradictory. The objective is to choose the design parameters (the factors that inﬂuence the quality) optimally to obtain the best quality product. In practice, the parameters μ and σ 2 are not known and are being estimated by X and S 2, respectively. This results in the Taguchi loss function L = k[(X − T )2 + S 2 ].

This loss function penalizes small deviations from T only slightly, while assessing a larger penalty for responses far from the target. The expected loss is similar to a mean squared error loss, which we have seen in regression analysis in the form of least squares.

492 CHAPTER 9 Design of Experiments

Why is controlling both bias and variance important? Suppose you want your community swimming pool temperature at 80◦ F, which is the T here. Suppose the temperature varies between 60◦ F and 100◦ F. Clearly the average (bias) is zero; however, it will be pretty uncomfortable to swim at 60◦ F or 100◦ F. Here the bias takes the ideal value of zero, but the variance is large. In another scenario, the variance may be small, but the average temperature may be farther away from the target value of 80◦ F (for example, the temperature is constant at 60◦ F). Hence, we want the pool temperature to be near to the target value of 80◦ F, with as small variance as possible (say, within 1◦ F to 2◦ F). Taguchi coined the term design parameters as the generic description for factors that may inﬂuence the quality and whose levels we want to optimize. Taguchi’s philosophy is to “design quality in” rather than to weed out the defective items after manufacturing. In order to obtain an optimal set of design parameters that affect the quality of the end product, the Taguchi method utilizes appropriately designed experiments. More speciﬁcally, orthogonal arrays are used for fractional factorial designs. Taguchi provides tables for these designs so that even a nonspecialist can use them. For two-level designs (high, low), we have a table for an L4 orthogonal array up to three factors; a table for an L8 orthogonal array up to seven factors; and so forth. Similar tables are available for three-level designs. We will not describe these design issues in this section. We refer the reader to specialized books on the subject for further details. We can summarize the Taguchi approach to quality design as follows: 1. Taguchi’s methods for experimental design are ready made and simple to use in the design of efﬁcient experiments, even by nonexperts. 2. Taguchi’s approach to total quality management is holistic and tries to design quality into a product rather than inspecting defects in the ﬁnal product. 3. Taguchi’s techniques can readily be applied to other ﬁelds such as management problems.

EXERCISES 9.5 9.5.1.

Suppose the following data represent thickness between and within silicon wafers (in microns), with a target value of 14.5 microns. 13.688 13.925

13.788 14.545

14.173 13.797

14.557 14.778

Compute the Taguchi loss function. 9.5.2.

One of the commonly used performance measures in the Taguchi method is log

(mean)2 s2

,

where s2 is the sample variance. In general, the higher the performance measure, the better the design. This measure is called robustness statistics. For the problem of Exercise 9.5.1, suppose that we run the experiment by controlling various factors affecting the thickness. Table 9.5.1 shows the data obtained in four different runs.

9.6 Chapter Summary 493

Table 9.5.1 Run 1: 14.158 14.754 14.412 14.065 13.802 14.424 14.898 14.187 Run 2: 13.676 14.177 14.201 14.557 13.827 14.514 13.897 14.278 Run 3: 13.868 13.898 14.773 13.597 13.628 14.655 14.597 14.978 Run 4: 13.668 13.788 14.173 14.557 13.925 14.545 13.797 14.778

(a) Using the robustness statistics given earlier, which of the processes gives us an improved performance? (b) Another commonly used performance statistic is − log(s2 ).

Using this robustness statistic, which of the processes gives us an improved performance? Compare this with the results of part (a).

9.6 CHAPTER SUMMARY In this chapter, we have learned some basic aspects of experimental design. Some fundamental deﬁnitions and tools for developing experimental designs such as randomization, replication, and blocking were introduced in Section 9.2. Basic concepts of factorial design were given in Section 9.3. In Section 9.4, we saw an example of optimal design. The Taguchi method was introduced in Section 9.5. In the next chapter, we introduce the analysis component. We have discussed only a very small collection of experimental designs in this chapter. There exist a wide variety of experimental designs to deal with a large number of treatments and to suit speciﬁc needs of research experiments in diverse ﬁelds. It is an exciting and growing area for the interested student to apply and explore. We list some of the key deﬁnitions introduced in this chapter: ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■ ■

Response variable (output variable) Independent variables (treatment variables or input variables or factors) Nuisance variables Noise Observational Experimental units Single-factor experiments Multifactor experiments Experimental error Blinding, double-blinding, and placebo Replication Block Randomization Completely randomized design Randomized complete block design k × k Latin square design

494 CHAPTER 9 Design of Experiments

■ ■

Greco-Latin square design parameters

In this chapter, we have also learned the following important concepts and procedures. ■ ■ ■ ■ ■ ■ ■ ■ ■

Procedure for random assignment Procedure for randomization in a randomized complete block design Procedure for a randomized complete block design with r replications Procedure for constructing a 4 × 4 Latin square One-factor-at-a-time design Full factorial design Fractional factorial design Choice of optimal sample size The Taguchi methods

9.7 COMPUTER EXAMPLES In this chapter, we present Minitab and SAS commands only. SPSS commands can be performed similarly to Minitab.

9.7.1 Minitab Examples Example 9.7.1 Obtain a random permutation of numbers 1 to n.

Solution Enter in C1 the numbers 1 to n, say n = 10. Then Calc > random data > samples from column. . . > enter sample 10 > rows from column(s) C1 > Store samples in: C2 > OK The result is a random permutation of numbers 1 to n(= 10). One such permutation is given by 8 5 9 7 10 6 4 3 2 1 Now if we need to generate blocks of random permutations of numbers 1 to n(= 10), in the foregoing steps, just store samples in C3, C4, . . . .

9.7.2 SAS Examples Example 9.7.2 For the data of Example 9.2.4, conduct a randomized complete block design using SAS.

9.7 Computer Examples 495

Solution We represent blocks that are reasons for pain by H = 1, M = 2, and CB = 3. Similarly five brands which are treatments by A = 1, B = 2, C = 3, D = 4, and E = 5. Then we can use the following code to generate a randomized complete block design.

options nodate nonumber; data a; do block = 1 to 3 ; do subject = 1 to 5; x = ranuni(0); output; end; end ; proc sort; by block x; data c; set a; trt = 1 + mod(N − 1, 5); /* mod = remainder of N/5 */ proc sort; by block subject; proc print; var block subject trt; run;

We get the following output. Completely randomized 2 × 3 design, 4 subjects per cell Obs 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

block 1 1 1 1 1 2 2 2 2 2 3 3 3 3 3

subject 1 2 3 4 5 1 2 3 4 5 1 2 3 4 5

trt 5 4 3 2 1 2 5 3 4 1 4 5 1 2 3

496 CHAPTER 9 Design of Experiments

Note that the numbers in the column corresponding to a block identify the type of pain, the numbers in the subject column correspond to the subjects, and the numbers in the column corresponding to trt identify the brands. Using the corresponding letters, we can rewrite the foregoing table in the familiar form shown in Table 9.14.

Table 9.14 H

M

CB

1(E)

1(B)

1(D)

2(D)

2(E)

2(E)

3(C)

3(C)

3(A)

4(B)

4(D)

4(B)

5(A)

4(A)

5(C)

The PLAN procedure constructs experimental designs. The PLAN procedure does not have a DATA= option in the PROC statement; in this procedure, both the input and output data sets are speciﬁed in the OUTPUT statement. We will use this to construct a Latin square design.

Example 9.7.3 A gasoline company is interested in comparing the effect of four gasoline additives (A, B, C, D) on the gas mileage achieved per gallon. Four cars (1, 2, 3, 4) and four drivers (I, II, III, IV) will be used in the experiment. Create a Latin square design.

Solution We can use the following program, where we represent the additives by 1 = A, 2 = B, 3 = C, and 4 = D. Options nodate nonumber; title ’Latin Square design for 4 additives’; proc plan seed=37432; factors rows=4 ordered cols=4 ordered/NOPRINT; treatments tmts=4 cyclic; output out=g rows cvals=(’car 1’ ’car 2’ ’car 3’ ’car 4’) random cols cvals=(’Driver 1’ ’Driver 2’ ’Driver 3’ ’Driver 4’) random tmts nvals=(1 2 3 4) random; run; proc tabulate; class rows cols;

Projects for Chapter 9 497

var tmts; table rows, cols*(tmts*f=1.); keylabel sum=’ ’; run;

PROJECTS FOR CHAPTER 9 9A. Sample Size and Power Suppose that the experimenter is interested in comparing the true means of two independent populations. If two similar treatments are to be compared, the assumption of equality of variances is not unreasonable. Hence, assume that the common variance of the two populations is σ 2 , and the experimenter has a prior estimate of the variance. We learned in Section 9.4 that in this case, the optimal design will be to take sample sizes n1 and n2 to be equal. Let n = n1 = n2 be the size of the random sample that the experimenter should take from each population. Now, suppose that the experimenter has decided to use the one-sided large sample test, H0 : μ1 = μ2 vs. Ha : μ1 > μ2 with a ﬁxed α = P(Type I error). He wants to choose n to be so large that if μ1 = μ2 + kσ, he will get a ﬁxed power (1 − β) of deciding μ1 > μ2 . Recall that power of a test is the probability of (correctly) rejecting H0 when H0 is false. Find the approximate value of n. Note that, for a given α, this will be an optimal sample size with a desired value of the power. In particular, what should be the sample size in the hypothesis testing problem, H0 : μ1 − μ2 = 0 vs. Ha : μ1 − μ2 = 3, if α = β = 0.05. Assume that σ = 7.

9B. Effect of Temperature on Spoilage of Milk Suppose you have observed that milk in your refrigerator spoils very fast. You may be wondering whether it has anything to do with the temperature settings. Design an experiment to study the effect of temperature on spoiled milk, with at least three meaningful settings of the temperature. (i) Write a possible hypothesis for your experiment. (ii) What are the independent and dependent variables? (iii) Which variables are being controlled in this experiment? (iv) Discuss how you used the three basic principles of replication, blocking, and randomization. (v) What conclusions can you make? Think through any possible ﬂaws in the design that may affect the integrity of your ﬁndings.

This page intentionally left blank

Chapter

10

Analysis of Variance Objective: To analyze the means of several populations by identifying sources of variability of the data. 10.1 Introduction 500 10.2 Analysis of Variance Method for Two Treatments (Optional) 501 10.3 Analysis of Variance for Completely Randomized Design 510 10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 10.5 Multiple Comparisons 536 10.6 Chapter Summary 543 10.7 Computer Examples 543 Projects for Chapter 10 554

526

John Wilder Tukey (Source: http://en.wikipedia.org/wiki/John_Tukey)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

499

500 CHAPTER 10 Analysis of Variance

John W. Tukey (1915–2000), a chemist-turned-topologist-turned statistician, was one of the most inﬂuential statisticians of the past 50 years. He is credited with inventing the word software. He worked as a professor at Princeton University and a senior researcher at AT&T’s Bell Laboratories. He made signiﬁcant contributions to the ﬁelds of exploratory data analysis and robust estimation. His works on the spectrum analysis of time series and other aspects of digital signal processing have been widely used in engineering and science. He coined the word bit, which refers to a unit of information processed by a computer. In collaboration with Cooley, in 1965, Tukey introduced the fast Fourier transform (FFT) algorithm that greatly simpliﬁed computation for Fourier series and integrals. Tukey authored or coauthored many books in statistics and wrote more than 500 technical papers. Among Tukey’s most far-reaching contributions was his development of techniques for “robust analysis,” an approach to statistics that guards against wrong answers in situations where a randomly chosen sample of data happens to poorly represent the rest of the data set. Tukey also made signiﬁcant contributions to the analysis of variance.

10.1 INTRODUCTION Suppose that we are interested in the effect of four different types of chemical fertilizers on the yield of rice, measured in pounds per acre. If there is no difference between the different types of fertilizers, then we would expect all the mean yields to be approximately equal. Otherwise, we would expect the mean yields to differ. The different types of fertilizers are called treatments and their effects are the treatment effects. The yield is called the response. Typically we have a model with a response variable that is possibly affected by one or more treatments. The study of these types of models falls under the purview of design of experiments, which we discussed in Chapter 9. In this chapter we concentrate on the analysis aspect of the data obtained from the designed experiments. If the data came from one or two populations, we could use the techniques learned in Chapters 6 and 7. Here, we introduce some tests that are used to analyze the data from more than two populations. These tests are used to deal with treatment effects, including tests that take into account other factors that may affect the response. The hypothesis that the population means are equal is considered equivalent to the hypothesis that there is no difference in treatment effects. The analytical method we will use in such problems is called the analysis of variance (ANOVA). Initial development of this method could be credited to Sir Ronald A. Fisher who introduced this technique for the analysis of agricultural ﬁeld experiments. The “green revolution” in agriculture would have been impossible without the development of theory of experimental design and the methods of analysis of variance. Analysis of variance is one of the most ﬂexible and practical techniques for comparing several means. It is important to observe that analysis of variance is not about analyzing the population variance. In fact, we are analyzing treatment means by identifying sources of variability of the data. In its simplest form, analysis of variance can be considered as an extension of the test of hypothesis for the equality of two means that we learned in Chapter 7. Actually, the so-called one-way analysis of variance is a generalization of the two-means procedure to a test of equality of the means of more than two independent, normally distributed populations.

10.2 Analysis of Variance Method for Two Treatments (Optional) 501

Recall that the methods of testing H0 : μ1 − μ2 = 0, such as the t-test, were discussed earlier. In this chapter, we are concerned with studying situations involving the comparison of more than two population or treatment means. For example, we may be interested in the question “Do the rates of heart attack and stroke differ for three different groups of people with high cholesterol levels (borderline high such as 150–199 mg/dL, high such as 200–239 mg/dL, very high such as greater than 240 mg/dL) and a control group given different dosage levels of a particular cholesterol-lowering drug (say, a particular statin drug)?” Let us consider four populations with means μ1 , μ2 , μ3 , and μ4 , and say that we wish to test the hypotheses μ1 = μ2 = μ3 = μ4 . That is, the mean rate is the same for all the four groups. The question here is: Why do we need a new method to test for differences among the four procedure population means? Why not use z- or t-tests for all possible pairs and test for differences in each pair? If any one of these tests leads to the rejection of the hypothesis of equal means, then we might conclude that at least two of the four population means differ. The problem with this approach is that our ﬁnal decision is based on results of 42 = 6 different tests, and any one of them can be wrong. For each of the six tests, let α = 0.10 be the probability of being wrong (type I error). Then the probability that at least one of the six tests leads to the conclusion that there is a difference leads to an error 1 − (0.9)6 = 0.46856, which clearly is much larger than 0.10, thus resulting in a large increase in the type I error rate. Hence, if an ordinary t-test is used to make several treatment comparisons from the same data, the actual α-value applying to the tests taken as a group will be larger than the speciﬁed value of α, and one is likely to declare signiﬁcance when there is none. Analysis of variance procedures were developed to eliminate the increase in error rates resulting from multiple t-tests. With ANOVA, we are able to set one alpha level and test whether any of the group means differ from one another. Given a sample from each of the populations, our interest is to answer the question: Are the observed discrepancies among the different sample means merely due to chance ﬂuctuations, or are they due to inherent differences among the populations? Analysis of variance separates the effect of purely random variations from those caused by existing differences among population means: The phrase “analysis of variance” springs from the idea of analyzing variability in the data to see how much can be attributed to differences in μ and how much is due to variability in the individual populations. The ANOVA method incorporates information on variability from all of the samples simultaneously. At the heart of ANOVA is the fact that variances can be partitioned, with each partition attributable to a speciﬁc source. The method inspects various sums of squares (which are measures of variation in a sample) calculated from the data. ANOVA looks at two types of sums of squares: sums of squares within groups and sums of squares between groups. That is, it looks at each of the distributions and compares the between-group differences (variation in group means) with the within-group differences (variation in individuals’ scores within groups).

10.2 ANALYSIS OF VARIANCE METHOD FOR TWO TREATMENTS (OPTIONAL) In this section, we present the simplest form of the analysis of variance procedure, the case of studying the means of two populations I and II. For comparing only two means, the ANOVA will result in the same conclusions as the t-test for independent random samples. The basic purpose of this section is to introduce the concept of ANOVA in simpler terms. Let us consider two random samples of size n1 and

502 CHAPTER 10 Analysis of Variance

n2 , respectively. That is, y11 , y12 , . . . , y1n1 from population I and y21 , y22 , . . . , y2n2 from population II. Let y1 =

y11 + y12 + · · · + y1n1 (sample mean from population I) n1

and y2 =

y21 + y22 + · · · + y2n2 (sample mean from population II). n2

These samples are assumed to be independent and come from normal populations with respective means μ1 , μ2 , and variances σ12 = σ22 . We wish to test the hypothesis H0 : μ1 = μ2 vs. Ha : μ1 = μ2 .

The total variation of the two combined response measurements about y (the sample mean of all n = n1 + n2 observations) is (SS is used for sum of squares) deﬁned by Total SS =

ni 2

2 yij − y .

(10.1)

i=1 j=1

That is, y=

y11 + y12 + · · · + y1n1 + y21 + y22 + · · · + y2n2 . n

The total sums of squares measures the total spread of scores around the grand mean, y. We can rewrite (10.1) as

Total SS =

ni 2

yij − y

2

i=1 j=1

=

n1

y1j − y

2

+

j=1

=

n1

n2

y2j − y

j=1

y1j − y1 + y1 − y

j=1

=

2

+

n2

y2j − y2 + y2 − y

2

j=1

n1

y1j − y1

2

n1 2 y1j − y1 + n1 y 1 − y + 2 y 1 − y

j=1

+

2

n2 j=1

j=1

y2j − y2

2

n2 2 + n2 y 2 − y + 2 y 2 − y y2j − y2 . j=1

10.2 Analysis of Variance Method for Two Treatments (Optional) 503

Note that

n1 j=1

n2 y1j − y1 = 0 = y2j − y2 . We obtain j=1

Total SS =

n1

y1j − y1

2

+

j=1

y2j − y2

2

j=1

+ n1 y 1 − y =

n2

2

2 + n2 y 2 − y

ni 2

yij − yi

2

i=1 j=1

+

2

2 ni y i − y .

(10.2)

i=1

Deﬁne SST, the sum of squares for treatment by 2

SST =

ni (yi − y)2 .

i=1

The SST measures the total spread of the group means yi with respect to the grand mean, y. Also, SSE represents the sum of squares of errors given by SSE =

ni 2

yij − yi

2

i=1 j=1

=

n1

y1j − y1

j=1

2

+

n2

y2j − y2

2

j=1

= (n1 − 1)s12 + (n2 − 1)s22

where s12 and s22 are the unbiased sample variances of the two random samples. Note that this connects the sum of squares to the concept of variance we have been using in previous chapters. We can now rewrite (10.2) as Total SS = SSE + SST.

It should be clear that the SSE measures the within-sample variation of the y-values (effects), whereas SST measures the variation among the two sample means. The logic by which the analysis of variance tests is as follows: If the null hypothesis is true, then SST as compared to SSE should be about the same, or less. The larger SST, the greater will be the weight of evidence to indicate a difference in the means μ1 and μ2 . The question then is, how large? To answer this question, let us suppose we have two populations that are normal. That is, let Yij be N μi , σ 2 distributed with values yij . Then the pooled unbiased estimate of σ 2 is given by 2 = sp

(n1 − 1) s12 + (n2 − 1) s22 SSE = . n1 + n 2 − 2 n1 + n 2 − 2

504 CHAPTER 10 Analysis of Variance

Hence,

2 =E σ 2 = E sp

SSE . n1 + n 2 − 2

Also, we can write 2 2 n1 n2 Y1j − Y1 Y2j − Y2 SSE = + σ2 σ2 σ2 j=1

j=1

which has a χ2 -distribution with (n1 + n2 − 2) degrees of freedom. Under the hypothesis that μ1 = μ2 , E (SST ) = σ 2 . Furthermore, Y1 − Y2 Z= 8 ∼ N (0, 1) . σ 2 n11 + n12

This implies that

Z2 =

1 1 + n1 n2

%

Y1 − Y2 σ2

& =

SST σ2

has a χ2 −distribution with 1 degree of freedom. It can be shown that SST and SSE are independent. From Chapter 4, we restate the following result. Theorem 10.2.1 If χ12 has υ1 degrees of freedom χ22 has υ2 degrees of freedom, and χ12 and χ22 are indeχ 2 /υ pendent, then F = χ12 υ1 has an F -distribution with υ1 numerator degrees of freedom and υ2 denominator 2/ 2 degrees of freedom. Using the foregoing result, we have > SST (1) σ 2 SST /1 = > SSE/(n1 + n2 − 2) SSE (n1 + n2 − 2) σ 2

which has an F -distribution with υ1 = 1 numerator degrees of freedom and υ2 = (n1 + n2 − 2) denominator degrees of freedom. Now, we introduce the mean square error (MSE), deﬁned by MSE = =

SSE (n1 + n2 − 2) (n1 − 1) s12 + (n2 − 1) s22 (n1 + n2 − 2)

10.2 Analysis of Variance Method for Two Treatments (Optional) 505

and the mean square treatment (MST) given by SST 1 2 2 . = n1 y1 − y + n2 y2 − y

MST =

Under the null hypothesis, H0 : μ1 = μ2 , both MST and MSE estimate σ 2 without bias. When H0 is false and μ1 = μ2 , MST estimates something larger than σ 2 and will be larger than MSE. That is, if H0 is false, then E(MST ) > E(MSE) and the greater the differences among the values of μ, the larger E(MST ) will be relative to E(MSE). Hence, to test H0 : μ1 = μ2 vs. Ha : μ1 = μ2 , we use the F -test given by F=

MST MSE

as the test statistic. Thus, for given α, the rejection region is {F > Fα }. It is important to observe that compared to the small sample t-test, here we work with variability. Now we summarize the analysis of variance procedure for the two-sample case.

ANALYSIS OF VARIANCE PROCEDURE FOR TWO TREATMENTS For equal sample sizes n = n1 = n2 , assume σ12 = σ22 . We test H0 : μ1 = μ2 vs. Ha : μ1 = μ2 . 1. Calculate: y1 , y2 ,

2 yij , yij , and ﬁnd ij

ij

SST =

2

2 ni y i − y .

i=1

Also calculate Total SS =

i

j

yij2 −

i

2 yij

j

n1 + n2

Then SSE = Total SS − SST .

.

506 CHAPTER 10 Analysis of Variance

2. Compute MST =

SST 1

MSE =

SSE . n1 + n2 − 2

3. Compute the test statistic, F=

MST . MSE

4. For a given α, ﬁnd the rejection region as RR : F > Fα , based on 1 numerator and (n1 + n2 − 2) denominator degrees of freedom. 5. Conclusion: If the test statistic F falls in the rejection region, conclude that the sample evidence supports the alternative hypothesis that the means are indeed different for the two treatments. Assumptions: Populations are normal with equal but unknown variances.

Example 10.2.1 The following data represent a random sample of end-of-year bonuses for lower-level managerial personnel employed by a large ﬁrm. Bonuses are expressed in percentage of yearly salary. Female 6.2 9.2 8.0 7.7 8.4 9.1 7.4 6.7 Male 8.9 10.0 9.4 8.8 12.0 9.9 11.7 9.8 The objective is to determine whether the male and female bonuses are the same. We can answer this question by connecting the following. (a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.05. (b) What assumptions are necessary for the test in part (a)? (c) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a).

Solution (a) We need to test H0 : μ1 = μ2 vs. Ha : μ1 = μ2 From the random sample, we obtain the following needed estimates, n1 = n2 = 8: y1 = 7.8375, y2 = 10.0625,

2 = 1319.34, yij

ij

SST =

2 i=1

ni (yi − y2 )2 = 19.8025.

ij

yij = 143.20

10.2 Analysis of Variance Method for Two Treatments (Optional) 507

Therefore, Total SS =

i

2 yij

2 − yij

i

j

= 1391.34 −

2

j

2n

(143.2)2 = 109.70. 16

Then SSE = Total SS − SST = 109.7 − 19.8025 = 89.8975, MST =

SST = 19.8025 1

and MSE =

89.8975 SSE = 2n1 − 2 14

= 6.42125. Hence, the test statistic F=

19.8025 MST = MSE 6.42125

= 3.0839. For α = 0.05, F0.05,14 = 4.60. Hence the rejection region is {F > 4.60}. Because 3.0839 is not greater than 4.60, H0 is not rejected. There is not enough evidence to indicate that the average bonuses are different for men and women at α = 0.05. (b) To solve the problem, we assumed that the samples are random and independent with n1 = n2 = 8, drawn from two normal populations with means μ1 and μ2 and common variance σ 2 . 2 = 6.42125. Also, y = 7.8375 and y = 10.0625. Then, (c) The value of MSE is the same as s2 = sp 1 2 the t-statistic is 7.8375 − 10.0625 y1 − y2 t= 8 = 8 = −1.756. 1 1 1 1 2 s n1 + n2 6.42125 8 + 8 Now, t0.025,14 = 2.415 and the rejection region is {t < −2.145}. Because −1.756 is not less than −2.45, H0 is not rejected, which implies that there is no significant difference between the bonuses for the males and the females. Note also that t 2 = F , that is, (−1.756)2 = 3.083 implying that in the two-sample case, the t-test and F -test lead to the same result.

508 CHAPTER 10 Analysis of Variance

It is not surprising that in the previous example, the conclusions reached using ANOVA and two sample t-tests are the same. In fact, it can be shown that for two sets of independent and normally distributed random variables, the two procedures are entirely equivalent for a two-sided hypothesis. However, a t-test can also be applied to a one-sided hypothesis, whereas ANOVA cannot. The purpose of this section is only to illustrate the computations involved in the analysis of variance procedures as opposed to simple t-tests. The analysis of variance procedure is effectively used for three or more populations, which is described in the next section.

EXERCISES 10.2 10.2.1.

The following information was obtained from two independent samples selected from two normally distributed populations with unknown but equal standard deviations. Do the data present sufﬁcient evidence to indicate that there is a difference in the mean for the two populations? Sample 1 Sample 2

1 2

2 5

3 2

3 4

1 3

2 1

1 2

3 3

1 3

(a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.05. (b) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a). 10.2.2.

The following information was obtained from two independent samples selected from two normally distributed populations with unknown but equal standard deviations. Do the data present sufﬁcient evidence to indicate that there is a difference in the mean for the two populations? Sample 1: Sample 2:

15 18

13 16

11 13

14 21

10 16

12 19

7 15

12 18

11 19

14 20

15 21

14

(a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.01. (b) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a). 10.2.3.

A company claims that its medicine, brand A, provides faster relief from pain than another company’s medicine, brand B. A random sample from each brand gave the following times (in minutes) for relief. Do the data present sufﬁcient evidence to indicate that there is a difference in the mean time to relief for the two populations? Brand A: Brand B:

47 44

51 48

45 42

53 45

41 44

55 42

50 49

46 46

45 45

51 48

53 39

50 49

48

(a) Use the ANOVA approach to test the appropriate hypotheses. Use α = 0.01. (b) What assumptions are necessary for the conclusion in part (a)?

10.2 Analysis of Variance Method for Two Treatments (Optional) 509

(c) Test the appropriate hypothesis by using the two-sample t-test for comparing population means. Compare the value of the t-statistic to the value of the F -statistic calculated in part (a). 10.2.4.

Table 10.2.1 gives mean SAT scores for math by state for 1989 and 1999 for 20 randomly selected states (source: The World Almanac and Book of Facts 2000).

Table 10.2.1 State

1989

1999

Arizona

523

525

Connecticut

498

509

Alabama

539

555

Indiana

487

498

Kansas

561

576

Oregon

509

525

Nebraska

560

571

New York

496

502

Virginia

507

499

Washington

515

526

Illinois

539

585

North Carolina

469

493

Georgia

475

482

Nevada

512

517

Ohio

520

568

New Hampshire

510

518

Using the ANOVA procedure, test that the mean SAT score for math in 1999 is greater than that in 1989 at α = 0.05. Assume that the variances are equal and the samples come from a normal distribution. 10.2.5.

Let X1 , . . . , Xn1 and Y1 , . . . , Yn2 be two sets of independent, normally distributed random variables with means μ1 and μ2 , and the common variance σ 2 . Show that the two-sample t-test and the analysis of variance are equivalent for testing H0 : μ1 = μ2 versus Ha : μ1 > μ2 .

510 CHAPTER 10 Analysis of Variance

10.3 ANALYSIS OF VARIANCE FOR COMPLETELY RANDOMIZED DESIGN In this section, we study the hypothesis testing problem of comparing population means for more than two independent populations, where the data are about several independent groups (different treatments being applied, or different populations being sampled). We have seen in Chapter 9 that the random selection of independent samples from k populations is known as a completely randomized experimental design or one-way classiﬁcation. Let μ1 , . . . , μk be the means of k normal populations with unknown but equal variance σ 2 . The question is whether the means of these groups are different or are all equal. The idea is to consider the overall variability in the data. We partition the variability into two parts: (1) between-groups variability and (2) within-groups variability. If between groups is much larger than that within groups, this will indicate that differences between the groups are real, not merely due to the random nature of sampling. Let independent samples be drawn of sizes ni , i = 1, 2, . . . , k and let N = n1 + · · · + nk . Let yij be the measured response on the jth experimental unit in the ith sample. That is, Yij is the jth observation from population i, i = 1, 2, . . . , k, and j = 1, 2, . . . , ni . Let y be the overall mean of all observations. The problem can be formulated as a hypothesis testing problem, where we need to test H0 : μ1 = μ2 = . . . = μk vs. Ha : Not all the μi s are equal.

The method of analysis of variance tests the null hypothesis H0 by comparing two unbiased estimates of the variance, σ 2 , an estimate based on variations from sample to sample and the other one based on variations within the samples. We will be rejecting H0 if the ﬁrst estimate is signiﬁcantly larger than the second, so that the samples cannot be assumed to come from the same population. We can write the total sum of squares of deviations of the response measurements about their overall mean for the k samples into two parts, from the treatment (SST) and from the error (SSE). This partition gives the fundamental relationship in ANOVA, where total variation is divided into two portions: between-sample variation and within-sample variation. That is, Total SS = SST + SSE.

The following derivations will make computation of these quantities simpler. The total SS can be written as Total SS =

ni k

yij − y

2

=

i=1 j=1 ni k

Note that y =

i=1 j=1

N

ni k

2 − 2y yij

i=1 j=1

yij

, and then we have Total SS =

ni k i=1 j=1

2 − CM yij

ni k i=1 j=1

yij + Ny2 .

10.3 Analysis of Variance for Completely Randomized Design 511

where CM is the correction factor for the correction for the means and is given by

ni k

2 yij

i=1 j=1

CM =

= Ny2 .

N

Let Ti =

ni

yij , be the sum of all the observations in the ith sample

j=1

and ni

Ti =

yij

j=1

ni

, the mean of the observations in the ith sample.

We can rewrite y as ni k

y=

k

yij

i=1 j=1

=

N

ni T i

i=1

N

Now, we introduce SST, the sum of squares for treatment (sometimes known as between group sum of squares, SSB) by SST =

k

2 ni Ti − y .

i=1

We note that Ti is the mean response due to its ith treatment and y is the overall mean. A large value of Ti − y is likely to be caused by the ith treatment effect being much different from the rest. Hence SST can be used to measure the differences in the treatment effects. Thus, the sum of squares of errors (SSE) is SSE = Total SS − SST.

We must state that the SSE is the sum of squares within groups (thus, sometimes SSE is referred to as within group sum of squares, SSW) and this can be seen from rewriting the expression as SSE =

ni k

2 yij − T i .

i=1 j=1

The decomposition of total sum of squares can be easily seen in Figure 10.1. Figure 10.2 represents one point for each observation against each sample, with SM representing the sample means and GM representing the grand mean. The dotted line between SMs and GM is the

512 CHAPTER 10 Analysis of Variance

Total sum of squares

SST (or between group sum of squares

SSE (or within group sum of squares ni

k

⫽

k

(yij ⫺Ti )2

i ⫽1 j ⫽1

2

ni (Ti ⫺ y )

⫽ i⫽1

Observations

■ FIGURE 10.1 Decomposition of total SS.

GM SM

0

SM

0 0

I

SM

II

III Sample

■ FIGURE 10.2 ANOVA decomposition.

distance between them. Taking this distances, squaring, multiplying by the corresponding sample sizes, and summing, we get SST. To obtain SSE, we take the distance from each group mean, SM, to each member of the group, square them, and add them. In addition, to give an idea of within-group variations, it is customary to draw side-by-side box plots. As mentioned earlier, SST estimates the variation among the μi s, and hence if all the μi s were equal, the Ti s would be similar and the SST would be small. It can be veriﬁed that the unbiased estimator of σ 2 based on (n1 + n2 + · · · + nk − k) degrees of freedom is S 2 = MSE = =

SSE . N −k

SSE (n1 + n2 + · · · + nk − k)

10.3 Analysis of Variance for Completely Randomized Design 513

Note that the quantity MSE is a measure of variability within the groups. If there were only one group with n observations, then the MSE is nothing but the sample variance, s2 . The fact that ANOVA deals simultaneously with all the k groups can be seen by rewriting MSE in the following form: MSE =

(n1 − 1) s12 + (n2 − 1) s22 + · · · + (nk − 1) sk2 (n1 − 1) + (n2 − 1) + · · · + (nk − 1)

.

The mean square for treatments with (k − 1) degrees of freedom is MST =

SST . k−1

The MST is a measure of the variability between the sample means of the groups. We now summarize the analysis of variance hypothesis testing method for two or more populations. ONE-WAY ANALYSIS OF VARIANCE FOR k ≥ 2 POPULATIONS We test H0 : μ1 = μ2 = . . . = μk

versus

Ha : At least two of the μi s are different. When H0 is true, we have E(MST ) = E(MSE ) The greater the differences among the μ s, the larger the E(MST ) will be relative to E(MSE ). Test statistic: MST . F= MSE Rejection region is RR : F > Fα with υ1 = (k − 1) numerator degrees of freedom and υ2 = ki=1 ni − k = N − k denominator degrees of k freedom, where N = i=1 ni . Assumptions: The observations Yij s are assumed to be independent and normally distributed with mean μi , i = 1, 2, . . . , k , and variance σ 2 .

Now we give a ﬁve-step computational procedure that we could follow for analysis of variance for the completely randomized design. ONE-WAY ANALYSIS OF VARIANCE PROCEDURE FOR k ≥ 2 POPULATIONS We test H0 : μ1 = μ2 = . . . = μk versus Ha : At least two of the μi s are different.

514 CHAPTER 10 Analysis of Variance

1. Compute Ti =

CM =

ni

ni k

yij , T =

j=1

i=1 j=1

2

ni k

yij

yij , and

ni k

yij2 ,

i=1 j=1

T2 , where N = ni , N k

i=1 j=1

=

N

i=1

T Ti = i , ni and Total SS =

ni k

yij2 − CM.

i=1 j=1

2. Compute the sum of squares between samples (treatments), SST =

k Ti2 i=1

=

k

ni

− CM

Ti − CM.

i=1

and the sum of squares within samples, SSE = Total SS − SST Let MST =

SST , k −1

MSE =

SSE . n−k

and

3. Compute the test statistic: F=

MST . MSE

4. For a given α, ﬁnd the rejection region as RR : F > Fα

10.3 Analysis of Variance for Completely Randomized Design 515 k with υ1 = k − 1 numerator degrees of freedom and υ2 = i=1 ni − k = N − k denominator degrees of freedom, where N = ki=1 ni . 5. Conclusion: If the test statistic F falls in the rejection region, conclude that the sample evidence supports the alternative hypothesis that the means are indeed different for the k treatments and are not all equal. Assumptions: The samples are randomly selected from the k populations in an independent manner. The populations are assumed to be normally distributed with equal variances σ 2 and means μ1 , . . . , μk .

10.3.1 The p-Value Approach Note that if we are using statistical software packages, the p-value approach can be used for the testing. Just compare the p-value and α to arrive at a conclusion. Refer to the computer examples in Section 10.7. The following example illustrates the ANOVA procedure.

Example 10.3.1 The three random samples in Table 10.1 represent test scores from three classes of statistics taught by three different instructors and are independently obtained. Assume that the three different populations are normal with equal variances. At the α = 0.05 level of signiﬁcance, test for equality of population means.

Table 10.1 Sample 1

Sample 2

Sample 3

64

56

81

84

74

92

75

69

84

77 80

Solution We test H0 : μ1 = μ2 = μ3 versus Ha : At least two of the μ s are different. Here, k = 3, n1 = 5, n2 = 3, and N = n1 + n2 + n3 = 11.

516 CHAPTER 10 Analysis of Variance

Also, Ti ni Ti

380 5 76

199 3 66.33

257 3 85.67

Clearly, the sample means are different. The question we are going to answer is: Is this difference due to just chance, or is it due to a real difference caused by different teaching styles? For this, we now compute the following: 2 yij (836)2 i j CM = = = 63,536 N 11 2 − CM yij Total SS = i

j

= 64,558 − 63,536 = 1022 SST =

T2

i − CM

i

ni

(199)2 (257)2 (380)2 + + − CM = 5 3 3 = 64,096.66 − 63,536 = 560.66 SSE = Total SS − SST = 1022 − 560.66 = 461.34. Hence, MST =

SST 560.66 = = 280.33, k−1 2

MSE =

461.34 SSE = = 57.67. N −k 8

and

The test statistic is F=

280.33 MST = = 4.86. MSE 57.67

From the F -table, F0.05,2,8 = 4.46. Therefore, the rejection region is given by RR : F > 4.46. Decision: Because the observed value of F = 4.86 falls in the rejection region, we do reject H0 and conclude that there is sufficient evidence to indicate a difference in the true means.

10.3 Analysis of Variance for Completely Randomized Design 517

If we want the p-value, we can see from the F -table that 0.025 < p-value < 0.05, indicating the rejection of the null hypothesis with α = 0.05. Using statistical software packages, we can get the exact p-value.

The calculations obtained in analyzing the total sum of squares into its components are usually summarized by the analysis-of-variance table (ANOVA table), given in Table 10.2. Sometimes, one may also add a column for the p-value, P(Fk−1,n−k ≥ observed F ), in the ANOVA table. For the previous example, we can summarize the computations by the ANOVA table shown in Table 10.3.

10.3.2 Testing the Assumptions for One-Way ANOVA The randomness assumption could be tested using the Wald–Wolfowitz test (see Project 12B). The assumption of independence of the samples is hard to test without knowing how the data are collected and should be implemented during collection of data in the design stage. Normality can be tested (this should be performed separately for each sample, not for the total data set) using probability plots or other tests such as the chi-square goodness-of-ﬁt-test. ANOVA is fairly robust against violation of this assumption if the sample sizes are equal. Also, if the sample sizes are fairly large, the central limit theorem helps. The presence of outliers is likely to increase the sample variance, thus decreasing

Table 10.2 Source of variation

Degree of freedom

Treatments

k−1

Sum of squares SST =

k i=1

Error

n−k

Total

n−k

Ti2 ni

Mean squares

− CM

SSE = Total SS − SST Total SS =

ni k

yij − y

Fstatistic

MST =

SST k−1

MSE =

SSE n−k

2

i=1 i=1

Table 10.3 Source of variation

Degree of freedom

Sum of squares

Mean square

F-statistic

p-Value

Treatments

2

560.66

280.33

4.86

0.042

Error

8

461.34

57.67

Total

10

1022

MST MSE

518 CHAPTER 10 Analysis of Variance

the value of the F -statistic for ANOVA, which will result in a lower power of the test. Box plots or probability plots could be used to identify the outliers. If the normality test fails, transforming the data (see Section 14.4.2) or a nonparametric test such as the Kruskal–Wallis test described in Section 12.5.1 may be more appropriate. If the sample sizes of each sample are equal, ANOVA is mostly robust for violation of homogeneity of the variances. A rule of thumb used for robustness for this condition is that the ratio of sample variance of the largest sample variance s2 to the smallest sample variance s2 should be no more than 3 : 1. Another popular rule of thumb used in one-way ANOVA to verify the requirement of equality of variances is that the largest sample standard deviation not be larger than two times the smallest sample standard deviation. Graphically, representing side-byside box plots of the samples can also reveal lack of homogeneity of variances if some box plots are much longer than others (see Figure 10.3e). For a signiﬁcance test on the homogeneity of variances (Levene’s test), refer to Section 14.4.3. If these tests reveal that the variances are different, then the populations are different, in spite of what ANOVA concludes about differences of the means. But this itself is signiﬁcant, because it shows that the treatments had an effect.

Example 10.3.2 In order to study the effect of automobile size on the noise pollution, the following data are randomly chosen from the air pollution data (source: A. Y. Lewin and M. F. Shakun, Policy Sciences: Methodology and Cases, Pergamon Press, 1976, p. 313). The automobiles are categorized as small, medium, large, and noise level reading (decibels) are given in Table 10.4.

Table 10.4 Size of automobile Small

Medium

Large

820

840

785

Noise level

820

825

775

(decibels)

825

815

770

835

855

760

825

840

770

At the α = 0.05 level of signiﬁcance, test for equality of population mean noise levels for different sizes of the automobiles. Comment on the assumptions.

Solution Let μ1 , μ2 , μ3 be population mean noise levels for small, medium, and large automobiles, respectively. First we test for the assumptions. Using Minitab, run tests for each of the samples; we can justify the assumption of randomness of the sample values. A normality test for each column gives the graphs shown in Figures 10.3a through 10.3c, through which we can reasonably assume the normality. Because the sample sizes are equal, we will use the one-way ANOVA method to analyze these data.

10.3 Analysis of Variance for Completely Randomized Design 519

Figure 10.3d indicates that the relative positions of the sample means are different, and Figure 10.3e (Minitab steps for creating side-by-side box plots are given at the end of Example 10.7.1) gives an indication of withingroup variations; perhaps the group 2 (medium-size) variance is larger. Now, we will do the analytic testing. Noise level for small size automobiles 0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 820

825

830

835

Small Average: 825 Std Dev: 6.12372 N: 5

Kolmogorov-Smirnov Normality Test D⫹: 0.200 D⫺: 0.149 D: 0.200 Approximate P -Value ⬎ 0.15

■ FIGURE 10.3(a) Normal plot for noise level of small automobiles.

Noise level for medium size automobiles 0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 815

Average: 835 Std Dev: 15.4110 N: 5

825

835 Medium

845

855

Kolmogorov-Smirnov Normality Test D⫹: 0.142 D ⫺: 0.127 D: 0.142 Approximate P -Value ⬎ 0.15

■ FIGURE 10.3(b) Normal plot for noise level of medium-sized automobiles.

520 CHAPTER 10 Analysis of Variance

Noise level for large size automobiles

0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 760

770

780 Large

Average: 772 Std Dev: 9.08295 N: 5

Kolmogorov-Smirnov Normality Test D⫹: 0.171 D⫺: 0.124 D: 0.171 Approximate P -Value ⬎ 0.15

■ FIGURE 10.3(c) Normal plot for noise level of large automobiles.

840 830

Mean

820 810 800 790 780 770 1

2 Sample

■ FIGURE 10.3(d) Mean decibel levels for three sizes of automobiles.

3

10.3 Analysis of Variance for Completely Randomized Design 521

860 850 840

Decibels

830 820 810 800 790 780 770 760 1

2 Size of auto

3

■ FIGURE 10.3(e) Side-by-side box plots for decibel levels for three sizes of automobiles.

We test H0 : μ1 = μ2 = μ3 versus Ha : At least two of the μ s are different. Here, k = 3, n1 = 5, n2 = 5, n3 = 5 and N = n1 + n2 + n3 = 15. Also Ti ni Ti

4125 5 825

4175 5 835

3860 5 772

In the following calculations, for convenience we will approximate all values to the nearest integer. 2 yij (12,160)2 i j = = 9,857,707 CM = N 15 2 − CM Total SS = yij i

j

= 12,893 SST =

T2

i − CM

i

ni

= 11,463 SSE = Total SS − SST = 1430.

522 CHAPTER 10 Analysis of Variance

Hence, MST =

SST 11,463 = = 5732 k−1 2

and MSE =

SSE 1430 = = 119. N −k 12

The test statistic is F=

MST 5732 = = 48.10. MSE 119

From the table, we get F0.05,2,12 = 3.89. Because the test statistic falls in the rejection region, we reject at α = 0.05 the null hypothesis that the mean noise levels are the same. We conclude that size of the automobile does affect the mean noise level.

It should be noted that the alternative hypothesis Ha in this section covers a wide range of situations, from the case where all but one of the population means are equal to the case where they are all different. Hence, with such an alternative, if the samples lead us to reject the null hypothesis, we are left with a lot of unsettled questions about the means of the k populations. These are called post hoc testing. This problem of multiple comparisons is the topic of Section 10.5.

10.3.3 Model for One-Way ANOVA (Optional) We conclude this section by presenting the classical model for one-way ANOVA. Because the variables Yij values are random samples from normal populations with E(Yij ) = μi and with common variance Var(Yij ) = σ 2 , for i = 1, . . . , k and j = 1, . . . , ni , we can write a model as Yij = μi + εij , j = 1, . . . , ni

where the error terms εij are independent normally distributed random variables with E(εij ) = 0 and Var(εij ) = σ 2 . Let αi = μ − μi be the difference of μi (ith population mean) from the grand mean μ. Then αi can be considered as the ith treatment effect. Note that the αi values are nonrandom. Because μ = i (ni μi /N), it follows that ki=1 αi = 0. This will result in the following classical model for one-way layout: Yij = μ + αi + εij ,

i = 1, . . . , k,

j = 1, . . . , ni .

With this representation, the test H0 : μ1 = μ2 = . . . . = μk reduces to testing the null hypothesis that there is no treatment effect, H0 : αi = 0, for i = 1, . . . , k.

EXERCISES 10.3 10.3.1.

In an effort to investigate the premium charged by insurance companies for auto insurance, an agency randomly selects a few drivers who are insured by one of three different companies. These individuals have similar cars, driving records, and levels of coverage.

10.3 Analysis of Variance for Completely Randomized Design 523

Table 10.3.1 gives the premiums paid per 6 months by these drivers with these three companies.

Table 10.3.1 Company I

Company II

Company III

396

348

378

438

360

330

336

522

294

318

474

432

(a) Construct an analysis-of-variance table and interpret the results. (b) Using the 5% signiﬁcance level, test the null hypothesis that the mean auto insurance premium paid per 6 months by all drivers insured for each of these companies is the same. Assume that the conditions of completely randomized design are met. 10.3.2.

Three classes in elementary statistics are taught by three different persons: a regular faculty member, a graduate teaching assistant, and an adjunct from outside the university. At the end of the semester, each student is given a standardized test. Five students are randomly picked from each of these classes, and their scores are as shown in Table 10.3.2.

Table 10.3.2 Faculty

Teaching assistant

Adjunct

93

88

86

61

90

56

87

76

73

75

82

90

92

58

47

(a) Construct an analysis-of-variance table and interpret your results. (b) Test at the 0.05 level whether there is a difference between the mean scores for the three persons teaching. Assume that the conditions of completely randomized design are met. 10.3.3.

Let n1 = n2 = . . . = nk = n . Show that

k n

yij − y

i=1 j=1

2

=

k n

yij − Ti

i=1 j=1

2

+n

k i=1

2 Ti − y .

524 CHAPTER 10 Analysis of Variance

10.3.4.

For the sum of squares for treatment SST =

k

2 n i Ti − y

i=1

show that E (SST ) = (k − 1) σ 2 +

k

ni (μi − μ)2

i=1

where μ =

1 N

k

ni μi .

i=1

[This exercise shows that the expected value of SST increases as the differences among the μi s increase.] 10.3.5.

(a) Show that SSE = 1 n−1

k i=1

ni

(ni − 1) Si2 =

ni k

2 Yij − Ti ,

i=1 j=1

2

where = provides an independent, unbiased estimator for j=1 Yij − Ti σ 2 in each of the k samples. > (b) Show that SSE σ 2 has a chi-square distribution with N − k degrees of freedom, where N = ki=1 ni . Si2

10.3.6.

Let each observation in a set of k independent random samples be normally distributed with means μ1 , . . . , μk and common variance σ 2 . If H0 = μ1 = μ2 = . . . = μk is true, show that F=

MST SST /(k − 1) = SSE/(n − k) MSE

has an F -distribution with k − 1 numerator and n − k denominator degrees of freedom. 10.3.7.

The management of a grocery store observes various employees for work productivity. Table 10.3.3 gives the number of customers served by each of its four checkout lanes per hour.

Table 10.3.3 Lane 1

Lane 2

Lane 3

Lane 4

16

11

8

21

18

14

12

16

22

10

17

17

21

10

10

23

15

14

13

17

10

15

10.3 Analysis of Variance for Completely Randomized Design 525

(a) Construct an analysis-of-variance table and interpret the results. Indicate any assumptions that were necessary. (b) Test whether there is a difference between the mean number of customers served by the four employees at the 0.05 level. Assume that the conditions of completely randomized design are met. 10.3.8.

Table 10.3.4 represents immunoglobulin levels (with each observation being the IgA immunoglobulin level measured in international units) of children under 10 years of age of a particular group. The children are grouped as follows: A: ages 1 to less than 3, B: ages 3 to less than 6, C: ages 6 to less than 8, and D: ages 8 to less than 10. Test whether there is a difference between the means for each of the age groups. Use α = 0.05. Interpret your results and state any assumptions that were necessary to solve the problem.

Table 10.3.4

10.3.9.

A

35

8

12

19

56

64

75

25

B

31

79

60

45

39

44

45

62

20

C

74

56

77

35

95

81

28

D

80

42

48

69

95

40

86

79

51

66

Table 10.3.5 gives rental and homeowner vacancy rates by U.S. region (source: U.S. Census Bureau) for 5 years.

Table 10.3.5 Rental units

1995

1996

1997

1998

1999

Northeast

7.2

7.4

6.7

6.7

6.3

Midwest

7.2

7.9

8.0

7.9

8.6

South

8.3

8.6

9.1

9.6

10.3

West

7.5

7.2

6.6

6.7

6.2

Test at the 0.01 level whether the true rental and homeowner vacancy rates by area are the same for all 5 years. Interpret your results and state any assumptions that were necessary to perform the analysis. 10.3.10.

Table 10.3.6 gives lower limits of income (approximated to the nearest $1000 and calculated as of March of the following year) of the top 5% of U.S. households by race from 1994 to 1998 (Source: U.S. Census Bureau). Test at the 0.05 level whether the true lower limits of income for the top 5% of U.S. households for each race are the same for all 5 years.

526 CHAPTER 10 Analysis of Variance

Table 10.3.6 Race

10.3.11.

Year 1994

1995

1996

1997

1998

All Races

110

113

120

127

132

White

113

117

123

130

136

Black

81

80

85

87

94

Hispanic

82

80

86

93

98

Table 10.3.7 gives mean serum cholesterol levels (given in milligrams per deciliter) by race and age in the United States between 1978 and 1980 (source: “Report of the National Cholesterol Education Program Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults,” Arch. Intern. Med. 148, January 1988).

Table 10.3.7 Race

Age 20–24

25–34

35–44

45–54

55–64

65–74

All Races

180

199

217

227

229

221

White

180

199

217

227

230

222

Black

171

199

218

229

223

217

Test at the 0.01 level whether the true mean cholesterol levels for all races in the United States between 1978 and 1980 are the same.

10.4 TWO-WAY ANALYSIS OF VARIANCE, RANDOMIZED COMPLETE BLOCK DESIGN A randomized block design, or the two-way analysis of variance, consists of b blocks of k experimental units each. In many cases we may be required to measure response at combinations of levels of two or more factors considered simultaneously. For example, we might be interested in gas mileage per gallon among four different makes of cars for both in-city and highway driving, or to examine weight loss comparing ﬁve different diet programs among whites, African Americans, Hispanics, and Asians according to their gender. In studies involving various factors, the effect of each factor on the response variable may be analyzed using one-way classiﬁcation. However, such an analysis will not be efﬁcient with respect to time, effort, and cost. Also, such a procedure would give no knowledge about the likely

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 527

interactions that may exist among different factors. In such cases, the two-way analysis of variance is an appropriate statistical method to use. In a randomized block design, the treatments are randomly assigned to the units in each block, with each treatment appearing exactly once in every block (that is, there is no interaction between factors). Thus, the total number of observations obtained in a randomized block design is n = bk. The purpose of subdividing experiments into blocks is to eliminate as much variability as possible, that is, to reduce the experimental error or the variability due to extraneous causes. Refer to Section 9.2.3 for a procedure to obtain completely randomized block design. The goal of such an experiment is to test the equality of levels for the treatment effect. Sometimes, it may also be of interest to test for a difference among blocks. We proceed to give a formal statistical model for the completely randomized block design. For i = 1, 2, . . . , k and j = 1, 2, . . . , b, let Yij = μ + αi + βj + εij , where Yij is the observation on treatment i in block j, μ is the overall mean, αi is the nonrandom effect of treatment i, βj is the nonrandom effect of block j, and εij are the random error terms such that εij are independent αi = 0, normally distributed random variables with E εij = 0 and Var εij = σ 2 . In this case, and βj = 0. The analysis of variance for a randomized block design proceeds similarly to that for a completely randomized design, the main difference being that the total sum of squares of deviations of the response measurements from their means may be partitioned into three parts: the sum of squares of blocks (SSB), treatments (SST), and error (SSE). Let Bj = ki=1 yij and Bj denote, respectively, the total sum and mean of all observations in block j. Represent the total for all observations receiving treatment i by Ti = bj=1 yij , and mean and T i , respectively. Let y = average of n = bk observations =

b k 1 yij n j=1 i=1

and 2 1 total of all observations n ⎛ ⎞2 b k 1 ⎝ ⎠ = yij . n

CM =

j=1 i=1

For convenience, we can represent the two-way classiﬁcation as in Table 10.5. Note that from the table we can obtain

k b j=1 i=1

yij =

b j=1

Bj . Hence, CM = (1/n)

b j=1 Bj

2

.

528 CHAPTER 10 Analysis of Variance

Table 10.5 Blocks 1

2

...

j

...

b

Total T i

Mean T i

Treatment 1

y11

y12

...

y1j

...

y1b

T1

T1

Treatment 2

y21

y22

...

y2j

...

y2b

T2

T2

yij

...

yib

Ti

Ti

Tk

Tk

. . . Treatment i

. . .

. . .

yi1

yi2

...

. . .

. . .

Treatment k

yk1

yk2

...

ykj

...

ykb

Total Bj

B1

B2

...

Bj

...

Bb

Mean Bj

B1

B2

...

Bj

...

Bb

y

Then for a randomized block design with b blocks and k treatments, we need to compute the following sums of squares. They are Total SS = SSB + SST + SSE =

b k

yij − y

2

b k

=

j=1 i=1

j=1 i=1 b

SSB = k

b

2 − CM yij

Bj − y

2

=

Bj2

j=1

k

j=1

− CM

and k

SST = b

k

Ti − y

2

=

i=1

i=1

Ti2

b

SSE = Total SS − SSB − SST.

We deﬁne SSB , b−1 SST , MST = k−1

MSB =

− CM

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 529

Table 10.6 Source

d.f.

SS

MS

Blocks

b−1

SSB

SSB b−1

Treatments

k−1

SST

SST k−1

Error

(b − 1)(k − 1) =n−b−k+1

SSE

SSE n−b−k+1

Total

n−1

Total SS

and MSE =

SSE . n−b−k+1

The analysis of variance for the randomized block design is presented in Table 10.6. The column corresponding to d.f. represents the degrees of freedom associated with each sum of squares. MS denotes the mean square. To test the null hypothesis that there is no difference in treatment means, that is, to test H0 : αi = 0, i = 1, . . . , k versus Ha : Not all αi s are zero

we use the F -statistic F=

MST MSE

and reject H0 if F > Fα based on (k − 1) numerator and (n − b − k + 1) denominator degrees of freedom. Although blocking lowers the experimental error, it also furnishes a chance to see whether evidence exists to indicate a difference in the mean response for blocks. In this case we will be testing the hypothesis H0 : βj = 0, j = 1, . . . , b versus Ha : Not all βj s are zero.

Under the assumption that there is no difference in the mean response for blocks, MSB provides an unbiased estimator for σ 2 based on (b − 1) degrees of freedom. If there is a real difference that exists among block means, MSB will be larger in comparison with MSE and F=

MSB MSE

will be used as a test statistic. The rejection region will be if F > Fα based on (b − 1) numerator and (n − b − k + 1) denominator degrees of freedom.

530 CHAPTER 10 Analysis of Variance

We now summarize the foregoing methodology in a step-by-step computational procedure. For a reasonable data size, we could use scientiﬁc calculators for handling the ANOVA calculations. For larger data sets, the use of statistical software packages is recommended.

COMPUTATIONAL PROCEDURE FOR RANDOMIZED BLOCK DESIGN 1. Calculate the following quantities: (i) Sum the observations for each row to form row totals: T1 , T2 , . . . , Tk , where Ti =

b

yij .

j=1

(ii) Sum the observations for each column to form column totals: B1 , B2 , . . . , Bb , where Bj =

k

yij .

i=1

(iii) Find the sum of all observations: b k

yij =

j=1 i=1

b

Bj .

j=1

2. Calculate the following quantities: (i) Square the sum of the totals for each column and divide it by n = bk to obtain ⎛ ⎞ b 1 ⎝ 2 ⎠ Bj . CM = n j=1

(ii) Find the sum of squares of the totals of each column and divide it by k to obtain b 1 2 Bj k j=1

and b Bj2

SSB =

j=1

k

− CM

and

MSB =

SSB . b−1

(iii) Find the sum of squares of the totals of each row and divide it by b to obtain k Ti2

i=1

b

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 531

and k Tj2

SST =

i=1

b

− CM

and

MSB =

SST . k −1

(iv) Find the sum of squares of individual observations: b k

yij2

j=1 i=1

Also compute Total SS =

b k

yij2 − CM.

j=1 i=1

(v) Using (ii), (iii), and (iv), ﬁnd SSE = Total SS − SSB − SST

and

MSE =

SSE . n−b−k +1

3. To test the null hypothesis that there is no difference in treatment means: (i) Compute the F -statistic, F=

MST . MSE

(ii) From the F-table, ﬁnd the value of Fα, υ1 , υ2 , where υ1 = (k − 1) is the numerator and υ2 = (n − b − k + 1) the denominator degrees of freedom. (iii) Decision: Reject H0 if F > Fα, υ1 , υ2 and conclude that there is evidence to conclude that there is a difference in treatment means at level α. 4. To test the null hypothesis that there is no difference in the mean response for blocks, (i) Compute the F-statistic, F=

MSB . MSE

(ii) From the F -table, ﬁnd the value of Fα, υ1 , υ2 , where υ1 = (b − 1) is the numerator and υ2 = (n − b − k + 1) the denominator degrees of freedom. (iii) Decision: Reject H0 if F > Fα, υ1 , υ2 and conclude that there is evidence to conclude there is a difference in the mean response for blocks at level α. Assumptions: The samples are randomly selected in an independent manner from n = bk populations. The populations are assumed to be normally distributed with equal variances σ 2 . Also, there are no interactions between the variables (two factors).

532 CHAPTER 10 Analysis of Variance

We have already discussed the assumptions and how to verify those assumptions in one-way analysis. The only new assumption in the randomized blocked design is about the interactions. One of the ways to verify the assumption of no interaction is to plot the observed values against the sample number. If there is no interaction, the line segments (one for each block) will be parallel or nearly parallel; see Figure 9.2. If the lines are not approximately parallel, then there is likely to be interaction between blocks and treatments. In the presence of interactions, the analysis of this section need to be modiﬁed. For details on those procedures, refer to more specialized books on ANOVA methods. We illustrate the randomized block design procedure with the following example.

Example 10.4.1 A furniture company wants to know whether there are differences in stain resistance among the four chemicals used to treat three different fabrics. Table 10.7 shows the yields on resistance to stain (a low value indicates good stain resistance). At the α = 0.05 level of signiﬁcance, is there evidence to conclude that there is a difference in mean resistance among the four chemicals? Is there any difference in the mean resistance among the materials? Give bounds for the p-values in each case.

Table 10.7 Chemical

Material I

II

III

Total

C1

3

7

6

16

C2

9

11

8

28

C3

2

5

7

14

C4

7

9

8

24

Total

21

32

29

82

Solution Here T1 = 16, T2 = 28, T3 = 14, and T4 = 24. Also, B1 = 21, B2 = 32, and B3 = 29. In addition, b = 3, k = 4, and n = bk = 12. Now ⎞2 ⎛ b 1 1 ⎝ ⎠ (82)2 = 560.3333. Bj = CM = n 12 j=1

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 533

We can compute the following quantities: b

SSB =

Bj2

j=1

k

SST =

− CM =

k

MSB =

16.1667 SSB = = 8.0834, b−1 2

− CM =

1812 − 560.3333 = 43.6667, 3

Ti2

i=1

b

2306 − 560.3333 = 16.1667, 4

and MST =

We have

SST 43.6667 = = 14.5556. k−1 3

k b 2 = 632. From this yij

j=1 i=1

Total SS =

b k

2 − CM = 632 − 560.3333 = 71.666 yij

j=1 i=1

SSE = Total SS − SSB − SST = 71.6667 − 16.1667 − 43.6667 = 11.8333 and MSE =

11.8333 SSE = = 1.9722. n−b−k+1 6

The F -statistic is F=

MST 14.5556 = = 7.3804 MSE 1.9722

From the F -table, F0.05,3,6 = 4.76. Because the observed value F = 7.3804 > 4.76, we reject the null hypothesis and conclude that there is a difference in mean resistance among the four chemicals. Because the F -value falls between α = 0.025 and α = 0.01, the p-value falls between 0.01 and 0.025. To test for the difference in the mean resistance among the materials, F=

MSB 8.0834 = = 4.0987. MSE 1.9722

From the F -table, F0.05,2,6 = 5.14. Because the observed value of F = 4.098 < 5.14, we conclude that there is no difference in the mean resistance among the materials. Because the F -value falls between α = 0.10 and 0.05, the p-value falls between 0.05 and 0.10.

534 CHAPTER 10 Analysis of Variance

EXERCISES 10.4 10.4.1.

Show that b k

yij − y

2

j=1 i=1

=

k b

yij − Ti − Bj − y

2

i=1 j=1

+b

k b 2 2 Ti − y + k Bj − y . i=1

j=1

[Hint: Use the identity yij − y = yij − Ti − Bj − y + Ti − y + Bj − y .] 10.4.2.

Show the following: (a) E(MSE) = σ 2 , (b) E(MSB) = (c) E(MST ) =

10.4.3.

k b−1 b k−1

b j=1 k i=1

Bj2 + σ 2 , τi2 + σ 2 .

The least-square estimators of the parameters μ, τi ’s, and βj ’s are obtained by minimizing the sum of squares W=

k b

yij − μ − τi − βj

2

i=1 j=1

with respect to μ, τi ’s, and βj ’s; subject to the restrictions:

k i=1

τi =

b

βj = 0. Show that the

j=1

resultant estimators are μ ˆ = y, τˆi = Ti − y, i = 1, 2, . . . , k,

and βˆ j = Bj − y,

10.4.4.

j = 1, . . . , b.

In order to test the wear on four hyperalloys, a test piece of each alloy was extracted from each of the three positions of a test machine. The reduction of weight in milligrams due to wear was determined on each piece, and the data are given in Table 10.4.1. At α = 0.05, test the following hypotheses, regarding the positions as blocks: (a) There is no difference in average wear for each material. (b) There is no difference in average wear for each position. (c) Interpret your ﬁnal result and state any assumptions that were necessary to solve the problem.

10.4 Two-Way Analysis of Variance, Randomized Complete Block Design 535

Table 10.4.1 Loss in Weights Due to Wear Testing of Four Materials (in mg) Position Type of alloy

1

2

3

1

241

270

274

2

195

241

218

3

235

273

230

4

234

236

227

10.4.5.

For the data of Exercise 10.3.10, test at the 0.05 level that the true income lower limits of the top 5% of U.S. households for each race are the same for all 5 years. Also, test at the 0.05 level that the true income lower limits of the top 5% of U.S. households for each year between 1994 and 1998 are the same.

10.4.6.

For the data of Exercise 10.3.11, test at the 0.01 level that the true mean cholesterol levels for all races in the United States during 1978–1980 are the same. Also, test at the 0.01 level that the true mean cholesterol levels for all ages in the United States during 1978–1980 are the same.

10.4.7.

In order to see the effect of hours of sleep on tests of different skill categories (vocabulary, reasoning, and arithmetic), tests consisting of 20 questions each in each category were given to 16 students, four each based on the hours of sleep they had on the previous night. Each right answer is given one point. Table 10.4.2 gives the cumulative scores of the each of the four students in each category.

Table 10.4.2 Hours of sleep

Category Vocabulary

Reasoning

Arithmetic

0

44

33

35

4

54

38

18

6

48

42

43

8

55

52

50

Test at the 0.05 level whether the true mean performance for different hours of sleep is the same. Also, test at the 0.05 level whether the true mean performance for each category of the test is the same.

536 CHAPTER 10 Analysis of Variance

10.5 MULTIPLE COMPARISONS The analysis of variance procedures that we have used so far showed whether differences among several means are signiﬁcant. However, if the equality of means is rejected, the F -test did not pinpoint for us which of the given means or group of means differs signiﬁcantly from another given mean or group of means. With ANOVA, when the null hypothesis of equality of means is rejected, the problem is to see whether there is some way to follow up (post hoc) this initial test H0 : μ1 = μ2 = . . . = μk by looking at subhypotheses, such as H0 : μ1 = μ2 . This involves multiple tests. However, the solution is not to use a simple t-test repeatedly for every possible combination taken two at a time. That, apart from introducing many tests, will considerably increase the signiﬁcance level, the probability of type I error. For example, to test four samples we will need 42 = 6 tests. If each one of the comparisons is tested with the same value of α = P (type I error), and if all the null hypotheses involving six comparisons are true, then the probability of rejecting at least one of them is P(at least one type I error) = 1 − (1 − α)6 .

In particular, if α = 0.01, then P(at least one type I error) = 0.077181, which is signiﬁcantly higher than the original error value of 0.01. One way to investigate the problem is to use a multiple comparison procedure. A good deal of work has been done on problems of multiple comparisons. There are a variety of techniques available in the literature, such as the Bonferroni procedure, Tukey’s method, and Scheffe’s method. We now describe one of the more popular procedures called Tukey’s method for completely randomized, one factor design. In this multiple comparison problem, we would like to test H0 : μi = μj versus Ha : μi = μj , for all i = j. Tukey’s method will be used to test all possible differences of means to decide whether at least one of the differences μi −μj is considerably different from zero. In this comparison problem, Tukey’s method makes use of conﬁdence intervals for μi − μj . If each conﬁdence interval has a conﬁdence level 1 − α, then the probability that all conﬁdence intervals include their respective parameters is less than 1 − α. We now describe this method where each of the k sample means is based on the common number of observations, n. Let N = kn be the total number of observations and let S2 =

k ni =n 2 1 Yij − Ti . N −k i=1 j=1

Let T max

= max T1 , . . . , Tk and T min = min T1 , . . . , Tk . Deﬁne the random variable Q=

T max − T min . √ S n

The distribution of Q under the null hypothesis H0 : μ1 = . . . = μk is called the Studentized range distribution, which depends on the number of samples k and the degrees of freedom υ = N − k = (n − 1)k. We denote the upper α critical value by qα,k,υ . The Studentized range distribution table gives

10.5 Multiple Comparisons 537

values for selected values of k, υ, and α = 0.01, 0.05, and 0.10. The following theorem, due to Tukey, deﬁnes the test procedure. Theorem 10.5.1 Let Ti , i = 1, 2, . . . , k be the k sample means in a completely randomized design. Let μi , i = 1, 2, . . . , k be the true means and let ni = n be the common sample size. Then the probability that all 2k differences μi − μj will simultaneously satisfy the inequalities

s s Ti − Tj − qα,k,υ √ ≤ μi − μj ≤ Ti − Tj + qα,k,υ √ , n n

is (1 − α), where qα,k,υ is the upper α critical value of the Studentized range distribution. If, for a given i and j, zero is not contained in the preceding inequality, H0 : μi = μj can be rejected in favor of Ha : μi = μj , at the signiﬁcance level of α. Now we give a step-by-step approach to implementing Tukey’s method discussed earlier. PROCEDURE TO FIND (1–α)100% CONFIDENCE INTERVALS FOR DIFFERENCE OF MEANS WITH COMMON SAMPLE SIZE N: TUKEY’S METHOD 1. There are k2 comparisons of μi versus μj . 2. Compute the following quantities: ni

Ti =

yij

j=1

ni

, i = 1, 2, . . . , k,

and s2 =

k ni =n 2 1 yij − Ti , where N = kn. N −k i=1 j=1

3. From the Studentized range distribution table, ﬁnd the upper α critical value, qα, k, υ , where υ = N − k = (n − 1)k . 4. For each of k2 pair (i, j), i = j, compute the Tukey’s interval Ti − Tj − qα, k, υ √s , Ti − Tj + qα, k, υ √s . n n 5. Let NR denote insufﬁcient evidence for rejecting H0 . Create the following table for each of k2 pairwise difference μi − μj , i = j, and do not reject if the Tukey interval contains the number 0. Otherwise reject.

Table 10.8 is used to summarize the ﬁnal calculations of the Tukey method. In practice, there are now numerous statistical packages available for Tukey’s purpose. The following example is solved using Minitab. The necessary Minitab commands are given in Example 10.7.3.

538 CHAPTER 10 Analysis of Variance

Table 10.8 μi − μj

Ti − Tj

Tukey interval

Observation

Conclusion

μ1 − μ2

T1 − T2

...

Doesn’t contain 0

Reject

μ1 − μ3

T1 − T3

...

Contains 0

Do not reject

. . .

. . .

. . .

. . .

. . .

Example 10.5.1 Table 10.9 shows the 1-year percentage total return of the top ﬁve stock funds for ﬁve different categories (source: Money, July 2000). Which categories have similar top returns and which are different? Use 95% Tukey’s conﬁdence intervals.

Table 10.9 Large-cap

Mid-cap

Small-cap

Hybrid

Specialty

110.1

299.8

153.8

68.3

181.6

102.9

139.0

139.8

67.1

159.3

93.1

131.2

138.3

42.5

138.3

83.0

110.5

121.4

40.0

132.6

83.3

129.2

135.9

41.0

135.7

Solution For simplicity of computation, we will use SPSS (Minitab steps are given in Example 10.7.2). The following is the output. One-way ANOVA RETURN

Between Groups Within Groups Total

Sum of Squares

df

Mean Square

F

Sig.

41243.698 27877.580 69121.278

4 20 24

10310.925 1393.879

7.397

.001

10.5 Multiple Comparisons 539

Post Hoc Tests Multiple Comparisons Dependent Variable: RETURN Tukey HSD (I) FUND

(J) FUND

Mean

Std. Error

Sig.

Difference

95% Confidence Interval Lower Bound

Upper Bound

(I-J) 1.00

2.00

3.00

4.00

5.00

2.00

−67.4600

23.61253

.066

−138.1175

3.1975

3.00

−43.3600

23.61253

.382

−114.0175

27.2975

4.00

42.7000

23.61253

.396

−27.9575

113.3575

5.00

−55.0200

23.61253

.177

−125.6775

15.6375

1.00

67.4600

23.61253

.066

−3.1975

138.1175

3.00

24.1000

23.61253

.843

−46.5575

94.7575

4.00

110.1600*

23.61253

.001

39.5025

180.8175

5.00

12.4400

23.61253

.984

−58.2175

83.0975

1.00

43.3600

23.61253

.382

−27.2975

114.0175

2.00

−24.1000

23.61253

.843

−94.7575

46.5575

4.00

86.0600*

23.61253

.012

15.4025

156.7175

5.00

−11.6600

23.61253

.987

−82.3175

58.9975

1.00

−42.7000

23.61253

.396

−113.3575

27.9575

2.00

−110.1600*

23.61253

.001

−180.8175

−39.5025

3.00

−86.0600*

23.61253

.012

−156.7175

−15.4025

5.00

−97.7200*

23.61253

.004

−168.3775

−27.0625

1.00

55.0200

23.61253

.177

−15.6375

125.6775

2.00

−12.4400

23.61253

.984

−83.0975

58.2175

3.00

11.6600

23.61253

.987

−58.9975

82.3175

4.00

97.7200*

23.61253

.004

27.0625

168.3775

* The mean difference is significant at the .05 level.

540 CHAPTER 10 Analysis of Variance

Homogeneous Subsets RETURN Tukey HSDa FUND 4.00 1.00 3.00 5.00 2.00 Sig.

N 5 5 5 5 5

Subset for alpha = .05 2

1 51.7800 94.4800

94.4800 137.8400 149.5000 161.9400 .066

.396

Means for groups in homogeneous subsets are displayed. a Uses Harmonic Mean Sample Size = 5.000. The Tukey intervals for pairwise differences (μi − μj ) are in the foregoing computer printout. For example, the Tukey interval for (μ1 − μ2 ) is (−138.1, 3.2) and for (μ2 − μ4 ) is (39.5, 180.8). Also, sample mean and standard deviation are given in the output. For example, 94.48 is the sample mean of the five data points of large-cap funds, and 11.97 is the sample standard deviation of the five data points of large-cap funds. If the Tukey interval for a particular difference (μj − μi ) contains the number zero, we do not reject the H0 : μi = μj . Otherwise, we reject the H0 : μi = μj . For example the interval for (μ4 − μ2 ) is (39.5 − 180.8) and does not contain zero. Hence we reject H0 : μ4 = μ2 . The complete table corresponding to step 5 is produced in Table 10.10, where N.R. represents ‘‘not reject.’’

Table 10.10 μi − μ j

Ti − Tj

Tukey interval

Reject or N.R.

μ1 − μ2

161.94 − 94.48

(−138.1, 3.2)

N.R.

μ1 = μ2

μ1 − μ3

137.84 − 94.48

(−114.0, 27.3)

N.R.

μ1 = μ3

μ2 − μ3

137.84 − 161.94

(−46.6, 94.8)

N.R.

μ3 = μ2

μ1 − μ4

51.78 − 94.48

(−27.9, 113.3)

N.R.

μ4 = μ1

μ2 − μ4

51.78 − 161.94

(39.5, 180.8)

R

μ4 = μ2

μ3 − μ4

51.78 − 137.84

(15.4, 156.7)

R

μ4 = μ3

μ1 − μ5

149.50 − 94.98

(−125.6, 15.6)

N.R.

μ5 = μ1

μ2 − μ5

149.50 − 161.94

(−58.2, 83.1)

N.R.

μ5 = μ2

μ3 − μ5

149.50 − 137.84

(−82.3, 59.0)

N.R.

μ5 = μ3

μ4 − μ5

149.50 − 51.78

(−168.3, −27.1)

R

μ5 = μ4

Conclusion

10.5 Multiple Comparisons 541

Based on the 95% Tukey intervals, the average top return of hybrid funds is different from those for mid-cap, small-cap, and specialty funds. All other returns are similar.

In Tukey’s method, the conﬁdence coefﬁcient for the set of all pairwise comparisons {μi − μj } is exactly equal to 1 − α when all sample sizes are equal. For unequal sample sizes, the conﬁdence coefﬁcient is greater than 1 − α. In this sense, Tukey’s procedure is conservative when the sample sizes are not equal. In the case of unequal sample sizes, one has to estimate the standard deviation for each pairwise comparison. Tukey’s procedure for unequal sample sizes is sometimes referred to as the Tukey–Kramer method.

EXERCISES 10.5 10.5.1.

A large insurance company wants to determine whether there is a difference in the average time to process claim forms among its four different processing facilities. The data in Table 10.5.1 represent weekly average number of days to process a form over a period of 4 weeks.

Table 10.5.1 Facility 1

Facility 2

Facility 3

Facility 4

1.50

2.25

1.30

2.0

0.9

1.85

2.75

1.5

1.12

1.45

2.15

2.85

1.95

2.15

1.55

1.15

(a) Test whether there is a difference in the average processing times at the 0.05 level. (b) Test whether there is a difference, using Tukey’s method to ﬁnd which facilities are different. (c) Interpret your results and state any assumptions you have made in solving the problem. 10.5.2.

Table 10.5.2 gives the rental vacancy rates by U.S. region (source: U.S. Census Bureau) for 5 years.

Table 10.5.2 Rental units

1995

1996

1997

1998

Northeast

7.2

7.4

6.7

6.7

1999 6.3

Midwest

7.2

7.9

8.0

7.9

8.6

South

8.3

8.6

9.1

9.6

10.3

West

7.5

7.2

6.6

6.7

6.2

542 CHAPTER 10 Analysis of Variance

(a) Test at the 0.01 level whether the true rental vacancy rates by region are the same for all 5 years. (b) If there is a difference, use Tukey’s method to ﬁnd which regions are different. 10.5.3.

Table 10.5.3 gives lower limits of income (approximated to nearest $1000 and calculated as of March of the following year) by race for the top 5% of U.S. households from 1994 to 1998. (Source: U.S. Census Bureau.)

Table 10.5.3 Race

1994

1995

1996

1997

All Races

110

113

120

127

1998 132

White

113

117

123

130

136

Black

81

80

85

87

94

Hispanic

82

80

86

93

98

(a) Test at the 0.05 level whether the true lower limits of income for the top 5% of U.S. households for each race are the same for all 5 years. (b) If there is a difference, use Tukey’s method to ﬁnd which is different. (c) Interpret your results and state any assumptions you have made in solving the problem. 10.5.4.

The data in Table 10.5.4 represent the mean serum cholesterol levels (given in milligrams per deciliter) by race and age in the United States from 1978 to 1980 (source: “Report of the National Cholesterol Education Program Expert Panel on Detection, Evaluation, and Treatment of High Blood Cholesterol in Adults,” Arch. Intern. Med. 148, Jan. 1988).

Table 10.5.4 Race

Age 20–24

25–34

35–44

45–54

55–64

65–74

All races

180

199

217

227

229

221

White

180

199

217

227

230

222

Black

171

199

218

229

223

217

(a) Test at the 0.01 level whether the true mean cholesterol levels for all races in the United States during 1978–1980 are the same. (b) If there is a difference, use Tukey’s method to ﬁnd which of the races are different with respect to the mean cholesterol levels.

10.7 Computer Examples 543

10.6 CHAPTER SUMMARY In this chapter, we have introduced the basic idea of analyzing various experimental designs. In Section 10.3, we explained the one-way analysis of variance for the hypothesis testing problem for more than two means (different treatments being applied, or different populations being sampled). The two-way analysis of variance, having b blocks and k treatments consisting of b blocks of k experimental units each, is discussed in Section 10.5. We also describe one popular procedure called Tukey’s method for completely randomized, one-factor design for multiple comparisons. We saw in Chapter 9 that there are other possible designs, such as the Latin square design or Taguchi methods. We refer to specialized books on experimental design (Hicks and Turner) for more details on how to conduct ANOVA on such designs. In the ﬁnal section, we give some computational examples. We now list some of the key deﬁnitions introduced in this chapter: ■ ■ ■ ■

Completely randomized experimental design Randomized block design Studentized range distribution Tukey–Kramer method

In this chapter, we also learned the following important concepts and procedures: ■ ■ ■ ■

■

Analysis of variance procedure for two treatments One-way analysis of variance for k ≥ 2 populations One-way analysis of variance procedure for k ≥ 2 populations Procedure to ﬁnd (1 − α)100% conﬁdence intervals for difference of means with common sample size n; Tukey’s method Computational procedure for randomized block design

10.7 COMPUTER EXAMPLES Minitab, SPSS, SAS, and other statistical programming packages are especially useful when we perform an analysis of variance. As we have experienced in earlier sections, an ANOVA computation is very tedious to complete by hand.

10.7.1 Minitab Examples Example 10.7.1 (One-way ANOVA): The three random samples in Table 10.11 are independently obtained from three different normal populations with equal variances. At the α = 0.05 level of signiﬁcance, test for equality of means.

544 CHAPTER 10 Analysis of Variance

Table 10.11 Sample 1

Sample 2

Sample 3

64

56

81

84

74

92

75

69

84

77 80

Solution Enter sample 1 data in C1, sample 2 in C2, and sample 3 in C3. Stat > ANOVA > One-way (unstacked). . . > in Responses (in separate columns): type C1 C2 C3 and click OK We get the following output: One-Way Analysis of Variance Analysis of Variance Source DF SS Factor 2 560.7 Error 8 463.3 Total 10 1024.0

Level C1 C2 C3

N 5 3 3

Pooled StDev =

MS 280.3 57.9

F 4.84

P 0.042

Individual 95% CIs For Mean Based on Pooled StDev Mean StDev ----+---------+---------+---------+-76.000 7.517 (-----*------) 66.333 9.292 (-------*--------) 85.667 5.686 (-------*--------) -----+---------+--------+---------+-7.610 60 72 84 96

We can see that the output contains, SS, MS, individual column means, and standard deviation values. Also, the F -value gives the value of the test statistic, and the p-value is obtained as 0.042. Comparing this p-value of 0.042 with α = 0.05, we will reject the null hypothesis. If we want to create side-by-side box plots to graphically test homogeneity of variances, we can do the following.

10.7 Computer Examples 545

Enter all the data (from all three samples) in C1, and enter the sample identifier number in C2 (that is, 1 if the data belong to sample 1, 2 for sample 2, and 3 for sample 3). Graph > Boxplot > in Y column, type C1 and in X column, type C2 > click OK Then as in Example 10.3.2, interpret the resulting box plots.

Example 10.7.2 Give Minitab steps for randomized block design for the data of Example 10.4.1.

Solution To put the data into the format for Minitab, place all the data values in one column (say, C2). Let numbers 1, 2, 3, 4 represent the chemicals and numbers 1, 2, 3 represent the fabric material. In one column (say, C1) place numbers 1 through 4 with respect to the data values identifying the factor (chemical) used. In another column (say, C3) place corresponding numbers 1 through 3 to identify the second factor (material) used. See Table 10.12.

Table 10.12 C1 chemical

C2 response

C3 material

1

3

1

2

9

1

3

2

1

4

7

1

1

7

2

2

11

2

3

5

2

4

9

2

1

6

3

2

8

3

3

7

3

4

8

3

546 CHAPTER 10 Analysis of Variance

Then do the following: Stat > ANOVA > Two-way. . . > in Response: type C2, in Row Factor: type C1, and in Column factor: type C3 > OK We will get the following output. Two-Way Analysis of Variance Analysis of Variance for Response Source DF SS MS F Chemical 3 43.67 14.56 7.38 Material 2 16.17 8.08 4.10 Error 6 11.83 1.97 Total 11 71.67

P 0.019 0.075

Note that the output contains p-values for the effect both of the chemicals and of the materials. Because the p-value of 0.019 is less than α = 0.05, we reject the null hypothesis and conclude that there is a difference in mean resistance among the four chemicals. For the materials, the p-value of 0.075 is greater than α = 0.05, so we cannot reject the null hypothesis and conclude that there is no difference in the mean resistance among the materials.

Example 10.7.3 Give the Minitab steps for using Tukey’s method for the data of Example 10.5.1.

Solution In order to use Tukey’s method, it is necessary to enter the data in a particular way. Enter all the data points in column C1; first five from large-cap, next five from mid-cap, and so on, with the last five from specialty. In column C2, enter the number identifying the data points; the first four numbers are 1 (identifying 1 as the data belonging to large-cap), next five numbers are 2, and so on; the last five numbers are 5. Then: Stat > ANOVA > One-way. . . > Comparisons. . . > click Tukey’s, family error rate: and type 5 (to represent 100α% error) > OK > in Response: type C1, and in Factor: type C2 > OK We will get the output similar to that given in the solution part of Example 10.5.1. For discussion of the output, refer to Example 10.5.1.

10.7.2 SPSS Examples Example 10.7.4 Conduct a one-way ANOVA for the data of Example 10.7.1. Use α = 0.05 level of signiﬁcance, and test for equality of means.

10.7 Computer Examples 547

Solution In SPSS, we need to enter the data in a special way. First name column C1 as Sample, and column C2 as Values. In the Sample column, enter the numbers to identify from which group the data comes. In this case, enter 1 in the first five rows, 2 in the next three rows, and 3 in the last three rows. In the Values column, enter sample 1 data in the first five rows, sample 2 data in the next five rows, and sample 3 data in the last three rows. Then: Analyze > Compare Means > One-way ANOVA. . . > Bring Values to Dependent List: and Sample to Factor: > OK We will get the following output. ANOVA VALUES Between Groups Within Groups Total

Sum of Squares 560.667 463.333 1024.000

df 2 8 10

Mean Square 280.333 57.917

F 4.840

Sig. .042

Because Sig. Value 0.042 is less than α = 0.05, we reject the null hypothesis.

Example 10.7.5 Give the SPSS steps for using Tukey’s method for the data of Example 10.5.1.

Solution First name column C1 as Fund and column C2 as Return. In the Fund column, enter the numbers to identify from which group the data comes. In this case, the first four numbers are 1 (identifying 1 as the data belonging to large-cap), the next four numbers are 2, and so on, until the last four numbers are 5. In the Return column, enter large-cap return data in the first four rows, mid-cap data in the next four rows, and so on; the last four from speciality. Then: Analyze > Compare Means > One-way ANOVA. . . > Bring Return to Dependent List: and Fund to Factor: > Click Post-Hoc. . . > click Tukey > click Continue > OK

We will get the output as in Example 10.5.1. Interpretation of output is given in Example 10.5.1. When the treatment effects are significant, as in this example where the p-value is 0.001, the means must then be further examined to determine the nature of the effects. There are procedures called post hoc tests to assist the researcher in this task. For example, looking at the output column Sig., we could observe that there are significant differences in the mean returns between funds 2 and 4, and funds 4 and 5.

548 CHAPTER 10 Analysis of Variance

10.7.3 SAS Examples Example 10.7.6 Using SAS, conduct a one-way ANOVA for the data of Example 10.7.1. Use α = 0.05 level of signiﬁcance, and test for equality of means.

Solution We could use the following code. Options nodate nonumber; options ls=80 ps=50; DATA Scores; INPUT Sample Value @@; DATALINES; 1 64 1 84 1 75 1 77 1 80 2 56 2 74 2 69 3 81 3 92 3 84 ; PROC ANOVA DATA=Scores; TITLE ’ANOVA for Scores’; CLASS Sample; MODEL Value=Sample; MEANS Sample; RUN; We will get the following output: ANOVA for Scores The ANOVA Procedure Class Level Information Class

Levels

Sample

Values

3

Number of observations

1 2 3 11

The ANOVA Procedure Dependent Variable: Value

Source

DF

Model

2

Sum of Squares 560.666667

Mean Square 280.333333

F Value

Pr > F

4.84

0.0419

10.7 Computer Examples 549

Error Corrected Total

8

463.333333

10

1024.000000

57.916667

R-Square

Coeff Var

Root MSE

Value Mean

0.547526

10.01355

7.610300

76.00000

Source

DF

Sample

2

Anova SS 560.6666667

Mean Square

F Value

280.3333333

4.84

Pr > F 0.0419

The ANOVA Procedure Level of Sample

------------Value-----------Mean Std Dev

N

1 2 3

5 3 3

76.0000000 66.3333333 85.6666667

7.51664819 9.29157324 5.68624070

Because the p-value 0.0419 is less than α = 0.05, we reject the null hypothesis. We could have used PROC GLM instead of PROC ANOVA to perform the ANOVA procedure. Usually, PROC ANOVA is used when the sizes of the samples are equal; otherwise PROC GLM is more desirable. The next example will show how to do the multiple comparison using Tukey’s procedure.

Example 10.7.7 Give the SAS commands for using Tukey’s method for the data of Example 10.5.1.

Solution We could use the following code. Options nodate nonumber; options ls=80 ps=50; DATA Mfundrtn; INPUT Fund Return @@; DATALINES; 1 110.1 2 299.8 1 102.9 2 139.0 1 93.1 2 131.2 1 83.3 2 129.2 1 83.0 2 110.5 ;

3 3 3 3 3

153.8 139.8 138.3 135.9 121.4

4 4 4 4 4

68.3 67.1 42.5 41.0 40.0

5 5 5 5 5

181.6 159.3 138.3 135.7 132.6

550 CHAPTER 10 Analysis of Variance

PROC GLM DATA=Mfundrtn; TITLE ’ANOVA for Mutual fund returns’; CLASS Fund; MODEL Return=Fund; MEANS Fund / tukey; RUN; ANOVA for Mutual fund returns The GLM Procedure Class Level Information Class

Levels

Fund

5

Values 1 2 3 4 5

Number of observations

25

ANOVA for Mutual fund returns The GLM Procedure Dependent Variable: Return Source

Sum of Squares

DF

Mean Square

Model

4

Error

20

27877.58000

Corrected Total

24

69121.27840

Source Fund Source Fund

41243.69840

10310.92460

Coeff Var

Root MSE

0.596686

31.34524

37.33469

Type I SS

4

41243.69840

DF

Type III SS

4

Mean Square 10310.92460 Mean Square

41243.69840

Pr > F

7.40

0.0008

1393.87900

R-Square

DF

F Value

10310.92460

Return Mean 119.1080 F Value 7.40

Pr > F 0.0008

F Value

Pr > F

7.40

0.0008

ANOVA for Mutual fund returns The GLM Procedure Tukey’s Studentized Range (HSD) Test for Return

10.7 Computer Examples 551

NOTE: This test controls the Type I experiment wise error rate, but it generally has a higher Type II error rate than REGWQ.

Alpha 0.05 Error Degrees of Freedom 20 Error Mean Square 1393.879 Critical Value of Studentized Range 4.23186 Minimum Significant Difference 70.658 Means with the same letter are not significantly different. Tukey Grouping

Mean

N

A A A A A A

161.94

5

2

149.50

5

5

137.84

5

3

B B B

A

Fund

94.48

5

1

51.78

5

4

The GLM Procedure Tukey’s Studentized Range (HSD) Test for Value NOTE: This test controls the Type I experiment wise error rate, but it generally has a higher Type II error rate than REGWQ. Alpha 0.05 Error Degrees of Freedom 20 Error Mean Square 1393.879 Critical Value of Studentized Range 4.23186 Minimum Significant Difference 70.658 Means with the same letter are not significantly different. Tukey Grouping

B B B

A A A A A A A

Mean

N

Sample

161.94

5

2

149.50

5

5

137.84

5

3

94.48

5

1

51.78

5

4

552 CHAPTER 10 Analysis of Variance

Looking at the p-value of 0.008, which is less than α = 0.05, we conclude that there is a difference in mutual fund returns. In the previous example, we used the post hoc test Tukey. We could have used other options such as DUNCAN, SNK, LSD, and SCHEFFE. The test is performed at the default value of α = 0.05. If we want to specify, say, α = 0.01, or 0.1, we could have done so by using the command MEANS Fund / Tuckey ALPHA=0.01;. If we need all the confidence intervals in the Tukey method, in the code just given, we have to modify ‘MEANS Fund / Tukey;’ to ‘MEANS Fund / LSD TUKEY CLDIFF;’ which will result in the following output. ANOVA for Mutual fund returns The GLM Procedure Class Level Information Class

levels

Fund

5

Values 1 2 3 4 5

Number of observations

25

ANOVA for Mutual fund returns The GLM Procedure Dependent Variable: Return

Source

DF

Model

4

Error

20

Sum of Squares 41243.69840

Fund Source Fund

7.40

Pr > F 0.0008

1393.87900

69121.27840

R-Square

Coeff Var

0.596686

31.34524

DF

F Value

10310.92460

27877.58000

Corrected Total 24

Source

Mean Square

Root MSE 37.33469

Return Mean

Type I SS

Mean Square

4

41243.69840

10310.92460

DF

Type III SS

Mean Square

4

41243.69840

10310.92460

119.1080 F Value 7.40 F Value 7.40

Pr > F 0.0008 Pr > F 0.0008

10.7 Computer Examples 553

ANOVA for Mutual fund returns The GLM Procedure t-tests (LSD) for Return NOTE: This test controls the Type I comparisonwise error rate, not the experiment wise error rate.

Alpha Error Degrees of Freedom Error Mean Square Critical Value of t Least Significant Difference

0.05 20 1393.879 2.08596 49.255

Comparisons significant at the 0.05 level are indicated by ***. Difference Fund Comparison

Between Means

95% Confidence Limits

2 2 2 2 5 5 5 5 3 3 3 3 1 1 1 1 4 4 4 4

12.44 24.10 67.46 110.16 –12.44 11.66 55.02 97.72 –24.10 –11.66 43.36 86.06 –67.46 –55.02 –43.36 42.70 –110.16 –97.72 –86.06 –42.70

–36.81 –25.15 18.21 60.91 –61.69 –37.59 5.77 48.47 –73.35 –60.91 –5.89 36.81 –116.71 –104.27 –92.61 –6.55 –159.41 –146.97 –135.31 –91.95

– – – – – – – – – – – – – – – – – – – –

5 3 1 4 2 3 1 4 2 5 1 4 2 5 3 4 2 5 3 1

61.69 73.35 116.71 159.41 36.81 60.91 104.27 146.97 25.15 37.59 92.61 135.31 –18.21 –5.77 5.89 91.95 –60.91 –48.47 –36.81 6.55

*** *** *** ***

*** *** *** *** *** ***

ANOVA for Mutual fund returns The GLM Procedure Tukey’s Studentized Range (HSD) Test for Return NOTE: This test controls the Type I experiment wise error rate.

554 CHAPTER 10 Analysis of Variance

Alpha Error Degrees of Freedom Error Mean Square Critical Value of Studentized Range Minimum Significant Difference

0.05 20 1393.879 4.23186 70.658

Comparisons significant at the 0.05 level are indicated by ***. Difference Fund Comparison 2 2 2 2 5 5 5 5 3 3 3 3 1 1 1 1 4 4 4 4

– – – – – – – – – – – – – – – – – – – –

5 3 1 4 2 3 1 4 2 5 1 4 2 5 3 4 2 5 3 1

Between Means 12.44 24.10 67.46 110.16 –12.44 11.66 55.02 97.72 –24.10 –11.66 43.36 86.06 –67.46 –55.02 –43.36 42.70 –110.16 –97.72 –86.06 –42.70

Simultaneous 95% Confidence Limits –58.22 –46.56 –3.20 39.50 –83.10 –59.00 –15.64 27.06 –94.76 –82.32 –27.30 15.40 –138.12 –125.68 –114.02 –27.96 –180.82 –168.38 –156.72 –113.36

83.10 94.76 138.12 180.82 58.22 82.32 125.68 168.38 46.56 59.00 114.02 156.72 3.20 15.64 27.30 113.36 –39.50 –27.06 –15.40 27.96

***

***

***

*** *** ***

EXERCISES 10.7 10.7.1.

For the data of Exercise 10.5.4, perform a one-way analysis of variance using any of the software (Minitab, SPSS, or SAS).

10.7.2.

For the data of Exercise 10.5.2, perform Tukey’s test using any of the software (Minitab, SPSS, or SAS).

10.7.3.

For the data of Exercise 10.5.4, perform Tukey’s test using any of the software (Minitab, SPSS, or SAS).

PROJECTS FOR CHAPTER 10 10A. Transformations The basic model for the analysis of variance requires that the independent observations come from normal populations with equal variances. These requirements are rarely met in practice, and the extent to which they are violated affects the validity of the subsequent inference. Therefore, it is important

Projects for Chapter 10 555

for the investigator to decide whether the assumptions are at least approximately satisﬁed and, if not, what can be done to rectify the situation. Hence it is necessary to (a) examine the data for marked departures from the model and, if necessary, (b) apply an appropriate transformation to the data to bring it more in line with the basic assumptions. A simple way to check for the equality of the population variances is to calculate the sample variances and plot against mean as in Figure 10.3. If the graph suggests a relation between sample mean and variance, then the relation very likely exists between population mean and variance, and hence the population from which the samples are taken may very well be nonnormal. If a study of sample means and variances reveals a marked departure from the model, the observations may be transformed into a new set to which the methods of ANOVA are better suited. Three commonly used transformations are the following: (a) The logarithmic transformation: If the graph of sample means against sample variance suggests a relation of the form 2 s2 = C X ,

replace each observation X by its logarithm to the base 10, Y = log 10 X;

or, if some X-values are zero, by Y = log 10 (X + 1). (b) The square root transformation: If the relation is of the form s2 = CX

replace X by its square root, Y=

√ X

or, if the values of X are very close to zero, by the square root of X + 1/2 . This relation is found in data from Poisson populations, where the variance is equal to the mean. (c) The angular transformation: If the observations are counts of a binomial nature, and pˆ is the observed proportion, replace pˆ by θ = arcsin

p, ˆ

which is the principal angle (in degrees or radians) whose sine is the square root of p. ˆ (i) To check for the equality of the population variances, calculate the sample variances for each of the data sets given in the exercises of Section 10.3 and plot against the corresponding mean. (ii) If there is assumptional violation, perform one of the transformations described earlier and do the analysis of variance procedure for the transformed data.

556 CHAPTER 10 Analysis of Variance

10B. Anova with Missing Observations In the two-way analysis of variance, we assumed that each block cell has one treatment value. However, it is possible that some observations in some block cells may be missing for various reasons, such as that the investigator failed to record the observations, the subject discontinued participation in the experiment, or the subject moved to a different place or died prior to completion of the experiment. In those cases, this project gives a method of inserting estimates of the missing values. Let y.. denote the total of all kb observations. If the observation corresponding to the ith row and the jth column, which is denoted by yij ., is missing, then all the sums of squares are calculated as before, except that the yij term is replaced by yˆ ij =

bBj + kTi − y .. (k − 1) (b − 1)

,

where Ti denotes the total of b−1 observations in the ith row, Bj denotes the total of k−1 observations in the jth column, and y .. denotes the sum of all kb − 1 observations. Using calculus, one can show that yˆ ij minimizes the error sum of squares. One should not include these estimates when computing relevant degrees of freedom. With these changes, proceed to perform the analysis as in Section 10.4. For more details on the method, refer to Sahai and Ageel (2000), p. 145. Perform the test of Example 10.4.1, now with a missing value for material III and chemical C4 . Does the conclusion change?

10C. ANOVA in Linear Models In order to determine whether the multiple regression model introduced in Section 8.5 is adequate for predicting values of dependent variable y, one can use the analysis of variance F -test. The model is Y = β0 + β1 x1 + β2 x2 + · · · + βk xk + ε,

where ε = (ε1 , ε2 , . . . , εn ) ∼ N 0, σ 2 and εi and εj are uncorrected if i = j. Deﬁne the multiple coefﬁcient of determination, R2 , by 2 yi − yˆ i 2 R =1− . (yi − y)2

The Analysis of Variance F-Test H0 : β1 = β2 = . . . = βk = 0 versus Ha : At least one of the parameters, β1 , β2 , . . . , βk , differs from 0.

Projects for Chapter 10 557

Test statistic: Mean square for model Mean square for error SS model /k = SSE/[n − (k + 1)] > R2 k = , > 1 − R2 [n − (k + 1)]

F=

where n = number of observations k = number of parameters in the model excluding β0 .

From the F -table, determine the value of Fα with k numerator d.f. and n − (k + 1) denominator d.f. Then the rejection region is {F > Fα }. If we reject the null hypothesis, then the model can be taken as useful in predicting values of y. For the data of Example 8.5.1, test the overall utility of the ﬁtted model y = 66.12 − 0.3794X1 + 21.4365X2

using the F -test described earlier.

This page intentionally left blank

Chapter

11

Bayesian Estimation and Inference Objective: To study Bayesian analysis methods and procedures that are becoming very popular in building statistical models for real-world problems. 11.1 Introduction 560 11.2 Bayesian Point Estimation 562 11.3 Bayesian Conﬁdence Interval or Credible Intervals 11.4 Bayesian Hypothesis Testing 584 11.5 Bayesian Decision Theory 588 11.6 Chapter Summary 596 11.7 Computer Examples 596 Projects for Chapter 11 596

579

The Reverend Thomas Bayes (Source: http:en.wikipedia.org/wiki/Thomas_Bayes)

Mathematical Statistics with Applications Copyright © 2009 by Academic Press, Inc. All rights of reproduction in any form reserved.

559

560 CHAPTER 11 Bayesian Estimation and Inference

The Reverend Thomas Bayes (1702–1761) was a Nonconformist minister. In the 1720s Bayes started working on the theory of probability. Even though he did not publish any of his works on mathematics during his lifetime, Bayes was elected a Fellow of the Royal Society in 1742. His famous work titled “Essay toward solving a problem in the doctrine of chances” was published in the Philosophical Transactions of the Royal Society of London in 1764, after his death. The paper was sent to the Royal Society by Richard Price, a friend of Bayes. Another mathematical publication on asymptotic series also appeared after his death.

11.1 INTRODUCTION Bayesian procedures are becoming increasingly popular in building statistical models for real-world problems. In recent years, the Bayesian statistical methods have been increasingly used in scientiﬁc ﬁelds ranging from archaeology to computing. Bayesian inference is a method of analysis that combines information collected from experimental data with the knowledge one has prior to performing the experiment. Bayesian and classical (frequentist) methods take basically different outlooks toward statistical inference. In this approach to statistics, the uncertainties are expressed in terms of probabilities. In the Bayesian approach, we combine any new information that is available with the prior information we have, to form the basis for the statistical procedure. The classical approach to statistical inference that we have studied so far is based on the random sample alone. That is, if a probability distribution depends on a set of parameters θ, the classical approach makes inferences about θ solely on the basis of a sample X1 , . . . , Xn . This approach to inference is based on the concept of a sampling distribution. To correctly interpret traditional inferential procedures, it is necessary to fully understand the notion of a sampling distribution. In this approach, we analyze only one set of sample values. However, we have to imagine what could happen if we drew a large number of random samples from the population. For example, consider a normal sample with known variance. We have seen that a 95% conﬁdence interval for the population mean μ is given by the random √ √ interval X − 1.96σ/ n, X + 1.96σ/ n . This means that when samples are repeatedly taken from the population, at least 95% of the random intervals contain the true mean μ. The classical inferential approach does not use any of the prior information we might have as a result of, say, our familiarity with the problem, or information from earlier studies. Scientists and engineers are faced with the problem that there is typically only a single data set, and they need to determine the value of the parameter at the time the data are taken. The basic question then is, “What is the best estimate of a parameter one can make from the data using one’s prior information?” Statistical approaches that use prior knowledge, possibly subjective, in addition to the sample evidence to estimate the population parameters are known as Bayesian methods. Bayesian statistics provides a natural method for updating uncertainty in the light of evidence. Data are still assumed to come from a distribution belonging to a known parametric family. However, the Bayesian outlook toward inference is founded on the subjective interpretation of probability. Subjective probability is a way of stating our belief in the validity of a random event. The following example will illustrate the idea. Suppose we are interested in the proportion of all undergraduate students at a particular university who take on out-of-campus jobs for at least 20 hours a week. Suppose we randomly select, say, 50 students from this university and obtain the proportion of

11.1 Introduction 561

students who have out-of-campus jobs for at least 20 hours a week. Let us assume that the sample proportion is 30/50 = 0.6. In a frequentist approach, all of the inferential procedures, such as point estimation, interval estimation, or hypothesis testing, are based on the sampling distribution. That is, even though we are analyzing only one data set, it is necessary to have the knowledge of the mean, standard deviation, and shape of this sampling distribution of the proportion for the correct interpretation in classical inferential procedures. In the subjective interpretation of probability, the proportion of undergraduates who work on an out-of-campus job for at least 20 hours a week is assumed to be unknown and random. A probability distribution, called the prior, that represents our knowledge or belief about the location of this proportion before any data collected is used. For instance, the college placement ofﬁce already may have an opinion on this proportion based on its earlier experience. The classical approach ignores this prior knowledge, whereas the Bayesian approach incorporates this knowledge with the current observed data to update the value of this proportion. That is, after the data are collected our opinion about the proportion may change. Using Bayes’ rule, we will compute the posterior probability distribution for the proportion, based on our prior belief and evidence from the data. All of our inferences about the proportion are made by computing appropriate statistics of the posterior distribution. The Bayesian approach seeks to optimally merge information from two sources: (1) knowledge that is known from theory or opinion formed at the beginning of the research in the form of a prior, and (2) information contained in the data in the form of likelihood functions. Basically, the prior distribution represents our initial belief, whereas the information in the data is expressed by the likelihood function. Combining prior distribution and likelihood function, we can obtain the posterior distribution. This expresses our revised uncertainty in light of the data. The main difference between the Bayesian approach and the classical approach is that in the Bayesian setting, the parameter is viewed as random variables, whereas the classical approach considers the parameter to be ﬁxed but unknown. The parameter is random in the sense that we can assign to it a subjective probability distribution that describes our conﬁdence about the actual value of the parameter. Some of the reasons for Bayesian approaches are as follows: (1) Most Bayesian inferential conclusions are made conditional on the observed data. Unlike the traditional approach, one need not be concerned with data sets other than the one that is observed. There is no need to discuss sampling distributions using the Bayesian approach. Also, (2) from a Bayesian viewpoint, it is legitimate to talk about the probability that the proportion falls in a speciﬁc interval, say (0.2, 0.6), or the probability that a hypothesis is true. Too often, traditional inferential conclusions are misstated; for example, if a conﬁdence interval computed from a sample for a parameter is (0.2, 0.6), it is common for the student to incorrectly state that the population parameter falls in the interval (0.2, 0.6) with probability at least 0.90. The Bayesian viewpoint provides a convenient model for implementing the scientiﬁc method. The prior probability distribution can be used to state initial beliefs about the population of interest, relevant sample data are collected, and the posterior probability distribution reﬂects one’s new updated beliefs about the population parameter in light of the new data that were collected. All inferences about the parameter are made by computing appropriate summaries of the posterior probability distribution. Because of formidable theoretical and computational challenges, the Bayesian approach has found relatively limited use. Recent advances in Bayesian analysis combined with the

562 CHAPTER 11 Bayesian Estimation and Inference

growing power of computers are making Bayesian methods practical and increasingly popular. The Markov chain Monte Carlo (MCMC) method described in Section 13.5 is one of the computationally intensive methods that is often useful in Bayesian estimation.

11.2 BAYESIAN POINT ESTIMATION The cornerstone of Bayesian methodology is the Bayes theorem. It helps us to update our beliefs in the form of probability statements about the parameters after the sample has been taken. The conditional distribution of the parameters after observing the data is called the posterior distribution that integrates the prior and the sample information. Suppose we have two discrete random variables, X and Y . Then the joint probability function (pmf ) can be written as p(x, y) = p(x |y)pY (y) , and the marginal probability density function of X is pX (x) = y p(x, y) = y p(x |y)pY (y) . Then Bayes’ rule for the conditional p(y |x ) is p (y |x) =

p (x, y) p (x |y) pY (y) p (x |y) pY (y) = = . pX (x) pX (x) p (x |y) pY (y) y

The denominator in this expression is a ﬁxed normalizing factor that ensures that the If Y is continuous, the Bayes theorem can be stated as p (y |x ) =

y

p (y |x ) = 1.

p (x |y) pY (y) , p (x |y) pY (y) dy

where the integral is over the range of values of y. These two equations are the Bayes formulas for random variables. In Bayesian terminology, pY (y) represents the probability statement of our prior belief, p(x|y) is the probability of the data x given our prior beliefs, which is called the likelihood, and the updated probability p(y|x) is the posterior. Because pX (x) (which is the likelihood accumulated over all possible prior values) is independent of y, we can express the posterior distribution as proportional (∝) to [(likelihood) × (prior distribution)], that is, p(y|x) ∝ p(x|y)p(y).

We use the notation f (x|θ) to represent a probability distribution whose population parameter is considered to be a random variable. Now one of the problems is of ﬁnding a point estimate of the parameter θ (possibly a vector) for the population with distribution f (x|θ), given θ. Assume that π(θ) is the prior distribution of θ, which reﬂect the experimenter’s prior belief about θ. We will not distinguish between the scalars and vectors, which will be clear based on the speciﬁc situation. Suppose that we have a random sample X = (X1 , . . . , Xn ) of size n from f (x|θ). Then the posterior distribution can be written as f (θ|X1 , . . . , Xn ) =

L(X1 , . . . , Xn |θ)π(θ) f (θ, X1 , . . . , Xn ) = , f (X1 , . . . , Xn ) f (X1 , . . . , Xn )

11.2 Bayesian Point Estimation 563

where L(X1 , . . . , Xn |θ) is the likelihood function. Letting C represent all terms that do not involve θ (in this case, C = 1/f (X1 , . . . , Xn )), we have f (θ |X1 , . . . , Xn ) = CL(X1 , . . . , Xn |θ )π(θ),

For speciﬁc sample values X1 = x1 , X2 = x2 , . . . , Xn = xn , the foregoing equation can be written in a compact form as f (θ |x ) ∝ f (x |θ )π(θ),

where

x = (x1 , x2 , . . . , xn ).

This can be expressed as

posterior distribution ∝ prior distribution × likelihood .

The full result including the normalization can be written as (posterior distribution) = [(prior distribution) × (likelihood)] /

(prior × likelihood)

where the denominator is a ﬁxed normalizing factor obtained by the likelihood accumulated over all possible prior values. We can now give a formal deﬁnition. Deﬁnition 11.2.1 The distribution of θ, given data x1 , x2 , . . . , xn , is called the posterior distribution, which is given by π(θ |x ) =

f (x |θ )π(θ) , g(x)

(11.1)

where g (x) is the marginal distribution of X. The Bayes estimate of the parameter θ is the posterior mean. The marginal distribution g(x) can be calculated using the formula

g(x) =

⎧ ⎪ f (x|θ)π(θ), ⎪ ⎪ ⎨θ

in discrete case

∞ ⎪ ⎪ ⎪ f (x|θ)π(θ)dθ, ⎩

in continuous case

−∞

where π(θ) is the prior distribution of θ. Here, the marginal distribution g (x) is also called the predictive distribution of X, because it represents our current predictions of the values of X taking into account both the uncertainty about the value of θ and the residual uncertainty about the random variable X when θ is known. In a Bayesian setting, all the information about θ from the observed data and from the prior knowledge is contained in the posterior distribution, π(θ|x). In almost all practical cases, because we are combining our prior information with the information contained in the data, the posterior distribution provides a more reﬁned estimation of θ than the prior. All inferences from Bayesian methods are based on the posterior probability distribution of the parameter θ. Using the explanation given later, we will take the Bayes estimate of a parameter as the posterior mean.

564 CHAPTER 11 Bayesian Estimation and Inference

Furthermore, consider a Bayesian statistical inference problem where the parameter is a population proportion. In the Bernoulli trials, the population contains two types called “successes” and “failures.” The proportion of successes in the population is denoted by θ. We take a random sample of size n from the population and observe s successes and f failures. The goal is to learn about the unknown proportion θ on the basis of these data. In this situation, a model is represented by the population proportion θ. We do not know its value. In Chapter 5, we have seen that we could use the maximum likelihood estimator (MLE) for estimating θ, which did not use any prior knowledge we may have about θ. Note that the maximum likelihood estimate is broadly equivalent to ﬁnding the mode of the likelihood. In a Bayesian setting, we represent our beliefs about location of θ in terms of a prior probability distribution. We introduce proportion inference by using a discrete prior distribution for θ. We can construct a prior by specifying a list of possible values for the proportion θ, and then assigning probabilities to these values that reﬂect our knowledge about θ. Then the posterior probabilities can be computed using the Bayes theorem. The following example illustrates this concept.

Example 11.2.1 It is believed that cross-fertilized plants produce taller offspring than the self-fertilized plants. In order to obtain an estimate on the proportion θ of cross-fertilized plants that are taller, an experimenter observes a random sample of 15 pairs of plants that are exactly the same age. Each pair is grown in the same conditions with some cross-fertilized and the others self-fertilized. Based on previous experience, the experimenter believes that the following are possible values of θ and that the prior probability for each value of θ (prior weight) is π(θ). θ: π(θ):

0.80 0.13

0.82 0.15

0.84 0.22

0.86 0.25

0.88 0.15

0.90 0.10

From the experiment, it is observed that in 13 of 15 pairs, cross-fertilized is taller. Create a table with columns of the prior π(θ), likelihood of L(X1 , X2 , . . . , Xn |θ) for different values of θ and for the given sample, prior times likelihood, and posterior probability of θ. Based on the posterior probabilities, what value of θ has the highest support? Also, ﬁnd E(θ) based on the posterior probabilities.

Solution The likelihood of obtaining 13 of 15 taller plants to the different prior values of π are given using the binomial 15 pdf θ 13 (1 − θ)2 . For example, if the prior value of θ is 0.80, then the likelihood of θ given the 13 sample is

15 f (x|θ) = (0.8)13 (0.2)2 = 0.2309. 13

11.2 Bayesian Point Estimation 565

Table 11.1 Prior values Prior Likelihood of θ Prior times Posterior of θ π(θ) given sample likelihood probability of θ 0.80

0.13

0.2309

3.0017×10−2

0.11029

0.82

0.15

0.2578

0.03867

0.14208

0.2787

6.1314×10−2

0.22528 0.2661

0.84

0.22

0.86

0.25

0.2897

7.2425×10−2

0.88

0.15

0.2870

0.4305

0.15817

0.90

0.10

0.2669

0.02669

0.098064

0.27217

0.9998 ≈ 1.0

Total

From Table 11.1 we obtain (prior × likelihood) = 0.27217. Hence, the normalized value corresponding to θ = 0.80 is the posterior probability f (θ|x), which is equal to (0.030017/0.27217) = 0.11029. Now, we can obtain the table of posterior distribution ofa proportion π using the discrete prior given in Table 11.1. 15 When we substitute in Bayes’ rule, the factor would be canceled. Hence, in the calculation of the 13 15 13 13 2 θ (1 − θ)2 . likelihood function, we could have just used θ (1 − θ) instead of the full expression 13 Thus, the Bayesian estimate of θ is E(θ) = (0.8)(0.11029) + (0.82)(0.14028) + (0.84)(0.22528) + (0.86)(0.2661) + (0.88)(0.15817) + (0.9)(0.098065) = 0.84879 ≈ 0.85. It may be noted that the MLE of θ is 13/15 = 0.867.

In Example 11.2.1, the priors are called informative priors, because it favored certain values of θ; for example for the value θ = 0.86, the prior value of π (θ) is 0.25, which is higher than all the rest of the values. If there was no information or no strong prior opinions, then we could select a noninformative prior, which would have assigned equal prior probability of 1/6 to each of the possible values of θ. A noninformative prior (also called a ﬂat or uniform prior) provides little or no information. Based on the situation, noninformative priors may be quite disperse, may avoid only impossible values of the parameter, and oftentimes give results similar to those obtained by classical frequentist methods.

566 CHAPTER 11 Bayesian Estimation and Inference

Example 11.2.2 Repeat the Example 11.2.1 using a noninformative prior, π(θ) = 1/6, for each given value of θ.

Solution Here π(θ) = 16 for each value of θ. See Table 11.2.

Table 11.2 Prior values of θ

Prior π(θ)

Likelihood of θ given sample

Prior times likelihood

Posterior probability of θ

0.80

1/6

0.2309

3.8483×10−2

0.14333

0.82

1/6

0.2578

4.2967×10−2

0.16003

0.84

1/6

0.2787

0.04645

0.86

1/6

0.2897

4.8283×10−2

0.17982

0.88

1/6

0.2870

4.7833×10−2

0.17815

0.90

1/6

0.2669

4.4483×10−2

0.16567

0.2685

1.0

Total

0.173

The Bayesian estimate for the noninformative prior is E(θ) = (0.8)(0.14333) + (0.82)(0.16003) + (0.84)(0.173) + (0.86)(0.17982) + (0.88)(0.17815) + (0.9)(0.16567) = 0.85173.

It should be noted that because the choice of priors in Example 11.2.1 is only mildly informative, we do not see much difference in the values of Bayesian estimates. In general, it is difﬁcult to construct an acceptable prior, because most often it has to be based on subjective experiences. Therefore, it is relatively easy to use a “noninformative” prior. For example, if we have no information on the values of proportion θ, then one type of standard “noninformative” prior is to take the proportion θ as one of the equally spaced values 0, 0.1, 0.2, . . . , 0.9, 1. We can assign for each value of θ the same probability, π(θ) = 1/11. This prior is convenient and may work reasonably well when we do not have many data. It is fairly easy to construct a prior when there exists considerable prior information about the proportion of interest. The posterior distribution gives us information regarding the likelihood of values of θ given sample data. Then the question is how to use this information to estimate θ. Instead of having explicit probabilities, the prior may be given through an assumed probability distribution. We illustrate the calculations involved to ﬁnd the posterior distribution in the following example.

11.2 Bayesian Point Estimation 567

Example 11.2.3 Let X be a binomial random variable with parameters n and p. Assume that the prior distribution of p is uniform on [0,1]. Find the posterior distribution, f (p|x).

Solution Because X is binomial, the likelihood function is given by n x f (x|p) = p (1 − p)n−x . x Because p is uniform on [0,1], π(p) = 1, 0 ≤ p ≤ 1. Then the posterior distribution is given by n x f (p|x) ∝ f (x|p)π(p) = p (1 − p)n−x , x = 0, 1, . . . , n x which is the same as the likelihood.

This example illustrates that if the prior is noninformative (uniform), then the posterior is essentially the likelihood function. In the case where the prior and posterior are of the same functional form, we call it a conjugate prior. Bayesian inference becomes simpler when the prior density has the same functional form as the likelihood (which is the case for the conjugate prior) or when data are an independent sample from an exponential family (such as normal, Poisson, or binomial). The following example demonstrates the method of ﬁnding posterior distribution for a continuous random variable.

Example 11.2.4 Suppose that X is a normal random variable with mean μ and variance σ 2 , where σ 2 is known and μ is unknown. Suppose that μ behaves as a random variable whose probability distribution (prior) is π(μ) and is also normally distributed with mean μp and variance σp2 , both assumed to be known or estimated. Find the posterior distribution f (μ|x).

Solution Using the Bayes theorem, we have f (μ|x) =

f (x|μ)π(μ) f (x|μ)π(μ)dμ

2 2 2 2 √ 1 e−(x−μ) /2σ √ 1 e−(μ−μp ) /2σp 2πσ 2πσp = 2 2 2 2 √ 1 e−(x−μ) /2σ √ 1 e−(μ−μp ) /2σp dμ 2πσ 2πσp ! " 2 (μ−μp )2 − (x−μ) 1 2 + 2

=

2πσσp

e

2σ

2σp

.

(11.2)

568 CHAPTER 11 Bayesian Estimation and Inference

2 (μ−μ )2 Consider the exponential term in (11.2), namely, (x−μ) + 2σ 2p . 2σ 2 p

(μ − μp )2 1 (x − μ)2 + = 2 2 2 2σ 2σp 1 = 2 1 = 2

%

1 1 + 2 2 σ σp

%

(μ − μp )2 (x − μ)2 + 2 σ σp2

μ2 − 2

μp x + 2 2 σp σ

&

μ+

μ2p x2 + σ2 σp2

&

& % 2 σp + σ 2 2 μ2p μp x2 x μ −2 + 2 μ+ + 2 σ 2 σp2 σp2 σ σ2 σp

% 2 2 σ 2 σp2 μp x 1 σp + σ 2 + 2 μ μ −2 2 = 2 σ 2 σp2 σp + σ 2 σp2 σ

& μ2p x2 + 2 + 2 σp + σ 2 σ 2 σp % 2 2 σp2 σ2 1 σp + σ 2 μp + 2 x μ μ −2 = 2 σ 2 σp2 σp2 + σ 2 σp + σ 2 2 ⎤ σp2 σ2 μp + 2 x ⎦ + σp2 + σ 2 σp + σ 2 σ 2 σp2

⎡ 2 2 2 μ2p σp2 σ2 1 σp + σ ⎣ x2 + 2 − x+ 2 μp + 2 σ 2 σp2 σ2 σp σp2 + σ 2 σp + σ 2 % &2 2 2 σp2 σ2 1 σp + σ ˜ μ− = μp + 2 x + K, 2 σ 2 σp2 σp2 + σ 2 σp + σ 2 where ⎡ 2 ⎤ 2 2 μ2p σp2 1 σp + σ ⎣ x2 σ2 ˜ = K + 2 − μp + 2 x ⎦. 2 σ 2 σp2 σ2 σp σ 2 + σp2 σ + σp2 From the foregoing derivation, we obtain − 12

f (μ| x) = Ke

2 +σ 2 σp 2 σ 2 σp

!

μ−

2 σp σ2 2 +σ 2 μp + σ 2 +σ 2 x σp p

"2

,

where K does not contain μ. This implies that the posterior density f (μ |x ) is the pdf of normal random variable with mean σp2 σ2 μp + 2 x σp2 + σ 2 σp + σ 2

11.2 Bayesian Point Estimation 569

and variance σ 2 σp2 σp2 + σ 2

.

If we let τp = σ12 and τ = σ12 , then the posterior density can be rewritten as the pdf of normal random p variable with mean τp1+τ τp μp + τx and variance τp1+τ . As an example, suppose that μp = 100, σp = 15, and σ = 10, x = 115. Then f (μ |x ) is the pdf of a normal random variable with Mean =

225 100 (100) + (115) = 110.4 100 + 225 100 + 225

and Variance =

(100)(225) = 69.2. 100 + 225

11.2.1 Criteria for Finding the Bayesian Estimate In the Bayesian approach to parameter estimation, we use both the prior and observations. This leads to an estimation strategy based on the posterior distribution. How do we know that the estimate thus obtained is “good”? To assess the quality of likely estimators, we use a loss function L (θ, a) that measures the loss incurred by using a as an estimate of θ. Here θ is the parameter being estimated (in real-world problems it is not known), and a is the estimate of θ. Then the “optimal” or “best” estimate a = θˆ is chosen so as to minimize the expected loss E[L(θ, θˆ )], where the expectation is taken over θ with respect to the posterior distribution f (θ |x ). Here we mention two types of commonly used loss functions: quadratic and absolute error loss functions and the resulting estimates. (1) A quadratic (or squared error) loss function is of the form L(θ, a) = (a − θ)2 . In this case, E [L(θ, a)] =

L(θ, a)f (θ|x1 , . . . , xn )dθ

=

(a − θ)2 f (θ|x1 , . . . , xn )dθ.

Differentiating with respect to a and equating to zero, we obtain 2

(a − θ) f (θ |x1 , . . . , xn ) dθ = 0

This implies

a=

θf (θ |x1 , . . . , xn ) dθ.

This is the posterior mean (expected value) of θ, E (θ |x1 , . . . , xn ). Hence the quadratic loss function is minimized by taking the estimate of θ, that is, θˆ , to be the posterior mean. In previous examples in this section, we used this value as the estimate θˆ . Note that what the quadratic loss function displays

570 CHAPTER 11 Bayesian Estimation and Inference

is that if the estimate θˆ and the true parameter θ are close to each other, the loss we expect is very small. Likewise, if the difference is larger, the expected loss in estimating θ with θˆ is going to be large. (2) An absolute error loss function is of the form L (θ, a) = |a − θ|. In this case, L(θ, a)f (θ |x1 , . . . , xn )dθ

E [L(θ, a)] =

a (a − θ) f (θ |x1 , . . . , xn )dθ

= θ=−∞

+

∞ (θ − a)f (θ |x1 , . . . , xn )dθ

θ=a

Differentiating with respect to a and equating to zero, we obtain a f (θ |x1 , . . . , xn ) dθ − θ=−∞

∞ f (θ |x1 , . . . , xn ) dθ = 0

θ=a

The minimum loss is attained when the values of both integrals are equal to 12 . This can be achieved by taking θˆ to be the posterior median. The following can be considered as a general Bayesian procedure for point parameter estimation.

BAYESIAN PARAMETER ESTIMATION PROCEDURE 1. Consider the unknown parameter θ as a random variable. 2. Use a probability distribution(prior) to describe the uncertainty about the unknown parameter. 3. Update the parameter distribution using the Bayes theorem: P(θ|Data) ∝ P(θ)P(Data|θ), that is, (posterior of θ) ∝ (prior of θ).(likelihood). 4. The Bayes estimator of θ is set to be the expected value of the posterior distribution P(θ |Data) under quadratic loss function. 5. The Bayes estimator of θ is set to be posterior median under absolute error loss function.

From the procedure of Bayesian estimation, it is clear that a bad choice of prior may result in a bad estimate. Generally, if the priors are based on a previous and trustworthy sample, Bayesian estimation methods are desirable. A schematic ﬁgure of steps involved in the Bayesian estimate is given in Figure 11.1.

11.2 Bayesian Point Estimation 571

Prior info, P()

Posterior P(| Data)

Loss function

Updated

Likelihood P(Data |) ■ FIGURE 11.1 Bayesian estimation procedure.

In this chapter, we use only the quadratic loss function unless it is explicitly stated otherwise. We also mention that this loss function is very popular because of its analytic tractability. We now derive Bayesian point estimates for some speciﬁc distributions. Whereas uniform priors are useful in the noninformative situations, the beta family of distributions is one of the commonly taken informative priors. Distributions in the beta family take values in the interval (0, 1). Recall that if X ∼ beta(α, β), then the pdf of X is given by f (x) =

(α+β) α−1 (1 − x)β−1 , (α) (β) x

0≤x 0, β > 0.

The beta pdf can be written as f (x) = Cxα−1 (1 − x)β−1 ∝ xα−1 (1 − x)β−1 ,

where C =

(α+β) (α) (β) .

We also know that E (X) =

α , α+β

and

Var (X) =

αβ . (α + β)2 ((α + β + 1)

Example 11.2.5 Let X1 , . . . , Xn be a sample from geometric distribution with parameter p, 0 ≤ p ≤ 1. Assume that the prior distribution of p is beta with α = 4, and β = 4. (a) Find the posterior distribution of p. (b) Find the Bayes estimate under quadratic loss function.

Solution (a) Because p is Beta(4, 4), the prior density is (8) p3 (1 − p)3 = 140p3 (1 − p)3 . (4) (4)

572 CHAPTER 11 Bayesian Estimation and Inference

Because the r.v.’s Xi s have geometric distribution with parameter p, the likelihood is given by L(X1 , . . . , Xn |θ ) =

n 7

n xi −n x −1 n i i=1 p (1 − p) = p (1 − p) .

i=1

The product of the likelihood function and the prior is given by n n xi −n xi −n+3 n 3 3 n+3 (1 − p)i=1 140p (1 − p) = 140p . p (1 − p)i=1

Because, (posterior of p) ∝ (prior ofp) . (likelihood), rewriting the normalizing constant in the denominator of Equation (11.1) as C, and letting C1 = 140C, the posterior distribution (because n xi − n + 4 . α − 1 = n + 3, and β − 1 = ni=1 xi − n + 3) is Beta n + 4, i=1

(b) Recall that for a Beta(α, β) random variable, the mean is [α/(α + β)]. Because the Bayes estimate n xi − n + 4 is is the posterior mean, the mean of Beta n + 4, i=1

%

n

n+4 = n xi + 8 xi − n + 4 + (n + 4) n+4 &

i=1

i=1

Note that for large n, the Bayes estimate is approximately n/ ni=1 xi , which is the MLE of p. In general, for a Bernoulli random variable with unknown probability of success p in [0,1], the usual conjugate prior is the beta distribution, where the parameters of the beta distribution are chosen to reflect any prior information that we have. We will follow the idea of the previous example in a binomial experiment of tossing a coin.

Example 11.2.6 Suppose we are ﬂipping a biased coin, where the probability of heads p could be any value between 0 and 1. Given a sequence of toss samples x1 , x2 , . . . , xn , we want to estimate P (H) = p. We may have two sources of information: our prior belief, which we will express as a beta distribution, and the data, which could come from counts of heads x in n = 20 independent ﬂips of the coin, say x = 13. Suppose that in six prior tosses, we observed three heads and three tails, which lead us to believe that the value of p is near 0.5. Obtain the posterior distribution of p.

Solution Here our prior belief or assumption can be written in terms of beta distribution as π (p) =

(α + β) α−1 (1 − p)β−1 p (α) (β)

where α = 4 and β = 4. That is (noting (n) = (n − 1)!) π(p) =

7! p3 (1 − p)3 . (3!)(3!)

11.2 Bayesian Point Estimation 573

Hence, π(p) ∝ p3(1 − p)3 . Because the mean of a beta distribution is α/(α + β) and the variance is αβ/ (α + β)2 (α + β + 1) , for the prior, Mean(p) =

4 = 0.5, 4+4

and Var(p) =

(4)(4) = 0.028. (4 + 4)2 (4 + 4 + 1)

Let X denote the number of heads in 20 flips of this coin. Then X has a binomial distribution, and the pmf is given by 20 x f (x|p) = p (1 − p)20−x , x = 0, 1, . . . , 20. x This we can write as f (x|p) ∝ px (1 − p)20−x . In the 20 flips we have observed 13 heads. Then fix x = 13, and we are interested in the likelihood, which is the relative value of the function at different values of p: f (13|p, 20) ∝ p13 (1 − p)7 . The posterior probability of p, given x = 13, is π(p|x = 13) ∝ f (x|p)π(p) = p13 (1 − p)20−13 p3 (1 − p)3 = p16 (1 − p)10 . Thus, the posterior is a beta distribution with α = 17 and β = 11. Consequently, we can now obtain the mean and variance of p as Mean(p) =

17 = 0.607 17 + 11

and Var(p) =

(17)(11) = 0.008. (17 + 11)2 (17 + 11 + 1)

Note that the prior was beta distribution with mean 0.5 and variance 0.028. Figure 11.2 gives the prior and posterior densities. Note that if we had ignored the prior and just took the point estimation, then the MLE of p is MLE(p) = pˆ = 13 = 0.65. Compare this with the Bayesian estimate of p = 0.607. Because Beta(1, 1) is the Uniform [0, 1], 20

574 CHAPTER 11 Bayesian Estimation and Inference

4.5 4 3.5 Posterior

(x)

3 2.5 2

Prior

1.5 1 0.5 0

0

0.2

0.4

0.6

0.8

1

p ■ FIGURE 11.2 Prior and posterior distributions for the proportions.

the method of the previous example can be used for noninformative priors. The method could also be used in many applications. For example, suppose p represents the proportion of infected individuals in a population, and x is the number of infected individuals in a sample of size n. Then with a noninformative prior, we can show that the posterior of p is Beta(x + 1, n − x + 1). This type of setting can be used for estimating the true proportion of infected individuals in the population.

Example 11.2.7 Suppose for the past million days we have been predicting whether the sun will rise the next morning or ˆ and we were right (R) all these days. not. Each evening we say that the sun will rise the next morning (R), Suppose on the 106 evenings we predicted that the sun will rise on the next day. What is the probability that the sun will rise the next day?

Solution The problem can be cast in the following table form. 1 ˆ R R

2 ˆ R R

... ... ...

106 ˆ R R

106 + 1 ˆ R ?

ˆ = 1 if we use the frequency method of estimation (for example the MLE). Let us now consider the P(R|R) Bayes method. Suppose the prior is uniform on [0,1]. That is, ⎧ ⎨1, if 0 ≤ p ≤ 1 π(p) = ⎩0, otherwise.

11.2 Bayesian Point Estimation 575

Suppose we predict n times and we succeed x times. Then n x f (x|p) = p (1 − p)n−x . x The joint pdf is given by f (x, p) = f (x|p)π(p) n x = p (1 − p)n−x , x

x = 0, 1, . . . , n;

0 ≤ p ≤ 1.

By the Bayes theorem, the posterior pdf π(p|x) is π(p|x) =

f (x|p)π(p) 1

f (x|p)π(p)dp

0

= K(n, x)px (1 − p)n−x ,

0 ≤ p ≤ 1,

0 ≤ x ≤ n,

which is a beta probability distribution. Recall that the beta density is given by f (y) =

1 yα−1 (1 − y)β−1 B(α, β)

α . Thus, and E(Y ) = α+β

E [π (p |x )] =

x+1 x+1 . = (x + 1) + (n − x) + 1 n+2

In our example, x = 106 , n = 106 , which implies that the posterior mean is given by pˆ β =

106 + 1 ≈ 1. 106 + 2

Example 11.2.8

Let X1 , X2 , . . . , Xn be N μ, σ 2 random variables with prior π (μ) having N μ0 , σ02 distribution with known σ 2 . (a) Obtain the posterior distribution of μ. (b) Suppose it is known from past experience that the weight loss for a particular combination of diet and exercise program (if followed for a month) is normally distributed with mean 10 lb and standard deviation of 2 lb. A random sample of ﬁve persons who went through this program for a month produced the following weight loss in pounds: 14

8

11

7

11

What is the point estimate of the mean, μ? Assume σ 2 = 4.

576 CHAPTER 11 Bayesian Estimation and Inference

Solution

# $ (a) Because π (μ) ∼ N μ0 , σ02 , π (μ) ∝ exp (μ − μ0 )2 /σ02 and we omit the terms that do not depend on μ. We have from the data x = (x1 , . . . , xn ), the likelihood function, 4 n 7 (xi − μ)2 L (x1 , . . . , xn |μ ) = f (x |μ ) ∝ exp − 2σ 2 i=1 4 n 2 2 (xi − μ) /2σ = exp − , i=1

where μ is determined by the posterior distribution. The product of the likelihood function and the prior gives the posterior, which is obtained (after some algebra) as follows: f (μ|x) ∝ π(μ)f (x|μ) ∝ exp − (μ − μ1 )2 /2σ12 where μ1 =

n x+ 1 μ σ2 σ02 0 n + 1 σ2 σ02

and 1 σ12 = n . 1 + 2 σ2 σ0 2 Thus, the posterior distribution of μ is N μ1 , σ1 . (b) Note that the sample mean x = 10.2 lb, and sample standard deviation s = 2.77 lb. Now from part (a), the posterior distribution of μ is normal with mean n x+ 1 μ 5 (10.2) + 1 (10) σ2 σ02 0 22 22 μ1 = = = 10.167 n + 1 5 + 1 σ2 22 22 σ02

and variance 1 1 σ12 = n = 5 = 0.66667. 1 + + 212 σ2 22 σ02

Thus, the point estimate of μ is the posterior mean, 10.167. Figure 11.3 represents the prior and posterior densities of μ. Sometimes, the inverse of variance in the normal distribution is called the precision of the normal distribution and denoted by τ = 1/σ 2 . Also note that in part (a) of the previous example, if the prior variance σ02 → ∞, then the prior ﬂattens out, π(μ) ∝ c, a constant. This basically amounts to saying that prior information on μ decreases, that is, all μ are equally probable. This corresponds 2 to a noninformative prior. Also, in this case as σ02 → ∞, σ12 → σn and μ1 → x. Hence, in the limit

11.2 Bayesian Point Estimation 577

0.5 0.45 0.4 0.35 ()

0.3

Posterior

0.25 0.2 Prior

0.15 0.1 0.05 0

4

6

8

10

12

14

16

■ FIGURE 11.3 Prior and posterior densities of μ.

(i.e., for noninformative priors), the posterior f (μ|x) will have an N(x, σ 2 /n) distribution, which is exactly the same inference as in classical statistics. In Bayesian inference problems, one of the questions is, which will have relatively more inﬂuence, prior or likelihood? As we observe a large amount of data, it can be shown that the posterior distribution is almost exclusively determined by the data. That is, asymptotically, observed data will have a larger inﬂuence compared to the choice of prior, and thus the prior will be irrelevant. Hence, we can make the following general observations. If the prior is noninformative and we have a large data set, then we can expect that the likelihood will have greater inﬂuence. Whereas, if we have a small data set and an informative prior, then the prior will have a larger inﬂuence on the updated posterior distribution. Bayesian estimators are more complicated to compute than calculating the maximum likelihood estimates in simple cases. However, in complex settings Bayesian statistics are often relatively easier to compute. One of the problems in using Bayesian analysis is choosing an appropriate prior. There are no speciﬁc rules available for this purpose. For instance, the following priors are commonly used in the literature. If data are in [0,1], we could use uniform or beta distribution. If the data are in [0, ∞), normal (with nonnegative and relatively large μ), gamma, or log-normal distributions are used. If the data are in (−∞, ∞), normal or t-distributions are commonly used.

EXERCISES 11.2 11.2.1.

Suppose in a casino, two kinds of dice are used, one kind of which 98% are fair, and 2% are loaded such that ﬁve comes up 60% of the time and the rest of the numbers are equally probable. We pick a die at random and roll it three times. We get three consecutive ﬁves. What is the probability that the die is loaded?

578 CHAPTER 11 Bayesian Estimation and Inference

11.2.2.

It is believed that cross-fertilized plants produce taller offspring than self-fertilized plants. In order to obtain an estimate on the proportion θ of cross-fertilized plants that are taller, an experimenter observes a random sample of 15 pairs of plants exactly the same age, with each pair grown in the same conditions with one cross-fertilized and the other self-fertilized. Based on previous experience, the experimenter believes that the following are possible values of π and prior probabilities for each value (prior weight), π(θ): θ: π (θ):

0.80 0.03

0.82 0.40

0.84 0.22

0.86 0.15

0.88 0.15

0.90 0.05

From the experiment, it is observed that in 13 of 15 pairs, the cross-fertilized is taller. (a) Create a table with columns for prior, likelihood of θ given sample, prior times likelihood, and posterior probability of θ. Based on the posterior probabilities, what value of θ has the highest support? Also, ﬁnd E(θ) based on the posterior probabilities. (b) Redo part (a) with a completely noninformative prior, that is, take the prior for the proportion θ as one of the equally spaced values 0, 0.1, 0.2, . . . , 0.9, 1. Also assign for each value of θ the same probability, π(θ) = 1/11. (c) Calculate the MLE of θ and compare it with the Bayesian estimate. 11.2.3.

Consider the problem of estimating p in a binomial distribution. Let X be number of successes in a sample of size n. (a) Let the prior distribution of p be given by Beta(3,1), that is π (p) =

3p2 , 0,

0 1.

586 CHAPTER 11 Bayesian Estimation and Inference

This method of hypothesis testing is called Jeffreys’ hypothesis testing criterion. It basically says that if the posterior odds ratio is greater than 1, we accept the null hypothesis; otherwise, we reject the null in favor of the alternative hypothesis. Because we cannot determine the probability of a single value in the continuous variable case, it should be noted that for a simple null hypothesis of the form θ equals some speciﬁed value cannot be dealt with easily in the Bayesian framework. Hence, unlike the classical framework, here we mostly deal with the composite hypotheses for both null and alternative.

Example 11.4.1 A student taking a standardized test is classiﬁed as gifted if he or she scores at least 100 out of a possible score of 150. Otherwise the student is classiﬁed as not gifted. Suppose the prior distribution of the scores of all students is a normal with mean 100 and standard deviation 15. It is believed that scores will vary each time the student takes the test and that these scores can be modeled as a normal distribution with mean μ and variance 100. Suppose the student takes the test and scores 115. Test the hypothesis that the student can be classiﬁed as a gifted student.

Solution The hypothesis testing problem can be phrased as H0 : θ < 100 vs. Ha : θ ≥ 100. Referring to the Example 11.2.8, we know that the posterior distribution f (θ|x) is a normal with mean 110.4 and variance 69.2. Because the prior is an N(100, 225), we have π0 = P(θ < 100) = 1/2 and π1 = P(θ ≥ 100) = 1/2. We can now compute α0 = P (θ < 100 |x = 115 )

100 − 110.4 θ − 110.4 < =P √ √ 69.2 69.2

10.4 = 0.106 = P z ≤ −√ 69.2 and α1 = P (θ ≥ 100 |x = 115 ) = 1 − P (θ < 100 |x = 115 ) = 1 − 0.106 = 0.894. Thus, α0 /α1 = (0.106/0.894) = 0.119 < 1, and we reject H0 .

11.4 Bayesian Hypothesis Testing 587

BAYESIAN HYPOTHESIS TESTING PROCEDURE To test H0 : θ ∈ 0 vs. H1 : θ ∈ 1 , where 0 and 1 are given sets: 1. Consider θ as a random variable with prior distribution π(θ). 2. Compute the posterior distribution f (θ |x1 , . . . , xn ) of θ given x1 , . . . , xn , using Bayes’ theorem. 3. Compute α0 and α1 using the following formulas: α0 = P (θ ∈ 0 |x1 , . . . , xn ) ⎧ ⎪ f (θ |x1 , . . . , xn ) dθ, ⎪ ⎨ 0 = ⎪ ⎪ f (θ |x1 , . . . , xn ) , ⎩

if continuous if discrete

θ∈0

and α1 = P (θ ∈ 1 |x1 , . . . , xn ) ⎧ ⎪ f (θ |x1 , . . . , xn ) dθ, ⎪ ⎨ 1 = ⎪ ⎪ f (θ |x1 , . . . , xn ) , ⎩

if continuous if discrete.

θ∈1

4. Reject H0 if the posterior odds ratio,

α0 < 1. Otherwise accept. α1

In the foregoing procedure, we assume that P (θ ∈ 0 ) and P (θ ∈ 1 ) are both greater than zero.

EXERCISES 11.4 11.4.1.

The following is random data from a normal distribution with variance 9. 0.92 1.05 7.42 1.76

5.53 3.64 0.01 2.69

−4.47 1.54

−2.60 3.97

0.71 1.34

−3.66 −1.63

1.38 −1.24

3.87 −4.78

(a) Test the hypothesis, H0 : μ ≤ 0 vs. Ha : μ > 0. Assume that the prior is N(0, 4), so that μ ≤ 0 and μ > 0 are equally probable. (b) Compare your decision with classical hypothesis testing, with α = 0.05. 11.4.2.

(a) For the data of Exercise 11.3.2, using the Bayesian method, test the hypothesis H0 : μ ≤ 170 vs. Ha : μ > 170. (b) Compare your decision with classical hypothesis testing, with α = 0.05.

11.4.3.

It is known that a certain disease affects 10% of a population. Of a random sample of 50 patients in the disease group who are exposed to a new treatment, we observe that 12 patients were hospitalized in a year. Let μ be the population rate that needs hospitalization in a year. Assume μ has a Gamma(0.1, 2) prior. Let μ ∼ Gamma(0.1, 2) and f (x|μ) ∼

588 CHAPTER 11 Bayesian Estimation and Inference

Poi(50μ). Given that x = 0.24 is an observation of X, test the hypothesis H0 : p ≤ 0.10 vs. Ha : p > 0.10. (If X is the number of patients admitted in a year, assume X ∼ Poi (50μ), the Poisson approximation of the binomial.) 11.4.4.

For an upcoming congressional election, suppose we want to estimate the amount of support for a particular candidate in a district. By previous experience and voter registration data, we can assume that the prior distribution, the proportion of support, p, is a beta distribution with α = 10, and β = 8 (i.e., π (p) ∼ Beta (10, 8)). We conducted a survey of 1000 randomly selected voters, of whom 600 support the candidate. Test the hypothesis H0 : p ≥ 0.60 vs. Ha : p < 0.60.

11.4.5.

For the data of Exercise 11.3.5, test the hypothesis H0 : μ ≤ 2400 mg vs. Ha : μ > 2400 mg for this ethnic group.

11.4.6.

Suppose we have a coin (not necessarily balanced) with p being the probability of heads. Assume a uniform prior for p. Suppose in 20 tosses of this coin, we obtained 12 heads. Test the hypothesis H0 : p ≥ 0.50 vs. Ha : p > 0.50.

11.5 BAYESIAN DECISION THEORY Bayesian methods in general are more concerned with problems of decision making than with problems of inference. Decision theory, as the name implies, is concerned with the problem of making decisions. Statistical decision theory is concerned with optimal decision making under uncertainty or when statistical knowledge is available only on some of the uncertainties involved in the decision problem. Uncertainty could be about the true value related to the decision, or, uncertainty could be about the actual state of the nature. Abraham Wald (1902–1950) laid the foundation for statistical decision theory. Original works on the decision theory emerged out of game theory considerations. Many books and articles have been written on the various aspects of decision theory. The Bayesian approach to the decision theory was introduced by Leonard Jimmie Savage in 1954. In this section, we introduce the general idea of decision theory. We basically deal with analytical procedures for the decision-making process. This will involve selection of an optimum decision from a choice of courses of action among two or more alternatives. The Bayesian decision theory quantiﬁes the trade-offs between different decisions using costs and probabilities that accompany such decisions. Consider, as an example, a company deciding whether or not to market a new brand of toothpaste with a whitening agent. Clearly many factors will affect the decision (for example, the proportion of people who are likely to switch to the new brand, and the likelihood of other competing companies introducing similar toothpastes). These factors are generally unknown, but estimates can be obtained from statistical investigations. The classical statistical approach relies exclusively on the data obtained from these statistical investigations, ignoring other relevant information such as the company’s past experiences in marketing similar products. Statistical decision theory tries to combine other relevant information with the sample information to arrive at the optimal decision. Therefore, a Bayesian setting seems to be more appropriate for decision theory.

11.5 Bayesian Decision Theory 589

One piece of relevant information that decision theory considers is the possible consequences of the decisions. Often these consequences can be quantiﬁed. That is, the loss or gain of each decision can be expressed as a number (called the loss or utility). A loss or utility to a decision maker is the effect of the interaction of two factors: (1) the decision or action selected by the decision maker; and (2) the event or state of the world that actually occurs. Classical statistics does not explicitly use a loss function or a utility (payoff ) function. A second source of information that decision theory utilizes is the prior information. Prior information could be based on past experiences of similar situations or on expert opinion. We can follow the procedure explained next as a guideline for decision making.

GENERAL DECISION THEORY PROCEDURE 1. Identify the objectives of the decision-making process. 2. Identify the set of actions and set of possible events (states of nature). 3. Assign probabilities to the occurrence of each possible state of nature (prior). If more observations are available, calculate the posterior probabilities to the occurrence of each possible state of nature. 4. For each possible event, assign a numerical value to the anticipated payoff (or loss) of each course of action. 5. Compute the expected value of the payoffs (utility or loss function). This could be done by either using the prior probabilities if there are no observations, or using the posterior probabilities. 6. Select the optimum decision among the available alternative courses of action that maximizes the expected value of the payoffs.

We now consider an example to illustrate the idea of statistical decision making.

Example 11.5.1 Suppose you own a small stall at a ﬂea market that is open only on weekends. If the weather is good, you make a proﬁt of $200, and if it is bad, you close your stall and you make no (zero) proﬁt. However, you have the option of buying, from an insurance company, weather insurance that costs $75. The company pays you $210 if the weather is bad. Suppose you believe that the probability of good weather on a particular weekend is p. Compute the expected gain if you insure and if you do not. What is the best course of action? Arrive at a decision.

Solution From the information in the problem, we can obtain the utility gain or profit table shown in Table 11.4, based on our decision to insure or not insure. Suppose that we model the state of weather as good or bad by means of a random variable defined as follows. θ=

1,

if the weather is good

0,

if the weather is bad.

590 CHAPTER 11 Bayesian Estimation and Inference

Table 11.4 Weather Parameter Space → Decision Space ↓D Insurance (I)(d1)

Good (θ1 )

Bad (θ2 )

$125 (200–75)

$135 (210–75)

$200

$0

No Insurance (NI)(d2)

Suppose for our example we believe that during a particular weekend P(θ = 1) = p, and P(θ = 0) = 1 − p. This can be considered as prior information. The different values of θ are called states of nature. We assign (perhaps subjectively) a probability structure for the states of nature defined by a prior distribution π(θ). Now we can compute the expected gain when we insure and when we do not. Using the values in the table, Expected gain given we insure = (125) p + (135) (1 − p) = 135 − 10p Expected gain when do not insure = (200) p + (0) (1 − p) = 200p Hence, insurance is preferable if 135 − 10p > 200p or p

M).

We consider the problem of testing the null hypothesis H0 : M = m0

versus

Ha : M > m0 .

Assume that the underlying population distribution is continuous so that P (X ≤ M) = 0.5. Let Xi be the ith observation and let N + be the number of observations that are greater than m0 . N + will be our test statistic. We will reject H0 if, n+ the observed value of N + , is too large. This test is called the sign test. A test at signiﬁcance level α will reject H0 if n+ ≥ k, where k is chosen such that P(N + ≥ k when M = m0 ) = α.

Similarly, if the alternative is of the form Ha : M = m0 , the critical region is of the form N + ≤ k or N + ≥ k1 , where P(N + ≤ k) + P(N + ≥ k1 ) = α. In order to determine such a k and k1 , we need to determine the distribution of N + . The test works on the principle that if the sample were to come from a population with a continuous distribution, then each of the observations falls above the median or below the median with probability 12 . Hence, the number of sample values falling below the median follows a binomial distribution with parameters n and p = 12 , n being the sample size. If a sample value equals the hypothesized median m0 , that observation will be discarded and the sample size will be adjusted accordingly (we remark that such values should be very few). Thus, when H0 is true, N + will have a binomial distribution with parameters n and p = 12 . For this reason, some authors call this test the binomial test. The following box summarizes the test procedure and the corresponding critical regions.

608 CHAPTER 12 Nonparametric Tests

SIGN TEST H0 : M = m0 Alternative Hypothesis Ha : M > m0

Critical Region

1 n n N + ≥ k, where =α 2 i=k i n

1 n n =α 2 i

Ha : M < m0

k N + ≤ k, where i=0

Ha : M = m0

n N + ≥ k1 , where i=k1

or k N + ≤ k, where i=0

1 n α n = 2 2 i

α 1 n n = 2 2 i

If α or α/2 cannot be achieved exactly, choose k (or k and k1 ) so that the probability comes as close to α (or α/2) as possible.

We now summarize the procedure of the sign test in the case of an upper tail alternative. The other two cases are similar. HYPOTHESIS TESTING PROCEDURE BY SIGN TEST We test H0 : M = m0 vs. H1 : M > m0 . 1. Replace each value of the observation that is greater than m0 by a plus sign and each sample value less than m0 by a minus sign. If the sample value is equal to m0 , discard the observation and adjust the sample size n accordingly. 2. Let n+ be the number of +’s in the sample. For n and p = 12 , from the binomial table, ﬁnd γ = P (N + ≥ n+ ). 3. Decision: If γ is less than α, H0 must be rejected. Based on the sample, we will conclude that the median of the population is greater than m0 at the signiﬁcance level α. Otherwise do not reject H0 . Assumptions: The population distribution is continuous. The number of ties is small (less than 10% of the sample).

12.3 Nonparametric Hypothesis Tests for One Sample 609

Note that the approach described in the foregoing procedure is nothing but the p-value method for hypothesis testing regarding a median using the sign test. Recall that the p-value is the probability of observing a test statistic as extreme or more extreme than what was really observed, under the assumption that the null hypothesis is true. In the sign test, we had assumed that the median is M = m0 , so 50% of the data should be less than m0 and 50% of the data greater than m0 . Thus, we expect half of the data to result in plus signs and half to result in minus signs. Hence, we can think of the data as following a binomial distribution with p = 1/2 under the null hypothesis. The p-value is computed from its deﬁnition given by the formula p-value = P (N + ≥ n+ ) =

n n 1 n i=k

i

2

= γ.

The p-value method is to reject the null hypothesis if the computed p-value is greater than α. These binomial probabilities can be obtained from the binomial tables, or statistical software packages. The following example illustrates how we apply the three-step procedure.

Example 12.3.1 For the given data from an experiment 1.51

1.35

1.69

1.48

1.29

1.27

1.54

1.39

1.45

test the hypothesis that H0 : M = 1.4 versus Ha : M > 1.4 at α = 0.05.

Solution We test H0 : M = 1.4 versus Ha : M > 1.4. Replacing each value greater than 1.4 with a plus sign and each value less than 1.4 with a minus sign, we have + − + + − − + − +. Thus, n+ = 5. From the binomial table with n = 9 and p = 12 , we have P(N + ≥ 5) = 0.50. Thus, the p-value is 0.5. Because α = 0.05 < 0.50, the null hypothesis is not rejected. We conclude that the median does not exceed 1.4.

When the sample size n is large, we can apply the normal approximation to the binomial distribution. That is, the test statistic N + is approximately normally distributed. Thus, under H0 , N + will

610 CHAPTER 12 Nonparametric Tests

have approximate normal distribution with mean np = z-transform, we have Z=

n 2

and variance of np (1 − p) =

n 4.

By the

N + − n/2 2N + − n ∼ N(0, 1). = √ √ n n/4

We could utilize this test if n is large, that is, if np ≥ 5 and n(1 − p) ≥ 5. Hence, under H0 , because p = 1/2, if n ≥ 10, we could use the large sample test. The following table summarizes the large sample sign test. A SIGN TEST FOR A LARGE RANDOM SAMPLE When the sample size is large (n ≥ 10), we can use the normal approximation to a binomial. This leads to the large sample sign test: H0 : M = m0 versus Alternative Hypothesis Ha : M > m0

Rejection Region z ≥ zα

Ha : M < m0 Ha : M = m0

z ≤ −zα |z| ≥ zα/2

The test statistic is Z=

2N + − n . √ n

Decision: Reject H0 , if the test statistic falls in the rejection region, and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 because there is not enough evidence to conclude that Ha is true for a given α, and more experiments are needed. Assumptions: (i) Population distribution is continuous. (ii) Sample size greater than or equal to 10 (after the removal of ties). (iii) The number of ties is small (less than 10% of the sample size).

We illustrate this procedure with the following example.

Example 12.3.2 In order to measure the effectiveness of a new procedure for pruning grapes, 15 workers are assigned to prune an acre of grapes. The effectiveness is measured in worker-hours/acre for each person. 5.2 4.2

5.0 5.3

4.8 4.9

3.9 4.7

6.1 4.9

4.2

4.4

5.5

5.8

4.5

Test the null hypothesis that the median time to prune an acre of grapes with this method is 4.5 hours against the alternative that it is larger. Use α = 0.05.

12.3 Nonparametric Hypothesis Tests for One Sample 611

Solution We test H0 : M = 4.5 versus H0 : M > 4.5. Replacing each value greater than 4.5 with a plus sign and each value less than 4.5 with a minus sign, we have +++−+−−++−++++. Because there is one observation that is equal to 4.5, we must discard it and take n = 14. Thus N + = 10, using the large sample approximation, the test statistic is Z=

20 − 14 2N + − n = 1.6. = √ √ n 14

For α = 0.05, from the standard normal table, the value of z0.05 = 1.645. Hence, the rejection region is z > 1.645. Because the observed value of the test statistic does not fall in the rejection region, we do not reject the null hypothesis at α = 0.05 and conclude that the median time to prune an acre of grapes is 4.5 hours.

12.3.2 Wilcoxon Signed Rank Test In the sign test, we have considered only whether each observation is greater than m0 or less than m0 without giving any importance to the magnitude of the difference from m0 . An improved version of the sign test is the Wilcoxon signed rank test, in which one replaces the observations by their ranks of the ordered magnitudes of differences, |xi − m0 |. The smallest observation is ranked as 1, the next smallest will be 2, and so on. However, the Wilcoxon signed rank test requires an additional assumption that the continuous population distribution is symmetric with respect to its center. Thus, if the data are ordinal, the Wilcoxon test cannot be used.

HYPOTHESIS TESTING PROCEDURE BY WILCOXON SIGNED RANK TEST We test H0 : M = m0 versus H1 : M = m0 . 1. Compute the absolute differences zi = |xi − m0 | for each observation. Replace each value of the observation that is greater than m0 by a plus sign and each sample value that is less than m0 by a minus sign. If the sample value is equal to m0 , discard the observation and adjust the sample size n accordingly. 2. Assign each zi a value equal to its rank. If two values of zi are equal, assign each zi a rank equal to the average of ranks each should receive if there were not a tie. 3. Let W + be the sum of the ranks associated with plus signs and W − be the sums of ranks with negative signs.

612 CHAPTER 12 Nonparametric Tests

4. Decision: If m0 is the true median, then the observations should be evenly distributed about m0 . For a size α critical region, reject H0 if W + ≤ c1 , where P (W + ≤ c1 ) =

α , 2

or α . 2 Assumptions: The population distribution is continuous and symmetrical. The number of ties is small, less than 10% of the sample size. W + ≥ c2 , where P (W + ≥ c2 ) =

The exact distribution of W + is considerably complicated and we will not derive it. However, for certain values of n, the distribution is given in the Wilcoxon signed rank test table. For the Wilcoxon signed rank test, the rejections region based on the alternative hypothesis is given next. For Ha : M > m0 , rejection region is W + ≥ c, where P (W + ≥ c) = α,

and for Ha : M < m0 , rejection region is W + ≤ c, where P (W + ≤ c) = α.

We illustrate the Wilcoxon signed rank test with the following examples.

Example 12.3.3 For the given data that resulted from an experiment 1.51

1.35

1.69

1.48

1.29

1.27

1.54

1.39

1.45

test the hypothesis that H0 : M = 1.4 versus Ha : M = 1.4. Use α = 0.05.

Solution We test H0 : M = 1.4 versus Ha : M = 1.4. Here, α = 0.05, and m0 = 1.4. The results of steps 1 to 3 are given in Table 12.1. Thus, we have W + = 29 and n = 9. From the Wilcoxon signed-rank test table in the appendix, we should reject H0 if W + ≤ 6 or W + ≥ 38 with actual size of α = 0.054. Because W + = 29 does not fall in the rejection region, we do not reject the null hypothesis that M = 1.4.

12.3 Nonparametric Hypothesis Tests for One Sample 613

Table 12.1 xi

zi = |xi − 1.4|

Sign

Rank

1.51

0.11

+

5.5

1.35

0.05

−

3

1.69

0.29

+

9

1.48

0.08

+

4

1.29

0.11

−

5.5

1.27

0.13

−

7

1.54

0.14

+

8

1.39

0.01

−

1.5

1.45

0.01

+

1.5

Example 12.3.4 Air pollution in large U.S. cities is monitored to see whether it conforms to requirements set by the Environmental Protection Agency. The following data, expressed as an air pollution index, give the air quality of a city for 10 randomly selected days. 57.3

58.1

58.7

66.7

58.6

61.9

59.0

64.4

62.6

64.9

Test the hypothesis that H0 : M = 65 versus Ha : M < 65. Use α = 0.05.

Solution We test H0 : M = 65 versus Ha : M < 65. Here, α = 0.05, and m0 = 65. The results of steps 1 to 3 are given in Table 12.2. Thus, W + = 3, and n = 10. Using the Wilcoxon signed rank test table, we should reject H0 if W + ≤ 10 with actual size of α = 0.042. Because the observed value of W + falls in the rejection region, we reject H0 and conclude that the sample evidence suggests that we conclude the median air pollution index is less than 65.

The Wilcoxon signed rank test is a nonparametric alternative to the one-sample t-test. The question then is, how do we decide which one to choose? Choose the one-sample t-test if it is reasonable to assume that the population follows a normal distribution. Otherwise, choose the Wilcoxon nonparametric test. However, the Wilcoxon test will have less power. For example, a normal probability plot of the data of Example 12.3.4 is given in Figure 12.4. Looking at this ﬁgure, we can see that the normality assumption is a suspect. It may make more sense to use the nonparametric method.

614 CHAPTER 12 Nonparametric Tests

Table 12.2 xi

zi = |xi − 65|

Sign

Rank

57.3

7.7

−

10

58.1

6.9

−

9

58.7

6.3

−

8

66.7

1.7

+

3

58.8

6.2

−

7

61.9

4.1

−

5

59.0

6.0

−

6

64.4

0.6

−

2

62.6

2.4

−

4

64.9

0.1

−

1

Normal probability plot 0.999

Probability

0.99 0.95 0.80 0.50 0.20 0.05 0.01 0.001 57

58

59

60

61

Average: 61.22 Std Dev: 3.32158 N: 10

62 63 Index

64

65

66

67

Kolmogorov-Smirnov Normality Test D⫹: 0.248 D⫺: 0.131 D: 0.248 Approximate P-Value: 0.081

■ FIGURE 12.4 Normal probability for air pollution index.

When sample size n is sufﬁciently large, under the assumption of H0 being true, the distribution of W + is approximately normal with mean E(W + ) =

1 n(n + 1) 4

12.3 Nonparametric Hypothesis Tests for One Sample 615

and variance Var(W + ) =

n(n + 1)(2n + 1) . 24

Hence, the test statistic is given by Z= √

W + − 14 n(n + 1)

n(n + 1)(2n + 1)/24

which is approximately the standard normal distribution. This approximation can be used when n > 20.

SUMMARY OF THE WILCOXON SIGNED RANK TEST FOR LARGE SAMPLES (N > 20) We test H0 : M = m0 versus M > m0 , upper tailed test Ha : M < m0 , lower tailed test M = m0 , two-tailed test. The test statistic: 1 n(n + 1) 4 Z= √ . n(n + 1)(2n + 1)/24 W+ −

Rejection region: ⎧ ⎪ ⎨ z > zα , z < −zα , ⎪ ⎩ |z| > z , α/2

upper tail RR lower tail RR two tail RR.

Decision: Reject H0 , if the test statistic falls in the RR, and conclude that Ha is true with (1 − α)100% conﬁdence. Otherwise, do not reject H0 , because there is not enough evidence to conclude that Ha is true for a given α and more experiments are needed. Assumptions: (i) The population distribution is continuous and symmetric about 0. (ii) Sample size is greater than or equal to 20. (iii) The number of ties is small, < 10% of the sample size.

We illustrate the Wilcoxon signed rank test with the following example.

616 CHAPTER 12 Nonparametric Tests

Example 12.3.5 The following data give the monthly rents (in dollars) paid by a random sample of 25 households selected from a large city. 425 960 1450 655 1025 750 670 975 660 880 1250 780 870 930 550 575 425 900 525 1800 545 840 765 950 1080 Using the large sample Wilcoxon signed rank test, test the hypotheses that the median rent in this city is $750 against the alternative that it is higher with α = 0.05.

Solution We test H0 : M = 750 versus Ha : M > 750. Here α = 0.05, and m0 = 750. The results of steps 1 to 3 are given in Table 12.3 (where the asterisk indicates zi = 0).

Table 12.3 xi

zi = |xi − 750|

Sign

425

325

−

19.5

960

210

+

15

1450

700

+

23

655

95

−

6

1025

302

+

18

750

0

∗

ignore

670

80

−

3

975

225

+

16.5

660

90

−

4.5

880

130

+

8

1250

500

+

22

780

30

+

2

870

120

+

7

930

180

+

11

550

200

−

12.5

Rank

(continued)

12.3 Nonparametric Hypothesis Tests for One Sample 617

Table 12.3 (continued) xi

zi = |xi − 750|

Sign

Rank

575

175

−

10

425

325

−

19.5

900

150

+

9

525

225

−

16.5

1800

1050

+

24

545

205

−

14

840

90

+

4.5

765

15

+

1

950

200

+

12.5

1080

330

+

21

Here, for n = 24, W + = 172.5, and the test statistic is 1 n(n + 1) 4 Z= √ n(n + 1)(2n + 1)/24 W+ −

1 172.5 − (24)(25) 4 = = 0.64286. 8 (24)(25)(49) 24 For α = 0.05, the rejection region is z > 1.645. Because the observed value of the test statistic does not fall in the rejection region, we do not reject the null hypothesis. There is not enough evidence to conclude that the median rent in this city is more than $750.

The rank tests are useful for situations when you suspect that the data do not follow the normal population. It is important to note that ignoring the tied observations reduces the effective sample size, which in turn reduces the power of the test (see Example 7.1.4 for the effect of n on the value of β). This loss is not signiﬁcant if there are only a few ties. However, if the ties are 10% or more, hypothesis testing using rank tests becomes considerably conservative. That is, they yield error probabilities that are signiﬁcantly high.

12.3.3 Dependent Samples: Paired Comparison Tests The sign test and the Wilcoxon signed rank test can also be used for paired comparisons. The experimental procedure typically consists of taking “before” and “after” type or otherwise matched as in

618 CHAPTER 12 Nonparametric Tests

the paired t-test case readings for each unit. Suppose there are n pairs of before and after observations and we are interested in testing the equality of the two medians. One way to test such observations is to consider the difference between the two observations for a unit to be a single observation on that unit. Thus, we can treat the sample as being n observations on a population of differences. For this new sample of differences, the testing problem becomes H0 : M = 0 versus Ha : M > 0(or M < 0, or M = 0).

Hence, the basic procedure could be summarized to ﬁrst ﬁnd the difference between the two units for each of the observations, and then follow the testing procedures explained earlier for the sign test or the Wilcoxon signed rank test. Both small sample and large sample cases can be handled as before. In the following example, we illustrate this concept for a large sample sign test.

Example 12.3.6 A dietary program claims that 3 months of its diet will reduce weight. In order to test this claim, a random sample of eight individuals who went through this program for 3 months is taken. The following table gives weight in pounds. Before After

180 172

199 191

175 172

226 230

189 178

205 199

169 171

211 201

Using a 5% signiﬁcance level, is there evidence to conclude that the program really reduces the population median weight?

Solution Let M denote the median of the population of difference of weights. We will use the difference as ‘‘after’’−’’before.’’ Then we will test H0 : M = 0

versus

Ha : M < 0.

We will use the large sample sign test. Replacing each value of the difference that is greater than zero by a + sign and less than zero by a − sign, we have Difference Sign

−8 −

−8 −

−3 −

4 +

−11 −

−6 −

2 +

−10 −

For n = 8 and N + = 2, the test statistic is given by Z=

2N + − n 4−8 = √ = −1.414. √ n 8

For α = 0.05, z0.05 = 1.645, and the rejection region is z < − 1.645. Because the observed value of the test statistic does not fall in the rejection region, we do not reject the null hypothesis. Thus, there is not enough evidence to conclude that the new program reduces the weight.

12.3 Nonparametric Hypothesis Tests for One Sample 619

EXERCISES 12.3 12.3.1.

It was reported that the median interest rate on 30-year ﬁxed mortgages in a certain large city is 7.75% on a particular day, with zero points. A random sample of nine lenders produced the following data of interest rates in percentage. 7.625 7.375 8.00 7.50 7.875 8.00 7.625 7.75 7.25 Test the hypothesis that the median interest rate in this city is different from 7.75%, using (a) the sign test, and (b) the Wilcoxon signed rank test. Use α = 0.01. Compare the two results.

12.3.2.

It is believed that a typical family spends 35% of its income on food and groceries. A sample of eight randomly selected families yielded the following data. 30

29

39

49

36

33

37

35

Test the hypothesis that the median percentage of family income spent for food and groceries is 35 against the alternative that it is less than 35. Use α = 0.05. 12.3.3.

The SAT scores (out of a maximum possible score of 1600) for a random sample of 10 students who took this test recently are: 1355 765 890 1089 986 1128 1157 1065 1224 567 Test the hypothesis that the median SAT score is 1000 against the alternative that it is greater using α = 0.05. Use both the sign test and the Wilcoxon signed rank test. Explain if the conclusions are different.

12.3.4.

The regulatory board of health in a particular state speciﬁes that the ﬂuoride levels in water must not exceed 1.5 parts per million (ppm). The 20 measurements given here represent the randomly selected daily early morning readings on ﬂuoride levels in water at a certain city. 0.88 0.82 0.71 0.92

0.97 1.11

0.95 0.84 0.90 0.81 0.97 0.85

0.87 0.97

0.78 0.75 0.83 0.91 0.78 0.87

Test the hypothesis that the median ﬂuoride level for this city is 0.90 against the alternative that the median is different from 0.9 at α = 0.01, using (a) the large sample sign test, and (b) the Wilcoxon signed rank test. Interpret the results. 12.3.5.

The following data give the weights (in pounds) for a random sample of 20 NFL players. 285 269

178 285

311 276 192 232 259 189 298 296 193 288 254 246 234 274

211 229

Test the hypothesis that the median weight of NFL players is 250 pounds against the alternative that it is greater at α = 0.05, using (a) the large sample sign test and (b) the Wilcoxon signed rank test.

620 CHAPTER 12 Nonparametric Tests

12.3.6.

The following data give the amount of money (in dollars) spent on textbooks by 18 students for the last academic year at a large university. 510 490

425 188

190 115

298 230

157 610

260 320 220 155

615 315

455 110

Test the hypothesis that the median amount spent on books at this university is $325 against the alternative that it is different using the large-sample sign test. Use α = 0.05. 12.3.7.

It is desired to study the effect of a special diet on systolic blood pressure. The following sample data are obtained for eight adults over 40 years of age before and after 6 months of this diet. Before After

185 188

222 235 217 229

198 190

224 226

197 185

228 225

234 231

At 95% conﬁdence level, is there evidence to conclude that the new diet reduces the systolic blood pressure in individuals of over 40 years old? Test (a) using the sign test, and (b) using the Wilcoxon signed rank test. Interpret the results. 12.3.8.

In an effort to study the effect on absenteeism of having a day-care facility at the workplace for women with newborn babies (less than 1 year old), a large company compared the number of absent days for a year for seven women with newborn children before and after instituting a day-care facility. Before After

20 16

18 9

35 22 17 24 15 22 28 19 13 10

At 99% conﬁdence level, is there evidence to conclude that having a day-care facility at the workplace reduces absenteeism for women with newborn children?

12.4 NONPARAMETRIC HYPOTHESIS TESTS FOR TWO INDEPENDENT SAMPLES In this section we learn how to test the equality of the medians of two independent samples from two populations. This is especially useful when one studies the treatment effects, such as the effect of a certain drug to treat a given medical condition when we have two groups—an experimental group and a control group—or the effect of a particular type of teaching method. We will describe the median test, which corresponds to the sign test, and the Wilcoxon rank sum test.

12.4.1 Median Test Let m1 and m2 be the medians of two populations 1 and 2, respectively, both with continuous distributions. Assume that we have a random sample of size n1 from population 1 and a random sample of size n2 from population 2. The median test can be summarized as follows.

12.4 Nonparametric Hypothesis Tests for Two Independent Samples 621

HYPOTHESIS TESTING PROCEDURE USING MEDIAN TEST We test m1 > m2 , upper tailed test H0 : m1 = m2

versus

Ha : m1 < m2 , lower tailed test m1 = m2 , two-tailed test.

1. Combine the two samples into a single sample of size n1 + n2 , keeping track of each observation’s original population. Arrange the n1 + n2 observations in increasing order and ﬁnd the median of this combined sample. If the median is one of the sample values, discard those observations and adjust the sample size accordingly. 2. Deﬁne N1b to be the number of observations of a sample from population 1 (under H0 we would expect this number to be around n1 /2). 3. Decision: If H0 is true, then we would expect N1b to be equal to some number around n1 /2. For Ha : m1 > m2 , rejection region is N1b ≤ c, where P(N1b ≤ c ) = α, for Ha : m1 < m2 , rejection region is N1b ≥ c, where P(N1b ≥ c ) = α, and for Ha : m1 = m2 , rejection region is N1b ≥ c1 , or N1b ≤ c2 , where