Sources of Error (CHAPTER 01.03)
Effect of Carrying Significant Digits
Learn the effect of carrying limited number of significant digits for a real-life problem. We regressed the data of coefficient of thermal expansion of steel as a function of temperature to a second order polynomial using Excel. We found that the general format of the regression model provided by Excel creates large round-off errors because of carrying limited significant digits. A solution to this issue is also presented.