Numeric Precision in FORTRAN Computing
Abstract
This article compares both real and complex outputs from sizeable numeric computations using identical code on several computer systems. The digital signal processing technique known as the modified covariance method was used as the computational engine. It is a recursive algorithm for solving the covariance equations of a linear predictor that seeks to predict an input signal by a linear combination of past signal samples. Single precision and double precision results are presented but the study focuses primarily on differences between the VAX Fortran 4.8 and MacFortran/020 compilers. Differences in the first digit for single precision arithmetic were found and double precision differences occurred in the eighth digit. Arithmetic with complex data types was found to be less precise than with real data types. Although differences exist among various computer systems, they all show the same order of magnitude accuracy with respect to CRAY-YMP results. The algorithm used here required a double precision implementation to obtain agreement between different computer systems. Acoustic propagation, sea ice, noise.
Document Details
- Document Type
- Technical Report
- Publication Date
- Oct 01, 1992
- Accession Number
- ADA258046
Entities
People
- Roger W. Meredith
Organizations
- United States Naval Research Laboratory