History of binary RUMP file format

RUMP, in early 1980's versions, stored RBS spectral data in a machine dependent format dictated by the FORTRAN compiler on each computer. Files written by one machine were generally incompatible with other machines due to variations in computer architectures and compiler implementation (byte ordering, integer length, real number format, record blocking, etc.). These incompatibilities became sufficiently annoying in the age of networked computers to force a transition in 1984 to a machine independent and extensible format.

Although a few of the old FORTRAN format data files may still exist, they have mostly disappeared since the commercial release of RUMP. Intel architecture versions of RUMP still carry code to read the old data format; continued validity however depends on the whims of non-portable compiler constructs. If necessary, source code for old VAX and PC format converters are available from CGS.

In 1984, a new machine independent and easily extendable data format was adopted and incorporated as the default in released versions of RUMP. The advantages of transparently moving data files between computers quickly guarenteed the general acceptance of the new format. With only very minor changes, this RBS file format has remained for over a decade. The current version is 100% backward compatible ensuring the long term accessibility of archival data.

• Version 1.0 [5/84]. Original specification.
• Version 1.0a [1/94]. Added time constant (tau) parameter to the MCA record in order to implement a more accurate pulse pileup algorithm in RUMP. This extra parameter is ignored by earlier readers as per standard specification.
• Version 1.1 [8/94].. Implemented additional data compression algorithm (structure 3) for zero element compression. Added record types 12h-15h for override of default compression. Added record type 20h for storing arrays or multiple spectra as a single data record. These formats were introduced to support efficient three dimensional RBS spectrum storage. (Not implemented in DOS versions of RUMP.)

RUMP, as of 7/97, reads at the 1.1 format, but by default writes only to the 1.0 standard (changable using the CONFIG WRITE_LEVEL command). The additional compression available at level 1.1 is really only useful for 3D time-of-flight spectral data which contain extremely long runs of zeros. Unless storage space is a critical issue, it is recommended that 1.1 features not be used to maintain back-level compatability.

General Specifications

Revision requirements

1. The term "word" below refers exclusively to a 32 bit data item, either in memory or as stored on any media, referred to generically as "disks".
2. Disk files are considered to be byte oriented. Each word is stored as 4 sequential bytes labeled in order of storage from 1 to 4. The individual bits of a word are labeled from 0 to 31 with bit 31 occupying the upper order bit of byte 1 (see below). Since word storage differs between processors, byte swapping may be necessary before and after file accesses. The standard does not support non-byte oriented storage devices.

   
       |31........24|23........16|15.........8|7..........0|  bit order
       |   Byte 1   |   Byte 2   |   Byte 3   |   Byte 4   |  byte order
       |  offset i  | offset i+1 | offset i+2 | offset i+3 |  disk storage
   

3. Integers are long signed integers (I*4) using the two's complement representation. Bit 31 is the sign bit and bit 0 is the least significant bit. Range is limited to -2^{31}-1 to 2^{31}-1 with sign bit only (80000000h) considered invalid by this specification. Where used, short and tiny integers refer to two's complement 16 bit and 8 bit values respectively with the high order bit remaining the sign bit. Two short or four tiny integers may be packed in a single word. The first short integer occupies bits 31-16 and the first tiny occupies bits 31-24.
4. Real numbers must conform to the 32 bit single precision IEEE (REAL*4) representation. Range allowed is 1.1754944E-38 to 3.4028235E38 fully normalized plus zero. Plus and minus zero are equivalent. The bit representation is shown below. The sign bit is 0 for positive numbers and 1 for negatives. The exponent is an 8 bit unsigned integer with a bias of 127 (the actual exponent is field value minus 127). An implicit 1 bit is assumed to the left of bit 22 and the binary decimal point occurs between the implicit 1 and bit 22.

   
       |31........24|23........16|15.........8|7..........0|  bit order
       |S| Exponent |              Mantissa                |  REAL*4
       |  offset i  | offset i+1 | offset i+2 | offset i+3 |  disk storage
   

5. Logical values are undefined and are not used. If required, logical values may be stored as integers where 0 will be interpreted as false and non-zero true.
6. Character strings are stored as a structure consisting of a integer specifying the number of valid characters in the string followed by the string itself, packed 4 characters per word. Characters are packed with the first character in byte 1 followed by the second character in byte 2, etc. Any unused bytes are undefined (do not assume a null value nor blank filled). The total number of words required to store a string of length n is 1+(n+3)/4 (using integer arithmetic). Character values 00h-7Fh are defined as the ASCII standard sequence. Characters 80h-FFh are considered valid but are not specified by this standard.

Record Representations

1. File format must be sequential and binary. Records must be stored on disk exactly as specified in this standard. No intervening blocking bytes, index entries, checksums or terminating markers are permitted.
2. All records will start and terminate on word boundaries.
3. Each record consists of 3 to 1027 words. The first word specifies the total length of the record (in words), the second defines the type of record, and the last is a checksum word.

   WORD 1:	Record length in words	(long integer)
   WORD 2:	Record type	(long integer)
   WORD 3:	First data element of record	(arbitrary)
   .....	.....	.....
   WORD 3+N-1:	Last of N data elements in record	(arbitrary)
   WORD 3+N:	Checksum	(long integer)

   All entries are treated as 32 bit unsigned integers for evaluating the checksum regardless of their actual types. The sum of all entries in the record (including the length, type and checksum) must equal 00000000h (overflow ignored).

4. Subsequent records begins immediately following the checksum.
5. Records are limited to 1024 data words. The total length of any record including the type, length and checksum is 1027 words (4108 bytes).
6. The first record in a file must be type 0000h as specified below.

Handling of excess or missing data in a record

1. If any record contains more data elements than expected, the extra should be ignored. This is to permit enhancements to record types by the addition of new parameters at the end.
2. If too few data elements are given, a program may either generate an error or proceed assuming zero, blank, or some default for the missing elements.
3. Unrecognized record types should be skipped.

Differential integer format

The differential integer format is efficient for storing integer values which change only slowly from one element to the next. It requires byte manipulation capabilities to implement. Records containing compressed data may be any length less than 1024 words. If compression would require more than 1024 words, the data must be written in uncompressed INTEGER or REAL format. Each block of compressed data must contain exactly 1024 data elements or the number of elements remaining to be read. For example, 1920 data elements would be packed with 1024 elements in the first 11h record and 896 elements in the second.

The first element in each block is the 4 byte value of the first data element. Subsequent values are encoded as changes from the previous value, either as byte offset, a 2 byte offsets, or a 4 byte absolute number. The next byte following one data element is the byte offset (-127 to +127) from the current value. If the offset is out of range, an 80h is written as the byte offset and the next two bytes are read as a 2 byte offset (-32767 to +32767). If the difference is still out of range, an 8000h is written and the next four bytes give the absolute value of the data element. The process continues until all 1024 bytes have been read or written.

Unpacked:        100   120      284   300       93275        93274
Packed:     00000064    14   8000A4    10    80800000016C5B     FF

In addition, at revision level 1.1, a zero compression algorithm has been implemented following the delta compression. This optional compression reduces runs of 00 bytes to a [FLAG][REPEAT] byte sequence. This compression is on a record-by-record basis; a record ZeroCompressed is identified by an initial byte of 80h followed by the [FLAG] byte pattern. True [FLAG] bytes in the data stream are stored as [FLAG][00] patterns.

Note that enabling of compression will cause a level 1.0 reader to break. This is unfortunate but accepted for the 10-50% additional compression, especially for extremely large spectra such as three dimensional microprobe or time-of-flight spectra.

Unpacked:        100   120      284   300       93275        93274
Packed:     00000064    14   8000A4    10    80800000016C5B     FF
Compressed: 80 81 81 03 64 14 80 00 A4 10 80 80 81 02 01 6C 5B FF

Record Types

• 0000h-00FFh - General data and record specifiers
• 0100h-01FFh - RUMP data files descriptors
• 0200h-0FFFh - reserved for Thompsonware

Program ID and data block descriptors

RUMP information, accelerator and MCA descriptors

RBS/FRES/PIXE spectrum descriptors

• simultaneous PIXE/RBS - different 0110h, 0112h and 012xh records for each
• simultaneous FRES/RBS - different 0110h, 0112h and 012xh records for each
• channeling/random RBS - possibly different 0120h records
• series of energy scans - different 0111h records

Programming Support

Example File

00000005 00000000 10211210 00010000 efddedeb                    ;Header
0000000b 00000002 00000019 'PC-RUMP data file [v 1.0]xxx'       ;Comment
                  f2e1ec83
0000000d 00000101 00000024 'Ni/NiSi/Si ....'  ced4b22f          ;ID string
00000008 00000102 0000000f 'LT= 857 CT= 860a' ffec1d26          ;LTCT string
0000000a 00000103 00000017 '18-JUN-1985 12:33:48.48 ' ada0c869  ;Date
00000009 00000111 404145d0 00000002 40800c56 00000002 41200000  ;Accelerator
                  41000000 fd1eacbc
00000007 00000112 409e6666 3fcccccd 00000000 414282e8 3e5248cc  ;MCA
00000008 00000120 00000000 40e00000 41100000 00000000 4059999a  ;Geometry
                  3db6653e
00000004 00000110 3f866666 c0799886                             ;CORR Factor
00000005 00000010 00000002 00000400 fffffbe9                    ;1024 compressed data
0000010e 00000011 00000000 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02
                  1b 8000c2 800106 80 012d 800107 8000b6 80008e 2b 58 35 0a ce
                  16 e0 2e a6 35 c6 d1 43 80ff68 5b 0b cb e1 da e5 d7 ed 46 d5
                  12 b8 1c f8 f4 de d5 13 27 d4 fa ba 04 0b de 2e 16 d8 10 f8
                  c3 28 05 05 fb de 06 e6 57 b8 06 01 da 06 ad 80008a c0 f7 01
                  e1 de 1b 34 03 cc 10 e8 06 f3 dd 1e df 25 05 dc ec 34 d9 3b
                  f5 ca 0e 12 0a e7 26 c3 0c f7 1d 21 cc 26 c5 fa 15 f9 1c e1
                  ......................
                  00 00 00 00 52dbd061


00000008 00000120 00000000 40e00000 41100000 00000000 4059999a  ;MCA parms
                  3db6653e
00000004 00000110 3f866666 c0799886                             ;Correction
00000005 00000010 00000000 00000400 fffffbeb                    ;1024 reals
00000403 00000011 00000000 00000000 00000000 00000000 00000000  ;Real data
                  00000000 00000000 be2f8af9 bfe66666 c1033333
                  4246a0ea 43590000 43f48ea1 4443c57c
                  4482399a 449af249 44a97333 44b36666
                  44bb3a84 44c2b8b0 44c24750 44c063a8
                  ....................................
                  441ce2be 44127e2c 440ba666 43f62f8b
                  00000000 00000000 53f717e6

API modules

Required #include files:

        #define _POSIX_SOURCE       /* POSIX compliance required */
        #include "preload.h"

        #include <stdio.h>
          .... other system include files ...

        #include "mytypes.h"
        #include "rump_rdw.h"

Required compile time switches:

In addition, when compiling rump_rdw.c, the following definitions must also be included to disable features unavailable in stand alone programs.
    -DLOCAL_MODE -DNO_EA_MODE -DNO_PIPE_MODE   
Alternatively, the #define for these may be uncommented in the source file.

SPECTRUM structure

typedef enum _SPECTRUM_TYPE {RBS, FRES, PIXE, NUCLEAR, OTHER} SPECTRUM_TYPE;
typedef enum _GEOMETRY_TYPE {
   CORNELL =  0,              /* Cornell theta/phi definitions */
   IBM     =  1,              /* IBM theta/phi definitions     */
   GENERAL = -1               /* Full theta/phi/psi definition */
} GEOMETRY_TYPE;

typedef struct _SPECTRUM {
   char filename[PATH_MAX];            /* Filename                         */
   char date[IDCHR];                   /* Date spectrum collected          */
   char ltct[IDCHR];                   /* Live time/Clock time information */
   char id[IDCHR];                     /* Identifier string                */
   SPECTRUM_TYPE type;                 /* Type of spectrum                 */
   FLOAT e0;                           /* Incident energy            (MeV) */
   int   zbeam;                        /* Atomic Z of incident             */
   FLOAT mbeam;                        /* Atomic mass of incident    (amu) */
   int   cbeam;                        /* |Charge| state of beam           */
   FLOAT q;                            /* Total accumulated charge    (uC) */
   FLOAT current;                      /* Average beam current        (nA) */
   FLOAT scale1,scale2;                /* Conversion MCA chan # -> keV     */
   FLOAT first;                        /* Channel number of first data pt  */
   FLOAT fwhm;                         /* Detector resolution        (keV) */
   FLOAT tau;                          /* MCA shaping time constant   (uS) */
   GEOMETRY_TYPE geom;                 /* Geometry identifier              */
   FLOAT phi,theta,psi;                /* Scattering angles      (degrees) */
   FLOAT omega;                        /* Detector solid angle       (mSr) */
   FLOAT corr;                         /* Random correction factor         */
   int   nspectra;                     /* Number of spectra in data        */
   int   npt;                          /* Number of data points            */
   int   nptmax;                       /* Dimensioned size of counts       */
   FLOAT *counts;                      /* Pointer to actual data           */
   int   dirty;                        /* Has data changed since read      */
   int   modify;                       /* Have parameters been modified    */
   int   iddone;                       /* Is ID been done for this spectra */
} SPECTRUM ;

API entries

RbsAllocateSpectrum

/* =========================================================================== -- Function to allocate space for a spectrum structure, and initialize to -- either default parameters or a copy of an existing spectrum. On return, -- the buf->count array is ready to be filled with spectral data. -- -- Usage: SPECTRUM *RbsAllocateSpectrum(SPECTRUM *proto, int NumChannels); -- -- Inputs: proto - prototype for spectrum header information. If -- NULL, default values will be used for all entries. -- NumChannels - number of channels for which to allocate space. -- If 0 or negative, buf->counts will be left NULL. -- -- Returns: Pointer to properly initialized structure or NULL on failure. =========================================================================== */

RbsFreeSpectrum

/* =========================================================================== -- Function to free all memory used by a SPECTRUM structure, including the -- structure itself (assuming it was malloc'd). -- -- Usage: int RbsFreeSpectrum(SPECTRUM *buf); -- -- Inputs: *buf - Rump spectrum structure -- -- Output: Free's spectrum data buffer and the structure itself. The pointer -- buf will no longer be valid after this call. -- -- Returns: 0 =========================================================================== */

RbsReadSpectrum

/* =========================================================================== -- Function to read a binary RBS file from disk. Modifies all entries in -- the structure that are defined in the file. -- -- Usage: SPECTRUM *RbsReadSpectrum(char *filename, SPECTRUM *spectra, int *err); -- -- Inputs: filename - character pointer to filename to be read -- spectra - pointer to a current spectrum structure. If this -- entry is NULL, a new structure will allocated, -- initialized with default values, and then filled. -- -- Output: err - If not NULL, will receive the error code -- 0 - successful read of data -- 1 - unable to open file -- 2 - unable to allocate buffer for reading -- -3 - unable to allocate buffers space for counts -- -2 - bad file format from the beginning -- -1 - bad file format somewhere during reading (after headers) -- -- Returns: pointer to spectrum (either spectra or the malloc'd space). On -- error, will return NULL. -- -- Notes: This routine is guarenteed to load all of the spectral data, -- resizing or allocating the counts buffer if necessary. =========================================================================== */

RbsWriteSpectrum

/* =========================================================================== -- Function to write a binary RBS file to disk. The filename is taken from -- the SPECTRUM structure. This is a standard write that stores all parameters -- currently relevent to RBS analysis. -- -- Usage: int RbsWriteSpectrum(SPECTRUM *spectra) -- -- Inputs: *spectra - pointer to a spectrum structure containing all of -- the current parameters and data. -- -- Output: Outputs record as specified -- -- Returns: 0 - successful write -- 1 - unable to open file for writing -- 2 - unable to allocate buffer space for writing -- 3 - pipe failed =========================================================================== */

RbsCopySpectrum

/* =========================================================================== -- Function to copy an existing spectrum structure to a new structure. -- -- Usage: SPECTRUM *RbsCopySpectrum(SPECTRUM *dest, SPECTRUM *source); -- -- Inputs: *source - spectrum to be copied -- -- Output: *dest - spectrum where data will be copied. If NULL, then -- a new structure will be allocated for the copy. -- -- Return: dest, or newly allocated SPECTRUM structure if dest was NULL -- NULL indicates error - usually unable to allocate space =========================================================================== */

RbsResizeSpectrum

/* =========================================================================== -- Function to re-allocate space for channel data in a SPECTRUM structure -- to a specified size -- -- Usage: int RbsResizeSpectrum(SPECTRUM *buf, int size) -- -- Inputs: *buf - Rump spectra structure -- size - New size desired. If 0, will dealloc counts buffer. -- buf->npt will be adjusted if the new size is less than -- than the currently data set. -- -- Output: Changes size of counts buffer to specified size -- -- Return: 0 if successful, 1 if could not realloc memory. =========================================================================== */

RbsSetFileWriteVersion

/* =========================================================================== -- Function to set the revision level for RbsWriteSpectrum(). The default -- mode is to write to the 1.0 revision level. This call will enable writing -- compressed data to the 1.1 specification. -- -- Usage: int RbsSetFileWriteVersion(int minor_level); -- -- Inputs: minor_level - level of writes tolerated -- 0 => limit to version 1.0 format -- 1 => limit to version 1.1 format -- <0 => no change, report level only -- -- Returns: previous minor_level enabled =========================================================================== */

Test program

/* Example usage of the RUMP read routines */

#define _POSIX_SOURCE
#include "preload.h"

#include <stdio.h>
#include <string.h>
#include <math.h>
#include <stdlib.h>

#include "mytypes.h"
#include "rump_rdw.h"

/* ===========================================================================
-- Routine to encode the mass, Z and charge state of an ion into a string
=========================================================================== */
static char *BeamCode(int zbeam, REAL mbeam, int cbeam) {

   int i;
   char *aptr;

   static char buffer[20];
   static char *ATOM_LIST[] = {"xx","H","He","Li","Be","B","C","N",
         "O","F","Ne","Na","Mg","Al","Si","P","S","Cl","Ar"};

   sprintf(buffer,"%d%s", (int) (mbeam+0.5), ATOM_LIST[zbeam] );
   aptr = buffer + strlen(buffer);
   if (cbeam > 0) {for (i=0; i<+cbeam; i++) *aptr++ = '+';}
   if (cbeam > 0) {for (i=0; i<-cbeam; i++) *aptr++ = '-';}
   *aptr = '\0';
   return(buffer);
}  

/* ===========================================================================
--  ShowActive(SPECTRUM *buf)
--
--  Prints header block to standard output
--     Filename:    c:\rump\example.rbs
--     Identifier:  Ni/NiSi/Si Annealed 90 min 295^~o^+C
--     LTCT Text:   LT= 857 CT= 860
--     Date:        18-JUN-1985 12:33:48.48
--     Beam:        3.000 MeV   4He++   xxxx.yy uCoul  @ xx.yy nA
--     Geometry:    General  Theta: -180.0  Phi: -109.0  Psi: +000.0
--     MCA:         Econv: xxx.yyy  xxx.yyy  First chan:  0.0  NPT: 1024
--     Detector:    FWHM: 35.0 keV   Omega: 3.400
--     Correction:  1.0041
=========================================================================== */
static void ShowActive(SPECTRUM *buf) {

   char *geometry, *filetype, *beam_string;
        
/* Create string of form 4He++ for the incident beam */
   beam_string = BeamCode(buf->zbeam, buf->mbeam, buf->cbeam);

/* Look and convert geometry ID into the appropriate string */
   if (buf->geom == CORNELL)      geometry = "Cornell";
   else if (buf->geom == IBM)     geometry = "IBM";
   else if (buf->geom == GENERAL) geometry = "General";
   else                           geometry = "Unknown";

/* Look and convert spectrum type into an appropriate string */
   if (buf->type == RBS)       filetype = "RBS";
   else if (buf->type == FRES) filetype = "FRES";
   else if (buf->type == PIXE) filetype = "PIXE";
   else                        filetype = "Unknown";

   printf(
      "  %-4s File:   %s\n"
      "  Identifier:  %s\n"
      "  LTCT Text:   %s\n"
      "  Date:        %s\n"
      "  Beam:       %6.3f MeV   %-7s% 7.2f uCoul  @ %5.2f nA\n"
      "  Geometry:    %7s  Theta: %7.2f  Phi: %7.2f  Psi: %7.2f\n"
      "  MCA:         Econv: %7.3f  %7.3f  First chan: %4.1f  NPT: %4d\n"
      "  Detector:    FWHM: %4.1f keV  Tau: %4.1f   Omega: %5.3f\n"
      "  Correction:  %6.4f\n",
         filetype,
         buf->filename, buf->id,      buf->ltct,  buf->date,
         buf->e0,       beam_string,
         buf->q,        buf->current,
         geometry,      buf->theta,   buf->phi,   buf->psi,
         buf->scale1,   buf->scale2,  buf->first, buf->npt,
         buf->fwhm,     buf->tau,     buf->omega,
         buf->corr);
   return;
}

/* ===========================================================================
-- Main routine.  Will print out the header for each file on the command line.
=========================================================================== */
int main(int argc, char *argv[]) {

   SPECTRUM *spectrum;                 /* Pointer to spectra structure */
   int rcode;

/* Require at least a single argument */
   if (argc < 2) {
      fprintf(stderr, "Usage: show <file1.rbs> <file2.rbs>\n");
      return(EXIT_FAILURE);
   }

/* Go through each argument on command line showing the header information */
   argc--; argv++;
   while (argc) {
      if ( (spectrum = RbsReadSpectrum(*argv, NULL, &rcode)) == NULL) {
         fprintf(stderr, "ERROR: Read failure on %s (%d)\n", *argv, rcode);
      } else {
         ShowActive(spectrum);                           /* Print identifier */
         RbsFreeSpectrum(spectrum);
      }
      argc--; argv++;
   }

   return(EXIT_SUCCESS);
}