3 S2-TCI system design considerations

  Due to the high data rate and high data volume implied by the S2 baseband processing system (e.g. a single S2 produces data at a rate of approximately 1 GByte/minute), overall baseband processing efficiency relies on methods of fast data transfer from tape to computer and fast data processing. With current technology, and processing data rates of 128 Mbits/s, it presently appears that even the highest performance Tape-to-Computer Interface and super-computing speeds may barely achieve real-time combined data throughput and processing times. We note that most super-computing applications to-date have been largely compute-intensive rather than data- (and input/output) intensive, but certainly the latter mode is being pursued by the challenging application of pulsar baseband data processing.

Presently, in general the high data rate of the S2 usually implies that data are most efficiently transferred from tape to computer in bursts, and via direct memory access (DMA). In such a scheme, tape data at a given total data rate and for some defined duration are passed to a typically small buffer memory in the DMA system and into computer RAM. Once in RAM, the following processing schemes may be considered:

1.

Data are processed immediately from (local) RAM;

2.

Data are transferred to an alternative medium, usually disk or output tape device (such as Exabyte or DLT), on the local computer for later processing;

3.

Data are transferred to RAM or disk of a more powerful remote computer, such as a super computer, for more efficient processing.

Scheme 1 is generally more appropriate for "smaller-scale" applications requiring "smaller" data volumes and processing times, such as single-pulse studies where data may be processed directly from RAM and only the final results, usually greatly compressed in volume, need to be saved to disk. Even for single-pulse studies, however, reasonable overall processing times demand a capable system both in RAM- and compute-capacity. In this scheme, the entire tape data volume is processed via multiple data transfer bursts to RAM. The primary advantage of this scheme is that the relatively slow transfer of data to disk, typically slower than transfer from S2 to computer RAM, is not needed, and that results are obtained directly. The disadvantage is that generally small data volumes must be processed at one time, and that reasonably large capacity RAM, which is currently cost-prohibitive, is required.

In scheme 2, data are written from RAM to disk or tape before processing. The advantage is that disks or tape devices, currently considerably more cost-effective than RAM, permit a larger data volume to be built up from S2 tape. The disadvantage is that writing to disk followed by subsequent reading from disk for processing generally decreases the overall data processing throughput. Bypassing the disk and writing directly to fast output tape devices (now becoming available) increase the data throughput rate. Furthermore, the advantage of transferring data from S2 tape to more standard tape media/formats such as Exabyte, DAT, DLT, etc., a function which may be performed at established S2 pulsar processing centers, is that S2 data may be made available in industry-standard formats which are generally supported by computers of all kinds. Thus, S2 data processing may effectively be performed at the user's home institution or (super) computer of choice.

Scheme 3 should be considered when the processing speed in the local computer is limited and a super computer can be used. If a direct interface from S2 tape to super computer is not available, data may be transferred from local RAM or disk across a data link such as ethernet, "fast ethernet", or the Fiber Distributed Data Interface (FDDI) to the super computer. The advantage of this scheme is the processing speed gain of the super computer. The disadvantage is the requirement for a high-speed link to the super computer. This scheme is clearly feasible only if the processing speed gain in the super computer exceeds the additional link transfer time. The S2-TCI system supports the three schemes discussed above.

In this section, we have described the most general system design considerations, with particular attention to modes of data transfer from tape recorder to local and remote computers. With regard to the system implementation, we note that this was largely independent of the precise choice of local (workstation) computer, as present technology at the time of writing provided sufficient capabilities (memory sizes, disk transfer speeds, etc.) for the tape-to-computer data transfer. (This implementation was, however, tied to a specific commercial vendor for the DMAC, which at the time of writing necessitated the Sun workstation line of computers.) For example, the first implementation utilized a modest configuration consisting of a "Sparc 20" workstation, with 32 MBytes of RAM, and commercial "disk striping" software to avoid the requirements on special systems (such as "RAID" systems) delivering fast disk speeds. In subsequent implementations regarding the local computer capability, increased RAM and sheer "number crunching" capability were most desired, and primarily for local pulsar signal processing speed in the absence of a more powerful remote computer.