“A Look at Apollo Ground Support Control”

by Captain Richard J. Stachurski

Published in Air University Review, March-April 1972.

“All stations, this is Network on Net Two. Lift-off, one three, plus three two, zero zero decimal seven eight. I repeat…” As the voice speaking from the Mission Control Center (MCC) in Houston, Texas, begins the second transmission, the lift-off time for another Apollo mission is relayed via landline, microwave, and communications satellites to the worldwide stations that make up the Manned Space Flight Network (MSFN) of the National Aeronautics and Space Administration (NASA). Simulations are complete, the launch countdown is over, and the remote stations settle down to support the long lunar mission.

The principal elements of NASA’s ground support system are the 85-foot antenna stations at Madrid, Spain; Honeysuckle Creek, Australia; and Goldstone, California. These stations are supplemented by a tracking ship, four instrumented aircraft, and eight 30-foot-antenna land stations located around the globe. In addition, the 210-foot antenna of the Jet Propulsion Laboratory station at Goldstone and the 210-foot antenna of the Australian Commonwealth Scientific and Industrial Research Organisation at Parkes, Australia, are used in support of the actual lunar landing and extravehicular activities whenever possible. Voice and data communications to all these stations from MCC are routed through the Goddard Space Flight Center in Maryland and subsequently through subsidiary switching centers at Canberra, London, Madrid, and Honolulu.

The elaborate communications system employed in support of each Apollo mission makes possible a centralized control of the manned space flight ground systems that was unknown during earlier programs. During the Gemini program, for instance, the majority of the stations in the tracking network were connected to MCC only by voice and 60- or 100-word-per-minute teletype circuits. To analyze mission data, it was necessary to deploy teams of vehicle and crew systems flight controllers to the various tracking stations. Although the Flight Director at MCC had ultimate mission responsibility, he and the other flight controllers at MCC were not able to view vehicle data in near real-time as were the remote station controllers. Thus, each station became a kind of semiautonomous control center, and the ground system was, for the most part, under the direct control of the deployed controllers.

Technical assistance on the Gemini ground systems was provided to the flight controllers by permanent station personnel and by a team of ground system specialists located at MCC. The group at MCC, known as the Network Support Team (NST), consisted of experts in command (transmission of control functions to the space vehicle), telemetry, tracking, remote station data processing, documentation, and station scheduling. With the exception of the tracking controller, who conducted C-band beacon handovers, the ground system experts exercised no direct control of the ground system. Their role was purely that of an expert staff. It is interesting to note also that the NST members were not permanently stationed at MCC; they were specialists who were normally stationed at Goddard Space Flight Center or the Air Force Eastern Test Range. The NST was only an ad hoc group, organized and deployed to MCC during actual mission periods.

During the initial stages of the Apollo program, NASA made several major changes in the flight control system that had been developed during the Mercury and Gemini projects. First, a decision was made to exercise control of future missions directly from the MCC. Flight controllers would no longer be deployed to the Manned Space Flight Network stations. The MSFN stations and their associated communications would henceforth serve as a medium for mission command and control, rather than as a series of semiautonomous control centers. Second, NASA decided to discontinue deployment of the NST to Houston during mission periods. Instead, a group of ground support specialists designated as the Instrumentation Support Team (IST) would be developed and permanently based in Houston.

Actually, the first of these changes was not a new idea. The MCC in Houston had been built with centralized mission control in mind, but lack of adequate communications forced the adoption of the somewhat decentralized system used during Gemini. But by the time the Apollo control system was developed, the idea of truly centralized control became practical because sufficiently reliable, high-capacity data circuits were then available.

The creation of the IST was largely motivated by the desire to have the ground support team available for integrated training with the vehicle and crew systems flight controllers during the entire premission simulation period. This kind of integrated training was thought to be particularly required because the upcoming Apollo missions would be more complex and difficult than any flown during the Mercury and Gemini periods.

The IST developed for the support of Apollo is the central element in a system of ground support control that differs radically from the Gemini system. The members of the IST are actively engaged in configuring and operating the MSFN ground systems rather than simply advising on technical aspects of the system. The IST collects and coordinates flight control data requirements, translates these requirements into ground system configurations, and then monitors and controls the receipt of the mission data. The way in which the IST functions in controlling the flow of data through the Apollo ground system can best be described by briefly outlining the functions of each operating position.

Like the NST, the IST positions were developed on a functional basis. Thus, each IST shift operating during an Apollo mission is made up of a team leader and his assistant, three command controllers, two telemetry controllers, two tracking controllers, two communications controllers, an air-to-ground controller, and specialists in scheduling and documentation.

The IST team leader, the network controller, has operational control of all the ground systems supporting a mission except for the Real Time Computer Complex (RTCC) at MCC. The functions of the RTCC have been judged to be so complex and critical that a special computer supervisor, assisted by a team of software specialists, is responsible for this system. Both the network controller and the computer supervisor report directly to the Flight Director, who has overall responsibility for the mission.

Principal tasks of the Network Controller

The principal tasks of the network controller during an Apollo mission fall into two broad categories. The first category includes those tasks associated with supervising the execution of the nominal ground support plan. This plan is prepared premission and provides a detailed timeline, which shows how available ground support resources are to be allocated in support of mission data requirements. A typical task in this area is the supervision of interface testing prior to a remote station’s support period, to ensure that the station is properly configured per the ground support plan.

The second broad category of network controller tasks involves replanning of ground support when the mission does not proceed according to the nominal plan. This replanning may be done on a long-term basis or on a very immediate basis. Apollo 13 provided the most striking example of the long-range or extensive replanning activity when the explosion of an oxygen tank forced abandonment of the original mission plan. Consequently, new view period tables and tracking assignments for the ground network had to be generated on a continuing basis throughout the mission.

In contrast to this type of activity, which covered many days, replanning of ground support may also be done on a real-time basis. This type of activity is probably familiar to operations personnel in all fields as the cause of well-remembered moments of sheer terror. Usually decisions of this kind are necessitated by failure in the ground system that must be immediately corrected. A classic example occurred during the critical translunar injection burn of Apollo 11. Command computer problems onboard the tracking ship Mercury forced the network controller to decide on a late handover of transmitting responsibility from the tracking ship Redstone to the Mercury.(1) The hand over was held until the last moment, until imminent loss of signal by the Redstone forced the issue. When the switch was finally made, Mercury experienced numerous signal dropouts, and the metric tracking data were virtually useless. Once again a decision was made to carry out a contingency handover to the Hawaii tracking station, which had just acquired the spacecraft signal. This whole process covered a time span of approximately three minutes.

During those hectic three minutes and in similar situations, the network controller, as IST leader, has final responsibility for the actions taken to alter the ground support configuration. But since the network controller is backed up by a team of specialists, all his decisions are based on the advice of his team members, and in most cases the decisions are implemented by the IST member. The contingency station handovers required during the Apollo 11 flight, for instance, were actually executed by one of the IST command controllers who has responsibility for control of the station transmitter on and off times.

Assistant Network Controller

Before we discuss the functions of the various IST system controllers in more detail, one special duty performed by the assistant team chief, known as the assistant network controller, is worth noting. As his title implies, this team member aids the network controller, with whom he shares a console, in carrying out the overall task of controlling the ground support system. In addition, the assistant network controller has particular responsibility for the operation, configuration, and troubleshooting of the supporting systems within the MCC. This responsibility includes every system within the control center except the RTCC, which is under a separate supervisor. Thus, the individual supervisors of the display, telemetry, power, and communications systems within MCC report their status and configuration to the assistant network controller, who then can monitor the internal configuration for consistency with the overall plan of operations for the ground system. The assistant network controller effectively serves the IST as an expert on the MCC portion of the overall Apollo system, and the sight of flickering lights or blank displays has been known to accelerate his pulse rate greatly, especially during a launch.

Other IST Controllers

The other IST controllers also have some excitement in their own areas of responsibility. The telemetry controllers, for example, lead a hectic life. They must see that vehicle system performance measurements and flight crew biomedical parameters are successfully received by the ground stations and transferred to Houston for subsequent processing and display. Since most of the critical phases of an Apollo mission involve multiple vehicles, the telemetry controllers must be certain that the various vehicle data streams are handled according to the priority established for that phase. They must also be certain that the proper combinations of vehicle measurements for that phase are shipped to MCC, since communications limitations prohibit the transfer of all the measurements available at any given time. Finally, the telemetry controllers must make certain that the data suffer no degradation en route from the remote station to the end display device at MCC.

The command controllers are responsible for data going in the opposite direction, from MCC to the spacecraft via a ground station. Data transferred from MCC include commands for controlling the remote station computers, commands for controlling vehicle functions, and updates to the spacecraft computers. The command controllers must ensure that the data are successfully transferred to the remote computer, properly formatted for transmission to the space vehicle, and then transmitted. If, after transmission, the spacecraft does not verify data receipt, the command controllers must work with the vehicle flight controllers to determine the reason for the rejection of the data.

Besides telemetry and command data, the IST is responsible for tracking data. The tracking controllers monitor the flow of S-band and C-band tracking data from the remote stations to the Houston RTCC, where they are finally used in computing trajectory solutions. The tracking controllers also make certain that the stations are provided with the pointing information they need to acquire each spacecraft they are assigned to track. Like the other IST controllers, the tracking controllers troubleshoot system problems, keep Network advised of their system status, and recommend alternate configurations whenever required.

In addition to their specific system-oriented function, the tracking controllers perform a second function that is a fundamental element in the Apollo ground support control plan. Ground support, as pointed out earlier, is planned in great detail before the mission ever lifts off. The gathering of requirements for this plan and the matching of MSFN resources against the requirements are carried out primarily by the IST tracking controller group at Houston. Once a mission countdown begins, the tracking controllers, working under the supervision of the network controller, become the chief instrument for executing the support plan. They gather last-minute requirements changes from the various vehicle and crew systems flight controllers and alter the basic plan accordingly. When the plan has been updated (once per shift or as required by mission deviations), the trackers use it to prepare a Site Configuration Message (or SCM, as it is known to MSFN controllers worldwide).

From a ground support point of view, the SCM is probably the most important message generated at MCC during an Apollo mission. An SCM is sent to each ground tracking station prior to the start of its view period. The message contains all of the basic information required to set up the station for the next spacecraft pass. This information includes the vehicles to be tracked, the tracking mode, i.e., transmit or receive only, carrier on and off times if the station is to transmit, and the required configuration for transmitting biomedical data. The SCM is transmitted by teletype to each station, and it can be updated by subsequent messages or by voice instruction if the situation requires. The accuracy and timeliness of the SCM are fundamental concerns in assuring successful ground support for a mission, and the role of the tracking controllers in achieving this success is clearly a large one.

Successful mission support depends also on the functioning of the ground communications system that links the remote stations with MCC. The voice and data circuits in this system are the responsibility of IST members who, logically enough, are designated communications controllers. Working with a communications manager at Goddard Space Flight Center, the MCC communications controllers must set up and check out the required circuit configuration for each mission phase. If failures occur, they must move immediately to restore service by using spare circuits or lower-priority circuits.

The highest-priority circuit during any Apollo mission is the one assigned the air-to-ground function. Maintenance of uninterrupted voice communication with the ground crew is considered so important that a special air-to-ground communications technician position was created to ensure this function. This technician, who uses the call sign “comm tech,” is theoretically an assistant to the communications controllers. However, in practice, the comm tech takes the majority of his direction from the network controller and the CAPCOM, who is an astronaut assigned the task of actually communicating with the crew. The life of a comm tech can be a difficult one, not only because of the high priority of the voice link but also because of the great flexibility of the air-to-ground system.

For instance, the system can be set up to allow the CAPCOM to hold a two-way conversation with crewmen on the lunar surface while the crewman orbiting in the Apollo command module does not hear the CAPCOM’S part of the conversation at all. If the CAPCOM wishes to speak to the command module pilot, he uses a separate path, and this conversation is not heard by the lunar-surface crewmen. It is sometimes difficult to tell “who’s on first,” and the comm tech’s job is demanding, especially since air-to-ground configurations are not included on the SCM. To assure direct control and flexibility, the comm tech controls his system strictly with voice instructions rather than by teletype message. He has to know exactly what he is talking about and who is supposed to be talking to whom.

The two IST positions that remain to be discussed are not nearly as hectic as that of the comm tech, but both these positions involve essential housekeeping functions. The scheduling controller, as his title indicates, is responsible to Network for calling up stations to support at the proper time and for releasing the stations at the end of their view period. The scheduler also maintains up-to-date status on the MSFN systems, for use by all the other IST members in planning or replanning ground system support.

The great amount of message traffic generated by the IST and the various stations providing ground support is the responsibility of the documentation controller. The documentation controller must screen incoming traffic, distribute it to the appropriate controllers, and maintain reference files for all essential messages. Outgoing messages also require some handling and filing. The documentation controllers, unlike most of us, are actually paid to shuffle papers, and they get a great deal of practice, handling some 4000 messages during a normal mission.

These, then, are the positions that constitute the Apollo Instrumentation Support Team, a team that is the focal point for preparing and executing ground operations in support of manned space flight missions. How well does the IST system of control work? One might simply look at the team’s record of success and answer that the system works very well indeed! However, if the Apollo system is to provide lessons for future programs, its advantages and disadvantages must be clearly examined.

One of the principal advantages of the IST system of ground support control has been its flexibility. Throughout the Apollo program, the MSFN has been called on to support multiple vehicles and flight systems. As the need arose to support these vehicles and systems, the IST was able to take the requirements of the various vehicle and system flight controllers and translate them into an effective ground support plan. The IST, in effect, provides an interface between the various flight controllers and the ground support system which permits the MSFN to provide coordinated support for multiple vehicles.

The flexibility of the IST system has led to a good deal of specialization and resultant efficiency. Vehicles and crew systems controllers have been able to concentrate their attention almost exclusively on the operation of their own system. The various spacecraft controllers have effectively been able to take the MSFN bus and leave the driving to the IST controllers. On the other hand, the specialization allowed by the IST approach has provided a reserve of controllers who are thoroughly familiar with the ground system and can respond rapidly to contingencies.

One further advantage of the Apollo ground control system as it presently exists is the centralization of overall ground system control at the network controller position. This position serves as a kind of clearinghouse for information on the status of MCC and MSFN systems. When a failure occurs, the remaining resources can be rapidly evaluated and allocated to minimize the effect of the failure.

On the negative side, the IST system, since its beginnings, has been plagued by problems involving clouded lines of authority and responsibility. The individual IST controllers are often caught between the proverbial “rock and a hard place.” To provide effective support, they must respond rapidly to the requirements of individual flight controllers. But, at the same time, they must operate under the supervision of their own team leader, who has to coordinate their individual activities and meet the overall ground support objectives. Obviously, the requirements of the individual flight controllers and the overall mission priorities do not always mesh, and then a compromise must be devised or a ruling sought from the Flight Director. This situation, although undesirable, has never seriously jeopardized mission support. It has resulted, of course, in more than one exchange of heated words, and these exchanges are likely to continue, since the only solution seems to be strict supervision by the network controller of all IST activities. The idea of this kind of supervision has generally been rejected by the controllers involved because of the inherent sacrifices in response time and flexibility.

Despite this difficulty, the IST system has indeed worked well. Until such time as programs like the proposed space shuttle provide a large degree of spacecraft independence from ground support systems, the IST approach might well be considered for use in any program, manned or unmanned, that requires extensive ground support.

Aeronautical Systems Division, AFSC

Note
1. The tracking network for Apollo 11 included four ships, Mercury, Redstone, Huntsville, and Vanguard. Only the Vanguard is still active.

Contributor:
Captain Richard J. Stachurski (B.A., Manhattan College) is a Special Systems Program Officer in the Drone/Remotely Piloted Vehicle Systems Program Office at Wright-Patterson AFB, Ohio. He spent two tours with NASA working on the Apollo Manned Spaceflight Network. Captain Stachurski was a primary flight controller for the Apollo 11 mission and a member of the Apollo 13 operations team. He is a graduate of Squadron Officer School.

(Headings and bold emphasis added.)