THIS OPINION WAS INITIALLY ISSUED UNDER PROTECTIVE ORDER AND IS BEING RELEASED TO THE PUBLIC IN REDACTED FORM ON MARCH 23, 1995 ______________________________________ GRANTED IN PART: July 15, 1994 ______________________________________ GSBCA 12824-P COMPUTER DATA SYSTEMS, INC., Protester, v. DEPARTMENT OF ENERGY, Respondent, and DYNCORP, Intervenor. Marcia G. Madsen, Andrew D. Ness, Brian W. Craver, and David F. Dowd of Morgan, Lewis & Bockius, Washington, DC; and Charles B. Machion and Peter A. Fish of Computer Data Systems, Inc., Rockville, MD, counsel for Protester. Patricia D. Graham and Ralph C. Oser, Office of General Counsel, Department of Energy, Washington, DC, counsel for Respondent. David S. Cohen, G. Brent Connor, C. Patteson Cardwell, IV, Laurel A. Heneghan, and William F. Savarino of Cohen & White, Washington, DC; and Stuart Young of DynCorp, Reston, VA, counsel for Intervenor. Before Board Judges DANIELS (Chairman), WILLIAMS, and GOODMAN. WILLIAMS, Board Judge. On May 2, 1994, Computer Data Systems, Inc. (CDSI) protested the award of a contract by the Department of Energy (DOE) to DynCorp for information resources management (IRM) support services. DynCorp intervened. CDSI is the incumbent contractor for most of the services being acquired. In its original fifteen-count complaint, CDSI alleged that DOE had relaxed the mandatory experience requirements for DynCorp, conducted an improper technical evaluation of both proposals, failed to conduct meaningful discussions, performed an improper responsibility determination for DynCorp, and conducted an arbitrary and capricious cost evaluation. Further, CDSI alleged that there was an impermissible organizational conflict of interest between DynCorp and DOE, and that the award decision was tainted by bias.1 On May 11, 1994, CDSI filed its first amended complaint, adding an allegation of technical transfusion/leveling. Subsequently, CDSI filed second and third amended complaints adding further factual allegations.2 For the reasons stated below, we grant the protest in part. We agree with CDSI that DOE conducted an improper technical evaluation of CDSI's and DynCorp's proposals in the following respects. The agency had no objective standards for defining strengths and weaknesses in proposals. The Source Evaluation Board (SEB) had no consistent definition of "similar size and scope" of reference contracts. The reference check for CDSI's contracts was inadequate and resulted in an unfair evaluation of those contracts. The evaluation of DynCorp's reference contracts was also erroneous in that DOE improperly credited DynCorp with more corporate experience than was warranted. We grant two aspects of CDSI's complaint regarding the cost evaluation, concluding that DynCorp's and CDSI's proposals were not treated equally. First, DynCorp was told that a Wyatt Data Services Survey would be used in evaluating its cost proposal, in particular in assessing the realism of direct labor rates; DynCorp was further told that any rates below the 50th percentile in the survey would be deemed unrealistically low and that the ____________________ 1 In a motion filed on May 24, 1994, protester sought summary relief on two grounds. First, protester contended that DynCorp's proposal failed to comply with the mandatory requirement that offerors describe their experience on at least three contracts similar in size and scope to the instant contract. Second, CDSI claimed that one of DynCorp's key personnel, its proposed administration manager, did not have the requisite two years of experience in administering a contract of similar size and scope within the last five years. Because we determined that these allegations raised genuine issues of material fact, we denied the motion on June 2, 1994. Conference Memorandum (June 3, 1994). 2 By letter dated June 20, 1994, CDSI withdrew Count XIV and paragraphs 56, 57, 82, and the second sentence in paragraph 114 of the third amended complaint. agency would look to the 50th percentile rates to establish a probable cost. CDSI was not given this information, yet the agency utilized the Wyatt Data Survey to evaluate direct labor rates for cost realism. Second, the agency gave disparate treatment to uncompensated overtime offered by DynCorp and CDSI. DOE, relying on conclusions of the Defense Contract Audit Agency (DCAA), indicated that CDSI should offer a diluted rate based upon a work week of hours because it paid its employees if they worked overtime. Although DynCorp had a similar system, DOE suggested no adjustments for DynCorp's proposed uncompensated overtime. The Board denies CDSI's allegation that the contracting officer improperly forced CDSI to its direct labor rates when CDSI offered as permitted by the request for proposals (RFP). We conclude that the contracting officer's efforts in seeking information regarding CDSI's were directed at assessing the cost realism of CDSI's proposal by ensuring that the individual salaries proposed for CDSI's rates were representative. In so concluding, we recognize the discretion accorded to contracting officers in performing a cost realism determination. We also deny CDSI's allegation that the contracting officer forced CDSI to "prequalify" individuals for positions regardless of the fact that the RFP did not require offerors to identify specific individuals for non-key positions. We find that the agency conducted improper discussions in three regards: it failed to advise CDSI that it only had contract of similar size and scope; it failed to advise both CDSI and DynCorp that one of each firm's key personnel did not meet the mandatory requirements of the RFP; and it gave DynCorp, but not CDSI, beneficial information with regard to the evaluation of its cost proposal. We deny CDSI's contention that there was an impermissible organizational conflict of interest between DynCorp and DOE, finding the relationship cited -- the performance of an unrelated maintenance and operations (M&O) contract by a subsidiary of DynCorp in Louisiana -- to be tenuous and the potential for conflict speculative. Importantly, if a conflict were to arise, we conclude that it could be mitigated after award by denying DynCorp access to certain data. We also deny CDSI's contention that DOE performed an improper responsibility determination for DynCorp due to DynCorp's alleged lack of financial stability and lack of business integrity. The record supports a conclusion that DynCorp has the financial capability to perform the contract. The contracting officer was not required to conclude that warranted a finding that DynCorp was nonresponsible. Nor do we conclude that the award decision was tainted by bias. CDSI has not proffered "well nigh irrefragable proof" that the evaluators failed to act in good faith. Finally, we find no violation of statute or regulation by virtue of the fact that DynCorp had internal CDSI data reflecting CDSI's organization and staffing of its incumbent contract. These documents were not utilized by DynCorp in its proposal, and it is questionable whether these materials remained proprietary. With regard to the remedy, we revise the agency's delegation of procurement authority (DPA) by directing DOE to empanel a new SEB, establish objective standards for evaluation, clarify its minimum needs and its cost evaluation mechanism, and proceed in accordance with statute and regulation. Findings of Fact The Solicitation On August 25, 1992, DOE issued request for proposals number DE-RP01-92AD32229. Consolidated RFP.3 The solicitation requested IRM support services for the Office of Information Technology Services and Operations (ITSO) and contemplated award of a cost-plus-award fee contract for a thirty-six month base period with two twelve-month option periods. Id. at C 012607-08. Section B.007 advised offerors that cost proposals would be evaluated on the assumption that the contractor would provide 4,185,000 total direct productive labor hours (DPLH) for the thirty-six-month base period. Consolidated RFP at C 012613. In addition, paragraph B.011 provided the estimated level of effort for the option period as follows: 1,488,000 DPLH for the first twelve-month option and 1,581,000 DPLH for the second twelve- month option. Id. at C 012617-18. Offerors were to provide their total estimated base fee and maximum award fee for the base and option periods. The RFP in Section C described the twenty-four IRM functions required to be supported: 1. Application Systems Development 2. Application Systems Maintenance 3. Automated Office Support System (AOSS) Services 4. Management Analysis 5. Secure Automated Communications Network (SACNET) Support 6. Computer System Software Support ____________________ 3 By agreement of the parties, protester submitted at the hearing a "consolidated" RFP which incorporated all amendments. This RFP shall be referred to as "Consolidated RFP." Citations to pages in the RFP shall be to the Bates-stamped numbers, consistent with practice during the hearing. 7. Computer System Operations, Maintenance and Customer Services 8. Communications Center 9. DOE Crisis Management and Emergency Management 10. Data Entry Support 11. Teleprocessing Services 12. Network Services 13. Video Teleconferencing Management 14. Radio Communications and Frequency Management 15. National Security/Emergency-Preparedness 16. DOE Open System Standards Program 17. Headquarters Telephone Switchboard Operations 18. Field IRM Support 19. Records Management 20. IRM Planning 21. Technical Training 22. Information Systems Security 23. Technology Assessment 24. Oversight Activities Consolidated RFP at C 012627-42. Paragraph H.012, Contractor Personnel, provided in pertinent part: (a) The contractor shall provide the skilled personnel and the supervision, management and administrative services necessary to meet the Government's requirements successfully. Personnel assigned to work on the contract must meet the minimum qualifications for the applicable labor category as stated in Part III, Section J, Attachment M of this contract except as authorized by the Contracting Officer. Consolidated RFP at C 012705. Paragraph H.012(e) also expressly provided that "[a]ll personnel shall be dedicated (assigned full time to this DOE contract only) in support of this contract . . . No overtime, compensated or uncompensated shall be accomplished without prior COR [contracting officer's representative] approval." Id. at C 012706. Paragraph H.014 of the solicitation, Key Personnel, required offerors to identify six key personnel, i.e., the contract manager, the oversight activities manager, the operations manager, the systems development and maintenance manager, the network manager, and the administration manager. Consolidated RFP at C 012708. No substitutions of key personnel were permitted in the first 365 days of the contract absent illness, death, or termination. Id. Paragraph H.024, Position Qualifications, provided: Contractor direct labor personnel assigned to the performance of this contract shall satisfy as a minimum the applicable labor category qualifications, both education and experience, set forth in the "Generic Position Descriptions" attachment J to this contract, except as the Contracting Officer may authorize. Consolidated RFP at C 012716. The position description for the administration manager set forth in Attachment J of the RFP mandated the following experience: A minimum of 10 years progressive responsibility is required in contract administration. At least 2 years of recent experience must be in the administration of a contract of similar size and scope. Consolidated RFP, Attachment J at C 012862. In the body of all of the other position descriptions for key personnel in the RFP, the term "recent" in the context of "recent experience" was expressly stated to mean "(within the last five years)." Id. at C 012857-61. The position description for the systems development and maintenance manager mandated that this individual have "at least 4 years experience managing a functional area staffed by a minimum of 100 people." Consolidated RFP at C 012860. The RFP in paragraph L.046(j), instructions for preparation of the offer, permitted offerors to take exceptions and deviations to the model contract but stated: (2) Any exceptions, etc., taken must contain sufficient amplification and justification to permit evaluation. The benefit to the Government shall be explained for each exception taken. Such exceptions will not, of themselves, automatically cause a proposal to be termed unacceptable. A large number of exceptions, or one or more significant exceptions not providing benefit to the Government, however, may result in rejection of your proposal(s) as unacceptable. Consolidated RFP at C 013052. Paragraph L.048, proposal preparation instructions for the technical proposal, provided in pertinent part: Part B. Technical Discussion. This section . . . should clearly address each of the Technical Proposal evaluation criteria in Section M, and at a minimum, cover the subordinate factors or subcriteria listed thereunder, if any. . . . Section 1. The offeror's corporate experience with contracts of similar scope and size. A. The offeror must describe corporate experience on not less than three, nor more than five IRM Support Services contracts similar in scope and size to this proposed contract. The corporate experience presented should reflect support services in ADP and Telecommunications as well as other functional areas described in the definition of IRM. Contracts submitted to describe company experience must have been performed within the last 5 years by the proposing entity. Give the following information on each contract: 1. Contracting entity -- name, address, and phone number of the Technical Officer and the Contracting Officer. 2. Contract number, type, and total contract value. 3. Period and place of performance. 4. Two page summary of scope of work which includes a section on how that contract is, or was, similar in scope and size to this proposed contract. Attach a list of the ADP and telecommunications hardware and software supported. B. Subcontractors proposed for whom the DPLH equals or exceeds 10 percent of the total estimated DPLH for this solicitation shall provide one contract reference. This reference shall reflect technical experience, performed within the last 5 years, similar in scope to the work proposed. Give the following information on the contract reference: . . . . 5. A summary of the scope of work, not to exceed two pages, which includes a section on how that subcontract is, or was, similar in scope to the work proposed. Attach a list of the ADP and telecommunications hardware and software supported in performance of this subcontract. The following notes apply to both prime and subcontractors: 1. Prime contractor versus subcontractor performance should be clearly indicated. 2. The Technical Officer, Contracting Officer, and other associated with any of the supplied references may be contacted as part of the evaluation process. Section 2. Capabilities and Experience. The offeror must describe the capabilities and directly related experience for performing the functions in the Statement of Work. The offeror must address all functions specified in the Statement of Work. For each function addressed the offeror shall cross reference the submitted contract references (i.e., Reference 1- 5). The offeror's management and approach should be addressed in section 3 and not in this section. Consolidated RFP at C 013053-55. Section 3 addressed the offeror's plan for contract management and mandated that the offeror discuss its staffing, quality management program, management approach, oversight, management reporting, cost management, and phase-in. Id. at C 013055-56. That section included the following specific instructions: b. Quality Management Program The offeror must describe, in a concise and comprehensive manner, the processes used to (1) detect problems and errors during the performance of assigned tasks and (2) deliver quality products and services on schedule and economically. c. Management Approach The offeror must describe, in a concise and comprehensive manner, the organization structure and plan to manage all facilities, equipment, and subcontractors used in support of this contract. . . . d. Oversight The offeror must describe . . . the authorities and management reporting of the Oversight Manager and his supporting staff. Id. at C 013056. None of these paragraphs referenced an offeror's "corporate" quality management program or oversight program. Id. Section 4, "Key Personnel," required that offerors submit resumes for the six key personnel and specified, in pertinent part: b. Candidates' resumes must demonstrate the management, experience, and expertise necessary for managing complex IRM systems. c. Candidates shall meet at a minimum the required position description qualifications in Section J. Consolidated RFP at C 013056-57. Paragraph L.049 set forth proposal preparation instructions for cost proposals. The primary element of cost under the solicitation was direct productive labor hours. The solicitation in Attachment 4, Exhibit I, listed thirty-nine labor categories and set out the estimated number of DPLH which would be ordered for each labor category for each year of the contract. Consolidated RFP at C 012985. The labor category of senior systems analyst contained the most hours under the contract, 1,404,300. A significant number of hours were attributable to the programmer analyst, 697,500 hours, the systems analyst, 558,000 hours, and the senior AOSS support specialist and AOSS support specialist, each of which required 306,900 hours. Id. The DPLH for the entire five-year period of performance totaled 7,254,000 hours. The hours listed in Exhibit I were mandatory, for purposes of estimating, and no deviations could be taken from those hours. Id. Section L.049 1.(a), paragraph L.049, included the following section on direct labor hours: (c) Direct Labor Hours: Direct labor hours shall be proposed in accordance with the labor categories and labor hour requirements described in Attachment 4, Exhibit-I of the solicitation. The manpower requirements are summarized in Attachment M. Direct labor shall be proposed on the basis of Direct Productive Labor Hours (DPLH), i.e., estimated number of hours on the job. Should an offeror's accounting system require nonproductive labor hours (vacations, holidays, sick leave, etc.) to be charged as direct labor, all such costs shall be separately identified and priced. Each offeror must propose the number of labor hours by labor category as specified by Exhibit-I. . . . . NOTE: Modification of the DPLH or the labor mix is not permitted, except for (1) clerical and/or administrative hours, as discussed above or (2) any additional labor categories or DPLH above that identified in Attachment 4, Exhibit I. Consolidated RFP at C 013058. Further, the RFP advised offerors to submit separate exhibits A through L as part of their cost proposals, including: 1. Exhibit-A: Standard Form 1411 . . . . 3. Exhibit-B: Labor Rates for Proposed Individuals and Categories This exhibit shall be prepared showing direct labor rates only. All named personnel, including key personnel . . . shall be separately identified on this exhibit. In those instances where the DPLH or a portion of the DPLH is to be performed by unnamed individuals, a category or composite rate may be used. The following applies when proposing other than individual rates. (i) The proposed category rates should reflect those identified in your accounting system. If your category titles do not correspond to the RFP categories and their applicable qualification then enter your corresponding category title in the second column of the Exhibit-B along with the RFP Category name in the first column. (ii) If it is necessary to propose more than one of your category titles to satisfy a required RFP Labor Category, then provide more than one (1) entry for each of the RFP category titles. For example, if you intend to satisfy a RFP Labor category of "Senior Analyst" with your categories of "Analyst-I" and "Analyst-II", then you would enter the RFP labor category of "Senior Analyst" twice, and provide the category rates for both "Senior Analyst/Analyst-I" and "Senior Analyst/Analyst-II". (iii) If your accounting system does not maintain or use true category rates for proposal and cost accounting purposes, then category rates should not be proposed. Instead, the development and use of a composite rate composed of individual rates which correspond to the RFP labor category requirements would be acceptable. If it is not possible to develop a composite RFP category rate, then individual rates should be proposed for each of the RFP categories. List the RFP labor category, the individual's name or the offeror's category/composite rate title, current hourly rate, "actual as of" date (the "actual as of" date for the current actual hourly rate identifies the date the last increase was given to an individual or category), the yearly escalation rates, and the projected rate for each year for the proposed contract. . . . . In an attachment to Exhibit-B show how any proposed category or composite rates were developed for each labor category and how the key/named personnel and the other personnel in the category were weighted. Also identify and discuss the basis and development of the rate and its application for the initial year and each year of use thereafter. Consolidated RFP at C 013059-61. On Exhibit B1 to the cost proposal, offerors were required to provide a summary of the direct labor cost for each contract year by listing the RFP labor category, composite rate, or individuals' rates from Exhibit-B, the number of hours per year per individual/category, and the cost (rates x hours) by category. Consolidated RFP at C 012975. A summary page showing total labor dollars by year was also required on the last page of Exhibit B1. Id. at C 012976. The pricing tables also contained another mandatory exhibit, Exhibit E, entitled "Reconciliation of RFP and Offeror Labor Categories," which provided: If the offeror's labor categories differ from the RFP labor categories, the offeror shall supply a cross reference between the RFP labor categories and the offeror's labor categories. If specific individuals, rather than an offeror's labor category are proposed (named within Exhibits-B, or B1), then those individuals shall be identified. Consolidated RFP at C 013063. Paragraph M.001(b) provided that "[a]ward will be made to that responsible offeror, whose offer, conforming to this RFP, is considered most advantageous to the Government, considering the Evaluation Criteria in this Section M." Consolidated RFP at C 013082-83. Paragraph M.009, EVALUATION CRITERIA - SUPPORT SERVICES (SEB), provided: (a) Technical Criteria: Technical aspects of the proposals will be evaluated in accordance with the following criteria and scored on a numerical basis. Criteria 1, 2, and 3 are approximately equal in importance. Criterion 4 is significantly less important than criterions 1, 2, and 3. CRITERION 1: The offeror's corporate experience in performing contracts of similar scope and size. CRITERION 2: The demonstrated ability and directly related experience for performing the functions of the statement of work. CRITERION 3: The adequacy and appropriateness for each of the following subcriteria of the offeror's plan for contract management. The subcriteria are of equal importance. a. Staffing b. Quality Management Program c. Management Approach d. Oversight e. Management Reporting f. Cost Management g. Phase In CRITERION 4: The Key Personnel qualifications, and management experience and expertise necessary for managing complex IRM systems. (b) Cost Criteria: The cost proposal will be evaluated to determine the probable cost to the Government including all options, taking into consideration the allowability, allocability, reasonableness, adequacy, and realism of proposed costs, and reasonableness of proposed fee. The Cost Proposal will not be point scored, assigned a numerical weight, or adjectivally rated. However, an unrealistic cost or price proposal may be evidence of the offeror's lack of understanding or poor understanding of the project. The cost proposal will be evaluated for determination of the probable cost to the Government for those offerors in the competitive range, subject to the provisions of FAR 15.609, following technical evaluation. Probable cost will be based upon cost analysis performed in accordance with the applicable cost provisions of FAR [Federal Acquisition Regulation] and DEAR [Department of Energy Acquisition Regulation]. (c) Offeror's Overall Proposal: The Government will evaluate the offeror's overall proposal (all volumes) as a representative product of the offeror's quality of work and quality assurance. This evaluation will not be point scored, but will be considered where points are applied to technical areas which are associated with the development of written products and quality assurance. Consolidated RFP at C 013084-85. Paragraph M.013, FAR 52.222-46 Evaluation of Compensation for Professional Employees (Feb 1993), provided: (a) Recompetition of service contracts may in some cases result in lowering the compensation (salaries and fringe benefits) paid or furnished professional employees. This lowering can be detrimental in obtaining the quality of professional services needed for adequate contract performance. It is therefore in the Government's best interest that professional employees, as defined in 29 CFR 541, be properly and fairly compensated. As a part of their proposals, offerors will submit a total compensation plan setting forth salaries and fringe benefits proposed for the professional employees who will work under the contract. The Government will evaluate the plan to assure that it reflects a sound management approach and understanding of the contract requirements. This evaluation will include an assessment of the offeror's ability to provide uninterrupted high-quality work. The professional compensation proposed will be considered in terms of its impact upon recruiting and retention, its realism, and its consistency with a total plan for compensation. Supporting information will include data, such as recognized national and regional compensation surveys and studies of professional, public and private organizations, used in establishing the total compensation structure. (b) The compensation levels proposed should reflect a clear understanding of work to be performed and should indicate the capability of the proposed compensation structure to obtain and keep suitably qualified personnel to meet mission objectives. The salary rates or ranges must take into account differences in skills, the complexity of various disciplines, and professional job difficulty. Additionally, proposals envisioning compensation levels lower than those of predecessor contractors for the same work will be evaluated on the basis of maintaining program continuity, uninterrupted high-quality work, and availability of required competent professional service employees. Offerors are cautioned that lowered compensation for essentially the same professional work may indicate lack of sound management judgment and lack of understanding of the requirement. (c) The Government is concerned with the quality and stability of the work force to be employed on this contract. Professional compensation that is unrealistically low or not in reasonable relationship to the various job categories, since it may impair the Contractor's ability to attract and retain competent professional service employees, may be viewed as evidence of failure to comprehend the complexity of the contract requirements. (d) Failure to comply with these provisions may constitute sufficient cause to justify rejection of a proposal. Consolidated RFP at C 13086-87. Vendor Questions and DOE's Answers On October 2, 1992, DOE issued Amendment 001 to the solicitation which included vendor's questions and DOE's responses. Consolidated RFP at C 012568-95. Question 1 and Response 1 stated: Question 1: REFERENCE: L.048, PROPOSAL PREPARATION INSTRUCTIONS, (b), PART B. TECHNICAL DISCUSSION SECTION 1, The Offeror's Corporate Experience with contracts of similar scope and size (pg L.22), and SECTION M.004(a)1, CRITERION 1, The offeror's corporate experience with contracts of similar scope and size (pg. M-3). If two or more companies choose to form a joint venture for this program, will the experience of individual joint venture partners be evaluated as experience of 'the proposing entity'? Response 1: Subcontractor(s)' (joint venture partner(s)) experience will be evaluated as experience of the prime contractor. However, for evaluation under Criterion 1 the prime contractor (one firm) must submit not less than three contracts of similar scope and size through its own corporate experience, i.e., two or more companies with contracts not of similar size and scope cannot be combined and considered as similar scope and size because between them they performed the total or majority of functions described within the RFP. Consolidated RFP at C 012568. Question 37 and Response 37 stated: Question 37: Part IV, Section M.009, Evaluation Criteria - Support Services states that proposals will be evaluated in accordance with four criteria, including 'Criterion 1: The offeror's corporate experience performing contracts of similar scope and size.' What range of level of effort or contract value constitutes similar size? What is the justification for requiring similar size if a firm has multiple contracts of similar scope, complexity and objective? Is this criterion intended to discourage bids from firms with corporate experience performing contracts of smaller size but similar scope? For example, will a firm that has performed contracts of similar scope but multiple contracts of one third to one half the size be significantly penalized under this criterion? Response 37: Similar in size will be judged against the estimated level of effort in Section B of the RFP, but no specific range has been established so as not to unduly restrict competition. Although not solely, scope is also considered in terms of volume as well as the types of ADP support services, not merely the types of ADP support services to be provided. While specific ratings could not be established on undefined contracts at this stage of the process, it would be fair to say that contract references which depict contracts similar in scope and size would score higher than contracts non-similar in size and scope. Consolidated RFP at C 012581. Question 40 stated: Refer to Section L.048 - Proposal Preparation Instructions - Technical Proposal, Section 2, Capabilities and Experience (page L-023). The solicitation states that the offeror must cross reference each function in the SOW to the submitted contract references. Does the Government mean that bidders may only describe their capabilities and directly related experience in accordance with the contracts they provide to comply with Section 1 of the RFP? Or may bidders demonstrate their capabilities and directly related experience by work they accomplished on other contracts provided that cross references to submitted contracts are cited appropriately? Please clarify. Response 40 stated: The offerors['] demonstrated ability and directly related experience shall be demonstrated only through the prime['s] three to five contract references and subcontractor references submitted in accordance with the instructions. The demonstrated ability and directly related experience will be verified through the Technical and Contracting Officers of the referenced contracts. Ability and experience on contracts other than those specified . . . in Section L will not be evaluated. Consolidated RFP at C 012582. Question 50 and its response by DOE provided: Question 50: Page 1 of Attachment J requires the Contract Manager to have experience '. . . managing a contract of similar size and scope.' Can this management experience encompass several concurrent contracts that, when taken in total, are similar in size and scope to this procurement? Response 50: Yes, but the contracts must have been or [be] in operation at the same time, not merely intersecting at completion and phase-in. Your attention is directed to the fact that this response does not apply to Criterion 1. Consolidated RFP at C 012586. DynCorp's proposed administration manager believed that DOE's response to vendor question 50, which allowed the contract manager position to have concurrent experience, also applied to the position of administration manager. Transcript at 296-97. Initial Proposals: October 23, 1993 CDSI's Initial Technical Proposal In response to criterion 1 of the technical evaluation criteria, corporate experience in performing contracts of similar contract. Transcript at 24; Protest File, Exhibit 23. CDSI interpreted the solicitation to require under criterion 1 that it describe its experience performing at least three but not more than five contracts similar in size and scope. Transcript at 21-22. The has a value of over over a five-year period. Transcript at 351; Protest File, Exhibit 80. The has a value of over a five-year period. Protest File, Exhibit 80. On the CDSI indicated that IRM functions being requested in the RFP were currently being performed. Protest File, Exhibit 58 at 48-49; Transcript at 66. With regard to the contract and the contract, CDSI endeavored to describe all functions of the twenty-four required functions which were being performed, and noted that it was performing of the twenty-four functions for and of the twenty-four functions for . Protest File, Exhibit 58 at 68; Transcript at 67. However, with respect to CDSI to describe all of the IRM functions being performed, those functions and the proposed new contract; CDSI wanted contract to highlight those functions. Transcript at 67-68, 70-71. DynCorp's Initial Technical Proposal In its initial and revised technical proposals, DynCorp submitted four contract references: (1) (2) (3) (4) Protest File, Exhibits 15, 51, 70. For its subcontractor, DynCorp proposed contract. Id. DynCorp's contract has a value of over a five-year period. Protest File, Exhibit 51 at 000054. DynCorp represented that this contract involved performance of of the twenty-four IRM functions in Section C of the RFP. Id. The contract itself, and a representative sampling of the task orders issued to DynCorp by which were produced by DynCorp during discovery, reference non-IRM services such as establishing "logistics procedures" for "proper purchasing/leasing, receiving and accounting and distribution of all supplies . . ." and putting on "conference workshops." Protester's Supplemental Protest File, Exhibit 7 at 00310-12. DynCorp's subcontract is a subcontract with which has a prime contract with Protest File, Exhibit 51 at 000060; Intervenor's Hearing Exhibit 29. DynCorp stated the value of the subcontract to be Protest File, Exhibit 51 at 000060. of the value of the DynCorp subcontract represents IRM-related services. Intervenor's Hearing Exhibit 29.4 DynCorp submitted a reference contract for its subcontractor, and cited experience in demonstrating experience for several functions under criterion 2, such as Automated Office Support Services and SACNET Support. Protest File, Exhibit 51 at 000070. was proposed for DPLH, out of the total of some of the total. Protest File, Exhibit 51 at 000191. ____________________ 4 The parties stipulated that DynCorp's division manager for the division which performed the subcontract, would have testified as follows if he had appeared at the hearing: DynCorp provides range engineering, data acquisition and reduction (REDAR) services, as well as marine automotive and aviation transportation services [non- ADPE], under the Subcontract. The total DynCorp personnel on the Subcontract is approximately people at present, and the number of full-time personnel assigned to REDAR functions and those assigned to transportation functions is accurately stated on the attached Monthly Status Report. . . . . The percentage of the total value of the Subcontract represented by the REDAR services and transportation services, respectively, is approximately proportional to the percentage of full-time personnel assigned to those respective services. Intervenor's Hearing Exhibit 29. The referenced monthly status report indicated that for October 1993, out of employees were performing ADPE services, and as of March 31, 1994, out of employees were performing ADPE services. DynCorp also cited the experience of its subcontractor under criterion 2. Protest File, Exhibit 51 at 000067, 000091. DynCorp stated in its proposal that both and were proposed for 10% or more of the total DPLH. Id. at 000066. In its initial proposal, DynCorp's proposed administration manager's resume did not specifically address the two-year requirement for recent experience in the administration of a contract of similar size and scope. Protest File, Exhibit 15 at 000418-22. CDSI's Initial Cost Proposal In its initial proposal, CDSI offered for each solicitation labor category other than key personnel. These were developed in accordance with CDSI's estimating policy. CDSI's vice president who developed its labor rates examined the RFP requirements and RFP labor categories, without any consideration of Transript at 653, 662-63, 669-71, 825-26; Protest File, Exhibit 23. For the category senior systems analyst, the representative sample consisted of Protest File, Exhibit 23 at 000035. The attachment to Exhibit B sets forth the RFP labor category, the names of the individuals who were selected as representative of the RFP skills, the current hourly rate for these individuals, and the composite "bid rate." Protest File, Exhibit 23 at 000035-39. In the notes to the cost proposal, CDSI stated: "CDSI's Energy Systems Division direct labor rates were derived using the Id. at 000007. CDSI derived the hourly rate for each individual by dividing the annual salary of representative employees who were currently assigned to the existing DOE contract by hours ( hours per week times 52 weeks). Protest File, Exhibit 23 at 000040. In developing the CDSI composite rates for the initial cost proposal, CDSI's vice president Transcript at 674-81, 749; Protester's Supplemental Protest File, Exhibit 72. On average, CDSI's rate for its composite rates in its initial offer was close to the Transcript at 681; Protest File, Exhibit 23; Protester's Supplemental Protest File, Exhibit 72. As part of its initial cost proposal, CDSI completed Exhibit E of the RFP exhibits. On Exhibit E, CDSI listed the RFP labor categories in one column, and for the "Offeror's Corresponding Labor Category," the possible CDSI corporate titles that an individual might have who possessed the skills to perform the DOE RFP labor category. Transcript at 673. CDSI did not use the information in Exhibit E in development of its direct labor rates. Protest File, Exhibit 23 at 000052; Transcript at 458-60, 673-74. Protest File, Exhibit 23 at 000185. Rather, its proposal states that Id. CDSI's total estimated cost in its initial proposal was Protest File, Exhibit 23 at 000023. CDSI's CDSI's offer of composite labor rates Transcript at 446-47. Id. at 440-46, 448-50, 649; Protester's Hearing Exhibit 9. Transcript at 453, 656. Id. at 487. CDSI's Cost Estimating Manual states at Section 4.2.2.1: Mapping Employees to Labor Categories Protester's Hearing Exhibit 9, 1994 Manual at 7.5 DCAA's January 1993 Audit of CDSI's Proposal DOE requested DCAA to audit all the offerors' cost proposals. In its January 11, 1993, Audit Report for CDSI, DCAA concluded: b. The proposed contract is a follow-on to contract DE-AC01-88MA33222. We compared, for six high- dollar labor categories, the current rates in the existing contract with the proposed rates. Protest File, Exhibit 28 at 000026. ____________________ 5 This provision does not differ substantially from the earlier versions. Protester's Hearing Exhibit 9, 1990 Manual at 11, 1993 Manual at 4-2. DynCorp's Initial Cost Proposal DynCorp proposed labor category rates in its initial proposal. Protest File, Exhibit 16. DynCorp developed its non- key personnel labor rates Id. at 000084. DynCorp stated that Protest File, Exhibit 16 at 000004. DynCorp continued: Id. The Evaluation Process The Rating Plan The SEB's internal rating plan provided that proposals would be evaluated by the SEB in accordance with that plan and the "Acquisition Regulations Handbook, Source Evaluation Board DOE/MA-0154." Protest File, Exhibit 30 at 22. The rating plan assigned the following maximum point scores to each of the four technical evaluation criteria: Corporate experience points Demonstrated ability and directly related experience points Contract management plan points Key personnel points Id. at 000024. The rating plan provided that cost proposals would not be point scored. Id. at 000055. The rating plan included a "contractor technical reference check" form to be used by the SEB in questioning references regarding technical criteria 1 and 2. Protest File, Exhibit 30 at 000058-61. Under the rating plan, which was not disclosed to offerors, an offeror could only receive a score of 0, 2, 5, 8, or 10. Transcript at 100. For example, if an individual evaluator had wanted to assign a score of 6.5, that was impossible. Id. The rating scale defined the point scores as follows: 10 - This proposal response is distinguished by its excellence in meeting the RFP's requirements as measured against the evaluation criteria. Such a response would normally evidence significant strength(s) and no weakness(es). 8 - This proposal response evidences a good understanding in meeting the RFP's requirements as measured against the evaluation criteria. Such a response would normally evidence significant strengths(s) [sic], and few, if any, weakness(es). 5 - This proposal response appears capable of meeting the RFP's requirements as measured against the evaluation criteria. Such a response would normally evidence no significant strength(s) or weakness(es). 2 - This proposal response omits major details and/or evidences a lack of understanding in meeting the RFP requirements, as measured against the evaluation criteria. Such a response would normally have few, if any, strength(s), but would evidence significant weakness(es). 0 - This proposal response can not [sic] be expected to meet the RFP's requirements as measured against the evaluation criteria. Such a response would normally have no strengths, but would evidence significant weakness(es). Protest File, Exhibit 30 at 000054. The rating plan contained the following provision regarding retention of documents by the SEB: The official SEB file will retain those documents containing the information or backup material upon which the Board's evaluation is based, (i.e., the rating plan and its attachments, and cost evaluation report). The Board member's individual worksheets and notes, if any, need not be retained after the Board has achieved consensus on all the evaluation criteria. If an individual evaluator desires any particular worksheets or notes be retained in the official SEB file, a memorandum from the individual evaluator should be attached to the document(s) stating the document(s) are to be retained. This memorandum and attached document(s) are to be placed in the official SEB file. Protest File, Exhibit 30 at 000052. DOE's Acquisition Regulations Handbook which governed this procurement included a section entitled "Transfer/Disposal of Files" which provided: During the course of its activities the [SEB] will accumulate certain data or documentation. Such documentation, or back-up material, normally will include: summaries of board meetings, the solicitation, proposals, working papers such as rating or scoring sheets and checklists, and committee reports. These papers constitute the basis for the board report and will be preserved. After the Board has completed its activities, the official SEB files will be purged of excess copies of material, including SEB reports, and the official SEB File will be transferred to the procurement office that will be responsible for negotiating the contract. A complete copy of each proposal received will be retained in the official files. . . . Protest File, Exhibit 95 at 000027. The SEB's Process There were six voting members and one nonvoting member of the SEB (DOE counsel). Transcript at 94-95. offers were submitted initially. Id. at 96. When the board first received proposals, each of the board members individually read the proposals and noted tentative strengths and weaknesses. Id. at 99. Proposals were maintained and reviewed in the board room; they could not be removed. Protest File, Exhibit 30 at 000022. The Individual Evaluations The individual members of the board were not provided with scoring sheets, but made their own informal notes as they individually reviewed proposals. Transcript at 100. There is no record of the individual evaluations because the individual notes of all members of the SEB were destroyed. Transcript at 100, 193-94. One member of the SEB testified as follows regarding the destruction of the notes: I don't recall why we -- we just felt we had -- once we had our board report, that represented our findings, and there was no reason to keep working papers and things like that which no longer beared on our consensus. I mean, I might have notes that might appear as a question; in other instances, a comment. And it may or may not have had anything to do with the evaluation. That was something we talked about. So once -- we didn't have any minority positions in the board so there was not need. There were many revisions. Id. at 205; see also id. at 100, 193, 1241-42, 1289-91. The notes were methodically destroyed at two different time frames during the evaluation -- just prior to the receipt of BAFOs and just prior to award. Id. at 1367-68. The SEB worked in a very small room and had a shredder, which was used to destroy the notes. Id. at 1367-68. The contracting officer testified that there were "official purges" when the SEB requested BAFOs on December 13, 1993, and when the SEB did its final report on February 4, 1994. Id. at 1705. When the board members finished reviewing the proposals, they met as a group to reach consensus on strengths and weaknesses of each proposal. Transcript at 99, 1363-65. They then assessed whether the strengths and weaknesses were "significant." Id. After reaching consensus on the significance of the strengths and weaknesses, the board reached another consensus and scored each technical criterion of each proposal. Id. There was never a minority position. Protester's Hearing Exhibit 8 at 32.6 The SEB determined which contracts were to be "strengths" and "weaknesses" without any agreed-upon, collective definition of those terms. Transcript at 164-69, 209-14, 242-43. One SEB member adopted the rating scale to determine what a "strength" was. Id. at 187, 193. He testified: Q What instruction were you given in deciding what was a strength and what was a weakness? A Well, that's what we decided to do. We were following the rating plan, and we were to read the -- we agreed as a group. If I said I was told, I'm sorry. I don't believe that we were told. No one told us. We as a group decided that's how we were going to approach the evaluation. Q The way you agreed to approach it was each of you look at the contracts and decide for yourself as a strength or a weakness, right? A Note those sorts of things using the model contract. Q Right. Did you discuss what the definition of 'strength' would be versus 'weakness'? ____________________ 6 By stipulation, one SEB member's deposition was entered into the record as Protester's Hearing Exhibit 8, in lieu of his testifying at the hearing. A I don't recall. Id. at 191-92. However, the chairman of the SEB considered the rating scale to be inapplicable to define what a strength might be. Id. at 217-18. One member of the SEB contradicted his own testimony regarding strengths and weaknesses. Initially, he testified that there were three categories, strength, weakness, and "adequate." Transcript at 180.7 Several minutes later this witness testified as follows: Q When you were deciding whether the contracts were strengths or weaknesses or adequate, did you have a definition of adequate in your mind at the time? A I did not do what you just described. Q What did you do . . . ? A As a board member identified strengths and weaknesses. Q Did you identify which contracts were adequates? A No. Q So it had to go into either strength or weakness; is that your testimony? A That is correct. Id. at 189. The SEB also did not articulate any objective guideline for determining whether the strengths or weaknesses were significant. Id. at 34-35, 182-83, 196-201. "Significance" ____________________ 7 This specific testimony was: Q So if it's similar in scope and size, it may not be a strength; is that your testimony? A I'm saying that it might be adequate. It depends. ____________________ It's relative. Transcript at 180 (emphasis added); see id. at 155. ("It's ___ ___ possible that an offeror gave us a contract where we determined that it was adequate, given that you will find no place in our evaluation describing adequate in terms of strengths or weaknesses. We addressed on these sheets strengths and weaknesses not adequates.") was assigned for the purpose of discriminating among the proposals and had an impact on scoring. Id. at 198. Another SEB member would adjust points for "significance" of strengths and weaknesses, but he was not sure by how much. He testified: Q Now, if . . . you make the contract, which is the first strength, . . . significant, what happens to the score? A Based upon what we've agreed, that would make the score go higher. . . . . A [I]t would set it as a higher number. What that number is, I don't know. It might be -- it might have been a 7.5. Or it might have been a 7. I'd say, well, I think it should be a 7.5 now, in my own mind. Protester's Hearing Exhibit 8 at 38-39. One member of the SEB testified as follows regarding "significance" of strengths and weaknesses: Q . . . . What was your purpose in assigning the significance to the strengths under criterion 1? A That it was far more than just meeting the requirements would be an example. Transcript at 196. He also stated: So it's a tough call when you say two strengths that are not significant. I'm not sure where I would take that, and I would work with the board to work that out. Transcript at 199-200. Initial Evaluations: Scoring The total point scores and evaluated costs at initial evaluation for offerors ultimately included in the competitive range were: MAXIMUM TECHNICAL POINTS OFFERORS CDSI CSC8 DynCorp 1. Corporate Experience 2. Demonstrated Ability and Directly Related Experience 3. Contract Management 4. Key Personnel Total Points PROPOSED COST CDSI CSC DynCorp Protest File, Exhibit 30 at 000024. Evaluation of CDSI's Initial Proposal For criterion 1, DOE concluded that CDSI's corporate experience in performing contracts of similar scope and size was Protest File, Exhibit 30 at 000029. Initial evaluations did not encompass reference checks; this was done after the competitive range determination. Transcript at 101, 1297-98. DOE recognized that CDSI's performance on the contract was virtually the same scope and size as the proposed contract. However, DOE further concluded that CDSI DOE stated: Protest File, Exhibit 30 at 000029. CDSI received a rating of on criterion 2 because of its experience on the contract. Id. With regard to criterion 3, DOE found the adequacy and appropriateness of CDSI's plan for contract management to be Id. at 000029. DOE concluded that ____________________ 8 Computer Sciences Corporation. CDSI had Id. at 000029-30. With regard to criterion 4, DOE concluded that CDSI's key personnel were but that its proposed systems development and maintenance manager's resume Protest File, Exhibit 30 at 000030. Citing the DCAA's determination, the SEB concluded that CDSI's cost proposal which DCAA had noted. Specifically, DOE characterized the DCAA findings as follows: [DCAA] has determined that CDSI's cost proposal contains The proposal contains These consist of the presence of [9] in labor category rate calculations, indirect rates . . . and failure to provide cost and pricing data for the Id. at 000030. With regard to direct labor the SEB questioned in direct labor costs. Protest File, Exhibit 30 at 000030. The SEB concluded that CDSI Finally, DOE questioned CDSI's estimating method for determining the proposed labor rates. DOE stated: If CDSI were to utilize their it would result in a It also appears that CDSI has failed to include all employees currently working in a specific category to determine the category average rate therefore the rate may not be representative of the actual average rate. Protest File, Exhibit 30 at 000030. In evaluating the initial cost proposals, the contracting officer basically looked at areas of inconsistency. Transcript at 1595-96. He explained that there were three charts that he used, Exhibit B, Exhibit B-1, and Exhibit E. Exhibit B listed the projected labor rates for proposed individuals and categories. For example, ____________________ 9 Exempt employees means employees who are exempt from the Fair Labor Standards Act. The attachment to Exhibit B, i.e., Exhibit B-1, listed the Protest File, Exhibit 23 at 000035. Exhibit E, entitled "Reconciliation of Labor Categories," tracking the title in the RFP, indicated for Protest File, Exhibit 23 at 000052. In evaluating CDSI's initial offer, the contracting officer was concerned with the discrepancy between the CDSI labor categories indicated in its Exhibit B and those indicated in its Exhibit E. He testified: So, they've got one on Exhibit B that's not in Exhibit E and they have an additional three on Exhibit E that's not represented on Exhibit B. So we have a disconnect. We have a question. There's a disconnect here [as] to what they are really offering in relation to their labor categories they're going to propose to meet our labor categories. Now that disconnect becomes important because now we go to the question as to -- ok, who now makes up the composite. Is it the individuals from labor categories identified on Exhibit B or is it individuals from the diverse labor categories identified on Exhibit E. So we now have several issues which we need to resolve with the offeror in relation to direct labor rates. Transcript at 1600. Evaluation of DynCorp's Initial Proposal Under criterion 1, DynCorp's corporate experience in performing contracts of similar scope and size was determined to be Three of its references -- -- were considered strengths. The contract was stated to have out of twenty-four functional areas addressed as to scope, and was rated a strength, but not a "significant" strength because the SEB could not determine what tasks DynCorp was actually performing on that contract. Protest File, Exhibit 30 at 000034; Transcript at 1435-41. DynCorp's demonstrated ability and directly related experience for performing the functions of the State of Work (SOW) plan for management were also considered Protest File, Exhibit 30 at 000035. Under criterion 4, DynCorp's proposed key personnel were determined to be its contract manager's experience was and its administration manager Id. at 000036. The DCAA found no deficiencies with DynCorp's cost proposal's estimating system and took no exceptions to DynCorp's direct labor rates. Protest File, Exhibit 30 at 000036. The May 12, 1993, Competitive Range Determination Because CDSI and DynCorp demonstrated previous corporate experience and the capability and experience necessary to perform the functions, as well as adequate management plans, the SEB recommended that they be included in the competitive range. Protest File, Exhibit 30 at 000041. Reference Checks After making the competitive range determination, the SEB performed reference checks of the contracts submitted under criterion 1. Transcript at 1297-98. The purpose of the corporate reference checks was to corroborate the contract references contained in each offeror's proposals. Id. at 104-105, 1298. The reference checks were not rated or point scored. Id. at 117, 1301. In general, two members of the SEB were supposed to perform reference checks jointly "to insure the accuracy of the contract reference checks." Transcript at 105; see also id. at 239. In doing reference checks, the members of the SEB preferred contacting the contracting officer's technical representative (COTR) because they believed that the COTR could provide better technical information concerning technical aspects of the contract. Id. at 105-07. In performing reference checks, the SEB members were supposed to use the same form to insure they were asking the same questions. Id. at 107-08; Protest File, Exhibit 30 at 000058-61. The "contractor technical reference check" contained a list of various tasks required under the RFP to be verified in the reference contracts. Protest File, Exhibit 30 at 000058. In addition, the form solicited the principal types of personnel utilized and comments on various aspects of performance. Id. at 000058-61. Typically, the SEB spent less than an hour for each contract reference. Transcript at 108. The SEB members only talked to a reference once. Id. CDSI's Reference Checks The chairman of the SEB performed CDSI's reference checks. Transcript at 219. He did this by himself because the member of the SEB who was supposed to assist him, the contracting officer, was starting his initial cost evaluations and may have been tied up on an environmental information administration proposal at the time. "His time was rather valuable," so the chairman decided to take the burden of the interview on himself. Id. at 239-40. The chairman of the SEB attempted to contact the technical officer on the contracts, but he was on leave. Transcript at 220. The contracting officer for both of those contracts agreed to assist him, but she was in a hurry. Id. The chairman of the SEB and the contracting officer on the contracts agreed to do both references "in tandem to save everybody a lot of time." Id. At hearing the chairman of the SEB initially testified to one methodology of performing CDSI's reference check, but then completely changed his testimony. First, the SEB chairman testified that in doing the reference check he used the form which everyone else used, which listed the twenty-four RFP functions and that the form was on his personal computer (PC). Id. at 221-22. He testified that he tailored the form to fit the references in the proposal and deleted from the form any of the twenty-four functions that a contract reference said it was not performing. Id. at 223. For some of the twenty-four functional areas, the chairman of the SEB went beyond simply asking the reference whether they were being performed; he asked what number of people were working on that function to test how significant that functional area was in the reference contract. Id. at 226. Finally, the chairman of the SEB testified that he discussed what was on the reference form with the contracting officer. Id. at 232-33. Then several days later at the hearing this same witness testified as follows: A Okay. When [the contracting officer on the said she was pressed for time and could I possibly do anything to speed the process up -- . . . I asked her to give me a few seconds and I was trying to tailor these forms. . . . Okay. By "tailor" what I meant to do was delete those functions that were not in CDSI's proposal because there was no need to discuss functions not in the proposal with her to corroborate them. Q So you started deleting the ones that were not checked off by CDSI at [CDSI's proposal]? A I went to that page in [CDSI's proposal]. I started trying to tailor the forms. It became too confusing an issue so I just used that page out of [CDSI's proposal]. Q So is it your testimony that you started deleting and you just never completed it? . . . A Well, I started deleting, you know, and I even got confused as well, maybe I didn't delete the right one. I may have added one or two. It became a total confusing mess. You know, so I just didn't want to use them. I wanted to make sure I was being accurate and corroborating the proper areas, so I used the page out of the [CDSI proposal]. Transcript at 1322-23. The reference form which was described as "a total confusing mess" was included as part of the SEB official file for all of the other members to utilize. Id. at 1325. This witness said he had intended to correct the form, but to his embarrassment he "just forgot it" and never did it." Transcript at 1324. In performing the and reference checks, the chairman of the SEB corroborated more functions being performed than were indicated in the proposal. On the contract, he corroborated functions, whereas CDSI had only listed Transcript at 1312-14. On the reference check, the SEB chairman confirmed functions, even though CDSI only listed in its proposal. Id. at 1314; Protest File, Exhibit 22 at 45. DynCorp's Reference Checks In the reference check for DynCorp's contract, the reference form indicated that DynCorp had no experience in of the twenty-four functional areas. Transcript at 114. The form further indicated that for of the SOW functions DynCorp's experience was limited. Id. at 115. In performing the reference check on DynCorp's subcontractor, the SEB was unable to verify any of the work that was proposed to do. Transcript at 127-28. One member of the SEB testified: What we got by talking with the contract reference is that they provided in total a turnkey type function, and obviously they were performing the work because where you're doing a billing function for, you know, it's a complete function. They're obviously doing the work. But we could not verify it, and we tried to be factual in what we reported on this sheet. Id. at 127-28. DOE's June 11, 1993, Clarification Questions To CDSI In the technical/business management proposal area, DOE requested that, inter alia, CDSI clarify the following: its its its and its Protest File, Exhibit 34 at 000008-9. DOE also asked CDSI to clarify its directly related experience in Id. at 000008. In the clarification questions regarding CDSI's proposed cost, DOE stated, in pertinent part: COST: The DOE concurs with DCAA's questions regarding CDSI's estimating method for determining the proposed labor rates. Please submit a schedule reflecting CDSI's labor classifications, individuals' names, current salaries, and the labor category's average rate for each RFP labor category. All individuals must be currently working within the labor category for which they are included. Protest File, Exhibit 34 at 000009. Following receipt of the contracting officer's June 11, 1993, letter, CDSI's vice president contacted the contracting officer to obtain a clearer understanding of the points for which DOE needed clarification. In these telephonic discussions, CDSI explained that CDSI corporate labor classifications or titles had nothing to do with CDSI's creation of its direct labor rates. CDSI also explained its understanding that what DCAA had requested was a comparison of the to the RFP labor categories. Transcript at 699-702. CDSI representatives subsequently met with the DCAA supervisory auditor. DCAA confirmed that it expected a comparison of the direct labor rates to the RFP labor categories because CDSI would be performing the same functions if it were awarded the follow-on contract pursuant to the RFP. CDSI understood DCAA to be requesting and communicated Transcript at 701-03; Protest File, Exhibit 39 at 000004. By letter dated June 25, 1993, CDSI responded to DOE's request for clarifications and discussions. Protest File, Exhibit 38. In its response CDSI addressed all of the questions raised in the June 11 letter. Id.; Transcript at 80-83. In response to DOE's question regarding CDSI detailed its experience in Protest File, Exhibit 38 at 000003-06. In response to DOE's comment that CDSI's proposed systems development and maintenance manager did not appear to meet the qualifications in the position description in that he did not demonstrate at CDSI substituted a different individual and resume. Id. 38 at 000026-33. This individual's resume indicated that he had been employed by CDSI since and had been the program manager on the contract, with a staff of since between and he was the Project Director/Deputy Project Director on a contract with the supervising between and CDSI and subcontractor staff. Id. In response to DOE's concern regarding the DCAA conclusions, CDSI stated that it of bidding and delivering lower hourly rates. CDSI stated: Protest File, Exhibit 39 at 000002. Id. In response to DOE's request that CDSI submit a schedule reflecting its labor classifications, CDSI submitted Attachment 2 to the June 25, 1993, letter in which it to produce labor rates on the new contract. Protest File, Exhibit 39 at 000016-27. Protest File, Exhibit 39 at 000004, 000016-27. Id. at 00004. the rate increased to Id. at 000016-27; Transcript at 706-10; Protester's Supplemental Protest File, Exhibit 72. CDSI contends that it treated this submission as an exercise to show the effect of literal compliance with DCAA's directive that Protester's Posthearing Brief at 58-59. To DynCorp By letter dated June 11, 1993, DOE advised DynCorp that its proposal was within the competitive range and enclosed a list of written questions and statements for which written discussions/clarifications were requested. DOE addressed the issue regarding the administration manager as follows: The proposed Administration Manager does not appear to meet the qualifications contained in the generic position description in that he does not meet the requirement of administering a contract of similar size and scope. Protest File, Exhibit 34 at 000015. On June 25, 1993, in response to DOE's question concerning the administration manager, DynCorp proposed a different administration manager. Protest File, Exhibit 36 at 000048. This individual had ten years of contract administration experience for He had been responsible for ADP contract management and administration for multiple contracts as the Id. According to his qualifications statement, this administration manager between and "had administrative responsibility for more than personnel supporting multiple contracts generating annual revenues exceeding " Id. at 000051. His efforts at DynCorp since 1992 included proposal efforts on the ITSO procurement. Id. at 000050. DOE's August 6, 1993, Letters By letter dated August 6, 1993, the contracting officer requested each offeror in the competitive range to provide a copy of the company's position descriptions, to include the experience and educational requirements for the company's corresponding labor categories identified in Exhibit E of each offeror's proposal. Protest File, Exhibit 41; Transcript at 1621-22. By letter dated August 18, 1993, CDSI supplied the requested corporate position descriptions and explained that its position descriptions are used only for internal purposes and not for purposes of proposal pricing. Protest File, Exhibits 42, 44; Transcript at 486-87, 1626-27. The contracting officer interpreted CDSI's cost proposal to be offering a composite rate and to draw individual employees from labor categories listed in Exhibit E. Transcript at 1633. The contracting officer believed that in order to propose acceptable composite rates, a vendor had to use individuals in deriving those composite rates who met each requirement of the RFP for a given labor category. Id. at 1639-40. Therefore, the contracting officer looked at all of CDSI's position descriptions in a given CDSI labor category covering all the individuals proposed to make up a composite rate to make sure that each position description met all of the requirements for the RFP labor category. Id. at 1640. In reviewing CDSI's August 18, 1993, submission of position descriptions offered for the corresponding RFP labor categories, the contracting officer was concerned that in the senior management analyst position none of the met the minimum requirements of the RFP's position description. Transcript at 1643-44. The contracting officer's concern went to "the realism of the composite rate, as to how it is developed, and whether it is really representative of the individuals and labor categories that will be used, to fill our RFP labor categories." Id. Oral Discussions On August 23, 1993, DOE held oral discussions with CDSI. Intervenor's Protest File Supplement, Exhibit 102. Most of the issues discussed with CDSI during oral discussions concerned Protest File, Exhibit 44; Transcript at 1645. The contracting officer prepared a prenegotiation memorandum listing the issues to be discussed with CDSI during oral discussions. Protest File, Exhibit 44 at 9-11. He read from this during discussions and made the following points: . The CDSI labor categories identified in Exhibit E of your proposal do not appear to be consistent with the RFP labor categories. i. Junior levels proposed for senior levels and vice versa. ii. CDSI appears to have proposed corresponding labor categories which are inconsistent with certain RFP labor categories such as . . . . CDSI appears to have used selective individuals to establish the average category rate instead of all individuals performing within the corporate category. (Individuals who do not possess the required educational and/or educational requirements should not be used to determine the weighted or composite labor rate.) The contracting officer further stated: . The attachment to Exhibit B shall include the RFP labor category, and the corporate corresponding labor category title for which each individual is employed under CDSI. . The hours must be included into the DPLH in addition to the minimum DPLH specified in Exhibit-I. Id. at 000009-11. During oral discussions, CDSI stated that it did not use rates and that, by requiring CDSI to submit its internal position descriptions, the contracting officer was essentially Transcript at 1646. Nonetheless, the contracting officer insisted that CDSI use, for each labor category, individuals who met every requirement of the RFP labor category. Id. at 1647. The contracting officer also raised the concern that it would make the rate nonrepresentative of the actual rate being proposed if CDSI proposed senior level personnel for junior level labor categories. Id. at 1652. As a result of oral discussions, CDSI understood that the following instructions had to be followed in development of its direct labor rates for the next cost proposal submission: We were instructed that in preparing the composite rates that we had to do it within the following boundaries. You would not necessarily use the people in DOE; that you would use all CDSI employees in the Washington metropolitan area; that you could not use a junior CDSI titled person in a senior DOE titled position and vice versa; that you had to have somebody in a senior DOE category -- senior CDSI title for a senior DOE category. Everybody had to meet the minimum DOE qualifications, but they couldn't be overqualified. Transcript at 714-15. With respect to the instructions concerning use of senior titled CDSI personnel in the rates for senior level RFP labor categories, CDSI understood from an example given at oral discussions that the CDSI corporate labor category minimum experience requirement did not have to be an exact match to the DOE labor category experience requirement in order for an individual's rate to be used in the composite rate for the senior RFP position, provided that the CDSI individual was drawn from a CDSI title with a "senior" designation and the individual personally met the experience and education requirements of the RFP position description. Transcript at 850; Intervenor's Protest File, Exhibit 102. CDSI understood that it would not have been acceptable for CDSI to use an individual in its rate for the RFP senior systems programmer who was drawn from a CDSI position which did not have a "senior" title, such as the Transcript at 2087-2107. CDSI believed this to be the case even if an individual CDSI whose rate was being used actually had the six years of experience. Id. The contracting officer denies that he was refusing to allow offerors to propose individuals simply on the basis of title. Id. at 1752-57. Rather, he would agree that persons could be used for the senior systems programmer so long as they had the actual experience even if they did not have the title "senior." Id. at 1795-1803. During oral discussions, DOE did not advise CDSI that the agency considered that CDSI's were not similar in size and scope to the procurement in question. Transcript at 1452. The contracting officer explained his rationale for this decision: Q Okay. And why did you not do that? A Well, Criterion 1 for CDSI was not identified as a deficient criterion. They in fact were capable of meeting the RFP requirements, based on evaluation of their corporate experience. The weakness that was associated with that did not constitute a deficiency, and for us to hold discussions, in regards to their corporate experience, as described on the reference contracts, would constitute leveling, because the only thing we would have been achieving was, allowing CDSI to increase their score, of which they have already met the capability of performing the requirements. Q Okay. This, CDSI was a on Criterion 1? A They were a on Criterion 1. So there were no uncertainties, in the board's minds, in regards to the evaluation of the proposals. We truly had a clear understanding of what the contract was, and it was represented. We clearly had an understanding of what contract was, as represented. We obviously had a clear understanding of the -- contract, and the first contract, and I think the other one was the contract. There was no uncertainty in relation to what those references constitute. So, by not having any uncertainty, in regards to the references, by not being deficient, for the criterion, overall, the only thing the discussions would have served, at that point, was to technically level the offeror, in Criterion 1. Id. at 1453-54. During discussions DOE did not advise CDSI or DynCorp that they each had proposed key personnel who did not meet the experience requirements of the RFP -- in CDSI's case the systems development and maintenance manager, and, in DynCorp's case, the administration manager. Protest File, Exhibits 44, 46. During discussions, the contracting officer stated that CDSI was argumentative. Respondent's Answer to Protester's Interrogatory No. 122; Protester's Supplemental Protest File, Exhibit 73. The contracting officer did not express similar opinions to the effect that other offerors were argumentative. Respondent's Answer to Protester's Interrogatory No. 124; Protester's Supplemental Protest File, Exhibit 73. The August 27, 1993, Letters Following Oral Discussions On August 27, 1993, the contracting officer sent CDSI a letter confirming his understanding of the oral negotiations. With respect to direct labor rates, the contracting officer stated, inter alia: The Attachment to Exhibit B [which shows the labor rate development] shall include a new column to reflect the CDSI corresponding labor category for which the individuals are performing; Junior and Intermediate Level personnel shall not be used in developing the composite rates for Senior Levels [sic] positions, unless it is clearly demonstrated that the corresponding CDSI labor category truly meets all the requirements; Senior Level personnel shall not be used in developing the composite rates for Intermediate/Junior Level positions, unless it is clearly demonstrated that the corresponding CDSI labor category truly meets all the requirements; CDSI will ensure that individuals used to develop the composite rates are performing in corporate labor categories which have responsibilities and duties relevant to that of the RFP labor categories, and which meet the minimum educational and experience requirements. Protest File, Exhibit 46 at 000005-6. The contracting officer's August 27, 1993, letter also addressed uncompensated overtime: It is the DOE's position that the additional hours will be added to the suggested DPLH specified in Exhibit-I to establish the probable cost. DOE advised CDSI that the additional hours per year for those pertinent labor categories or DPLH per man year will be used to establish the probable cost instead of the 1,860 DPLH per man year to eliminate any possibility of a buy-in. This position is further substantiated by the comments of CDSI that while they do not have Protest File, Exhibit 46 at 000006. In its August 27, 1993, letter to DynCorp, DOE stated, in pertinent part: DynCorp needs to address the requirements of L.049.2.(b)3.b regarding compensation to professional employees to assist the DOE to determine whether Dyn[C]orp has a sound management approach. Dyn[C]orp will review its methodology for developing its direct labor rates and will make a business decision on its appropriateness. It is the DOE's position that the percentile and percentile hourly rates and annual salaries of the stated wage and salary surveys in the proposal are not realistic or reasonable rates. The DOE will use the 50th percentile to establish the probable cost in those instances where any rates are proposed below the 50th percentile. Dyn[C]orp shall submit the pertinent portions of the surveys used to establish its proposed rates. In the event Dyn[C]orp fails to submit the pertinent data (surveys or specific employee salaries) the DOE will use the Wyatt Data Services survey and other data deemed necessary to establish the probable cost. It was discussed that Dyn[C]orp should use its actual labor rates for individuals working in the same labor categories and within this regional area to assist in the development of a composite rate instead of relying solely on the survey data when historical data does exist. Protest File, Exhibit 46 at 000000-01. Revised Proposals: September 16, 1993 On September 16, 1993, CDSI submitted revised cost and technical proposals. In the cost proposal CDSI attempted to follow the instructions of the contracting officer regarding development of direct labor rates given during oral discussions and in his August 27, 1993, letter. CDSI's total estimated cost for this proposal was Transcript at 856-57; Protest File, Exhibit 59 at 000027. In developing the direct labor rates for this revised proposal, CDSI drew upon personnel from the Washington, D.C. metropolitan area. Transcript at 734-35. understood by CDSI as a result of oral discussions and the August 27, 1993, letter. Id. at 736-39. CDSI's revised proposal listed the Protest File, Exhibit 59 at 000047-55. CDSI's Exhibit E listed only those corresponding CDSI labor categories that matched the CDSI corporate titles for each RFP labor category contained on the attachment to Exhibit B. Id. at 000075. but was attempting to comply with DOE's instructions. Transcript at 734-37; Protest File, Exhibit 59 at 000015. CDSI included in its proposal a note to Exhibit E: Protest File, Exhibit 59 at 000076. In CDSI's view, the effect of complying with DOE's instructions, particularly the senior to senior directive, was to exclude qualified individuals and drive up the rate for certain senior RFP categories. For example, CDSI's composite rate for the senior systems analyst increased from an hourly rate of in the initial proposal to in the revised proposal. Protest File, Exhibits 23 at 000035, 59 at 000049. In its revised proposal, CDSI diluted its rates as the contracting officer had instructed in his August 27, 1993, letter. Protest File, Exhibit 59 at 000017. Id. at 000017. The November 4, 1993, Audit of CDSI's Revised Proposal In a November 4, 1993, audit report, DCAA concluded: Our audit showed that CDSI CDSI's representative first told us that the reason they In our opinion, because the estimated direct labor costs for the follow-on contract may not be realistic. The labor rates of We obtained a listing of employees who worked on the previous contract, and we select Sr. System Analyst labor category (object code 0124) for review. This category has the largest proposed labor cost per contract year. Our audit showed only out of employees who worked for the previous contract are included in the rate computation. The proposed labor rate for this category is derived from the average rate of employees. This means that employees did not work under this labor category. Protest File, Exhibit 65 at 000008. The Evaluation of Revised Proposals: December 7, 1993 The SEB reviewed the revised proposals and issued an addendum to its SEB report dated December 7, 1993. Protest File, Exhibit 67. CDSI CDSI's scoring for criterion 1 remained a and its score for criterion 2 remained For each of criteria 3(b), 3(c), and 3(d), CDSI continued to receive a score of Protest File, Exhibit 67 at 000028-30. CDSI received a score of on its key personnel and the SEB noted as a weakness that its systems development and maintenance manager failed to meet the requirement of Id. at 000034. The SEB determined CDSI's costs "may not be realistic" due to the fact that CDSI had demonstrated "uncertainties" in staffing of the contract. Protest File, Exhibit 67 at 000010. The SEB report did not mention the contracting officer's instructions at oral discussions and in the August 27, 1993, letter regarding drawing personnel from certain corporate labor categories and using all personnel in the Washington, D.C. metropolitan area (as opposed to Energy Systems Division personnel only) in the direct labor rates. In its December 7, 1993, report, the SEB also conducted an "Analysis of CDSI's Corresponding Labor Categories for Professional Employees" pursuant to Section M.013 of the solicitation entitled "Evaluation of Compensation for Professional Employees." Protest File, Exhibit 67 at 000056-67. In this analysis, the SEB concluded CDSI failed to professional compensation. The SEB stated: "CDSI's policy of not using its established job classification system to correlate job titles and develop its salary structure Protest File, Exhibit 67 at 000056. The SEB compared CDSI's corporate labor categories to the labor categories contained in the Wyatt Data Services Wage Survey (WDSWS) dated March 1993. Based on this comparison, the SEB concluded that CDSI's rates for of twenty professional labor categories Id. at 000011. For example, the SEB determined that the RFP labor category senior systems analyst corresponded with a systems analyst level five in the Wyatt Data Services Wage Survey. The survey's average salary range is a minimum, middle, and maximum of $21.59, $27.16, and $32.74. Protest File, Exhibit 67 at 000056. The weighted average hourly rate is $27.45. Id. Because CDSI's composite rate of included hourly rates ranging from a low of to a high of and because of the actual salaries used have hourly rates less than the minimum average rate of the WDSWS, the SEB questioned the corresponding labor categories used to develop the rate. The SEB also questioned CDSI's composite rate for its Id. at 000066-67. In its analysis of professional compensation pursuant to Section M.013, DOE questioned CDSI's failure to correspond its corporate labor categories to of the RFP labor categories. Id. at 000011. In its December 7, 1993, report, the SEB stated that uncompensated overtime was not mandatory and would not be demanded by DOE. However, the SEB stated: Since all employees are required to be dedicated to this project (i.e., work a 5 day week, 8 hour day) and the DOE does not require uncompensated overtime, it is realistic and reasonable to assume that a professional employee's full annual salary will be billed to this contract based upon the dedicated concept. To allow a diluted rate to be proposed based upon uncompensated overtime, and not account for a professional employee's full salary, would allow an offeror to understate its proposal. Protest File, Exhibit 67 at 000015. DynCorp The chairman of the SEB testified that DynCorp's score from initial to revised proposal was increased to for criterion 1 based upon confirmation that DynCorp was actually performing the work it was stated to be doing under the contract. Transcript at 1317-18. Instead of being simply a strength, the contract was now a significant strength. Protest File, Exhibit 67 at 000046. With regard to DynCorp's reference, which was initially cited as a strength, the SEB, after clarifying what functional areas the subcontractor would perform rated that reference a weakness. Id. In evaluating the revised proposals, the SEB again concluded that DynCorp's proposed "Administration Manager does not appear to have administered a contract of similar size and scope within the last 5 years." Protest File, Exhibit 67 at 000055. Between initial proposals and the revised proposals, the SEB raised DynCorp's technical score on criterion 3(b), Quality Management Program, from a to a even though DynCorp's proposal had not changed substantively. In the initial evaluation, DOE had noted no strengths and no weaknesses for this criterion. In the revised evaluation, the SEB noted the following significant strength: The offeror's Protest File, Exhibit 67 at 000049. One member of the SEB explained that the change in DynCorp's scoring stemmed from a review of Transcript at 1342. Id. at 1463-64. As a result of its struggle to attempt to understand what entailed, the SEB reviewed all quality management programs a second time and attempted to prepare flow charts to aid in understanding their components. Id. One member of the SEB testified: Id. at 1342-43. With the SEB saw less in its program and dropped its rating Id. at 1343. The SEB was unable to do a flow chart for CDSI's TQM (total quality management) because "there were individual pieces that could be put in boxes on a piece of paper, but there were no interrelationships of the components discussed." Id. at 1343; see also, id. at 1466-69. DynCorp's score under criterion 2 remained at its score under criterion 4 was increased from to a Protest File, Exhibits 30 at 000110, 67 at 000055. The contracting officer testified that the reference went from a strength to a weakness because in doing the reference check the SEB ascertained that in fact relationship with was not a contract but an outsourcing of a billing process. Transcript at 1474. The references could not attest to anything other than the fact that they received their bills. Id. at 1476. However, this same contract became a strength under criterion 2 because the SEB found out more information about what was being done. Id. at 1477-78. The contracting officer testified that had three data centers where it does this outsourcing. Id. He continued: Their data centers are larger than ours . . . more encompassing than anything we have. And they in fact do have all these systems analysts, programmer, data entry, and everything else, and they actually are performing all these functions. They are developing the software, and maintaining the software for the maintenance or billing process to actually work. So, although cannot attest to it, we know they are doing it, and they demonstrate that they are doing it, but the weakness in criterion 1 is only associated with evaluating corporate experience in performing the contract. Id. at 1477-78. In terms of cost evaluation, the SEB report noted that the DCAA audit had disclosed no question of unsupported cost. However, the SEB did question DynCorp's direct labor rates proposed and the multiple surveys used. Protest File, Exhibit 67 at 000022. The December 13, 1993, Request for BAFOs In its request for a BAFO, DOE advised CDSI: The following comments and statements are provided to highlight areas of the revised cost proposal questioned by the Department of Energy (DOE) and/or Defense Contract Audit Agency (DCAA). There are no comments or statements associated with the technical proposal. . . . . 2. CDSI expended a large effort in discussing uncompensated overtime and the basis for establishing its direct labor rates. CDSI has explicitly stated However, CDSI made a business decision to propose a diluted rate, in its revised proposal, based upon uncompensated overtime, or a standard work week of hours. The DOE has no preference as to how CDSI proposes its direct labor rates, or whether it proposes uncompensated overtime or not. The only requirement by the DOE is that the offeror propose direct labor rates in accordance with its disclosure statement. To ensure clarity, and to assist the DOE in its review of CDSI's proposed direct labor rates and its methodology for accumulating and estimating direct labor costs, CDSI should submit a copy of its disclosure statement with its BAFO. CDSI is advised that should it propose a diluted rate based upon uncompensated hours, or a standard work week of hours instead of 40 hours, the additional hours will be included in the DOE's probable cost. 3. CDSI has taken three unacceptable deviations to the contract as follows: i) The proposal to invoice the DOE a is unacceptable. The terms and conditions regarding the invoicing for payment of the total award fee earned for the period evaluated remain unchanged. ii) The proposal that the fee be adjusted for any variance in The terms and conditions for reducing or negotiating any increase of the award fee remain unchanged. iii) The proposal to invoice is unacceptable. The terms and conditions for invoicing remain unchanged. 4. The DOE and DCAA question the direct labor rates proposed. See the audit report for additional data. CDSI was advised at oral discussions to use all of the employees in the Washington, DC regional area of the proposing entity, be that CDSI or a Division thereof. CDSI's initial proposal proposed category average rates which were not representative of the actual rate that would be experienced. The Energy Systems Division is the proposing entity, therefore, representative labor rates from that Division should be used, not labor rates from other Divisions. CDSI states in its revised proposal that qualified individuals will be drawn from other labor categories than those identified as corresponding labor categories reflected in Exhibit E. By not including those labor rates and corresponding labor categories in the proposal, the DOE considers CDSI's The SEB evaluated the offeror's corresponding labor categories and proposed rates for professional employees . . . . Of the 20 professional labor categories evaluated, the DOE questions CDSI's corresponding labor categories proposed for of the professional labor categories. of the 20 labor categories evaluated for professional labor reflected compensation levels which appear Protest File, Exhibit 69 at 000010-11. The SEB did not indicate to CDSI that it had used the Wyatt Data Survey to determine the reasonableness of professional compensation. Id. After receiving the BAFO request, CDSI asked the contracting officer to identify which corresponding labor categories were questionable. CDSI also asked the contracting officer to identify which of the twenty professional labor categories were "unrealistically low" and "unreasonable" and reflected salaries that were inadequate. The contracting officer refused to disclose this information to CDSI. Transcript at 532-33. CDSI's December 21, 1993, Response to the November 4 DCAA Audit Report By letter dated December 21, 1993, and stamped "DRAFT," CDSI replied to the DCAA's audit report. CDSI stamped its reply letter draft to informally discuss its concerns with DCAA. Transcript at 425-26. In response to DCAA's concern that CDSI had bid direct labor in a way contrary to its normal procedure, CDSI explained that it was directed to do so. Id. at 427. In its informal response, CDSI stated, in pertinent part: The whole issue of direct labor costing is a direct reflection of the government's instruction to us. Our initial bid contained representative labor costs from the current contract based on the RFP requirements. At subsequent oral discussions, we were instructed to only bid staff who meet the minimum qualifications of the RFP, and to utilize CDSI staff from the greater Washington Metropolitan Area. The proposed individuals, which represent actual costs to CDSI, were used to determine estimated labor costs for the contract. Whether they came from another division or not is irrelevant. It would be no different had we used the Webber/Hansen or the Mercer salary surveys to determine the estimated labor cost. In each and every proposal submission, CDSI prepared its proposed direct labor rates based upon personnel who met the RFP criteria. . . . In addition, the Government required CDSI to bid based on CDSI corporate titles which CDSI had established for management control. In many cases, our titles did not specifically match the Government's, nor were they ever intended to. This prevented CDSI from using individuals who, although they qualified for a particular labor category, did not have the 'proper' CDSI title. Protest File, Exhibit 75 at 000044-45. This draft letter to the DCAA was subsequently included in CDSI's BAFO. Transcript at 425-31; Protest File, Exhibit 75 at 000044-45. However, DOE did not consider this letter as part of CDSI's BAFO because it was stamped "DRAFT." Transcript at 555; Protester's Supplemental Protest File, Exhibit 78 at C 012524. Best and Final Offers CDSI's BAFO On December 30, 1993, CDSI submitted its BAFO. Protest File, Exhibit 75. No changes to CDSI's technical proposal were submitted. Id. at 000000. In its BAFO, in discussing its direct labor rates, CDSI explained: [CDSI] used only those representative personnel who fit the correlated category by meeting the RFP educational and experience requirements as well as your requirements regarding CDSI titles and contract labor categories. CDSI has again reviewed the DOE requested correlation of CDSI titles to DOE [RFP] labor categories and has complied with this request to the best of our ability. . . . CDSI's job titles were developed for CDSI corporate management purposes and we do not normally try to correlate them to contract labor categories. However, to comply with your instructions in both the oral negotiations and the August 27, 1993 letter, we have developed and proposed composite direct labor rates. We remain concerned that this methodology will disallow a qualified employee from working on the contract simply because he does not have a 'proper' CDSI title which corresponds to a RFP labor category. [S]everal [RFP] categories fall on the cusp of CDSI's title descriptions. This is especially true of RFP positions requiring five years of experience. Protest File, Exhibit 75 at 000055-56. The evidence as a whole leads us to conclude that even though CDSI did not use its internal labor categories for bidding and estimating, CDSI's employees meet the minimum skill, education, and experience requirements of the CDSI positions to which they are assigned. Transcript at 485-86, 595-96. In the BAFO, the attachment to Exhibit B listed, for each RFP labor category, the specific individuals whose salaries had been included in the rate with their CDSI labor categories. Protest File, Exhibit 75 at 000086-92. Pursuant to DOE's instructions, the rates included only Energy Systems Division personnel who: Were drawn from a CDSI position whose minimum requirements met the minimum requirements of the RFP position description and whose CDSI title matched the DOE RFP title 'senior to senior' and 'junior to junior;' except that where the CDSI title carried a senior designation, CDSI could use personnel from corporate positions whose minimums were less than the RFP; provided that the individuals themselves met the RFP education and experience requirements; Met the minimum educational and experience standards of the RFP position descriptions; Were not 'overqualified;' and Were not part-time employees or consultants. Transcript at 754, 757-68. CDSI gave its employees raises between initial cost proposal and BAFOs. Transcript at 1160. For example, one of the individuals in the initial cost proposal's labor pool for senior systems analyst had a raise in compensation from per hour at the time of initial proposals to at BAFO. Id. With regard to uncompensated overtime, in its BAFO, CDSI stated: Even though CDSI has no 'uncompensated overtime policy' in that all full-time employees are required to work a 40-hour week, in accordance with DOE's letter of December 13, 1993, CDSI has added additional hours per week for each Exempt employee. The additions to the DPLH's are shown in Exhibit B.1 and explained in Preface section 2.6. While we understand DOE's concern that, without this increase the rates may be diluted during contract execution, it is assumed that any contractor bidding uncompensated overtime in its proposal will have the additional hours added to the evaluated cost. Additionally, as personnel assigned to the contract are now allowed to work more than forty hours a week without approval, this methodology serves to penalize the contractor by inflating the number of hours which need to be worked to achieve full fee. Protest File, Exhibit 75 at 000005. CDSI further stated that the dilution: creates an increase in the proposed price, because it presumes that all of the uncompensated overtime is direct to the DOE contract. Protest File, Exhibit 75 at 000058-59. CDSI uses Transcript at 2167. CDSI's estimated cost at BAFO increased by over CDSI's initial proposal. Transcript at 1003-10; Protester's Hearing Exhibits 5 at 10-14. DynCorp's BAFO No changes were made to DynCorp's technical proposal at BAFO. Protest File, Exhibit 70 at 000000. At BAFO, DynCorp again did not use uncompensated overtime in developing its proposed direct labor rates. Protest File, Exhibit 70 at 000062, 000252. Protest File, Exhibit 16 at 000004. When exempt employees work in excess of forty hours per week, the hourly rate is reduced to reflect uncompensated overtime. Id. In its BAFO, DynCorp stated: We understand the DOE may be using the 1993 Wyatt Survey as a benchmark for salary data. We also have reviewed the Wyatt Survey to validate the reasonableness of our proposed labor rates. In most cases, our proposed labor rates are comparable to those in the Wyatt Survey. In a few cases, differences still exist: our proposed rates are higher for some categories and lower for others. In these few instances, we have examined the differences and believe the 1993 Mercer, Washington Technical Personnel Forum, and Washington Personnel Association provide more realistic estimates. . . . The salary levels cited in the Mercer, Washington Technical Personnel Forum, and Washington Personnel Association tracked closely with our own experience. We examined the participant base of the Wyatt Survey and determined that the government contracting firms were underrepresented in Wyatt. The majority of participants are in other industries such as banking, petroleum refining, and retailing. The Mercer, Washington Technical Personnel Forum, and Washington Personnel Association have a high representation of government high-technology firms, and we believe they represent a more accurate estimate of salary levels. Protest File, Exhibit 70 at 000009. The Final SEB Report: February 4, 1994 Technical Proposals In evaluating similar size and scope under criterion 1, the SEB did not use any standard, but rather determined that an all- encompassing definition of similar scope and size was impossible. Transcript at 1371-74. A member of the SEB who is currently the Deputy Office Director of the Office of Information Technology Services and Operations testified that there were three factors involved in determining whether a contract was similar in size and scope, i.e., number of people, cost, and type of contract. Id. at 148-49, 159-60. The SEB had no dollar cutoff in deciding whether a contract was similar in size. Id. at 165. Nor was there a cutoff of a particular number of people to measure whether a contract was similar in size. Id. at 168. With respect to scope, the SEB looked at the number of the twenty-four functions a contractor was performing on a reference contract. Transcript at 176. However, there was no standard to suggest how many functions out of the twenty-four a referenced contract had to have in order to be similar in scope. Id. One evaluator testified that he "lumped scope and size together in making one judgment as to whether a contract was similar in scope and size." Id. at 181. In evaluating criteria 1 and 2 the SEB did not look at the actual reference contracts themselves. Transcript at 104. Rather, the only information the SEB had was the description in the proposals and the information gathered as a result of reference checks. Id. DOE concluded that two of DynCorp's referenced contracts -- and contracts -- were similar in size and scope to the instant contract. The SEB evaluation criteria and worksheet for DynCorp's technical proposal on criterion 1 provided: Criterion 1: The offeror's corporate experience in performing contracts of similar scope and size. Strengths: The reference is similar in size and very similar in scope with 20 of the 24 functional areas addressed. (Significant) The reference is similar in size and very similar in scope with 20 of 24 functional areas addressed. (Significant) Weaknesses: corporate references are very small in size and appear limited in scope. Neither contract is a cost type contract, although the reference does include some contract line items which are billed as Time & Materials. The contract reference for is for the outsourcing of a billing process. It is not representative of the scope of work proposed for Exceptions/Deviations: None Protest File, Exhibit 80 at 000064. The contract reference sheet on the DynCorp contract indicated that DynCorp did not perform eight of the twenty-four functions at all. Protest File, Exhibit 33 at 000010; Transcript at 114. Four of the functions were being performed by DynCorp in "limited" or partial fashion. Transcript at 115; Protest File, Exhibit 33 at 000010. Nonetheless, in the final evaluation report for criterion 1 it was stated that twenty of the twenty- four functional areas were being addressed. Protest File, Exhibit 80 at 000064; Transcript at 116. One member of the SEB explained this discrepancy as follows: Q Having looked at the reference sheet and counting up the 'nones' and the 'limiteds,' do you have any explanation for why the final SEB report states that DynCorp's was very similar in scope with 20 of the 24 functional areas being addressed? A No. Why it states 20? No. Only the explanation, when we discussed this before in deposition, I think, is that we not only looked at the reference sheets, we also looked at all the information that was provided at the time and then developed the consensus. So, the reference sheets were . . . we used more than reference sheets. The reference sheets only provided one input. They were used, I hate to use the word 'verify', they were used to now in a general sense see what the contract was and what was being performed. . . . Q So, am I correct in thinking that the board, after performing the reference check, to determine if a proposal was accurate, could override the reference check by reading the proposal? A And discussing it. Transcript at 116-17. The DynCorp reference sheets for the contract indicated that of the twenty-four SOW functions, DynCorp provided none of the reference work for five functions and limited work for four. Transcript at 120-21. However, in the final SEB report the board stated: the contract "is similar in size and very similar in scope with 20 of 24 function areas addressed." Protest File, Exhibit 80 at 000064. One of the SEB members who performed this reference check testified as follows regarding the discrepancy: Q Now, having . . . gone through the 21 functions listed in [DynCorp's reference check for noting there were five nones and three limiteds, how do you reconcile . . . [the SEB report] which states that the [ contract] is similar in size and very [similar] in scope to 20 of the 24 function areas addressed? A I can't. Between the twenty to twenty-four there, but like we said before, we didn't just use the reference checks. The reference checks -- it would have been so simple that we could just have done the reference checks, we could have helped -- we could have saved a lot of time. Q So, am I correct in thinking that, as you did, in the although the reference check stated that DynCorp did not have the experience set forth in its proposal, you used DynCorp's proposal to override the reference check? A We used the proposal, plus we then discussed several things, again within consensus. How we arrived at the 20 out of the 24 numbers, I can't tell you today sitting here, what -- how we arrived at those numbers. But if you go to other places where the summary is in the proposal, it talks about more of a range, and I think you could get to it. Transcript at 123-24. The SEB member who performed DynCorp's reference check on the contract believed that DynCorp had people performing IRM work on that contract and that the contract value was million for DynCorp's work, when in fact the contract value and personnel represented IRM work. Transcript at 118-19; Intervenor's Hearing Exhibit 29. The SEB evaluated DynCorp's proposed key personnel in the same fashion as it evaluated the revised proposal: The Key Personnel proposed in the initial proposal were considered The offeror's proposed key personnel in its revised proposal/BAFO are considered to be although some non-significant weaknesses were noted as stated below. The proposed Administration Manager does not appear to meet the requirement of recent experience in the administration of a contract of similar scope and size. Protest File, Exhibit 80 at 000037. DynCorp's proposed administration manager testified that based upon his resume and experience from 1988 until the present he did not believe that he had experience administering a contract of similar scope and size. Transcript at 282. CDSI received the same technical scores in the final SEB report as it had in the December 1993 evaluation of revised proposals. Protest File, Exhibits 67 at 000024-34, 80 at 000074-84. In summarizing the significant differences among the proposals, the SEB stated: CSC demonstrated corporate experience and directly related experience. CSC's contract management plan was due to The CSC didn't reflect an Dyn[C]orp demonstrated corporate experience. Although Dyn[C]orp was not able to translate that into Dyn[C]orp received only an in the directly related experience. Dyn[C]orp presented a contract management plan. Dyn[C]orp was the only offeror who maintained consistency in its methodologies and procedures in every area of its proposals throughout the competition. Dyn[C]orp is the only offeror who submitted direct labor costs which reflect what may actually be experienced on the contract. CDSI only had corporate reference which was similar in size and scope to the model contract, Therefore, CDSI only received in the area of corporate experience. CDSI demonstrated directly related experience as a result of its The weakest area of CDSI's proposal was its contract management plan. While CDSI did not demonstrate any single deficiency in its contract management plan, the multitude of significant weaknesses made the plan on the whole. CDSI was the only offeror who received a rating lower than an rating for an overall evaluation criteria. Protest File, Exhibit 80 at 000028. The final technical scores were: TECHNICAL EVALUATION SCORES BEST AND FINAL OFFERS MAXIMUM OFFERORS TECHNICAL POINTS CSC DynCorp CDSI 1. Corporate Experience 2. Demonstrated Ability and Directly Related Experience 3. Contract Management 4. Key Personnel Total Points Protest File, Exhibit 80 at 000030. Cost Proposals With regard to direct labor, the SEB in the final report reiterated that because of the uncertainty of the outcome of CDSI's decision associated with staffing the contract, the estimated direct labor costs may not be realistic. Protest File, Exhibit 80 at 000043. The SEB report continued: CDSI used a different methodology to develop its direct labor costs at each stage of the competition. DCAA audit report [dated November 4] reflects that CDSI used employee rates from other than the Energy Systems Division (ESD), which is the proposing entity, to develop its direct labor rates. The initial DCAA audit report [dated November 4, 1993] stated that the category average rates were not representative of the actual rate that would be experienced. CDSI used only ESD personnel to develop its direct labor rates in its BAFO. Id. In the final cost evaluation, the SEB did not mention that CDSI had been given instructions during oral discussions or in the August 23, 1993, letter. In advance of its evaluation, the SEB cross referenced the Wyatt Data Services Salary Survey to the labor categories in the RFP. Transcript at 1781. Then when the SEB evaluated offerors' rates, it compared them to the Wyatt Survey. Id. at 1781-82. The contracting officer testified that with regard to CDSI's evaluation, the SEB did not use the existing contract rates. Id. at 1783. Nor did he look to the existing contract for retention of professional employees or turnover rate. Id. With regard to final evaluation of CDSI's labor rates for professional employees, the SEB concluded: The SEB questions of the twenty corresponding labor categories proposed. of the twenty labor categories evaluated for professional labor reflected compensation levels which appear unrealistically low and unreasonable. of the labor categories with questioned rates included actual salaries ranging from to below the average minimum salaries in the Wyatt Data Services Wage Survey (WDSWS). The WDSWS was the survey used by the SEB to evaluate compensation levels and corresponding labor categories. The labor categories reflected above and other labor categories reflect compensation levels that The SEB considers CDSI's direct labor rates not to be representative of what would be incurred under the contract should CDSI be selected for award. CDSI billed the DOE against the current contract for the period of December 1993, a higher average rate for of the twenty labor categories included in the aforementioned analysis, than it proposed in its BAFO dated December 1993. Protest File, Exhibit 80 at 000043-44. Finally, the SEB report concludes that CDSI did not demonstrate with respect to establishing its compensation levels. Protest File, Exhibit 80 at 000046. In so concluding the SEB reiterated that it was not a for CDSI to develop a and then not use it to assist in determining salaries and correlate labor categories to meet its client's needs. Id. at 000047. Specifically, the SEB stated: CDSI states the job titles are used to generically describe its grade structure which provides for the differences in education and experience, and skills required. CDSI grade structure provides for differences in education, experience and skills required to establish job levels. The jobs included within the grade structure are formally evaluated by senior CDSI management and its Human Resources Division. The vehicle for this formal evaluation is a formal job description outlining the duties and responsibility of the position, skills necessary and education/experience required. The job levels within the grade structure are given a title. CDSI states that these job titles, which include a formal job description and responsibilities of the position, skills necessary and education/experience required are not used to correlate CDSI's labor categories to contract labor categories, or to develop salaries. However, CDSI also states, 'each employee assigned to this contract will be assigned an appropriate CDSI title. The basis for placement will be individuals' compliance with CDSI job title requirements . . . CDSI contradicts itself, and its process appears to be one of reverse correlation not one of no correlation. It would appear that CDSI's grade structure, which defines job levels in terms of education, experience, skills, responsibilities, and duties, would be the foundation of its compensation plan program. But CDSI elects not to use its grade structure in the development of salaries. Instead it uses a contract to establish its foundation. It does not appear to constitute for a company to develop a comprehensive grade structure, and then not use it to assist in determining salaries and correlate labor categories to meet its client's needs. Id. at 000046-47. The evaluated cost for each offeror was: CSC DynCorp CDSI Best & Final Offer Probable Cost Protest File, Exhibit 80 at 000030. The SSO's (Source Selection Official's) Analysis In performing the final evaluation, the SSO first excluded CDSI because it had the Transcript at 255-56. DynCorp and CSC on Id. at 256-61. The SSO explained his determination of which proposal was most advantageous to the Government as follows: We're sitting here basically with some fairly equal contractors from a technical point, if you -- well, all three -- but we had quite a different range in cost estimates. And if you read in detail the board's report it tended to give a flavor that -- that the DynCorp proposal was a stronger estimate, if you will, even though it was lower, it was still -- it had strength behind it more than the other two proposals which, if you read the words in the report, the labor rate problem they had, some softness in other areas. It just -- it gave your confidence that . . . . DynCorp, although being lower, was a stronger more valid estimate. Transcript at 1562-63. The SSO had no knowledge of the directions given to CDSI by the contracting officer in oral discussions, in the August 27, 1993, letter, or the December 13, 1993, letter. Transcript at 1571. On April 22, 1994, DOE awarded the contract to DynCorp. Third Amended Protest Complaint 39. CDSI's Proprietary Information Which Was Allegedly Turned Over to DynCorp Fifteen pages of documents which show how CDSI runs its incumbent contract came into the possession of DynCorp. Transcript at 56; Supplemental Protest File, Exhibit 76. These documents were not received in response to a Freedom of Information Act (FOIA) request. Transcript at 289. DynCorp's proposed administration manager testified that he came across CDSI's documents while working in DynCorp's proposal development room on DynCorp's premises. Id. at 284. These documents reflect the CDSI management solution to running the existing contract and show who is responsible for which function and how many people are at various sites. Id. at 56. CDSI considers these documents proprietary. Id. at 57. These documents are hard copies of slides presented at a briefing for new employees done by CDSI's contract administration manager and security manager. Id. The new employees are informed that these documents include proprietary information and should not be disclosed outside the contract. Supplemental Protest File, Exhibit 76. These briefings have been given since 1988 to the majority of 700 people working on the current contract. Transcript at 59-60. Over a hundred people have left the contract since then. Id. at 60. DynCorp's administration manager testified that DynCorp did not use CDSI's organizational chart in preparing its proposal because the organization in the chart was not what was suggested by DOE in the RFP. Transcript at 316-17. The Contracting Officer's Responsibility Determination for DynCorp At the time the contracting officer signed the contract with DynCorp, he made an affirmative responsibility determination for DynCorp. Transcript at 1209. In concluding that DynCorp was a responsible offeror, the contracting officer reviewed proposal data, information from DCAA and GSA, and relied upon the advice of DOE counsel. Id. Prior to making his responsibility determination, the contracting officer became aware of an ongoing investigation of DynCorp through an article in the Washington Post. Transcript at 1211. The article concerned alleged fraudulent billing practices by DynCorp at Fort Belvoir, Virginia. The contracting officer also was aware that the Form 10-K in DynCorp's proposal provided: "The company was advised in November of 1991 that it might be a target of possible fraud and false claim prosecution." Id. at 1218. The contracting officer did not pursue any of the open investigations because until there was an indictment or a charge he did not want to influence himself or the SEB with inappropriate data. Id. at 1211-12. The contracting officer asked DCAA to assess the financial capability of the offerors. Transcript at 1214. DynCorp's financial information presented at the hearing indicated that DynCorp has a line of credit available outside of its corporation which would be available to finance of this contract. Id. at 1261. In addition, DynCorp has the availability of additional funding, Id. at 1260; Intervenor's Hearing Exhibit 1. DynCorp's financial records are audited by Arthur Anderson and Company. Transcript at 1273. Arthur Anderson has never advised DynCorp that its viability as a going concern is in danger. Id. Generally accepted auditing standards would require Arthur Anderson to so state if that is the case. Id. at 1274. Facts Relevant to the Organizational Conflict of Interest (OCI) Allegation DynMcDermott Petroleum Operations Company (DynMcDermott) is a Louisiana corporation, 60% of which is owned by DynCorp. Transcript at 1194. DynMcDermott is currently operating the strategic petroleum reserve (SPRO) for DOE in Louisiana. Id. Under the SPRO contract, DynMcDermott, a management and operations (M&O) contractor, operates and maintains facilities which are involved in unloading oil from oil tankers and pumping it through pipelines, maintaining pumps, and designing and maintaining the caverns where the oil is stored. Id. at 1195. All of the sites for the SPRO are in Texas and Louisiana. Although computers are involved in the SPRO contract in its wide area network and related equipment, none of those computers is maintained under the ITSO contract. Transcript at 1196. ITSO does not access any data bases used by SPRO. Id. at 1197. ITSO does not review or evaluate any work conducted under the SPRO contract or draft any specifications to be used in SPRO's work. Id. Within DOE, SPRO reports to the Assistant Secretary for Fossil Energy, and ITSO reports to the Assistant Secretary for Administration and Management. Transcript at 1197. The Office of the Assistant Secretary for Fossil Energy is supported by the DOE ITSO contract and the ITSO contractor could obtain access to computers and information on those computers at DOE headquarters. Id. at 1206. The contracting officer considered that DynCorp would have access to information regarding the SPRO contract when he made his organizational conflict of interest determination. Id. at 1227. The contracting officer contacted the program offices in an effort to ascertain whether there would be any areas of organizational areas of conflicts of interest. Id. at 1226. The contracting officer explained: It came down to the fact that in discussions with the program officials, and I say that's usually branch managers, . . . that to have an organizational conflict of interest with an M&O in Louisiana for the SPRO contract, that the awardee would actually have to be in a DOE policy decision-making billet, and that would never happen. Id. at 1228. The contracting officer further testified that he had specifically considered that DynCorp would have access to confidential information involving M&O contractors in providing maintenance, systems development, and software development. Transcript at 1229. However, he stated that there were ways to avoid a conflict by prohibiting access to certain files and machines, and by using removable hard disks. Id. at 1229. The contracting officer testified that he would advise his contracting officer's representative to ensure that DynCorp would not have access to such data involving M&O contractors. Id. at 1234. The contracting officer understood that perhaps entire segments or sections of DOE may be "off limits" to DynCorp. Id. at 1235. Testimony Relating to Purported Bias In the course of responding to a question about the purpose of preparing a prenegotiation memorandum, the contracting officer offered the following comment: By the way, here's those -- the notes that I talked about being on the specific CDSI document . . . that I had written down in relation to their discussions . . . . It says 'It has come to our attention that [a CDSI employee], who was proposed as the operations manager a key personnel slot, has been proposed as a contracts manager for a Social Security contract.' That was a concern of our[s] because it was an example of bait and switch. Transcript at 1626-27. No evidence was adduced at trial concerning the so-called bait and switch. The contracting officer also stated that there was an open investigation of CDSI regarding fraudulent billing. Transcript at 1211-12. When asked who told him that CDSI was under investigation, he testified: "It was either a procurement official or legal counsel." Id. at 1212. He further stated: "Who gave me the information, I don't know. I didn't take it any farther because . . . there wasn't any indictment, there wasn't any formal charge." Id. at 1213. He continued: Q Do you know if it concerned an investigation regarding CDSI or issues that CDSI brought to the attention of DOE to investigate? A I don't know any specifics about the investigation. Q So, as far as you know, it could have been regarding a situation that CDSI brought to the attention of DOE, correct? A Could have been. Id. at 1214. This testimony prompted CDSI counsel to seek to call the DOE counsel to whom the remark might have been made as a rebuttal witness. The Board denied this request, but directed DOE counsel to make a representation regarding (1) her conversation with the contracting officer and (2) her personal knowledge of any investigation of CDSI regarding fraudulent billing. In response to that request, counsel for DOE said that she had no personal knowledge of any investigation of CDSI involving fraudulent billing. Transcript at 1891. She further represented that she did not disclose the existence of any such investigation to the contracting officer. Id. at 1891-94. No evidence to corroborate the contracting officer's testimony that CDSI was under investigation for fraudulent billing was introduced. The Expert Testimony Protester's Expert on Similarity in Scope and Size of the Reference Contracts Protester's expert in IRM support services opined that both CDSI's reference contracts were similar in size and scope to the DOE model contract. Transcript at 349-50, 369. In rendering his opinion, protester's expert read the contracts themselves, a sampling of task orders, the reference checks performed by the DOE SEB members, the technical proposals, and other information in the SEB reports and the record. Id. at 349-51. He also took into account his with the contracts -- between 1985-1991 he had served as Id. at 335-36, 2073-75, 2082.10 Protester's expert first derived a standard for ascertaining whether a contract was similar in size. Id. at 352-53. He used principally contract value and determined that if a contract had approximately one-half of the value of the model contract it would be similar in size. The nature of the IRM contract was a secondary consideration. Id. at 353. Therefore, the expert believed that a contract valued at $100 million was similar in size to the DOE contract. Id. The expert viewed the contract as similar in size because it was approximately million in value and was an IRM services contract. Id. at 349-51. In looking at similarity in scope, protester's expert again devised his own method for defining such similarity. Transcript at 355. He first determined the proportion of the total number of hours in the RFP that were assigned to each of the twenty-four ____________________ 10 Protester's expert testified that based upon his role as source selection official and his oversight over the he did not view them as schedule contracts. Transcript at 363. functions. Id. at 354-62.11 He then determined from the reference contracts and their task orders which of the twenty- four IRM functions were represented and added up the percentage of total hours associated with each represented function. Id. at 356-64, 2072-74; Protester's Hearing Exhibit 4. He established 50% of total hours as the threshold for similarity in scope. Using this method, protester's expert determined that the contract, with functions being performed, represented of the total hours in the DOE model contract. Transcript at 2072-73. The contract, with functions, covered of the hours. Id. at 2073. Protester's expert also considered CDSI's contract to be similar in size with a value of million. Transcript at 369-71, 374-75. The contracting officer reviewed the additional documentation in this protest regarding CDSI's contracts. He testified that even if CDSI had provided the SEB with such additional information concerning both the and the contracts, DOE's assessment would not have changed. Transcript at 1384-1426. He performed this review personally; the other members of the SEB did not perform such a review. Id. at 1384, 1412-15. Protester's IRM expert further opined that DynCorp's contract was similar in scope but not similar in size to the DOE model contract. Transcript at 376. With regard to size, the expert reviewed all the sample task orders which DynCorp provided. Id. at 378. The expert divided the task orders into three categories -- IRM, mixed IRM and totally non-IRM. Id. at 379-80. He was unable to accurately determine the value of these components. The total value of all the sample task orders was approximately $16 million, and the value of the task orders for the non-IRM services was approximately $5 million or one-third. Id. at 380. Subtracting out one-third for non-IRM services of the total value of the contract, he determined that the value of IRM services was $81-$82 million, which was under his $100 million threshold for similarity in size. Id. at 376-81. With regard to scope of the DynCorp's contract, protester's expert testified that because approximately of similar tasks were being performed, the contract was similar in scope. Transcript at 381. Protester's expert opined that the contract was similar in neither size nor scope to the DOE model contract. Transcript at 382. As to size, protester's expert ascertained that even though the contract was stated to be valued at ____________________ 11 For example, the first function in the Statement of Work, "Systems Development," has 727,260 hours assigned to it, which is 10% of the total hours. Transcript at 360, Protester's Hearing Exhibit 4. it included a number of non-IRM services being provided, such as automotive services. Id. at 384.12 Protester's expert further concluded that not more than out of DynCorp personnel at provided IRM-related services. Id. at 388, 390. Therefore, he believed that approximately half of the value of the contract was attributable to IRM. Id. With regard to scope, he reasoned that the contract represented approximately of the requirements to be acquired under the DOE contract, which was significantly less than the 50% threshold he had established. Id. at 393. Protester's expert agreed with the SEB that DynCorp's subcontractor was not performing similar work on its subcontract. Transcript at 394. He reasoned that was simply providing its own data center and was essentially selling computer time to in contrast to actually running DOE's computer center. Id. at 394-95. Intervenor's Expert on Compensation Intervenor's compensation expert testified that the SEB was reasonable in its analysis of CDSI's cost proposal. Transcript at 1837. In arriving at his opinion, this expert analyzed fifteen distinct wage surveys. Id. at 1841-42. In its evaluation of CDSI's professional compensation, the SEB computed a minimum, middle, and maximum salary for each composite and compared these figures to the weighted average for an analogous labor category reported in the Wyatt Survey. Transcript at 1661. The SEB next determined whether any salary in the composite rate was 15% or more below the 50th percentile for the analogous job classification in the Wyatt Survey. Id. at 1662. Intervenor's compensation expert testified that he became concerned whenever compensation decreased more than 10% below the 50th percentile. Id. at 1880. Intervenor's compensation expert was also concerned with the range of salary rates used by CDSI to comprise certain RFP labor categories.13 For example, in the category senior systems analyst, CDSI's rates ranged from a low of to a high of In converting these hourly rates to annual salaries, the expert testified that there was a difference from approximately in annual salary to almost in annual salary. He, thus, questioned the representation that the persons fulfilling ____________________ 12 The subcontract between DynCorp and described both IRM services and automotive aviation services. Transcript at 387-88. 13 Intervenor's compensation expert recognized that the rates proposed by CDSI were diluted rates due to the treatment of uncompensated overtime, whereas the market survey rates were undiluted. Transcript at 1871. the requirements of this labor category were being paid a difference of as much as in annual salary. Transcript at 2019. In his analysis of CDSI's BAFO, intervenor's compensation expert pointed out that categories were more than 10% below the pertinent 50th percentile wage rate. Transcript at 2000-02; Intervenor's Hearing Exhibit 9. The systems analyst was less than the 50th percentile wage rate, the systems programmer, roughly less, the programmer analyst, less, the computer specialist, less, the senior operator, less, the electronics engineer, the senior electronics technician, the technical writer-editor, Transcript at 2001-02. Protester's Accounting Expert Protester's accounting expert testified that DCAA misinterpreted the use of uncompensated overtime at CDSI. Transcript at 1026. The expert testified that DCAA and the SEB caused CDSI to artificially lower its hourly rates by requiring CDSI to estimate based upon using a hour week. Id. at 1033-34. The diluted rate which resulted from the 41.3-hour week was approximately less than what it otherwise would have been. Id. at 1036. Protester's accounting expert stated that the SEB improperly compared the Wyatt Data Services Survey rates, which were undiluted, to CDSI's diluted rates in assessing whether CDSI's individual labor rates were reasonable and realistic. Id. at 1040-41. Protester's accounting expert opined that DCAA did not treat CDSI and DynCorp in the same manner in their audit reports with respect to overtime. With respect to CDSI, the expert testified that DCAA calculated the hours worked by exempt employees divided by the number of exempt employees to calculate a total number of hours worked of hours and then recommended that CDSI be required to use the hours to develop its hourly rate. Transcript at 2182-83; Protest File, Exhibit 28 at 000025. With respect to DynCorp, the DCAA auditors did not do a similar calculation of the number of hours worked by DynCorp's exempt employees divided by the number of exempt employees to come up with what uncompensated overtime there might be, despite the fact that DynCorp's proposal clearly indicated there was uncompensated overtime. Transcript at 2183; Protest File, Exhibit 28 at 000046-48. Discussion Is CDSI an Interested Party? For the first time in its posthearing brief, DynCorp challenges CDSI's standing to bring this protest. Intervenor's Posthearing Brief at 51. DynCorp contends that because CDSI is not next in line for award, it has no direct economic interest in the outcome of this protest. DynCorp cites in support of this position United States v. International Business Machines Corp., 892 F.2d 1006 (Fed. Cir. 1989), and Berkshire Computer Products v. Department of Commerce, GSBCA 12172-P, 93-2 BCA 25,724, 1993 BPD 36. We reject intervenor's argument. One of the cited cases involved a sealed bid and the other a negotiated procurement in which award was to be made on the basis of low cost among technically acceptable proposals. Thus, it was a simple matter to ascertain which vendor was next in line for award. Here, because award involves a subjective best value assessment, comparing differing approaches to providing complex IRM services, it is impossible, as a threshold matter, to conclude that a particular vendor is next in line for award. As we recognized in Federal Computer Corp. v. Department of Justice, GSBCA 12560-P, 94-1 BCA 26,442, at 131,563, 1993 BPD 264, at 5, "[t]he existence of a final reliable ranking is essential to the application of the IBM rule." CDSI's protest allegations challenge the conduct of discussions and technical and cost rankings of offerors. The application of the IBM rule is inappropriate where the protest allegations, if proven, will result in relief which alters or cancels the original ranking of offerors. If CDSI prevails as to its allegations regarding discussions, the likely remedy is further, proper discussions. If CDSI prevails as to its allegations regarding rankings of offerors, the agency's best value analysis must be redone. Federal Computer Corp., 1993 BPD 264, at 5 (citations omitted). Thus, DynCorp's standing argument must fail. The Technical Evaluation: Overview CDSI's challenges to the SEB's technical evaluation will be addressed in turn. We note at the outset, however, that this procurement was characterized by a lack of objective standards for evaluating proposals, inconsistent evaluation standards applied by the individual evaluators, a lack of any record of independent individual evaluations prior to "consensus," a "consensus" evaluation process which remains shrouded in mystery, inequitable and erroneous reference checks for CDSI, reference checks for DynCorp which were not accurately reflected in the evaluation, and a lack of meaningful discussions. In particular, there was no consistent methodology for determining the strengths and weaknesses of the proposals and no mechanism for establishing whether strengths and weaknesses were significant. This Board has recognized that it is inappropriate for evaluators to apply their own subjective standards such as "good feelings" in evaluating proposals. See CBIS Federal, Inc. v. Department of the Interior, GSBCA 12092-P, 93-2 BCA 25,643, at 127,606, 1992 BPD 386, at 13. Although the solicitation did not itself articulate objective criteria to be used in an analysis of strengths and weaknesses, the agency should have established a consistent methodology for evaluating proposals and failed to do so. Did DOE Change a "Mandatory" Experience Requirement Involving Contracts of Similar Scope and Size? In Count I, CDSI alleged that DOE improperly changed the mandatory requirements of the RFP by permitting offerors to submit only one contract of similar scope and size, rather than requiring three to five as specified in paragraph L.048 in the RFP.14 CDSI contends that DynCorp failed to meet this mandatory minimum requirement and, therefore, should not have received the award. As a threshold matter, we must ascertain whether the requirements of the RFP involving corporate experience were mandatory. In assessing whether a given requirement is mandatory, we read the RFP as a whole. Hughes Advanced Systems Co., GSBCA 9601-P, 88-3 BCA 21,115, 1988 BPD 185. The solicitation in Section C details the twenty-four functional areas in which IRM services must be provided. The provision of these services is clearly and expressly stated to be mandatory. Section C contains no mention that an offeror must possess a requisite level of corporate experience to be eligible for award. We next look to Section M. Paragraph M.0009, Evaluation Criteria, specified: Technical aspects of the proposals will be evaluated in accordance with the following criteria and scored on a numerical basis. Criteria 1, 2, and 3 are approximately equal in importance. Criterion 4 is significantly less important. . . . CRITERION 1: The offeror's corporate experience in performing contracts of similar scope and size. There is no language in this paragraph or any other part of Section M suggesting that offerors who do not meet the corporate experience requirements are ineligible for award. We turn to the section relied upon by CDSI in urging that criterion 1 experience is a mandatory requirement, paragraph L.048(b), contained in the "Format and Content" section of the instructions on technical proposals. To place those instructions in context, we review the general provisions of paragraph L.048(a). There, the solicitation instructs: ____________________ 14 CDSI further claimed that if DOE had informed it that it was not required to describe a minimum of three to five contracts similar in scope and size to the proposed contract to demonstrate its corporate experience, CDSI would have selected different contracts to address the twenty-four functions evaluated under criterion 2. Volume II - Technical Proposal consists of the offeror's proposal addressing the technical and management aspects of the procurement, capabilities and directly related experience that will be used to satisfy the requirements of the Statement of Work. Since the Technical Proposal will be evaluated to determine such matters as the offeror's understanding of the work to be performed, the management approach, quality, and the ability to complete the desired work, it should be specific and complete in every detail. The proposal should be practical, be prepared simply and economically and represent a quality product. Paragraph L.048(b), the linchpin of CDSI's argument, goes on to state in pertinent part: Technical Discussion. This section must contain the major portion of the Technical Proposal. It should clearly address each of the Technical Proposal evaluation criteria in section M, and at a minimum, cover the subordinate factors or subcriteria listed thereunder, if any. It should be presented in as much detail as practicable and include the following four major sections. Section 1. The offeror's corporate experience with contracts of similar scope and size. a. The offeror must describe corporate experience on not less than three nor more than five IRM support services contracts similar in scope and size to this proposed contract. Reading these paragraphs together, we conclude that Section L.048 set forth mandatory instructions for proposal preparation -- something very different from dictating the mandatory minimum criteria which a contractor must meet in order to be eligible for award. This interpretation is bolstered by the RFP provision "Content of Resultant Contract," which informed offerors that the instructions in Section L would not become part of the resultant contract. This case is thus unlike OAO Corp. v. General Services Administration, GSBCA 12484-P, 94-1 BCA 26,392, 1993 BPD 247, cited by protester. In OAO, there was a mandatory requirement stated in both Section C and Section L that key personnel be available on the date of award. Importantly, the Section L provisions in OAO expressly advised offerors: "To be acceptable, an offer must have acceptable resumes for all required positions. . . . The offeror . . . shall certify that the individuals named shall be available . . . on the date the contract is effective." OAO, 94-1 BCA at 131, 293, 1993 BPD 247, at 4 (emphasis added). Thus, in OAO, compliance with the Section L instructions directly affected the acceptability of an offer. Here, while the solicitation did "require" offerors to adhere to instructions in the preparation of their proposals, i.e., to describe three to five contracts similar in scope and size, the failure to adhere fully to those instructions did not operate to disqualify an offeror. Rather, an offeror's failure to describe contracts which were ultimately determined to be dissimilar in size and scope would subject the proposal to a downgrading under paragraph M.0009. Nor do the answers to vendor questions 1 and 37 suggest that the corporate experience requirements under criterion 1 were mandatory. In vendor question 1, an offeror asked whether, if two or more companies formed a joint venture, their joint experience would be evaluated as that of "the proposing entity." DOE responded that the prime contractor "must submit not less than three contracts of similar scope and size through its own corporate experience." Consolidated RFP at C 012568 (emphasis added). The requirement was thus one of proposal submission. In question 37, a vendor asked the range of level of effort or contract value which would constitute similarity in size. In responding, DOE stated: While specific ratings could not be established on undefined contracts at this stage of the process, it would be fair to say that contract references which depict contracts similar in scope and size would score higher than contracts non-similar in size and scope. Consolidated RFP at C 012568 (emphasis added). This latter language indicates that the three to five reference contracts were to be scored and not used as a bright-line, pass-fail elimination mechanism. In sum, "[i]f respondent had intended that the proposals meet each specification requirement or be summarily eliminated from the competition, respondent would have specifically so stated, as agencies typically do." Hughes Advanced Systems Co., GSBCA 9601-P, 88-3 BCA at 106,602, 1988 BPD 185, at 9. We, thus, conclude that DOE neither changed nor waived a mandatory solicitation requirement as to corporate experience. Did DOE Change a Mandatory Experience Requirement for Key Personnel? CDSI argues that the RFP contained mandatory requirements for all key personnel and that DynCorp's proposed administration manager failed to satisfy the minimum mandatory requirements for this position. Protester's Posthearing Brief at 9-12. Applying the same rationale articulated above and reviewing the solicitation as a whole, we conclude that the key personnel experience requirements are mandatory and that DynCorp's administration manager does not appear to possess the requisite experience. The solicitation required offerors to submit six resumes of named individuals to serve in key positions as part of their technical proposals. These six individuals were to supervise over 700 people involved in performing a contract worth over $225 million. No substitutions of key personnel were permitted within the first 365 days of contract performance, absent sudden illness, death, or termination of employment. As this Board recognized in OAO: "In a contract for services of this magnitude, it is hard to imagine a provision more material than one which requires the offeror to have acceptable key personnel available to work on the contract." OAO, 94-1 BCA at 131,297, 1993 BPD 247, at 12-13.15 The key personnel qualifications were expressly stated to be mandatory in Section L and Attachment J. In contrast to the Section L provision on describing corporate experience, the section on key personnel was not directed to the description of a key personnel's experience, but rather specified minimum mandatory experience which the employee had to possess to be acceptable. Paragraph L.048, Section 4(c) (emphasis added) stated: "[c]andidates shall meet at a minimum the required position description qualifications in Section J." The position description in Section J required the administration manager to have "at least two years of recent experience in the administration of a contract of similar size and scope." As we recognized in U.S. Sprint Communications Co. Limited Partnership, Networks Systems Division, GSBCA 10684-P, 91-1 BCA 23,333, at 117,025, 1990 BPD 264, at 14, solicitation language stating "the minimum technical requirements and performance required" left no choice to the vendor. We thus conclude that the requirement that the administration manager possess at least two years of recent experience in the administration of a contract of similar size and scope was mandatory. DynCorp failed to meet this requirement. DOE itself concluded that DynCorp's administration manager did not appear "to meet the requirement of recent experience in the administration of a contract of similar scope and size." DynCorp's proposed administration manager admitted that he had not administered a contract of similar size and scope from 1988 until the present. The SEB also concluded that CDSI's systems development and maintenance manager failed to meet the requirement of four years experience managing a functional area staffed by a minimum of 100 people. CDSI has not established that this conclusion was erroneous. Thus, we conclude that both CDSI and DynCorp should ____________________ 15 While we recognize that evaluation of key personnel was the fourth criteria in the technical evaluations, and was stated to be significantly less important than the other three criteria, in examining the RFP as a whole we nonetheless recognize the criticality of key personnel. have been advised during discussions that their proposed key personnel did not meet mandatory requirements. As we recognized in CBIS Federal, where an offeror was ineligible for award because the resumes of key personnel did not comply with minimum RFP requirements, the offeror should have been advised of this through discussions and given an opportunity to supplement the resumes or replace proposed personnel. 93-2 BCA at 127,607, 1992 BPD 386, at 15. Did DOE Improperly Evaluate CDSI's Corporate Experience? CDSI contends that DOE's determination that only of CDSI's reference contracts was similar in size and scope was erroneous and adversely affected CDSI's evaluation. We agree. DOE's reference checks on CDSI were flawed. It is undisputed that for all offerors except CDSI two members of the SEB performed the reference checks to insure their validity. With CDSI, however, only one member of the SEB, the chairman, performed the reference check. Furthermore, when this individual contacted the reference, he did not speak with the technical officer, the preferred contact. Rather, he spoke with a contracting officer who was admittedly in a hurry. This contracting officer was the contracting officer on contracts, and to speed things up, she performed reference checks simultaneously. The chairman of the SEB admitted that in doing this reference check he did not use the form used for all the other reference checks because he was attempting to hurry and it got too confusing when he did the reference simultaneously on two contracts. He testified: A Okay. When [the contracting officer on the said she was pressed for time and could I possibly do anything to speed the process up -- . . . I asked her to give me a few seconds and . . . what I meant to do was delete those functions that were not in CDSI's proposal . . . . . . . . Well, I started deleting, you know, and I even got confused as well, maybe I didn't delete the right one. I may have added one or two. It became a total confusing mess. You know, so I just didn't want to use them. I wanted to make sure I was being accurate and corroborating the proper areas, so I used the page out of the [CDSI proposal].16 ____________________ 16 Transcript at 1322-23. In this colloquy, the chairman of the SEB contradicted his earlier testimony at the hearing regarding his reference checks on CDSI's Transcript at 218-28, 1318-25. Then he included the reference form, which was "a total confusing mess," as part of the SEB official file for all of the other SEB members to utilize, and never mentioned the problem. This witness said he had intended to correct the form but, to his embarrassment, he "just forgot it" and "never did it." Transcript at 1324. The bottom line, of course, is that the procedure which DOE utilized in performing the reference checks on CDSI's contracts was woefully deficient and unfair. The entire process of ascertaining whether contracts were similar in size and scope was flawed by the failure of the SEB to follow an objective standard defining similarity in size or similarity in scope. "Similar" is not defined in the RFP, and no parameters for determining similarity of size and scope were stated in the RFP or in instructions to evaluators.17 For size, one member considered dollar value, numbers of personnel, and "type" of contract (e.g., whether it was cost plus award fee -- like the model contract).18 The SEB chairman, on the other hand, considered value, number of people, scope, "complexity," and "location." Another member took number of people and dollar value into account. The SEB members did not utilize any cutoff in determining the size of the contracts. Where dollar value was considered, no particular dollar value was established as a baseline for assessing similarity. In determining similarity of scope, the SEB members considered the extent to which the twenty-four IRM functions in Section C of the RFP were performed under the reference contracts. However, no cutoff was established for determining how many or which functions had to be performed in order for a reference contract to be similar. ____________________ 17 We recognize that the response to vendor question 37 advised offerors that similarity in size would be judged against the level of effort in the RFP and similarity in scope would be considered in terms of volume and types of ADP services. However, we conclude that this guidance does not establish an objective standard for assessing similarity. 18 CDSI contends also that the use of the "type of contract" in assessing similarity of size was an improper, undisclosed evaluation criterion. Protester's Posthearing Brief at 27. We agree with CDSI. The type of contract is not rationally related to the size and scope of a contract. It is a distinct, new consideration which cannot reasonably be subsumed into the overall inquiry on similarity in size and scope. This adds support to our conclusion that the evaluation of CDSI's proposal under criterion 1 was improper and should be redone. DOE and DynCorp attempt to minimize the adverse effect of this irregularity on the evaluation of CDSI's reference contracts. First, they contend that because CDSI's own proposal only identified Second, they contend that the contracting officer's testimony conclusively established that the were not in fact similar in scope and size to the DOE contract. We disagree with both analyses. First, if the reference check had been done properly, would have been identified as having been performed, just as occurred in the reference checks of contracts. Second, the contracting officer is not himself the entire SEB. The fact that he thought the contracts were dissimilar does not mean that the SEB as a whole would have so concluded. CDSI would have the Board determine that CDSI should have received additional points due to the similarities in scope and size of the contracts and the proposed DOE contract. While it appears to the Board that DOE overlooked certain similarities between the contracts, the Board declines CDSI's invitation to usurp the SEB's function and point score a proposal. E.g., Sperry Corp., GSBCA 8208-P et al., 86-1 BCA 18,704, at 94,081, 1986 BPD 9, at 3. Did DOE Perform an Improper Technical Evaluation of DynCorp's Experience? CDSI alleges that DynCorp did not describe any reference contracts that would qualify as similar in scope and size to the proposed contract. DOE had concluded that DynCorp's contract was similar in size and scope. However, although DynCorp indicated in its proposal that the contract value of the contract was and that employees were performing on that contract, in fact, the employees were performing IRM work, and of the contract amount related to IRM work. Thus, the SEB did not possess crucial information in evaluating the size and scope of the contract. An incomplete evaluation was also performed as to the similarity in size and scope of the contract. The record indicates that there were non-IRM functions and services performed under the contract which were not taken into consideration by the SEB. Specifically, the contract itself and task orders issued under it referenced services such as establishing logistical procedures or "proper purchasing/leasing, receiving and accounting and distribution of all supplies" and putting on "conference workshops." While we do not adopt protester's expert's conclusions that IRM-related services and functions represent only two-thirds of the total value of the contract, we are persuaded that there are significant non-IRM functions in that contract which should have been taken into account. Moreover, in translating contract reference scores into the SEB report, the SEB improperly raised DynCorp's scores as compared to its reference form with regard to both the and contracts. The SEB members were unable to explain why the number of functions being performed was far fewer in the actual reference contracts than in the SEB report with regard to both these references. The post hoc vague explanation that these scores were developed in consensus and the proposals themselves were reviewed does not persuade us that the SEB reports are accurate. In sum, DOE's conclusion that the and contracts were significant strengths is not supported by the record. Did DOE Improperly Evaluate Offers Under Criterion 2? Under criterion 2, offerors were required to demonstrate their capabilities and directly related experience for performing the twenty-four functions in the RFP. Only references submitted under criterion 1, including subcontractor references, could be relied upon under criterion 2. In addition, offerors could propose a contract reference for a proposed subcontractor if the direct productive labor hours for that subcontractor exceeded 10% of the total DPLH in the proposal. CDSI contends that the SEB improperly permitted DynCorp to use two subcontractors, and as references under criterion 2. CDSI contends that was not assigned 10% of the DPLH and, therefore, its experience could not properly be considered. Even though experience was listed as a weakness, CDSI argued that DynCorp should have received no experience for these functions and should have been graded a "significant" weakness, decreasing its score from to under criterion 2. In reply DynCorp states that CDSI has misread the solicitation provision which states: "Subcontractors proposed for whom the DPLH equals or exceeds 10 percent of the total estimated DPLH for this solicitation shall provide one contract reference." DynCorp contends that this provision "does not prohibit subcontractors with less than 10% of DPLH from being used as a reference; rather it states that subcontractors over 10% must provide a reference." Intervenor's Reply Brief at 11. We agree with DynCorp. The RFP stated an additional requirement that subcontractors for whom the DPLH equals or exceeds 10% were required to provide an additional contract reference.19 ____________________ 19 In Count IV, CDSI alleged that DynCorp was impermissibly allowed to submit contract references in addition to the number permitted in criterion 1 of the RFP to satisfy the requirements in criterion 2. Consistent with the preceding Further, the RFP's provision on reference contracts speaks in terms of "offerors" submitting three reference contracts of similar size and scope. Read in conjunction with DOE's response to vendor question 1, the RFP permits subcontractors' experience -- regardless of their percentage of the DPLH -- to be evaluated as the experience of the prime. The response to vendor question 1 states, in pertinent part: Subcontractor(s)' . . . experience will be evaluated as experience of the prime contractor. However, for the evaluation under Criterion 1 the prime contractor (one firm) must submit not less than three contracts of similar scope and size to its own corporate experience. Thus, if the prime chose to submit three reference contracts, an additional two presumably could be from subcontractors regardless of the percentage of DPLH being performed by those firms. We also conclude that the reference was properly considered. CDSI contends because the contract was found to be dissimilar to the instant contract under criterion 1, it should not have been considered as a strength under criterion 2. We disagree. The reference contracts were not required to be graded in the same manner for both criteria 1 and 2. What was a weakness in similarity in size and scope was reasonably found to be a strength under criterion 2 in that it demonstrated directly related experience. Did the SEB Improperly Evaluate Criterion 3? CDSI has alleged two improprieties in the technical evaluation regarding criterion 3. First, CDSI challenged the increase in DynCorp's score from a to a for subcriterion 3(b), quality management program, between DynCorp's initial and revised proposals even though DynCorp made no substantive changes in its revised proposal. Protester's Posthearing Brief at 25-26. DOE and intervenor contend that DOE properly rescored all proposals under subcriterion 3(b) prompted by DOE's initial evaluation of Specifically, in trying to understand what DOE made flowcharts for the same thing for CDSI and DynCorp. DOE could not create a flowchart for CDSI. DOE maintains that the strength of DynCorp's program became apparent after the flowcharts were created. Intervenor's Posthearing Brief at 59-60. ____________________ analysis, we deny this count. We note that in the initial evaluation worksheet, the SEB listed no strengths and no weaknesses for DynCorp's quality management program and scored the proposal at While we are troubled about the apparent lack of a thorough evaluation at the outset, we find the SEB's explanation and revised evaluation credible, and we cannot conclude that the final score is unjustified. Second, CDSI challenges DOE's determination to downgrade CDSI under criterion 3 because it did not have a plan, a program, and a when the solicitation contains no requirement for such plans and programs. DOE assigned CDSI a under subcriterion 3(b), quality management program, for its failure to have CDSI contends that by insisting on these features, DOE applied undisclosed evaluation criteria nowhere stated in the RFP. Protester's Posthearing Brief at 27-28. Ideally, if an agency requires offerors to possess and utilize plans and programs, the agency should clearly disclose that in the solicitation. Here, the agency did not use the adjective in describing criterion 3 components. However, the agency did request, in vendor questions to CDSI, a description of the programs, and CDSI provided such a description. Therefore, CDSI was on notice that the agency was assessing plans in evaluating criterion 3. See Grumman Data Systems Corp. v. Department of the Air Force, GSBCA 11635-P, 1994 BPD 64, at 42-43 (Mar. 19, 1992). The agency was legitimately interested in an offeror's programs. Contract management, quality management, management approach, and oversight are items which are readily susceptible to plans and programs and we conclude, on the whole, that it was not unreasonable for the agency to request such information and utilize it in its evaluation of these subfactors. Id. Did DOE Fail to Conduct Meaningful Discussions with CDSI? CDSI claimed that DOE improperly failed to discuss the following deficiencies with it during discussions: (1) only of its reference contracts qualified as similar in scope and size to the proposed contract; and (2) CDSI's systems development and maintenance manager did not satisfy one of the mandatory requirements for this position. We agree that DOE failed to conduct meaningful discussions with CDSI in these areas. First, DOE concluded that CDSI had a under criterion 1 because it had only contract which was similar in size and scope. CDSI received out of a possible points for this criterion. The contracting officer testified that he did not discuss this with CDSI because CDSI had received a score of which was under the rating plan. Regardless of the label applied by the contracting officer, criterion 1 was a significant part of the evaluation, and the failure to advise CDSI that of its reference contracts were not similar in size or scope was error. System Automation Corp., GSBCA 8204-P, 86-1 BCA 18,654, at 93,817-20, 1985 BPD 15, at 20-24. Similarly, DOE should have held discussions with CDSI concerning the failure of its systems development and maintenance manager to meet the minimum requirements. As we have recognized, the experience requirements for key personnel were minimum mandatory requirements. Finally, as explained more fully below, in the context of the cost evaluation, DOE conferred an unfair competitive advantage on DynCorp by disclosing in its post negotiation letter of August 27, 1993, that the Wyatt Data Survey would be utilized in evaluating cost realism and that direct labor rates below the 50th percentile of that wage survey would be deemed unrealistically low. Did DOE Conduct an Arbitrary and Capricious Cost Evaluation? As a prelude to our cost discussion, we note that in the context of a cost reimbursement contract, the evaluation of the cost proposal should not be a primary consideration. FAR 15.605(d), 48 CFR 15.605(d) (1993). This is because, regardless of the cost proposed, the Government is bound to pay the contractor its actual and allowable costs. FAR 15.605(d) provides: (d) In awarding a cost-reimbursement contract, the cost proposal should not be controlling, since advance estimates of cost may not be valid indicators of final actual costs. There is no requirement that cost-reimbursement contracts be awarded on the basis of lowest proposed cost, lowest proposed fee, or the lowest total proposed cost plus fee. The award of cost-reimbursement contracts primarily on the basis of estimated costs may encourage the submission of unrealistically low estimates and increase the likelihood of cost overruns. The primary consideration should be which offeror can perform the contract in a manner most advantageous to the Government, as determined by the evaluation of proposals according to the established evaluation criteria. In construing this FAR provision, the General Accounting Office recognized in AmerInd, Inc., B-248324, 92-2 CPD 85, at 4 (Aug. 6, 1992): "Consequently, a cost realism analysis must be performed by the agency to determine the extent to which an offeror's proposed cost represents what the contract should cost, assuming reasonable economy and efficiency." The solicitation here provided that an unrealistic cost or price proposal may be evidence of the offeror's lack of understanding or poor understanding of the project. The solicitation continued: "The cost proposal will be evaluated for determination of the probable cost to the Government for those offerors in the competitive range . . . ." As both the GAO and this Board have recognized, the contracting agency is in the best position to make a cost realism determination and, therefore, the review of an agency's judgment is limited to determining whether the agency's cost realism evaluation was reasonably based. AmerInd, at 4; SofTech, Inc. v. Department of the Air Force, GSBCA 11708-P, 92-3 BCA 25,080, at 125,017, 1992 BPD 116, at 18 (". . . in general, cost realism determinations are 'by their nature business judgments which call for the exercise of discretion by Government officials.' Thus, unless we find that the exercise of this judgment was 'clearly defective' we will not upset it 'even if the protester's contrary opinion is not unreasonable.' Program Resources, Inc., GSBCA 8879-P, 87-2 BCA 19,816, 1987 BPD 65."). After considering the myriad allegations which CDSI has raised against DOE's cost evaluation in this protest, we conclude that the agency made two errors, not in its cost realism determination, but in its disparate treatment of offerors. First, DOE disclosed to DynCorp, but not CDSI, that the Wyatt Wage Data Survey would be used as a standard for assessing the cost realism of direct labor rates and that any salaries below the 50th percentile would be considered unrealistically low. In so advising DynCorp, the SEB gave DynCorp an unfair competitive advantage that it did not bestow on other offerors. We recognize that DynCorp had utilized wage surveys to derive its rates and that the SEB in disclosing this information was addressing that portion of DynCorp's proposal. Nonetheless, DOE went too far in its advice and skewed the playing field in DynCorp's favor. We further recognize that the RFP in general terms in paragraph M.13 advised offerors that in discussing professional compensation evaluation "supporting information will include data such as recognized national and regional compensation surveys . . . ." However, this general statement does not come close to advising offerors of the specific "benchmark" DOE disclosed to DynCorp. While CDSI was told nothing in its discussions regarding wage surveys, DynCorp was told that DOE would use the 50th percentile of the Wyatt Data Survey to establish the probable cost in those instances where rates proposed were below the 50th percentile, and that in the event DynCorp failed to submit its own surveys or specific employee salaries, the DOE "will use the Wyatt Data Surveys and other data deemed necessary to establish the probable cost." DynCorp was thus given more specific guidance regarding what direct labor rates would be considered reasonable than was CDSI. DynCorp used this information regarding the Wyatt Wage Survey at BAFO. For the most part, DynCorp was able to correlate its direct labor rates to the Wyatt Survey, but in those few instances where it could not, it took great pains to distinguish away the Wyatt Survey and explain why its own experience and other surveys were more relevant checkpoints. In its BAFO CDSI referenced the Wyatt Survey as follows: "Moreover, to ensure the rates are reasonable for the work described in the position descriptions, we have compared the proposed rates to applicable categories in the Mercer, Wyatt-ECS, Washington Personnel Association, Washington Compensation Forum, and Washington Technical Personnel Forum surveys. These rates are, we believe, the minimum reasonable rates to attract and retain qualified contract personnel, prevent loss of continuity of the DOE environment, and fulfill the requirement of M.013 of the Solicitation." Unlike DynCorp, CDSI did not correlate its wage rates primarily to the Wyatt Data Survey or distinguish that survey from any of its own wage rates which were not within the appropriate range of that survey. The SEB used exclusively the Wyatt Data Survey to ascertain the reasonableness of rates. We believe this disparity in treatment gave DynCorp an unfair competitive advantage. As we recognized in Communications Network Systems, Inc. v. Department of Commerce, GSBCA 12705-P, 1994 BPD 63, at 24 (Feb. 24, 1994) (citations omitted), "'[t]he sine qua non of competitive negotiations, and indeed of all public contracting is that all offerors be afforded an equal opportunity to compete.' This requires that offerors receive equal information during negotiations." Here, the disclosure of use of the 50th percentile methodology as well as the identity of one survey which would be used in evaluating proposals, gave DynCorp a competitive edge. The second element of disparate treatment in the evaluation of cost proposals concerned the use of uncompensated overtime. Although and both offerors indicated in their proposals that they were not proposing uncompensated overtime, the agency suggested that CDSI dilute its rates based on its use of uncompensated overtime, but did not apply the same standard to DynCorp.20 Although we deny the remainder of CDSI's allegations regarding cost, some bear comment. During the hearing and subsequent briefing, CDSI focused on two contentions: (1) The contracting officer's improper conclusion that there was no ____________________ 20 We recognize that in the request for BAFOs, DOE left the treatment of uncompensated overtime up to CDSI. However, we view the pre-BAFO instructions to have influenced CDSI's proposal of uncompensated overtime at BAFO, and to have reasonably caused CDSI to dilute its rates. difference between composite rates and category rates, and his restricted CDSI's ability to provide its best proposal.21 (2) The contracting officer's misconception that individuals whose rates were used for the development of direct labor rate estimates were the actual individuals who would be performing work under the contract. CDSI contends that as a result of this methodology, the contracting officer forced CDSI to exclude 200 personnel from the existing ITSO contract from CDSI's estimate, thus preventing CDSI from proposing its own best estimate for the cost of performing the work. Protester's Posthearing Brief at 37. In considering these allegations, we are cognizant that the contracting officer's cost realism determination should not be upset unless clearly defective. The evidence as a whole does not support a finding that the contracting officer insisted that CDSI Rather, the evidence suggests that the contracting officer took great pains to ensure that those individuals whose salaries were actually qualified for the RFP labor categories in all respects. The RFP clearly permitted offerors 22 ____________________ 21 DynCorp argues that CDSI's contentions regarding its direct labor rate are untimely. We disagree. DynCorp contends that CDSI knew the impact of DOE's instructions long before contract award and should have protested before that time. Intervenor's Posthearing Brief at 152. Although CDSI was aware of the contracting officer's instructions and attempted to comply with them, CDSI was not aware of the way the contracting officer intended to use these instructions and how they would impact his evaluation. We view CDSI's challenge regarding its direct labor rates as a challenge to the evaluation and not the instruction. __________ Therefore, the allegations are timely. 22 The DCAA Audit Manual, paragraph 9-505, Evaluation of Estimated Direct Labor Rates, recognizes that direct labor rates used to estimate direct labor costs may be at expected average rates which may be either separately estimated for each proposal or pre-established for pricing many proposals submitted over a given period of time. Protester's Posthearing Brief at 50-51. Id. The contracting officer, on the other hand, CDSI has also lodged numerous additional attacks against DOE's instructions regarding CDSI's preparation of its revised cost proposal and its BAFO. In essence, CDSI contends that the directives of the contracting officer were misleading and caused it improperly to raise its rates or to offer labor rates for certain RFP categories. We do not agree that the record supports these contentions and, in the interest of judicial economy, we do not discuss these contentions given our conclusion that in two regards the agency did improperly evaluate CDSI's cost proposal, thus warranting corrective action in any event. Did DOE Perform an Improper Responsibility Determination of DynCorp? CDSI claims that the contracting officer had inadequate information on which to base an affirmative responsibility finding. In particular, CDSI alleges that the contracting officer failed to consider that there were ongoing investigations involving allegations that DynCorp falsely claimed to have done work that was never completed at Ft. Belvoir. Finally, CDSI notes that the 10-K report furnished with DynCorp's proposal indicated that in November 1991 DynCorp was advised that it might be the target of possible fraud and false claim prosecution. CDSI contends that because DOE failed to consider information which was relevant to DynCorp's integrity and ethical business practices, it failed to meet its obligation prior to determining that DynCorp was responsible. Protester's Posthearing Brief at 98-99. DOE and intervenor respond that the responsibility determination was adequate because the contracting officer ascertained that DynCorp was not under indictment, and was not on GSA's list of debarred and suspended contractors. In addition, the contracting officer checked with DynCorp to see whether a new certification needed to be submitted. Respondent's Posthearing Brief at 58; Intervenor's Posthearing Brief at 164-67. Intervenor persuasively contends that open investigations are not, by themselves, reasons to find an offeror nonresponsible. As a matter of law, this Board has traditionally accorded a contracting officer's responsibility determination deference and tempered its de novo review. Executone Information Systems, Inc., v. Department of Health and Human Services, GSBCA 12402-P, 94-1 BCA 26,274, at 130,730, 1993 BPD 203, at 17. Protester has cited no statute, regulation, or other authority which has been violated by the contracting officer in declining to rely on open investigations or a settlement. The Federal Acquisition Regulation establishes the mandatory requirements which contracting officers must follow in performing responsibility determinations. FAR 9.404. Those regulations establish the GSA's list of debarred and suspended offerors as the checkpoint for ascertaining an offeror's responsibility. So long as an offeror in a negotiated procurement is not on the ineligible list, it is free to submit a proposal. Cf. Integrated Systems Group v. Department of the Navy, GSBCA 12784-P (May 12, 1994) (offeror's response to a request for quotation could be considered, even though at the time the response was submitted offeror was on the debarred and suspended list, so long as offeror was removed from the list prior to award). Nor was there any substance to CDSI's contentions that DynCorp lacks financial responsibility. The contracting officer reviewed DynCorp's 10-K and relied upon a report by DCAA. At hearing, DynCorp established that it had access to sufficient capital to perform the ITSO contract. CDSI has not demonstrated that DynCorp failed to provide accurate and complete financial information to the contracting officer. Did DOE Award to an Offeror Which Had an Impermissible Organizational Conflict of Interest (OCI)? CDSI contends that because DynCorp's affiliate, DynMcDermott, is a DOE management and operations (M&O) contractor responsible for operating DOE's Strategic Petroleum Reserve (SPRO) in Louisiana, there exists an impermissible organizational conflict of interest. CDSI reasons that if permitted to perform the ITSO contract DynCorp will have access to proprietary information concerning M&O contractors, which "may include DOE evaluations, concerns, future plans, or investigations concerning M&O contractors generally or DynCorp specifically. In either case, this information would be of value to DynCorp's commercial interests as [an] M&O contractor." Specifically, CDSI claims that the existence of DynCorp's roles in both the ITSO contract and the SPRO contract creates an impermissible conflicting role for DynCorp under DOE Acquisition Regulation (DEAR) 9.09.570-4(a), which provides: Two questions should generally be asked in determining whether organizational conflicts of interest exist: (1) Are there conflicting roles which might bias a contractor's judgment in relation to its work for the Department? (2) Is the contractor being given an unfair competitive advantage based on the performance of the contract? 48 CFR 909.570-4(a) (emphasis added). Citing the first question CDSI contends that DynCorp's dual role might bias DynCorp's judgment as the ITSO contractor concerning the confidentiality of M&O information. Protester's Posthearing Brief at 92. DEAR 909.570-4(a) continues: The ultimate determination as to whether organizational conflicts of interest exist should be made in the light of common sense and good business judgment based upon the relevant facts and the work to be performed. . . . Department personnel must pay particular attention to proposed contractual requirements which call for the rendering of advice, or consultation or evaluation services or similar activities that lay direct groundwork for the Department's decisions on future acquisitions, research and development programs, production and regulatory activities. Applying a common sense approach, we believe that CDSI has not met its burden of proving there was an improper OCI in this case. DynCorp will not under the ITSO contract be evaluating any work performed under the SPRO contract. The SPRO contract is totally unrelated to the ITSO statement of work, and is physically far removed from ITSO performance. Even the contention that DynCorp will have access to information concerning the SPRO contract is an assumption. Further, the contracting officer was aware that DynCorp's affiliate was the M&O contractor for the SPRO, but he reasonably concluded that any organizational conflict of interest arising due to DynCorp's affiliate's M&O contract could be mitigated after award. In sum, the relationship between DynMcDermott as the SPRO M&O contractor and DynCorp as the ITSO contractor is too remote, and any OCI too speculative, to support a conclusion of violation of statute or regulation. Was DOE Biased Against CDSI? CDSI contends that DOE harbored impermissible bias against CDSI which permeated the conduct of the entire procurement. In reviewing alleged bias, the Board presumes "that Government officials acting in their official capacity do so in good faith." SMS Data Products Group, Inc., GSBCA 10587-P, 90-3 BCA 23,034, at 115,658, 1990 BPD 141, at 9. A protester "must come forward with evidence to overcome the presumption." Id. The evidence which a protester must proffer is quite substantial. "Well-nigh irrefragable proof" is required to induce us to abandon a presumption of good faith dealing. See Schaefer v. United States, 633 F.2d 945, 949 (Ct. Cl. 1980); Trans-Atlantic Industries, Inc., GSBCA 10803, 91-1 BCA 23,412, at 117,458. CDSI cites the following indicia of DOE's bias: the manifold procurement violations; the standardless methodology of evaluation; the fact that the SEB knew that DynCorp's proposal was throughout the entire evaluation process; the SEB's acceptance of DynCorp's representations regarding even though it could not confirm them; the waiver of the minimum of three reference contracts for DynCorp; the of DynCorp's score on criterion 3(b) from to without any change in its proposal; the downgrading of CDSI's proposal for not having features it liked in DynCorp's proposal like the the SEB's discussions with DynCorp about 23 and lack of discussion with CDSI about the contracts; the contracting officer's testimony at hearing that CDSI was being investigated for fraudulent billing; his testimony regarding bait and switch; and the SEB's destruction of all of its notes. CDSI goes on to allege evidence of bias in the cost evaluation, such as the disparate treatment of uncompensated overtime, mischaracterization of the DCAA audits, the failure to advise CDSI about the Wyatt Data Survey, the contracting ____________________ 23 We did not reach this contention given our conclusion that DOE was required to conduct discussions with CDSI regarding the officer's characterization of CDSI as "argumentative," and the failure of the contracting officer to consider CDSI's response to the DCAA audit letter because it was marked draft. Protester's Posthearing Brief at 103-05. Although this litany of allegations, some of which we have accepted, is suggestive of a flawed procurement, it does not constitute "well-nigh irrefragable proof" of bias, and the Board does not conclude that bias tainted the award process. Was there Impermissible Technical Transfusion/Leveling? CDSI claims that DOE provided DynCorp with and that DynCorp obtained source selection information, procurement sensitive information, and proprietary information pertaining to CDSI prior to award. CDSI cites its organizational chart and documents indicating its current subcontractors and task assignments. This argument fails for several reasons. First, there is no proof that DOE provided DynCorp with the organizational chart and related documents. The record suggests only that the documents mysteriously appeared in DynCorp's proposal room. Second, the documents were disseminated to hundreds of CDSI employees, approximately one hundred of whom have left the contract. Third, and most importantly, the documents were not used by DynCorp in this procurement in any way. As DynCorp's administration manager testified, the organization depicted by CDSI was different from that called for by the RFP. Thus, there has been no violation of statute or regulation in the context of this procurement and no prejudice to CDSI in this procurement. Decision The protest is GRANTED IN PART. Specifically, Counts 1, 4, 8, 11, 12, 13, 15, and 16 are DENIED; Counts 2, 3, 5, 6, and 7 are GRANTED; Counts 9 and 10 are GRANTED IN PART. Our suspension of the agency's delegation of procurement authority lapses by its terms. The Board is unable to ascertain whether the awardee's proposal was properly determined to have the best value or not. Since additional discussions must be held, another round of BAFOs must be submitted and evaluated. Because this evaluation should begin on a clean slate, unaffected by this particularly intense and difficult litigation process, we direct the establishment of a new source evaluation board. See Communications Network System, Inc. v. Department of Commerce, GSBCA 12705-P, 1994 BPD 63, at 27 (Feb. 24, 1994); Denro, Inc. v. Department of Transportation, GSBCA 11736-P-R, 1992 BPD 381 (Aug. 31, 1992). The SSO need not be replaced. The new SEB shall establish objective standards for ascertaining strengths, weaknesses, and the import of a designation of a "significant" strength or weakness. Similarly, the SEB shall establish objective baseline-type standards for determining similarity in size and scope for the reference contracts. We strongly recommend, but do not order, that the DOE rethink its policy of permitting wholescale destruction of individual evaluators' notes during the evaluation. This prevents a reviewer such as the SSO from ascertaining whether the consensus ratings bear any relationship to independent, individual evaluations. Preserving the individual evaluators' worksheets would enhance the accountability of the process. Next, the SEB must begin anew to perform its reference checks for CDSI's contracts and DynCorp's and contracts and to rescore criteria 1 and 2. The SEB shall also reanalyze proposals with regard to the use of uncompensated overtime and treat all offerors equally. After reevaluating proposals in that fashion, the SEB shall engage in discussions as necessary, and disclose to all offerors the standard it will use in the evaluation of cost BAFOs. The SEB shall then call for another round of BAFOs and proceed in accordance with statute and regulation. We recognize that the empaneling of a new SEB may cause the agency inconvenience. Nonetheless, after careful consideration, we conclude that this is the only manner in which to ensure that further competition is on an equitable basis. _____________________________ MARY ELLEN COSTER WILLIAMS Board Judge We concur: ____________________________ ____________________________ STEPHEN M. DANIELS ALLAN H. GOODMAN Board Judge Board Judge