Independent Evaluation of the Kosovo Reliable Energy Landscape Project (RELP)

95332419Q0065 Amendment 0001 Kosovo Reliable Energy Landscape Project (RELP) Evaluation 1. The estimated budget for the project is $1,920,000 over the five year project. Is the $310k data plug in addition to or included in the overall estimate? Answer - The data collection plug of $310k is included in the estimated budget of $1,920,000. 2. Can the evaluation contractor suggest a different spread of the data collection plug between option years? Answer - Yes. The timing of data collection at this stage is notional and will be heavily dependent on the implementation timelines of the RELP project. The evaluator is expected to work closely with MCC and the Implementing Contractor(s) to identify the data needs not being fulfilled by the Implementing Contractor(s). The evaluation contractor can modify the proposed timing of data collection in the Evaluation Design Report. 3. C.1.4 notes that "While the implementer is in charge of baseline and subsequent data collection, the evaluator will serve as a critical data quality check on the baseline data collection and consequently will use the baseline data to inform their analysis". ? Can MCC confirm that the implementer will also collect endline data? ? Does MCC envision that the evaluation contractor will serve as a data quality check on all data collected by the implementer, including data that may not be of use to the evaluation? ? Can MCC specify how many rounds of data collection for which the evaluator will be responsible for data quality? Answer - - The Implementing Contractor(s) will also collect end line data for the AER and HER sub-activities within the PIEE Activity. See section C.3 - Evaluation Questions and Guidance on Methodology - Expected Data Sources for details on expected data collection by the Implementing Contractor by sub-activity. - No, not all data. The Evaluator will serve as a data quality check on beneficiary and control group data collected by the Implementing Contractor for use in the evaluation. The expectation is that the data needs for the evaluation will be larger than the data needs for the Implementing Contractor. Depending on the timing of this award, the evaluator may need to prioritize review of the Implementing Contractor's baseline survey strategy and instruments as the very first task (see Section Task 1 in Section C.5). - See section C.3 - Evaluation Questions and Guidance on Methodology - Expected Data Sources for details on expected data collection by the Implementing Contractor by sub-activity. The Evaluator will be responsible for serving as data quality check on the specified data collection activities by the Implementing Contractor(s). 4. C.1.4 notes that the implementer will employ an iterative design framework for the HER intervention package which is based on the principles of behavior economics. Can MCC confirm that the evaluator will have no role in the design or analysis of this? If not, what role does MCC envisage? Answer - The Evaluator is expected to assess the quality and effectiveness of the HER design iteration process, so should not expect to influence that design. The Evaluator will not have any role in the design of the intervention to ensure independence of the evaluation. However, the Evaluator is expected to play a quality control role in the measurement of the outcomes of this iterative activity. The PIEE/HER sub-activity is the only iterative sub-activity in this Project. Given the remaining timeline of the Threshold Project, MCC expects no more than two rounds of program iteration requiring three rounds of data collection by the Implementing Contractor (baseline, interim after first iteration, and endline after second iteration). The Evaluator is expected to ensure that the Implementing Contractor's survey strategy and instruments are of high quality to ensure that evaluation questions will be answerable. (See section C.3 - Evaluation Questions and Guidance on Methodology - Expected Data Sources for details on expected data collection by the Implementing Contractor by sub-activity. See Task 1 in Section C.5 for details about data quality check and oversight). 5. C.5 references interim and endline data collection. Should the reference be to baseline and endline? Or does MCC envision a baseline, interim and endline? Answer - The data collection referred to in Section C.5 includes primary data collection by the evaluator (likely qualitative but TBD per Evaluation Design Report), which will be distinct from and in addition to the quantitative data collected by the Implementing Contractor. In case of qualitative data, the expectation is that the evaluator will conduct rolling data collection based on a thoughtful evaluation design. MCC expects two broad rounds of data collection by the evaluator. We refer here to "interim" and endline data collection because we do not expect pre-implementation primary data collection by the evaluator (therefore it will not be baseline) based on the timing of this award. It may be possible to establish baselines for certain outcomes using recall data during the first round of "interim" data collection. 6. We note that the implementing contractor collects data but the evaluation contractor is to obtain IRB approval (C.5 Task 4). Will the implementer also obtain IRB approval? Will the implementer be bound by the evaluator's IRB? Answer - The Implementing Contractor(s) are expected to obtain their own IRB approval. The evaluator will also obtain their own IRB approval for primary data collection, which will be distinct from and in addition to the quantitative data collected by the Implementing Contractor. The Implementing Contractor will not be bound by the evaluator's IRB. 7. Can MCC provide further information on the financial allocation by sub-activity (AER, HER, and WEE) within PIEE? Answer - The total financial allocation for PIEE is between $13.5 - $14.5 million. The total financial allocation for DHM is between $8 - $9 million. The final breakdown of funds for the PIEE and DHM sub-activities will be determined by the respective Implementing Contractors. 8. C.1.4 (p. 15): This section states that "In order to implement iterative study for HER, some randomization will be required to select households for interventions as well as comparison group (control or counterfactual), and baseline data will need to be collected before implantation of the EE retrofits occurs." Please clarify evaluator's role in this randomization exercise. Answer - The Evaluator will be required to conduct a technical quality check on the randomization and overall sampling strategy. See first bullet of Task 1 in Section C.5. 9. C.5 (p.22): Letter e denotes that the task of the independent evaluator is to "manage the data collection firms". However, in the preceding section it was noted that the implementing contractor holds responsibility for collecting baseline, interim, and final rounds of data. Please clarify whether the evaluator should manage the data collection firm used by the IC for their data collection, or whether the data collection firm managed by the evaluator refers to firms that evaluator will use for the qualitative data collection. Answer - Letter e refers to management of the data collection conducted directly by the evaluator. Should the evaluator choose to subcontract this data collection to an outside firm, then the evaluator will be responsible for managing that firm. The data collection referred to in Section C.5 letter e includes primary data collection by the evaluator (likely qualitative but TBD per Evaluation Design Report), which will be distinct from and in addition to the quantitative data collected by the Implementing Contractor. Note that the Implementing Contractor will not collect all data that the evaluator will need to answer the evaluation questions posed in this RFQ (see section C.3 - Evaluation Questions and Guidance on Methodology - Expected Data Sources for details on expected data collection by the Implementing Contractor by sub-activity). 10. L.8.1 (p.57): The RFP states "Appendices to the Technical Quotation shall be submitted as a separate electronic file." Please confirm if offerors are to submit one separate appendix document for CVs, letters of commitment, and CPARS; or if each item (CVs, letters of commitment and CPARS) should be in separate appendices. Answer - The Appendices, including CVs, letters of commitment, and CPARS, shall be included in one appendix file document. 11. L.9.6 (p.58): Please confirm if the key personnel qualifications chart and the whole team qualification factors chart are excluded from the 30-page limit of the technical volume. Answer - The key personnel qualifications chart and the whole team qualification factors chart are included in the 30-page limit of the technical volume. The note included in L.8.1 discusses the items that are excluded from the 30-page limit for the technical quotation. 12. L.9.7 (p.60): If we include past performance projects for consultants contracted by the prime contractor, does that count as a reference for the prime? Answer - No. A consultant's past performance reference does not count as a reference for the prime. The note under L.9.7 indicates "At least two of the three references must be for the prime contractor; one of the three may be for a significant subcontractor/ partner." 13. C.3.4. poses a research question, "Did the District Heating Metering (DHM) Activity result in a change in energy consumption in the intervention areas? What was the contribution of various components of the project towards any changes in energy consumption (technical assistance to key institutions, behavior change and outreach activities, installation of energy efficient equipment, and implementation of consumption-based billing)? How many new households shifted to District Heating?" Earlier, C.3 at the top of page 20 notes, "Activity 1.2 DHM: Household level quantitative data collection (baseline and endline) to be conducted by the IC (beneficiary and control households). Evaluator to collect primary rigorous qualitative data. Evaluator to include administrative data in outcome analysis." The research question implies a pre-post comparison while the activity description implies a comparison. Could MCC clarify its proposed approach to the evaluation of this activity? Answer - MCC expects the presence of a comparison (counterfactual) group as well as the presence of baseline and endline data in preparation for a difference-in-difference analysis to answer the evaluation question. The evaluator is expected to use the data collected by the Implementing Contractor (household level quantitative baseline and endline data collection for beneficiary and control households), as well as administrative data from the utility to answer the first two sub-questions: Did the District Heating Metering (DHM) Activity result in a change in energy consumption in the intervention areas? What was the contribution of various components of the project towards any changes in energy consumption (technical assistance to key institutions, behavior change and outreach activities, installation of energy efficient equipment, and implementation of consumption-based billing)? The last question will require use of utility administrative data (How many new households shifted to District Heating?). (See section C.3 - Evaluation Questions and Guidance on Methodology - Expected Data Sources for details on expected data collection by the Implementing Contractor by sub-activity.) 14. Given the 4th of July holiday, is MCC amenable to extending the submission deadline to July 12, 2019? Answer - The quotation due date has been extended to July 10, 2019 at 12:00pm Eastern Time. 15. Is MCC able to provide timeline estimates of the RELP activity workplans? In developing the evaluation design, it would be helpful for Offerors to better understand the sequencing and duration of sub-activity implementation schedules. Answer - As of June 2019, the selection of both RELP PIEE and DHM Implementing Contractors is in the procurement phase. The hope is to have the Evaluator come on board concurrently with both Implementing Contractors to ensure maximum coordination. 16. Will each of the three RELP activities be implemented by a different Contractor? Answer - Yes. Expectation is to have three Implementing Contractors: one each for PIEE, DHM, and IPP. 17. Can MCC confirm whether each RELP activity implementer will have an in-country office? Answer - These details are unknown since the selection of both PIEE and DHM Implementing Contractors is in the procurement phase. The expectation would be yes. 18. On pg. 24 under Task 6: Disseminate Interim Package, MCC notes that "The Evaluator may be asked to participate in other MCC-financed dissemination and training events, such as MCC M&E College and MCC Evaluation Summits." Can MCC please clarify whether the Offeror should budget for MCC-financed events and whether the Offeror will be required to prepare a separate deliverable such as a research paper or policy note? Answer - The Offeror should not expect such events to require significant level of effort. In terms of deliverables, the Offeror would only be asked to present materials already produced for the evaluation (e.g. results dissemination presentations). Participation in such activities will be discussed with MCC ahead of time with consideration of the entire work portfolio and the decision will be based on the availability of funds remaining in the contract. As such, the Offeror should not explicitly budget for this. ====================================================== Kosovo Reliable Energy Landscape Project (RELP) Independent Evaluation Full and Open Competition (F&OC) 95332419Q0065 This is a combined synopsis/solicitation for commercial items prepared in accordance with the format in FAR Subpart 12.6 and conducted per the procedures of FAR Subpart 13.5, as supplemented with additional information included in this notice. The Millennium Challenge Corporation (MCC) is a U.S. Government corporation whose mission is to provide assistance that will support economic growth and poverty reduction in carefully selected countries that demonstrate a commitment to just and democratic governance economic freedom, and investments in their citizenry. For more information on the MCC, please visit www.mcc.gov. This announcement constitutes the only solicitation; quotes are being requested. This solicitation is not set-aside and is being conducted as a full and open competition. The solicitation number is 95332419Q0065. This solicitation is issued as a Request for Quotations (RFQ) and is attached in full to this announcement. The scope of this contract is to provide independent evaluation services in support of the Reliable Energy Landscape Project. The evaluation will cover the Pilot Incentives for Household Investment in Energy Efficiency (PIEE) activity and sub-activities, the District Heating Metering (DHM) activity, and the Independent Power Producer (IPP) activity. The evaluation of RELP seeks to understand (a) the degree to which planned activities are implemented, (b) the validity of the program logics including its assumptions and outcome/results, (c) lessons learned from project implementation and (d) the sustainability of those results. The design and implementation of the evaluation will be closely coordinated with the Millennium Foundation of Kosovo (MFK) (Note: MFK website: https://millenniumkosovo.org/), the accountable entity in Kosovo responsible for management of the Threshold Program. More information on the Kosovo Threshold Program and a copy of the Threshold agreement are available at https://www.mcc.gov/where-we-work/program/kosovo-threshold-program. The following evaluation criteria are hereby included in paragraph (a) of provision 52.212-2 Evaluation - Commercial Items. A. Technical Capability a. Sub-Factor 1: Staffing b. Sub-Factor 2: Technical Approach/Methodology B. Past Performance C. Cost Offers must be submitted via email to Mr. David F. Fuller at contact@example.gov with a copy to Ms. Jacqueline H. Naranjo at contact@example.gov no later than 12:00pm ET on Wednesday, July 03, 2019. Questions regarding the solicitation must be submitted via email to Mr. Fuller with a copy to Ms. Naranjo no later than 12:00pm ET on Monday, June 17, 2019. Please ensure that the solicitation number "95332419Q0065" is included in the subject line of your e-mail response. Failure to do so may cause a quotation not to be considered.