OPEN-SOURCE DATA LAKE PLATFORM FOR THE SECRETARIAT DIGITAL HUB UNGSC Chief Procurement Officer

Description This RFI is issued solely for market research and information-gathering purposes. It does not constitute a solicitation for proposals, bids, or quotes, nor does it commit UNGSC to pay for any costs incurred in responding. Responses will not be evaluated for selection or award, and no contractual obligations will arise from this RFI. Information provided will help UNGSC understand available open-source-based data lake platforms and related services in the market and contribute to developing a scope of requirements for a potential future Request for Proposal (RFP). All responses will be treated as confidential where indicated by respondents. Background The Infrastructure and Operations Section of the Service for Geospatial Information and Telecommunication Technology (SGITT), United Nations Global Service Centre (UNGSC), Department of Operational Support, installs, operates and maintains Information and Communications Technology (ICT) systems for the United Nations Secretariat, including the Secretariat Digital Hub. The Secretariat generates and collects growing volumes of structured and unstructured data from databases, applications, files, logs and streaming sources. A data lake platform is required to ingest, store, process, analyse and share this data in a secure environment that remains under the full control of the Organization, in line with its data protection and information security requirements. The Organization has assessed consumption-priced proprietary cloud analytics services, whose capacity-based charges recur every month for the duration of use. It intends instead to deploy a modular platform based on open-source software on UN-managed infrastructure, supplied, integrated and supported by an external contractor, and to retain ownership of the resulting configuration and know-how. UNGSC is issuing this RFI to gather information on open-source-based data lake platforms, and on the design, integration, support and training services offered around them, available in the market. Interested suppliers are invited to provide detailed information on their solution’s components, architecture, security, integration capabilities, implementation approach, licensing and support services. This information will inform UNGSC’s understanding of commercial options and support the development of requirements for the Request for Proposal (RFP) for an open-source data lake platform for the Secretariat Digital Hub. Key areas of interest include • Modular Structure: The requirement is envisaged in three lots: A. Core Platform (design, deployment and integration), B. Object Storage (optional), and C. AI/ML Services (optional), together with support and training services. • Open Source Without Recurring Charges: All platform components under OSI-approved open-source licences, with a software bill of materials, and no paid licence, subscription or capacity fee required to run them. • Deployment on UN-Managed Infrastructure: Containerised deployment on UN-provided servers, storage and network, on a cluster of at least five nodes, with no UN data leaving UN-managed infrastructure. • Security and Governance: Single sign-on, role-based access control, secrets management, security monitoring, encryption, hardening and audit logging, aligned with UN Secretariat information security and data protection requirements. • Ownership and Knowledge Transfer: Scripts, configurations, infrastructure code, pipelines and documentation becoming the property of the Organization, with knowledge transfer to UNGSC IT staff and handover at the end of the contract. • Support and Service Levels: An annual support and maintenance subscription with defined service levels, and the ability to deliver on-site services at both Brindisi and Valencia as well as remotely. Information Requested Suppliers are encouraged to address the following sections in their responses, providing examples, documentation, or references where possible. Responses should focus on how their solution meets or approaches these areas, including any relevant case studies from similar organizations. A. Functional Fit • Data ingestion and streaming: flow-based ingestion with flow versioning, event streaming, and connectors for relational databases, REST/SOAP APIs, files, logs and CDC sources, with data lineage (reference: Apache NiFi, NiFi Registry, Apache Kafka or equivalent). • Storage: distributed file storage and relational database, open file and table formats, with backup and recovery (reference: HDFS, PostgreSQL, Parquet, ORC, Avro, Iceberg or equivalent). • Processing, orchestration and query: batch and stream processing, distributed SQL engine and workflow orchestration (reference: Apache Spark, Trino, Apache Airflow or equivalent). • Analytics and BI: dashboards, multi-user notebooks, SQL exploration and export of results (reference: Apache Superset, JupyterHub, Apache Zeppelin, Hue or equivalent). • Optional modules: object storage (reference: MinIO, Ceph, Apache Ozone or equivalent) and AI/ML services (reference: Ollama, Open WebUI, MLflow, vector databases or equivalent). B. Technical & Integration Capabilities • Architecture and deployment model, including container orchestration on a cluster of at least five nodes (reference: Docker Swarm or equivalent), and the infrastructure sizing required (cores, memory, storage, network). • Gateway and platform operations: reverse proxy, monitoring, log management, source control and CI/CD (reference: Nginx, Grafana, Graylog, OpenSearch, GitLab, Jenkins or equivalent). • Integration with UN identity, monitoring, logging and CI/CD services. • Open-source components used, versions, upstream communities and licences. C. Usability & Adoption • Quality of UX/UI for data engineers, analysts and administrators. • Accessibility and self-service capabilities. • Learning curve, documentation, and training materials for administrators, data engineers and analysts. D. Security & Compliance • Single sign-on (OIDC/SAML), multi-factor authentication and role-based access control (reference: Keycloak or equivalent). • Secrets management and security monitoring (reference: Passbolt, Wazuh or equivalent). • Encryption in transit and at rest, hardening to a recognised benchmark (e.g. CIS) and audit logging. • Vulnerability and supply-chain management for open-source components. • Alignment with UN security policies and standards. E. Implementation & Support • Implementation methodology and indicative timeline, from design and project plan through deployment, pilot data pipelines, testing and acceptance, to documentation and training. • Annual support and maintenance subscription: patching, upgrades, security advisories, incident handling, and proposed service levels and service credits. • Support and training services: professional consultancy, platform administration training (online), on-site training and knowledge transfer, and remote and on-site support. • End-of-contract handover and orderly transition. • Experience and references in the design, deployment and support of production open-source data lake or big-data platforms. F. Cost & Licensing • Open-source licensing and any optional commercial support subscriptions for upstream components. • Pricing structure for each lot, the annual support subscription, and support and training services. • Total cost of ownership (TCO) over a three to five-year period, and flexibility in scaling capacity. Annex A – Market Information Questionnaire provides further questions to support market research on the proposed platform. Respondents are encouraged to address these questions in their PDF response, identifying relevant open-source components, commercial add-ons, alternative approaches and any important limitations. The questions are intended to guide information gathering and are not evaluation or pass/fail criteria. Response Instructions • Format: Submit responses in PDF format, not exceeding 20 pages (excluding appendices). Structure your main response with the following sections: company overview, solution summary, and sections A to F, as listed under Information Requested. Detailed answers to Annex A and supporting technical documentation may be included as appendices. • Content: Include your company’s overview, solution summary, and responses to each of sections A to F. Mark any proprietary information as confidential. • Annex A: Address all questions in Annex A within your PDF response, following the question numbering and providing brief descriptions, relevant explanations or supporting references, as appropriate. A separately completed Annex A is not required. • Open source transparency: Identify the open-source projects and licences on which your solution is based, and provide references (URL or document included in your response) to support your answers. • Submission: Email responses to •••••@•••••.•••Sign up to see this detail by the closing date stated above. Please ensure that your response includes the name of your company, the name and functional title of the designated contact person, their email address and telephone number, and the reference number of this RFI. Late submissions may not be considered. UNGSC appreciates your participation and looks forward to learning about innovative open-source-based data lake platforms in the market. This RFI is issued for market research purposes only, without any immediate procurement intent. Specific Requirements / Information The requirement is envisaged in three lots (A. core platform; B. object storage, optional; C. AI/ML services, optional) plus support and training, delivered at UNGSC Brindisi (Italy) or Valencia (Spain). Respondents are asked to state clearly which components of the proposed platform are open source (naming the upstream project and licence) and which are proprietary or commercial add-ons. ANNEX A – MARKET INFORMATION QUESTIONNAIRE Request for Information on an Open-Source Data Lake Platform for the Secretariat Digital Hub, United Nations Global Service Centre (UNGSC). Instructions to respondents The purpose of this questionnaire is to gather market information to inform a possible future procurement. Sections A to H are classified as Mandatory, while Section I is classified as Desirable. These classifications reflect UNGSC’s priorities and are provided solely to guide respondents; they do not constitute evaluation or disqualification criteria. Respondents should address the questions in their PDF response, following the question numbering and describing available solutions, options, assumptions and limitations where relevant. Supporting documentation, technical references or URLs may be provided where appropriate. UNGSC is seeking solutions based exclusively on open-source software, without software licensing, subscription or capacity-based fees for the use of the platform. Alternative technologies may be proposed, provided they meet these open-source requirements. Respondents should clearly identify the proposed software components and their respective open-source licences. A. Organization & Experience 1. Relevant experience Please briefly describe your organization’s experience in designing, implementing or supporting open-source data lake or big-data solutions. You may provide examples of relevant assignments and the technical expertise available. Response: Brief description / references (if available): 2. Service locations Can your organization provide remote and/or on-site services in Brindisi, Italy, and Valencia, Spain? Please indicate the available delivery arrangements and any limitations. Response: Service coverage / limitations: 3. Open-source licensing Please indicate the licensing approach for the proposed technologies, including any open-source licence conditions, commercial dependencies, subscription charges or capacity-based fees, if applicable. Response: Brief description / relevant licences: 4. Ownership and handover Please describe your standard arrangements for handing over project deliverables, such as scripts, configurations, infrastructure code, data pipelines and documentation, including any intellectual-property or reuse restrictions. Response: Brief description / restrictions (if any): 5. Information security and data protection Can the solution be deployed and operated entirely within UN-managed infrastructure, without transferring UN data to external environments? Please outline relevant security and data protection capabilities and any dependencies. Response: Yes / No / Subject to conditions: Explanation (if applicable): 6. Service levels What support coverage and service-level arrangements can be offered, including incident response, resolution targets, security patching and availability? Response: Brief description / indicative service levels: B. Foundation, Access & Security 7. Container platform Please describe supported deployment and container-orchestration approaches, including clustering, scalability and high availability. Indicate whether a deployment across five or more nodes is supported and whether a different architecture is recommended. Response: Brief description / recommended approach: 8. Gateway Please identify the available gateway or reverse-proxy capabilities, including secure access, TLS termination and routing for user-facing services. Response: Brief description / indicative components: 9. Identity, SSO and access control Please describe support for integration with an enterprise identity provider, single sign-on (such as OIDC or SAML), multi-factor authentication and role-based access control. Response: Brief description / integration considerations: 10. Secrets management and security monitoring Please describe the available approaches to centralised secrets and key management, intrusion detection, file integrity monitoring and vulnerability monitoring. Response: Brief description / indicative components: 11. Encryption, hardening and audit Please outline capabilities for encryption in transit and at rest, system hardening against recognised benchmarks, and logging of user and administrative activities. Response: Brief description / limitations (if any): C. Data Ingestion & Streaming 12. Flow-based ingestion Please describe available tools for visual or flow-based data ingestion, routing and transformation, including version control and promotion between environments. Response: Brief description / indicative components: 13. Event streaming Please indicate whether the solution supports event streaming and describe relevant capabilities, including replication and data-retention controls. Response: Brief description / indicative components: 14. Connectors and lineage Please describe supported connectors and integration methods for relational databases, APIs, files, logs and streaming or change-data-capture sources, together with data provenance and lineage capabilities. Response: Brief description / key supported sources: D. Storage 15. Distributed file storage Please describe the storage options suitable for a scalable data lake, including distributed file storage, redundancy and resilience. Response: Brief description / recommended approach: 16. Relational database Please indicate available relational database options for metadata and operational or analytical needs, including high availability and backup capabilities. Response: Brief description / indicative components: 17. Open formats, backup and recovery Please describe support for open data and table formats, and the proposed approach to backup, restoration and disaster recovery for stateful components. Response: Brief description / supported formats: E. Processing, Orchestration & Query 18. Batch and stream processing Please describe the available batch and stream processing capabilities, including operational monitoring and access to processing history. Response: Brief description / indicative components: 19. SQL query engine Please describe capabilities for distributed SQL queries over data held in the data lake, including relevant interfaces and supported formats. Response: Brief description / indicative components: 20. Orchestration and resource isolation Please outline options for workflow scheduling, retries, alerting, access controls and workload or resource isolation between batch and interactive activities. Response: Brief description / recommended approach: F. Analytics, Exploration & BI 21. Dashboards and self-service BI Please describe the available dashboarding and self-service business intelligence capabilities. Response: Brief description / indicative components: 22. Notebooks Please indicate the availability of shared or multi-user notebook environments, including support for Python and R where available. Response: Brief description / supported environments: 23. SQL exploration and export Please describe available SQL exploration interfaces and options for exporting query results, including common formats such as CSV, Excel and PDF. Response: Brief description / supported formats: G. Platform Operations 24. Monitoring and log management Please outline capabilities for metrics collection, dashboards, alerting, centralised logging and log search across platform components. Response: Brief description / indicative components: 25. Source control and CI/CD Please describe the available integration with source control and automation tools for building, testing and deploying code and configurations. Response: Brief description / indicative components: 26. Lifecycle management Please describe the typical approach to upgrades, patching, compatibility management and rollback of platform components. Response: Brief description / key considerations: H. Implementation, Support & Training 27. Design and project plan Please outline your typical approach to solution architecture, infrastructure sizing, network and security design, project planning and indicative deployment timelines. Where possible, identify key assumptions and dependencies for computing, memory, storage and networking. Response: Brief description / indicative timeline and assumptions: 28. Integration and pilot Please describe how the proposed solution could be integrated with existing enterprise identity, monitoring, logging and CI/CD services, and how a pilot could be structured. Response: Brief description / indicative pilot scope: 29. Testing and acceptance Please outline your typical methods for functional, performance, security and resilience testing and for demonstrating solution readiness. Response: Brief description / testing approach: 30. Documentation, training and knowledge transfer Please indicate the documentation, training and knowledge-transfer services available for administrators and users, including remote and on-site options. Response: Brief description / delivery options: 31. Support subscription and services Please describe available support and maintenance models, advisory and consultancy services, training options, and remote or on-site assistance. Where possible, indicate the main elements that influence indicative costs. Response: Brief description / indicative commercial model: I. Optional Capabilities (Potential Lots B and C) 32. Object storage (potential Lot B) Please indicate whether the solution can incorporate S3-compatible or other scalable object storage and describe how it would integrate with processing and query services. Response: Available: Yes / No / Optional Brief description (if available): 33. Private LLM serving and chat interface (potential Lot C) Please indicate whether locally hosted large language models and a web-based chat interface can be provided within UN-managed infrastructure, including integration with single sign-on and controls to prevent external data transfer. Response: Available: Yes / No / Optional Brief description (if available): 34. Experiment tracking and model registry (potential Lot C) Please indicate the availability of experiment tracking and model registry capabilities and their possible integration with notebooks and storage services. Response: Available: Yes / No / Optional Brief description (if available): 35. Vector database (potential Lot C) Please indicate whether vector storage and similarity search capabilities are available, including possible integration with the broader data platform. Response: Available: Yes / No / Optional Brief description (if available): Instructions for Responding to Annex A Respondents are requested to address the questions above as part of their overall RFI response, submitted in PDF format. Responses should follow the question numbering to facilitate review and cross-referencing. Where a description is requested, please provide a brief explanation of the relevant capabilities, proposed approach or available options. Supporting documentation, technical references or URLs may be included where appropriate. Respondents are encouraged to identify alternative solutions, limitations and other considerations that may assist UNGSC in understanding market offerings. A separately completed copy of this questionnaire is not required.