Data indexing software

SECTION I: Purpose Product, Service or Outcome Needed: The purpose of this procurement is to purchase a solution that will improve the utilization and analysis of log data by automatically indexing all the data, including complex multi-line application logs, enabling searches on all the data without the need for custom connectors, and without limitations inherent in database schemas. It will quickly search and report on this data and interpret the data during searches to provide a more complete context. It will provide seamless integration and tracking of log data from numerous data sources for trending analysis, trending visualization, current and historical data mining and predictive analytics. This solution will also support the creation of actionable intelligence. The product must support at least ten users, have visual interfaces, capable of data mining enormous amounts of data in order to understand relationships, patterns, and trends. The result is a more flexible and complete approach to using and analyzing log data, enabling diagnosis of issues and troubleshooting of security incidents and investigations faster, and providing repeatable and affordable compliance. Scope of the Product, Service, or Outcome: This product will help Focused Operations (FO) perform indexing of all the data, including complex multi-line application logs, enabling searches on all the data without the need for custom connectors, and without limitations inherent in database schemas. It will quickly search and report on this data and interpret the data during searches to provide a more complete context. This product will help the FO team to automate the acquisition of IT log data from multiple sources and perform log, trending analysis, correlation and predictive analysis on current data and historical data through data mining and creating actionable intelligence. Utilizing visual interfaces, this tool will seamlessly data mine large amounts of log data in real-time in order to understand relationships, patterns, and trends. The tool must expeditiously (in minutes, not hours) collect, index and harness 15 to 20GB of uncompressed data per day generated from a complex IT infrastructure of multiple log sources and formats (e.g. SMTP, Blackberry, Web Proxy, Chat). The tool must also be able to easily export log data in a readable format (i.e. MS Office product formats, PDF, MSOutlook etc.). SECTION II: Background The Transportation Security Administration (TSA/Office of Information Technology (OIT)/Information Assurance and Cyber Security Division (IAD)/ Technical Services Section/Focused Operations Team (FO also known as digital forensics) performs cyber investigations on malicious attacks, cyber threats, support for legal and law enforcement cases. The FO team collects multiple tactics, techniques, and procedures (TTPs) in order to perform trending and predictive analysis through data mining of log data from multiple sources. In an effort to expand its forensic and investigative capabilities, the FO team requires a real-time log indexing and analysis tool to be able to perform these functions. FO has custom built a Linux server that automatically collects logs from various sources to support its investigative mission. This custom built solution has proven to be of extreme value during investigations but is no longer supportable. FO requires a supported COTS product to replace the custom built solution. SECTION III: Technical Requirements/Tasks/Outcomes Software: TSA has a need for software that is designed to assist in performing the analysis of log data by automatically indexing all the data, quickly searching and reporting on this data, and interpret the data during searches to provide a more complete context. It will provide seamless integration and tracking of log data from numerous data sources for trending analysis, trending visualization, current and historical data mining and predictive analytics. 1) Index logs in any format (structured, unstructured, and disparate) from any data source (including system data) in real time. 2) Ability to create apps that help visualize data geographically or that provide pre-canned compliance views; apps for different technologies such as Windows, Linux, Unix, virtualization, networking and more. 3) Ability to save and schedule searches or monitor and alert on specific events, patterns and thresholds. 4) Collect, use, search and analyze 15 to 20 GB of uncompressed log data per day in real time from any source to one central location daily and setup custom reports and dashboards that summarize log activity. 5) Automatically index all data, including complex multi-line application logs, enabling the search on all data without the need for custom connectors and database schema limitations. 6) Ability to manipulate, customize and report log data in common Microsoft Office product formats (i.e. MSWord, PDF, MSExcel, MSOutlook, system application files etc.) 7) Support and install on Windows operating systems, Mac OS, Linux, and FreeBSD 8) Multiple user access - support at least 12 users. SECTION IV: Submission instructions Submissions shall not exceed 5 pages, single sided, in length. Submissions should include detailed information that communicates the products(s) ability to meet the requirements described in the Section III of this RFI.