Standing Up the Largest Search Engine Deployment of Its Kind.

About the Client

Company E launched its search engine simultaneously in the United States and China in December 2004.

Industry Context

By the mid-2000s, the web search market was dominated by Google and Yahoo!, but a wave of well-funded challengers was attempting to differentiate through specialized relevance technology, vertical search across news and business content, or strategic international partnerships. Norway-based FAST Search & Transfer had emerged as a leading enterprise and web-scale search platform vendor, later acquired by Microsoft in 2008, and deploying its technology at genuine web scale was a formidable undertaking in infrastructure and domain expertise that few startups could execute with internal teams alone. That gap created real demand for specialist implementation partners who combined deep platform expertise with the technical program discipline needed to hit an aggressive public launch date.

Current Situation

Because Company E was a startup, it lacked the in-house expertise to launch its search engine offering at the speed and scale needed to ensure the new venture’s commercial viability. It needed substantial DataSearch implementation experience, deep search domain expertise, and experienced technical project management. Iknow was asked to take a leadership role on the project because of its specific experience deploying technology infrastructures, its solid understanding of FAST’s products, its skills in optimizing search engine performance, and its experience establishing technical project management offices.

• A launch at genuinely unprecedented technical scale. Company E had selected FAST’s DataSearch platform for a launch at unprecedented scale — seven billion URLs discovered, two billion documents crawled, one billion indexed, and 450 servers — the largest FAST implementation ever attempted.
• No in-house expertise for a launch of this size. As a two-year-old startup, Company E lacked the substantial DataSearch implementation experience, search domain expertise, and technical project management needed to execute at that scale.
• Rapid growth straining organizational stability. The company’s rapid growth created organizational and hiring pressures running in parallel with an already demanding technical build.
• No margin for error against entrenched competitors. Differentiating from entrenched leaders Google and Yahoo! left little room for delay or technical shortfalls in a costly, high-visibility public launch.

• Deploy FAST Search & Transfer’s DataSearch 4.1 at the scale required for Company E’s public launch, on time and within budget.
• Establish technical project management leadership across the architecture, indexing, and operations workstreams.
• Provide deep DataSearch platform and search domain expertise throughout the engagement.
• Support Company E’s organizational growth through recruiting assistance and IT organizational design.

How Iknow Structured the Work

Iknow structured the project into several parallel workstreams — architecture (logical, physical, and data center build-out), the Web index, the news index, the business index, and operations — and assisted the company across all of them, providing resources for roles ranging from technical project management to deep search domain expertise.

Key Activities & Decisions

• Implementation approach and milestone planning. Iknow developed the project implementation approach, including phases, major milestones, and critical path identification.
• Project, resource, and risk planning. Iknow developed detailed project plans and resource plans, and performed risk analysis and mitigation planning across the engagement.
• Requirements, specifications, and roadmap scheduling. Iknow documented the business requirements and product specifications for each index, and developed product roadmap implementation schedules for the phased introduction of new features.
• Data center architecture and platform expertise. Iknow developed the data center architecture modeling and procurement schedule, and provided DataSearch expertise and search domain knowledge throughout.
• Crawler seeding strategy. Iknow developed crawler seeding approaches to support the Web, news, and business indexes at the scale Company E’s launch required.
• Recruiting and organizational support. Iknow assisted the company with its recruiting and selection processes, helping Company E build the internal IT organization needed to sustain the platform going forward.

Stakeholders & Collaboration

Iknow served as prime contractor, taking a leadership role directly with Company E’s technical and executive teams across every workstream of the engagement, from architecture and indexing through operations and organizational design.

Executing the Largest FAST Implementation Ever Attempted

Discovering seven billion URLs, crawling two billion documents, indexing one billion, and deploying 450 servers represented a scale of DataSearch implementation no one had attempted before, leaving little precedent to rely on. Iknow addressed this by structuring the engagement into dedicated architecture, indexing, and operations workstreams, backed by rigorous project and resource planning and risk mitigation, so the unprecedented scale was broken into workstreams the team could actually execute and sequence with confidence.

Maintaining Stability Through a Period of Rapid Company Growth

Building a launch-ready search platform while Company E’s organization was simultaneously scaling created real risk that technical decisions and hiring would fall out of step with each other. Iknow addressed this by pairing its technical project management leadership with direct support for recruiting and IT organizational design, so the internal team growing around the platform was being built deliberately rather than reactively.

Quantitative Outcomes

• Search index scale: Seven billion URLs discovered, two billion documents crawled, and one billion documents indexed.
• Infrastructure scale: 450 servers deployed, making this the largest FAST DataSearch implementation ever attempted.
• Program scope: Five parallel workstreams delivered: architecture, Web index, news index, business index, and operations.
• Engagement duration: Six-month assignment, culminating in launch on time and within budget.

Qualitative Outcomes

Company E successfully deployed FAST’s DataSearch search engine on time and within budget, and Iknow contributed to the program's successful launch across many dimensions. Beyond the technical build, Iknow helped with Company E’s roadmap for introducing new features and with the personnel recruiting and organizational design needed to support the company’s rapidly growing IT organization — serving, in the company’s own words, as a stabilizing influence during a period of rapid growth. The launch helped position Company E for the visibility it achieved shortly thereafter, including recognition on Time magazine’s 50 Coolest Websites of 2006 list.

Timeline to Impact

Iknow delivered the full multi-workstream program within the six-month engagement, and Company E launched its public search engine on time and within budget at the conclusion of the project.

Core Skills

• Search engine implementation and optimization
• Technical program and PMO leadership
• Enterprise-scale infrastructure deployment
• Search domain and platform expertise

Methods & Frameworks

• Multi-workstream program management
• Risk analysis and mitigation planning
• Product roadmap and requirements documentation

Technologies & Tools

• FAST Search & Transfer DataSearch 4.1
• Web-scale crawling and indexing infrastructure