Sentiment Analysis Software Product Evaluation for Social Media Monitoring.

About the Client

Company N provides content and tools that allow companies to better understand their customers and make faster, better-informed decisions, and its media monitoring products combine research tools, quality content, text-mining methods, and visualization capabilities to help companies understand their products’ performance and competitive position across both traditional and social media. Iknow has worked with Company N since 2010 across a range of technology consulting and systems integration engagements, including prior work on the company’s metadata strategy and semantic technology initiatives, giving Iknow deep familiarity with Company N’s content processing environment heading into this evaluation.

Industry Context

By 2013, the sentiment analysis and social media monitoring market was young and highly fragmented, with more than 50 vendors — including firms such as Attensity, Clarabridge, Crimson Hexagon, Lexalytics, and NetBase — competing with meaningfully different approaches to sentiment scoring, data source coverage, and performance characteristics, and no single dominant, clearly superior product. For a media monitoring business like Company N’s, the accuracy and speed of sentiment scoring directly determined the credibility of the insights it could sell to its own customers, making the choice of platform a real product and reputational risk. Because sentiment analysis output can vary widely across text samples and “sentiment” itself is genuinely subjective, evaluating vendors credibly required an objective, human-validated benchmark rather than relying on vendor-reported accuracy claims alone.

Current Situation

Until this project, Company N had used a legacy application to analyze and report on the sentiments expressed in news and social media. The purpose of this project was to modernize this capability and add new analytic functionality, and the project started with a review of Company N’s requirements to determine the functionality that would best meet its customers’ needs.

• An outdated legacy sentiment analysis capability. Company N’s legacy sentiment analysis application no longer met the analytic functionality its media monitoring business and customers needed.
• A large, fragmented, immature vendor market. The sentiment analysis software market was large and fragmented, with more than 50 vendors and no clear, obviously superior product.
• No independent basis for validating vendor accuracy claims. Vendor-reported accuracy claims could not be trusted without independent, objective validation against real news and social media content.
• Accuracy alone was not a sufficient evaluation criterion. Company N needed to know not just which product performed best in terms of accuracy, but also whether it could meet real-world latency requirements at scale.

• Review Company N’s requirements to determine the functionality that would best meet customer needs.
• Evaluate the commercial sentiment analysis software market and narrow a large vendor field to a manageable set of finalists.
• Rigorously and objectively test finalist products against real news and social media content.
• Deliver a vendor-neutral recommendation and implementation roadmap.

How Iknow Structured the Work

Iknow structured the engagement as a funnel, moving from requirements definition through a broad market scan of more than 50 products to rigorous, lab-based testing of 10 finalists — reflecting Iknow’s vendor and platform selection methodology for turning an overwhelming, fragmented vendor field into a defensible, evidence-based recommendation.

Key Activities & Decisions

• Requirements review and market scan. Iknow reviewed Company N’s requirements to determine the functionality that would best meet its customers’ needs, then evaluated more than 50 sentiment analysis software products on the market, selecting 10 for more exhaustive testing.
• Lab environment and vendor presentations. Iknow installed the selected software packages in a Sentiment Analysis Lab environment specially built for the analysis, and invited each of the chosen software vendors to present their product’s benefits to an evaluation panel.
• Control group and sentiment dictionary. Iknow developed a control group of articles and social media objects, manually reviewing the samples and assigning a sentiment score to each object, and prepared a Company N dictionary of sentiment-bearing words for use in the lab environment.
• Accuracy and performance testing. Iknow ran each software package through three sets of social media and news artifacts with varying degrees of sentiment expressed about different types of entities, and conducted performance testing on each product to determine whether it would meet Company N’s latency requirements.
• Results synthesis and recommendations. Iknow compared the results of the analysis, presented the quantitative results to Company N, and prepared a recommendations report and an implementation roadmap.

Stakeholders & Collaboration

Iknow served as prime contractor, working with Company N’s media monitoring business and an evaluation panel that reviewed vendor presentations and testing results throughout the engagement.

Objectively Comparing Products on an Inherently Subjective Measure

Comparing more than 50 products, then narrowing to 10 finalists, on something as inherently subjective as sentiment risked producing results that simply reflected each vendor’s marketing claims. Iknow addressed this by building a manually scored control group of real articles and social media objects, along with a custom Company N sentiment dictionary, giving every product a consistent, human-validated benchmark to measure against.

Balancing Accuracy With Real-World Performance Requirements

A highly accurate product that could not process content fast enough would not work for a media monitoring business operating at scale. Iknow addressed this by running dedicated performance and latency testing on each finalist product alongside its sentiment accuracy testing, rather than evaluating accuracy in isolation.

Quantitative Outcomes

• Initial market scan: More than 50 sentiment analysis software products initially evaluated.
• Finalists tested: 10 finalist products selected for exhaustive lab testing.
• Testing infrastructure delivered: A custom-built Sentiment Analysis Lab environment created for the engagement.
• Testing scope: 3 sets of social media and news artifacts used for accuracy testing, spanning varying sentiment and entity types.
• Engagement duration: Two-month assignment.

Qualitative Outcomes

The value of this project to Company N included an unbiased quantitative assessment of 10 sentiment analysis software products, a sentiment analysis lab environment for continued evaluation of social media monitoring software, and a set of metrics for comparing the efficacy of sentiment analysis software. It also included recommendations on which products best met Company N’s requirements, as well as an implementation and integration roadmap. Company N came away not just with a single vendor recommendation but with a durable evaluation capability — the lab environment and metrics — it could reuse for future technology decisions.

Timeline to Impact

Within the two-month engagement, Iknow moved Company N from a field of more than 50 vendors, through rigorous, objective lab testing, to a vendor-neutral recommendation and an implementation roadmap ready to modernize its media-monitoring sentiment capability.

Core Skills

• Vendor-neutral software evaluation
• Sentiment analysis and text analytics assessment
• Testing lab design and benchmark development
• Performance and latency testing

Methods & Frameworks

• Requirements-driven vendor narrowing
• Human-validated control group benchmarking
• Combined accuracy and performance testing methodology

Technologies & Tools

• Custom Sentiment Analysis Lab environment and sentiment-word dictionary
• Commercial sentiment analysis and social media monitoring software