Content Matters

Twitter Buys Gnip; What Are the Repercussions?

Twitter is buying social media firehose Gnip, it's largest distributor and among the largest social data aggregators.

On the surface, this could make sense. Twitter has been looking to monetize its data, and simply being a wholesaler doesn't let that business scale. As Rob Passarella notes, the deal is a clear sign that Twitter is focused on its data business.

But the acquisition raises a few interesting questions.

Data aggregation platforms are typically most successful when they are independent. Aggregators thrive because they can be "Switzerland", neutral and unaffiliated with any providers. It's what makes acquisition of aggregators difficult. Once they are acquired, they lose that independence. I've seen this happen many times in the financial data business.

So, the two questions this deal raises in my mind are:

1. What will the other Gnip data partners do, now that Twitter has acquired them? Beyond Twitter, Gnip partners include Facebook, Google+, FourSquare, Tumblr, YouTube, Vimeo, Flickr, Instagram, Disqus, StockTwits, Estimize and more.

While many of these partners may be content to allow their data to flow through Twitter-owned Gnip, I can see some (Facebook/Instagram, Google/YouTube) looking to exit the service.

2. What will this mean for DataSift? Gnip competitor DataSift is the other leading provider of the Twitter firehose, along with Facebook, Tumblr, G+ and many others. Will Twitter still sell its firehose via DataSift? Will Gnip gain strategic advantages due to its new parent? And, will Google or Facebook to jump in with an offer to buy DataSift in response to Twitter's acquisition of Gnip?

 

 

 

 

 

 

 

 

The Four Types of Data Scientists

If “Big Data” was the hot buzzword of 2012, it’s pretty apparent that “Data Scientist” has now become the hot position to recruit for in 2013.

A search on Indeed shows 9,158 current openings for Data Scientist coming from employers like Netflix, The New York Times, Roche and others.  In fact, Indeed itself is looking for a data scientist to focus on personalization efforts for its jobseeker database. Defense contractor Booz Allen alone has 30 current positions for data scientists, though none specify whether they’re part of the PRISM program. 

So, what is a data scientist and what do they do?

That question is the focus of a new report from O’Reilly Strata called Analyzing the Analyzers. The report (free with registration) clusters data scientists into four groups, based on a survey of data scientists, focused on skills and career experiences. While there were some commonalities across those four categories, they were actually quite different from one another in many ways. The four categories the paper identifies are: 

Data Businesspeople: while they have strong technical skills, data businesspeople are focused on using data to drive profits within an organization. They tend to be more senior and have an entrepreneurial focus.

Data Creatives tend to have substantial academic experience and excel at machine learning, big data and programming skills. Avid users of open source, Data Creatives tend to have broad-based skills, and can move from role to role more easily.

Data Developers tend to focus on the technical issues involved in managing data. They tend to be coders with strong programming and machine learning skills, with less of a focus on business or statistics.

Data Researchers typically come from the academic world and have deep backgrounds in statistics or the physical or social sciences. More than the other groups identified, Data Researchers frequently hold a PhD (more than half of those in the survey) and tend to have weaker sills in machine learning, programming or business.

The survey is fascinating and it's easy to think of people I know who clearly fit into one of these four buckets. So, if you’re thinking of recruiting a data scientist, before you start the process, think about what skills you really need. Is it a statistician, a coder, a technical business lead or a jack-of-all-trades?

The full paper is available from the O’Reilly Strata site. 

PRISM or How I Learned To Stop Worrying And Love Information Awareness

In the last 24 hours, everyone has suddenly become an expert on privacy and national security, weighing in on the NSA's PRISM program.

Personally, I have mixed feelings about the program. While we have government assurances that the program is limited in scope, does not target US citizens, and is under close supervision, I'm not sure that I'm completely comforted by that. Rules governing use are hard to enforce, particularly when cloaked in complete secrecy. So, I think that any program like this can be easily misused, which clearly concerns me.

At the same time, on a practical level, do we not expect our intelligence community to use information and technology in their efforts to keep us safe? Whether it's the Boston bombing or Benghazi, there's a fairly strong outcry asking "how come our intelligence efforts couldn't prevent this"?

Following 9/11, I worked for a company that provided semantic technology used to identify potential relationships between entities. While they were not our primary market, the Intelligence Community quickly saw value in what we did, and our product was part of the Poindexter-led Total Information Awareness ("TIA") project. TIA was clearly a bit in-your-face (the all-seeing eye of God was a bit much, not to mention hiring John Poindexter to oversee the effort) and the program was officially killed in 2003 when Congress refused to fund it. But the program never died. Our software licenses were simply shifted to a different project, with different funding, but where the same analysts were the users.

Technology has come a long way in ten years. Entity extraction, data mining and analytics are commonplace tools in the business environment. It shouldn't shock anyone to see those tools are used by our government agencies, particularly the one tasked with "signal intelligence". 

Accessing voice, email and social media data takes that to another level. And I think it's good that we're having this public debate. Personally, I am very conflicted about this issue. But many of those now voicing the loudest criticism will be among those asking "how did we not know about this" when the next incident occurs.

Data Publishing With Content Curation and Aggregation

The traditional boundaries of publishing have vanished. 

Data publishers now create editorial, traditional "content" publishers want to publish data, and brands are hiring journalists to better engage their customers. At the same time, most publishers have stumbled when it comes to content marketing and social media.

For b2b publishers, this emerging world can be challenging. Yet, there are some simple ways for publishers to embrace aggregation and curation to drive engagement and to improve their content marketing efforts.

I'll be speaking on curation and aggregation this Thursday at the Specialized Information Publishers Assocation (SIPA) "Pathways to Publishing Profit" conference in DC. Here's a preview of some of what I'll be exploring.

The value of curation continues to increase as a means of filtering content for an audience. Curation is not new, of course. B2B publishers have long served as a filter for information. Whether you need to track legislative activity on a topic or market-moving news about companies, b2b publishers have filled that need. What's changed is that automation can be used in conjunction with human editorial to provide that filtering at a much greater scale. And, leveraging the right tools, there's no need for a large editorial staff to filter the content. Curation can be done by small teams or a single person.

A good example is Jason Hirschhorn's Media ReDEFined. Delivered via an enewsletter or via social media, Media ReDEF is a must-read for anyone in the publishing or digital media space. Jason is a one-man shop. Using RSS readers, he scans an estimated 3,000 - 4,000 stories per day and selects the 30-40 that he feels most important. And there are lots of ways to read Media ReDEF. I follow MediaReDEF on Twitter, but prefer to read it in Flipboard, as it's a great platform for news consumption. But you can read Media ReDEF via email, Facebook, RSS or whatever method you prefer.

 

Working at a slightly larger scale is Alacra's new Dealmeme, currently in beta. Alacra uses technology to aggregate thousands of news feeds, with probably 100,000 or more stories a day. Using a mix of semantic technology and human editorial, they identify M&A rumors, deal ideas and deal news, then display the information in a Techmeme-like interface. Dealmeme leverages a number of advanced technologies, but operationally, the team is a fraction of the size of what it would take to do the same thing using just manual editorial.

Effectively curated content can be used in various ways. At its simplest, it can be used to attract a following and engage them on a daily basis. Jason Hirschhorn's Media ReDEF has more than 7,000 Twitter followers. And most of those followers are big fans of the free service. Most b2b publishers have struggled to gain a foothold in social media. By providing a stream of focused content each day, you'll quickly grow that following.

Leveraging technology for aggregation and filtering, combined with editorial expertise to pick the right stories and topics, publishers can today deliver a focused set of information to its audience, without the need for a massive editorial team.

We'll cover much more at the SIPA conference. Hope to see you there.

 

I Moved

I’m thrilled to share that I’ve just joined Connotate as Vice President, Product Strategy. 

Connotate is the leader in the web content extraction space. Publishers, ecommerce retailers, financial institutions, government agencies and others rely upon Connotate to aggregate, organize, analyze and distribute web data at scale.

Connotate agents can mine content from websites, extracting critical pieces of data, then structure, aggregate and organize that information for use in analytical applications. Clients use Connotate to monitor prices on tens of thousands of items, run background checks on potential new hires, and aggregate data to update information services.

In many ways, Connotate is the perfect home for me. I’ve spent much of the past fifteen years helping publishers and media companies create content-driven products. Extracting insights from the web has been a big part of that - from my early days at semantic pioneer ClearForest to harvesting and tagging news and social media content at Alacra. 

While many technology companies have latched onto the big data theme, Connotate has been focused on extracting data at scale since the start. Whether you’re automating background checks, analyzing the sentiment of customer comments or pulling pricing data from competitor websites, Connotate can deliver highly accurate results across a wide range of disparate sources.

Connotate goes way beyond simple web scraping. Today’s complex websites, built with AJAX, Javascript and HTML5 don’t allow for easy content harvesting. The Connotate technology leverages feature-based machine learning with heuristics to do intelligent extraction. And the system is highly resilient. The feature-based approach allows Connotate extraction algorithms to continue to perform accurately, even when website designs change. 

Last year, Connotate acquired Fetch, its largest direct competitor, and is currently positioned for strong growth. There are a number of interesting market opportunities which we’ll be exploring in the coming weeks and months. If you have web data challenges, I’d love to get your thoughts on areas where we might focus. You can reach me at bgraubart-at-connotate-dot-com.

You can read the formal announcement of my joining Connotate here.