Skip to content

Social media collections: Twitter

Introduction

Twitter is a social media site focused publicly sharing short messages.

A Twitter gathering template is included as part for Funnelback's social media collections support to allow content from Twitter to be gathered and then presented within Funnelback search results.

Please note that your usage of Funnelback to gather content from Twitter must comply with Twitter's terms of service.

Setting up collection

To create a Twitter collection, you will need to create a social media collections and select the Twitter template.

Once you have created the collection you will need to fill out the collection.cfg file. To do this go to the Administer tab of the admin home page and then click on the Browse Collection Configuration Files link.

Getting authentication keys and secrets

Before you can crawl Twitter, ensure that you have:

Once complete, note your OAuth consumer key/consumer secret and OAuth access token / token secret.

Configuration options

Twitter's gathering template will read the configuration from collection.cfg. The following settings are supported:

  • twitter.oauth.consumer-key: OAuth consumer key.
  • twitter.oauth.consumer-secret: OAuth consumer secret.
  • twitter.oauth.access-token: OAuth access token.
  • twitter.oauth.token-secret: OAuth token secret.
  • twitter.users: Comma delimited list of user names to crawl.
  • twitter.debug: Boolean flag to enable debug mode. When debug mode is enabled the gathering script will print out the crawled records in XML form.

Example

twitter.oauth.consumer-key=...
twitter.oauth.consumer-secret=...
twitter.oauth.access-token=...
twitter.oauth.token-secret=...
twitter.users=funnelback
twitter.debug=false

Metadata mappings

The Twitter gathering template includes a number of Twitter specific metadata mappings:

Class IDTypeBehaviourExplanationMetadata fields included
authortextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/screenName
authorImagetextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/profileImageUrl
ctextcontentTweet/com.funnelback.socialmedia.twitter.TwitterXmlRecord/tweet
countrytextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/country
ddatedateDate/com.funnelback.socialmedia.twitter.TwitterXmlRecord/createdDate
hashtagtextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/hashtags/Hashtag/text
identifiertextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/id
imagetextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedMediaURLs/MediaURL/thumbnail/pictureUrl
isReTweettextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/isReTweet
latLonggeospatial x/y co-ordinateN/A/com.funnelback.socialmedia.twitter.TwitterXmlRecord/latLong
linkedDisplayUrltextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedURLs/URL/displayURL
linkedExpandedUrltextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedURLs/URL/expandedURL
linkedShortUrltextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedURLs/URL/shortURL
locationtextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/placeName
usernametextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/username

Limits

Please note that Twitter applies limits to the volume of content which can be retrieved from their APIs, and so in the case of large Twitter streams Funnelback may be unable to gather all historical content.

See also

top

Funnelback logo
v15.14.0