Skip to content

Twitter collections

Introduction

Twitter is a social media site focused publicly sharing short messages.

Please note that your usage of Funnelback to gather content from Twitter must comply with Twitter's terms of service.

Getting authentication keys and secrets

Before you can crawl Twitter, ensure that you have:

Once complete, note your OAuth consumer key/consumer secret and OAuth access token / token secret.

Configuration options

With the details above, you can create a new Twitter collection from the 'Create collection' menu.

The following settings are supported as part of the collection setup.

  • twitter.oauth.consumer-key: OAuth consumer key.
  • twitter.oauth.consumer-secret: OAuth consumer secret.
  • twitter.oauth.access-token: OAuth access token.
  • twitter.oauth.token-secret: OAuth token secret.
  • twitter.users: Comma delimited list of user names to crawl.

Metadata mappings

Twitter collections include a number of default Twitter specific metadata mappings:

Class IDTypeBehaviourExplanationMetadata fields included
authortextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/screenName
authorImagetextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/profileImageUrl
ctextcontentTweet/com.funnelback.socialmedia.twitter.TwitterXmlRecord/tweet
countrytextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/country
ddatedateDate/com.funnelback.socialmedia.twitter.TwitterXmlRecord/createdDate
hashtagtextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/hashtags/Hashtag/text
identifiertextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/id
imagetextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedMediaURLs/MediaURL/thumbnail/pictureUrl
isReTweettextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/isReTweet
latLonggeospatial x/y co-ordinateN/A/com.funnelback.socialmedia.twitter.TwitterXmlRecord/latLong
linkedDisplayUrltextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedURLs/URL/displayURL
linkedExpandedUrltextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedURLs/URL/expandedURL
linkedShortUrltextdisplay/com.funnelback.socialmedia.twitter.TwitterXmlRecord/linkedURLs/URL/shortURL
locationtextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/placeName
usernametextcontent/com.funnelback.socialmedia.twitter.TwitterXmlRecord/username

Use the -SF query processor option to access these metadata fields on the search response and in the templates (i.e. `-SF=[author,hashtag]).

Limits

Please note that Twitter applies limits to the volume of content which can be retrieved from their APIs, and so in the case of large Twitter streams Funnelback may be unable to gather all historical content.

Working with the fetched data

Funnelback will crawl Twitter and convert responses into XML. You can use the metadata customisation tool to map elements to a metadata class.

Note: To preview the crawled records please enable debug mode by setting twitter.debug=true in collection.cfg file.

The XML that Funnelback generates for a Twitter collection is as follows:

<com.funnelback.socialmedia.twitter.TwitterXmlRecord>
  <id>tweet_id</id>
  <username>username</username>
  <screenName>some username</screenName>
  <profileImageUrl/>
  <tweet>tweet content</tweet>
  <createdDate>2018-06-20 14:58:03.0 UTC</createdDate>
  <url>https://twitter.com/user_name/status/tweet_id</url>
  <hashtags>
    <Hashtag>
      <start>110</start>
      <end>119</end>
      <text>hashtag conetnt</text>
    </Hashtag>
  </hashtags>
  <linkedURLs>
    <URL>
      <start>133</start>
      <end>156</end>
      <shortUrl>https://t.co/qwert</shortUrl>
      <expandedURL>http://bit.ly/qwert</expandedURL>
      <displayURL>bit.ly/qwert</displayURL>
    </URL>
  </linkedURLs>
  <isReTweet>false</isReTweet>
  <linkedMediaURLs>
    <MediaURL>
      <baseUrl>http://pbs.twimg.com/media/qwert.jpg</baseUrl>
      <thumbnail>
        <pictureUrl>http://pbs.twimg.com/media/qwert.jpg:thumb</pictureUrl>
        <width>150</width>
        <height>150</height>
        <resizeMethod>CROP</resizeMethod>
      </thumbnail>
      <small>
        <pictureUrl>http://pbs.twimg.com/media/qwert.jpg:small</pictureUrl>
        <width>430</width>
        <height>430</height>
        <resizeMethod>FIT</resizeMethod>
      </small>
      <medium>
        <pictureUrl>http://pbs.twimg.com/media/qwert.jpg:medium</pictureUrl>
        <width>430</width>
        <height>430</height>
        <resizeMethod>FIT</resizeMethod>
      </medium>
      <large>
        <pictureUrl>http://pbs.twimg.com/media/qwert.jpg:large</pictureUrl>
        <width>430</width>
        <height>430</height>
        <resizeMethod>FIT</resizeMethod>
      </large>
    </MediaURL>
  </linkedMediaURLs>
</com.funnelback.socialmedia.twitter.TwitterXmlRecord>

top

Funnelback logo
v15.16.0