YouTube collections
Introduction
YouTube is a social media site focused on video sharing.
Please note that your usage of Funnelback to gather content from YouTube must comply with YouTube's terms of service.
Getting API key
To be able to gather content from YouTube you must first use your Google account to create an API key. Google provides instructions on creating an API key which will provide the most up-to-date instructions for creating the key.
Configuration options
YouTube collections support the following settings:
- youtube.api-key: API key retrieved from the Google API console.
- youtube.channel-ids: Comma delimited list of YouTube channel IDs to crawl.
- youtube.liked-videos: Boolean flag to enable fetching of videos liked by a channel ID. It'll apply to all channels provided in
youtube.channel-ids
. - youtube.playlist-ids: Comma delimited list of YouTube playlist IDs to crawl.
Metadata mappings
YouTube collections includes a number of default YouTube specific metadata mappings:
Class ID | Type | Behaviour | Explanation | Metadata fields included |
---|---|---|---|---|
author |
text | content | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/channelTitle |
|
authorUrl |
text | display | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/channelUrl |
|
c |
text | content | Description | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/description |
category |
text | content | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/category |
|
commentCount |
number | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/commentCount |
|
d |
date | date | Date | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/publishedDate |
dislikeCount |
number | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/dislikes |
|
duration |
text | display | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/durationPretty |
|
durationInSeconds |
number | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/durationInSeconds |
|
favoriteCount |
number | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/favoriteCount |
|
identifier |
text | display | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/videoId |
|
imageSmall |
text | display | Image URL - small | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/thumbnails/veryDefault/thumbNailUrl |
imageMedium |
text | display | Image URL - medium | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/thumbnails/medium/thumbNailUrl |
imageLarge |
text | display | Image URL - large | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/thumbnails/high/thumbNailUrl |
latLong |
geospatial x/y co-ordinate | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/latLong |
|
likeCount |
number | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/likes |
|
t |
text | content | Title | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/title |
viewCount |
number | N/A | /com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord/viewCount |
Use the -SF
query processor option to access these metadata fields on the
search response and in the templates (i.e. -SF=[author,viewCount]
).
Limits
Please note that YouTube applies limits to the volume of content which can be retrieved from their APIs, and so in the case of large channels Funnelback may be unable to gather all historical content.
Working with the fetched data
Funnelback will crawl YouTube and convert responses into XML. You can use the metadata customisation tool to map elements to metadata classes.
Note: To preview the crawled records please enable debug mode by setting youtube.debug=true
in collection.cfg
file.
The XML that Funnelback generates for a YouTube collection is as follows:
<com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord>
<url>http://www.youtube.com/watch?v=video_id&feature=youtube_gdata_player</url>
<title>some title</title>
<description>some description</description>
<videoId>video_id</videoId>
<viewCount>1</viewCount>
<commentCount>0</commentCount>
<thumbnails>
<veryDefault>
<thumbNailUrl>https://i.ytimg.com/vi/video_id/default.jpg</thumbNailUrl>
<thumbNailUrlWidth>120</thumbNailUrlWidth>
<thumbNailUrlHeight>90</thumbNailUrlHeight>
<thumbNailUrlExtras class="linked-list"/>
</veryDefault>
<medium>
<thumbNailUrl>https://i.ytimg.com/vi/video_id/mqdefault.jpg</thumbNailUrl>
<thumbNailUrlWidth>320</thumbNailUrlWidth>
<thumbNailUrlHeight>180</thumbNailUrlHeight>
<thumbNailUrlExtras class="linked-list"/>
</medium>
<high>
<thumbNailUrl>https://i.ytimg.com/vi/video_id/hqdefault.jpg</thumbNailUrl>
<thumbNailUrlWidth>480</thumbNailUrlWidth>
<thumbNailUrlHeight>360</thumbNailUrlHeight>
<thumbNailUrlExtras class="linked-list"/>
</high>
<stndard>
<thumbNailUrl>https://i.ytimg.com/vi/video_id/sddefault.jpg</thumbNailUrl>
<thumbNailUrlWidth>640</thumbNailUrlWidth>
<thumbNailUrlHeight>480</thumbNailUrlHeight>
<thumbNailUrlExtras class="linked-list"/>
</stndard>
<maxres>
<thumbNailUrl>https://i.ytimg.com/vi/video_id/maxresdefault.jpg</thumbNailUrl>
<thumbNailUrlWidth>1280</thumbNailUrlWidth>
<thumbNailUrlHeight>720</thumbNailUrlHeight>
<thumbNailUrlExtras class="linked-list"/>
</maxres>
</thumbnails>
<publishedDate>2016-08-16T23:14:04.000Z</publishedDate>
<category/>
<durationInSeconds>378</durationInSeconds>
<durationPretty>00:06:18</durationPretty>
<likes>1</likes>
<dislikes>0</dislikes>
<favoriteCount>0</favoriteCount>
<channelTitle>channel_name</channelTitle>
<channeld>channel_id</channeld>
<channelUrl>https://www.youtube.com/channel/channel_id</channelUrl>
<embedHtml><iframe width="480" height="270" src="//www.youtube.com/embed/video_id" frameborder="0" allow="autoplay; encrypted-media" allowfullscreen></iframe></embedHtml>
<extras class="linked-list"/>
</com.funnelback.socialmedia.youtube.v3.YouTubeXmlRecord>