Languages
This tutorial will explain how to set the language property for various nodes and file objects when using the ricecooker framework.
Explore language objects and language codes
First we must import the le-utils pacakge. The languages supported by Kolibri and the Content Curation Server are provided in le_utils.constants.languages.
[1]:
from le_utils.constants import languages
# can lookup language using language code
language_obj = languages.getlang('en')
language_obj
[1]:
Language(native_name='English', primary_code='en', subcode=None, name='English', text_direction='ltr')
[2]:
# can lookup language using language name (the new le_utils version has not shipped yet)
language_obj = languages.getlang_by_name('English')
language_obj
[2]:
Language(native_name='English', primary_code='en', subcode=None, name='English', text_direction='ltr')
[3]:
# all `language` attributed (channel, nodes, and files) need to use language code
language_obj.code
[3]:
'en'
[4]:
from le_utils.constants.languages import getlang_by_native_name
lang_obj = getlang_by_native_name('français')
print(lang_obj)
print(lang_obj.code)
Language(native_name='Français, langue française', primary_code='fr', subcode=None, name='French', text_direction='ltr')
fr
The above language code is an internal representaiton that uses two-letter codes, and sometimes has locale information, e.g., pt-BR for Brazilian Portuiguese. Sometimes the internal code representaiton for a language is the three-letter vesion, e.g., zul for Zulu.
[ ]:
Create chef class
We now create subclass of ricecooker.chefs.SushiChef and defined its get_channel and construct_channel methods.
For the purpose of this example, we’ll create three topic nodes in different languages that contain one document in each.
[5]:
from ricecooker.chefs import SushiChef
from ricecooker.classes.nodes import ChannelNode, TopicNode, ContentNode
from le_utils.constants import licenses
from le_utils.constants.languages import getlang
class MultipleLanguagesChef(SushiChef):
"""
A sushi chef that creates a channel with content in EN, FR, and SP.
"""
channel_info = {
'CHANNEL_TITLE': 'Languages test channel',
'CHANNEL_SOURCE_DOMAIN': 'ricecooker-docs-example.org', # where you got the content
'CHANNEL_SOURCE_ID': '<unique id for channel>', # channel's unique id CHANGE ME!!
'CHANNEL_LANGUAGE': getlang('mul').code, # set global language for channel
'CHANNEL_DESCRIPTION': 'This channel contains nodes in multiple languages',
'CHANNEL_THUMBNAIL': None, # (optional)
}
def construct_channel(self, **kwargs):
# create channel
channel = self.get_channel(**kwargs)
# create the English topic, add a ContentNode to it
topic = TopicNode(
source_id="<en_topic_id>",
title="New Topic in English",
language=getlang('en').code,
)
doc_node = ContentNode(
source_id="<en_doc_id>",
title='Some doc in English',
description='This is a sample document node in English',
uri='samplefiles/documents/doc_EN.pdf',
license=licenses.PUBLIC_DOMAIN,
language=getlang('en').code,
)
topic.add_child(doc_node)
channel.add_child(topic)
# create the Spanish topic, add a ContentNode to it
topic = TopicNode(
source_id="<es_topic_id>",
title="Topic in Spanish",
language=getlang('es-MX').code,
)
doc_node = ContentNode(
source_id="<es_doc_id>",
title='Some doc in Spanish',
description='This is a sample document node in Spanish',
uri='samplefiles/documents/doc_ES.pdf',
license=licenses.PUBLIC_DOMAIN,
language=getlang('es-MX').code,
)
topic.add_child(doc_node)
channel.add_child(topic)
# create the French topic, add a ContentNode to it
topic = TopicNode(
source_id="<fr_topic_id>",
title="Topic in French",
language=languages.getlang('fr').code,
)
doc_node = ContentNode(
source_id="<fr_doc_id>",
title='Some doc in French',
description='This is a sample document node in French',
uri='samplefiles/documents/doc_FR.pdf',
license=licenses.PUBLIC_DOMAIN,
language=getlang('fr').code,
)
topic.add_child(doc_node)
channel.add_child(topic)
return channel
Unable to load pyppeteer, using phantomjs for JS loading.
Run of you chef by creating an instance of the chef class and calling it’s run method:
[6]:
mychef = MultipleLanguagesChef()
args = {
'command': 'dryrun', # use 'uploadchannel' for real run
'verbose': True,
'token': 'YOURTOKENHERE9139139f3a23232'
}
options = {}
mychef.run(args, options)
INFO In SushiChef.run method. args={'command': 'dryrun', 'verbose': True, 'token': 'YOURTO...'} options={}
INFO
***** Starting channel build process *****
INFO Calling construct_channel...
INFO Setting up initial channel structure...
INFO Validating channel structure...
INFO Languages test channel (ChannelNode) (<unique id for channel>): 6 descendants
INFO New Topic in English (TopicNode) (<en_topic_id>): 1 descendant
INFO Some doc in English (ContentNode): uri: samplefiles/documents/doc_EN.pdf
INFO Topic in Spanish (TopicNode) (<es_topic_id>): 1 descendant
INFO Some doc in Spanish (ContentNode): uri: samplefiles/documents/doc_ES.pdf
INFO Topic in French (TopicNode) (<fr_topic_id>): 1 descendant
INFO Some doc in French (ContentNode): uri: samplefiles/documents/doc_FR.pdf
INFO Tree is valid
INFO
INFO Downloading files...
INFO Processing content...
INFO All files were successfully downloaded
INFO Command is dryrun so we are not uploading channel.
Congratulations, you put three languages on the internet!
[ ]:
Example 2: YouTube video with subtitles in multiple languages
You can use the library youtube_dl to get lots of useful metadata about videos and playlists, including the which language subtitle are vailable for a video.
[7]:
import yt_dlp as youtube_dl
ydl = youtube_dl.YoutubeDL({
#'quiet': True,
'no_warnings': True,
'writesubtitles': True,
'allsubtitles': True,
})
youtube_id = 'FN12ty5ztAs'
info = ydl.extract_info(youtube_id, download=False)
subtitle_languages = info["subtitles"].keys()
print(subtitle_languages)
[youtube] Extracting URL: FN12ty5ztAs
[youtube] FN12ty5ztAs: Downloading webpage
[youtube] FN12ty5ztAs: Downloading android vr player API JSON
[info] FN12ty5ztAs: Downloading subtitles: en-5IebwaT_cAk, fr-5IebwaT_cAk, zu-5IebwaT_cAk
dict_keys(['en-5IebwaT_cAk', 'fr-5IebwaT_cAk', 'zu-5IebwaT_cAk'])
[ ]:
Full sushi chef example
The YoutubeVideoWithSubtitlesSushiChef class below shows how to create a channel with youtube video and upload subtitles files with all available languages.
[8]:
from ricecooker.chefs import SushiChef
from ricecooker.classes import licenses
from ricecooker.classes.nodes import ChannelNode, TopicNode, ContentNode
from ricecooker.classes.files import YouTubeSubtitleFile
from ricecooker.utils.youtube import is_youtube_subtitle_file_supported_language
import yt_dlp as youtube_dl
ydl = youtube_dl.YoutubeDL({
'quiet': True,
'no_warnings': True,
'writesubtitles': True,
'allsubtitles': True,
})
# Define the license object with necessary info
TE_LICENSE = licenses.SpecialPermissionsLicense(
description='Permission granted by Touchable Earth to distribute through Kolibri.',
copyright_holder='Touchable Earth Foundation (New Zealand)'
)
class YoutubeVideoWithSubtitlesSushiChef(SushiChef):
"""
A sushi chef that creates a channel with content in EN, FR, and SP.
"""
channel_info = {
'CHANNEL_SOURCE_DOMAIN': 'ricecooker-docs-example.org', # where you got the content
'CHANNEL_SOURCE_ID': '<unique id for channel>', # channel's unique id CHANGE ME!!
'CHANNEL_TITLE': 'Youtube subtitles downloading chef',
'CHANNEL_LANGUAGE': 'en',
'CHANNEL_THUMBNAIL': 'https://edoc.coe.int/4115/postcard-47-flags.jpg',
'CHANNEL_DESCRIPTION': 'This is a test channel to make sure youtube subtitle languages lookup works'
}
def construct_channel(self, **kwargs):
# create channel
channel = self.get_channel(**kwargs)
# get all subtitles available for a sample video
youtube_id ='FN12ty5ztAs'
info = ydl.extract_info(youtube_id, download=False)
subtitle_languages = info["subtitles"].keys()
print('Found subtitle_languages = ', subtitle_languages)
# create video node - the pipeline infers the video kind from the youtube.com URL
video_node = ContentNode(
source_id=youtube_id,
title='Youtube video',
license=TE_LICENSE,
derive_thumbnail=True,
uri='https://www.youtube.com/watch?v={}'.format(youtube_id),
)
# add subtitles in whichever languages are available.
# uri alone can't express subtitles - add them explicitly, language is required
for lang_code in subtitle_languages:
if is_youtube_subtitle_file_supported_language(lang_code):
video_node.add_file(
YouTubeSubtitleFile(
youtube_id=youtube_id,
language=lang_code
)
)
else:
print('Unsupported subtitle language code:', lang_code)
channel.add_child(video_node)
return channel
[9]:
chef = YoutubeVideoWithSubtitlesSushiChef()
args = {
'command': 'dryrun', # use 'uploadchannel' for real run
'verbose': True,
'token': 'YOURTOKENHERE9139139f3a23232'
}
options = {}
chef.run(args, options)
INFO In SushiChef.run method. args={'command': 'dryrun', 'verbose': True, 'token': 'YOURTO...'} options={}
INFO
***** Starting channel build process *****
INFO Calling construct_channel...
INFO Setting up initial channel structure...
INFO Validating channel structure...
INFO Youtube subtitles downloading chef (ChannelNode) (<unique id for channel>): 1 descendant
INFO Youtube video (ContentNode): 3 files
INFO Tree is valid
INFO
INFO Downloading files...
INFO Processing content...
INFO Initiating DOWNLOAD for https://edoc.coe.int/4115/postcard-47-flags.jpg with kwargs {'default_ext': 'png'}
INFO Initiating DOWNLOAD for https://www.youtube.com/watch?v=FN12ty5ztAs with kwargs {'yt_dlp_settings': {'format': 'bestvideo[height<=480][ext=mp4]+bestaudio[ext=m4a]/bestvideo[height<=480][ext=webm]+bestaudio[ext=webm]/best[height<=480][ext=mp4]'}}
Found subtitle_languages = dict_keys(['en-5IebwaT_cAk', 'fr-5IebwaT_cAk', 'zu-5IebwaT_cAk'])
[youtube] Extracting URL: https://www.youtube.com/watch?v=FN12ty5ztAs
[youtube] FN12ty5ztAs: Downloading webpage
INFO Completed DOWNLOAD for https://edoc.coe.int/4115/postcard-47-flags.jpg with kwargs {'default_ext': 'png'} saved to docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg
INFO Initiating CONVERT for docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg
INFO Completed CONVERT for docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg saved to docs/examples/storage/e/c/ec609ff23af81c9ffc2eb23103ee1d6e.jpg
WARNING: [youtube] No supported JavaScript runtime could be found. Only deno is enabled by default; to use another runtime add --js-runtimes RUNTIME[:PATH] to your command/config. YouTube extraction without a JS runtime has been deprecated, and some formats may be missing. See https://github.com/yt-dlp/yt-dlp/wiki/EJS for details on installing one
[youtube] FN12ty5ztAs: Downloading android vr player API JSON
[info] FN12ty5ztAs: Downloading 1 format(s): 134+140
[download] Destination: /tmp/bd09424388b8b8b966a8afa144edec7d.f134.mp4
[download] 100% of 3.81MiB in 00:00:00 at 3.88MiB/s
[download] Destination: /tmp/bd09424388b8b8b966a8afa144edec7d.f140.m4a
[download] 100% of 941.23KiB in 00:00:00 at 1.97MiB/s
[Merger] Merging formats into "/tmp/bd09424388b8b8b966a8afa144edec7d.mp4"
Deleting original file /tmp/bd09424388b8b8b966a8afa144edec7d.f140.m4a (pass -k to keep)
Deleting original file /tmp/bd09424388b8b8b966a8afa144edec7d.f134.mp4 (pass -k to keep)
INFO Completed DOWNLOAD for https://www.youtube.com/watch?v=FN12ty5ztAs with kwargs {'yt_dlp_settings': {'format': 'bestvideo[height<=480][ext=mp4]+bestaudio[ext=m4a]/bestvideo[height<=480][ext=webm]+bestaudio[ext=webm]/best[height<=480][ext=mp4]', 'outtmpl': {'default': '/tmp/bd09424388b8b8b966a8afa144edec7d.mp4', 'chapter': '%(title)s - %(section_number)03d %(section_title)s [%(id)s].%(ext)s'}, 'js_runtimes': {'deno': {}}, 'remote_components': set(), 'compat_opts': set(), 'http_headers': {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36', 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8', 'Accept-Language': 'en-us,en;q=0.5', 'Sec-Fetch-Mode': 'navigate'}, 'forceprint': {}, 'print_to_file': {}}} saved to docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4
INFO Initiating CONVERT for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 with kwargs {'ffmpeg_settings': {}}
INFO Completed CONVERT for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 with kwargs {'ffmpeg_settings': {}} saved to docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4
INFO Initiating EXTRACT_METADATA for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4
INFO Video preset from docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 = low resolution
INFO Completed EXTRACT_METADATA for docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4 saved to docs/examples/storage/7/4/74b08dc1997c69bbb2a111559a3299cc.mp4
INFO All files were successfully downloaded
INFO Command is dryrun so we are not uploading channel.
[ ]:
[ ]:
[ ]:
[ ]: