2017年3月30日 星期四

webvtt format

https://dev.opera.com/articles/zh-cn/an-introduction-to-webvtt-and-track/

https://developer.mozilla.org/zh-TW/docs/Web/API/Web_Video_Text_Tracks_Format


時間標籤
  • 時間標籤中的時間,必須比該節點的開始時間晚、比同節點先前之其他時間標籤晚,並且早於該節點的結束時間。 當前文字(active text) 代表此時間標籤到下個時間標籤(或文字內容的結尾)之間的文字,其他同節點中,比 當前文字 早的文字都稱為 先前文字(previous text) ,而晚於 當前文字 的便稱為 未來文字(future text) 。如此便可做出卡拉OK字幕效果
Example 12 - Karaoke style text
1
00:16.500 --> 00:18.500
When the moon <00:17.500>hits your eye

1
00:00:18.500 --> 00:00:20.500
Like a <00:19.000>big-a <00:19.500>pizza <00:20.000>pie

1
00:00:20.500 --> 00:00:21.500
That's <00:00:21.000>amore


When this specification says that a user agent is to collect a WebVTT timestamp, the user agent must run the following steps:
  1. Let input and position be the same variables as those of the same name in the algorithm that invoked these steps.
  2. Let most significant units be minutes.
  3. If position is past the end of input, return an error and abort these steps.
  4. If the character indicated by position is not an ASCII digit, then return an error and abort these steps.
  5. Collect a sequence of characters that are ASCII digits, and let string be the collected substring.
  6. Interpret string as a base-ten integer. Let value1 be that integer.
  7. If string is not exactly two characters in length, or if value1 is greater than 59, let most significant units be hours.
  8. If position is beyond the end of input or if the character at position is not a U+003A COLON character (:), then return an error and abort these steps. Otherwise, move position forwards one character.
  9. Collect a sequence of characters that are ASCII digits, and let string be the collected substring.
  10. If string is not exactly two characters in length, return an error and abort these steps.
  11. Interpret string as a base-ten integer. Let value2 be that integer.
  12. If most significant units is hours, or if position is not beyond the end of input and the character at position is a U+003A COLON character (:), run these substeps:
    1. If position is beyond the end of input or if the character at position is not a U+003A COLON character (:), then return an error and abort these steps. Otherwise, move position forwards one character.
    2. Collect a sequence of characters that are ASCII digits, and let string be the collected substring.
    3. If string is not exactly two characters in length, return an error and abort these steps.
    4. Interpret string as a base-ten integer. Let value3 be that integer.
    Otherwise (if most significant units is not hours, and either position is beyond the end of input, or the character at position is not a U+003A COLON character (:)), let value3 have the value of value2, then value2have the value of value1, then let value1 equal zero.
  13. If position is beyond the end of input or if the character at position is not a U+002E FULL STOP character (.), then return an error and abort these steps. Otherwise, move position forwards one character.
  14. Collect a sequence of characters that are ASCII digits, and let string be the collected substring.
  15. If string is not exactly three characters in length, return an error and abort these steps.
  16. Interpret string as a base-ten integer. Let value4 be that integer.
  17. If value2 is greater than 59 or if value3 is greater than 59, return an error and abort these steps.
  18. Let result be value1×60×60 + value2×60 + value3 + value4∕1000.
  19. Return result.






https://www.w3.org/community/texttracks/2013/09/11/carriage-of-webvtt-and-ttml-in-mp4-files/


By default, the line is set to auto.




A line
The line defines positioning of the cue box.
The line offsets the cue box from the top, the right or left of the video viewport as defined by the writing direction, the snap-to-lines flag, or the lines occupied by any other showing tracks.
The line is set either as a number of lines, a percentage of the video viewport height or width, or as the special value auto, which means the offset is to depend on the other showing tracks.
By default, the line is set to auto.
If the writing direction is horizontal, then the line percentages are relative to the height of the video, otherwise to the width of the video.
WebVTT cue has a computed line whose value is that returned by the following algorithm, which is defined in terms of the other aspects of the cue:
  1. If the line is numeric, the WebVTT cue snap-to-lines flag of the WebVTT cue is false, and the line is negative or greater than 100, then return 100 and abort these steps.
    Although the WebVTT parser will not set the line to a number outside the range 0..100 and also set the WebVTT cue snap-to-lines flag to false, this can happen when using the DOM API’s snapToLines and line attributes.
  2. If the line is numeric, return the value of the WebVTT cue line and abort these steps. (Either the WebVTT cue snap-to-lines flag is true, so any value, not just those in the range 0..100, is valid, or the value is in the range 0..100 and is thus valid regardless of the value of that flag.)
  3. If the WebVTT cue snap-to-lines flag of the WebVTT cue is false, return the value 100 and abort these steps. (The line is the special value auto.)
  4. Let cue be the WebVTT cue.
  5. If cue is not in a list of cues of a text track, or if that text track is not in the list of text tracks of a media element, return −1 and abort these steps.
  6. Let track be the text track whose list of cues the cue is in.
  7. Let n be the number of text tracks whose text track mode is showing and that are in the media element’s list of text tracks before track.
  8. Increment n by one.
  9. Negate n.
  10. Return n.






A line alignment
An alignment for the cue box’s line, one of:
Start alignment
The cue box’s top side (for horizontal cues), left side (for vertical growing right), or right side (for vertical growing left) is aligned at the line.
Center alignment
The cue box is centered at the line.
End alignment
The cue box’s bottom side (for horizontal cues), right side (for vertical growing right), or left side (for vertical growing left) is aligned at the line.
By default, the line alignment is set to start.
The line alignment is separate from the text alignment — right-to-left vs. left-to-right cue text does not affect the line alignment.

2017年1月25日 星期三

2017年1月10日 星期二

Potential race condition issue of GStreamer


It is the issue report & proposed patch description for GStreamer; as the link below.

https://bugzilla.gnome.org/show_bug.cgi?id=777206




fig 1: download loop thread













fig 2: update loop thread



fig 3: base source thread

[Test URL & settings]:

[Issue]: 

Wrong codec settings leads to mosaic (ex, use the codec setting of high bit-rate representation to decode data of low bit-rate representation ).


[Root cause]: 

1. Race condition between threads.
2. Slow start of gst_mpd_client_setup_streaming() when updating manifest.


[Background]

Generally we have three kinds of threads within adaptive demuxer for a live (type = dynamical) streaming; they are listed below.
1. download thread: 
The flow controller to update fragment info, wait until the target fragment is available, create base source thread to download, wait for completeness, check EOS. 
2. update loop thread:
Update manifest according "minimumUpdatePeriod".
3. base source thread:
Actually download bit-stream of each fragment (segment).


[Detailed description]: 

Please refer to fig 4, an issued case is illustrated to understand where the problem arises.

Originally the representation is of 250000 bits/sec. Then the update loop thread locks the manifest_lock and by default launches from slow start (the lowest bit-stream) upon the updated manifest. It makes cur_representation to be of 125000 bits/sec.

After update done, manifest_lock is unlocked.
Since the download loop thread is still waiting for the signal from src thread to inform the completeness of download, the next thread which will get manifest_lock is src thread. 

At _src_event when src thread has completed download, gst_adaptive_demux_eos_handling() is executed with manifest_lock locked.

At B.4 gst_adaptive_demux_stream_advance_fragment_unlocked() is executed & the next download bit-rate is set to 125000 bits/sec. At B.7 gst_adaptive_demux_stream_select_bitrate() (in fact, gst_dash_demux_stream_select_bitrate ()) the check of "if (new_index != active_stream->representation_idx)" is false. It is because that the slow start of update manifest has changed active_stream->representation_idx to the lowest one. As the result, the new caps as well as the Boolean variable need_header will NOT be set. It makes the switch of bit-rate without re-passing necessary codec data.

Finally, it leads to mosaic by applying wrong codec data (of 250000 bits/sec) to decode 125000 bits/sec. As at C1 where the next URL is composed of the lowest bit-rate = 125000 but we do NOT pass codec data & header down for this bit-rate switch .







fig 4: an issued case


[Proposed solution]:

As fig 1 & fig 5, to download next fragment, at first download loop thread will update fragment info (by gst_adaptive_demux_stream_update_fragment_info()). It results in the update to 
stream->fragment of the type GstAdaptiveDemuxStreamFragment which includes the URI info.

Once GstAdaptiveDemuxStreamFragment has been updated, it will NOT be changed. To take use of this fact, we keep the bit-rate we have downloaded previously and compare it to current target. If they are different, we pass the info of header & caps down.

To avoid passing redundant header & caps, we only do the check if 
1. It is a live (type = dynamic) streaming.
2. If the "stream->need_header == FALSE" is TRUE.

Finally as figure 6, within gst_dash_demux_stream_update_fragment_info(), the bit-rate of current chosen representation could be known from: dashstream->active_stream->cur_representation->bandwidth.



 

Fig 5: Relationship of GstAdaptiveDemuxStream, GstActiveStream & GstRepresentationNode.

















Fig 6: How could we get bit-rate of current target fragment within gst_dash_demux_stream_update_fragment_info()



[1st Review]:

as reviewer's kindly suggestion:
This should be done not only when the bitrate changed but whenever the stream (representation) was changed

Also, according to ISO 23009-1:
specifies an identifier for this Representation. The identifier shall be unique within a Period unless the Representation is functionally identically to another Representation in the same Period.
s

2016年12月22日 星期四

Improve the performance of far seek for EXOPlayer

ExoPlayer

Issue report = https://github.com/google/ExoPlayer/issues/2253

Check in:

Merged #2318.


Environment:


1.     [Movie]: 4K timer MPEG DASH streaming from https://www.youtube.com/watch?v=uo9dAIQR3g8.
2.     [Device]: HTC One X9u with Android 6.0 API = 23.
3.     [Code Base]: Branch = release-v2


Issue description:


Here we check the side of upstream and see if we could do optimize where.



fig 1: overview of the player pipeline

Fig 1 is the overview of the player pipeline. Please refer to http://programmingmemojohnchang.blogspot.tw/2016/12/improve-performance-of-short-seek-of.html.


fig 2

As figure 2, the rectangle with a dashed border is the latest media chunk we download. It has not been completely downloaded yet so we represent it by dashed border.

Originally if the seek target is behind sampleQueue at the upstream side, we will clean up the whole sampleQueue. Then do refetch from the media chunk which contains the search target.
The refetch media chunk is exactly the same as the last media chunk in dashed border.

Obviously, there is a waste in downloading redundant data as the part shown in green. As the result, if we keep the the data of the last partially downloaded media chunk, it could save some. 

If the conditions below are satisfied, we could try to locate the last key frame within sampleQueue and drop the out-of-range (decoded only) samples at renderer.
1.     The search target locates within the last media chunk.
2.     The nearest key frame precedes the search target has been either within sampleQueue or sent to decoder (or rendered).

To take use of this fact, we create a new function named skipToLastKeyframe(). It tries to find the last key sample within the sampleQueue. Also, we create a function named isWithinLastChunk() to check whether the seek target locates within the last media chunk.


Test environment:


1. [Movie]: 4K timer MPEG DASH streaming from: 
https://www.youtube.com/watch?v=uo9dAIQR3g8
2. [Device]: HTC One X9u with Android 6.0 API = 23.
3. [Code Base]: Branch = release-v2
4. Fix the bitrate to the max one = 22361348 bits/sec.


Test result:



fig 3


fig 4


fig 5: comparison


The experimental result is analyzed here.

Here we explain the meanings of the result within fig 3 & 4, and how we decide the test condition.


How we choose the target seek?

Every time triggering a seek, we call getBufferedPositionUs() to get the buffered position =  T and seek the T+(50ms). It insures the seek target always be out of the range of sampleQueue.
The example is as below (I put it in seekToInternal()).

    if (Util.upstreamOptimizationExploration) {
      /*In most of the cases video will have min buffered data*/
      periodPositionUs = loadingPeriodHolder.mediaPeriod.getBufferedPositionUs() + 50000 /*50ms*/;
    }


1. [search target,  media chunk start] 
represents the average distance between the seek target and the start of last media chunk when doing test.
2. data loss
represent how much data we flushed from the beginning of last media chunk to the end of sampleQueue.
3. AllRenderersReady:
How much time it spends to all renderers are ready (all of them have rendered the first frame) after seek is delivered. At this time you could see the 1st frame of the seek target time but is still NOT ready to play. 
4. HaveSufficientBuffer:
At this time the playback actually starts.
5. network speed:
The network speed when we probe the improvement performance.

Explanation to the result:
By uniform distribution, since a segment typically is ~ 5.2 seconds, in average the seek target will be away from the start of the containing media chunk by ~ 2.6 seconds. Therefore, the item (1) & (2) match with the fact.

(3) could also be explained by the fact that the bitrate is not uniformly distributed within a segment. Since the key frame at the front of a media chunk is usually larger than other samples, the bitrate distribution of the first half part is larger than the second half one. Hence the time spent in downloading the first half part will be greater than the time spent in downloading the other part.

(4) reflects the relationship between network speed and download bitrate by
(22361348 * 2.6) / 19380250 ~ 2.999935, compared to 2.914 of item 4.

Finally, the experimental result shows we can improve about:
1.     3162.865 us to render the first frame.
2.     1233.17 us to start playback.

Totally, it improves ~ 4396 us for the 4K test case, or speed up by:
((4228.225+2914) /(1065.36+1680.33)) ~ 252.5%.

Figure 6 & 7 summarizes the difference between original & proposed scheme.
The saving gain could be understood there.





fig 6: original method - redownload the last chunk 





fig 7: proposed method - avoid the redundant download