To demonstrate the spatiotemporal annotation capabilities of the Video Ontology, assume a video scene of a medical procedure, namely, Nd:YAG1 laser vitreolysis. This procedure is used to treat bothersome vitreous opacities, which negatively affect the quality of life (Shah and Heier, 2017). The selected scene was described using spatiotemporal segmentation, which involved the annotation of the region of interest, i.e., an eye floater, as a moving region, and the description of the medical video scene and the concept depicted in the scene. By using standard Media Fragment URIs,2 the spatiotemporal segmentation can be performed by specifying the positions in Normal Play Time format (as defined in RFC 2326),3 and the region of interest was represented by its minimum bounding box, as shown in Fig. 1.

Spatiotemporal Annotation

Figure 1 Spatial annotation of the region of interest depicting a Weiss ring floater by the top left
corner coordinates and dimensions of its minimum bounding box. Video by James H. Johnson4

This scene can be described in Turtle as follows:

@prefix dbpedia: <http://dbpedia.org/resource/> .
@prefix mpeg7: <http://purl.org/ontology/mpeg7/> .
@prefix snomedct: <http://purl.bioontology.org/ontology/SNOMEDCT/> .
@prefix temporal: <http://swrl.stanford.edu/ontologies/built-ins/3.3/temporal.owl> .
@prefix video: <http://purl.org/ontology/video/> .
@prefix xsd: <http://www.w3.org/2001/XMLSchema#> .
<https://www.youtube.com/watch?v=hZibbnjiGuY> a mpeg7:Video .
<https://www.youtube.com/watch?v=hZibbnjiGuY#t=0:00:00,0:00:50> a video:Scene ; video:temporalSegmentOf <https://www.youtube.com/watch?v=hZibbnjiGuY> ;
video:sceneFrom <https://www.youtube.com/watch?v=hZibbnjiGuY> ;
temporal:hasStartTime "00:00:0"^^xsd:time ;
temporal:duration "PT00M00S50"^^xsd:duration ;
temporal:hasFinishTime "00:00:50"^^xsd:time .
<https://www.youtube.com/watch?v=hZibbnjiGuY#t=0:00:48,0:00:50&xywh=721,207,358,323> a mpeg7:StillRegion ;
video:spatioTemporalSegmentOf <https://www.youtube.com/watch?v=hZibbnjiGuY> ;
video:depicts snomedct:15013002 , dbpedia:Floater .

Note that highly specialized domain knowledge is captured using other ontologies as usual. The formal definition of the medical term used in this video scene description was retrieved from SNOMED CT and DBpedia. To maximize interoperability, terms of the MPEG-7 Ontology and the SWRL Temporal Ontology were reused for spatiotemporal segmentation, and XSD for datatype declarations. Further upper and domain ontologies can be used similarly, such as the 3D Modeling Ontology for 3D scenes.


Shah, C. P., Heier, J. S. (2017). YAG laser vitreolysis vs sham YAG vitreolysis for symptomatic vitreous floaters: a randomized clinical trial. JAMA Ophthalmology, 135, 918–923. doi:10.1001/jamaophthalmol.2017.2388

1Neodymium-doped yttrium aluminum garnet (Nd:Y3Al5O12)
2https://www.w3.org/TR/media-frags/
3https://www.ietf.org/rfc/rfc2326.txt
4https://www.youtube.com/watch?v=hZibbnjiGuY