media-workshop November 2015

media-workshop@linuxtv.org

6 participants
2 discussions

[PATCH v8.4 00/83] Media MC patches
by Mauro Carvalho Chehab 11 Dec '15

11 Dec '15

As requested by Sakari and Hans, I'm re-sending the entire 83 patch series that were already sent to the media ML. NOTE: as agreed on IRC, after getting the ack for those patches, eventually adding newer fixups where needed, we should add this to the media-controller topic branch of our devel tree. Before merging this on the "master" branch, we'll: 1) add a patch series that Sakari is working to allow not using the entity_id for the graph traversal; 2) add a patch converting MC next gen to use a single range, as agreed on IRC; 3) add a patch increasing the maximum number of objects, in order for the graph traversal to work with DVB drivers; 4) get ALSA maintainer's acks for the ALSA changes (on this series and on Shuah's series); 5) add Shuah ALSA patches. I'll do the above work after getting the acks for the patches already sent. Javier Martinez Canillas (13): [media] staging: omap4iss: get entity ID using media_entity_id() [media] omap3isp: get entity ID using media_entity_id() [media] media: use entity.graph_obj.mdev instead of .parent [media] media: remove media entity .parent field [media] omap3isp: separate links creation from entities init [media] omap3isp: create links after all subdevs have been bound [media] staging: omap4iss: separate links creation from entities init [media] v4l: vsp1: create pad links after subdev registration [media] v4l: vsp1: separate links creation from entities init [media] uvcvideo: create pad links after subdev registration [media] smiapp: create pad links after subdev registration [media] media-entity: init pads on entity init if was registered before media: don't try to empty links list in media_entity_cleanup() Mauro Carvalho Chehab (70): [media] media: create a macro to get entity ID [media] media: add a common struct to be embed on media graph objects [media] media: use media_gobj inside entities [media] media: use media_gobj inside pads [media] media: use media_gobj inside links [media] media: add messages when media device gets (un)registered [media] media: add a debug message to warn about gobj creation/removal [media] media: rename the function that create pad links [media] uapi/media.h: Declare interface types for V4L2 and DVB [media] media: add functions to allow creating interfaces [media] uapi/media.h: Declare interface types for ALSA [media] media: Don't accept early-created links [media] media: convert links from array to list [media] media: make add link more generic [media] media: make media_link more generic to handle interace links [media] media: make link debug printk more generic [media] media: add support to link interfaces and entities [media] media-entity: add a helper function to create interface [media] dvbdev: add support for interfaces [media] media: add a linked list to track interfaces by mdev [media] dvbdev: add support for indirect interface links [media] uapi/media.h: Fix entity namespace [media] replace all occurrences of MEDIA_ENT_T_DEVNODE_V4L [media] replace all occurrences of MEDIA_ENT_T_DEVNODE_DVB [media] media: add macros to check if subdev or V4L2 DMA [media] media: use macros to check for V4L2 subdev entities [media] omap3/omap4/davinci: get rid of MEDIA_ENT_T_V4L2_SUBDEV abuse [media] s5c73m3: fix subdev type [media] s5k5baf: fix subdev type [media] davinci_vbpe: stop MEDIA_ENT_T_V4L2_SUBDEV abuse [media] omap4iss: stop MEDIA_ENT_T_V4L2_SUBDEV abuse [media] v4l2-subdev: use MEDIA_ENT_T_UNKNOWN for new subdevs [media] media controller: get rid of entity subtype on Kernel [media] media.h: don't use legacy entity macros at Kernel [media] DocBook: update descriptions for the media controller entities [media] dvb: modify core to implement interfaces/entities at MC new gen [media] media: report if a pad is sink or source at debug msg [media] uapi/media.h: Add MEDIA_IOC_G_TOPOLOGY ioctl [media] media: Use a macro to interate between all interfaces [media] media: move mdev list init to gobj [media] media-device: add pads and links to media_device [media] media_device: add a topology version field [media] media-device: add support for MEDIA_IOC_G_TOPOLOGY ioctl [media] media-entity: unregister entity links [media] remove interface links at media_entity_unregister() [media] media-device: remove interfaces and interface links [media] au0828: unregister MC at the end [media] v4l2-core: create MC interfaces for devnodes [media] media-entity.h: document all the structs [media] tuner-core: add an input pad [media] au0828: add support for the connectors [media] au0828: Create connector links [media] media-device: supress backlinks at G_TOPOLOGY ioctl [media] media-controller: enable all interface links at init [media] media.h: create connector entities for hybrid TV devices [media] dvbdev: returns error if graph object creation fails [media] dvb core: must check dvb_create_media_graph() [media] media-entity: enforce check of interface and links creation [media] cx231xx: enforce check for graph creation [media] au0828:: enforce check for graph creation [media] media-entity: must check media_create_pad_link() [media] media-entity.h: rename entity.type to entity.function [media] media-device: export the entity function via new ioctl [media] uapi/media.h: Rename entities types to functions [media] DocBook: update entities documentation [media] dvbdev: move indirect links on dvr/demux to a separate function [media] dvbdev: Don't create indirect links [media] media_entity: remove gfp_flags argument [media] media-device: use unsigned ints on some places [media] media-entity.c: get rid of var length arrays .../DocBook/media/v4l/media-ioc-enum-entities.xml | 67 +-- Documentation/media-framework.txt | 2 +- Documentation/video4linux/v4l2-framework.txt | 4 +- drivers/media/common/siano/smsdvb-main.c | 6 +- drivers/media/dvb-core/dmxdev.c | 4 +- drivers/media/dvb-core/dvb_ca_en50221.c | 2 +- drivers/media/dvb-core/dvb_frontend.c | 11 +- drivers/media/dvb-core/dvb_net.c | 2 +- drivers/media/dvb-core/dvbdev.c | 392 +++++++++++++---- drivers/media/dvb-core/dvbdev.h | 17 +- drivers/media/dvb-frontends/au8522_decoder.c | 2 +- drivers/media/firewire/firedtv-ci.c | 2 +- drivers/media/i2c/adp1653.c | 2 +- drivers/media/i2c/adv7180.c | 2 +- drivers/media/i2c/as3645a.c | 2 +- drivers/media/i2c/cx25840/cx25840-core.c | 2 +- drivers/media/i2c/lm3560.c | 2 +- drivers/media/i2c/lm3646.c | 2 +- drivers/media/i2c/m5mols/m5mols_core.c | 2 +- drivers/media/i2c/noon010pc30.c | 2 +- drivers/media/i2c/ov2659.c | 2 +- drivers/media/i2c/ov9650.c | 2 +- drivers/media/i2c/s5c73m3/s5c73m3-core.c | 8 +- drivers/media/i2c/s5k4ecgx.c | 2 +- drivers/media/i2c/s5k5baf.c | 8 +- drivers/media/i2c/s5k6aa.c | 2 +- drivers/media/i2c/smiapp/smiapp-core.c | 22 +- drivers/media/i2c/tvp514x.c | 2 +- drivers/media/i2c/tvp7002.c | 2 +- drivers/media/media-device.c | 272 ++++++++++-- drivers/media/media-entity.c | 478 +++++++++++++++++---- drivers/media/pci/bt8xx/dst_ca.c | 3 +- drivers/media/pci/ddbridge/ddbridge-core.c | 2 +- drivers/media/pci/ngene/ngene-core.c | 2 +- drivers/media/pci/ttpci/av7110.c | 2 +- drivers/media/pci/ttpci/av7110_av.c | 4 +- drivers/media/pci/ttpci/av7110_ca.c | 2 +- drivers/media/platform/exynos4-is/common.c | 3 +- drivers/media/platform/exynos4-is/fimc-capture.c | 5 +- drivers/media/platform/exynos4-is/fimc-isp-video.c | 9 +- drivers/media/platform/exynos4-is/fimc-lite.c | 18 +- drivers/media/platform/exynos4-is/media-dev.c | 25 +- drivers/media/platform/exynos4-is/media-dev.h | 8 +- drivers/media/platform/omap3isp/isp.c | 202 +++++---- drivers/media/platform/omap3isp/ispccdc.c | 39 +- drivers/media/platform/omap3isp/ispccdc.h | 1 + drivers/media/platform/omap3isp/ispccp2.c | 35 +- drivers/media/platform/omap3isp/ispccp2.h | 1 + drivers/media/platform/omap3isp/ispcsi2.c | 33 +- drivers/media/platform/omap3isp/ispcsi2.h | 1 + drivers/media/platform/omap3isp/isppreview.c | 48 ++- drivers/media/platform/omap3isp/isppreview.h | 1 + drivers/media/platform/omap3isp/ispresizer.c | 46 +- drivers/media/platform/omap3isp/ispresizer.h | 1 + drivers/media/platform/omap3isp/ispvideo.c | 17 +- drivers/media/platform/s3c-camif/camif-capture.c | 2 +- drivers/media/platform/s3c-camif/camif-core.c | 4 +- drivers/media/platform/vsp1/vsp1_drv.c | 34 +- drivers/media/platform/vsp1/vsp1_rpf.c | 29 +- drivers/media/platform/vsp1/vsp1_rwpf.h | 5 + drivers/media/platform/vsp1/vsp1_video.c | 15 +- drivers/media/platform/vsp1/vsp1_wpf.c | 40 +- drivers/media/platform/xilinx/xilinx-dma.c | 10 +- drivers/media/platform/xilinx/xilinx-vipp.c | 4 +- drivers/media/usb/au0828/au0828-core.c | 98 ++++- drivers/media/usb/au0828/au0828-dvb.c | 8 +- drivers/media/usb/au0828/au0828-video.c | 84 +++- drivers/media/usb/au0828/au0828.h | 3 +- drivers/media/usb/cx231xx/cx231xx-cards.c | 46 +- drivers/media/usb/cx231xx/cx231xx-dvb.c | 6 +- drivers/media/usb/cx231xx/cx231xx-video.c | 10 +- drivers/media/usb/dvb-usb-v2/dvb_usb_core.c | 4 +- drivers/media/usb/dvb-usb/dvb-usb-dvb.c | 6 +- drivers/media/usb/uvc/uvc_entity.c | 25 +- drivers/media/v4l2-core/tuner-core.c | 10 +- drivers/media/v4l2-core/v4l2-dev.c | 112 ++++- drivers/media/v4l2-core/v4l2-device.c | 16 +- drivers/media/v4l2-core/v4l2-flash-led-class.c | 2 +- drivers/media/v4l2-core/v4l2-subdev.c | 8 +- drivers/staging/media/davinci_vpfe/dm365_ipipe.c | 9 +- drivers/staging/media/davinci_vpfe/dm365_ipipeif.c | 15 +- drivers/staging/media/davinci_vpfe/dm365_isif.c | 15 +- drivers/staging/media/davinci_vpfe/dm365_resizer.c | 33 +- .../staging/media/davinci_vpfe/vpfe_mc_capture.c | 10 +- drivers/staging/media/davinci_vpfe/vpfe_video.c | 17 +- drivers/staging/media/omap4iss/iss.c | 123 ++++-- drivers/staging/media/omap4iss/iss_csi2.c | 46 +- drivers/staging/media/omap4iss/iss_csi2.h | 1 + drivers/staging/media/omap4iss/iss_ipipe.c | 9 +- drivers/staging/media/omap4iss/iss_ipipeif.c | 42 +- drivers/staging/media/omap4iss/iss_ipipeif.h | 1 + drivers/staging/media/omap4iss/iss_resizer.c | 40 +- drivers/staging/media/omap4iss/iss_resizer.h | 1 + drivers/staging/media/omap4iss/iss_video.c | 9 +- include/media/media-device.h | 34 +- include/media/media-entity.h | 322 ++++++++++++-- include/media/tuner.h | 8 + include/media/v4l2-dev.h | 1 + include/uapi/linux/media.h | 214 ++++++++- 99 files changed, 2535 insertions(+), 831 deletions(-) -- 2.4.3

7 179

Etherpad notes of the KS meeting in Seoul
by Mauro Carvalho Chehab 03 Nov '15

03 Nov '15

Those are the final version of the Etherpad notes we took during the Media Workshop at the Kernel Summit. Please review. I should be preparing the final report of the meeting latter this week. Regards, Mauro --- Media Summit Seoul 2015 Attendees list: * Shuah Khan - Samsung OSG shuahkh(a)osg.samsung.com * Laurent Pinchart - Ideas on Board * Hans Verkuil - Cisco Systems Norway - hverkuil(a)xs4all.nl * Javier Martinez Canillas - Samsung OSG * Tomasz Figa - Google - tfiga(a)chromium.org * Mauro Carvalho Chehab - Samsung OSG - mchehab(a)osg.samsung.com * David Howells - Red Hat - dhowells(a)redhat.com * Seung-Woo Kim - Samsung Software R&D Center - sw0312.kim(a)samsung.com * Inki Dae - Samsung Software R&D Center - inki.dae(a)samsung.com * Junghak Sung - Samsung Visual Display Business - jh1009.sung(a)samsung.com * Geunyoung Kim - Samsung Visual Display Business - nenggun.kim(a)samsung.com * Rany Kwon - Samsung Visual Display Business - rany.kwon(a)samsung.com * Minsong Kim - Samsung Visual Display Business - ms17.kim(a)samsung.com * Ikjoon Kim - Samsung Visual Display Business - ikjoon.kim(a)samsung.com * Thiago - Samsung OSG * Reynaldo - Samsung OSG * Luis de Bittencourt - Samsung OSG * Pawel Osciak - Google * Vinod Koul - Intel * Mark Brown - Linaro * Arnd Bergmann - Linaro 1. Pawel: Codec API: Stream API The original V4L2 codec API was developed along with the Exynos codec driver. As the device implements high-level operations in hardware the resulting API was high-level as well with drivers accepting unprocessed raw streams. This matches older ARM SoCs where CPU power wasn't deemed enough to implement stream parsing. Drivers implement two V4L2 buffer queues, one on the uncompressed side and one on the compressed side. The two queues operate independently, without a 1:1 correspondence between consumed and produced buffers (for instance reference frames need to be accumulated when starting video encoding before producing any output or bitstream header parsed before CAPTURE buffers can be allocated). The mem2mem V4L2 kernel framework thus can't be used to implement such codec drivers as it hardcodes this 1:1 correspondence. This is a kernel framework issue, not a V4L2 userspace API issue. (For stream API fundamentals see Pawel's slides) * Frame API (Slice API) CPUs are getting faster in the ARM world. The trend is to implement lower-level hardware codecs that require stream parsing on the CPU. CPU code needs to slice the stream, extract information, process them and pass the result to a shader-like device. This is the model used on Intel platforms and implemented in the VA API library. Drivers still implement two V4L2 buffer queues, but the encoded stream is split into frames of slices, and a large number of codec-specific controls need to be set from parsed stream information. Stream parsing and parameters calculation is better done in userspace. Userspace is responsible for managing reference frames and their life time, and for passing data to the codec in such a way that an input buffer wlil always produce an output buffer. The two queues operate together with a 1:1 correspondence between buffers. The mem2mem framework is thus usable. Sources buffer contain only slice data (macroblocks + coefficient data). Controls contain information extracted from stream parsing, list of reference frames and DPB (Decoded Picture Buffer). The request API can be used to associate controls with source buffers. Keeping references to reference frames is one of the remaining problems (especially with DMABUF, and possibly with MMAP in the future when we'll have the ability to destroy MMAP buffers while streaming). This problem also exists for the stream API. More discussion is needed to design a solution. Pawel promissed to upstream the ChromeOS codec drivers code this year (and you should be also nagging Tomasz to do it...). For encoders header generation should probably be done in userspace as well as the code is complex and doesn't require a kernel implementation. Userspace code should be implemented as libv4l2 plugins to interface between the frame API exposed by the kernel and a stream API exposed to applications. * Status See Pawel's slides. * Discussion points To be discussed on Tuesday at 11:30. References: Intel libVA SDK: https://bugs.freedesktop.org/show_bug.cgi?id=92533 Request API: https://lwn.net/Articles/641204/ Chromium Code (user): https://code.google.com/p/chromium/codesearch#chromium/src/content/common/g… 2: kABI kABI needs more review by devs. New kernel APIs need to be documented, adding e.g. new fields to undocumented structures doesn't require the submitter to document the full struct (although it obviously would be appreciated). Suggest to use kernel doc for 'top-level' documentation: drivers/gpu/drm/drm_atomic_helper.c (look for "DOC: overview") and Documentation/DocBook/drm.tmpl (look for "!Pdrivers/gpu/drm/drm_atomic_helper.c overview"). 3: uAPI for DVB * videobuf2 No DMABUF support at the moment: recommended that that's added, at the very least planned in the API and likely implemented too. Support for DVB output ('mem2mem demux') is also recommended (not necessary for the first version) Streamon/off is automatically called in the DVB API: should this be available as ioctl? No need seen for this. DVB framework needs to be extended to see whether userspace uses read() or stream I/O: V4L2 does this as well, so the same method should be used as V4L2. No need to add (or remove) buffers at runtime (à la CREATE_BUF) is foreseen. * SoC pipelines Samsung SoCs have full hardware pipelines from the tuner output to the screen without requiring any intervention at runtime. This requires connecting a DVB device to a V4L2 or DRM device. Hardware pipelines include hardware-configured scalers for seamless resolution change, in order to achieve real-time operation when the input resolution changes. 4. v4l2_buffer & Y2038 Should we create a new structure and fix several other issues in one go ? Yes. There might be a cache-related problem due to the way DMABUF is handled. DMABUF buffers are only unmapped when applications queue a different DMABUF fd on the same V4L2 buffer index. By that time the driver doesn't own access to the buffer but still tries to clean the cache. The root cause needs to be investigated. A related issue is how V4L2 keeps DMABUF mappings around for performance improvement. This prevents buffer eviction by the GPU. This should be discussed with DRM developers and dma-buf maintainers. Should we add support for sub-streams in a buffer ? Possibly for 3D capture, for other cases there's not so much interest. Use cases would be needed. Formats are specified for buffers and makes it difficult to transport meta-data in a plane along with image planes. Redesign of v4l2_buffer and v4l2_plane should take the use case into account. 5. poll() for output streams The current poll() behaviour is not usable for codecs. As mem2mem is considered as the main use case for output devices we need to optimize for that, and implement write emulation specific code in poll() as a special case. We'll revert to the old implementation and update the documentation accordingly. 6. poll for capture streams q->waiting_for_buffers is v4l2 specific, but after the vb2 split it is now part of the vb2 core: should this be moved to the v4l2 part or be kept in the vb2 core? After discussing this the opinion is that we should make this v4l2 specific and allow DVB (and other subsystems) to use the standard behavior. Proposals are welcome on ways to make userspace select the 'waiting_for_buffers' behavior for V4L2 since it is awkward there as well (but needed for backwards compatibility) 7. CSC 8. CEC 9. REQUEST_BUFS: v4l2_format validation: expect that userspace calls TRY_FMT or similar to fill the format field, only use sizeimage field in drivers. 10. MC Look at being able to set a topology using 'cat file >/dev/mediaX' (idea from alsa): depends on the details of the proposed format. Atomic configuration updates across subsystems (MC for links, V4L2/ALSA for other parameters, ...) needs to be taken into account. For audio devices routing changes ordering matters unless the hardware can apply routing changes atomically. 11. VIDIOC_SUBDEV_QUERYCAP Suggestion: MEDIA_IOC_INTERFACE_INFO: check that it is easy to implement in various subsystems (dvb, alsa, drm, iio) 12. DELETE_BUFFERS Yes, we'd like to have it. Ditto for more than 32 buffers (don't limit this). 13. Workshop format If we have two days available, then post a CFP to various mailinglists (including apps like gstreamer). Co-locate with alsa workshop.

2 1

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

media-workshop November 2015