-- Cheers, Mauro Media brainstorming meeting 2015 https://etherpad.fr/p/media-brainstorming-2015 https://etherpad.mozilla.org/YoMTPHKPmf 2015-07-29--31 Hans Verkuil Mauro Chehab Antti Laakso Lars-Peter Clausen Sakari Ailus Tuukka Toivonen Laurent Pinchart Shuah Khan Morning 29th Summary: Stuff is complicated Control and data planes: http://etherealmind.com/controller-based-networks-for-data-centres/ Multiple options to express association between device nodes and entities: - Same IOCTL API as for data links, difference made between data and control links - Different IOCTL API to enumerate interfaces, with associated entities - Properties Terminology: - Entity: Functional unit, typically a hardware component or subunit in a hardware component - Pad: Data Input/Output of a entity - Link: Connection between a source pad and a sink pad - Interface: Software control access to a entity (e.g. v4l2-subdev) - Association: Mapping between interface and entity One interface can control multiple entities One entity can be controlled by multiple interfaces Afternoon 29th Summary: Things are still complicated Media interfaces - A concept of user space facing interfaces - An object of its own right (such as media entities) - At the same level than entities - Mauro's concern is that there will be lots of code duplication in the kernel Working proposal - Introduce struct media_interface as an object type alongside with entities in the kernel and in the user space - See how the patches will look like; start with V4L2 and proceed with DVB - How much will there be code duplication? - Target 4.3 at earliest, more likely 4.4 - Mauro: no patches, drivers or framework, implementing MC features merged until this is fixed (fixes are an exception) - Hans: I'll try do the work next week Userspace API Requirements: - Enumerate Entities + pads/links (exists today) - Enumerate Interfaces + entity associations (+ which interface is the default) - For entities: enumerate associated interfaces - Two application models: Entity-centric vs Interface-centric Future: - MC: 'versioning number' so apps are informed if the topology has changed. - MC: Notification events for topology changes Points to solve: - entity.type - should be renamed to reflect what's there; - How to represent DVB entities/interfaces? - DVB network interfaces? - How to represent ALSA entities? - What fields are needed for media interfaces at the public API? - changes at media-ctl userspace application Random notes - man 7 netdevice - man 7 rtnetlink - Network interface index is constant for a network device - Network interface index can be converted to name etc. by an IOCTL - IOCTL design: http://blog.ffwll.ch/2013/11/botching-up-ioctls.html - Not widely used outside DRM - Most sub-systems have their own conventions TODO: - Add graph topology version to the enumeration (and possibly other) ioctls QUESTION: SHOULD WE REDESIGN EVERYTHING, TO SUPPORT TOPOLOGY CHANGES? ========================================================================= Address for tomorrow morning, at Samsung Research UK, in ESPOO Innopoli 3, Vaisalantie 4, Espoo. Morning 30th VB2 - Presentation made by Samsung via videoconf about the scheduled changes for VB2 DVB pipelines - Presentation made by Samsung via videoconf about the pipelines: tuner 1 -> -> demux -> Video tuner 2 -> cam 1 -> demux -> Audio tuner 3 -> cam 2 -> demux -> Data -> V4L2 -> ALSA ASoC Generic DT machine driver can support around half of DT based devices Not all required data available in ACPI, so ACPI based devices typically have their own drivers DAPM does power management in ASoC much like the current omap3isp driver implementation does Walks the graph Supports large numbers of entities and cyclical graphs Contains ASoC dependencies, but the core of the algorithm is generic Stack based, iterative version in works MC has generic graph walking code as well Iterative Less concerned with performance omap3isp driver has PM code, which could be more generic Ideally we should have single implementation for graph walking and for power management Updates to ALSA and au0828 to share resources: 1. state->pads[AU8522_PAD_AUDIO_OUT].flags = MEDIA_PAD_FL_SINK; Need to be MEDIA_PAD_FL_SOURCE; 2. ALSA: to use on a non-atomic context: pcm->substream->nonatomic IRQ safe start/stop pipeline isn't necessary. Not a good idea to change graph_mutex to spinlock. I2C drivers will break. This will reduce the work and patch series v6 becomes smaller. 3. enable_source is necessary at media_device level, but find a better name 4. Write helper function to hde decoder and tuner details from ALSA 5. remove link is needed - implement - Shuah can do that as part of ALSA work 6. Managed Media Controller API - devres - change it to get reference and put reference. It isn't safe to delete media_device device resource. media_device_get() and media_device_put() Keep /dev/media device open and unplug the usb device - kernel would crash Afternoon 30th Keysign party: Mauro: F909 AE68 FC11 DF09 C175 5C00 085F 3EBD 8EE4 E115 (m.chehab@samsung.com and mchehab@osg.samsung.com) BDBC 259D BFF2 2603 F5E0 B67E 1FCD 9060 (mchehab@infradead.org) Shuah: 3D92 B29B F459 B8E0 8678 0FE4 0B02 44C4 0D0D 431C Lars: 5C1B A1C6 C4B6 1D76 3462 D775 E8F7 68B4 9C45 B399 Laurent: 9423 1B98 0100 EC61 9AC1 0E10 F045 C2B9 6991 256E Hans: 052C DE7B C215 053B 689F 1BCA BD2D 6148 6614 3B4C Sakari Ailus sakari.ailus@iki.fi: 270C DCD3 5B9D 7E26 C2D7 2097 BF53 AD52 308A D4EC Sakari Ailus sakari.ailus@linux.intel.com: F0D0 377A 0D4F 25A7 9238 EFE5 6D40 361B 6E28 C193 IIO - Industrial I/O - ADC, DAC, accelerometer, gyro, light sensor, pressure, manometer - typically: low speed < 10kbaud, I2C/SPI - changes: up to 5Gbaud, separate control and data bus (LVDS, JESD204b) - can be used by multi-channel SDR Morning 31st Media controller discussion Generic graph framework in the kernel? This would require review on linux-media, and take a long time. Different requirements could pop up. Perhaps not this time. An agreement was arrived for the UAPI interface. Some adjustments will be left to happen tomorrow. What was agreed: media interfaces will be a separate graph element; links will be used to connect both entities (pads to pads) and media interfaces; PADs will use unique IDs preliminary header changes: https://etherpad.mozilla.org/QNyfP34Ee6 V4L2 applications need to know if they can capture from an input/tuner. To make that easy for them we should add a new flag to VIDIOC_ENUMINPUT and whether the tuner is in use (and/or the tuner state) through VIDIOC_G_TUNER. Allow for bidirectional pads (needed for connectors like a coax that are bidirectional using time or frequency multiplexing) TODO: Connector entities Afternoon 31st Entity types Connector, DMA engine, processing block and controller block have been proposed Whather an entity is a connector needs to be told All others can be derived from other information already in the graph? Entity functional properties Most of the properties at entity level would be used to signal capabilities that can be figured out using sub-system specific interfaces the application already uses Interfaces Agreement: interfaces do have a type Types include V4L2, ALSA, DVB etc. Sub-types are used to tell between different interface types inside the sub-system A device can be a device node, network interface or a sysfs directory G_TOPOLOGY IOCTL should not be used to convey arbitrary sysfs paths G_INTERFACE to pass information on single interface; there, a single pointer set by the user would be feasible. The struct can be the same. Finding entities Especially device specific applications often need to find an exact entity the application requires This may be a certain scaler in the graph, for instance Configuration may involve e.g. private IOCTLs only a certain sub-device implements Name is only 32 bytes at the moment In practice, external entities have used i2c bus number and i2c address as part of the name This does not scale for DT based devices where the device is uniquely identified by a string that can be very long Blocks that are a part of an ISP there's been just a name of the block without any device specific information This causes that if one has multiple such devices in the system, the name is no longer unique The entity name is not enough to uniquely find an entity in a general case In a general case it is not possible to recognise an entity Hot pluggable buses and entities that have no serial numbers or something equivalent In order to best help the user space, all the information that matters should be provided to the user Device name, such as "omap3isp resizer" Bus Serial number Sysfs name (this may not be stable as sysfs has no official API stability, but may be still better than bus in some cases) This should enable most of the use cases where entities need to be found Properties Array of properties as an IOCTL argument has the problem that the user would have to allocate enough memory for each key-value pair in the array. This is unfeasible. The original proposal of using a text based format requires parsing by the user, albeit the kernel interface would remain simple The new proposal combines the best of both: a single memory area amended with an array of properties that refer to the memory area Using strings the key type has benefits Hierarchy is possible (not with u32 for instance) Should values be always strings or should other types be accepted? Probably having other types is a good idea. Different options for arrays As part of the key: entity.0/name As part of the property: add a length field. This is convenient except for strings, where it works as well --- just count the nul characters. Properties could be seen as re-implementing sysfs However they're much more light-weight No atomic access of all sysfs properties Sysfs is limited to page size Properties can be queried by media object, in which case the media object id would be provided The entire tree of properties can be obtained using object id 0 Another option for the kernel interface: key/length/value E.g. type (u32) | length (u32) | value (string) | key (string) Media objects Media objects have an unique ID in the system Objects can be entities, pads or interfaces, for instance A few bits can be reserved for object types, e.g. entity == 0; interface == 1, pad == 2 Action Items: - media_interface/media_topology: RFC for userspace API: Hans - RFC patch series: internal implementation for interface/topology: Mauro - Migration: add v4l-subdev media_interface: Laurent - Migration: add explicit DMA Engine entity: Laurent - media_properties: RFC userspace API: Sakari - MC core support for dynamically adding/removing from the media graph: Samsung - Clarify/implement DVB network interface: Mauro - MC support for Alsa: Lars: Need RFC before Alsa summit during ELCE - MC support for DVB: Mauro - MC default core support for interface-centric V4L2: Hans (?) - MC support for IIO: Lars - VIDIOC_ENUMINPUTS/G_TUNER: tell if the tuner/input is busy: RFC Hans, Shuah has a backend for this. - Fix media_device broken alloc/remove (devres): Shuah New include: /* * Multimedia device API * * Copyright (C) 2010 Nokia Corporation * * Contacts: Laurent Pinchart laurent.pinchart@ideasonboard.com * Sakari Ailus sakari.ailus@iki.fi * * This program is free software; you can redistribute it and/or modify * it under the terms of the GNU General Public License version 2 as * published by the Free Software Foundation. * * This program is distributed in the hope that it will be useful, * but WITHOUT ANY WARRANTY; without even the implied warranty of * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the * GNU General Public License for more details. * * You should have received a copy of the GNU General Public License * along with this program; if not, write to the Free Software * Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA */ #ifndef __LINUX_MEDIA_H #define __LINUX_MEDIA_H #include <linux/ioctl.h> #include <linux/types.h> #include <linux/version.h> #define MEDIA_API_VERSION KERNEL_VERSION(0, 1, 0) struct media_device_info { char driver[16]; char model[32]; char serial[40]; char bus_info[32]; __u32 media_version; __u32 hw_revision; __u32 driver_version; __u32 reserved[31]; }; #define MEDIA_ENT_ID_FLAG_NEXT (1 << 31) #define MEDIA_ENT_TYPE_SHIFT 16 #define MEDIA_ENT_TYPE_MASK 0x00ff0000 #define MEDIA_ENT_SUBTYPE_MASK 0x0000ffff #define MEDIA_ENT_T_DEVNODE (1 << MEDIA_ENT_TYPE_SHIFT) #define MEDIA_ENT_T_DEVNODE_V4L (MEDIA_ENT_T_DEVNODE + 1) #define MEDIA_ENT_T_DEVNODE_FB (MEDIA_ENT_T_DEVNODE + 2) #define MEDIA_ENT_T_DEVNODE_ALSA (MEDIA_ENT_T_DEVNODE + 3) /* Legacy symbol. Use it to avoid userspace compilation breakages */ #define MEDIA_ENT_T_DEVNODE_DVB MEDIA_ENT_T_DEVNODE_DVB_FE #define MEDIA_ENT_T_V4L2_SUBDEV (2 << MEDIA_ENT_TYPE_SHIFT) #define MEDIA_ENT_T_V4L2_SUBDEV_SENSOR (MEDIA_ENT_T_V4L2_SUBDEV + 1) #define MEDIA_ENT_T_V4L2_SUBDEV_FLASH (MEDIA_ENT_T_V4L2_SUBDEV + 2) #define MEDIA_ENT_T_V4L2_SUBDEV_LENS (MEDIA_ENT_T_V4L2_SUBDEV + 3) /* A converter of analogue video to its digital representation. */ #define MEDIA_ENT_T_V4L2_SUBDEV_DECODER (MEDIA_ENT_T_V4L2_SUBDEV + 4) #define MEDIA_ENT_T_V4L2_SUBDEV_TUNER (MEDIA_ENT_T_V4L2_SUBDEV + 5) Possible entity types: - connector - source-from-mem/sink-to-mem or memory-transport or dma engine - processing block - controller block - Time division multiplexed busses? Function Properties: - tuner - analog demod, digital demod - analog video decoder/encoder - video receiver/transmitter - audio ADC/DAC - mux/demux - image sensor - flash controller - lens/iris/etc. controllers - network (de)packetizer - audio mixer - deinterlaced - memory transport - scaler - CA - IR - amplifier #define MEDIA_ENT_FL_DEFAULT (1 << 0) struct media_entity_desc { __u32 id; char name[32]; __u32 type; __u32 revision; __u32 flags; __u32 group_id; __u16 pads; __u16 links; __u16 associations; __u16 reserved[7]; union { /* Node specifications */ struct { __u32 major; __u32 minor; } dev; /* Sub-device specifications */ /* Nothing needed yet */ __u8 raw[184]; }; }; #define MEDIA_PAD_FL_SINK (1 << 0) #define MEDIA_PAD_FL_SOURCE (1 << 1) #define MEDIA_PAD_FL_MUST_CONNECT (1 << 2) struct media_pad_desc { __u32 entity; /* entity ID */ __u16 index; /* pad index */ __u32 flags; /* pad flags */ __u32 reserved[2]; }; #define MEDIA_LNK_FL_ENABLED (1 << 0) #define MEDIA_LNK_FL_IMMUTABLE (1 << 1) #define MEDIA_LNK_FL_DYNAMIC (1 << 2) struct media_link_desc { struct media_pad_desc source; // source or controller pad struct media_pad_desc sink; // sink or controllee pad __u32 flags; __u32 reserved[2]; }; struct media_links_enum { __u32 entity; /* Should have enough room for pads elements */ struct media_pad_desc __user *pads; /* Should have enough room for links elements */ struct media_link_desc __user *links; __u32 reserved[4]; }; ////////////////////////////////////////////////////////// #define MEDIA_INTF_T_DVB_FE (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_DVB_DEMUX (MEDIA_ENT_T_DEVNODE + 5) #define MEDIA_INTF_T_DVB_DVR (MEDIA_ENT_T_DEVNODE + 6) #define MEDIA_INTF_T_DVB_CA (MEDIA_ENT_T_DEVNODE + 7) #define MEDIA_INTF_T_DVB_NET (MEDIA_ENT_T_DEVNODE + 8) // network interface: requires MC support for dynamically // adding/deleting these interface/entities/links/etc. #define MEDIA_INTF_T_NETIF_DVB (MEDIA_ENT_T_DEVNODE + 8) // ? #define MEDIA_INTF_T_V4L_VIDEO (MEDIA_ENT_T_DEVNODE + 8) #define MEDIA_INTF_T_V4L_VBI (MEDIA_ENT_T_DEVNODE + 8) #define MEDIA_INTF_T_V4L_RADIO (MEDIA_ENT_T_DEVNODE + 8) #define MEDIA_INTF_T_V4L_SUBDEV (MEDIA_ENT_T_DEVNODE + 8) #define MEDIA_INTF_T_V4L_SWRADIO (MEDIA_ENT_T_DEVNODE + 8) #define MEDIA_INTF_T_ALSA_PCM_CAPTURE (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_ALSA_PCM_PLAYBACK (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_ALSA_CONTROL (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_ALSA_COMPRESS (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_ALSA_RAWMIDI (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_ALSA_HWDEP (MEDIA_ENT_T_DEVNODE + 4) #if 0 // sysfs directory #define MEDIA_INTF_T_IIO_DEVICE (MEDIA_ENT_T_DEVNODE + 4) #define MEDIA_INTF_T_IIO_BUFFER (MEDIA_ENT_T_DEVNODE + 4) #endif struct media_interface { __u32 id; __u32 type; // 'Default interface' concept should be documented // in more detail. __u32 flags; __u32 num_interface_links; union { /* Node specifications */ // Split up node spec + subsystem info in separate // unions. struct { __u32 major; __u32 minor; } devnode; // struct { // __s32 if_index; // TBC // } netif; // struct { // __u32 id; // } iio_device; // Full pathname or similar should be passed as a property __u32 raw[16]; }; }; struct media_entity { __u32 id; __u32 flags; // Add connector flag //__u32 group_id; // removed: use properties instead __u16 num_pads; }; struct media_pad { __u32 entity_id; __u16 id; /* pad id (assuming unique pad ID) */ __u16 num_links; __u32 flags; }; // Should we have unique pad IDs? // If so, should we drop the source/sink_id fields? struct media_link { __u32 id; __u32 source_id; __u32 source_pad_id; __u32 sink_id; __u32 sink_pad_id; __u32 flags; // NON_DIRECTIONAL, BIDIRECTIONAL, UNIDIRECTIONAL // Also available in pads }; struct media_interface_link { __u32 id; __u32 interface_id; __u32 entity_id; __u32 flags; }; enum { MEDIA_PROP_T_STRING, MEDIA_PROP_T_U32, MEDIA_PROP_T_FOO, }; struct media_property { const char *key; __u32 object_id; // ? __u32 type; union { // based on type const char *string; __u32 *val; struct foo *foo_val; }; }; // id == entity/interface/pad // Idea: use the lower X bits to determine the type. // Not expose this to userspace struct media_properties { __u32 object_id; __u32 num_props; __u32 blob_size; struct media_property *props; __u8 *blob; }; //blob = malloc(blob_size); struct media_topology { __u32 topology_version; __u32 num_entities; struct media_entity *entities; __u32 num_interfaces; struct media_interface *interfaces; __u32 num_pads; struct media_pad *pads; __u32 num_links; struct media_link *links; __u32 num_interface_links; struct media_interface_link *interface_links; struct media_properties properties; }; //struct media_links { // __u32 topology_version; // __u32 num_links; // struct media_link *links; // __u32 num_interface_links; // struct media_interface_link *interface_links; //}; #define MEDIA_IOC_DEVICE_INFO _IOWR('|', 0x00, struct media_device_info) #define MEDIA_IOC_ENUM_ENTITIES _IOWR('|', 0x01, struct media_entity_desc) #define MEDIA_IOC_ENUM_LINKS _IOWR('|', 0x02, struct media_links_enum) #define MEDIA_IOC_SETUP_LINK _IOWR('|', 0x03, struct media_link_desc) #define MEDIA_IOC_G_TOPOLOGY _IOWR('|', 0x04, struct media_topology) //#define MEDIA_IOC_SETUP_LINKS _IOWR('|', 0x05, struct media_links) #endif /* __LINUX_MEDIA_H */