openwrt/upstream - upstream openwrt

diff options

author	Felix Fietkau <nbd@openwrt.org>	2012-10-06 16:01:06 +0000
committer	Felix Fietkau <nbd@openwrt.org>	2012-10-06 16:01:06 +0000
commit	44cd20b8023f6f2fca50c32772229590b179756d (patch)
tree	814262a8291ca31c2261c2028238ecb44447ab75 /package/madwifi/patches/310-noise_get.patch
parent	5c91b34ed732afddfcb15fdf857f6939c050dfb1 (diff)
download	upstream-44cd20b8023f6f2fca50c32772229590b179756d.tar.gz upstream-44cd20b8023f6f2fca50c32772229590b179756d.tar.bz2 upstream-44cd20b8023f6f2fca50c32772229590b179756d.zip

build: rework the package/install step - collect package install lists during package/compile, then install all packages at once

SVN-Revision: 33631

Diffstat (limited to 'package/madwifi/patches/310-noise_get.patch')

0 files changed, 0 insertions, 0 deletions

=== How the Blkif Drivers Work === Andrew Warfield andrew.warfield@cl.cam.ac.uk The intent of this is to explain at a fairly detailed level how the split device drivers work in Xen 1.3 (aka 2.0beta). The intended audience for this, I suppose, is anyone who intends to work with the existing blkif interfaces and wants something to help them get up to speed with the code in a hurry. Secondly though, I hope to break out the general mechanisms that are used in the drivers that are likely to be necessary to implement other drivers interfaces. As a point of warning before starting, it is worth mentioning that I anticipate much of the specifics described here changing in the near future. There has been talk about making the blkif protocol a bit more efficient than it currently is. Keir's addition of grant tables will change the current remapping code that is used when shared pages are initially set up. Also, writing other control interface types will likely need support from Xend, which at the moment has a steep learning curve... this should be addressed in the future. For more information on the driver model as a whole, read the "Reconstructing I/O" technical report (http://www.cl.cam.ac.uk/Research/SRG/netos/papers/2004-xenngio.pdf). ==== High-level structure of a split-driver interface ==== Why would you want to write a split driver in the first place? As Xen is a virtual machine manager and focuses on isolation as an initial design principle, it is generally considered unwise to share physical access to devices across domains. The reasons for this are obvious: when device resources are shared, misbehaving code or hardware can result in the failure of all of the client applications. Moreover, as virtual machines in Xen are entire OSs, standard device drives that they might use cannot have multiple instantiations for a single piece of hardware. In light of all this, the general approach in Xen is to give a single virtual machine hardware access to a device, and where other VMs want to share the device, export a higher-level interface to facilitate that sharing. If you don't want to share, that's fine. There are currently Xen users actively exploring running two completely isolated X-Servers on a Xen host, each with it's own video card, keyboard, and mouse. In these situations, the guests need only be given physical access to the necessary devices and left to go on their own. However, for devices such as disks and network interfaces, where sharing is required, the split driver approach is a good solution. The structure is like this: +--------------------------+ +--------------------------+ | Domain 0 (privileged) | | Domain 1 (unprivileged) | | | | | | Xend ( Application ) | | | | Blkif Backend Driver | | Blkif Frontend Driver | | Physical Device Driver | | | +--------------------------+ +--------------------------+ +--------------------------------------------------------+ | X E N | +--------------------------------------------------------+ The Blkif driver is in two parts, which we refer to as frontend (FE) and a backend (BE). Together, they serve to proxy device requests between the guest operating system in an unprivileged domain, and the physical device driver in the physical domain. An additional benefit to this approach is that the FE driver can provide a single interface for a whole class of physical devices. The blkif interface mounts IDE, SCSI, and our own VBD-structured disks, independent of the physical driver underneath. Moreover, supporting additional OSs only requires that a new FE driver be written to connect to the existing backend. ==== Inter-Domain Communication Mechanisms ==== ===== Event Channels ===== Before getting into the specifics of the block interface driver, it is worth discussing the mechanisms that are used to communicate between domains. Two mechanisms are used to allow the construction of high-performance drivers: event channels and shared-memory rings. Event channels are an asynchronous interdomain notification mechanism. Xen allows channels to be instantiated between two domains, and domains can request that a virtual irq be attached to notifications on a given channel. The result of this is that the frontend domain can send a notification on an event channel, resulting in an interrupt entry into the backend at a later time. The event channel between two domains is instantiated in the Xend code during driver startup (described later). Xend's channel.py (tools/python/xen/xend/server/channel.py) defines the function def eventChannel(dom1, dom2): return xc.evtchn_bind_interdomain(dom1=dom1, dom2=dom2) which maps to xc_evtchn_bind_interdomain() in tools/libxc/xc_evtchn.c, which in turn generates a hypercall to Xen to patch the event channel between the domains. Only a privileged domain can request the creation of an event channel. Once the event channel is created in Xend, its ends are passed to both the front and backend domains over the control channel. The end that is passed to a domain is just an integer "port" uniquely identifying the event channel's local connection to that domain. An example of this setup code is in linux-2.6.x/drivers/xen/blkfront/blkfront.c in blkif_connect(), which receives several status change events as the driver starts up. It is passed an event channel end in a BLKIF_INTERFACE_STATUS_CONNECTED message, and patches it in like this: blkif_evtchn = status->evtchn; blkif_irq = bind_evtchn_to_irq(blkif_evtchn); if ( (rc = request_irq(blkif_irq, blkif_int, SA_SAMPLE_RANDOM, "blkif", NULL)) ) printk(KERN_ALERT"blkfront request_irq failed (%ld)\n",rc); This code associates a virtual irq with the event channel, and attaches the function blkif_int() as an interrupt handler for that irq. blkif_int() simply handles the notification and returns, it does not need to interact with the channel at all. An example of generating a notification can also be seen in blkfront.c: static inline void flush_requests(void) { DISABLE_SCATTERGATHER(); wmb(); /* Ensure that the frontend can see the requests. */ blk_ring->req_prod = req_prod; notify_via_evtchn(blkif_evtchn); } }}} notify_via_evtchn() issues a hypercall to set the event waiting flag on the other domain's end of the channel. ===== Communication Rings ===== Event channels are strictly a notification mechanism between domains. To move large chunks of data back and forth, Xen allows domains to share pages of memory. We use communication rings as a means of managing access to a shared memory page for message passing between domains. These rings are not explicitly a mechanism of Xen, which is only concerned with the actual sharing of the page and not how it is used, they are however worth discussing as they are used in many places in the current code and are a useful model for communicating across a shared page. A shared page is set up by a front end guest first allocating and passing the address of a page in its own address space to the backend driver. Consider the following code, also from blkfront.c. Note: this code is in blkif_disconnect(). The driver transitions from STATE_CLOSED to STATE_DISCONNECTED before becoming CONNECTED. The state automata is in blkif_status(). blk_ring = (blkif_ring_t *)__get_free_page(GFP_KERNEL); blk_ring->req_prod = blk_ring->resp_prod = resp_cons = req_prod = 0; ... /* Construct an interface-CONNECT message for the domain controller. */ cmsg.type = CMSG_BLKIF_FE; cmsg.subtype = CMSG_BLKIF_FE_INTERFACE_CONNECT; cmsg.length = sizeof(blkif_fe_interface_connect_t); up.handle = 0; up.shmem_frame = virt_to_machine(blk_ring) >> PAGE_SHIFT; memcpy(cmsg.msg, &up, sizeof(up)); blk_ring will be the shared page. The producer and consumer pointers are then initialised (these will be discussed soon), and then the machine address of the page is send to the backend via a control channel to Xend. This control channel itself uses the notification and shared memory mechanisms described here, but is set up for each domain automatically at startup. The backend, which is a privileged domain then takes the page address and maps it into its own address space (in linux26/drivers/xen/blkback/interface.c:blkif_connect()): void blkif_connect(blkif_be_connect_t *connect) ... unsigned long shmem_frame = connect->shmem_frame; ... if ( (vma = get_vm_area(PAGE_SIZE, VM_IOREMAP)) == NULL ) { connect->status = BLKIF_BE_STATUS_OUT_OF_MEMORY; return; } prot = __pgprot(_PAGE_PRESENT | _PAGE_RW | _PAGE_DIRTY | _PAGE_ACCESSED); error = direct_remap_area_pages(&init_mm, VMALLOC_VMADDR(vma->addr), shmem_frame<<PAGE_SHIFT, PAGE_SIZE, prot, domid); ... blkif->blk_ring_base = (blkif_ring_t *)vma->addr }}} The machine address of the page is passed in the shmem_frame field of the connect message. This is then mapped into the virtual address space of the backend domain, and saved in the blkif structure representing this particular backend connection. NOTE: New mechanisms will be added very shortly to allow domains to explicitly grant access to their pages to other domains. This "grant table" support is in the process of being added to the tree, and will change the way a shared page is set up. In particular, it will remove the need of the remapping domain to be privileged. Sending data across shared rings: Shared rings avoid the potential for write interference between domains in a very cunning way. A ring is partitioned into a request and a response region, and domains only work within their own space. This can be thought of as a double producer-consumer ring -- the ring is described by four pointers into a circular buffer of fixed-size records. Pointers may only advance, and may not pass one another. resp_cons----+ V +----+----+----+----+----+----+----+ | | | free(A) |RSP1|RSP2| +----+----+----+----+----+----+----+ req_prod->| | --------> |RSP3| +----+ +----+ |REQ8| | |<-resp_prod +----+ +----+ |REQ7| | | +----+ +----+ |REQ6| <-------- | | +----+----+----+----+----+----+----+ |REQ5|REQ4| free(B) | | | +----+----+----+----+----+----+----+ req_cons---------^ By adopting the convention that every request will receive a response, not all four pointers need be shared and flow control on the ring becomes very easy to manage. Each domain manages its own consumer pointer, and the two producer pointers are visible to both (xen/include/public/io/blkif.h): /* NB. Ring size must be small enough for sizeof(blkif_ring_t) <=PAGE_SIZE.*/ #define BLKIF_RING_SIZE 64 ... /* * We use a special capitalised type name because it is _essential_ that all * arithmetic on indexes is done on an integer type of the correct size. */ typedef u32 BLKIF_RING_IDX; /* * Ring indexes are 'free running'. That is, they are not stored modulo the * size of the ring buffer. The following macro converts a free-running counter * into a value that can directly index a ring-buffer array. */ #define MASK_BLKIF_IDX(_i) ((_i)&(BLKIF_RING_SIZE-1)) typedef struct { BLKIF_RING_IDX req_prod; /* 0: Request producer. Updated by front-end. */ BLKIF_RING_IDX resp_prod; /* 4: Response producer. Updated by back-end. */ union { /* 8 */ blkif_request_t req; blkif_response_t resp; } PACKED ring[BLKIF_RING_SIZE]; } PACKED blkif_ring_t; As shown in the diagram above, the rules for using a shared memory ring are simple. 1. A ring is full when a domain's producer and consumer pointers are equal (e.g. req_prod == resp_cons). In this situation, the consumer pointer must be advanced. Furthermore, if the consumer pointer is equal to the other domain's producer pointer, (e.g. resp_cons = resp_prod), then the other domain has all the buffers. 2. Producer pointers point to the next buffer that will be written to. (So blk_ring[MASK_BLKIF_IDX(req_prod)] should not be consumed.) 3. Consumer pointers point to a valid message, so long as they are not equal to the associated producer pointer. 4. A domain should only ever write to the message pointed to by its producer index, and read from the message at it's consumer. More generally, the domain may be thought of to have exclusive access to the messages between its consumer and producer, and should absolutely not read or write outside this region. Thus the front end has exclusive access to the free(A) region in the figure above, and the back end driver has exclusive access to the free(B) region. In general, drivers keep a private copy of their producer pointer and then set the shared version when they are ready for the other end to process a set of messages. Additionally, it is worth paying attention to the use of memory barriers (rmb/wmb) in the code, to ensure that rings that are shared across processors behave as expected. ==== Structure of the Blkif Drivers ==== Now that the communications primitives have been discussed, I'll quickly cover the general structure of the blkif driver. This is intended to give a high-level idea of what is going on, in an effort to make reading the code a more approachable task. There are three key software components that are involved in the blkif drivers (not counting Xen itself). The frontend and backend driver, and Xend, which coordinates their initial connection. Xend may also be involved in control-channel signalling in some cases after startup, for instance to manage reconnection if the backend is restarted. ===== Frontend Driver Structure ===== The frontend domain uses a single event channel and a shared memory ring to trade control messages with the backend. These are both setup during domain startup, which will be discussed shortly. The shared memory ring is called blkif_ring, and the private ring indexes are resp_cons, and req_prod. The ring is protected by blkif_io_lock. Additionally, the frontend keeps a list of outstanding requests in rec_ring[]. These are uniquely identified by a guest-local id number, which is associated with each request sent to the backend, and returned with the matching responses. Information about the actual disks are stored in major_info[], of which only the first nr_vbds entries are valid. Finally, the global 'recovery' indicates that the connection between the backend and frontend drivers has been broken (possibly due to a backend driver crash) and that the frontend is in recovery mode, in which case it will attempt to reconnect and reissue outstanding requests. The frontend driver is single-threaded and after setup is entered only through three points: (1) read/write requests from the XenLinux guest that it is a part of, (2) interrupts from the backend driver on its event channel (blkif_int()), and (3) control messages from Xend (blkif_ctrlif_rx). ===== Backend Driver Structure ===== The backend driver is slightly more complex as it must manage any number of concurrent frontend connections. For each domain it manages, the backend driver maintains a blkif structure, which describes all the connection and disk information associated with that particular domain. This structure is associated with the interrupt registration, and allows the backend driver to have immediate context when it takes a notification from some domain. All of the blkif structures are stored in a hash table (blkif_hash), which is indexed by a hash of the domain id, and a "handle", really a per-domain blkif identifier, in case it wants to have multiple connections. The per-connection blkif structure is of type blkif_t. It contains all of the communication details (event channel, irq, shared memory ring and indexes), and blk_ring_lock, which is the backend mutex on the shared ring. The structure also contains vbd_rb, which is a red-black tree, containing an entry for each device/partition that is assigned to that domain. This structure is filled by xend passing disk information to the backend at startup, and is protected by vbd_lock. Finally, the blkif struct contains a status field, which describes the state of the connection. The backend driver spawns a kernel thread at startup (blkio_schedule()), which handles requests to and from the actual disk device drivers. This scheduler thread maintains a list of blkif structures that have pending requests, and services them round-robin with a maximum per-round request limit. blkifs are added to the list in the interrupt handler (blkif_be_int()) using add_to_blkdev_list_tail(), and removed in the scheduler loop after calling do_block_io_op(), which processes a batch of requests. The scheduler thread is explicitly activated at several points in the code using maybe_trigger_blkio_schedule(). Pending requests between the backend driver and the physical device drivers use another ring, pending_ring. Requests are placed in this ring in the scheduler thread and issued to the device. A completion callback, end_block_io_op, indicates that requests have been serviced and generates a response on the appropriate blkif ring. pending reqs[] stores a list of outstanding requests with the physical drivers. So, control entries to the backend are (1) the blkio scheduler thread, which sends requests to the real device drivers, (2) end_block_io_op, which is called as serviced requests complete, (3) blkif_be_int() handles notifications from the frontend drivers in other domains, and (4) blkif_ctrlif_rx() handles control messages from xend. ==== Driver Startup ==== Prior to starting a new guest using the frontend driver, the backend will have been started in a privileged domain. The backend initialisation code initialises all of its data structures, such as the blkif hash table, and starts the scheduler thread as a kernel thread. It then sends a driver status up message to let xend know it is ready to take frontend connections. When a new domain that uses the blkif frontend driver is started, there are a series of interactions between it, xend, and the specified backend driver. These interactions are as follows: The domain configuration given to xend will specify the backend domain and disks that the new guest is to use. Prior to actually running the domain, xend and the backend driver interact to setup the initial blkif record in the backend. (1) Xend sends a BLKIF_BE_CREATE message to backend. Backend does blkif_create(), having been passed FE domid and handle. It creates and initialises a new blkif struct, and puts it in the hash table. It then returns a STATUS_OK response to xend. (2) Xend sends a BLKIF_BE_VBD_CREATE message to the backend. Backend adds a vbd entry in the red-black tree for the specified (dom, handle) blkif entry. Sends a STATUS_OK response. (3) Xend sends a BLKIF_BE_VBD_GROW message to the backend. Backend takes the physical device information passed in the message and assigns them to the newly created vbd struct. (2) and (3) repeat as any additional devices are added to the domain. At this point, the backend has enough state to allow the frontend domain to start. The domain is run, and eventually gets to the frontend driver initialisation code. After setting up the frontend data structures, this code continues the communications with xend and the backend to negotiate a connection: (4) Frontend sends Xend a BLKIF_FE_DRIVER_STATUS_CHANGED message. This message tells xend that the driver is up. The init function now spin-waits until driver setup is complete in order to prevent Linux from attempting to boot before the disks are connected. (5) Xend sends the frontend an INTERFACE_STATUS_CHANGED message This message specifies that the interface is now disconnected (instead of closed). The domain updates it's state, and allocates the shared blk_ring page. Next, (6) Frontend sends Xend a BLKIF_INTERFACE_CONNECT message This message specifies the domain and handle, and includes the address of the newly created page. (7) Xend sends the backend a BLKIF_BE_CONNECT message The backend fills in the blkif connection information, maps the shared page, and binds an irq to the event channel. (8) Xend sends the frontend an INTERFACE_STATUS_CHANGED message This message takes the frontend driver to a CONNECTED state, at which point it binds an irq to the event channel and calls xlvbd_init to initialise the individual block devices. The frontend Linux is stall spin waiting at this point, until all of the disks have been probed. Messaging now is directly between the front and backend domain using the new shared ring and event channel. (9) The frontend sends a BLKIF_OP_PROBE directly to the backend. This message includes a reference to an additional page, that the backend can use for it's reply. The backend responds with an array of the domains disks (as vdisk_t structs) on the provided page. The frontend now initialises each disk, calling xlvbd_init_device() for each one.


context:
space:
mode: