openwrt/upstream - upstream openwrt

	Commit message (Expand)	Author	Age	Files	Lines
*	kernel: add generic image_cmdline hack to MIPS targets	Gabor Juhos	2009-09-10	1	-0/+1
*	initial support for the AP-R3052-V22RW-2X2 board	Gabor Juhos	2009-09-02	1	-0/+1
*	remove unused *_MACH_GENERIC config options	Gabor Juhos	2009-09-01	1	-1/+0
*	register GPIO LEDs on the WHR-G300N board	Gabor Juhos	2009-08-31	1	-0/+1
*	rt305x: remove HW_HAS_PCI config option	Gabor Juhos	2009-08-31	1	-1/+0
*	initial support for RT288x/RT305x	Gabor Juhos	2009-08-30	2	-0/+167

One of the biggest challenges to getting started with embedded devices is that you cannot just install a copy of Linux and expect to be able to compile a firmware. Even if you did remember to install a compiler and every development tool offered, you still would not have the basic set of tools needed to produce a firmware image. The embedded device represents an entirely new hardware platform, which is most of the time incompatible with the hardware on your development machine, so in a process called cross compiling you need to produce a new compiler capable of generating code for your embedded platform, and then use it to compile a basic Linux distribution to run on your device. The process of creating a cross compiler can be tricky, it is not something that is regularly attempted and so there is a certain amount of mystery and black magic associated with it. In many cases when you are dealing with embedded devices you will be provided with a binary copy of a compiler and basic libraries rather than instructions for creating your own -- it is a time saving step but at the same time often means you will be using a rather dated set of tools. Likewise, it is also common to be provided with a patched copy of the Linux kernel from the board or chip vendor, but this is also dated and it can be difficult to spot exactly what has been modified to make the kernel run on the embedded platform. \subsection{Building an image} OpenWrt takes a different approach to building a firmware; downloading, patching and compiling everything from scratch, including the cross compiler. To put it in simpler terms, OpenWrt does not contain any executables or even sources, it is an automated system for downloading the sources, patching them to work with the given platform and compiling them correctly for that platform. What this means is that just by changing the template, you can change any step in the process. As an example, if a new kernel is released, a simple change to one of the Makefiles will download the latest kernel, patch it to run on the embedded platform and produce a new firmware image -- there is no work to be done trying to track down an unmodified copy of the existing kernel to see what changes had been made, the patches are already provided and the process ends up almost completely transparent. This does not just apply to the kernel, but to anything included with OpenWrt -- It is this one simple understated concept which is what allows OpenWrt to stay on the bleeding edge with the latest compilers, latest kernels and latest applications. So let's take a look at OpenWrt and see how this all works. \subsubsection{Download OpenWrt} OpenWrt can be downloaded via subversion using the following command: \begin{Verbatim} $ svn checkout svn://svn.openwrt.org/openwrt/trunk openwrt-trunk \end{Verbatim} Additionally, there is a trac interface on \href{https://dev.openwrt.org/}{https://dev.openwrt.org/} which can be used to monitor svn commits and browse the source repository. \subsubsection{The directory structure} There are four key directories in the base: \begin{itemize} \item \texttt{tools} \item \texttt{toolchain} \item \texttt{package} \item \texttt{target} \end{itemize} \texttt{tools} and \texttt{toolchain} refer to common tools which will be used to build the firmware image, the compiler, and the C library. The result of this is three new directories, \texttt{build\_dir/host}, which is a temporary directory for building the target independent tools, \texttt{build\_dir/toolchain-\textit{<arch>}*} which is used for building the toolchain for a specific architecture, and \texttt{staging\_dir/toolchain-\textit{<arch>}*} where the resulting toolchain is installed. You will not need to do anything with the toolchain directory unless you intend to add a new version of one of the components above. \begin{itemize} \item \texttt{build\_dir/host} \item \texttt{build\_dir/toolchain-\textit{<arch>}*} \end{itemize} \texttt{package} is for exactly that -- packages. In an OpenWrt firmware, almost everything is an \texttt{.ipk}, a software package which can be added to the firmware to provide new features or removed to save space. Note that packages are also maintained outside of the main trunk and can be obtained from subversion using the package feeds system: \begin{Verbatim} $ ./scripts/feeds update \end{Verbatim} Those packages can be used to extend the functionality of the build system and need to be symlinked into the main trunk. Once you do that, the packages will show up in the menu for configuration. You would do something like this: \begin{Verbatim} $ ./scripts/feeds search nmap Search results in feed 'packages': nmap Network exploration and/or security auditing utility $ ./scripts/feeds install nmap \end{Verbatim} To include all packages, issue the following command: \begin{Verbatim} $ make package/symlinks \end{Verbatim} \texttt{target} refers to the embedded platform, this contains items which are specific to a specific embedded platform. Of particular interest here is the "\texttt{target/linux}" directory which is broken down by platform \textit{<arch>} and contains the patches to the kernel, profile config, for a particular platform. There's also the "\texttt{target/image}" directory which describes how to package a firmware for a specific platform. Both the target and package steps will use the directory "\texttt{build\_dir/\textit{<arch>}}" as a temporary directory for compiling. Additionally, anything downloaded by the toolchain, target or package steps will be placed in the "\texttt{dl}" directory. \begin{itemize} \item \texttt{build\_dir/\textit{<arch>}} \item \texttt{dl} \end{itemize} \subsubsection{Building OpenWrt} While the OpenWrt build environment was intended mostly for developers, it also has to be simple enough that an inexperienced end user can easily build his or her own customized firmware. Running the command "\texttt{make menuconfig}" will bring up OpenWrt's configuration menu screen, through this menu you can select which platform you're targeting, which versions of the toolchain you want to use to build and what packages you want to install into the firmware image. Note that it will also check to make sure you have the basic dependencies for it to run correctly. If that fails, you will need to install some more tools in your local environment before you can begin. Similar to the linux kernel config, almost every option has three choices, \texttt{y/m/n} which are represented as follows: \begin{itemize} \item{\texttt{<*>} (pressing y)} \\ This will be included in the firmware image \item{\texttt{<M>} (pressing m)} \\ This will be compiled but not included (for later install) \item{\texttt{< >} (pressing n)} \\ This will not be compiled \end{itemize} After you've finished with the menu configuration, exit and when prompted, save your configuration changes. If you want, you can also modify the kernel config for the selected target system. simply run "\texttt{make kernel\_menuconfig}" and the build system will unpack the kernel sources (if necessary), run menuconfig inside of the kernel tree, and then copy the kernel config to \texttt{target/linux/\textit{<platform>}/config} so that it is preserved over "\texttt{make clean}" calls. To begin compiling the firmware, type "\texttt{make}". By default OpenWrt will only display a high level overview of the compile process and not each individual command. \subsubsection{Example:} \begin{Verbatim} make[2] toolchain/install make[3] -C toolchain install make[2] target/compile make[3] -C target compile make[4] -C target/utils prepare [...] \end{Verbatim} This makes it easier to monitor which step it's actually compiling and reduces the amount of noise caused by the compile output. To see the full output, run the command "\texttt{make V=99}". During the build process, buildroot will download all sources to the "\texttt{dl}" directory and will start patching and compiling them in the "\texttt{build\_dir/\textit{<arch>}}" directory. When finished, the resulting firmware will be in the "\texttt{bin}" directory and packages will be in the "\texttt{bin/packages}" directory. \subsection{Creating packages} One of the things that we've attempted to do with OpenWrt's template system is make it incredibly easy to port software to OpenWrt. If you look at a typical package directory in OpenWrt you'll find several things: \begin{itemize} \item \texttt{package/\textit{<name>}/Makefile} \item \texttt{package/\textit{<name>}/patches} \item \texttt{package/\textit{<name>}/files} \end{itemize} The patches directory is optional and typically contains bug fixes or optimizations to reduce the size of the executable. The package makefile is the important item, provides the steps actually needed to download and compile the package. The files directory is also optional and typicall contains package specific startup scripts or default configuration files that can be used out of the box with OpenWrt. Looking at one of the package makefiles, you'd hardly recognize it as a makefile. Through what can only be described as blatant disregard and abuse of the traditional make format, the makefile has been transformed into an object oriented template which simplifies the entire ordeal. Here for example, is \texttt{package/bridge/Makefile}: \begin{Verbatim}[frame=single,numbers=left] include $(TOPDIR)/rules.mk PKG_NAME:=bridge PKG_VERSION:=1.0.6 PKG_RELEASE:=1 PKG_SOURCE:=bridge-utils-$(PKG_VERSION).tar.gz PKG_SOURCE_URL:=@SF/bridge PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd PKG_CAT:=zcat PKG_BUILD_DIR:=$(BUILD_DIR)/bridge-utils-$(PKG_VERSION) include $(INCLUDE_DIR)/package.mk define Package/bridge SECTION:=net CATEGORY:=Base system TITLE:=Ethernet bridging configuration utility URL:=http://bridge.sourceforge.net/ endef define Package/bridge/description Manage ethernet bridging: a way to connect networks together to form a larger network. endef define Build/Configure $(call Build/Configure/Default, \ --with-linux-headers="$(LINUX_DIR)" \ ) endef define Package/bridge/install $(INSTALL_DIR) $(1)/usr/sbin $(INSTALL_BIN) $(PKG_BUILD_DIR)/brctl/brctl $(1)/usr/sbin/ endef $(eval $(call BuildPackage,bridge)) \end{Verbatim} As you can see, there's not much work to be done; everything is hidden in other makefiles and abstracted to the point where you only need to specify a few variables. \begin{itemize} \item \texttt{PKG\_NAME} \\ The name of the package, as seen via menuconfig and ipkg \item \texttt{PKG\_VERSION} \\ The upstream version number that we are downloading \item \texttt{PKG\_RELEASE} \\ The version of this package Makefile \item \texttt{PKG\_SOURCE} \\ The filename of the original sources \item \texttt{PKG\_SOURCE\_URL} \\ Where to download the sources from (no trailing slash), you can add multiple download sources by separating them with a \\ and a carriage return. \item \texttt{PKG\_MD5SUM} \\ A checksum to validate the download \item \texttt{PKG\_CAT} \\ How to decompress the sources (zcat, bzcat, unzip) \item \texttt{PKG\_BUILD\_DIR} \\ Where to compile the package \end{itemize} The \texttt{PKG\_*} variables define where to download the package from; \texttt{@SF} is a special keyword for downloading packages from sourceforge. There is also another keyword of \texttt{@GNU} for grabbing GNU source releases. If any of the above mentionned download source fails, the OpenWrt mirrors will be used as source. The md5sum (if present) is used to verify the package was downloaded correctly and \texttt{PKG\_BUILD\_DIR} defines where to find the package after the sources are uncompressed into \texttt{\$(BUILD\_DIR)}. At the bottom of the file is where the real magic happens, "BuildPackage" is a macro set up by the earlier include statements. BuildPackage only takes one argument directly -- the name of the package to be built, in this case "\texttt{bridge}". All other information is taken from the define blocks. This is a way of providing a level of verbosity, it's inherently clear what the contents of the \texttt{description} template in \texttt{Package/bridge} is, which wouldn't be the case if we passed this information directly as the Nth argument to \texttt{BuildPackage}. \texttt{BuildPackage} uses the following defines: \textbf{\texttt{Package/\textit{<name>}}:} \\ \texttt{\textit{<name>}} matches the argument passed to buildroot, this describes the package the menuconfig and ipkg entries. Within \texttt{Package/\textit{<name>}} you can define the following variables: \begin{itemize} \item \texttt{SECTION} \\ The section of package (currently unused) \item \texttt{CATEGORY} \\ Which menu it appears in menuconfig: Network, Sound, Utilities, Multimedia ... \item \texttt{TITLE} \\ A short description of the package \item \texttt{URL} \\ Where to find the original software \item \texttt{MAINTAINER} (optional) \\ Who to contact concerning the package \item \texttt{DEPENDS} (optional) \\ Which packages must be built/installed before this package. To reference a dependency defined in the same Makefile, use \textit{<dependency name>}. If defined as an external package, use \textit{+<dependency name>}. For a kernel version dependency use: \textit{@LINUX\_2\_<minor version>} \item \texttt{BUILDONLY} (optional) \\ Set this option to 1 if you do NOT want your package to appear in menuconfig. This is useful for packages which are only used as build dependencies. \end{itemize} \textbf{\texttt{Package/\textit{<name>}/conffiles} (optional):} \\ A list of config files installed by this package, one file per line. \textbf{\texttt{Build/Prepare} (optional):} \\ A set of commands to unpack and patch the sources. You may safely leave this undefined. \textbf{\texttt{Build/Configure} (optional):} \\ You can leave this undefined if the source doesn't use configure or has a normal config script, otherwise you can put your own commands here or use "\texttt{\$(call Build/Configure/Default,\textit{<first list of arguments, second list>})}" as above to pass in additional arguments for a standard configure script. The first list of arguments will be passed to the configure script like that: \texttt{--arg 1} \texttt{--arg 2}. The second list contains arguments that should be defined before running the configure script such as autoconf or compiler specific variables. To make it easier to modify the configure command line, you can either extend or completely override the following variables: \begin{itemize} \item \texttt{CONFIGURE\_ARGS} \\ Contains all command line arguments (format: \texttt{--arg 1} \texttt{--arg 2}) \item \texttt{CONFIGURE\_VARS} \\ Contains all environment variables that are passed to ./configure (format: \texttt{NAME="value"}) \end{itemize} \textbf{\texttt{Build/Compile} (optional):} \\ How to compile the source; in most cases you should leave this undefined. As with \texttt{Build/Configure} there are two variables that allow you to override the make command line environment variables and flags: \begin{itemize} \item \texttt{MAKE\_FLAGS} \\ Contains all command line arguments (typically variable overrides like \texttt{NAME="value"} \item \texttt{MAKE\_VARS} \\ Contains all environment variables that are passed to the make command \end{itemize} \textbf{\texttt{Build/InstallDev} (optional):} \\ If your package provides a library that needs to be made available to other packages, you can use the \texttt{Build/InstallDev} template to copy it into the staging directory which is used to collect all files that other packages might depend on at build time. When it is called by the build system, two parameters are passed to it. \texttt{\$(1)} points to the regular staging dir, typically \texttt{staging\_dir/\textit{ARCH}}, while \texttt{\$(2)} points to \texttt{staging\_dir/host}. The host staging dir is only used for binaries, which are to be executed or linked against on the host and its \texttt{bin/} subdirectory is included in the \texttt{PATH} which is passed down to the build system processes. Please use \texttt{\$(1)} and \texttt{\$(2)} here instead of the build system variables \texttt{\$(STAGING\_DIR)} and \texttt{\$(STAGING\_DIR\_HOST)}, because the build system behavior when staging libraries might change in the future to include automatic uninstallation. \textbf{\texttt{Package/\textit{<name>}/install}:} \\ A set of commands to copy files out of the compiled source and into the ipkg which is represented by the \texttt{\$(1)} directory. Note that there are currently 4 defined install macros: \begin{itemize} \item \texttt{INSTALL\_DIR} \\ install -d -m0755 \item \texttt{INSTALL\_BIN} \\ install -m0755 \item \texttt{INSTALL\_DATA} \\ install -m0644 \item \texttt{INSTALL\_CONF} \\ install -m0600 \end{itemize} The reason that some of the defines are prefixed by "\texttt{Package/\textit{<name>}}" and others are simply "\texttt{Build}" is because of the possibility of generating multiple packages from a single source. OpenWrt works under the assumption of one source per package Makefile, but you can split that source into as many packages as desired. Since you only need to compile the sources once, there's one global set of "\texttt{Build}" defines, but you can add as many "Package/<name>" defines as you want by adding extra calls to \texttt{BuildPackage} -- see the dropbear package for an example. After you have created your \texttt{package/\textit{<name>}/Makefile}, the new package will automatically show in the menu the next time you run "make menuconfig" and if selected will be built automatically the next time "\texttt{make}" is run. \subsection{Creating binary packages} You might want to create binary packages and include them in the resulting images as packages. To do so, you can use the following template, which basically sets to nothing the Configure and Compile templates. \begin{Verbatim}[frame=single,numbers=left] include $(TOPDIR)/rules.mk PKG_NAME:=binpkg PKG_VERSION:=1.0 PKG_RELEASE:=1 PKG_SOURCE:=binpkg-$(PKG_VERSION).tar.gz PKG_SOURCE_URL:=http://server PKG_MD5SUM:=9b7dc52656f5cbec846a7ba3299f73bd PKG_CAT:=zcat include $(INCLUDE_DIR)/package.mk define Package/binpkg SECTION:=net CATEGORY:=Network TITLE:=Binary package endef define Package/bridge/description Binary package endef define Build/Configure endef define Build/Compile endef define Package/bridge/install $(INSTALL_DIR) $(1)/usr/sbin $(INSTALL_BIN) $(PKG_BUILD_DIR)/* $(1)/usr/sbin/ endef $(eval $(call BuildPackage,bridge)) \end{Verbatim} Provided that the tarball which contains the binaries reflects the final directory layout (/usr, /lib ...), it becomes very easy to get your package look like one build from sources. Note that using the same technique, you can easily create binary pcakages for your proprietary kernel modules as well. \subsection{Creating kernel modules packages} The OpenWrt distribution makes the distinction between two kind of kernel modules, those coming along with the mainline kernel, and the others available as a separate project. We will see later that a common template is used for both of them. For kernel modules that are part of the mainline kernel source, the makefiles are located in \textit{package/kernel/modules/*.mk} and they appear under the section "Kernel modules" For external kernel modules, you can add them to the build system just like if they were software packages by defining a KernelPackage section in the package makefile. Here for instance the Makefile for the I2C subsytem kernel modules : \begin{Verbatim}[frame=single,numbers=left] I2CMENU:=I2C Bus define KernelPackage/i2c-core TITLE:=I2C support DESCRIPTION:=Kernel modules for i2c support SUBMENU:=$(I2CMENU) KCONFIG:=CONFIG_I2C_CORE CONFIG_I2C_DEV FILES:=$(MODULES_DIR)/kernel/drivers/i2c/*.$(LINUX_KMOD_SUFFIX) AUTOLOAD:=$(call AutoLoad,50,i2c-core i2c-dev) endef $(eval $(call KernelPackage,i2c-core)) \end{Verbatim} To group kernel modules under a common description in menuconfig, you might want to define a \textit{<description>MENU} variable on top of the kernel modules makefile. \begin{itemize} \item \texttt{TITLE} \\ The name of the module as seen via menuconfig \item \texttt{DESCRIPTION} \\ The description as seen via help in menuconfig \item \texttt{SUBMENU} \\ The sub menu under which this package will be seen \item \texttt{KCONFIG} \\ Kernel configuration option dependency. For external modules, remove it. \item \texttt{FILES} \\ Files you want to inlude to this kernel module package, separate with spaces. \item \texttt{AUTOLOAD} \\ Modules that will be loaded automatically on boot, the order you write them is the order they would be loaded. \end{itemize} After you have created your \texttt{package/kernel/modules/\textit{<name>}.mk}, the new kernel modules package will automatically show in the menu under "Kernel modules" next time you run "make menuconfig" and if selected will be built automatically the next time "\texttt{make}" is run. \subsection{Conventions} There are a couple conventions to follow regarding packages: \begin{itemize} \item \texttt{files} \begin{enumerate} \item configuration files follow the convention \\ \texttt{\textit{<name>}.conf} \item init files follow the convention \\ \texttt{\textit{<name>}.init} \end{enumerate} \item \texttt{patches} \begin{enumerate} \item patches are numerically prefixed and named related to what they do \end{enumerate} \end{itemize} \subsection{Troubleshooting} If you find your package doesn't show up in menuconfig, try the following command to see if you get the correct description: \begin{Verbatim} TOPDIR=$PWD make -C package/<name> DUMP=1 V=99 \end{Verbatim} If you're just having trouble getting your package to compile, there's a few shortcuts you can take. Instead of waiting for make to get to your package, you can run one of the following: \begin{itemize} \item \texttt{make package/\textit{<name>}/clean V=99} \item \texttt{make package/\textit{<name>}/install V=99} \end{itemize} Another nice trick is that if the source directory under \texttt{build\_dir/\textit{<arch>}} is newer than the package directory, it won't clobber it by unpacking the sources again. If you were working on a patch you could simply edit the sources under the \texttt{build\_dir/\textit{<arch>}/\textit{<source>}} directory and run the install command above, when satisfied, copy the patched sources elsewhere and diff them with the unpatched sources. A warning though - if you go modify anything under \texttt{package/\textit{<name>}} it will remove the old sources and unpack a fresh copy. Other useful targets include: \begin{itemize} \item \texttt{make package/\textit{<name>}/prepare V=99} \item \texttt{make package/\textit{<name>}/compile V=99} \item \texttt{make package/\textit{<name>}/configure V=99} \end{itemize} \subsection{Using build environments} OpenWrt provides a means of building images for multiple configurations which can use multiple targets in one single checkout. These \emph{environments} store a copy of the .config file generated by \texttt{make menuconfig} and the contents of the \texttt{./files} folder. The script \texttt{./scripts/env} is used to manage these environments, it uses \texttt{git} (which needs to be installed on your system) as backend for version control. The command \begin{Verbatim} ./scripts/env help \end{Verbatim} produces a short help text with a list of commands. To create a new environment named \texttt{current}, run the following command \begin{Verbatim} ./scripts/env new current \end{Verbatim} This will move your \texttt{.config} file and \texttt{./files} (if it exists) to the \texttt{env/} subdirectory and create symlinks in the base folder. After running make menuconfig or changing things in files/, your current state will differ from what has been saved before. To show these changes, use: \begin{Verbatim} ./scripts/env diff \end{Verbatim} If you want to save these changes, run: \begin{Verbatim} ./scripts/env save \end{Verbatim} If you want to revert your changes to the previously saved copy, run: \begin{Verbatim} ./scripts/env revert \end{Verbatim} If you want, you can now create a second environment using the \texttt{new} command. It will ask you whether you want to make it a clone of the current environment (e.g. for minor changes) or if you want to start with a clean version (e.g. for selecting a new target). To switch to a different environment (e.g. \texttt{test1}), use: \begin{Verbatim} ./scripts/env switch test1 \end{Verbatim} To rename the current branch to a new name (e.g. \texttt{test2}), use: \begin{Verbatim} ./scripts/env rename test2 \end{Verbatim} If you want to get rid of environment switching and keep everything in the base directory again, use: \begin{Verbatim} ./scripts/env clear \end{Verbatim}