aboutsummaryrefslogtreecommitdiffstats
path: root/include/download.mk
Commit message (Collapse)AuthorAgeFilesLines
* download: improve handling of invalid local filesEneas U de Queiroz2021-11-011-2/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | 4e19cbc5533: [download: handle possibly invalid local tarballs] added a FORCE rule to downloaded files, so that they will be always checked by download.pl. As a side-effect, check-compile will fail, forcing unnecessary package rebuilds. The check-compile.txt log shows (for libxml2 for example): Considering target file '.../dl/libxml2-2.9.12.tar.gz'. ... prerequisite 'FORCE' of target '.../dl/libxml2-2.9.12.tar.gz' does not exist. Must remake target '.../dl/libxml2-2.9.12.tar.gz'. ... Giving up on target file '...libxml2-2.9.12/.prepared_...'. Giving up on target file '...libxml2-2.9.12/.configured_...'. Giving up on target file '...libxml2-2.9.12/.built'. Giving up on target file '...stamp/.libxml2_installed'. Giving up on target file '.compile'. Then the package is rebuilt even if it is not otherwise needed. To fix this, instead of always forcing the download target to be remade, check its hash first: if it matches, then the FORCE is not added. Signed-off-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
* build: introduce $(MKHASH)Leonardo Mörlein2021-05-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this commit, it was assumed that mkhash is in the PATH. While this was fine for the normal build workflow, this led to some issues if make TOPDIR="$(pwd)" -C "$pkgdir" compile was called manually. In most of the cases, I just saw warnings like this: make: Entering directory '/home/.../package/gluon-status-page' bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found [...] While these were only warnings and the package still compiled sucessfully, I also observed that some package even fail to build because of this. After applying this commit, the variable $(MKHASH) is introduced. This variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the correct path. Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
* download: add mirror alias for DebianDavid Bauer2021-02-261-1/+1
| | | | | | | Add an alias for Debian packages and download them from the Debian mirror redirector. Signed-off-by: David Bauer <mail@david-bauer.net>
* treewide: unify OpenWrt hosted source via @OPENWRTPaul Spooren2021-02-051-1/+1
| | | | | | | | | | | Multiple sources are hosted on OpenWrts source server only. The source URLs to point to the server vary based on different epochs in OpenWrts history. Replace all by @OPENWRT which is an "empty" mirror, therefore using the fallback servers sources.cdn.openwrt.org and sources.openwrt.org. Signed-off-by: Paul Spooren <mail@aparcar.org>
* build: use SPDX license tagsPaul Spooren2021-02-051-4/+1
| | | | | | | | | The license folder is a core part of OpenWrt and all GPL-2.0 licensed. Use SPDX license tags to allow machines to check licenses. Signed-off-by: Paul Spooren <mail@aparcar.org> [rebase, keep some Copyright lines, sharpen commit message] Signed-off-by: Adrian Schmutzler <freifunk@adrianschmutzler.de>
* build: prevent excessive re-evaluation of PKG_VERSIONFelix Fietkau2020-10-161-3/+5
| | | | | | | version_abbrev uses $(shell) and the ?= is causing make to run the command over and over again, causing a significant build slowdown Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: add zstd support to pack/unpack functionsMatthias Schiffer2020-05-311-0/+1
| | | | Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
* build: add PKG_SOURCE_URL_FILE supportKamil Wcislo2019-11-231-0/+1
| | | | | | | | | | | | | | It seems that there is a missing PKG_SOURCE_URL_FILE support. This little fix adds the support for packages to change the name of the downloaded file. Sometimes it is desirable to change the downloaded archive file name, like for mitigating name conflicts for different packages (some files on the server could be named like, e.g. 2018-01-01.tar.gz) or for the cases that there is no name for the file in the URL (e.g. http://someserver.com/download). Signed-off-by: Kamil Wcislo <kamil.wcislo@lpnplant.io> Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
* Revert faulty tree pushFelix Fietkau2019-08-121-1/+0
| | | | | | | | | | | | Revert "mac80211: add new minstrel_ht patches to improve probing on mt76x2" (9861050b85e5381c93d73ffdbc24c130400e0fb8) Revert "kernel: use bulk free in kfree_skb_list to improve performance" (98b654de2e7502507b31f0fb82befbb48f9c8542) Revert "ramips: add preliminary support for WIO ONE" (085141dc5ba5e763e1bf4c1a2fd6957160810ccc) Revert "ramips: add preliminary support for SGE AP-MTKH7-0006 developer board" (b1db6d0539511f60bf65fbde28b16afa31180e34) Revert "build: use config.site generated by autoconf-lean, drop hardcoded sitefiles" (363ce4329d4f6c00b4a7cb83dd1821fafc0d40ad) Revert "toolchain: add autoconf-lean" (fdb30eed03ae56cd87078b64d9b2fac00799e783) Revert "build: allow overriding the filename on the remote server when downloading" (6fa0e07758524eddf20c9b66dfcb55c490e1961e) Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: allow overriding the filename on the remote server when downloadingFelix Fietkau2019-08-121-0/+1
| | | | | | | | Github releases usually don't contain the project name in the release filename, which makes them very inconvenient to use from the build system. Add support for naming the local file differently. Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: remove sgid permission from tarEneas U de Queiroz2019-04-061-1/+2
| | | | | | | | | | Otherwise tar will keep the sgid bit when running from a sgid-set directory, resulting in a different file being generated. Signed-off-by: Eneas U de Queiroz <cote2004-github@yahoo.com> Signed-off-by: Christian Lamparter <chunkeey@gmail.com> [reworked commit message, removed DMARC]
* download.mk: enable DownloadMethod/github_archiveYousong Zhou2018-07-051-8/+14
| | | | Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* scripts/dl_github_archive.py: rename from download.pyYousong Zhou2018-07-051-7/+7
| | | | | | | | | | | - Make the code more GitHub-specific - Requires mirror hash to work with .gitattributes - Use different API depending on whether PKG_SOURCE_VERSION is a complete commit id or other ref types like tags - Fix removing symbolic link - pre-clean dir_untar for possible leftovers from previous run Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* download.mk: add more commentsYousong Zhou2018-07-051-2/+5
| | | | Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* download.mk: restore the old dl_method implementationYousong Zhou2018-06-291-1/+17
| | | | | | | | Seems like the python download.py dl_method call causes serious performance regression for fresh "make defconfig" as reported in FS#1621. GitHub tarball download will also be disabled with this Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* build: download code from github using archive APIYousong Zhou2018-06-271-31/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | A new python script scripts/download.py is added to fetch tarballs using GitHub archive API [1], then repack in a reproducible way same as the current DownloadMethod/git GitHub imposes a 60 reqs/hour rate limit on unauthenticated API access[2]. This affects fetching commit date for feeding tar --mtime= argument. However, observation indicates that archive download is NOT subject to this limit at the moment. In the rare cases where download fails because of this, we will falback to using DownloadMethod/git The missing piece in the GitHub API is that it cannot provide in the tarball dependent submodules's source code. In that case, the implementation will also fallback to using DownloadMethod/git [1] Get archive link, https://developer.github.com/v3/repos/contents/#get-archive-link [2] Rate limiting, https://developer.github.com/v3/#rate-limiting v2 <- v1: - allow passing multiple urls with --urls argument - add commit ts cache. can be helpful on retry Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* downloads.mk: introduce name-agnostic PROJECT_GIT variableJo-Philipp Wich2018-01-101-2/+4
| | | | | | | | | | Introduce a name-agnostic PROJECT_GIT variable poiting to https://git.openwrt.org/ and declare LEDE_GIT and OPENWRT_GIT as aliases to it. After some transition time we can drop this alias variables. Signed-off-by: Jo-Philipp Wich <jo@mein.io>
* build: remove @ as it's causing an errorPhilip Prindeville2017-09-201-1/+1
| | | | | | | | Since $(DownloadMethod/unknown) is being invoked in the expansion of $(call locked ...) anyway, you can't have an @ because the shell doesn't know what to do with it. Signed-off-by: Philip Prindeville <philipp@redfish-solutions.com>
* build: let make check warn about use of legacy PKG_MD5SUM variable in feedsFelix Fietkau2017-01-131-3/+0
| | | | | | | The variable rename change has been merged in OpenWrt now, so it's possible to convert the feeds as well. Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: use mkhash to replace various quirky md5sum/openssl callsFelix Fietkau2017-01-051-1/+1
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: add defaults for PKG_SOURCE, PKG_SOURCE_SUBDIR, PKG_VERSIONFelix Fietkau2016-12-221-0/+6
| | | | | | | | This makes it easier to unify versioning of git based package downloads. PKG_SOURCE_DATE along with an 8-character abbreviation of the git hash is used as PKG_VERSION, PKG_RELEASE should be used like normal packages. Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: add FIXUP option for make checkFelix Fietkau2016-12-171-1/+18
| | | | | | | | This will attempt to automatically fix common mistakes like using MD5 instead of SHA256, using the MD5SUM variable instead of HASH, or even a missing mirror file hash. Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: implement make check and make package/X/checkFelix Fietkau2016-12-171-9/+60
| | | | | | | | | | | | | | | | This is intended to be used for a wide array of package sanity checks. The first check that is implemented is for the hash of downloaded files. It checks: - Missing hash - Use of SHA256 instead of MD5 - dl/<file> hash not matching hash in makefile - deprecated MD5SUM variable The deprecated MD5SUM variable check is skipped for feeds/ until OpenWrt is updated as well Signed-off-by: Felix Fietkau <nbd@nbd.name>
* include/package.mk: sync default value for hash fallback with mirror hashFelix Fietkau2016-12-161-1/+1
| | | | | | Simplifies dealing with PKG_HASH vs PKG_MD5SUM Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: remove duplicate Download/default definition from include/host-build.mkFelix Fietkau2016-12-161-0/+13
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: replace MD5SUM variables with HASHFelix Fietkau2016-12-161-2/+4
| | | | | | | Since we've switched to preferring SHA256 over MD5, the old variable name is misleading. Packages using the old name remain compatible. Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download.mk: improve download tarball reproducibilityFelix Fietkau2016-12-131-1/+1
| | | | | | | Store only numeric user/group id. Group 0 is 'root' on linux and 'wheel' on BSD and Mac OS X. Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download.mk: remove code duplication in $(TAR) callFelix Fietkau2016-12-131-9/+12
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download.mk: use $(error) instead of a regular shell errorFelix Fietkau2016-12-131-1/+1
| | | | | | Useful for further rework Signed-off-by: Felix Fietkau <nbd@nbd.name>
* include/download.mk: Allow specify DownloadMethod specific optionsFlorian Fainelli2016-12-041-1/+2
| | | | | | | | This is going to be used to migrate the hand rolled git clone for the kernel into using the git download method. The kernel uses custom options that we may have to pass down. Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>
* include/download.mk: Use -7e compression instead of -6 by defaultDaniel Engberg2016-10-051-1/+1
| | | | | | Adds a slightly higher compression level to xz by default which roughly raises memory usage from 100MiB to about 200MiB during compression, about 10MiB for decompression. (Source: xz manpage) Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
* include/download.mk: generate reproducable SCM tarballsJo-Philipp Wich2016-10-051-4/+9
| | | | | | | | | | | | | | | | | Apply a number of changes to the tarball generation in order to produce identical files on different systems: 1) Use an explicit `gzip -cn` to avoid storing file mtime in the gzip header 2) Instruct `tar` to unconditionally use uid and gid 0 for archive members 3) Instruct `tar` to sort archive members by file name 4) For SCMs that do not preserve file modification times like Git or Mercurial, use the date of the last commit to the repository and pass it as `--mtime` value to `tar` After these changes, locally produced tarballs generated from SCM checkouts should be identical on any system, simplifying the mirroring of cache archives. Signed-off-by: Jo-Philipp Wich <jo@mein.io>
* download.mk: use HTTPS for git.lede-project.orgJo-Philipp Wich2016-06-011-1/+1
| | | | | | | | Some Git versions have issues following the HTTP->HTTPS redirect and since the keyring package is fetched from this host, switching to HTTPS is a sensible choice anyway. Signed-off-by: Jo-Philipp Wich <jo@mein.io>
* include: add lede git server urlJohn Crispin2016-04-291-0/+1
| | | | Signed-off-by: John Crispin <john@phrozen.org>
* download: add @GITHUB download facilityÁlvaro Fernández Rojas2016-04-031-0/+1
| | | | | | Define a new alias (@GITHUB) for downloading raw files from github repos. Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
* download: add @GITHUB download facilityJohn Crispin2016-04-091-1/+1
| | | | | | | | Define a new alias (@GITHUB) for downloading raw github repository files Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com> SVN-Revision: 49138
* buildroot: improve git submodule handling for packagesFelix Fietkau2016-02-291-2/+3
| | | | | | | | | | | | | | | | | | | Move the `--recursive` switch from `git clone` to `git submodule` so that submodules are cloned for upstream branches where the PKG_SOURCE_VERSION commit-ish has a different .gitmodules configuration than the repository default. This is, for example, required when the master branch for a source package does not use submodules, but its topic branch for OpenWRT does. This changes the buildroot dependency from git-1.6.2 to git 1.7.12.2, which was released September 2012. Signed-off-by: Darik Horn <dajhorn@vanadac.com> Signed-off-by: Karl Palsson <karlp@tweak.net.au> SVN-Revision: 48830
* download.mk: fix packed checkout mirroring supportJo-Philipp Wich2016-02-171-1/+1
| | | | | | | | | | | Changeset r48416 broke the downloading of mirrored, packed scm checkouts. Fix this by removing the "@" sign in front of the download command which is now executed as part of a larger shell command under flock. Signed-off-by: Jo-Philipp Wich <jow@openwrt.org> SVN-Revision: 48733
* build: download.mk: bugfix download.pl argument call orderFelix Fietkau2016-01-211-1/+1
| | | | | | | | | Change wrong download.pl argument call order. See download.pl argument list. Signed-off-by: Florian Eckert <Eckert.Florian@googlemail.com> SVN-Revision: 48427
* build: add locking for downloads (fixes race conditions with multiple variants)Felix Fietkau2016-01-201-1/+1
| | | | | | Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 48416
* build: add @APACHE download facilityFelix Fietkau2016-01-171-1/+1
| | | | | | | | | | The Apache Software Foundation offers diverse download mirros. For packaging Apache software a new alias @APACHE is defined. Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> SVN-Revision: 48270
* build: add a variable pointing to the main openwrt git repositories (useful ↵Felix Fietkau2016-01-041-0/+2
| | | | | | | | if we want to support using a mirror later) Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 48117
* build: add support for choosing a different url filename part than the ↵Felix Fietkau2015-11-221-2/+3
| | | | | | | | output file Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 47591
* include/download.mk: Add download mirrors for tools from GNU Savannah (bug ↵John Crispin2014-10-081-1/+1
| | | | | | | | | | | #15184) I defined a new download method @SAVANNAH in include/download.mk and scripts/download.pl, and converted quilt and qemu to use that method. Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi> SVN-Revision: 42840
* build: add support for packing version control downloads with xzFelix Fietkau2014-04-101-0/+1
| | | | | | Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 40446
* Incorrect checkout of git submodulesJo-Philipp Wich2013-10-101-1/+1
| | | | | | | | | | | When checking out git packages, buildroot doesn't seem to track the revisions correctly of any submodules referenced by that project. As a result, the submodule stays at whatever revision was referenced by the head of the master branch. Running a 'git submodule update' after the checkout fixes this problem. Signed-off-by: Owen Kirby <osk@exegin.com> SVN-Revision: 38359
* include: download.mk: correct tar argument order when packing checkoutsJo-Philipp Wich2012-12-021-2/+2
| | | | | | | GNU TAR v1.26 will fail with "no such file or directory" errors when the output file does not immediately follow the -f switch. SVN-Revision: 34460
* download.mk: refuse to download mirror tarballs from packages using version ↵Felix Fietkau2012-06-061-1/+1
| | | | | | control if no mirror md5sum was provided SVN-Revision: 32079
* Enable recursive download of git sources.Jens Muecke2012-03-171-2/+2
| | | | SVN-Revision: 30967
* add support for md5sum checks for mirrored tarballs of packages with version ↵Felix Fietkau2011-03-211-1/+2
| | | | | | control source urls SVN-Revision: 26260
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-01 16:07:56 +0000