aboutsummaryrefslogtreecommitdiffstats
path: root/scripts/download.pl
Commit message (Collapse)AuthorAgeFilesLines
* scripts/download.pl: fix downloads with wgetPetr Štetiar2022-09-161-2/+8
| | | | | | | | | | | | | | | | | | | | | Several users of wget for downloads (curl is not available in the system) have reported broken download functionality: wget --tries=5 --timeout=20 --output-document=- https://cdn.kernel.org/pub/linux/kernel/v5.x/linux-5.10.142.tar.xz http://: Invalid host name. Thats all happening due to '' was passed as an argument, which got later expanded to http://. In the context of a list constructor '' is not nothing, it is an empty string element. So fix it by using () as it will yield "nothing" and thus not introduce an empty string element. Fixes: #10692 Fixes: 90c6e3aedf16 ("scripts: always check certificates") Signed-off-by: Jo-Philipp Wich <jo@mein.io> [shellwords() -> ()] Signed-off-by: Petr Štetiar <ynezz@true.cz> (cherry picked from commit 50a48faa1b8424e6b4b436b7118fffa2cba14b18)
* scripts/download.pl: silence can't exec curl warningPetr Štetiar2022-09-161-1/+1
| | | | | | | | | | | | When running build in verbose mode `make V=s` we can see a lot of following warnings when curl is not available in the system: Can't exec "curl": No such file or directory at scripts/download.pl line 77. So lets fix it by redirecting of the stderr to null hole. Signed-off-by: Petr Štetiar <ynezz@true.cz> (cherry picked from commit c836ca84e8f641e10a8349a8f9b7432b33d6cec1)
* scripts: always check certificatesJosh Roys2022-09-161-2/+4
| | | | | | | | | | | | | | | | Remove flags from wget and curl instructing them to ignore bad server certificates. Although other mechanisms can protect against malicious modifications of downloads, other vectors of attack may be available to an adversary. TLS certificate verification can be disabled by turning oof the "Enable TLS certificate verification during package download" option enabled by default in the "Global build settings" in "make menuconfig" Signed-off-by: Josh Roys <roysjosh@gmail.com> [ add additional info on how to disable this option ] Signed-off-by: Christian Marangi <ansuelsmth@gmail.com> (cherry picked from commit 90c6e3aedf167b0ae1baf376e7800a631681e69a)
* toolchain: binutils: add support for version 2.38Rui Salvaterra2022-03-091-1/+2
| | | | | | | | | | | | While at it, also fix the download.xs4all.nl and add ftp.gnu.org to the GNU mirrors, as it seems to be updated faster. Deleted (upstreamed): 600-Close_the_file_descriptor.patch [1] [1] https://sourceware.org/git/?p=binutils-gdb.git;a=commit;h=5a98fb7513b559e20dfebdbaa2a471afda3b4742 Signed-off-by: Rui Salvaterra <rsalvaterra@gmail.com>
* treewide: drop use of whichAnsuel Smith2022-01-171-3/+1
| | | | | | | | | | | | | | Ubuntu started to flag which as deprecated and it seems which is not really standard and may vary across Distro. Drop the use of which and use the standard 'command -v' for this simple task. Which is still present in the prereq if some package/script still use which. A utility script called command_all.sh is implemented that will just mimic the output of which -a. Signed-off-by: Ansuel Smith <ansuelsmth@gmail.com>
* build: introduce $(MKHASH)Leonardo Mörlein2021-05-131-3/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this commit, it was assumed that mkhash is in the PATH. While this was fine for the normal build workflow, this led to some issues if make TOPDIR="$(pwd)" -C "$pkgdir" compile was called manually. In most of the cases, I just saw warnings like this: make: Entering directory '/home/.../package/gluon-status-page' bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found bash: line 1: mkhash: command not found [...] While these were only warnings and the package still compiled sucessfully, I also observed that some package even fail to build because of this. After applying this commit, the variable $(MKHASH) is introduced. This variable points to $(STAGING_DIR_HOST)/bin/mkhash, which is always the correct path. Signed-off-by: Leonardo Mörlein <me@irrelefant.net>
* download: add mirror alias for DebianDavid Bauer2021-02-261-0/+4
| | | | | | | Add an alias for Debian packages and download them from the Debian mirror redirector. Signed-off-by: David Bauer <mail@david-bauer.net>
* download: use mirror redirector for GNOME downloadsDavid Bauer2021-02-261-0/+1
| | | | | | | Use the GNOME mirror redirector as the primary download source for GNOME packages. Signed-off-by: David Bauer <mail@david-bauer.net>
* download: remove broken mirrorsDavid Bauer2021-02-221-2/+0
| | | | | | These mirrors don't exist anymore. Remove them. Signed-off-by: David Bauer <mail@david-bauer.net>
* treewide: unify OpenWrt hosted source via @OPENWRTPaul Spooren2021-02-051-0/+2
| | | | | | | | | | | Multiple sources are hosted on OpenWrts source server only. The source URLs to point to the server vary based on different epochs in OpenWrts history. Replace all by @OPENWRT which is an "empty" mirror, therefore using the fallback servers sources.cdn.openwrt.org and sources.openwrt.org. Signed-off-by: Paul Spooren <mail@aparcar.org>
* scripts: sources CDN as fallback in download.plPaul Spooren2021-01-271-1/+1
| | | | | | | | | In case the default sources for a package fail use the CDN rather than our own mirror. In case the CDN fails, fallback to our mirror. Also remove mirror1 which isn't available anymore. Signed-off-by: Paul Spooren <mail@aparcar.org>
* download.pl: properly cleanup intermediate .hash filePetr Štetiar2020-11-271-1/+1
| | | | | | | | | | It seems like after a build the /dl dir seems to now contain a .hash file for each source file due to inproper cleanup so fix it by removing those intermediate files before leaving the download action. Fixes: 4e19cbc55335 ("download: handle possibly invalid local tarballs") Reported-by: Hannu Nyman <hannu.nyman@iki.fi> Signed-off-by: Petr Štetiar <ynezz@true.cz>
* download: handle possibly invalid local tarballsPetr Štetiar2020-11-271-0/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently it's assumed, that already downloaded tarballs are always fine, so no checksum checking is performed and the tarball is used even if it might be corrupted. From now on, we're going to always check the downloaded tarballs before considering them valid. Steps to reproduce: 1. Remove cached tarball rm dl/libubox-2020-08-06-9e52171d.tar.xz 2. Download valid tarball again make package/libubox/download 3. Invalidate the tarball sed -i 's/PKG_MIRROR_HASH:=../PKG_MIRROR_HASH:=ff/' package/libs/libubox/Makefile 4. Now compile with corrupt tarball source make package/libubox/{clean,compile} Signed-off-by: Petr Štetiar <ynezz@true.cz>
* scripts: download.pl: retry download using filenameDavid Bauer2020-11-201-2/+6
| | | | | | | | | | | | | | | | With this commit, the download script will try downloading source files using the filename instead of the url-filename in case the previous download attempt using the url-filename failed. This is required, as the OpenWrt sources mirrors serve files using the filename files might be renamed to after downloading. If the original mirror for a file where url-filename and filename do not match goes down, the download failed prior to this patch. Further improvement can be done by performing this only for the OpenWrt sources mirrors. Signed-off-by: David Bauer <mail@david-bauer.net>
* scripts: download.pl: fix indentationDavid Bauer2020-09-111-3/+2
| | | | Signed-off-by: David Bauer <mail@david-bauer.net>
* Revert "scripts/download: add sources CDN as first mirror"Jo-Philipp Wich2020-08-241-1/+0
| | | | | | | | | | This reverts commit c737a9ee6a9c47b6e553ac81bf293b1161e59799. The source CDN has been discontinued in its current form and will take a while to be reestablished. Even then it makes little sense to put a CDN before other CDNs such as kernel.org, apache.org, sourceforge etc. Signed-off-by: Jo-Philipp Wich <jo@mein.io>
* scripts/download: add sources CDN as first mirrorPaul Spooren2020-04-141-0/+1
| | | | | | | | | | | | | | | | | OpenWrt now has a CDN for sources at sources.cdn.openwrt.org which mirrors sources.openwrt.org. Downloading sources outside Europe or US (mainland) could result in low throughput, extremely slowing down the first compilation of the build system. This patch adds sources.cdn.openwrt.org as the first mirror to offer worldwide fast download speeds by default. If the CDN goes down for whatever reason, the script jumps to the next available mirror and downloads requested files as before (in regional varying speed). Signed-off-by: Paul Spooren <mail@aparcar.org> Acked-by: Eneas U de Queiroz <cotequeiroz@gmail.com>
* scripts/dowload.pl: add archive.apache.org to apache mirror listJiri Kastner2019-12-231-0/+1
| | | | | | | | apache mirrors holds only latest releases, to download older releases, one must use archive.apache.org to get them. Signed-off-by: Jiri Kastner <cz172638@gmail.com>
* download.dl: fix @KERNEL mirror urlsYousong Zhou2019-09-161-4/+4
| | | | | | | - Prepend "/pub" for mirror.rackspace.com - Use https for download.xs4all.nl and mirrors.mit.edu Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* download.pl: use https://source.openwrt.orgYousong Zhou2019-08-181-2/+1
| | | | | | | | https://sources.lede-openwrt.org now redirects to there https://downloads.openwrt.org/sources returns 404, so remove it here Signed-off-by: Yousong Zhou <yszhou4tech@gmail.com>
* download.pl: Change OpenWrt mirrors to HTTPS.Rosen Penev2018-05-011-4/+4
| | | | | | These have supported HTTPS for quite a while. I have not seen any obvious breakage. Signed-off-by: Rosen Penev <rosenp@gmail.com>
* download.pl: Change SourceForge address to HTTPS.Rosen Penev2018-05-011-1/+1
| | | | | | | | SourceForge has supported HTTPS for its downloads for a long time now. I have not been able to see any failures resulting from this change. Signed-off-by: Rosen Penev <rosenp@gmail.com>
* base-files: allow skipping of hash verificationJo-Philipp Wich2017-12-141-1/+1
| | | | | | | | | | | | | | | | | | | | | When calling a download target, hash verification is now completely skipped if we set PKG_HASH=skip. This allows to easily bump package version: $ make package/<mypackage>/download PKG_HASH=skip V=s $ make package/<mypackage>/check FIXUP=1 V=s This will download the new version of the package, and then automatically update PKG_HASH with the hash of the new version. Of course, it is still the responsibility of the packager to ensure that the new tarball is legitimate, because it is downloaded from a possibly untrusted source. Fixes: b30ba14e ("scripts/download.pl: fail loudly if provided hash is unsupported") Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org> Signed-off-by: Jo-Philipp Wich <jo@mein.io> Acked-by: Stijn Tintel <stijn@linux-ipv6.be> Signed-off-by: John Crispin <john@phrozen.org>
* scripts/download.pl: fail loudly if provided hash is unsupportedBaptiste Jonglez2017-09-111-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | Currently, if the provided hash is unsupported (length different from 32 or 64 bytes), we happily download the requested file without any kind of checksum verification. This is quite dangerous and may provide a false sense of security, because a single typo in the hash (e.g. one character deleted by mistake) may skip checksum verification entirely. Instead, fail immediately if we don't support the provided hash. In particular, if an external package repository decides to change the hash algorithm one day, we will now fail loudly instead of skipping checksum verification without complaints. Note: if some users of scripts/download.pl knowingly provide an empty hash because they don't need checksum verification, this change will break them. This does not seem to be the case currently, but if this feature is ever needed, an option should be added to download.pl instead of relying on the hash being empty. Fixes: eaa4eba10a89 ("scripts/download.pl: add SHA-256 support") Signed-off-by: Baptiste Jonglez <git@bitsofnetworks.org>
* scripts/dowload.pl: use glob to expand target dirZoltan Gyarmati2017-08-301-1/+1
| | | | | | | | | | | | | | If CONFIG_DOWNLOAD_FOLDER is set to for example "~/dl", the download script fails to create the .hash and .dl files with the following errors: Cannot create file ~/dl/dropbear-2017.75.tar.bz2.dl: No such file or directory sh: 1: cannot create ~/dl/dropbear-2017.75.tar.bz2.hash: Directory nonexistent If the tarball already exists in the ~/dl dir, it's properly found and used, so this issue only affects the download.pl script. This patch calls glob() on the target dir parameter, which will expand `~`. Signed-off-by: Zoltan Gyarmati <mr.zoltan.gyarmati@gmail.com>
* scripts/download.pl: Adjust URLsDaniel Engberg2017-07-211-5/+5
| | | | | | | | | | | | Internet2 isn't considered a trusted issuer meaning that https links to rit.edu will fail. The host mirror.csclub.uwaterloo.ca has a trusted SSL cert and peering is good so it can replace rit.edu without performance issues. Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net> [Jo-Philipp Wich: rewrapped commit message] Signed-off-by: Jo-Philipp Wich <jo@mein.io>
* scripts/download.pl: print the command used to download filesFelix Fietkau2017-05-251-0/+1
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* scripts/download.pl: fix error message on hash mismatchFelix Fietkau2017-04-041-1/+1
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download.pl: fix detecting download errors with curlFelix Fietkau2017-01-311-1/+1
| | | | | | Avoid treating error pages as successfully downloaded content Signed-off-by: Felix Fietkau <nbd@nbd.name>
* build: use mkhash to replace various quirky md5sum/openssl callsFelix Fietkau2017-01-051-6/+2
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download.pl: use curl in preference to wgetBrian J. Murrell2017-01-031-4/+22
| | | | | | | | | | | | | | | Because wget doesn't know how to do Negotiate authentication with a proxy and curl does, use curl if it's present. The user is expected to have a ~/.curlrc that sets the options necessary for any proxy authentication. A ~/.curlrc is completely optional however and curl will work in exactly the same manner as wget without one. Signed-off-by: Brian J. Murrell <brian@interlinx.bc.ca> [Jo-Philipp Wich: Rework code to detect curl usability by checking --version, Use vararg style open() to bypass the shell when downloading, Use Text::ParseWords to decompose env vars into arguments] Signed-off-by: Jo-Philipp Wich <jo@mein.io>
* download.pl: check for existing file before the first download attemptFelix Fietkau2016-12-161-2/+1
| | | | | | | | | | | | In the build system, flock will prevent multiple concurrent downloads for the same file. However, if one download request for the same file is waiting for another one to finish, it will result in downloading the same file twice consecutively. Prevent this issue by exiting immediately if the file has already been downloaded Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download.pl: Rework URLsDaniel Engberg2016-05-281-15/+32
| | | | | | | | Provide HTTPS URL when possible, try to keep 8 mirrors per entry and spread over several locations of the world. Since most active contributors are in US/CA and/or EU prioritize mirrors that are within those regions if possible. Signed-off-by: Daniel Engberg <daniel.engberg.lists@pyret.net>
* scripts/download.pl: Use CDN for kernel downloadsPetr Štetiar2016-05-231-2/+2
| | | | | | More info at https://www.kernel.org/introducing-fastly-cdn.html Signed-off-by: Petr Štetiar <ynezz@true.cz>
* download.pl: use http://sources.lede-project.org as download mirrorFelix Fietkau2016-05-131-0/+1
| | | | Signed-off-by: Felix Fietkau <nbd@nbd.name>
* download: add @GITHUB download facilityÁlvaro Fernández Rojas2016-04-031-0/+1
| | | | | | Define a new alias (@GITHUB) for downloading raw files from github repos. Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com>
* download: add @GITHUB download facilityJohn Crispin2016-04-091-0/+5
| | | | | | | | Define a new alias (@GITHUB) for downloading raw github repository files Signed-off-by: Álvaro Fernández Rojas <noltari@gmail.com> SVN-Revision: 49138
* scripts/download.pl: fix sha256 hash command (#21931)Felix Fietkau2016-02-271-1/+1
| | | | | | Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 48809
* build: add @APACHE download facilityFelix Fietkau2016-01-171-0/+6
| | | | | | | | | | The Apache Software Foundation offers diverse download mirros. For packaging Apache software a new alias @APACHE is defined. Signed-off-by: Heinrich Schuchardt <xypron.glpk@gmx.de> SVN-Revision: 48270
* download.pl: fix openssl sha256 digest outputFelix Fietkau2016-01-161-1/+1
| | | | | | Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 48259
* scripts/download.pl: add SHA-256 supportFelix Fietkau2016-01-161-20/+38
| | | | | | Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 48253
* scripts/download.pl: cleanup GNOME source location definitionsFelix Fietkau2015-12-101-7/+2
| | | | | | | | | | | Cleanup the @GNOME source download location definitions: * remove dead and stale mirrors * adjust to changes at directory structure * add one new working mirror Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi> SVN-Revision: 47825
* build: add support for choosing a different url filename part than the ↵Felix Fietkau2015-11-221-2/+6
| | | | | | | | output file Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 47591
* script: downlaod: change mirror for kernel.orgHauke Mehrtens2015-09-111-2/+2
| | | | | | | | | kernel.org now suggests a different mirror address. this one also support IPv6 connections and was faster for me. Signed-off-by: Hauke Mehrtens <hauke@hauke-m.de> SVN-Revision: 46875
* scripts/download.pl: add support for adding mirrors through an environment ↵Felix Fietkau2014-12-121-0/+3
| | | | | | | | variable Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 43654
* include/download.mk: Add download mirrors for tools from GNU Savannah (bug ↵John Crispin2014-10-081-0/+5
| | | | | | | | | | | #15184) I defined a new download method @SAVANNAH in include/download.mk and scripts/download.pl, and converted quilt and qemu to use that method. Signed-off-by: Hannu Nyman <hannu.nyman@iki.fi> SVN-Revision: 42840
* scripts/download.pl: prefer the GNU mirror redirect over the primary site ↵Felix Fietkau2013-12-131-1/+1
| | | | | | | | (#14603) Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 39045
* remove broken mirrors for linux kernelMirko Vogt2013-07-141-4/+0
| | | | | | | | | ftp.de.kernel.org / ftp.fr.kernel.org don't resolv anymore, ftp.all.kernel.org is DNS round robin anyway Signed-off-by: Mirko Vogt <mirko@openwrt.org> SVN-Revision: 37323
* scripts/download.pl: use HTTP instead of FTP for the default GNU siteFelix Fietkau2013-07-091-1/+1
| | | | | | Signed-off-by: Felix Fietkau <nbd@openwrt.org> SVN-Revision: 37213
* scripts: update GNU mirrorsLuka Perkov2013-02-281-4/+1
| | | | SVN-Revision: 35838
generated by cgit v1.2.3 (git 2.25.1) at 2024-11-10 20:01:29 +0000