Tue Mar 31 02:10:38 CEST 2009 - MirrorBrain 2.8 - Improvements in the scanner, mainly with regard to the definition of patterns for files (and directories) that are to be included from scanning. Old, hardcoded stuff from the scanner has been removed. Now, excludes can be defined in /etc/mirrorbrain.conf by scan_exclude and scan_exclude_rsync directive. The former takes regular expressions and is effective for FTP and HTTP scans, while the latter takes rsync patterns, which are passed directly to the remote rsync daemon. See http://mirrorbrain.org/archive/mirrorbrain-commits/0140.html for details. This can decrease the size of the database (>20% for openSUSE), and for many mirrors it considerably shortens the scan time. - Fixed a bug where the scanner aborted when encountering filenames in (valid or invalid) UTF-8 encoding. See https://bugzilla.novell.com/show_bug.cgi?id=490009 - Improved the implementation of exclusions as well as the top-level-inclusion pattern, which were not correctly implemented to work in subdir scans. - The documentation was enhanced in some places. - mod_autoindex_mb (which is based on mod_autoindex) was rebased on httpd-2.2.11. - mb dirs: new subcommand for showing directories that the database contains, useful to tune scan exclude patterns. - mb export: implement a new output format, named "vcs". Can be used to commit changes to a subversion repository and get change notifications from it. See http://mirrorbrain.org/archive/mirrorbrain-commits/0152.html - Partial deletions (for subdir scans) have been implemented. - mb list accept --country --region --prefix --as --prio options to influence which details are output by it. - mb file: support for probing files, with optional md5 hash check of the downloaded content. - The latter three changes have already been described in more detail at http://mirrorbrain.org/news_items/2.7_mb_toolchain_work Wed Mar 4 16:40:06 CET 2009 - MirrorBrain 2.7 - Completely reworked the file database. It is 5x faster and one third the size. Instead of a potentially huge relational table including timestamps (48 bytes per row), files and associations are now in a single table, using smallint arrays for the mirror ids. This makes the table 5x faster and 1/3 the size. In addition, we need only a single index on the path, which is a small and very fast b-tree. This also gives us a good search, and the chance to do partial deletions (e.g. for a subtree). - With this change, MySQL is no longer supported. The core, mod_mirrorbrain, would still work fine, but the toolchain around is quite a bit specific to the PostgreSQL database scheme now. If there's interest, MySQL support in the toolchain can be maintained as well. - many little improvements in the toolchain were made. - Notably, the scanner has been improved to be more efficient and give better output. - mirror choice can be influenced for testing with a query parameter (as=), specifying the autonomous system number. Fri Feb 13 08:18:42 CET 2009 - MirrorBrain 2.6 - supports additional, finer mirror selection, based on network topological criteria, network prefix and autonomous system number, using mod_asn and global routing data. Fri Feb 13 02:39:45 CET 2009 - updated database schemes and toolchain -- PostgreSQL support is solid now - work on installation documentation for both MySQL and PostgreSQL (the latter is recommended now, because it allows for nifty features in the future. The mb tool has an export subcommand now, perfect to migrate the database.) - toolchain work Tue Feb 3 10:38:02 CET 2009 - version 2.5 - working on PostgreSQL support - working on the INSTALL documentation - scanner: 0.22 - more efficient SQL statement handling - output much improved - added SQL logging option for debugging - mb (mirrorbrain tool): - bugfix in the file command: make patterns work which have a wildcard as first character. - extend "mb scan" to accept -v and --sql-debug and pass it to the scanner Fri Jan 23 17:20:00 CET 2009 - version 2.4 - rename mod_zrkadlo to mod_mirrorbrain - use mod_geoip for GeoIP lookups, instead of doing it ourselves. We can now use the GeoIP city database for instance - handle satellite "country" called A2 - auto-reenable dead mirrors - geoiplookup_city added, new tool to show details from GeoIP city databases - geoip-lite-update tool updated, with adjusted URL for GeoLite databases. It also downloads the city database now. - deprecate "clientip" query parameter, which can no longer work once we use mod_geoip. Implement 'country" parameter that can be used instead. - make memcache support optional at compile time Sat Dec 13 20:04:53 CET 2008 - add commandline tool to edit marker files. (Marker files are used to generate mirror lists. Each marker file is used to determine whether a mirror mirrors a certain subtree.) Sat Dec 13 13:51:33 CET 2008 - improvements and few features in the toolchain: - the mirrorprobe now does GET requests instead of HEAD requests. - mb, the mirrorbrain tool, has a powerful "probefile" command now that can check for existance of a file on all mirrors, probing all URLs. This is especially useful for checking whether the permission setup for staged content is correct on all mirrors. - new database fields: public_notes, operator_name, operator_url - new database tables: country & region Sat Dec 6 12:15:13 CET 2008 - generate mirror lists Sat Nov 22 15:58:46 CET 2008 - release MirrorBrain 2.2: - simplified database layout, with additional space save. Sun Nov 9 15:20:03 CET 2008 - release MirrorBrain 2.1 - simplified the Apache configuration: It is no longer needed to configure a database query. At the same time it's less error-prone and avoids trouble if one forgets to update the query, when the database schema changes. - specific mirrors can be now configured to get only requests for files < n bytes Mon Nov 3 11:04:04 CET 2008 - release MirrorBrain 2.0 - implement better fallback mirror selection - add tool to list/add/rm files in the mirror database Sun Oct 26 18:07:52 CET 2008 - mod_zrkadlo 1.9: - add bittorrent links (to found .torrent files) into metalinks - embed PGP signatures (.asc files) into metalinks - add configurable CSS stylesheet to mirror lists Fri Sep 19 21:08:42 CET 2008 - implement the redirection exceptions (file too small, mime type not allowed to be redirected etc) for transparently negotiated metalinks. - add Vary header on all transparently negotiated resources. - new, better implementation of rsyncusers tool - improved mirrordoctor tool - bugfixes in the scanner, mainly for scanning via HTML - installation instructions updated Fri Aug 22 17:22:14 CEST 2008 - allow to use the apache module and all tools with multiple instances of the mirrorbrain. Now, one machine / one Apache can host multiple separate instances, each in a vhost. - a number of small bugs in the tools were fixed and several improvements added. Wed Jun 11 01:20:12 CEST 2008 - added "mirrordoctor", a commandline tool to maintain mirror entries in the database. Finally! Mon Jun 2 17:30:45 CEST 2008 - mod_zrkadlo 1.8: use mod_memcache for the configuration and initialization of memcache - metalink-hasher script added, to prepare hashes for injection into metalink files - rsyncusers analysis tool added - scanner bugfix regarding following redirects for large file checks Fri May 2 18:54:09 CEST 2008 - failover testbed for text mirrorlists implemented - rsyncinfo tool added Wed Apr 30 19:34:31 CEST 2008 - metalinks: switch back to RFC822 format - new ZrkadloMetalinkPublisher directive - fix issue with element Sun Apr 27 23:02:42 CEST 2008 - now there is another (more natural) way to request a metalink: by appending ".metalink" to the filename. Sun Apr 27 12:06:47 CEST 2008 - change metalink negotiation to look for application/metalink+xml in the Accept header (keep Accept-Features for now, but it is going to be removed probably) Mon Apr 21 17:35:27 CEST 2008 - new terse text-based mirrorlist - allow clients to use RFC2295 Accept-Features header to select variants (metalink or mirrorlist-txt) Mon Apr 21 01:35:31 CEST 2008 metalink hash includes can now be out-of-tree Sat Apr 19 18:21:52 CEST 2008 mod_autoindex_mb Sat Apr 19 18:21:52 CEST 2008 content-disposition header