imacat/arclog
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Compare commits

base: imacat:main
Branches Tags
imacat:main
..
compare: imacat:3c7030099be0393df9dae187751e3a06011f244d
Branches Tags
imacat:main

No commits in common. "main" and "3c7030099be0393df9dae187751e3a06011f244d" have entirely different histories.
main ... 3c7030099b

10 changed files with 243 additions and 881 deletions
Show Stats Expand all files Collapse all files

5
Build.PL
View File

@ -62,9 +62,8 @@ my $build = Module::Build->new(
}, },
recommends => { recommends => {
"File::MMagic" => 0, "File::MMagic" => 0,
"IO::Compress::Gzip" => 0, "Compress::Zlib" => 0,
"IO::Compress::Bzip2" => 2, "Compress::Bzip2" => 2,
"IO::Compress::Xz" => 0,
"Term::ReadKey" => 0, "Term::ReadKey" => 0,
}, },
build_requires => { build_requires => {

7
Changes
View File

@ -1,12 +1,5 @@
arclog change log arclog change log
2022-03-19 version 3.06
1. Added support for the xz compression.
2. Replaced Compress::Zlib with IO::Compress::Gzip and
IO::Uncompress::Gunzip.
3. Replaced Compress::Bzip2 with IO::Compress::Bzip2 and
IO::Uncompress::Bunzip2.
2020-02-06 version 3.05 2020-02-06 version 3.05
1. Replaced the messy GNU-styled, multi-lingual documents with a 1. Replaced the messy GNU-styled, multi-lingual documents with a
single Markdown README.md. single Markdown README.md.

5
Makefile.PL
View File

@ -52,9 +52,8 @@ EOT
our (%OPT_PREREQ); our (%OPT_PREREQ);
%OPT_PREREQ = ( %OPT_PREREQ = (
"File::MMagic" => 0, "File::MMagic" => 0,
"IO::Compress::Gzip" => 0, "Compress::Zlib" => 0,
"IO::Compress::Bzip2" => 2, "Compress::Bzip2" => 2,
"IO::Compress::Xz" => 0,
"Term::ReadKey" => 0, "Term::ReadKey" => 0,
); );
delete $OPT_PREREQ{$_} delete $OPT_PREREQ{$_}

142
README.md
View File

@ -12,9 +12,9 @@ space and prevents potential attacks on log files.
Currently, `arclog` supports [Apache] access log, Syslog, [NTP], Currently, `arclog` supports [Apache] access log, Syslog, [NTP],
Apache 1 SSL engine log, and my own bracketed, modified ISO date/time Apache 1 SSL engine log, and my own bracketed, modified ISO date/time
log file formats, and gzip, bzip2, and xz compression methods. log file formats, and gzip and bzip2 compression methods. Several
Several software projects log (or can log) in a format compatible with software projects log (or can log) in a format compatible with the
the Apache access log, like [CUPS], [ProFTPD], [Pure-FTPd]… etc., and Apache access log, like [CUPS], [ProFTPD], [Pure-FTPd]… etc., and
`arclog` can archive their Apache-like log files, too. `arclog` can archive their Apache-like log files, too.
Caution Caution
@ -84,10 +84,10 @@ Caution
* I suggest that you install [File::MMagic] instead of counting on the * I suggest that you install [File::MMagic] instead of counting on the
`file` executable. The internal magic file of File::MMagic works `file` executable. The internal magic file of File::MMagic works
better than the `file` executable. `arclog` treats everything not better than the `file` executable. `arclog` treats everything not
gzip, bzip2, or xz compressed as plain text. When a compressed log gzip nor bzip2 compressed as plain text. When a compressed log file
file is wrongly recognized as an image, `arclog` treats it as plain is wrongly recognized as an image, `arclog` treats it as plain text,
text, reads directly from it, and fails. This does not hurt the reads directly from it, and fails. This does not hurt the source
source log files, but is still annoying. log files, but is still annoying.
[Date::Parse]: https://metacpan.org/release/TimeDate [Date::Parse]: https://metacpan.org/release/TimeDate
[File::MMagic]: https://metacpan.org/release/File-MMagic [File::MMagic]: https://metacpan.org/release/File-MMagic
@ -174,23 +174,22 @@ System Requirement
the [GnuWin32] home page. Be sure to save it as `file.exe` the [GnuWin32] home page. Be sure to save it as `file.exe`
somewhere in your `PATH`. somewhere in your `PATH`.
* [IO::Compress::Gzip] and [IO::Uncompress::Gunzip] * [Compress::Zlib]
They are used to support reading/writing the gzip compressed This is used to support reading/writing the gzip compressed
files. It is only needed when gzip compressed files are files. It is only needed when gzip compressed files are
encountered. If they are not available, `arclog` tries the encountered. If it is not available, `arclog` tries the `gzip`
`gzip` executable instead. If that is not available, too, executable instead. If that is not available, too, `arclog`
`arclog` fails. You should not worry about IO::Compress::Gzip fails. You should not worry about Compress::Zlib since it comes
since it comes with Perl since version 5.9.3. If not, it is with Perl since version 5.9.3. If not, it is contained in the
contained in the [IO-Compress] distribution. You can download [IO-Compress] distribution. You can download and install it from
and install it from the CPAN archive, or install it with the the CPAN archive, or install it with the CPAN shell:
CPAN shell:
cpan IO::Compress::Gzip cpan Compress::Zlib
or with the CPANPLUS shell: or with the CPANPLUS shell:
cpanp i IO::Compress::Gzip cpanp i Compress::Zlib
For Debian/Ubuntu: For Debian/Ubuntu:
@ -212,80 +211,43 @@ System Requirement
[the gzip website]. Be sure to save it as `gzip.exe` somewhere [the gzip website]. Be sure to save it as `gzip.exe` somewhere
in your `PATH`. in your `PATH`.
* [IO::Compress::Bzip2] and [IO::Uncompress::Bunzip2] * [Compress::Bzip2] version 2 or above.
They are used to support reading/writing the bzip2 compressed This is used to support reading/writing the bzip2 compressed
files. They are only needed when bzip2 compressed files are files. It is only needed when bzip2 compressed files are
encountered. If they are not available, `arclog` tries the encountered. If it is not available, `arclog` tries the `bzip2`
`bzip2` executable instead. If that is not available, too, executable instead. If that is not available, too, `arclog`
`arclog` fails. You should not worry about IO::Compress::Bzip2 fails. Notice that older versions before 2 does not work, since
since it comes with Perl since version 5.10.1. If not, it is the file I/O compression was not implemented yet. You can
contained in the [IO-Compress] distribution. You can download download and install Compress::Bzip2 from the CPAN archive, or
and install it from the CPAN archive, or install it with the install it with the CPAN shell:
CPAN shell:
cpan IO::Compress::Bzip2 cpan Compress::Bzip2
or with the CPANPLUS shell: or with the CPANPLUS shell:
cpanp i IO::Compress::Bzip2 cpanp i Compress::Bzip2
For Debian/Ubuntu: For Debian/Ubuntu:
sudo apt install libio-compress-perl sudo apt install libcompress-bzip2-perl
For Red Hat/Fedora/CentOS: For Red Hat/Fedora/CentOS:
sudo yum install perl-IO-Compress sudo yum install perl-Compress-Bzip2
For FreeBSD: For FreeBSD:
ports install p5-IO-Compress ports install p5-Compress-Bzip2
For ActivePerl: For ActivePerl:
ppm install IO-Compress ppm install Compress-Bzip2
The alternative `bzip2.exe` for MS-Windows can be obtained from The alternative `bzip2.exe` for MS-Windows can be obtained from
[the bzip2 website]. Be sure to save it as `bzip2.exe` somewhere [the bzip2 website]. Be sure to save it as `bzip2.exe` somewhere
in your `PATH`. in your `PATH`.
* [IO::Compress::Xz] and [IO::Uncompress::UnXz]
They are used to support reading/writing the xz compressed
files. It is only needed when xz compressed files are
encountered. If it is not available, `arclog` tries the `xz`
executable instead. If that is not available, too, `arclog`
fails. They are contained in the [IO-Compress-Lzma] distribution.
You can download and install it from the CPAN archive, or install
them with the CPAN shell:
cpan IO::Compress::Xz
or with the CPANPLUS shell:
cpanp i IO::Compress::Xz
For Debian/Ubuntu:
sudo apt install libio-compress-lzma-perl
For Red Hat/Fedora/CentOS:
sudo yum install perl-IO-Compress-Lzma
For FreeBSD:
ports install p5-IO-Compress-Lzma
For ActivePerl:
ppm install IO-Compress-Lzma
The alternative `xz.exe` for MS-Windows can be obtained from
[the XZ Utils website]. Be sure to save it as `xz.exe`
somewhere in your `PATH`.
* [Term::ReadKey] * [Term::ReadKey]
This is used to display the progress bar. The progress bar is a This is used to display the progress bar. The progress bar is a
@ -322,17 +284,11 @@ System Requirement
[Date::Parse]: https://metacpan.org/pod/Date::Parse [Date::Parse]: https://metacpan.org/pod/Date::Parse
[File::MMagic]: https://metacpan.org/pod/File::MMagic [File::MMagic]: https://metacpan.org/pod/File::MMagic
[GnuWin32]: http://gnuwin32.sourceforge.net [GnuWin32]: http://gnuwin32.sourceforge.net
[IO::Compress::Gzip]: https://metacpan.org/pod/IO::Compress::Gzip [Compress::Zlib]: https://metacpan.org/pod/Compress::Zlib
[IO::Uncompress::Gunzip]: https://metacpan.org/pod/IO::Uncompress::Gunzip
[the gzip website]: https://www.gzip.org [the gzip website]: https://www.gzip.org
[IO-Compress]: https://metacpan.org/dist/IO-Compress [IO-Compress]: https://metacpan.org/release/IO-Compress
[IO::Compress::Bzip2]: https://metacpan.org/pod/IO::Compress::Bzip2 [Compress::Bzip2]: https://metacpan.org/pod/Compress::Bzip2
[IO::Uncompress::Bunzip2]: https://metacpan.org/pod/IO::Uncompress::Bunzip2
[the bzip2 website]: http://www.bzip.org [the bzip2 website]: http://www.bzip.org
[IO::Compress::Xz]: https://metacpan.org/pod/IO::Compress::Xz
[IO::Uncompress::UnXz]: https://metacpan.org/pod/IO::Uncompress::UnXz
[IO-Compress-Lzma]: https://metacpan.org/dist/IO-Compress-Lzma
[the XZ Utils website]: https://tukaani.org/xz/
[Term::ReadKey]: https://metacpan.org/pod/Term::ReadKey [Term::ReadKey]: https://metacpan.org/pod/Term::ReadKey
@ -474,8 +430,8 @@ Options
* `logfile` * `logfile`
The log file to be archived. Specify `-` to read from `STDIN`. The log file to be archived. Specify `-` to read from `STDIN`.
You can specify multiple log files. `gzip`, `bzip2`, or `xz` You can specify multiple log files. `gzip` or `bzip2` compressed
compressed files are supported. files are supported.
* `output` * `output`
@ -496,25 +452,17 @@ Options
* `g`, `gzip` * `g`, `gzip`
Compress with `gzip`. This is the default. `arclog` can use Compress with `gzip`. This is the default. `arclog` can use
`IO::Compress::Gzip` to compress instead of calling `gzip`. `Compress::Zlib` to compress instead of calling `gzip`. This can
This can be safer and faster for not calling foreign binaries. be safer and faster for not calling foreign binaries. If
If `IO::Compress::Gzip` is not installed, it tries `gzip` `Compress::Zlib` is not installed, it tries `gzip` instead. If
instead. If `gzip` is not available, either, it fails. `gzip` is not available, either, it fails.
* `b`, `bzip2` * `b`, `bzip2`
Compress with `bzip2`. `arclog` can use `IO::Compress::Bzip2` Compress with `bzip2`. `arclog` can use `Compress::Bzip2` to
to compress instead of calling `bzip2`. This can be safer and compress instead of calling `bzip2`. This can be safer and faster
faster for not calling foreign binaries. If for not calling foreign binaries. If `Compress::Bzip2` is not
`IO::Compress::Bzip2` is not installed, it tries `bzip2` installed, it tries `bzip2` instead. If `bzip2` is not available,
instead. If `bzip2` is not available, either, it fails.
* `x`, `xz`
Compress with `xz`. `arclog` can use `IO::Compress::Xz` to
compress instead of calling `xz`. This can be safer and faster
for not calling foreign binaries. If `IO::Compress::Xz` is not
installed, it tries `xz` instead. If `xz` is not available,
either, it fails. either, it fails.
* `n`, `none` * `n`, `none`

715
arclog
View File

File diff suppressed because it is too large Load Diff

6
t/01-exhaust.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w #! /usr/bin/perl -w
# Test all the possible combination of options # Test all the possible combination of options
# Copyright (c) 2007-2022 imacat. # Copyright (c) 2007-2021 imacat.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -21,7 +21,7 @@ use warnings;
use diagnostics; use diagnostics;
use Test; use Test;
BEGIN { plan tests => 3600 } BEGIN { plan tests => 2160 }
use File::Basename qw(basename); use File::Basename qw(basename);
use File::Path qw(mkpath rmtree); use File::Path qw(mkpath rmtree);
@ -35,7 +35,7 @@ $WORKDIR = catdir($FindBin::Bin, "logs");
$arclog = catfile($FindBin::Bin, updir, "blib", "script", "arclog"); $arclog = catfile($FindBin::Bin, updir, "blib", "script", "arclog");
$tno = 0; $tno = 0;
# 1-3600: All possible option combinations # 1-2160: All possible option combinations
# Test each log file format # Test each log file format
foreach my $fmt (@LOG_FORMATS) { foreach my $fmt (@LOG_FORMATS) {
# Test each source log file type # Test each source log file type

33
t/02-several.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w #! /usr/bin/perl -w
# Test archiving several log files at once # Test archiving several log files at once
# Copyright (c) 2007-2022 imacat. # Copyright (c) 2007-2021 imacat.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -21,7 +21,7 @@ use warnings;
use diagnostics; use diagnostics;
use Test; use Test;
BEGIN { plan tests => 25 } BEGIN { plan tests => 16 }
use File::Basename qw(basename); use File::Basename qw(basename);
use File::Path qw(mkpath rmtree); use File::Path qw(mkpath rmtree);
@ -35,15 +35,14 @@ $WORKDIR = catdir($FindBin::Bin, "logs");
$arclog = catfile($FindBin::Bin, updir, "blib", "script", "arclog"); $arclog = catfile($FindBin::Bin, updir, "blib", "script", "arclog");
$tno = 0; $tno = 0;
# 1-25: Archiving several log files at once # 1-16: Archiving several log files at once
foreach my $rt (@RESULT_TYPES) { foreach my $rt (@RESULT_TYPES) {
my $skip; my $skip;
$skip = 0; $skip = 0;
# 1: Source log files listed as the arguments # 1: Source log files listed as the arguments
$_ = eval { $_ = eval {
if ( ($$rt{"type"} eq TYPE_GZIP && has_no_gzip) if ( ($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)) {
|| ($$rt{"type"} eq TYPE_XZ && has_no_xz)) {
$skip = 1; $skip = 1;
return; return;
} }
@ -57,15 +56,13 @@ foreach my $rt (@RESULT_TYPES) {
$title = join ", ", "several log files", "all listed as arguments", $title = join ", ", "several log files", "all listed as arguments",
$$fmt{"title"}, $$rt{"title"}; $$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files # (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2) $_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
+ (has_no_xz? 0: 2);
$num = $_ + int rand $_; $num = $_ + int rand $_;
my %types = qw(); my %types = qw();
# At least 2 files for each available compression # At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) { foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip) next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st; $types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
@ -76,8 +73,7 @@ foreach my $rt (@RESULT_TYPES) {
do { do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES]; $types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip) } until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
} }
@st = map $types{$_}, (0...$num-1); @st = map $types{$_}, (0...$num-1);
@fs = qw(); @fs = qw();
@ -150,17 +146,15 @@ foreach my $rt (@RESULT_TYPES) {
skip($skip, $_, 1, $@); skip($skip, $_, 1, $@);
clean_up $_ || $skip, $WORKDIR, ++$tno; clean_up $_ || $skip, $WORKDIR, ++$tno;
# 2-5: One of the source log files is read from STDIN # 2-4: One of the source log files is read from STDIN
# The file type at STDIN # The file type at STDIN
foreach my $st_stdin (@SOURCE_TYPES) { foreach my $st_stdin (@SOURCE_TYPES) {
$skip = 0; $skip = 0;
$_ = eval { $_ = eval {
if ( ($$rt{"type"} eq TYPE_GZIP && has_no_gzip) if ( ($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$rt{"type"} eq TYPE_XZ && has_no_xz)
|| ($$st_stdin{"type"} eq TYPE_GZIP && has_no_gzip) || ($$st_stdin{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st_stdin{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$st_stdin{"type"} eq TYPE_BZIP2 && has_no_bzip2)) {
|| ($$st_stdin{"type"} eq TYPE_XZ && has_no_xz)) {
$skip = 1; $skip = 1;
return; return;
} }
@ -174,15 +168,13 @@ foreach my $rt (@RESULT_TYPES) {
$title = join ", ", "several log files", "one read from STDIN", $title = join ", ", "several log files", "one read from STDIN",
"STDIN " . $$st_stdin{"title"}, $$rt{"title"}; "STDIN " . $$st_stdin{"title"}, $$rt{"title"};
# (2-4 times available compression) log files # (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2) $_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
+ (has_no_xz? 0: 2);
$num = $_ + int rand $_; $num = $_ + int rand $_;
my %types = qw(); my %types = qw();
# At least 2 files for each available compression # At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) { foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip) next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st; $types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
@ -193,8 +185,7 @@ foreach my $rt (@RESULT_TYPES) {
do { do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES]; $types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip) } until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
} }
# Choose the STDIN from the matching compression # Choose the STDIN from the matching compression
@_ = grep ${$types{$_}}{"type"} eq $$st_stdin{"type"}, (0...$num-1); @_ = grep ${$types{$_}}{"type"} eq $$st_stdin{"type"}, (0...$num-1);

26
t/03-fallback.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w #! /usr/bin/perl -w
# Test fallback behavior # Test fallback behavior
# Copyright (c) 2007-2022 imacat. # Copyright (c) 2007-2021 imacat.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -50,21 +50,18 @@ foreach my $kt (@KEEP_MODES) {
do { do {
$rt = $RESULT_TYPES[int rand @RESULT_TYPES]; $rt = $RESULT_TYPES[int rand @RESULT_TYPES];
} until !($$rt{"type"} eq TYPE_GZIP && has_no_gzip) } until !($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
&& !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !($$rt{"type"} eq TYPE_XZ && has_no_xz);
$title = join ", ", "STDIN keep fall back", $$kt{"title"}, $title = join ", ", "STDIN keep fall back", $$kt{"title"},
$$fmt{"title"}, $$rt{"title"}; $$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files # (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2) $_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
+ (has_no_xz? 0: 2);
$num = $_ + int rand $_; $num = $_ + int rand $_;
$stdin = int rand $num; $stdin = int rand $num;
my %types = qw(); my %types = qw();
# At least 2 files for each available compression # At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) { foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip) next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st; $types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
@ -75,8 +72,7 @@ foreach my $kt (@KEEP_MODES) {
do { do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES]; $types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip) } until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
} }
@st = map $types{$_}, (0...$num-1); @st = map $types{$_}, (0...$num-1);
@fs = qw(); @fs = qw();
@ -164,21 +160,18 @@ $_ = eval {
do { do {
$rt = $RESULT_TYPES[int rand @RESULT_TYPES]; $rt = $RESULT_TYPES[int rand @RESULT_TYPES];
} until !($$rt{"type"} eq TYPE_GZIP && has_no_gzip) } until !($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
&& !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !($$rt{"type"} eq TYPE_XZ && has_no_xz);
$title = join ", ", "STDIN override ask fall back", $title = join ", ", "STDIN override ask fall back",
$$fmt{"title"}, $$rt{"title"}; $$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files # (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2) $_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
+ (has_no_xz? 0: 2);
$num = $_ + int rand $_; $num = $_ + int rand $_;
$stdin = int rand $num; $stdin = int rand $num;
my %types = qw(); my %types = qw();
# At least 2 files for each available compression # At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) { foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip) next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st; $types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
@ -189,8 +182,7 @@ $_ = eval {
do { do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES]; $types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip) } until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
} }
@st = map $types{$_}, (0...$num-1); @st = map $types{$_}, (0...$num-1);
@fs = qw(); @fs = qw();

11
t/04-errors.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w #! /usr/bin/perl -w
# Test the errors that should be captured. # Test the errors that should be captured.
# Copyright (c) 2007-2022 imacat. # Copyright (c) 2007-2021 imacat.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -140,15 +140,13 @@ $_ = eval {
$title = join ",", "A same log file is specified more than once", $title = join ",", "A same log file is specified more than once",
$$fmt{"title"}, $$rt{"title"}; $$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files # (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2) $_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
+ (has_no_xz? 0: 2);
$num = $_ + int rand $_; $num = $_ + int rand $_;
my %types = qw(); my %types = qw();
# At least 2 files for each available compression # At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) { foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip) next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2) || ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st; $types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1); @_ = grep !exists $types{$_}, (0...$num-1);
@ -159,8 +157,7 @@ $_ = eval {
do { do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES]; $types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip) } until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2) && !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
} }
@st = map $types{$_}, (0...$num-1); @st = map $types{$_}, (0...$num-1);
@fs = qw(); @fs = qw();

174
t/_helper.pm
View File

@ -1,6 +1,6 @@
# _helper.pm - A simple test suite helper # _helper.pm - A simple test suite helper
# Copyright (c) 2007-2022 imacat. # Copyright (c) 2007-2021 imacat.
# #
# Licensed under the Apache License, Version 2.0 (the "License"); # Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License. # you may not use this file except in compliance with the License.
@ -24,12 +24,12 @@ $VERSION = "0.01";
@EXPORT = qw( @EXPORT = qw(
read_file read_raw_file write_file write_raw_file read_file read_raw_file write_file write_raw_file
run_cmd where_is file_type list_files preserve_source clean_up run_cmd where_is file_type list_files preserve_source clean_up
has_no_file has_no_gzip has_no_bzip2 has_no_xz has_no_file has_no_gzip has_no_bzip2
make_log_file make_log_file
make_apache_log_file make_syslog_log_file make_apache_log_file make_syslog_log_file
make_ntp_log_file make_apache_ssl_log_file make_modified_iso_log_file make_ntp_log_file make_apache_ssl_log_file make_modified_iso_log_file
random_word random_word
TYPE_TEXT TYPE_GZIP TYPE_BZIP2 TYPE_XZ TYPE_TEXT TYPE_GZIP TYPE_BZIP2
@LOG_FORMATS @SOURCE_TYPES @RESULT_TYPES @KEEP_MODES @OVERRIDE_MODES); @LOG_FORMATS @SOURCE_TYPES @RESULT_TYPES @KEEP_MODES @OVERRIDE_MODES);
# Prototype declaration # Prototype declaration
sub this_file(); sub this_file();
@ -46,7 +46,6 @@ sub clean_up($$$);
sub has_no_file(); sub has_no_file();
sub has_no_gzip(); sub has_no_gzip();
sub has_no_bzip2(); sub has_no_bzip2();
sub has_no_xz();
sub make_log_file($$$@); sub make_log_file($$$@);
sub make_apache_log_file($;$); sub make_apache_log_file($;$);
sub make_syslog_log_file($;$); sub make_syslog_log_file($;$);
@ -72,17 +71,15 @@ use File::Temp qw(tempfile);
use Time::Local qw(timelocal); use Time::Local qw(timelocal);
$Data::Dumper::Indent = 1; $Data::Dumper::Indent = 1;
our (%WHERE_IS, $HAS_NO_FILE, $HAS_NO_GZIP, $HAS_NO_BZIP2, $HAS_NO_XZ); our (%WHERE_IS, $HAS_NO_FILE, $HAS_NO_GZIP, $HAS_NO_BZIP2);
%WHERE_IS = qw(); %WHERE_IS = qw();
undef $HAS_NO_FILE; undef $HAS_NO_FILE;
undef $HAS_NO_GZIP; undef $HAS_NO_GZIP;
undef $HAS_NO_BZIP2; undef $HAS_NO_BZIP2;
undef $HAS_NO_XZ;
use constant TYPE_TEXT => "text/plain"; use constant TYPE_TEXT => "text/plain";
use constant TYPE_GZIP => "application/x-gzip"; use constant TYPE_GZIP => "application/x-gzip";
use constant TYPE_BZIP2 => "application/x-bzip2"; use constant TYPE_BZIP2 => "application/x-bzip2";
use constant TYPE_XZ => "application/x-xz";
our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES); our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES);
# All the log format information # All the log format information
@ -110,11 +107,7 @@ our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES);
{ "title" => "bzip2 source", { "title" => "bzip2 source",
"type" => TYPE_BZIP2, "type" => TYPE_BZIP2,
"suf" => ".bz2", "suf" => ".bz2",
"skip" => has_no_bzip2, }, "skip" => has_no_bzip2, }, );
{ "title" => "xz source",
"type" => TYPE_XZ,
"suf" => ".xz",
"skip" => has_no_xz, }, );
# All the result type information # All the result type information
@RESULT_TYPES = ( @RESULT_TYPES = (
{ "title" => "default compress", { "title" => "default compress",
@ -132,11 +125,6 @@ our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES);
"suf" => ".bz2", "suf" => ".bz2",
"skip" => has_no_bzip2, "skip" => has_no_bzip2,
"opts" => [qw(-c b)], }, "opts" => [qw(-c b)], },
{ "title" => "xz compress",
"type" => TYPE_XZ,
"suf" => ".xz",
"skip" => has_no_xz,
"opts" => [qw(-c x)], },
{ "title" => "no compress", { "title" => "no compress",
"type" => TYPE_TEXT, "type" => TYPE_TEXT,
"suf" => "", "suf" => "",
@ -227,21 +215,29 @@ sub read_file($) {
# a gzip compressed file # a gzip compressed file
if ($file =~ /\.gz$/) { if ($file =~ /\.gz$/) {
# IO::Uncompress::Gunzip # Compress::Zlib
if (eval { require IO::Uncompress::Gunzip; 1; }) { if (eval { require Compress::Zlib;
my $gz; Compress::Zlib->import(qw(gzopen));
1; }) {
use Compress::Zlib qw(gzopen);
my ($FH, $gz);
$content = ""; $content = "";
$gz = IO::Uncompress::Gunzip->new($file) open $FH, $file or die this_file . ": $file: $!";
or die this_file . ": $file: $IO::Uncompress::Gunzip::GunzipError"; $gz = gzopen($FH, "rb") or die this_file . ": $file: $!";
$content = join "", <$gz>; while (1) {
$gz->close or die this_file . ": $file: $IO::Uncompress::Gunzip::GunzipError"; ($gz->gzread($_, 10240) != -1)
or die this_file . ": $file: " . $gz->gzerror;
$content .= $_;
last if length $_ < 10240;
}
$gz->gzclose and die this_file . ": $file: " . $gz->gzerror;
return $content; return $content;
# gzip executable # gzip executable
} else { } else {
my ($PH, $CMD); my ($PH, $CMD);
$CMD = where_is "gzip"; $CMD = where_is "gzip";
$CMD = "\"$CMD\" -cdf \"$file\""; $CMD = "\"$CMD\" -cd \"$file\"";
open $PH, "$CMD |" or die this_file . ": $CMD: $!"; open $PH, "$CMD |" or die this_file . ": $CMD: $!";
$content = join "", <$PH>; $content = join "", <$PH>;
close $PH or die this_file . ": $CMD: $!"; close $PH or die this_file . ": $CMD: $!";
@ -250,44 +246,29 @@ sub read_file($) {
# a bzip compressed file # a bzip compressed file
} elsif ($file =~ /\.bz2$/) { } elsif ($file =~ /\.bz2$/) {
# IO::Uncompress::Bunzip2 # Compress::Bzip2
if (eval { require IO::Uncompress::Bunzip2; 1; }) { if (eval { require Compress::Bzip2;
my $bz; Compress::Bzip2->import(2.00);
Compress::Bzip2->import(qw(bzopen));
1; }) {
my ($FH, $bz);
$content = ""; $content = "";
$bz = IO::Uncompress::Bunzip2->new($file) open $FH, $file or die this_file . ": $file: $!";
or die this_file . ": $file: $IO::Uncompress::Bunzip2::Bunzip2Error"; $bz = bzopen($FH, "rb") or die this_file . ": $file: $!";
$content = join "", <$bz>; while (1) {
$bz->close or die this_file . ": $file: $IO::Uncompress::Bunzip2::Bunzip2Error"; ($bz->bzread($_, 10240) != -1)
or die this_file . ": $file: " . $bz->bzerror;
$content .= $_;
last if length $_ < 10240;
}
$bz->bzclose and die this_file . ": $file: " . $bz->bzerror;
return $content; return $content;
# bzip2 executable # bzip2 executable
} else { } else {
my ($PH, $CMD); my ($PH, $CMD);
$CMD = where_is "bzip2"; $CMD = where_is "bzip2";
$CMD = "\"$CMD\" -cdf \"$file\""; $CMD = "bzip2 -cd \"$file\"";
open $PH, "$CMD |" or die this_file . ": $CMD: $!";
$content = join "", <$PH>;
close $PH or die this_file . ": $CMD: $!";
return $content;
}
# an xz compressed file
} elsif ($file =~ /\.xz$/) {
# IO::Uncompress::UnXz
if (eval { require IO::Uncompress::UnXz; 1; }) {
my $xz;
$content = "";
$xz = IO::Uncompress::UnXz->new($file)
or die this_file . ": $file: $IO::Uncompress::UnXz::UnXzError";
$content = join "", <$xz>;
$xz->close or die this_file . ": $file: $IO::Uncompress::UnXz::UnXzError";
return $content;
# xz executable
} else {
my ($PH, $CMD);
$CMD = where_is "xz";
$CMD = "\"$CMD\" -cdf \"$file\"";
open $PH, "$CMD |" or die this_file . ": $CMD: $!"; open $PH, "$CMD |" or die this_file . ": $CMD: $!";
$content = join "", <$PH>; $content = join "", <$PH>;
close $PH or die this_file . ": $CMD: $!"; close $PH or die this_file . ": $CMD: $!";
@ -330,14 +311,16 @@ sub write_file($$) {
# a gzip compressed file # a gzip compressed file
if ($file =~ /\.gz$/) { if ($file =~ /\.gz$/) {
# IO::Compress::Gzip # Compress::Zlib
if (eval { require IO::Compress::Gzip; 1; }) { if (eval { require Compress::Zlib;
my $gz; Compress::Zlib->import(qw(gzopen));
$gz = IO::Compress::Gzip->new($file, -Level => 9) 1; }) {
or die this_file . ": $file: $IO::Compress::Gzip::GzipError"; my ($FH, $gz);
($gz->write($content) == length $content) open $FH, ">$file" or die this_file . ": $file: $!";
or die this_file . ": $file: $IO::Compress::Gzip::GzipError"; $gz = gzopen($FH, "wb9") or die this_file . ": $file: $!";
$gz->close or die this_file . ": $file: $IO::Compress::Gzip::GzipError"; ($gz->gzwrite($content) == length $content)
or die this_file . ": $file: " . $gz->gzerror;
$gz->gzclose and die this_file . ": $file: " . $gz->gzerror;
return; return;
# gzip executable # gzip executable
@ -353,44 +336,24 @@ sub write_file($$) {
# a bzip compressed file # a bzip compressed file
} elsif ($file =~ /\.bz2$/) { } elsif ($file =~ /\.bz2$/) {
# IO::Compress::Bzip2 # Compress::Bzip2
if (eval { require IO::Compress::Bzip2; 1; }) { if (eval { require Compress::Bzip2;
my $bz; Compress::Bzip2->import(2.00);
$bz = IO::Compress::Bzip2->new($file, BlockSize100K => 9) Compress::Bzip2->import(qw(bzopen));
or die this_file . ": $file: $IO::Compress::Bzip2::Bzip2Error"; 1; }) {
($bz->write($content) == length $content) my ($FH, $bz);
or die this_file . ": $file: $IO::Compress::Bzip2::Bzip2Error"; open $FH, ">$file" or die this_file . ": $file: $!";
$bz->close or die this_file . ": $file: $IO::Compress::Bzip2::Bzip2Error"; $bz = bzopen($FH, "wb9") or die this_file . ": $file: $!";
($bz->bzwrite($content, length $content) == length $content)
or die this_file . ": $file: " . $bz->bzerror;
$bz->bzclose and die this_file . ": $file: " . $bz->bzerror;
return; return;
# bzip2 executable # bzip2 executable
} else { } else {
my ($PH, $CMD); my ($PH, $CMD);
$CMD = where_is "bzip2"; $CMD = where_is "bzip2";
$CMD = "\"$CMD\" -c9f > \"$file\""; $CMD = "\"$CMD\" -9f > \"$file\"";
open $PH, "| $CMD" or die this_file . ": $CMD: $!";
print $PH $content or die this_file . ": $CMD: $!";
close $PH or die this_file . ": $CMD: $!";
return;
}
# an xz compressed file
} elsif ($file =~ /\.xz$/) {
# IO::Compress::Xz
if (eval { require IO::Compress::Xz; 1; }) {
my $xz;
$xz = IO::Compress::Xz->new($file, Extreme => 1)
or die this_file . ": $file: $IO::Compress::Xz::XzError";
($xz->write($content) == length $content)
or die this_file . ": $file: $IO::Compress::Xz::XzError";
$xz->close or die this_file . ": $file: $IO::Compress::Xz::XzError";
return;
# xz executable
} else {
my ($PH, $CMD);
$CMD = where_is "xz";
$CMD = "\"$CMD\" -c9f > \"$file\"";
open $PH, "| $CMD" or die this_file . ": $CMD: $!"; open $PH, "| $CMD" or die this_file . ": $CMD: $!";
print $PH $content or die this_file . ": $CMD: $!"; print $PH $content or die this_file . ": $CMD: $!";
close $PH or die this_file . ": $CMD: $!"; close $PH or die this_file . ": $CMD: $!";
@ -489,7 +452,6 @@ sub file_type($) {
$_ = File::MMagic->new->checktype_filename($file); $_ = File::MMagic->new->checktype_filename($file);
return "application/x-gzip" if /gzip/; return "application/x-gzip" if /gzip/;
return "application/x-bzip2" if /bzip2/; return "application/x-bzip2" if /bzip2/;
return "application/x-xz" if /xz/;
# All else are text/plain # All else are text/plain
return "text/plain"; return "text/plain";
} }
@ -498,7 +460,6 @@ sub file_type($) {
$_ = join "", `"$_" "$file"`; $_ = join "", `"$_" "$file"`;
return "application/x-gzip" if /gzip/; return "application/x-gzip" if /gzip/;
return "application/x-bzip2" if /bzip2/; return "application/x-bzip2" if /bzip2/;
return "application/x-xz" if /: XZ/;
# All else are text/plain # All else are text/plain
return "text/plain"; return "text/plain";
} }
@ -572,31 +533,22 @@ sub has_no_file() {
# If we have gzip support somewhere # If we have gzip support somewhere
sub has_no_gzip() { sub has_no_gzip() {
$HAS_NO_GZIP = eval { require IO::Compress::Gzip; require IO::Uncompress::Gunzip; 1; } $HAS_NO_GZIP = eval { require Compress::Zlib; 1; }
|| defined where_is "gzip"? || defined where_is "gzip"?
0: "IO::Compress::Gzip or gzip executable not available" 0: "Compress::Zlib or gzip executable not available"
if !defined $HAS_NO_GZIP; if !defined $HAS_NO_GZIP;
return $HAS_NO_GZIP; return $HAS_NO_GZIP;
} }
# If we have bzip2 support somewhere # If we have bzip2 support somewhere
sub has_no_bzip2() { sub has_no_bzip2() {
$HAS_NO_BZIP2 = eval { require IO::Compress::Bzip2; require IO::Uncompress::Bunzip2; 1; } $HAS_NO_BZIP2 = eval { require Compress::Bzip2; Compress::Bzip2->import(2.00); 1; }
|| defined where_is "bzip2"? || defined where_is "bzip2"?
0: "IO::Compress::Bzip2 v2 or bzip2 executable not available" 0: "Compress::Bzip2 v2 or bzip2 executable not available"
if !defined $HAS_NO_BZIP2; if !defined $HAS_NO_BZIP2;
return $HAS_NO_BZIP2; return $HAS_NO_BZIP2;
} }
# If we have xz support somewhere
sub has_no_xz() {
$HAS_NO_XZ = eval { require IO::Compress::Xz; require IO::Uncompress::UnXz; 1; }
|| defined where_is "xz"?
0: "IO::Compress::Xz or xz executable not available"
if !defined $HAS_NO_XZ;
return $HAS_NO_XZ;
}
# Create a random existing log file # Create a random existing log file
sub make_log_file($$$@) { sub make_log_file($$$@) {
local ($_, %_); local ($_, %_);