imacat/arclog
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Compare commits

base: imacat:main
Branches Tags
imacat:main
..
compare: imacat:3c7030099be0393df9dae187751e3a06011f244d
Branches Tags
imacat:main

No commits in common. "main" and "3c7030099be0393df9dae187751e3a06011f244d" have entirely different histories.
main ... 3c7030099b

10 changed files with 243 additions and 881 deletions
Show Stats Expand all files Collapse all files

5
Build.PL
View File

@ -62,9 +62,8 @@ my $build = Module::Build->new(
},
recommends => {
"File::MMagic" => 0,
"IO::Compress::Gzip" => 0,
"IO::Compress::Bzip2" => 2,
"IO::Compress::Xz" => 0,
"Compress::Zlib" => 0,
"Compress::Bzip2" => 2,
"Term::ReadKey" => 0,
},
build_requires => {

7
Changes
View File

@ -1,12 +1,5 @@
arclog change log
2022-03-19 version 3.06
1. Added support for the xz compression.
2. Replaced Compress::Zlib with IO::Compress::Gzip and
IO::Uncompress::Gunzip.
3. Replaced Compress::Bzip2 with IO::Compress::Bzip2 and
IO::Uncompress::Bunzip2.
2020-02-06 version 3.05
1. Replaced the messy GNU-styled, multi-lingual documents with a
single Markdown README.md.

5
Makefile.PL
View File

@ -52,9 +52,8 @@ EOT
our (%OPT_PREREQ);
%OPT_PREREQ = (
"File::MMagic" => 0,
"IO::Compress::Gzip" => 0,
"IO::Compress::Bzip2" => 2,
"IO::Compress::Xz" => 0,
"Compress::Zlib" => 0,
"Compress::Bzip2" => 2,
"Term::ReadKey" => 0,
);
delete $OPT_PREREQ{$_}

142
README.md
View File

@ -12,9 +12,9 @@ space and prevents potential attacks on log files.
Currently, `arclog` supports [Apache] access log, Syslog, [NTP],
Apache 1 SSL engine log, and my own bracketed, modified ISO date/time
log file formats, and gzip, bzip2, and xz compression methods.
Several software projects log (or can log) in a format compatible with
the Apache access log, like [CUPS], [ProFTPD], [Pure-FTPd]… etc., and
log file formats, and gzip and bzip2 compression methods. Several
software projects log (or can log) in a format compatible with the
Apache access log, like [CUPS], [ProFTPD], [Pure-FTPd]… etc., and
`arclog` can archive their Apache-like log files, too.
Caution
@ -84,10 +84,10 @@ Caution
* I suggest that you install [File::MMagic] instead of counting on the
`file` executable. The internal magic file of File::MMagic works
better than the `file` executable. `arclog` treats everything not
gzip, bzip2, or xz compressed as plain text. When a compressed log
file is wrongly recognized as an image, `arclog` treats it as plain
text, reads directly from it, and fails. This does not hurt the
source log files, but is still annoying.
gzip nor bzip2 compressed as plain text. When a compressed log file
is wrongly recognized as an image, `arclog` treats it as plain text,
reads directly from it, and fails. This does not hurt the source
log files, but is still annoying.
[Date::Parse]: https://metacpan.org/release/TimeDate
[File::MMagic]: https://metacpan.org/release/File-MMagic
@ -174,23 +174,22 @@ System Requirement
the [GnuWin32] home page. Be sure to save it as `file.exe`
somewhere in your `PATH`.
* [IO::Compress::Gzip] and [IO::Uncompress::Gunzip]
* [Compress::Zlib]
They are used to support reading/writing the gzip compressed
This is used to support reading/writing the gzip compressed
files. It is only needed when gzip compressed files are
encountered. If they are not available, `arclog` tries the
`gzip` executable instead. If that is not available, too,
`arclog` fails. You should not worry about IO::Compress::Gzip
since it comes with Perl since version 5.9.3. If not, it is
contained in the [IO-Compress] distribution. You can download
and install it from the CPAN archive, or install it with the
CPAN shell:
encountered. If it is not available, `arclog` tries the `gzip`
executable instead. If that is not available, too, `arclog`
fails. You should not worry about Compress::Zlib since it comes
with Perl since version 5.9.3. If not, it is contained in the
[IO-Compress] distribution. You can download and install it from
the CPAN archive, or install it with the CPAN shell:
cpan IO::Compress::Gzip
cpan Compress::Zlib
or with the CPANPLUS shell:
cpanp i IO::Compress::Gzip
cpanp i Compress::Zlib
For Debian/Ubuntu:
@ -212,80 +211,43 @@ System Requirement
[the gzip website]. Be sure to save it as `gzip.exe` somewhere
in your `PATH`.
* [IO::Compress::Bzip2] and [IO::Uncompress::Bunzip2]
* [Compress::Bzip2] version 2 or above.
They are used to support reading/writing the bzip2 compressed
files. They are only needed when bzip2 compressed files are
encountered. If they are not available, `arclog` tries the
`bzip2` executable instead. If that is not available, too,
`arclog` fails. You should not worry about IO::Compress::Bzip2
since it comes with Perl since version 5.10.1. If not, it is
contained in the [IO-Compress] distribution. You can download
and install it from the CPAN archive, or install it with the
CPAN shell:
This is used to support reading/writing the bzip2 compressed
files. It is only needed when bzip2 compressed files are
encountered. If it is not available, `arclog` tries the `bzip2`
executable instead. If that is not available, too, `arclog`
fails. Notice that older versions before 2 does not work, since
the file I/O compression was not implemented yet. You can
download and install Compress::Bzip2 from the CPAN archive, or
install it with the CPAN shell:
cpan IO::Compress::Bzip2
cpan Compress::Bzip2
or with the CPANPLUS shell:
cpanp i IO::Compress::Bzip2
cpanp i Compress::Bzip2
For Debian/Ubuntu:
sudo apt install libio-compress-perl
sudo apt install libcompress-bzip2-perl
For Red Hat/Fedora/CentOS:
sudo yum install perl-IO-Compress
sudo yum install perl-Compress-Bzip2
For FreeBSD:
ports install p5-IO-Compress
ports install p5-Compress-Bzip2
For ActivePerl:
ppm install IO-Compress
ppm install Compress-Bzip2
The alternative `bzip2.exe` for MS-Windows can be obtained from
[the bzip2 website]. Be sure to save it as `bzip2.exe` somewhere
in your `PATH`.
* [IO::Compress::Xz] and [IO::Uncompress::UnXz]
They are used to support reading/writing the xz compressed
files. It is only needed when xz compressed files are
encountered. If it is not available, `arclog` tries the `xz`
executable instead. If that is not available, too, `arclog`
fails. They are contained in the [IO-Compress-Lzma] distribution.
You can download and install it from the CPAN archive, or install
them with the CPAN shell:
cpan IO::Compress::Xz
or with the CPANPLUS shell:
cpanp i IO::Compress::Xz
For Debian/Ubuntu:
sudo apt install libio-compress-lzma-perl
For Red Hat/Fedora/CentOS:
sudo yum install perl-IO-Compress-Lzma
For FreeBSD:
ports install p5-IO-Compress-Lzma
For ActivePerl:
ppm install IO-Compress-Lzma
The alternative `xz.exe` for MS-Windows can be obtained from
[the XZ Utils website]. Be sure to save it as `xz.exe`
somewhere in your `PATH`.
* [Term::ReadKey]
This is used to display the progress bar. The progress bar is a
@ -322,17 +284,11 @@ System Requirement
[Date::Parse]: https://metacpan.org/pod/Date::Parse
[File::MMagic]: https://metacpan.org/pod/File::MMagic
[GnuWin32]: http://gnuwin32.sourceforge.net
[IO::Compress::Gzip]: https://metacpan.org/pod/IO::Compress::Gzip
[IO::Uncompress::Gunzip]: https://metacpan.org/pod/IO::Uncompress::Gunzip
[Compress::Zlib]: https://metacpan.org/pod/Compress::Zlib
[the gzip website]: https://www.gzip.org
[IO-Compress]: https://metacpan.org/dist/IO-Compress
[IO::Compress::Bzip2]: https://metacpan.org/pod/IO::Compress::Bzip2
[IO::Uncompress::Bunzip2]: https://metacpan.org/pod/IO::Uncompress::Bunzip2
[IO-Compress]: https://metacpan.org/release/IO-Compress
[Compress::Bzip2]: https://metacpan.org/pod/Compress::Bzip2
[the bzip2 website]: http://www.bzip.org
[IO::Compress::Xz]: https://metacpan.org/pod/IO::Compress::Xz
[IO::Uncompress::UnXz]: https://metacpan.org/pod/IO::Uncompress::UnXz
[IO-Compress-Lzma]: https://metacpan.org/dist/IO-Compress-Lzma
[the XZ Utils website]: https://tukaani.org/xz/
[Term::ReadKey]: https://metacpan.org/pod/Term::ReadKey
@ -474,8 +430,8 @@ Options
* `logfile`
The log file to be archived. Specify `-` to read from `STDIN`.
You can specify multiple log files. `gzip`, `bzip2`, or `xz`
compressed files are supported.
You can specify multiple log files. `gzip` or `bzip2` compressed
files are supported.
* `output`
@ -496,25 +452,17 @@ Options
* `g`, `gzip`
Compress with `gzip`. This is the default. `arclog` can use
`IO::Compress::Gzip` to compress instead of calling `gzip`.
This can be safer and faster for not calling foreign binaries.
If `IO::Compress::Gzip` is not installed, it tries `gzip`
instead. If `gzip` is not available, either, it fails.
`Compress::Zlib` to compress instead of calling `gzip`. This can
be safer and faster for not calling foreign binaries. If
`Compress::Zlib` is not installed, it tries `gzip` instead. If
`gzip` is not available, either, it fails.
* `b`, `bzip2`
Compress with `bzip2`. `arclog` can use `IO::Compress::Bzip2`
to compress instead of calling `bzip2`. This can be safer and
faster for not calling foreign binaries. If
`IO::Compress::Bzip2` is not installed, it tries `bzip2`
instead. If `bzip2` is not available, either, it fails.
* `x`, `xz`
Compress with `xz`. `arclog` can use `IO::Compress::Xz` to
compress instead of calling `xz`. This can be safer and faster
for not calling foreign binaries. If `IO::Compress::Xz` is not
installed, it tries `xz` instead. If `xz` is not available,
Compress with `bzip2`. `arclog` can use `Compress::Bzip2` to
compress instead of calling `bzip2`. This can be safer and faster
for not calling foreign binaries. If `Compress::Bzip2` is not
installed, it tries `bzip2` instead. If `bzip2` is not available,
either, it fails.
* `n`, `none`

715
arclog
View File

File diff suppressed because it is too large Load Diff

6
t/01-exhaust.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w
# Test all the possible combination of options
# Copyright (c) 2007-2022 imacat.
# Copyright (c) 2007-2021 imacat.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -21,7 +21,7 @@ use warnings;
use diagnostics;
use Test;
BEGIN { plan tests => 3600 }
BEGIN { plan tests => 2160 }
use File::Basename qw(basename);
use File::Path qw(mkpath rmtree);
@ -35,7 +35,7 @@ $WORKDIR = catdir($FindBin::Bin, "logs");
$arclog = catfile($FindBin::Bin, updir, "blib", "script", "arclog");
$tno = 0;
# 1-3600: All possible option combinations
# 1-2160: All possible option combinations
# Test each log file format
foreach my $fmt (@LOG_FORMATS) {
# Test each source log file type

33
t/02-several.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w
# Test archiving several log files at once
# Copyright (c) 2007-2022 imacat.
# Copyright (c) 2007-2021 imacat.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -21,7 +21,7 @@ use warnings;
use diagnostics;
use Test;
BEGIN { plan tests => 25 }
BEGIN { plan tests => 16 }
use File::Basename qw(basename);
use File::Path qw(mkpath rmtree);
@ -35,15 +35,14 @@ $WORKDIR = catdir($FindBin::Bin, "logs");
$arclog = catfile($FindBin::Bin, updir, "blib", "script", "arclog");
$tno = 0;
# 1-25: Archiving several log files at once
# 1-16: Archiving several log files at once
foreach my $rt (@RESULT_TYPES) {
my $skip;
$skip = 0;
# 1: Source log files listed as the arguments
$_ = eval {
if ( ($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$rt{"type"} eq TYPE_XZ && has_no_xz)) {
|| ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)) {
$skip = 1;
return;
}
@ -57,15 +56,13 @@ foreach my $rt (@RESULT_TYPES) {
$title = join ", ", "several log files", "all listed as arguments",
$$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2)
+ (has_no_xz? 0: 2);
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
$num = $_ + int rand $_;
my %types = qw();
# At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
@_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1);
@ -76,8 +73,7 @@ foreach my $rt (@RESULT_TYPES) {
do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
}
@st = map $types{$_}, (0...$num-1);
@fs = qw();
@ -150,17 +146,15 @@ foreach my $rt (@RESULT_TYPES) {
skip($skip, $_, 1, $@);
clean_up $_ || $skip, $WORKDIR, ++$tno;
# 2-5: One of the source log files is read from STDIN
# 2-4: One of the source log files is read from STDIN
# The file type at STDIN
foreach my $st_stdin (@SOURCE_TYPES) {
$skip = 0;
$_ = eval {
if ( ($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$rt{"type"} eq TYPE_XZ && has_no_xz)
|| ($$st_stdin{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st_stdin{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$st_stdin{"type"} eq TYPE_XZ && has_no_xz)) {
|| ($$st_stdin{"type"} eq TYPE_BZIP2 && has_no_bzip2)) {
$skip = 1;
return;
}
@ -174,15 +168,13 @@ foreach my $rt (@RESULT_TYPES) {
$title = join ", ", "several log files", "one read from STDIN",
"STDIN " . $$st_stdin{"title"}, $$rt{"title"};
# (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2)
+ (has_no_xz? 0: 2);
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
$num = $_ + int rand $_;
my %types = qw();
# At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
@_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1);
@ -193,8 +185,7 @@ foreach my $rt (@RESULT_TYPES) {
do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
}
# Choose the STDIN from the matching compression
@_ = grep ${$types{$_}}{"type"} eq $$st_stdin{"type"}, (0...$num-1);

26
t/03-fallback.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w
# Test fallback behavior
# Copyright (c) 2007-2022 imacat.
# Copyright (c) 2007-2021 imacat.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -50,21 +50,18 @@ foreach my $kt (@KEEP_MODES) {
do {
$rt = $RESULT_TYPES[int rand @RESULT_TYPES];
} until !($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
&& !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !($$rt{"type"} eq TYPE_XZ && has_no_xz);
&& !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2);
$title = join ", ", "STDIN keep fall back", $$kt{"title"},
$$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2)
+ (has_no_xz? 0: 2);
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
$num = $_ + int rand $_;
$stdin = int rand $num;
my %types = qw();
# At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
@_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1);
@ -75,8 +72,7 @@ foreach my $kt (@KEEP_MODES) {
do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
}
@st = map $types{$_}, (0...$num-1);
@fs = qw();
@ -164,21 +160,18 @@ $_ = eval {
do {
$rt = $RESULT_TYPES[int rand @RESULT_TYPES];
} until !($$rt{"type"} eq TYPE_GZIP && has_no_gzip)
&& !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !($$rt{"type"} eq TYPE_XZ && has_no_xz);
&& !($$rt{"type"} eq TYPE_BZIP2 && has_no_bzip2);
$title = join ", ", "STDIN override ask fall back",
$$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2)
+ (has_no_xz? 0: 2);
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
$num = $_ + int rand $_;
$stdin = int rand $num;
my %types = qw();
# At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
@_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1);
@ -189,8 +182,7 @@ $_ = eval {
do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
}
@st = map $types{$_}, (0...$num-1);
@fs = qw();

11
t/04-errors.t
View File

@ -1,7 +1,7 @@
#! /usr/bin/perl -w
# Test the errors that should be captured.
# Copyright (c) 2007-2022 imacat.
# Copyright (c) 2007-2021 imacat.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -140,15 +140,13 @@ $_ = eval {
$title = join ",", "A same log file is specified more than once",
$$fmt{"title"}, $$rt{"title"};
# (2-4 times available compression) log files
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2)
+ (has_no_xz? 0: 2);
$_ = 2 + (has_no_gzip? 0: 2) + (has_no_bzip2? 0: 2);
$num = $_ + int rand $_;
my %types = qw();
# At least 2 files for each available compression
foreach my $st (@SOURCE_TYPES) {
next if ($$st{"type"} eq TYPE_GZIP && has_no_gzip)
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2)
|| ($$st{"type"} eq TYPE_XZ && has_no_xz);
|| ($$st{"type"} eq TYPE_BZIP2 && has_no_bzip2);
@_ = grep !exists $types{$_}, (0...$num-1);
$types{$_[int rand @_]} = $st;
@_ = grep !exists $types{$_}, (0...$num-1);
@ -159,8 +157,7 @@ $_ = eval {
do {
$types{$_} = $SOURCE_TYPES[int rand @SOURCE_TYPES];
} until !(${$types{$_}}{"type"} eq TYPE_GZIP && has_no_gzip)
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2)
&& !(${$types{$_}}{"type"} eq TYPE_XZ && has_no_xz);
&& !(${$types{$_}}{"type"} eq TYPE_BZIP2 && has_no_bzip2);
}
@st = map $types{$_}, (0...$num-1);
@fs = qw();

174
t/_helper.pm
View File

@ -1,6 +1,6 @@
# _helper.pm - A simple test suite helper
# Copyright (c) 2007-2022 imacat.
# Copyright (c) 2007-2021 imacat.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
@ -24,12 +24,12 @@ $VERSION = "0.01";
@EXPORT = qw(
read_file read_raw_file write_file write_raw_file
run_cmd where_is file_type list_files preserve_source clean_up
has_no_file has_no_gzip has_no_bzip2 has_no_xz
has_no_file has_no_gzip has_no_bzip2
make_log_file
make_apache_log_file make_syslog_log_file
make_ntp_log_file make_apache_ssl_log_file make_modified_iso_log_file
random_word
TYPE_TEXT TYPE_GZIP TYPE_BZIP2 TYPE_XZ
TYPE_TEXT TYPE_GZIP TYPE_BZIP2
@LOG_FORMATS @SOURCE_TYPES @RESULT_TYPES @KEEP_MODES @OVERRIDE_MODES);
# Prototype declaration
sub this_file();
@ -46,7 +46,6 @@ sub clean_up($$$);
sub has_no_file();
sub has_no_gzip();
sub has_no_bzip2();
sub has_no_xz();
sub make_log_file($$$@);
sub make_apache_log_file($;$);
sub make_syslog_log_file($;$);
@ -72,17 +71,15 @@ use File::Temp qw(tempfile);
use Time::Local qw(timelocal);
$Data::Dumper::Indent = 1;
our (%WHERE_IS, $HAS_NO_FILE, $HAS_NO_GZIP, $HAS_NO_BZIP2, $HAS_NO_XZ);
our (%WHERE_IS, $HAS_NO_FILE, $HAS_NO_GZIP, $HAS_NO_BZIP2);
%WHERE_IS = qw();
undef $HAS_NO_FILE;
undef $HAS_NO_GZIP;
undef $HAS_NO_BZIP2;
undef $HAS_NO_XZ;
use constant TYPE_TEXT => "text/plain";
use constant TYPE_GZIP => "application/x-gzip";
use constant TYPE_BZIP2 => "application/x-bzip2";
use constant TYPE_XZ => "application/x-xz";
our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES);
# All the log format information
@ -110,11 +107,7 @@ our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES);
{ "title" => "bzip2 source",
"type" => TYPE_BZIP2,
"suf" => ".bz2",
"skip" => has_no_bzip2, },
{ "title" => "xz source",
"type" => TYPE_XZ,
"suf" => ".xz",
"skip" => has_no_xz, }, );
"skip" => has_no_bzip2, }, );
# All the result type information
@RESULT_TYPES = (
{ "title" => "default compress",
@ -132,11 +125,6 @@ our (@LOG_FORMATS, @SOURCE_TYPES, @RESULT_TYPES, @KEEP_MODES, @OVERRIDE_MODES);
"suf" => ".bz2",
"skip" => has_no_bzip2,
"opts" => [qw(-c b)], },
{ "title" => "xz compress",
"type" => TYPE_XZ,
"suf" => ".xz",
"skip" => has_no_xz,
"opts" => [qw(-c x)], },
{ "title" => "no compress",
"type" => TYPE_TEXT,
"suf" => "",
@ -227,21 +215,29 @@ sub read_file($) {
# a gzip compressed file
if ($file =~ /\.gz$/) {
# IO::Uncompress::Gunzip
if (eval { require IO::Uncompress::Gunzip; 1; }) {
my $gz;
# Compress::Zlib
if (eval { require Compress::Zlib;
Compress::Zlib->import(qw(gzopen));
1; }) {
use Compress::Zlib qw(gzopen);
my ($FH, $gz);
$content = "";
$gz = IO::Uncompress::Gunzip->new($file)
or die this_file . ": $file: $IO::Uncompress::Gunzip::GunzipError";
$content = join "", <$gz>;
$gz->close or die this_file . ": $file: $IO::Uncompress::Gunzip::GunzipError";
open $FH, $file or die this_file . ": $file: $!";
$gz = gzopen($FH, "rb") or die this_file . ": $file: $!";
while (1) {
($gz->gzread($_, 10240) != -1)
or die this_file . ": $file: " . $gz->gzerror;
$content .= $_;
last if length $_ < 10240;
}
$gz->gzclose and die this_file . ": $file: " . $gz->gzerror;
return $content;
# gzip executable
} else {
my ($PH, $CMD);
$CMD = where_is "gzip";
$CMD = "\"$CMD\" -cdf \"$file\"";
$CMD = "\"$CMD\" -cd \"$file\"";
open $PH, "$CMD |" or die this_file . ": $CMD: $!";
$content = join "", <$PH>;
close $PH or die this_file . ": $CMD: $!";
@ -250,44 +246,29 @@ sub read_file($) {
# a bzip compressed file
} elsif ($file =~ /\.bz2$/) {
# IO::Uncompress::Bunzip2
if (eval { require IO::Uncompress::Bunzip2; 1; }) {
my $bz;
# Compress::Bzip2
if (eval { require Compress::Bzip2;
Compress::Bzip2->import(2.00);
Compress::Bzip2->import(qw(bzopen));
1; }) {
my ($FH, $bz);
$content = "";
$bz = IO::Uncompress::Bunzip2->new($file)
or die this_file . ": $file: $IO::Uncompress::Bunzip2::Bunzip2Error";
$content = join "", <$bz>;
$bz->close or die this_file . ": $file: $IO::Uncompress::Bunzip2::Bunzip2Error";
open $FH, $file or die this_file . ": $file: $!";
$bz = bzopen($FH, "rb") or die this_file . ": $file: $!";
while (1) {
($bz->bzread($_, 10240) != -1)
or die this_file . ": $file: " . $bz->bzerror;
$content .= $_;
last if length $_ < 10240;
}
$bz->bzclose and die this_file . ": $file: " . $bz->bzerror;
return $content;
# bzip2 executable
} else {
my ($PH, $CMD);
$CMD = where_is "bzip2";
$CMD = "\"$CMD\" -cdf \"$file\"";
open $PH, "$CMD |" or die this_file . ": $CMD: $!";
$content = join "", <$PH>;
close $PH or die this_file . ": $CMD: $!";
return $content;
}
# an xz compressed file
} elsif ($file =~ /\.xz$/) {
# IO::Uncompress::UnXz
if (eval { require IO::Uncompress::UnXz; 1; }) {
my $xz;
$content = "";
$xz = IO::Uncompress::UnXz->new($file)
or die this_file . ": $file: $IO::Uncompress::UnXz::UnXzError";
$content = join "", <$xz>;
$xz->close or die this_file . ": $file: $IO::Uncompress::UnXz::UnXzError";
return $content;
# xz executable
} else {
my ($PH, $CMD);
$CMD = where_is "xz";
$CMD = "\"$CMD\" -cdf \"$file\"";
$CMD = "bzip2 -cd \"$file\"";
open $PH, "$CMD |" or die this_file . ": $CMD: $!";
$content = join "", <$PH>;
close $PH or die this_file . ": $CMD: $!";
@ -330,14 +311,16 @@ sub write_file($$) {
# a gzip compressed file
if ($file =~ /\.gz$/) {
# IO::Compress::Gzip
if (eval { require IO::Compress::Gzip; 1; }) {
my $gz;
$gz = IO::Compress::Gzip->new($file, -Level => 9)
or die this_file . ": $file: $IO::Compress::Gzip::GzipError";
($gz->write($content) == length $content)
or die this_file . ": $file: $IO::Compress::Gzip::GzipError";
$gz->close or die this_file . ": $file: $IO::Compress::Gzip::GzipError";
# Compress::Zlib
if (eval { require Compress::Zlib;
Compress::Zlib->import(qw(gzopen));
1; }) {
my ($FH, $gz);
open $FH, ">$file" or die this_file . ": $file: $!";
$gz = gzopen($FH, "wb9") or die this_file . ": $file: $!";
($gz->gzwrite($content) == length $content)
or die this_file . ": $file: " . $gz->gzerror;
$gz->gzclose and die this_file . ": $file: " . $gz->gzerror;
return;
# gzip executable
@ -353,44 +336,24 @@ sub write_file($$) {
# a bzip compressed file
} elsif ($file =~ /\.bz2$/) {
# IO::Compress::Bzip2
if (eval { require IO::Compress::Bzip2; 1; }) {
my $bz;
$bz = IO::Compress::Bzip2->new($file, BlockSize100K => 9)
or die this_file . ": $file: $IO::Compress::Bzip2::Bzip2Error";
($bz->write($content) == length $content)
or die this_file . ": $file: $IO::Compress::Bzip2::Bzip2Error";
$bz->close or die this_file . ": $file: $IO::Compress::Bzip2::Bzip2Error";
# Compress::Bzip2
if (eval { require Compress::Bzip2;
Compress::Bzip2->import(2.00);
Compress::Bzip2->import(qw(bzopen));
1; }) {
my ($FH, $bz);
open $FH, ">$file" or die this_file . ": $file: $!";
$bz = bzopen($FH, "wb9") or die this_file . ": $file: $!";
($bz->bzwrite($content, length $content) == length $content)
or die this_file . ": $file: " . $bz->bzerror;
$bz->bzclose and die this_file . ": $file: " . $bz->bzerror;
return;
# bzip2 executable
} else {
my ($PH, $CMD);
$CMD = where_is "bzip2";
$CMD = "\"$CMD\" -c9f > \"$file\"";
open $PH, "| $CMD" or die this_file . ": $CMD: $!";
print $PH $content or die this_file . ": $CMD: $!";
close $PH or die this_file . ": $CMD: $!";
return;
}
# an xz compressed file
} elsif ($file =~ /\.xz$/) {
# IO::Compress::Xz
if (eval { require IO::Compress::Xz; 1; }) {
my $xz;
$xz = IO::Compress::Xz->new($file, Extreme => 1)
or die this_file . ": $file: $IO::Compress::Xz::XzError";
($xz->write($content) == length $content)
or die this_file . ": $file: $IO::Compress::Xz::XzError";
$xz->close or die this_file . ": $file: $IO::Compress::Xz::XzError";
return;
# xz executable
} else {
my ($PH, $CMD);
$CMD = where_is "xz";
$CMD = "\"$CMD\" -c9f > \"$file\"";
$CMD = "\"$CMD\" -9f > \"$file\"";
open $PH, "| $CMD" or die this_file . ": $CMD: $!";
print $PH $content or die this_file . ": $CMD: $!";
close $PH or die this_file . ": $CMD: $!";
@ -489,7 +452,6 @@ sub file_type($) {
$_ = File::MMagic->new->checktype_filename($file);
return "application/x-gzip" if /gzip/;
return "application/x-bzip2" if /bzip2/;
return "application/x-xz" if /xz/;
# All else are text/plain
return "text/plain";
}
@ -498,7 +460,6 @@ sub file_type($) {
$_ = join "", `"$_" "$file"`;
return "application/x-gzip" if /gzip/;
return "application/x-bzip2" if /bzip2/;
return "application/x-xz" if /: XZ/;
# All else are text/plain
return "text/plain";
}
@ -572,31 +533,22 @@ sub has_no_file() {
# If we have gzip support somewhere
sub has_no_gzip() {
$HAS_NO_GZIP = eval { require IO::Compress::Gzip; require IO::Uncompress::Gunzip; 1; }
$HAS_NO_GZIP = eval { require Compress::Zlib; 1; }
|| defined where_is "gzip"?
0: "IO::Compress::Gzip or gzip executable not available"
0: "Compress::Zlib or gzip executable not available"
if !defined $HAS_NO_GZIP;
return $HAS_NO_GZIP;
}
# If we have bzip2 support somewhere
sub has_no_bzip2() {
$HAS_NO_BZIP2 = eval { require IO::Compress::Bzip2; require IO::Uncompress::Bunzip2; 1; }
$HAS_NO_BZIP2 = eval { require Compress::Bzip2; Compress::Bzip2->import(2.00); 1; }
|| defined where_is "bzip2"?
0: "IO::Compress::Bzip2 v2 or bzip2 executable not available"
0: "Compress::Bzip2 v2 or bzip2 executable not available"
if !defined $HAS_NO_BZIP2;
return $HAS_NO_BZIP2;
}
# If we have xz support somewhere
sub has_no_xz() {
$HAS_NO_XZ = eval { require IO::Compress::Xz; require IO::Uncompress::UnXz; 1; }
|| defined where_is "xz"?
0: "IO::Compress::Xz or xz executable not available"
if !defined $HAS_NO_XZ;
return $HAS_NO_XZ;
}
# Create a random existing log file
sub make_log_file($$$@) {
local ($_, %_);