arclog version 3.04 is released. Documentation fixes. Upgrade is not necessary if you have installed arclog version 3.00 or later. Download arclog version 3.04.
arclog version 3.03 is released. Test suite fixes. Required Perl version number compatible to older versions, to work with the Perl 5.10 warning. Upgrade is not necessary if you have installed arclog version 3.00 or later. Download arclog version 3.03.
arclog version 3.02 is released. Test suite improvement, in order to deal with daylight saving time. Upgrade is not necessary if you have installed arclog version 3.00 or later. Download arclog version 3.02.
arclog version 3.01 is released. Test suite improvement, in order to catch CPAN tester failures. Upgrade is not necessary if you have installed arclog version 3.00. Download arclog version 3.01.
arclog version 3.00 is released. The code is completely new borrowed from reslog, with the name changed from arclog.pl to arclog, too. The new object-oriented handlers deal with different compression methods, formats nicely. Perl’s s build system ExtUtils::MakeMaker and Module::Build are used instead of GNU autoconf. And there is a complete test suite that help quality assurance. License updated to GPLv3. Download arclog version 3.00.
arclog.pl version 2.1.0 is released. This is the final 2.1.0 release. It adds support for bzip2 compression, gzip compression with gzip binary instead of Compress::Zlib, and GNU autoconf configuration script. Download arclog.pl version 2.1.0.
arclog.pl version 2.1.0dev2 is released. This release corrects several documentation errors. It also adds SourceForge in the documentations as one of the sources of arclog.pl. You can think of it as a Source Forge Memorial Release
. :p You do nott have to upgrade to this version in a rush. Download arclog.pl version 2.1.0dev2.
arclog.pl is hosted at SourceForge now! Congratulations! (Although I’m still trying hard to get it working at this time… ^^; )
arclog archives the log files monthly. It strips off previous months’ log records from the log file, and save them to compressed archive files named logfile.yyyymm. It then saves the hard disk space and prevents potential attacks on log files.
Currently, arclog supports Apache access log, Syslog, NTP, Apache 1 SSL engine log and my own bracketed, modified ISO date/time log file formats, and gzip and bzip2 compression methods. Several software projects log (or can log) in a format compatible with the Apache access log, like CUPS, ProFTPD, Pure-FTPd… etc., and arclog can archive their Apache-like log files, too.
Archiving takes time. To reduce the time occupying the source log file, arclog copies the content of the source log file to a temporary working file and restart the source log file first. Then arclog can take its time working on the temporary working file. However, please note:
If you have a huge log file (several hundreds of MBs), merely copying still takes a lot of time. You had better stop logging first, archive the log file and restart logging, to avoid racing condition in writing. If you archive the log file periodly, it shall not grow too big.
Ifarclog stops in the middle of the execution, it will leave a temporary working file. The next time arclog runs, it will stop when it sees that temporary working file. You have to process that temporary working file first. That temporary working file is merely a copy of the original log file. You can rename and archive it like an ordinary log file to solve this.
Do not sort unless you have a particular reason. Sorting has the following potential problem:
Sorting may eat huge memory on large log files. The amount of the memory required depends on the number of records in each archived month. Modern Linux and MS-Windows have memory consuming protection by killing processes that eats too much memory, but it still takes minutes, and your system will hang during that time. I do not know the memory consuming protection on other operating systems. If you try, you are at your own risk.
The time units of all recognized log formats are second. Log records happen in a same second will be sorted by the log file order (if you are archiving several log files at a time) and then the log record order. I try to ensure that the sorted archived records are in a correct order of the happening events, but I cannot guarantee. You have to watch out if the order in a second is important.
Be careful on the Syslog and NTP log files: Syslog and NTP does not record the year. arclog uses Date::Parse to parse the date, which assums the year between this month and last next month if the year is missing. For ex., if today is 2001-06-08, it will then assume the year between 2001-06-30 back to 2000-07-01. I think this is smart enough. However, if you do have a Syslog or NTP log file that has records older than one year, do not use arclog. It will destroy your log file.
If read from STDIN, please note:
You must specify the output prefix if you want to read from STDIN, since what it needs is an output pathname prefix, not an output file.
STDIN cannot be deleted, restarted or partially kept. If you read from STDIN, the keep mode will fall back to keep all
. If you archive several source log files including STDIN, the keep mode will fall back to keep all
for all source log files, to prevent disaster.
The answers of the ask mode is obtained from STDIN, too. Since you have only one STDIN, you cannot specify the ask mode while reading from STDIN. It will fall back to the fail mode in that case.
I suggest you to install File::MMagic instead of counting on the file executable. The internal magic file of File::MMagic seems to work better than the file executable. arclog treats everything not gzip nor bzip2 compressed as plain text. When a compressed log file is wrongly recognized as an image, arclog will treat it as plain text, read log records directly from it and fail. This failure does not hurt the source log files, but is still annoying.
Perl, version 5.8.0 or above. arclog uses 3-argument open() to duplicate filehandles, which is only supported since 5.8.0. I have not successfully port this onto earlier versions yet. Please tell me if you made it. You can run perl -v
to see your current Perl version. If you do not have Perl, or if you have an older version of Perl, you can download and install/upgrade it from Perl website. If you are using MS-Windows, you can download and install ActiveState ActivePerl.
Required Perl modules:
This is used to parse the timestamp of the log records. You can download and install Date::Parse from the CPAN archive, or install it with the CPAN shell by running cpan Date::Parse
, or install it with the CPANPLUS shell by running cpanp i Date::Parse
. Debian/Ubandu Linux users can install it by running apt-get install libtimedate-perl
. Red Hat/Fedora/CentOS Linux users can install it by running yum install perl-TimeDate
. FreeBSD users can install it by running ports install p5-TimeDate
. ActivePerl users can install it by running ppm install Date::Parse
.
If you install arclog with the CPAN or CPAN shell, the above shall be automatically installed.
Optional Perl modules:
This is used to check the file type. If this is not available, arclog will look for the file executable instead. If that is not available, too, arclog will judge the file type by its name suffix (extension). In that case arclog will fail when reading from STDIN. You can download and install File::MMagic from the CPAN archive, or install it with the CPAN shell by running cpan File::MMagic
, or install it with the CPANPLUS shell by running cpanp i Term::ReadKey
. Debian/Ubandu Linux users can install it by running apt-get install libfile-mmagic-perl
. Red Hat/Fedora/CentOS Linux users can install it by running yum install perl-File-MMagic
. FreeBSD users can install it by running ports install p5-File-MMagic
. ActivePerl users can install it by running ppm install File::MMagic
. The alternative file.exe for MS-Windows can be obtained from the GnuWin32 home page. Be sure to save it as file.exe somewhere in your PATH.
It is preferred to use File::MMagic. The file executable seems to make mistakes occationally.
This is used to support read/write of gzip compressed files. It is only needed when gzip compressed files are encountered. If it is not available when needed, arclog will try to use the gzip executable instead. If that is not available, too, arclog will fail. You can download and install Compress::Zlib from the CPAN archive, or install it with the CPAN shell by running cpan Compress::Zlib
, or install it with the CPANPLUS shell by running cpanp i Compress::Zlib
. Debian/Ubandu Linux users can install it by running apt-get install libcompress-zlib-perl
. Red Hat/Fedora/CentOS Linux users can install it by running yum install perl-Compress-Zlib
. FreeBSD users can install it by running ports install p5-Compress-Zlib
. ActivePerl users can install it by running ppm install Compress::Zlib
. The alternative gzip.exe for MS-Windows can be obtained from the gzip home page. Be sure to save it as gzip.exe somewhere in your PATH.
This is used to support read/write of bzip2 compressed files. It is only needed when bzip2 compressed files are encountered. If it is not available when needed, arclog will try to use the bzip2 executable instead. If that is not available, too, arclog will fail. Notice that older versions before 2 does not work, since file I/O compression were not implemented yet. You can download and install Compress::Bzip2 from the CPAN archive, or install it with the CPAN shell by running cpan Compress::Bzip2
, or install it with the CPANPLUS shell by running cpanp i Compress::Bzip2
. Debian/Ubandu Linux users can install it by running apt-get install libcompress-bzip2-perl
. Red Hat/Fedora/CentOS Linux users can install it by running yum install perl-Compress-Bzip2
. FreeBSD users can install it by running ports install p5-Compress-Bzip2
. ActivePerl users can install it by running ppm install Compress::Bzip2
. The alternative bzip2.exe for MS-Windows can be obtained from the bzip2 home page. Be sure to save it as bzip2.exe somewhere in your PATH.
This is used to display the progress bar. Without this arclog will not display the progress bar, but nothing else is different. The progress bar is a good visual representation of what arclog is currently doing. You can download and install Term::ReadKey from the CPAN archive, or install it with the CPAN shell by running cpan Term::ReadKey
, or install it with the CPANPLUS shell by running cpanp i Term::ReadKey
. Debian/Ubandu Linux users can install it by running apt-get install libterm-readkey-perl
. Red Hat/Fedora/CentOS Linux users can install it by running yum install perl-TermReadKey
. FreeBSD users can install it by running ports install p5-Term-ReadKey
. ActivePerl users can install it by running ppm install Term::ReadKey
.
arclog’s official websites is at…
You can always download the newest version of arclog from…
imacat’s PGP public key is at…
If you are upgrading from arclog.pl 2.1.1dev4 or earlier, please read UPGRADE for some upgrade instruction.
arclog uses standard Perl installation with ExtUtils::MakeMaker. Follow these steps:
% perl Makefile.PL % make % make test % make install
When running make install
, make sure you have the priviledge to write to the installation location. This usually requires the root priviledge.
If you are using ActivePerl under MS-Windows, you should use nmake instead of make. nmake can be obtained from the Microsoft FTP site.
If you want to install into another location, you can set the PREFIX. For example, to install into your home when you are not root:
% perl Makefile.PL PREFIX=/home/jessica
Refer to the docuemntation of ExtUtils::MakeMaker for more installation options (by running perldoc ExtUtils::MakeMaker
).
You can install with Module::Build instead, if you prefer. Follow these steps:
% perl Build.PL % ./Build % ./Build test % ./Build install
When running ./Build install
, make sure you have the priviledge to write to the installation location. This usually requires the root priviledge.
If you want to install into another location, you can set the --prefix. For example, to install into your home when you are not root:
% perl Build.PL --prefix=/home/jessica
Refer to the docuemntation of Module::Build for more installation options (by running perldoc Module::Build
).
./arclog [options] logfile… [output] ./arclog [-h|-v]
The log file to be archived. Specify -
to read from STDIN. Multiple log files are supported. gzip or bzip2 compressed files are supported, too.
The prefix of the output files. The output files will be named as output.yyyymm, i.e., output.200101, output.200102. If not specified, the default is the same as the log file. You must specify this if you want to read from STDIN. You cannot specify -
(STDIN), since this is only a name prefix, not the output file.
Specify the compression method for the archived files. Log files usually have large number of simular lines. Compress them saves you lots of disk spaces. (And this is why we want to archive them.) Currently the following compression methods are supported:
Compress with gzip. This is the default. arclog can use Compress::Zlib to compress instead of calling gzip. This can be safer and faster for not calling foreign binaries. But if Compress::Zlib is not installed, it will try to use gzip binary instead. If gzip binary is not available, either, but gzip compression is required, the program will fail.
Compress with bzip2. arclog can use Compress::Bzip2 to compress instead of calling bzip2. This can be safer and faster for not calling foreign binaries. But if Compress::Bzip2 is not installed, it will try to use bzip2 binary instead. If bzip2 binary is not available, either, but bzip2 compression is required, the program will fail.
No compression at all. (Why? :p)
Do not compress the archived files. This is equal to --compress none
.
Sort the records by time (and then the record order). Sorting eats huge memory and CPU, so it is disabled by default. See the description above for a detailed illustration on sorting.
Do not sort the records. This is the default.
Whether we should overwrite the existing archived files. Currently the following modes are supported:
Overwrite existing target files. You will lost these existing records. Use with care. This is helpful if you are sure the master log file has the most complete records.
Append the records to the existing target files. You may destroy the log file completely by putting irrelevant entries altogether accidently. Use with care. This is helpful if you append want to merge 2 or more log files, for ex., 2 log files of different periods.
Ignore any existing target file, and discard all the records of those months. You will lost these log records. Use with care. This is helpful if you are supplying log records for the missing months, or if you are merging the log records in a complex manner.
Stop processing whenever a target file exists, to prevent destroying existing files by accident. This should be mostly wanted when run from some automatic mechanism, like crontab. So, this is the default if no terminal is found at STDIN.
Ask you what to do when a target file exists. This should be most wanted if you are running arclog interactively. So, this is the default if a terminal is found at STDIN. The answers are read from STDIN. Since you have only one STDIN, you cannot specify this mode if you want read the log file from STDIN. In that case, it will fall back to the fail mode. Also, if arclog cannot get its answer from STDIN, for ex., on a closed STDIN like crontab, it will fall back to fail mode.
What to keep in the source file. Current the following modes are supported:
Keep the source file after records are archived.
Restart the source log file after records are archived.
Delete the source log file after records are archived.
Archive and strip records of previous months off from the log file. Keep the records of this month in the source log file, to be archived next month. This is designed to be run from crontab monthly, so this is the default.
Show the detailed debugging messages.
Shihhhhhh. Only yell when errors.
Display the help message and exit.
Output version information and exit.
Copyright © 2001-2007 imacat..
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see <http://www.gnu.org/licenses/>.
Please read the NEWS for the new functions and bug fixes.
arclog is hosted on SourceForge, CPAN and Tavern IMACAT’s. For the latest infomation, see
There is a arclog mailing list hosted at SourceForge. Please submit your questions, suggestions or bug reports there. It is a Mailman mailing list. For more information, see the arclog mailing list page. Alternatively, you can send a mail to the e-mail command mailbox with the subject help for a list of available e-mail commands.