506 lines
15 KiB
Markdown
506 lines
15 KiB
Markdown
|
`reslog` - Reverse-resolve IP in Apache log files
|
|||
|
===============================================
|
|||
|
|
|||
|
|
|||
|
Description
|
|||
|
-----------
|
|||
|
|
|||
|
`reslog` reverse-resolves IP in the [Apache] log files. These log
|
|||
|
files can then be analyzed by another program, like [Analog]. You can
|
|||
|
think of it as a replacement of Apache `HostNameLookups` directive, in
|
|||
|
the sense that it batch resolves the client IP once a day.
|
|||
|
|
|||
|
[Apache]: https://httpd.apache.org
|
|||
|
[Analog]: https://www.c-amie.co.uk/software/analog
|
|||
|
|
|||
|
|
|||
|
Caution
|
|||
|
-------
|
|||
|
|
|||
|
* *Resolving takes long time*. This is mainly caused by the look up:
|
|||
|
Network packets may be filtered by firewalls; DNS servers may not be
|
|||
|
correctly configured; may not be up working; may sit in slow network
|
|||
|
sections; may be old slow machines; may have traffic jam… etc. All
|
|||
|
these problems are beyond our control.
|
|||
|
|
|||
|
* If it stops in the middle of its execution, as when the user hits a
|
|||
|
`Ctrl-Break`, it may leave a temporary working file. The next time
|
|||
|
it runs, it stops when it sees that temporary working file at the
|
|||
|
first sight. Please process that file first. You can resolve it
|
|||
|
again, just like an ordinary log file.
|
|||
|
|
|||
|
* `reslog` needs temporary working space. Disk space is cheaper and
|
|||
|
is more available than memory. However, this means that it needs
|
|||
|
free temporary disk space about 2 times of the size of the
|
|||
|
uncompressed source log file (10 times if using memory). Please
|
|||
|
make sure you have that much free space.
|
|||
|
|
|||
|
* `reslog` does not support IPv6 yet.
|
|||
|
|
|||
|
* I suggest that you install [File::MMagic] instead of counting on the
|
|||
|
`file` executable. The internal magic file of File::MMagic works
|
|||
|
better than the `file` executable. `reslog` treats everything not
|
|||
|
gzip nor bzip2 compressed as plain text. When a compressed log file
|
|||
|
is wrongly recognized as an image, `reslog` treats it as plain text,
|
|||
|
reads directly from it, and fails. This does not hurt the source
|
|||
|
log files, but is still annoying.
|
|||
|
|
|||
|
[File::MMagic]: https://metacpan.org/release/File-MMagic
|
|||
|
|
|||
|
|
|||
|
System Requirement
|
|||
|
------------------
|
|||
|
|
|||
|
1. Perl, version 5.8.0 or above. `reslog` uses 3-argument open() to
|
|||
|
duplicate file handles, which is only supported since 5.8.0. I
|
|||
|
have not successfully port this onto earlier versions yet. Please
|
|||
|
tell me if you made it. You can run `perl -v` to see your current
|
|||
|
Perl version. If you do not have Perl, or if you have an older
|
|||
|
version of Perl, you can download and install/upgrade it from the
|
|||
|
[Perl website]. If you are using MS-Windows, you can download and
|
|||
|
install [ActiveState ActivePerl].
|
|||
|
|
|||
|
2. Required Perl modules: None.
|
|||
|
|
|||
|
3. Optional Perl modules:
|
|||
|
|
|||
|
* [File::MMagic]
|
|||
|
|
|||
|
This is used to check the file type. If this is not available,
|
|||
|
`reslog` tries the `file` executable instead. If that is not
|
|||
|
available, too, `reslog` judges the file type by its name suffix
|
|||
|
(extension). In that case `reslog` fails when reading from
|
|||
|
`STDIN`. You can download and install File::MMagic from the CPAN
|
|||
|
archive, or install it with the CPAN shell:
|
|||
|
|
|||
|
cpan File::MMagic
|
|||
|
|
|||
|
or with the CPANPLUS shell:
|
|||
|
|
|||
|
cpanp i File::MMagic
|
|||
|
|
|||
|
For Debian/Ubuntu:
|
|||
|
|
|||
|
sudo apt install libfile-mmagic-perl
|
|||
|
|
|||
|
For Red Hat/Fedora/CentOS:
|
|||
|
|
|||
|
sudo yum install perl-File-MMagic
|
|||
|
|
|||
|
For FreeBSD:
|
|||
|
|
|||
|
ports install p5-File-MMagic
|
|||
|
|
|||
|
For ActivePerl:
|
|||
|
|
|||
|
ppm install File-MMagic
|
|||
|
|
|||
|
The alternative `file.exe` for MS-Windows can be obtained from
|
|||
|
the [GnuWin32] home page. Be sure to save it as `file.exe`
|
|||
|
somewhere in your `PATH`.
|
|||
|
|
|||
|
* [Compress::Zlib]
|
|||
|
|
|||
|
This is used to support reading/writing the gzip compressed
|
|||
|
files. It is only needed when gzip compressed files are
|
|||
|
encountered. If it is not available, `reslog` tries the `gzip`
|
|||
|
executable instead. If that is not available, too, `reslog`
|
|||
|
fails. You can download and install Compress::Zlib from the CPAN
|
|||
|
archive, or install it with the CPAN shell:
|
|||
|
|
|||
|
cpan Compress::Zlib
|
|||
|
|
|||
|
or with the CPANPLUS shell:
|
|||
|
|
|||
|
cpanp i Compress::Zlib
|
|||
|
|
|||
|
For Debian/Ubuntu:
|
|||
|
|
|||
|
sudo apt install libcompress-zlib-perl
|
|||
|
|
|||
|
For Red Hat/Fedora/CentOS:
|
|||
|
|
|||
|
sudo yum install perl-Compress-Zlib
|
|||
|
|
|||
|
For FreeBSD:
|
|||
|
|
|||
|
ports install p5-Compress-Zlib
|
|||
|
|
|||
|
For ActivePerl:
|
|||
|
|
|||
|
ppm install Compress-Zlib
|
|||
|
|
|||
|
The alternative `gzip.exe` for MS-Windows can be obtained from
|
|||
|
[the gzip website]. Be sure to save it as `gzip.exe` somewhere
|
|||
|
in your `PATH`.
|
|||
|
|
|||
|
* [Compress::Bzip2] version 2 or above.
|
|||
|
|
|||
|
This is used to support reading/writing the bzip2 compressed
|
|||
|
files. It is only needed when bzip2 compressed files are
|
|||
|
encountered. If it is not available, `reslog` tries the `bzip2`
|
|||
|
executable instead. If that is not available, too, `reslog`
|
|||
|
fails. Notice that older versions before 2 does not work, since
|
|||
|
the file I/O compression was not implemented yet. You can
|
|||
|
download and install Compress::Bzip2 from the CPAN archive, or
|
|||
|
install it with the CPAN shell:
|
|||
|
|
|||
|
cpan Compress::Bzip2
|
|||
|
|
|||
|
or with the CPANPLUS shell:
|
|||
|
|
|||
|
cpanp i Compress::Bzip2
|
|||
|
|
|||
|
For Debian/Ubuntu:
|
|||
|
|
|||
|
sudo apt install libcompress-bzip2-perl
|
|||
|
|
|||
|
For Red Hat/Fedora/CentOS:
|
|||
|
|
|||
|
sudo yum install perl-Compress-Bzip2
|
|||
|
|
|||
|
For FreeBSD:
|
|||
|
|
|||
|
ports install p5-Compress-Bzip2
|
|||
|
|
|||
|
For ActivePerl:
|
|||
|
|
|||
|
ppm install Compress-Bzip2
|
|||
|
|
|||
|
The alternative `bzip2.exe` for MS-Windows can be obtained from
|
|||
|
[the bzip2 website]. Be sure to save it as `bzip2.exe` somewhere
|
|||
|
in your `PATH`.
|
|||
|
|
|||
|
* [Term::ReadKey]
|
|||
|
|
|||
|
This is used to display the progress bar. The progress bar is a
|
|||
|
good visual feedback of what `reslog` is currently doing, but
|
|||
|
`reslog` is safe without it. You can download and install
|
|||
|
Term::ReadKey from the CPAN archive, or install it with the
|
|||
|
CPAN shell:
|
|||
|
|
|||
|
cpan Term::ReadKey
|
|||
|
|
|||
|
or with the CPANPLUS shell:
|
|||
|
|
|||
|
cpanp i Term::ReadKey
|
|||
|
|
|||
|
For Debian/Ubuntu:
|
|||
|
|
|||
|
sudo apt install libterm-readkey-perl
|
|||
|
|
|||
|
For Red Hat/Fedora/CentOS:
|
|||
|
|
|||
|
sudo yum install perl-TermReadKey
|
|||
|
|
|||
|
For FreeBSD:
|
|||
|
|
|||
|
ports install p5-Term-ReadKey
|
|||
|
|
|||
|
For ActivePerl:
|
|||
|
|
|||
|
ppm install TermReadKey
|
|||
|
|
|||
|
[Perl website]: https://www.perl.org
|
|||
|
[ActiveState ActivePerl]: https://www.activestate.com
|
|||
|
[File::MMagic]: https://metacpan.org/release/File-MMagic
|
|||
|
[GnuWin32]: http://gnuwin32.sourceforge.net
|
|||
|
[Compress::Zlib]: https://metacpan.org/release/Compress-Zlib
|
|||
|
[the gzip website]: https://www.gzip.org
|
|||
|
[Compress::Bzip2]: https://metacpan.org/release/Compress-Bzip2
|
|||
|
[the bzip2 website]: http://www.bzip.org
|
|||
|
[Term::ReadKey]: https://metacpan.org/release/TermReadKey
|
|||
|
|
|||
|
|
|||
|
Download
|
|||
|
--------
|
|||
|
|
|||
|
`reslog` is hosted is on…
|
|||
|
|
|||
|
* [reslog project on GitHub]
|
|||
|
|
|||
|
* [reslog project on SourceForge]
|
|||
|
|
|||
|
You can always download the newest version of `reslog` from…
|
|||
|
|
|||
|
* [reslog download on SourceForge]
|
|||
|
|
|||
|
* [Tavern IMACAT’s FTP directory]
|
|||
|
|
|||
|
imacat’s PGP public key is at…
|
|||
|
|
|||
|
* [imacat’s PGP key at Tavern IMACAT’s]
|
|||
|
|
|||
|
[reslog project on GitHub]: https://github.com/imacat/reslog
|
|||
|
[reslog project on SourceForge]: https://sf.net/p/reslog
|
|||
|
[reslog download on SourceForge]: https://sourceforge.net/projects/reslog/files
|
|||
|
[Tavern IMACAT’s FTP directory]: https://ftp.imacat.idv.tw/pub/reslog/
|
|||
|
[imacat’s PGP key at Tavern IMACAT’s]: https://www.imacat.idv.tw/me/pgpkey.asc
|
|||
|
|
|||
|
|
|||
|
Install
|
|||
|
-------
|
|||
|
|
|||
|
If you are upgrading from `reslog` 3.10 or earlier, or if you are
|
|||
|
upgrading from `reslog.pl` 3.02 or earlier, please read the upgrade
|
|||
|
instruction later in this document.
|
|||
|
|
|||
|
### Install with [ExtUtils::MakeMaker]
|
|||
|
|
|||
|
`reslog` uses standard Perl installation with ExtUtils::MakeMaker.
|
|||
|
Follow these steps:
|
|||
|
|
|||
|
% perl Makefile.PL
|
|||
|
% make
|
|||
|
% make test
|
|||
|
% make install
|
|||
|
|
|||
|
When running `make install`, make sure you have the privilege to
|
|||
|
write to the installation location. This usually requires the `root`
|
|||
|
privilege.
|
|||
|
|
|||
|
If you are using ActivePerl under MS-Windows, you should use `nmake`
|
|||
|
instead of `make`. [nmake can be obtained from the Microsoft FTP site.]
|
|||
|
|
|||
|
If you want to install into another location, you can set the
|
|||
|
`PREFIX`. For example, to install into your home when you are not
|
|||
|
`root`:
|
|||
|
|
|||
|
% perl Makefile.PL PREFIX=/home/jessica
|
|||
|
|
|||
|
Refer to the documentation of ExtUtils::MakeMaker for more
|
|||
|
installation options (by running `perldoc ExtUtils::MakeMaker`).
|
|||
|
|
|||
|
### Install with [Module::Build]
|
|||
|
|
|||
|
You can install with Module::Build instead, if you prefer. Follow
|
|||
|
these steps:
|
|||
|
|
|||
|
% perl Build.PL
|
|||
|
% ./Build
|
|||
|
% ./Build test
|
|||
|
% ./Build install
|
|||
|
|
|||
|
When running `./Build install`, make sure you have the privilege to
|
|||
|
write to the installation location. This usually requires the `root`
|
|||
|
privilege.
|
|||
|
|
|||
|
If you want to install into another location, you can set the
|
|||
|
`--prefix`. For example, to install into your home when you are not
|
|||
|
`root`:
|
|||
|
|
|||
|
% perl Build.PL --prefix=/home/jessica
|
|||
|
|
|||
|
Refer to the documentation of Module::Build for more installation
|
|||
|
options (by running `perldoc Module::Build`).
|
|||
|
|
|||
|
[ExtUtils::MakeMaker]: https://metacpan.org/release/ExtUtils-MakeMaker
|
|||
|
[nmake can be obtained from the Microsoft FTP site.]: ftp://ftp.microsoft.com/Softlib/MSLFILES/nmake15.exe
|
|||
|
[Module::Build]: https://metacpan.org/release/Module-Build
|
|||
|
|
|||
|
|
|||
|
Upgrade Instruction
|
|||
|
-------------------
|
|||
|
|
|||
|
Here are a few hints for people upgrading from 3.10 or earlier:
|
|||
|
|
|||
|
### The Default Installation Location Is at `/usr/bin`
|
|||
|
|
|||
|
Also, the man page is at `/usr/share/man/man1/reslog.1`. This is to
|
|||
|
follow Perl’s standard convention, and to avoid breaking
|
|||
|
ExtUtils::MakeMaker with future versions.
|
|||
|
|
|||
|
When you run `perl Makefile.PL` or `perl Build.PL`, it hints a
|
|||
|
list of existing old files to be removed. Please delete them
|
|||
|
manually.
|
|||
|
|
|||
|
If you saved them in other places, you have to delete them yourself.
|
|||
|
|
|||
|
Also, if you have any scripts or cron jobs that are running `reslog`,
|
|||
|
remember to modify your script for the new `reslog` location. Of
|
|||
|
course, you can copy `reslog` to the original location. It still
|
|||
|
works.
|
|||
|
|
|||
|
|
|||
|
Here are a few hints for people upgrading from 3.02 or earlier:
|
|||
|
|
|||
|
### The Script Name is Changed from `reslog.pl` to `reslog`
|
|||
|
|
|||
|
This is obvious. If you have any scripts or cron jobs that are
|
|||
|
running `reslog`, remember to modify your script for the new name.
|
|||
|
Of course, you can rename `reslog` to `reslog.pl`. It still works.
|
|||
|
|
|||
|
The reason I changed the script and project name is that: A dot `.`
|
|||
|
in the project name is not valid everywhere. At least SourceForge
|
|||
|
don’t accept it. Besides, `reslog` is enough for a script name under
|
|||
|
UNIX. The `.pl` file name suffix/extension may be convenient on
|
|||
|
MS-Windows, but MS-Windows users won’t run it with explorer file name
|
|||
|
association anyway, and there is a `pl2bat` to convert `reslog` to
|
|||
|
`reslog.bat`, which would make more sense. The only disadvantage is
|
|||
|
that I was using `UltraEdit`, which depends on the file name extension
|
|||
|
for the syntax highlighting rules. I can manually set it anyway. I’m
|
|||
|
using `gedit` on Linux now. This is not a problem anymore.
|
|||
|
|
|||
|
|
|||
|
### You Need Perl 5.8.0 or Above
|
|||
|
|
|||
|
`reslog` now has threading to speed up resolving, which requires
|
|||
|
Perl’s `ithreads` threading module support that’s only available since
|
|||
|
5.8.0. You can still disable threading if it causes troubles to you,
|
|||
|
but the code itself need it. If you are using a Perl before 5.8.0,
|
|||
|
please upgrade it. You can run `perl -v` to see your current Perl
|
|||
|
version.
|
|||
|
|
|||
|
|
|||
|
### The Default Keep Mode is Now `delete`
|
|||
|
|
|||
|
The documentation said the default keep mode is `delete`, but `reslog`
|
|||
|
actually did a `restart`. :p This is fixed. If you are running with
|
|||
|
the default keep mode, remember to fix it.
|
|||
|
|
|||
|
|
|||
|
### The Argument of `--keep` and `--override` Options Are Required Now
|
|||
|
|
|||
|
Support for omitting the `--keep` or `--override` arguments are
|
|||
|
removed. This helps to avoid confusion for the log file name and the
|
|||
|
option arguments.
|
|||
|
|
|||
|
|
|||
|
### Specifying One `STDIN` No Longer Trigger Everything to `STDOUT`
|
|||
|
|
|||
|
When resolving multiple files, `STDIN` can output to `STDOUT` now,
|
|||
|
with other files output to where they should be. Specifying one
|
|||
|
`STDIN` no longer writes everything to `STDOUT`. If you want to write
|
|||
|
everything to `STDOUT`, be sure to add the `--stdout` option.
|
|||
|
|
|||
|
|
|||
|
Options
|
|||
|
-------
|
|||
|
|
|||
|
./reslog [options] [logfile…]
|
|||
|
./reslog [-h|-v]
|
|||
|
|
|||
|
* `logfile`
|
|||
|
|
|||
|
The log file to be resolved. You can specify multiple log files.
|
|||
|
If not specified, it reads from `STDIN` and outputs to `STDOUT`.
|
|||
|
You can also specify `-` to read from `STDIN`. Result of `STDIN`
|
|||
|
goes to `STDOUT`. `gzip` or `bzip2` compressed files are supported.
|
|||
|
|
|||
|
* `-k`, `--keep mode`
|
|||
|
|
|||
|
What to keep in the source file. The following modes are supported:
|
|||
|
|
|||
|
* `a`, `all`
|
|||
|
|
|||
|
Keep the source file after records are archived.
|
|||
|
|
|||
|
* `r`, `restart`
|
|||
|
|
|||
|
Restart the source file after records are resolved.
|
|||
|
|
|||
|
* `d`, `delete`
|
|||
|
|
|||
|
Delete the source file after records are resolved. This is the
|
|||
|
default.
|
|||
|
|
|||
|
* `-o`, `--override mode`
|
|||
|
|
|||
|
What to do with the existing resolved files. The following modes
|
|||
|
are supported:
|
|||
|
|
|||
|
* `o`, `overwrite`
|
|||
|
|
|||
|
Overwrite existing target files.
|
|||
|
|
|||
|
* `a`, `append`
|
|||
|
|
|||
|
Append the records to existing target files.
|
|||
|
|
|||
|
* `f`, `fail`
|
|||
|
|
|||
|
Stop processing whenever a target file exists, to prevent
|
|||
|
destroying existing files by accident. This is the default.
|
|||
|
|
|||
|
* `-s`, `--suffix suf`
|
|||
|
|
|||
|
The suffix to be appended to the output file. If not specified,
|
|||
|
the default is `.resolved`.
|
|||
|
|
|||
|
* `-t`, `--trim-suffix suf`
|
|||
|
|
|||
|
The suffix to be trimmed from the input file name before appending
|
|||
|
the above suffix. Default is none. If you are running several log
|
|||
|
file filters, this can help you trim the suffix of the previous one.
|
|||
|
|
|||
|
* `-n`, `--num-threads num`
|
|||
|
|
|||
|
Number of threads to run simultaneously. The default is 10. Use 0
|
|||
|
to disable threading. Your system must support threading itself.
|
|||
|
This option has no effect for systems that do not support threading.
|
|||
|
|
|||
|
* `-c`, `--stdout`
|
|||
|
|
|||
|
Output the result to `STDOUT`.
|
|||
|
|
|||
|
* `-d`, `--debug`
|
|||
|
|
|||
|
Show the detailed debugging messages. More `-d` to be more
|
|||
|
detailed.
|
|||
|
|
|||
|
* `-q`, `--quiet`
|
|||
|
|
|||
|
Hush! Only yell on error.
|
|||
|
|
|||
|
* `-h`, `--help`
|
|||
|
|
|||
|
Display the help message and exit.
|
|||
|
|
|||
|
* `-v`, `--version`
|
|||
|
|
|||
|
Output version information and exit.
|
|||
|
|
|||
|
|
|||
|
Documentation
|
|||
|
-------------
|
|||
|
|
|||
|
Type `perldoc reslog` to read the `reslog` manual.
|
|||
|
|
|||
|
|
|||
|
News, Changes and Updates
|
|||
|
-------------------------
|
|||
|
|
|||
|
Refer to the `Changes` for changes, bug fixes, updates, new functions,
|
|||
|
etc.
|
|||
|
|
|||
|
|
|||
|
Support
|
|||
|
-------
|
|||
|
|
|||
|
The `reslog` project is hosted on GitHub. Address your issues on the
|
|||
|
GitHub issue tracker https://github.com/imacat/reslog/issues.
|
|||
|
|
|||
|
|
|||
|
License
|
|||
|
-------
|
|||
|
|
|||
|
Copyright (C) 2000-2021 imacat.
|
|||
|
|
|||
|
Licensed under the Apache License, Version 2.0 (the "License");
|
|||
|
you may not use this file except in compliance with the License.
|
|||
|
You may obtain a copy of the License at
|
|||
|
|
|||
|
http://www.apache.org/licenses/LICENSE-2.0
|
|||
|
|
|||
|
Unless required by applicable law or agreed to in writing, software
|
|||
|
distributed under the License is distributed on an "AS IS" BASIS,
|
|||
|
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|||
|
See the License for the specific language governing permissions and
|
|||
|
limitations under the License.
|
|||
|
|
|||
|
|
|||
|
imacat ^_*'
|
|||
|
2007/12/4
|
|||
|
<imacat@mail.imacat.idv.tw>
|
|||
|
https://www.imacat.idv.tw
|