nsftools - Using AWStats To Report On Domino Web Logs

Using AWStats To Report On Domino Web Logs

Lotus Notes/Domino doesn't have very good built-in web log reporting functionality (and by that I'm referring to logs of website traffic, not blogs). As a web server, Domino can send traffic information to text files or to a DomLog database, but there's no real reporting built-in, so you're stuck with reams of information and no good way to look at it.

Many people make modifications to the DomLog.nsf database design, which can be useful, but logging stats to a database can cause performance problems on busy servers, and a large DomLog database is pretty slow and unwieldy in the middle of the day.

Fortunately, if you send your web log stats to a text file, it gets written in a standard logging format which can be parsed and sliced and diced by a number of popular reporting applications. One of the nicer log reporting tools available is the free and open-source AWStats application, which not only has really nice default reporting options, but is also very customizable and has a number of plug-ins. It can also be used on Windows, Unix, or Linux servers, because it's just a series or Perl scripts (very complex ones, but that's what they are). As an example for how nice the reports look, you can look at the stats for the AWStats site.

Here are some pretty basic step-by-step directions on how to set up your Lotus Domino server to use the AWStats tool (note that when I make reference to "%DOMINO_DATA_DIR%" below, I mean the Domino data directory on your server -- usually something like "d:\domino\data"). I encourage you to also refer to both the Domino Administrator Help and the AWStats documentation for more information and explanation.

Instructions for setting up AWStats on Domino

1. Set up your Domino server to log to text files. Open the server document in your Public Name and Address Book/Domino Directory and go to the "Internet Protocols - HTTP" tab, and set the following fields:

Access Log Format = "Extended Common"

Time Format = "LocalTime"

Log File Duration = "Daily"

Max Log Entry Length = "10 kb" (release 6 and higher)

Max size of access log = "0" (release 6 and higher)

Enable logging to log files = "Enabled"

Enable logging to Domlog.nsf = "Disabled"

Directory for log files = "d:\domino\weblogs"

Save and close the server document, and restart HTTP on the server.

2. Download AWStats from http://awstats.sourceforge.net

3. Extract AWStats.zip to a directory of your choice.

4. Create a directory called "awstats" under the %DOMINO_DATA_DIR%\domino\html directory on your server (for example, "d:\domino\data\domino\html\awstats").

5. Copy the "wwwroot\icon" directory from the awstats files to %DOMINO_DATA_DIR%\domino\html\awstats (for example: "d:\domino\data\domino\html\awstats\icon")

6. Copy the contents of the "wwwroot\cgi-bin" directory from the awstats files (and all subdirectories beneath it) to your %DOMINO_DATA_DIR%\domino\cgi-bin directory on the Notes server (for example, "d:\domino\data\domino\cgi-bin"). Also copy the contents of the "tools" directory from awstats to %DOMINO_DATA_DIR%\domino\cgi-bin -- in other words, take all of the .pl files from awstats and put them in your "cgi-bin" directory.

7. From the Domino "cgi-bin" directory, make a copy of the awstats.model.conf file. Call it "awstats.domino.conf". If the file is marked as "Read-Only", you'll want to go to the file properties and uncheck that flag so you can edit it. Open the file in a text editor and make the following changes (please note that Windows users will not be able to read the file very well using Notepad because it uses Unix-style linefeeds, so you'll want to open it in Wordpad or Crimson Editor or something like that):

Change the "LogFile" entry to be:
LogFile="perl ./logresolvemerge.pl d:/domino/weblogs/access*.log |"
where "d:/domino/weblogs" is the path to the log files you defined in step 1 (make sure you use "/" as a directory separator, even on Windows). Notice that the command ends with a vertical bar (pipe) character -- this is important.
Change "LogType" to W
Change "LogFormat" to 1
Change "SiteDomain" to be the main domain name for this server (like "www.servername.com")
Change "DirIcons" to the relative HTTP path of your icon directory from step 5 above ("../awstats/icon" should work).
Change "AllowToUpdateStatsFromBrowser" to "1" if you want to be able to update the stats from a browser (DANGER -- this could be resource intensive on a busy server); otherwise, if you will be doing updates programmatically, it should be "0".
optionally add the sample Notes file tracking report additions (see below) to the # EXTRA SECTIONS section near the bottom of the file.

8. If you don't have Perl installed on your server, download and install it. For Windows, you can use the free ActivePerl distribution at http://www.activestate.com/ActivePerl. The install for ActivePerl doesn't require a reboot, so you can actually install it on a live Domino server if you're feeling brave.

9. Run your initial update by opening a command prompt, navigating to the "cgi-bin" directory from step 6, and entering the following command:

perl awstats.pl -config=domino -update

Where "domino" in the "-config=domino" parameter refers to the middle name in the "awstats.domino.conf" filename from step 7. This will parse your log files (as defined in the "LogFile" parameter in your .conf file, from step 7) and create an index file that's used to build the reports. The first time you run the update process against a large set of data, it may take a few minutes.

If you get errors when you try to run this command, one of the following things probably happened:

The directory you're sending log files to in step 1 doesn't correspond to the directory you specified for the "LogFile" entry in step 7.

The log file directory doesn't exist yet, either because you forgot to enable logging to text files in step 1, or because you forgot to restart the HTTP process after making the settings in step 1, or because no one has accessed your Domino server via the web since you did both of those things.

The files in the log file directory are empty. This can happen if you just enabled logging to text files and people have accessed the server, but not enough log information has been collected yet to populate the log file. The log information is cached for efficiency, so it will only get written to the physical files when a certain amount of information has been accumulated, or when the HTTP process (or the entire server) has been shut down or restarted.

10. Once you've run the update command and you have an index file, you can manually create a report as an HTML page by running one of the following commands from the command prompt in the "cgi-bin" directory:

to build a single page for this month, run this from the "cgi-bin" directory (it will create a file in your DOMINO_DATA_DIR%\domino\html\awstats directory):

perl awstats.pl -config=domino -output -staticlinks > ../html/awstats/awstats.html

to build a single page for January 2004, run this from the "cgi-bin" directory (remove the line break, if there is one):

perl awstats.pl -config=domino -output -staticlinks -month=01 
-year=2004 > ../html/awstats/awstats200401.html

to build all stat pages for this month, run this from the "cgi-bin" directory:

perl awstats_buildstaticpages.pl -config=domino -dir=../html/awstats/

if you have the htmldoc program installed, you can also convert the html pages to PDF by adding the -buildpdf switch to this command:

perl awstats_buildstaticpages.pl -config=domino -dir=../html/awstats/ -buildpdf

Note that if you're running and viewing these files locally (not on your Domino web server), and you don't see any graphs or charts, you may need to adjust the "DirIcons" setting from step 7 and re-run the report.

11. To schedule your static reports to be built on a regular basis, use your favorite scheduling program and set awstats.pl or awstats_buildstaticpages.pl to run with whatever options you want from step 10, and include the "-update" switch.

12. Normally, you will want the "-update" process from step 9 to run every night, after midnight. The update process is the processor-intensive activity in this case -- it actually parses all the log files and writes the compiled information to an index file. Creating a report based on the index file is actually pretty quick.

While you could schedule this process through a Server Program document in Notes, you should probably use a scheduling program that runs separately from Notes (like "AT" on Windows, or "cron" on *nix), so you don't run into problems with using the Notes server memory space to run the updates.

13. Because it's pretty quick to run a report, you can also pull up the reports directly from a browser by navigating to this URL:

http://www.myserver.com/cgi-bin/awstats.pl?config=domino

where "www.myserver.com" is, obviously, the DNS name for your Domino server. This is the way that most web servers that use AWStats for reporting seem to work (including the AWStats site itself). One nice thing about this is that you can quickly and easily view prior month reports by adding parameters to the query string. For example:

http://www.myserver.com/cgi-bin/awstats.pl?config=domino&month=01&year=2004

will bring up the report for January 2004. In fact, the current version of AWStats allows you to choose a month and year from a drop-down menu when you run the report dynamically like that. Because you're pulling the data from an index file that's already been built, the report generation is fairly fast.

14. OPTIONAL: create a File Protection document to restrict access to the AWStats files on the server (see the Domino Administrator Help topic "Protecting files on a server from Web client access" for more information). You should definitely protect the "cgi-bin" directory, and if you're outputting any static reports to a directory, you can protect those as well.

NOTES:

you should use "/" instead of "\" for directory separators in the .conf file and on the command line arguments, even on a Windows machine.

you can optionally exclude image files from being logged, to reduce log file sizes. To do this, open your server document and go to the "Internet Protocols - HTTP" tab (as in step 1 above) and set the "Exclude from logging URLs" field to be "*.gif*; *.jpg*; *.bmp*; *.jpeg*; *.ico*".

if you modify the EXTRA SECTIONS reports in the .conf file, you may need to delete the index file and re-run -update for the changes to take effect for existing data.

if you want to be able to run or update the stats from the web, you should put the .pl files in the Domino cgi-bin directory (as described above) and use a File Protection document to secure them. Otherwise, you can move them to a different directory that's not accessible via the web.

you should run some kind of program/batch job that archives or deletes old log files after a certain amount of time. This will make AWStats run faster (because it won't have to parse and discard so many old entries), and save disk space on your server. I have an example of how to do this at the end of my Unfinished LotusScript Book, which is available as a free download.

Accessing awstats.pl directly from a browser requires perl.exe to be on the PATH of your server. As long as your Domino server can find perl.exe, it will automatically run cgi-bin files with a .pl extension as Perl scripts. See the Domino Designer Help files for more information (for some reason I only found this in the Designer Help files, but not the Administrator Help files).

If you get a large number of parsing errors when you run your updates, you may need to adjust the "LogFormat" setting in your awstats.domino.conf file (see step 7 above). Using a LogFormat of 1 worked for me in my limited testing, but the AWStats Documentation indicates that you may have to use a custom setting for some Domino servers. See the FAQ-SET250 section of the AWStats documentation for more details.

Additional Reports For Domino

Here are some example additions to the AWStats .conf file for Notes-specific file tracking (you may have to delete the index and re-run -update for them to take effect):

# report on the .nsf files that have been accessed
ExtraSectionName1="List of most active databases (top 50)"
ExtraSectionCodeFilter1="200 304"
ExtraSectionCondition1="URL,\.nsf\/"
ExtraSectionFirstColumnTitle1="Database Filename"
ExtraSectionFirstColumnValues1="URL,(.*\.nsf)"
ExtraSectionFirstColumnFormat1="%s"
ExtraSectionStatTypes1=PHB
MaxNbOfExtra1=50
MinHitExtra1=1

# report on the .gif, .jpg, and .jpeg files that have been accessed
ExtraSectionName2="List of most active images (top 20)"
ExtraSectionCodeFilter2="200 304"
ExtraSectionCondition2="URL,\.gif|URL,\.jp[e]?g"
ExtraSectionFirstColumnTitle2="Image Filename"
ExtraSectionFirstColumnValues2="URL,(.*\.gif)|URL,(.*\.jp[e]?g)"
ExtraSectionFirstColumnFormat2="%s"
ExtraSectionStatTypes2=HB
MaxNbOfExtra2=20
MinHitExtra2=1

# report on the top file downloads, assuming the URL had /$FILE/ in it
ExtraSectionName3="List of most active $FILES (top 20)"
ExtraSectionCodeFilter3="200 304"
ExtraSectionCondition3="URL,\/\$FILE\/"
ExtraSectionFirstColumnTitle3="Image Filename"
# this returns the whole line
#ExtraSectionFirstColumnValues3="URL,(.*)"
# this returns everything up to and including $FILE
#ExtraSectionFirstColumnValues3="URL,(.*\$FILE)"
# this returns everything before /$FILE
#ExtraSectionFirstColumnValues3="URL,(.*)(?=\/\$FILE)"
# this returns everything after /$FILE
ExtraSectionFirstColumnValues3="URL,\/\$FILE\/([^&]+)"
ExtraSectionFirstColumnFormat3="%s"
ExtraSectionStatTypes3=HB
MaxNbOfExtra3=20
MinHitExtra3=1