I installed and configured Maxmind's GeoIP database so AWStats could map IP addresses to countries. That explains the $1 " " $2 bit in my awk program and the use of sed in that other blog post. The %time2 field requires the concatenation of the first two CloudFront fields, date and time.
If your log files are different from mine, refer to the LogFormat documentation for help. If I didn't do that, there would be about 20 extra columns that I'd have to tell AWStats to ignore using the %other identifier. The LogFormat value is shorter because of the awk filtering I did in the bash script. There are several things in this file that need to be explained: LoadPlugin="geoip GEOIP_STANDARD /usr/share/GeoIP/GeoIP.dat" LogFormat="%time2 %bytesd %host %method %url %code %referer %ua" When I first installed AWStats, the relevant files were found in /usr/share/awstats/LogType=W
#Awstats io how to
There aren't many hints as to how to create a configuration file manually.īelow is the actual AWStats configuration file I am using for this website, saved to /var/The file needs to be in the same location as. The awstats_ script seems to want to extract information from the web server's configuration, but without an actual web server, that approach won't work. The most confusing part of the AWStats instructions is the configuration. log" rm " $TEMP_OUTPUT " AWStats Configuration #!/bin/bash LOG_DIR = "/local/DATA/aws_cloudfront_logs" # create an empty output file TEMP_OUTPUT = "/tmp/temp_merged. The critical outcome is that the everything is combined into one file and that the logs are sorted by timestamp. The result of this script could have been accomplished in other ways. There's some extra nonsense in there to mark each gzip file as "done" so I don't read the same log messages over and over. The sorting could have been done with but I chose not to do it that way. The script sorts the logs, as properly sorted logs are necessary for AWStats to work. The bash script also does some data cleaning to remove the unneeded columns and comments. Instead I wrote a bash script to read each gzip file one at a time and append to a single merged log file. bash: /usr/bin/gzip: Argument list too long
One could try to combine them with a command like zcat *.gz > /tmp/combined_logs.log but there are too many little files for that to work: $ gzip *.gz > /tmp/combined_logs.log The second step is to combine all of the data into one log file. My use of the -exclude and -include parameters limit the syncing to files from this year. The log data is stored in many small gzip files.
#Awstats io code
This accomplishes the same task as the Python code in that other blog post I found.
#Awstats io download
I use the aws s3 sync command to download only new log files to my computer. Standard CloudFront logging writes the log files to a S3 bucket. Obviously the first step is to obtain the log files. This blog post will document what I learned while getting this to work for me. I was able to find a single blog post from 2011 documenting how to process CloudFront logs with AWStats, and although that post was helpful, I believe more needs to be said about how to shoehorn CloudFront logs into something AWStats can use. That isn't the case when using CloudFront. The AWStats documentation seems to assume that you are using AWStats on the actual web server generating the logs, or at least that you have access to normal web server logs. How does one use AWStats to analyze AWS CloudFront logs?ĪWStats is a widely used tool to analyze website logs, but unfortunately there is not much information available on how to use it with AWS's (Standard) CloudFront logs.
0 kommentar(er)
