loki-enhance-middleware
loki-enhance-middleware hijacks log push requests sent to loki and modifies it.
Deploy
docker-compose.yaml
services:
loki:
image: grafana/loki:2.6.1
restart: always
expose:
- "3100"
# ...
enhance_middleware:
image: nihiue/loki_enhance_middleware:latest
restart: always
environment:
- LOKI_HOST=http://loki:3100
- WORKER_COUNT=1
- PORT=3100
expose:
- 3100
Config LogAgent
LogAgent needs to:
- Send requests to middleware
- Add
placeholderto log line for middle to process, seeModule - Geo IPfor example
promtail-config.yaml
clients:
- url: http://enhance_middleware:3100/loki/api/v1/push
Module - Geo IP
Injects GeoIP info for any log source
Powered by maxmind and maxmind-npm
promtail-config.yaml
scrape_configs:
- job_name: caddy
pipeline_stages:
- json:
expressions:
level:
status:
host: request.host
method: request.method
url: request.uri
remote_addr: request.remote_addr
- labels:
level:
status:
method:
- template:
source: output_msg
template: 'url="{{ .url }}" host="{{ .host }}" GeoIP_Source="{{.remote_addr }}"'
- output:
source: output_msg
GeoIP_Source="[IP]" is the placeholder, and it will be replaced by geo-ip fileds.
Log line
foo=bar GeoIP_Source="22.22.22.22" abc=xyz
Result
foo=bar geo_ip_asn="HostSlick" geo_ip_continent="North America" geo_ip_city="Ashburn" geo_ip_city_geoname_id="4744870" geo_ip_country="United States" geo_ip_country_geoname_id="6252001" geo_ip_country_iso_code="US" geo_ip_latitude="39.018" geo_ip_longitude="-77.539" abc=xyz
Module - UserAgent Detect
Parse user_agent field to structure data.
Powered by device-detector-js
Placeholder: Device_UA_Source="[UA]"
Log line - Normal
Device_UA_Source="Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36"
Result
ua_client="Chrome Mobile;90.0" ua_device="Google;Pixel 5" ua_os="Android;11.0"
Log line - Search Engine Bot
Device_UA_Source="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
Result
ua_bot="Googlebot"
28 Dec 20, 2022
4 May 3, 2022
14 Jan 3, 2023
1 Jan 3, 2022
9 Jan 2, 2023
13 Nov 3, 2022