StatsRaven

Sending data

Everything StatsRaven stores belongs to a project: a site, an app or a service. Each way of sending data below needs an ingest token for that project, sent as Authorization: Bearer <token>. The browser snippet is the exception: it uses a site key, see Browser snippet.

The quickest start is Get started in the app, or Settings, Projects, Add a source: they show each setup below with your address and a new token filled in, and a light that turns green when the first event arrives.

Projects and tokens

JSON over HTTP

Send one JSON object, an array, or one object per line to /ingest. Add ?source=name to label where it came from.

curl -X POST https://stats.example.com/ingest \
  -H 'Authorization: Bearer YOUR_TOKEN' \
  -H 'Content-Type: application/json' \
  -d '{"message": "hello from shop", "level": "info", "source": "test"}' 

StatsRaven reads these fields by name and keeps every other field as an attribute you can query:

MeaningField names
Timets, timestamp, time, @t, @timestamp, date
Event nameevent, event_name, name
Levellevel, severity, lvl, @l
Messagemessage, msg, @m, @mt
Requestpath, url, route, method, status

A record with an event name counts as a product event; one without is an operational log line.

OpenTelemetry

StatsRaven takes OTLP logs and traces over HTTP at /v1/logs and /v1/traces, as protobuf or JSON, and over gRPC on port 4317. In an OpenTelemetry Collector:

exporters:
  otlphttp/statsraven:
    endpoint: https://stats.example.com
    headers:
      Authorization: Bearer YOUR_TOKEN
service:
  pipelines:
    logs:
      exporters: [otlphttp/statsraven]

Add traces to the pipelines to send spans too. service.name becomes the source, and spans keep their trace and span IDs and duration. Spans with gen_ai.* attributes are marked as agent activity. For gRPC, send the same header as authorization metadata to port 4317, which listens on STATSRAVEN_OTLP_GRPC.

Fluent Bit

[INPUT]
    Name   tail
    Path   /var/log/nginx/access.log

[OUTPUT]
    Name   http
    Match  *
    Host   stats.example.com
    Port   443
    TLS    On
    URI    /ingest
    Format json
    Header Authorization Bearer YOUR_TOKEN

Vector

[sources.logs]
type = "file"
include = ["/var/log/nginx/access.log"]

[sinks.statsraven]
type = "http"
inputs = ["logs"]
uri = "https://stats.example.com/ingest"
encoding.codec = "json"
auth.strategy = "bearer"
auth.token = "YOUR_TOKEN"

Cloudflare Workers

One tail Worker forwards the logs of every Worker you choose. Make a Worker named statsraven-tail:

export default {
  async tail(events, env) {
    const items = events.filter((e) => e.scriptName);
    if (items.length === 0) return;
    await fetch("https://stats.example.com/ingest/workers", {
      method: "POST",
      headers: { "content-type": "application/json", authorization: `Bearer ${env.INGEST_TOKEN}` },
      body: JSON.stringify(items),
    });
  },
};

Deploy it with npx wrangler deploy, store the token with npx wrangler secret put INGEST_TOKEN, then add this to the wrangler config of each Worker to measure and deploy it:

"tail_consumers": [{ "service": "statsraven-tail" }]

Each Worker becomes a source named after its script. To file one Worker under another project, use Settings, Projects, or statsraven route-worker --script api --project shop --via main. Logpush jobs can send to the same address too; while the tail Worker sends a script's logs, Logpush copies of it are dropped.

Log files on the StatsRaven server

For a log on the machine StatsRaven runs on, open the project in Settings, Projects, choose Log file and give its absolute path. StatsRaven follows it from then on, through log rotation, and new lines arrive within ten seconds. Formats: web server access log, application log, or the systemd journal (all of it, or one unit such as nginx.service).

The same from the command line:

statsraven watch --project shop --source nginx --format access /var/log/nginx/access.log
statsraven watch-journal --project shop --source app --unit app.service

StatsRaven will not follow its own data folder, /proc, /sys or /dev. The service installed by --service can read /var/log and the journal; for logs elsewhere, check the service user can read them. For logs on other machines, use Vector or Fluent Bit.

To load old files once, stop the server first and run statsraven ingest --project shop --format access access.log.1 access.log.2.gz. Formats: access, json, workers and text.

Answers and limits