Logging¶
phēnix features a centralized logging system that aggregates logs from the core daemon, internal Go services, and external user applications.
Architecture¶
The logging architecture is designed to be centralized. The phēnix core acts as the aggregator, capturing stderr from all running applications, parsing the output as JSON, and ingesting it into the centralized logging system.
graph TD
subgraph "phēnix core"
Core[phēnix Daemon]
Plog[plog Package]
FileLog[File Logger]
ConsoleLog[Console Logger]
UILog[UI Logger]
Config[Viper Config]
end
subgraph "Go Apps"
GoApp[Go Application]
GoSlog[slog JSON Handler]
end
subgraph "Python Apps"
PyApp[Python Application]
PyLog[Python Logger]
PySink[phēnix Stderr Sink]
end
Config -->|Watches config.yaml| Plog
Core -->|Uses| Plog
Plog -->|Write JSON/Text| FileLog
Plog -->|Write Text| ConsoleLog
Plog -->|Publish| UILog
GoApp -->|Log| GoSlog
GoSlog -->|JSON via Stderr| Core
PyApp -->|Log| PyLog
PyLog -->|Log| PySink
PySink -->|JSON via Stderr| Core
Core -->|ProcessStderrLogs| Plog
CLI Verbosity (--log.level)¶
You can temporarily increase or decrease log verbosity for any command by using the global --log.level flag.
This is useful when troubleshooting because it only affects that command invocation.
Available Levels¶
phēnix supports the following log levels for --log.level:
debug: Most verbose output for troubleshooting.info: Default level. Normal operational messages.warn: Warnings and errors.error: Errors only.none: Suppress all standard log output.
Note
Level names are case-insensitive (for example, DEBUG and debug are equivalent).
Examples¶
Use higher verbosity to inspect behavior while running a command:
# Run the UI with debug logging for this invocation
phenix ui --log.level=debug
# Show only warnings and errors while listing experiments
phenix exp list --log.level=warn
# Keep command output quiet unless an error occurs
phenix config get --log.level=error Role/vm-admin
To change the default level for all future commands, use settings:
To revert and use the configured default again for subsequent commands:
The App Contract¶
All applications (Go or Python) running under phēnix must adhere to the following contract to ensure their logs are correctly parsed and displayed in the UI:
- Output: Logs must be written to
stderr. - Format: Logs must be single-line JSON objects.
- Required Fields:
level:DEBUG,INFO,WARN,ERRORmsg: The log message string.time: Timestamp (RFC3339 or similar).
- Optional Fields:
traceback: For exceptions/panics (string).
Tip
Check out the Example Applications for complete, runnable reference implementations in Go and Python.
Python Apps¶
Use the phenix_apps.common.logger. It is pre-configured to output JSON to stderr.
Important
For fatal errors, raise an exception (e.g., ValueError, RuntimeError) instead of calling sys.exit(1). The AppBase class wraps execution in a try/except block and will automatically catch the exception, log the traceback as structured JSON, and exit cleanly.
from phenix_apps.common.logger import logger
def my_func():
try:
logger.bind(custom_field="value").info("Starting operation")
# ... code ...
except Exception:
# Automatically captures traceback and formats as JSON
logger.exception("Operation failed")
Go Apps¶
Use log/slog with a JSON handler.
import (
"log/slog"
"os"
)
func main() {
logger := slog.New(slog.NewJSONHandler(os.Stderr, nil))
slog.SetDefault(logger)
slog.Info("Application started", "app", "my-app")
}
phēnix Core¶
Use the phenix/util/plog package for logging within the core application.
import "phenix/util/plog"
func MyFunc() {
// Always provide a LogType enum for UI filtering
plog.Info(plog.TypeSystem, "System initialized", "version", "1.0")
}
The log types are AUDIT, SECURITY, SOH, SCORCH, PHENIX-APP, ACTION,
HTTP, MINIMEGA and SYSTEM. Web handlers log AUDIT records with the
phenix/web/audit package rather than with plog directly (see
Audit Events).
HTTP Request Logging¶
To view detailed HTTP request logs (method, path, status, duration), start the UI with the --log-requests flag.
Logs Page¶
On the Logs page, select a log message to open its details. The details show
the level, the time, the type, the message text, and each key=value field in
a list. Use Copy to copy the full message. Use Previous and Next to
go to the adjacent messages in the filtered list.
Audit Events¶
The web API logs an audit event, a record of type AUDIT, for each
security-relevant action.
Audited Actions¶
event.action |
event.category |
event.type |
Routes |
|---|---|---|---|
sign-in |
authentication |
start |
GET, POST /login, by password or through a proxy (phenix.auth.method); POST /signup, once the user is created |
sign-out |
authentication |
end |
POST /logout |
user-create |
iam |
creation |
POST /users, POST /signup (as the user signing up) |
user-update |
iam |
change |
PATCH /users/{username} changing a first or last name (phenix.user.changed) |
user-role-change |
iam |
change |
PATCH /users/{username} with a role_name; refused, and the role left as it is, without patch on users/roles (a role_name equal to the user's current role is not refused) |
user-password-change |
iam |
change |
PATCH /users/{username} with a new_password |
user-delete |
iam |
deletion |
DELETE /users/{username}; refused for the user's own account (event.reason own user) |
user-token-create |
iam |
creation |
POST /users/{username}/tokens |
role-create |
iam |
creation |
POST /roles (phenix.role.name); refused for a role that grants a permission the requester does not hold (event.reason privilege escalation) |
role-update |
iam |
change |
PATCH /roles/{name} (phenix.role.name, and phenix.role.users for the users whose copy of the role changed, or could not change); refused as role-create is |
role-delete |
iam |
deletion |
DELETE /roles/{name} (phenix.role.name) |
request-auth |
authentication |
info |
any API or file server request refused for its token (event.reason), at most once a minute for each client address and reason |
config-create |
configuration |
creation |
POST /configs, POST /workflow/configs/{branch} for a new config (and its refusal by a role that may neither create nor update configs; phenix.workflow.variables when the request used a variable, also in a refusal for the config name: the sorted names, never values), and PUT /configs/{kind}/{name} refusals of the new name it renames a config to |
config-update |
configuration |
change |
PUT /configs/{kind}/{name}, POST /workflow/configs/{branch} for a stored config (phenix.workflow.variables when the request used a variable, also in a refusal for the config name: the sorted names, never values) |
config-delete |
configuration |
deletion |
DELETE /configs/{kind}/{name} |
workflow-apply |
configuration |
change |
POST /workflow/apply/{branch} (phenix.workflow.branch, phenix.workflow.action, and phenix.workflow.variables when the request used a variable: the sorted names of the variables that the workflow config or an included topology file used, never their values) |
settings-update |
configuration |
change |
POST /settings (phenix.settings.changed) |
experiment-create |
configuration |
creation |
POST /experiments |
experiment-update |
configuration |
change |
PATCH /experiments/{name}, and the VLAN pin and range routes PUT and DELETE .../vlans/aliases/{alias}, PUT and DELETE .../vlans/range, POST .../vlans/pin-live |
experiment-schedule |
configuration |
change |
POST /experiments/{name}/schedule |
experiment-delete |
configuration |
deletion |
DELETE /experiments/{name} |
experiment-start |
host |
start |
POST /experiments/{name}/start |
experiment-stop |
host |
end |
POST /experiments/{name}/stop |
experiment-restart |
host |
change |
POST /experiments/{name}/restart |
app-trigger |
process |
start |
POST /experiments/{name}/trigger (phenix.app.names, phenix.app.stage) |
app-trigger-cancel |
process |
end |
DELETE /experiments/{name}/trigger |
scorch-start |
process |
start |
POST .../scorch/pipelines/{run}, POST .../scorch/pipelines/{run}/cleanup (phenix.scorch.cleanup_only) |
scorch-cancel |
process |
end |
DELETE .../scorch/pipelines/{run} |
scorch-clear |
process |
change |
POST .../scorch/pipelines/{run}/clear |
scorch-terminal-exit |
process |
end |
POST .../scorch/terminals/{pid}/exit/{id} |
vm-start |
host |
start |
POST .../vms/{name}/start, POST .../vms/actions with start (one event per VM) |
vm-stop |
host |
end |
POST .../vms/{name}/stop, POST .../vms/actions with pause |
vm-shutdown |
host |
end |
POST .../vms/{name}/shutdown, POST .../vms/actions with shutdown |
vm-restart |
host |
change |
POST .../vms/{name}/restart, POST .../vms/actions with restart |
vm-reset-disk |
host |
change |
POST .../vms/{name}/reset, POST .../vms/actions with resetDisk |
vm-redeploy |
host |
change |
POST .../vms/{name}/redeploy |
vm-kill |
host |
end |
DELETE .../vms/{name}, POST .../vms/actions with kill |
vm-update |
configuration |
change |
PATCH .../vms/{name}, PATCH /experiments/{exp}/vms and POST /experiments/{exp}/vms/resetSettings (one event per VM), PUT and DELETE .../vms/{name}/interfaces/{iface}/qos (qos), naming the settings changed in phenix.vm.changed; refused (event.reason port forwards not allowed) for annotations that start port forwards the role may not create |
vm-cdrom-change, vm-cdrom-eject |
host |
change |
POST .../vms/{name}/cdrom (the disc in file.path), DELETE .../vms/{name}/cdrom |
capture-start |
network |
start |
POST .../vms/{name}/captures, POST .../captureSubnet (one event per VM), POST .../captureVlan (one event per capture, phenix.capture.vlan), POST /experiments/{exp}/captures (phenix.capture.id) |
capture-stop |
network |
end |
DELETE .../vms/{name}/captures, POST .../stopCaptureSubnet (one event per VM), POST .../stopCaptureVlan (one event per capture, phenix.capture.vlan), DELETE /experiments/{exp}/captures/{id} (phenix.capture.id) |
capture-stream |
network |
access |
GET .../captures/{iface}/stream, GET /experiments/{exp}/captures/{id}/stream (phenix.capture.id), GET /webshark/streams/{token} |
capture-webshark-open |
network |
access |
POST /experiments/{exp}/webshark (phenix.capture.id for an experiment capture) |
capture-webshark-read |
network |
access |
GET /webshark/captures/{name} (once per grant), GET /webshark/json?method=download (once per download) |
capture-token-create |
iam |
creation |
POST /experiments/{exp}/captureTokens |
capture-token-revoke |
iam |
deletion |
DELETE /experiments/{exp}/captureTokens/{id} |
netflow-start, netflow-stop |
network |
start, end |
POST /experiments/{exp}/netflow, DELETE /experiments/{exp}/netflow |
netflow-download |
file |
access |
GET /experiments/{exp}/netflow/captures/{id} (once per download, not per resumed range or HEAD), naming the capture and format in phenix.netflow.capture and phenix.netflow.format, and for CSV and JSONL the flows written and lines skipped in phenix.netflow.flows and phenix.netflow.skipped |
file-download |
file |
access |
GET /experiments/{name}/files/{filename} (downloads and the Files tab's viewer), POST .../files/download (one event per file sent, including those sent before the download failed), GET .../vms/{name}/files/download, GET /jobs/{id}/artifact of an archive job (one event per file archived); refusals of GET .../files/{filename}/checksum and POST .../files/archive, and archive jobs that fail or are canceled |
file-upload |
file |
creation |
the file server's upload page (one event per file) |
file-delete |
file |
deletion |
DELETE /experiments/{name}/files/{filename}, POST .../files/delete (one event per file) |
console-create |
process |
start |
POST /console |
console-attach |
process |
access |
GET /console/{pid}/ws; refusals of GET /console/{pid} and POST /console/{pid}/size, and of another user's console on any console route |
console-terminate |
process |
end |
DELETE /console/{pid} |
console-end |
process |
end |
none: a console ended, for any reason |
vnc-attach |
session |
start |
GET .../vms/{name}/vnc/ws, at most once in 5 minutes for each user and VM, and its failures at most once a minute for each user, VM and error (phenix.suppressed counts the events not logged); refusals of GET .../vms/{name}/vnc. Each refusal is logged. |
port-forward-attach |
network |
connection |
GET .../vms/{name}/forwards/{host}/{port}/ws |
disk-snapshot, disk-clone |
file |
creation |
POST /disks/snapshot, POST /disks/clone, POST /disks/clone/jobs |
disk-rename, disk-commit, disk-rebase, disk-resize |
file |
change |
POST /disks/rename, /disks/commit, /disks/rebase, /disks/resize |
disk-delete |
file |
deletion |
DELETE /disks; refused (event.reason in use) for an image that other images or experiments use, and phenix.disk.force for a forced delete |
disk-upload |
file |
creation |
POST /disks |
disk-download |
file |
access |
GET /disks/download; refusals of GET /disks/checksum |
vm-snapshot, vm-commit, vm-memory-snapshot |
file |
creation |
POST .../vms/{name}/snapshots, /commit, /memorySnapshot |
vm-screenshot |
file |
creation |
POST .../vms/{name}/screenshots, naming the saved PNG file in phenix.file.target_path |
vm-restore |
file |
change |
POST .../vms/{name}/snapshots/{snapshot} |
vm-mount, vm-unmount |
session |
start, end |
POST .../vms/{name}/mount, DELETE .../vms/{name}/unmount |
vm-file-upload, vm-file-copy |
file |
creation |
PUT .../vms/{name}/files/upload, POST .../vms/{name}/files/copy |
log-download |
file |
access |
GET /logs/files/{name}, naming the log file in phenix.log.name |
vlan-blacklist-add, vlan-blacklist-remove |
configuration |
creation, deletion |
PUT /vlans/blacklist/{id}, DELETE /vlans/blacklist/{id}, naming the VLAN ID in phenix.vlan.id |
vlan-release |
configuration |
deletion |
DELETE /vlans/allocations/{namespace}, naming the minimega namespace in phenix.vlan.namespace |
minimega-command |
process |
start |
POST /minimega/commands, naming the command line in phenix.minimega.command |
runtime-setting-change |
configuration |
change |
PUT /runtimeSettings/{key}, DELETE /runtimeSettings/{key}, naming the setting in phenix.setting.key |
Troubleshooting
For common logging issues and solutions, please see the Troubleshooting page.