Troubleshooting¶
This page consolidates common issues and their solutions when working with phēnix.
Logging Issues¶
Logs Not Appearing in UI¶
If your application logs are not showing up in the phēnix web UI, check the following:
- Check the stream: Ensure your app writes logs to
stderr.stdoutis reserved for returning modified experiment configuration JSON to phēnix. - Check the format: The phēnix core expects valid, single-line JSON objects. If your app prints raw text (like
Starting app...), it may be logged as a warning or error by the core but will not be indexed correctly for the UI. - Check the environment: Verify
PHENIX_LOG_FILE=stderris set in your app's environment (phēnix does this automatically for apps it launches). - Check for skipped lines: When the server logs faster than it can send to the browser, the Logs page's live view skips lines and shows a
WARNentry such as120 log lines skipped while the server was busy. The lines are still in the log file: reload the Logs page, or callGET /logs, to see them. The server also logs aWebSocket broker dropped slow clients or live log lineswarning at most every 30 seconds.
JSON Parsing Errors¶
If you see "malformed log entry" warnings in the main phēnix system log, it's likely because your application is writing non-JSON text to stderr.
- Debug Prints: Ensure you are not using
print()(Python) orfmt.Println()(Go) for debugging. These pollute thestderrstream. Use the structured logger. - Tracebacks: Ensure exception tracebacks are not being printed as raw text.
- Python: Use
logger.exception("message")inside anexceptblock. This correctly captures the traceback as a JSON field. Standardprint(e)ortraceback.print_exc()will break the parser. - Go: Use structured logging fields for errors (e.g.,
slog.Error("message", "err", err)).
- Python: Use
Inconsistent Timestamps¶
Ensure timestamps in your JSON logs are in a consistent format that phēnix can parse, such as RFC3339. The core now enforces a 2006-01-02 15:04:05.000 format for its own file logs for easier reading, but is flexible on ingest.
Packet Capture Issues¶
Experiment Captures Are Not Available¶
Experiment captures need minimega 3.4.0 or later on every node. phēnix checks the minimega version of the head node. For the disabled controls, a refused start, and a skew warning, see Experiment Packet Captures.
Configuration Issues¶
Do Not Delete config.yaml¶
If you delete the config.yaml file while phēnix is running, the internal file watcher will break, and dynamic configuration changes will no longer apply.
To reset your configuration, use the command phenix settings unset --all instead of deleting the file.