Skip to content

phenix doctor

Every command on this page also accepts the global flags.

phenix doctor

Check the phenix configuration without a running server

Synopsis

Check the phenix configuration without a running server

  Prints a header for bug reports (the phenix and minimega versions, OS and
  kernel, user, working directory, the config file read, and whether phenix
  runs in a container), then every setting with its effective value and where
  that value came from (flag, env, config file or default), then each check,
  then a summary that names each failed and warned check, with a hint for
  each failure.

  The checks read the config and users files and report unknown keys, list
  every config and users file on the search path (the working directory,
  $HOME/.config/phenix unless root, then /etc/phenix), name the file phenix
  settings set writes, check each set environment variable phenix reads
  (PHENIX_*, PATH, EDITOR, SUDO_USER and SUDO_COMMAND) as phenix parses it,
  warn about PHENIX_* variables that name no setting or override a different
  config file value, check the settings every other phenix command refuses or
  misreads (bridge-mode, deploy-mode, experiment.disk-check, log.level and
  log.system.*), check that the store and log files can be written, that the
  store file is a whole BoltDB file this user owns and that no root phenix ui
  uses another store, and that the base directories exist, check what phenix
  ui reads at startup (the JWT signing key and lifetime, initial users, proxy
  auth, the TLS key pair and certificate, the listen endpoints, and
  the unix socket's directory and group), ask a running phenix ui on the unix
  socket for its version and its bridge and deploy modes, and check that
  minimega answers on its socket and that its files directory, base directory
  and mesh suit phenix. The disk-chains check reads the headers of the disk
  images in that files directory, for at most --timeout, and warns about each
  image whose backing chain is broken. The disk-formats check reads the same
  headers, and warns about each image in an old or unsupported format, such as
  qcow2 version 2, or one that qemu refuses. The tools and features checks look for
  qemu-img, vmdb2 and filebeat on PATH, measure free disk space, report
  whether phenix ui turns on tunneler downloads and WebShark (relative to the
  working directory), report whether minimega supports experiment captures
  (3.4.0 or later), and check ui.features and
  ui.logs.minimega-path. --skip takes a check or a group of
  checks; an unknown name fails and lists the valid ones. Secret values, such
  as the JWT signing key and initial users, are redacted. doctor exits 1 when
  a check fails, or with --strict when one warns, and 0 otherwise.

  --report prints the whole output, text or JSON, in a fenced block to paste
  into a bug report. It replaces host names, IP addresses, user names and email
  addresses with tokens, such as <host-1>. It replaces passwords, secrets and
  the credentials in URLs with <redacted>. The same value always gets the same
  token, and a path keeps its shape, such as /home/<user-1>/images. It keeps
  localhost, 127.0.0.1, ::1, 0.0.0.0 and ::, the users root, phenix, minimega,
  nobody and admin@foo.com, and a hostname suffix, such as -phenix in
  <host-1>-phenix.

  --problems-only (-p) shows only the checks that warn or fail. The summary
  still counts every check and gives the number of hidden checks, and the exit
  status does not change. The text output colors the statuses, the summary
  and the hints when --color allows it, but never with -o json or --report.

  The kernel-modules, nbd, openvswitch and image-build checks look at this
  host, the head node, and only on Linux. kernel-modules warns when
  /lib/modules has no directory for the running kernel, or when a reboot is
  pending. nbd checks what minimega disk inject needs for injects: qemu-nbd,
  the nbd module with max_part, and a free /dev/nbdN. It also names the
  devices that a failed inject can leave. openvswitch runs ovs-vsctl show,
  and reports the Open vSwitch version and the experiment bridge. image-build
  checks the programs phenix image build needs, root and a free loop device.
  It also checks the debootstrap script and keyring of the release of each
  stored image config.

  The server group also looks at the state of phenix ui. stuck-status names
  the app runs, SCORCH runs and delayed starts that the store marks as in
  progress for a running experiment, when no phenix process can own them. It
  also names the interrupted starts, and the starts in progress that no
  phenix process runs. ui-limits (Linux only) warns when phenix ui has more
  than 80 percent of its soft limit of open files open. In the
  tls group, ui.tls-cert warns when the TLS certificate expires within 30
  days, or when its names do not cover the host that users reach. It fails
  when the certificate expired or is not valid yet. In the env group,
  proxy-env warns when a proxy variable is set and NO_PROXY does not cover
  the hosts of phenix ui. It names the variables, never their values.

  The images group also looks for problems that stopped or deleted
  experiments leave. writable-images warns when a VM with snapshot: false
  writes to an image that backs other images, or that more than one VM uses.
  leftovers names the snapshot files and directories of deleted experiments,
  VM file system mounts with no running VM, minimega taps and Open vSwitch
  ports that no VM owns, and mirrors that no capture owns. disk-space
  measures the filesystems of the images directory, minimega's -base and its
  log, and names the dstImg directories that a failed inject can leave.
  image-locks (Linux only) names the qemu processes that hold an image of a
  stopped experiment open, so qemu cannot lock it at the start.
  inject-partitions reads the partition table of each image that gets
  injects, and warns when the inject_partition of a VM names no partition,
  or one that minimega cannot write files to. It suggests a partition. It
  also warns when an NTFS inject partition is marked dirty or has a log that
  is not clean (Windows did not shut down fully), or when an ext file system
  is marked with errors.

  The minimega group also compares minimega with this host and the store.
  experiment-files warns when <base-dir.phenix>/images is not minimega's
  -filepath, because the Files tab then does not show the files that phenix
  writes for an experiment. minimega-limits (Linux only) reads the open files
  limit of the minimega process from /proc, and warns when it is too low for
  its VMs on this node. It also warns when this host is short of memory or
  swap, or when the VMs ask for more memory than the host has. state-drift
  names the experiments that the store marks as running with no VMs in
  minimega, and the namespaces with VMs that no stored experiment owns. vlans
  warns when two experiments pin the same VLAN ID, or when a stopped
  experiment cannot get its VLANs now. bridge-trunk warns when a mesh of more
  than one node has an experiment bridge with no trunk or tunnel port.
  schedule-hosts names the VM schedules on nodes that are not in the mesh.
  cc-clients names the running VMs that phenix needs miniccc in but that have
  no client, and the clients that minimega logged with another miniccc
  version after their VMs launched.

  The apps group lists the user apps and SCORCH components on PATH.
  scenario-apps names each app and SCORCH component type of a stored scenario
  or experiment that is not built in and not on PATH. app-runtimes names each
  of those programs that this user cannot run, such as a program whose #!
  line names an interpreter that does not exist.

  Other checks compare the stored configs with this host. scenario-hosts
  names the hosts of scenario apps that are not nodes of the topology.
  inject-sources names the inject sources that do not exist. image-configs
  names the overlay directories of stored image configs that do not exist.
  rbac-users, only with sign-in on, names the users with an unknown role, and
  the names of deleted experiments and users in roles. store-fs warns when the
  store is on a network or FUSE filesystem, or when a large store has many
  free pages. secret-files warns when the group or others can read a file
  that holds a secret, such as the TLS key, and never reads the contents.

  The store-file and store-fs checks read the pages of the store file that
  they need without the lock. The other checks that read the store open it
  through BoltDB, once for each kind of config in a run. doctor opens it
  read-only, and waits at most --timeout for the lock. After one read waits
  for the lock in vain, the other reads do not wait. doctor opens no log
  file, writes no file and binds no port, so it works while phenix ui is
  stopped. Before any command runs, phenix creates the config directory with
  an empty config.yaml when it finds no config file, and the default log
  directory.
  doctor connects to the unix socket as a client only, never creating or
  removing it. It sends minimega only commands that change nothing (version,
  args, mesh status, mesh list all and namespace, then vm info, tap, capture,
  vlans, vlans range and cc clients in the namespaces that minimega lists, and
  vlans blacklist). It also reads /proc and the end of minimega's log file.
  The only programs it runs are qemu-img --version, ovs-vsctl show and
  ovs-vsctl list, which change nothing. Run it with the same flags,
  environment and user as the phenix command you are checking.
phenix doctor [flags]

Examples

# In the Docker deployment
docker exec phenix phenix doctor

# As JSON, waiting up to 10 seconds for minimega
phenix doctor -o json --timeout 10s

# Without minimega or the settings list, failing on any warning
phenix doctor --skip minimega -x --strict

# Only the problems
phenix doctor -p -x

# Output to paste into a bug report, with host data replaced
docker exec phenix phenix doctor --report

Options

  -h, --help                help for doctor
  -x, --no-print-settings   leave out the settings list
  -o, --output string       Output format (text, json) (default "text")
  -p, --problems-only       show only the checks that warn or fail (the summary still counts every check)
      --report              print the output in a fenced block to paste into a bug report, with host names, addresses, user names and secrets replaced
      --skip strings        checks or groups of checks to skip, comma-separated or repeated (such as minimega or ui.jwt-signing-key)
      --strict              exit 1 when a check warns, not only when one fails
      --timeout duration    how long to wait for each answer of minimega, phenix ui or ovs-vsctl, for the store lock, and for each check that reads many files, such as disk-chains, leftovers and image-locks (default 3s)

SEE ALSO

  • phenix - A cli application for phÄ“nix