Troubleshooting#

#Startup Problems

#No available configuration file

After fresh installation the Proxy requires a valid configuration file. If it's missing - following log entries will be in the logs (use docker logs farm-proxy -f to check them) and Proxy will be waiting to receive a configuration file:

INFO farm_proxy: Welcome to Farm Proxy 24.06, rev=cb0be07558, arch=x86_64
INFO farm_proxy: Loading configuration file: /config/config_cache/last_used_config.toml
WARN farm_proxy: Default configuration file not found. Awaiting configuration via HTTP /configure... path=/config/config_cache/last_used_config.toml
INFO farm_proxy::service: Farm Proxy main application started pid=1
INFO farm_proxy::service: No initial usable configuration provided. Waiting for HTTP /configure
INFO farm_proxy::http_api: HTTP server bound and started socket=0.0.0.0:8080

Check the Configuration section, how to configure the proxy.

#Structure error of configuration file

When there's an error in the syntax of the configuration TOML file, output of docker compose up farm-proxy-configurator will throw an error:

Attaching to farm-proxy, farm-proxy-configurator
farm-proxy-configurator  | Error: Failed to load configuration file: Invalid configuration: '/config/profiles/active_profile.toml'
farm-proxy-configurator exited with code 1

#Invalid settings in the configuration file

When the TOML structure is fine, but there's wrong configuration (e.g. wrong setting or parameter name), output of docker compose up farm-proxy-configurator will throw an error similar to this one:

Attaching to farm-proxy, farm-proxy-configurator
farm-proxy               | 2024-06-26T13:33:58.631719Z  WARN farm_proxy::http_api::configure: Configuration rejected on HTTP /configure. Error: Configuration is invalid: Unable to load routings from config file: Routing #0: invalid server reference 'S1' in routing's `from` field
farm-proxy-configurator  | Rejected. Invalid config.
farm-proxy-configurator exited with code 0

Also Farm Proxy log will throw an error:

2024-06-26T13:33:58.631719Z  WARN farm_proxy::http_api::configure: Configuration rejected on HTTP /configure. Error: Configuration is invalid: Unable to load routings from config file: Routing #0: invalid server reference 'S1' in routing's 'from' field

#Problems with downstream connectivity

If there is a problem connecting to Farm Proxy, the BOSminer can take a while to reconnect. The longer the problem persists, the longer it takes for the retries. It has graceful slowdown: retry immediately, if fail, retry after 1 second, if fail after 2 second, if fail after 4 seconds... It can go all the way to 1 hour between retries. In such cases the solution is either restart BOSminer or wait.

#Hashrate drops

If you see a drop in the hashrate, it should go hand in hand with drop in number of connections. There could be several reasons for that:

1. The primary target went down and there is no backup target. Proxy disconnects all the miners as it has no target to route the hashrate to. Make sure to configure backup target or preferably 2 levels of backups when the second level is completely different pool (not just another port).
2. Proxy lost connection with outside world. It can be a momentarily loss, but again the connection to the miners is closed. Some miners' firmwares do not reconnect immediately when the connection is back, but after some while. If this is a repeating issue, set up a probe to regularly check on your connections to outside world. Simple ping is fine, but a stratum ping utility is even better. There should be visible outages in your network infrastructure which need to be addressed.
3. Miners lost connection to the Proxy. Ideally Farm Proxy and your miners should be on the same local network. More networking between your miners and the Proxy means higher chance of disconnects. This is visible in the Debug Dashboard - Downstream connections close side where external means the connection was closed by the miner (whereas internal means it was closed by the Farm Proxy).
4. Primary target is up, but it keeps rejecting the hashrate. This should not happen on the major pools. But should you mine against some tiny or private pools this could happen. Make sure to define backup pool.
5. Farm Proxy can get stuck. It can get overloaded (e.g. combination of too many miners and weak nodes specs), there could be a programming error. Farm Proxy will become unresponsive or the consumed resources (cpu, memory) will increase a lot.

Whatever reason might cause the hashrate drops, it's always necessary to configure backup targets also on the individual miners (if your network allows that) to prevent loss of the hashrate.

#Proxy appears dead for the miner

• check the IP address is reachable from the miner
• check the Farm Proxy is not in restart loop via docker ps, status should be something like Up 5 minutes (healthy) and not Restarting (3 seconds ago)
• check there is a valid configuration (see above)
• check the target did not reject the extranonce size requirements in the Farm Proxy log

2024-06-26T14:08:13.809370Z ERROR ii_stratum_router::controllers::v1_upstream: Insufficient extranonce obtained from server. Consider lowering extranonce size requirement. obtained=6 configured_minimum=7 target_name=Braiins Pool

• check the username is valid at the target pool in the Farm Proxy log

2024-06-26T14:09:29.571940Z  WARN ii_stratum_router::router::target_quality: Upstream authorization failed endpoint="stratum+tcp://159.223.18.138:3333" identity="nonexistinguser.worker"

• check the port configured in the server section of the configuration file and ports exposed in the .env file match

#Raspberry Pi

Prometheus and Grafana are quite demanding when it comes to Raspberry Pi resources. For the sake of the proxy running smoothly, it is advised to remove Prometheus and Grafana from the docker-compose.yml and run it on a different device.

#Rejected hashrate

In case of detecting rejected hashrate from downstream or upstream, please contact Braiins support. To solve the issues, information from Grafana "Debug Dashboard" and logs will provide much help to our support team.

#When requesting support

We will need following:

1. Description of the problem
2. Configuration .toml file (active_profile.toml)
3. Screenshots of both dashboards from the given timeframe. Open all sections of the dashboards and use Chrome plugin GoFullPage for screenshot.
4. Farm Proxy log from the given timeframe.
5. Snapshots of the Prometheus data curl -XPOST 127.0.0.1:9090/api/v1/admin/tsdb/snapshot. Snapshot will be located in /var/lib/docker/volumes/farm-proxy_prometheus_data/_data/snapshots.