Update on Ray CVE-2023-48022: New Verification Tooling Available

Last fall, we alerted the Ray community in a blog to the existence of potentially misconfigured Ray open source clusters. More recently we were contacted by security researchers from Oligo in reference to the same CVE report, stating that they had evidence of malicious exploitation of some misconfigured Ray open source clusters. Our earlier blog post explains the behavior as well as how to protect your Ray clusters by following our documentation. As noted there, this potential issue does not affect Anyscale-hosted Ray clusters, which have authentication enabled.

In light of reports of malicious activity, we have moved quickly to provide tooling to allow users to verify proper configuration of their clusters to avoid accidental exposure.

We’re providing a client-side script and server-side code. All of the source code and further details for this tooling can be found here: https://github.com/ray-project/ray-open-ports-checker. 

The repo is made available under Apache2. 

We’ve also pre-configured the defaults of the client side script to reach out to a server we have set up to simplify the process of determining whether or not ports are unexpectedly open.  

Most granular details are provided in the above repo’s README but at a high level:

• The tool consists of a simple client plus server.

• The client can be deployed to any Ray Head or Worker Node.

• It will return all existing ports used by Ray using existing Ray APIs.

  • This works across all versions of Ray.

• It can also be prompted to send a test network call through those ports to a lightweight web server.

  • Anyscale will host for the community to explicitly test these network paths*.  

  • You may also optionally configure the script to send to your own server instead.

• The script will then return one of the following:

If it returns “[OK] No open ports detected checked_ports=<list of ports> node=<Ray Node ID>”: 

• It means that the server could not establish a connection to any ports identified as open on the machine reached at the IP address of the machine that sent the information to the server.

• Note that because we can’t necessarily know how you have your network configured, a “no successful network call” returned response does not guarantee that no ports are open (e.g., you may have used firewall or NAT rules to route different external ports to the identified port on the machine, which would result in a false negative).

If it returns “[WARNING] open ports detected open_ports=<list of ports> node=<Ray Node ID>”:

• It means that the server identified that something is open on the port identified by the script at the IP address of the machine that sent the call to the server.

• It does not necessarily mean that your port is open to unauthenticated traffic, as the script does not attempt to validate what is running on the identified open port, and so will not be able to identify whether or not it failed to authenticate or even whether Ray is what is listening on that port.

The easiest way to do this is to simply copy the client side code into your Ray Clusters’s Head Node and run it there; if you want to be extra thorough you can also run it explicitly on every Worker Node. 

We’ve also provided the server side code in case you want to self-host to test your network traffic through whatever network topology you want.

We are also including these capabilities in Ray 2.11, expected in April.

*Anyscale may collect information in accordance with our privacy policy sent to the server (e.g., IP address, open ports) to help improve Ray and determine the extent to which these misconfigurations continue to be an issue.