hops.experiment_impl.distribute package

Submodules

hops.experiment_impl.distribute.mirrored module

Utility functions to retrieve information about available services and setting up security for the Hops platform.

These utils facilitates development by hiding complexity for programs interacting with Hops services.

hops.experiment_impl.distribute.mirrored_reservation module

class hops.experiment_impl.distribute.mirrored_reservation.Reservations(required)

Bases: object

Thread-safe store for node reservations.

__init__(required)

Initialize self. See help(type(self)) for accurate signature.

add(meta)

Add a reservation.

Args:

meta

a dictonary of metadata about a node

done()

Returns True if the required number of reservations have been fulfilled.

get()

Get the list of current reservations.

remaining()

Get a count of remaining/unfulfilled reservations.

class hops.experiment_impl.distribute.mirrored_reservation.MessageSocket

Bases: object

Abstract class w/ length-prefixed socket send/receive functions.

receive(sock)

Receive a message on sock Args:

sock:

Returns:

send(sock, msg)

Send msg to destination sock Args:

sock: msg:

Returns:

class hops.experiment_impl.distribute.mirrored_reservation.Server(count)

Bases: hops.experiment_impl.distribute.mirrored_reservation.MessageSocket

Simple socket server with length prefixed pickle messages

done = False

__init__(count)

Initialize self. See help(type(self)) for accurate signature.

reservations = None

await_reservations(sc, status={}, timeout=600)

Block until all reservations are received.

Args:

sc: status: timeout:

Returns:

start()

Start listener in a background thread

Returns:

address of the Server as a tuple of (host, port)

stop()

Stop the Server’s socket listener.

class hops.experiment_impl.distribute.mirrored_reservation.Client(server_addr)

Bases: hops.experiment_impl.distribute.mirrored_reservation.MessageSocket

Client to register and await node reservations.

__init__(server_addr)

Args:

server_addr

a tuple of (host, port) pointing to the Server.

sock = None

socket to server TCP connection

server_addr = None

address of server

close()

Close the client socket.

register(reservation)

Register reservation with server.

Args:

reservation:

Returns:

get_reservations()

Get current list of reservations.

await_reservations()

Poll until all reservations completed, then return cluster_info.

request_stop()

Request server stop.

hops.experiment_impl.distribute.mirrored_reservation_client module

hops.experiment_impl.distribute.parameter_server module

Utility functions to retrieve information about available services and setting up security for the Hops platform.

These utils facilitates development by hiding complexity for programs interacting with Hops services.

hops.experiment_impl.distribute.parameter_server_client module

hops.experiment_impl.distribute.parameter_server_reservation module

class hops.experiment_impl.distribute.parameter_server_reservation.Reservations(required)

Bases: object

Thread-safe store for node reservations.

__init__(required)

Args:

required:

add(meta)

Add a reservation.

Args:

meta

a dictonary of metadata about a node

done()

Returns True if the required number of reservations have been fulfilled.

get()

Get the list of current reservations.

remaining()

Get a count of remaining/unfulfilled reservations.

class hops.experiment_impl.distribute.parameter_server_reservation.WorkerFinished(required)

Bases: object

Thread-safe store for node reservations.

__init__(required)

Args:

required

expected number of nodes in the cluster.

add()

Add a reservation.

Args:

meta

a dictonary of metadata about a node

done()

Returns True if the required number of reservations have been fulfilled.

remaining()

Get a count of remaining/unfulfilled reservations.

class hops.experiment_impl.distribute.parameter_server_reservation.MessageSocket

Bases: object

Abstract class w/ length-prefixed socket send/receive functions.

receive(sock)

Receive a message on sock

Args:

sock:

Returns:

send(sock, msg)

Send msg to destination sock.

Args:

sock: msg:

Returns:

class hops.experiment_impl.distribute.parameter_server_reservation.Server(count)

Bases: hops.experiment_impl.distribute.parameter_server_reservation.MessageSocket

Simple socket server with length prefixed pickle messages

done = False

__init__(count)

Args:

count:

reservations = None

await_reservations(sc, status={}, timeout=600)

Block until all reservations are received.

Args:

sc: status: timeout:

Returns:

start()

Start listener in a background thread

Returns:

address of the Server as a tuple of (host, port)

stop()

Stop the Server’s socket listener.

class hops.experiment_impl.distribute.parameter_server_reservation.Client(server_addr)

Bases: hops.experiment_impl.distribute.parameter_server_reservation.MessageSocket

Client to register and await node reservations.

Args:

server_addr

a tuple of (host, port) pointing to the Server.

__init__(server_addr)

Initialize self. See help(type(self)) for accurate signature.

sock = None

socket to server TCP connection

server_addr = None

address of server

close()

Close the client socket.

register(reservation)

Register reservation with server.

Args:

reservation:

Returns:

register_worker_finished()

Register worker as finished with server.

Returns:

await_all_workers_finished()

Poll until all reservations completed, then return cluster_info.

Returns:

get_reservations()

Get current list of reservations.

Returns:

await_reservations()

Poll until all reservations completed, then return cluster_info.

request_stop()

Request server stop.

Module contents