https://github.com/KevenLi8888/wire-protocols

Engineering Notebook - Wire Protocol

Re-implement the chat application you built for the first design exercise, but replace using sockets and a custom wire protocol/JSON with gRPC or some similar RPC package. Does the use of this tool make the application easier or more difficult? What does it do to the size of the data passed? How does it change the structure of the client? The server? How does this change the testing of the application?

python -m grpc_tools.protoc -Igenerated=./protos --python_out=. --grpc_python_out=. --pyi_out=. ./protos/chat.proto

Design Decision: Two ways of adding gRPC

When considering how to add gRPC support for the application, we thought of two approaches.

1. Minimal effort approach: Using gRPC as a transport layer with a generic send/receive method

   In our current design, we use a “request” and “response” fashion for each type of messages. We can use gRPC for transmitting data between client and server, replacing JSON or our custom protocol, with only one RPC method: Send. We then parse the response according to the message type, and perform corresponding operations. This allows us to reuse most of our existing code (message processing, sending, receiving, etc.), but it defeats the purpose of the assignment.

   service ChatService {
     rpc SendMessage (MessageRequest) returns (MessageResponse);
   }
   
   message MessageRequest {
     int32 message_type = 1;
     map<string, string> data = 2;
   }
   
   message MessageResponse {
     int32 message_type = 1;
     map<string, string> data = 2;
   }
   

2. Proper RPC approach: Creating specific RPC methods for each operation type

   This is the way we used to introduce gRPC to our project, which is a more conventional approach. There’s a RPC method corresponding to each type of operations/messages.

   service ChatService {
     rpc CreateAccount(CreateAccountRequest) returns (CreateAccountResponse);
     rpc Login(LoginRequest) returns (LoginResponse);
     rpc DeleteAccount(DeleteAccountRequest) returns (BasicResponse);
     rpc GetUsers(GetUsersRequest) returns (GetUsersResponse);
     rpc SendMessage(SendMessageRequest) returns (SendMessageResponse);
     rpc SearchUsers(SearchUsersRequest) returns (SearchUsersResponse);
     rpc GetRecentChats(GetRecentChatsRequest) returns (GetRecentChatsResponse);
     rpc GetPreviousMessages(GetPreviousMessagesRequest) returns (GetPreviousMessagesResponse);
     rpc GetChatUnreadCount(GetChatUnreadCountRequest) returns (GetChatUnreadCountResponse);
     rpc GetUnreadMessages(GetUnreadMessagesRequest) returns (GetUnreadMessagesResponse);
     rpc DeleteMessages(DeleteMessagesRequest) returns (BasicResponse);
   }
   

Implementation and Changes

Protocol Definition

The protocol definition in chat.proto defines the service interface with specific methods for each operation, as well as the message structures.

Client-Side Changes

1. Structure Changes:
   • Added GRPCClient class that wraps the generated stubs
   • Modified the Client class to use either gRPC or socket-based communication based on configuration
   • Added a communication abstraction layer to handle protocol differences
2. Connection Management:
   • For gRPC, connections are managed through channels instead of sockets
   • The client establishes an insecure channel with the server: self.channel = grpc.insecure_channel(f'{host}:{port}')

On client’s side, we modified our send() method so that when client is trying to sending out some type of messages, the corresponding RPC method stub will be called, instead of serializing and sending the message out using the socket.