@flyweight2017 Nice questions to add up. Please see my inline comments for the same -
- Uniqueness and scalability of URL generation - how do we guarantee the uniqueness at massive scale, say, to accommodate thousands (or more) of simultaneous pasting requests per second? A single server to generate URL most likely would not be sufficient. But if we distribute the generation across multiple servers, what schemes should we use to ensure the URL uniqueness? -
Please see below basic high level view of architecture. In order to generate unique urls we can utilize the concept of Key Generation Service for creating unique key for each url and storing it in Key DB. Key Generation Service will make sure all the keys inserted in key DB are unique. In order to implement such scenario Key Generation Service can use two or three tables to store keys corresponding to url, one for keys that are not used yet and one for all the used keys. As soon as Key Generation Service provide keys to application server, it can move these to the used keys table. In order to make this process fast Key Generation Service can keep some keys in the memory so that whenever a application server needs them, it can quickly provide them. Thus, whenever Key Generation Service loads some keys in memory, it can move them to used keys table, this way we can make sure each server gets unique keys.
- Storage of the content - Interviewers are (or should be) more interested in how candidates would design scalable solutions for the content storage rather than just using an existing relational/nosql/object database. -
Please see below basic high level view of architecture. We can use block storage to store the paste. This storage is needed when we want to store volume of data.
- Content retrieval - In a distributed design, how do we distribute the content retrieval based on a URL? -
Upon receiving a read request for the paste, the application service layer contacts the datastore. The datastore searches for the key, and if it is found, returns the paste’s contents by first looking into paste cache and than moving to Paste storage. Otherwise, an error code is returned.
- For any solutions with distributed nature, how do we scale up if more users join? How do we handle any of the compute/storage server failures in those solutions? -
We can use database partitioning and consistent hashing techniques to scale up the architecture.