A high-performance URL lookup engine for URL filtering systems

URL filtering systems provide a simple and effective way to prevent people from browsing undesirable or malicious websites. These systems require a well-designed URL lookup method as the core operation. A high-performance URL lookup engine is proposed in this paper for URL filtering systems. It combines a URL compression algorithm with a multiple string matching based (Wu-Manber-like) matching algorithm. Using this method, the proposed URL lookup engine can achieve high URL lookup performance and efficient memory utilization for storing the ever-increasing URL blacklist with the ability of prefix matching. Experiments with actual URL blacklists and requested URL sets show that our engine can save about 80% memory usage for storing URL blacklists, and reduce 58%-162% URL lookup time compared with the state-of-the-art URL lookup methods.