----------------------------------------
Smart Pinot Table Config Recommendation Engine
----------------------------------------
Pinot has several table and system level tuning options. Currently there is no automation around recommending optimal values of these options when a user wants to create a new table on Pinot or wants to optimize the config of an existing table. Either the customers themselves make a call on picking the right config (indexes etc) or the Pinot developer involved with the customer does a manual parse of information (table schema, query pattern etc) provided by the customer and tries to make decisions on the configuration either by consulting other developers and/or utilizing past knowledge or docs that have some information on how a particular config can help. This mechanism doesn't prove to be the de facto means of smartly recommending optimal config with minimal manual effort. Since this is manual, the interpretation of optimal config rules by people can vary from time to time and it gets hard to make this process standardized without having some form of automation. Each person (customer or Pinot developer) tries to recommend config based on their individual assessment of the information related to the table.
To automate this process, we have built a rule engine (aka recommendation engine) to recommend quite a few parts of the table config. The rule engine is fully customizable. It has a default set of rules to recommend parts of table config (inverted index, sorted index etc). Rules can be easily modified/tuned and new rules can be added to change the recommendation engine's algorithms. The talk will go through the details of this engine, deep dive into some design/algorithms, usability etc
----------------------------------------
Presented by:
----------------------------------------
Jia Guo
Software Engineer at LinkedIn
Jia works at LinkedIn as a member of Pinot team. Prior to joining as a full-time, Jia had interned twice on the team, during which he prototyped, designed, and implemented the initial version of config recommendation engine. Jia received his PhD degree this June from the Ohio State University on the study of parallel and distributed computing. He’s interested in performance modeling and building software with high parallelism and locality.
----------------------------------------
Resources
----------------------------------------
StarTree: https://www.startree.ai/
Pinot Meetup https://www.meetup.com/apache-pinot/
Pinot website http://pinot.apache.org/
Pinot docs https://docs.pinot.apache.org/
Pinot Slack https://communityinviter.com/apps/apache-pinot/apache-pinot
Pinot Github https://github.com/apache/incubator-pinot
Pinot twitter: https://twitter.com/ApachePinot
0 Comments