Abstract: Pretrain-Finetune paradigm recently becomes prevalent in many NLP tasks: question answering, text classification, sequence labeling and so on. As the state-of-the-art model, BERT pre-trained on the general corpus (e.g., Wikipedia) have been widely used in these tasks. However, these BERT-style models still show limitations on some scenarios, especially for two: a corpus that contains very different text from the general corpus Wikipedia, or a task that learns embedding spacial distribution for specific purpose (e.g., approximate nearest neighbor search). In this paper, to tackle the above dilemmas we also encounter in an industrial e-commerce search system, we propose novel customized pre-training tasks for two critical modules: user intent detection and semantic embedding retrieval. The customized pre-trained models with specific fine-tuning, being less than 10% of BERT-base's size in order to be feasible for cost-efficient CPU serving, significantly improves its other counterparts on both offline evaluation metrics and online benefits. We have open sourced our datasets for the sake of reproducibility and future works.