1.ElasticSearch概念
官网介绍:https://www.elastic.co/cn/what-is/elasticsearch/
官网学习文档:https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
1.1.ElasticSearch与MySQL的比较
- MySQL有事务性,而ElasticSearch没有事务性,所以你删了的数据是无法恢复的。
- ElasticSearch没有物理外键这个特性,如果你的数据强一致性要求比较高,还是建议慎用
- ElasticSearch和MySql分工不同, MySQL负责存储数据, ElasticSearch负责搜索数据
1.2.为什么要使用Elasticsearch?
因为在我们商城中的数据,将来会非常多,所以采用以往的模糊查询,模糊查询前置配置,会丢弃索引,导致商品查询是全表扫描,在百万级别的数据库中,效率非常低下,而我们使用ES做一个全文索引,我们将经常查询的商品的某些字段,比如说商品名,描述、价格还有id这些字段我们放入我们索引库里,可以提高查询速度。
1.3.ES中核心概念
1.4.倒排索引机制
倒排索引:将各个文档中的内容,进行分词,形成词条。然后记录词条和数据的唯一标识(id)的对应关系,形成的产物 。
倒排索引是搜索引擎的核心。搜索引擎的主要目标是在查找发生搜索条件的文档时提供快速搜索。倒排索引是一种像数据结构一样的散列图,可将用户从单词导向文档或网页。它是搜索引擎的核心。其主要目标是快速搜索从数百万文件中查找数据。
1.5.ElasticSearch数据的存储和搜索原理
2.Docker安装
Support Matrix:https://www.elastic.co/cn/support/matrix#matrix_compatibility
- Elasticsearch 7.10.1 存储和检索数据
- Kibana 7.10.1 可视化检索数据
2.1.下载镜像文件
docker pull elasticsearch:7.10.1 #存储和检索数据
docker pull kibana:7.10.1 #可视化检索数据
2.2.创建实例
2.2.1.ElasticSearch
# 查看虚拟机内存,建议调整到 4096m
free -mmkdir -p /mydata/elasticsearch/config
mkdir -p /mydata/elasticsearch/data
mkdir -p /mydata/elasticsearch/plugins# 修改文件夹权限
chmod -R 777 /mydata/elasticsearch/echo "http.host: 0.0.0.0">>/mydata/elasticsearch/config/elasticsearch.yml# 9200 http请求端口,9300 集群节点之间通信端口
docker run --name elasticsearch -p 9200:9200 -p 9300:9300 \
--restart=always \
-e "discovery.type=single-node" \
-e ES_JAVA_OPTS="-Xms1024m -Xmx1024m" \
-v
/mydata/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/el
asticsearch.yml \
-v /mydata/elasticsearch/data:/usr/share/elasticsearch/data \
-v /mydata/elasticsearch/plugins:/usr/share/elasticsearch/plugins \
-d elasticsearch:7.10.1# 查看容器启动日志
docker logs elasticsearch # 容器名或容器id都可以
访问:http://192.168.139.10:9200
查看es所有节点:http://192.168.229.116:9200/_cat/nodes
2.2.2.Kibana
# 注意:http://192.168.139.10:9200 为es的http访问地址
docker run --name kibana \
--restart=always \
-e ELASTICSEARCH_HOSTS=http://192.168.139.10:9200 \
-p 5601:5601 \
-d kibana:7.10.1
访问:http://192.168.139.10:5601
点击 Explore on my own
2.3.安装Nginx
2.3.1.复制配置
启动一个Nginx实例,复制出配置
docker run -p 80:80 --name nginx -d nginx:1.18.0
将容器内的配置文件拷贝到当前目录
cd /mydata
mkdir nginx
docker container cp nginx:/etc/nginx .
停止并删除容器
docker stop nginx
docker rm nginx
修改文件名称
mv nginx conf
mkdir nginx
mv conf/ nginx/
2.3.2. 创建实例
docker run -p 80:80 --name nginx \
--restart=always \
-v /mydata/nginx/html:/usr/share/nginx/html \
-v /mydata/nginx/logs:/var/log/nginx \
-v /mydata/nginx/conf:/etc/nginx \
-d nginx:1.18.0
2.3.3.访问测试
cd /mydata/nginx/htmlvim index.html
<h1>com.atguigu.gmall<h1>
访问:http://192.168.139.10
3.文本分词
一个tokenizer(分词器)接收一个字符流,将之分割为独立的tokens(词元,通常是独立的单词),然后输出tokens流。
例如,whitespace tokenizer遇到空白字符时分割文本。它会将文本 Quick brown fox! 分割为
[Quick,brown,fox!] 。该tokenizer还负责记录各个 term(词条)的顺序或 position位置(用于
phrase短语和word proximity词近邻查询),以及term(词条)所代表的原始word(单词)的start(起始)和end(结束)的character offsets(字符偏移量),用于高亮显示搜索的内容。
ElasticSearch提供了很多内置的分词器,可以用来构建custom analyzers(自定义分词器)。
官方文档:https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-standard-tokenizer.html
3.1.安装ik分词器
GitHub:https://github.com/medcl/elasticsearch-analysis-ik
注意:ik分词器的版本一定要对应es版本安装
https://github.com/medcl/elasticsearch-analysis-ik/releases/tag/v7.10.1
# 进入 plugins 目录
cd /mydata/elasticsearch/plugins# 下载
wget https://github.com/medcl/elasticsearch-analysis-ik/releases/download/v7.10.1/elasticsearch-analysis-ik-7.10.1.zip# 解压
unzip elasticsearch-analysis-ik-7.10.1.zip -d ik# 删除zip文件
rm -rf *.zip# 修改ik文件夹权限
chmod -R 777 ik/# 确认是否安装好了分词器,进入容器bin目录
docker exec -it elasticsearch /bin/bash
cd bin
# 列出系统的分词器
elasticsearch-plugin list
ik# 重启容器
docker restart elasticsearch
3.2.测试分词器:
使用默认
GET _analyze
{"text":"我是中国人"
}
使用分词器 ik_smart
GET _analyze
{"analyzer":"ik_smart","text":"我是中国人"
}
另一个分词器 ik_max_word
GET _analyze
{"analyzer":"ik_max_word","text":"我是中国人"
}
3.3.自定义词库
配置远程词库,在nginx的 html 目录下新创建自定义词库
# 在html目录下创建es文件夹
mkdir es
# 创建新的分词并保存
vim participle.txt
尚硅谷
谷粒商城
修改 /mydata/elasticsearch/plugins/ik/config/ 中的 IKAnalyzer.cfg.xml
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE properties SYSTEM "http://java.sun.com/dtd/properties.dtd">
<properties><comment>IK Analyzer 扩展配置</comment><!-- 用户可以在这里配置自己的扩展字典 -><entry key="ext_dict"></entry><!-- 用户可以在这里配置自己的扩展停止词字典-><entry key="ext_stopwords"></entry><!-- 用户可以在这里配置远程扩展字典,这里使用的是nginx来访问 -><entry key="remote_ext_dict">http://192.168.139.10/es/participle.txt</entry><!-- 用户可以在这里配置远程扩展停止词字典-><!-- <entry key="remote_ext_stopwords">words_location</entry>->
</properties>
重启elasticsearch容器
docker restart elasticsearch
3.4.测试自定义词库
更新词库完成后,es只会对新增的数据用新词分词。历史数据是不会重新分词的,如果想要历史数据重新分词,需要执行:
POST my_index/_update_by_query?conflicts=proceed
4.创建检索服务模块
4.1ElasticSearch-Rest-Client
1)9300:TCP
spring-data-elasticsearch:transport-api.jar
- Spring Boot 版本不同,transport-api.jar不同,不能适配 elasticsearch 版本
- 官方7.x已经不建议使用,8以后就要废弃
2)9200:HTTP
- JestClient:非官方,更新慢
- RestTemplate:模拟发HTTP请求,ES很多操作需要自己封装,很麻烦
- HttpClient/OkHttp:模拟发HTTP请求,ES很多操作需要自己封装,很麻烦
- ElasticSearch-Rest-Client:官方RestClient,封装了ES操作,API层次分明,上手简单
官方文档:https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/index.html
4.2.创建检索服务模块
4.2.1.新建Module gmall-search
1)聚合模块
<modules><module>gmall-search</module>
</modules>
2)导入版本依赖
<properties><elasticsearch.version>7.10.1</elasticsearch.version>
</properties>
<dependency><groupId>com.atguigu.gmall</groupId><artifactId>gmall-common</artifactId><version>0.0.1-SNAPSHOT</version><exclusions><exclusion><groupId>com.baomidou</groupId><artifactId>mybatis-plus-boot-starter</artifactId></exclusion><exclusion><groupId>mysql</groupId><artifactId>mysql-connector-java</artifactId></exclusion></exclusions>
</dependency>
<dependency><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-high-level-client</artifactId><version>${elasticsearch.version}</version>
</dependency>
3)加入到Nacos注册中心和配置中心
application.yml
server:port: 20000
spring:application:name: gmall-searchcloud:nacos:discovery:server-addr: 192.168.139.10:8848namespace: 36854647-e68c-409b-9233-708a2d41702c
bootstrap.properties
spring.application.name=gmall-search
spring.cloud.nacos.config.server-addr=192.168.139.10:8848
spring.cloud.nacos.config.namespace=873d6587-5969-47dd-accb-a4d33a13817d
spring.cloud.nacos.config.group=dev
4)网关路由配置
spring:cloud:gateway:routes:- id: search_routeuri: lb://gmall-searchpredicates:- Path=/api/search/**filters:- RewritePath=/api/(?<segment>.*), /$\{segment}
4.2.3.配置ElasticSearch
编写ElasticSearch配置类 ElasticSearchConfig
package com.atguigu.gmall.search.config;import org.apache.http.HttpHost;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;/*** ElasticSearch 配置类 {@link ElasticSearchConfig}* * @author zhangwen* @email: 1466787185@qq.com*/
@Configuration
public class ElasticSearchConfig {public static final RequestOptions COMMON_OPTIONS;/*** https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-low-usage-requests.html#java-rest-low-usage-request-options*/static {RequestOptions.Builder builder = RequestOptions.DEFAULT.toBuilder();COMMON_OPTIONS = builder.build();}@Beanpublic RestHighLevelClient restHighLevelClient () {RestHighLevelClient restHighLevelClient = new RestHighLevelClient(RestClient.builder(// es集群模式下,可以指定多个 HttpHostnew HttpHost("192.168.139.10", 9200, "http")));return restHighLevelClient;}
}
# 5.商品上架到ES
-
上架的商品才可以在网站展示
-
上架的商品可以被检索
5.1.API
POST /product/spuinfo/{spuId}/up
5.2.后台接口实现
SpuInfoController
/*** 商品上架* @param spuId* @return*/@PostMapping("/{spuId}/up")public R spuUp(@PathVariable("spuId") Long spuId) {spuInfoService.up(spuId);return R.ok();}
SpuInfoServiceImpl
/*** 商品上架* @param spuId*/
@Override
public void up(Long spuId) {// 组装数据// 查询当前sku的所有可以被用来检索规格属性List<ProductAttrValueEntity> baseAttrs = productAttrValueService.listBaseAttrForSpu(spuId);List<Long> attrIds = baseAttrs.stream().map(ProductAttrValueEntity::getAttrId).collect(Collectors.toList());// 在指定的属性集合里面,查找出能够被检索的属性List<Long> searchAttrIds = attrService.selectSearchAttrIds(attrIds);Set<Long> idSet = new HashSet<>(searchAttrIds);List<SkuEsModel.Attrs> attrsList = baseAttrs.stream().filter(attr -> {return idSet.contains(attr.getAttrId());}).map(attr -> {SkuEsModel.Attrs attrs = new SkuEsModel.Attrs();BeanUtils.copyProperties(attr, attrs);return attrs;}).collect(Collectors.toList());// 查询出当前spuId对应的所有sku信息List<SkuInfoEntity> skus = skuInfoService.getSkusBySpuId(spuId);// 发送远程调用,库存系统查询是否有库存List<Long> skuIds =skus.stream().map(SkuInfoEntity::getSkuId).collect(Collectors.toList());Map<Long, Boolean> stockMap = null;try {List<SkuHasStockTO> tos = wareFeignService.getSkuHasStock(skuIds);stockMap = tos.stream().collect(Collectors.toMap(SkuHasStockTO::getSkuId, value ->value.getHasStock()));} catch (Exception e) {log.error("调用远程库存服务 gmall-ware 查询异常:{}", e);}// 封装每个sku的信息Map<Long, Boolean> finalStockMap = stockMap;List<SkuEsModel> skuEsModelList = skus.stream().map(skuInfoEntity -> {SkuEsModel skuEsModel = new SkuEsModel();BeanUtils.copyProperties(skuInfoEntity, skuEsModel);skuEsModel.setSkuPrice(skuInfoEntity.getPrice());skuEsModel.setSkuImg(skuInfoEntity.getSkuDefaultImg());// 远程调用异常,默认设置有库存if (finalStockMap == null) {skuEsModel.setHasStock(true);} else {skuEsModel.setHasStock(finalStockMap.get(skuInfoEntity.getSkuId()));}// TODO 热度评分(应该设计为后台可控的复杂操作)skuEsModel.setHotScore(0L);BrandEntity brandEntity = brandService.getById(skuEsModel.getBrandId());skuEsModel.setBrandName(brandEntity.getName());skuEsModel.setBrandImg(brandEntity.getLogo());CategoryEntity categoryEntity = categoryService.getById(skuEsModel.getCatalogId());skuEsModel.setCatalogName(categoryEntity.getName());// 设置检索属性skuEsModel.setAttrs(attrsList);return skuEsModel;}).collect(Collectors.toList());// 将数据发送给 ES保存R r = searchFeignService.productUp(skuEsModelList);if (r.getCode() == 0) {// 远程调用成功// 更新spu状态为已上架状态baseMapper.updateSpuStatus(spuId, ProductConstant.StatusEnum.SPU_UP.getCode());} else {// 远程调用失败// TODO 重试机制?接口幂等性?}}
5.3.远程接口
5.3.1.查询sku是否有库存
WareFeignService
package com.atguigu.gmall.product.feign;import com.atguigu.common.to.SkuHasStockTO;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;import java.util.List;/*** Ware 仓储服务远程接口 {@link WareFeignService}** @author zhangwen* @email: 1466787185@qq.com*/
@FeignClient("gmall-ware")
public interface WareFeignService {/*** 查询sku是否有库存* @param skuIds* @return*/@PostMapping("/ware/waresku/hasstock")List<SkuHasStockTO> getSkuHasStock(@RequestBody List<Long> skuIds);
}
远程接口实现
WareSkuController
/*** 查询sku是否有库存* @param skuIds* @return*/@PostMapping("/hasstock")public List<SkuHasStockTO> getSkuHasStock(@RequestBody List<Long> skuIds) {List<SkuHasStockTO> tos = wareSkuService.getSkuHasStock(skuIds);return tos;}
WareSkuServiceImpl
/*** 查询sku是否有库存* @param skuIds* @return*/
@Override
public List<SkuHasStockTO> getSkuHasStock(List<Long> skuIds) {List<SkuHasStockTO> tos = skuIds.stream().map(skuId -> {SkuHasStockTO to = new SkuHasStockTO();to.setSkuId(skuId);// 查询当前sku的库存量long count = baseMapper.getSkuStock(skuId);to.setHasStock(count > 0);return to;}).collect(Collectors.toList());return tos;
}
5.3.2.商品上架到ES库
SearchFeignService
package com.atguigu.gmall.product.feign;import com.atguigu.common.to.es.SkuEsModel;
import com.atguigu.common.utils.R;
import org.springframework.cloud.openfeign.FeignClient;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;import java.util.List;/*** Search 检索远程服务接口 {@link SearchFeignService}** @author zhangwen* @email: 1466787185@qq.com*/
@FeignClient("gmall-search")
public interface SearchFeignService {/*** 商品上架* @param skuEsModelList* @return*/@PostMapping("/search/save/product")R productUp(@RequestBody List<SkuEsModel> skuEsModelList);
}
远程接口实现
ElasticSaveController
package com.atguigu.gmall.search.controller;import com.atguigu.common.exception.BizCode;
import com.atguigu.common.to.es.SkuEsModel;
import com.atguigu.common.utils.R;
import com.atguigu.gmall.search.service.ProductServcie;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RequestBody;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RestController;import java.io.IOException;
import java.util.List;/*** ElasticSearch 存储 {@link ElasticSaveController}** @author zhangwen* @email: 1466787185@qq.com*/
@Slf4j
@RequestMapping("/search/save")
@RestController
public class ElasticSaveController {@Autowiredprivate ProductServcie productServcie;/*** 商品上架* @param skuEsModelList* @return*/@PostMapping("/product")public R productUp(@RequestBody List<SkuEsModel> skuEsModelList) {boolean flag = false;try {// 返回 false,说明商品上架没有异常flag = productServcie.productUp(skuEsModelList);} catch (IOException e) {log.error("ElasticSaveController商品上架错误:{}", e);return R.error(BizCode.PRODUCT_UP_EXCEPTION.getCode(), BizCode.PRODUCT_UP_EXCEPTION.getMessage());}if (!flag) {return R.ok();} else {return R.error(BizCode.PRODUCT_UP_EXCEPTION.getCode(), BizCode.PRODUCT_UP_EXCEPTION.getMessage());}}
}
ProductServiceImpl
package com.atguigu.gmall.search.service.impl;import com.atguigu.common.to.es.SkuEsModel;
import com.atguigu.gmall.search.config.ElasticSearchConfig;
import com.atguigu.gmall.search.service.ProductServcie;
import io.micrometer.core.instrument.util.JsonUtils;
import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.action.bulk.BulkItemResponse;
import org.elasticsearch.action.bulk.BulkRequest;
import org.elasticsearch.action.bulk.BulkResponse;
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;import java.io.IOException;
import java.util.Arrays;
import java.util.List;
import java.util.stream.Collectors;/*** 商品服务 {@link ProductServiceImpl}** @author zhangwen* @email: 1466787185@qq.com*/
@Slf4j
@Service
public class ProductServiceImpl implements ProductServcie {@Autowiredprivate RestHighLevelClient restHighLevelClient;/*** 商品上架* @param skuEsModelList*/@Overridepublic boolean productUp(List<SkuEsModel> skuEsModelList) throws IOException {// 给 es 中建立索引,并建立好映射关系// ES批量保存BulkRequest bulkRequest = new BulkRequest();skuEsModelList.forEach(skuEsModel -> {// 指定索引IndexRequest indexRequest = new IndexRequest(EsConstant.PRODUCT_INDEX);// 指定索引idindexRequest.id(skuEsModel.getSkuId().toString());// 转换为jsonString jsonString = JsonUtils.objectToJson(skuEsModel);// 设置数据indexRequest.source(jsonString, XContentType.JSON);bulkRequest.add(indexRequest);});BulkResponse bulk = restHighLevelClient.bulk(bulkRequest, ElasticSearchConfig.COMMON_OPTIONS);// TODO 处理批量保存错误// true 有错误,false 没有错误boolean b = bulk.hasFailures();// 记录日志List<String> collect = Arrays.stream(bulk.getItems()).map(BulkItemResponse::getId).collect(Collectors.toList());log.info("商品上架:{}", collect);return b;}
}
5.4.ES库查询
在Kibana中查询商品是否成功入库