起因
- 公司最近有个客户需要把2-3w台设备各类数据存放到我们平台，这么多设备带来的数据量一年下来单表大概会达到720w，这样会使得平台某些分页查询或相关业务效率变慢。所以想让客户自己去阿里云买服务器，但是客户不想管理，想丢在我们平台。那也没办法，客户是上帝，能做是能做，不过得加钱！！！ 哈哈哈
- 要帮客户存储数据，那得知道买多大的服务器合适，或者租多大服务器，一年得多少钱，这个得有一定的评估。所以就得生成一年的数据量了，进行存储和效率测试，系统后台的性能调优。
思路
- 要生成720w数据得有基础的2w台设备基础信息，所以现同普通方式生成了2w基础数据
- 720w = 2w * 360 每个表一天一笔数据，一年按照360算，一共720w
- 数据量有这么大，于是我 Ctrl+Alt+Del 打开任务管理器,看了看电脑的配置，评估一下CPU能扛得住怎样的摩擦，发现6核12处理器
- 那就给他来十个线程,每个线程处理2000基础数据生成72w，应该可以生成出来。
实践
1.新建定时器
@Slf4j @Configuration @EnableScheduling public class InsertBatchMeterArchiveTask implements SchedulingConfigurer { @Autowired private MeterArchiveService meterArchiveService; @Override public void configureTasks(ScheduledTaskRegistrar taskRegistrar) { taskRegistrar.addTriggerTask(() -> { try { meterArchiveService.insertBatchBaseInfo(); } catch (Exception e) { log.error("insertBatchBaseInfo meter error：{}", e.getMessage()); } }, triggerContext -> { String cron = "0 37 12 * * ? "; return new CronTrigger(cron).nextExecutionTime(triggerContext); }); } }
- 业务实现
public static List<Date> dateList = new ArrayList<>(360); @Override public void insertBatchBaseInfo() throws InterruptedException { for (int i = 0; i < 360; i++) { Calendar instance = Calendar.getInstance(); instance.setTime(CommonUtil.getCurrentDate()); instance.add(Calendar.DAY_OF_YEAR, -i); dateList.add(instance.getTime()); } // id 20062 --> 40061 System.out.println(System.currentTimeMillis()); List<MeterArchiveDO> meterArchiveDOS = meterArchiveMapper.selectList(new QueryWrapper<MeterArchiveDO>() .eq("area_id", 9).gt("id", 100)); int listSize = meterArchiveDOS.size(); int toIndex = 2000; //用map存起来新的分组后数据 Map<String, List<MeterArchiveDO>> map = new HashMap(); int keyToken = 0; for (int i = 0; i < meterArchiveDOS.size(); i += 2000) { //作用为toIndex最后没有2000条数据则剩余几条newList中就装几条 toIndex = (i + 2000 > listSize ? listSize - i : toIndex); List newList = meterArchiveDOS.subList(i, i + toIndex); map.put("keyName" + keyToken, newList); keyToken++; } System.out.println(System.currentTimeMillis()); // 处理生成数据 dealGenerateData(map); }
- 这里之前发生了线程不安全问题
原因是在线程实现里面用到了Calendar进行时间处理，导致时间错乱，因为Calendar是单例的，每个线程任务都使用了这个变化的时间。所以通过声明全局变量，把360天都生成出来，传到线程处理业务中，就不会有这种情况了。
- 处理生成数据
/** * 处理生成数据 * * @param map */ private void dealGenerateData(Map map) throws InterruptedException { //线程池10个线程 ExecutorService executorService = Executors.newFixedThreadPool(10); //第一批十个任务 List<StartAgent> agentsStart = new ArrayList(); for (int i = 0; i < 10; i++) { agentsStart.add(new StartAgent((List<MeterArchiveDO>) map.get("keyName" + i), meterDayFlowWaterMapper,dateList)); } List<List<StartAgent>> task = new ArrayList<>(); task.add(agentsStart); //记录任务执行时间 long t1 = System.currentTimeMillis(); CountDownLatch c; //循环任务组 for (List<StartAgent> startList : task) { //定义线程阻塞为10 c = new CountDownLatch(11); for (StartAgent agent : startList) { agent.setCountDownLatch(c); executorService.submit(agent); } c.await(); } executorService.shutdown(); }
- 具体线程内业务实现
@Data public class StartAgent implements Runnable { private CountDownLatch countDownLatch; private List<MeterArchiveDO> meterArchiveDOS; private MeterDayFlowWaterMapper meterDayFlowWaterMapper; private List<Date> dateList; public StartAgent(List<MeterArchiveDO> meterArchiveDOS, MeterDayFlowWaterMapper meterDayFlowWaterMapper, List<Date> dateList) { this.meterArchiveDOS = meterArchiveDOS; this.meterDayFlowWaterMapper = meterDayFlowWaterMapper; this.dateList = dateList; } @Override public void run() { try { System.out.println("开始启动节点：" + Thread.currentThread().getName()); MeterDayFlowWaterDO meterDayFlowWaterDO = new MeterDayFlowWaterDO(); for (MeterArchiveDO meterArchiveDO : meterArchiveDOS) { for (int i = 0; i < 360; i++) { BeanUtils.copyProperties(meterArchiveDO, meterDayFlowWaterDO); meterDayFlowWaterDO.setMeterReadTime(dateList.get(i)); meterDayFlowWaterDO.setMeterSaveTime(dateList.get(i)); meterDayFlowWaterDO.setMeterPositiveFlow("6.66"); meterDayFlowWaterDO.setMeterReverseFlow("0.05"); meterDayFlowWaterDO.setMeterIncrementFlow("1.38"); meterDayFlowWaterDO.setSync(1); meterDayFlowWaterDO.setRemainingAmount("100"); meterDayFlowWaterMapper.insert(meterDayFlowWaterDO); } } System.out.println(Thread.currentThread().getName() + "执行完毕"); } catch (Exception e) { e.printStackTrace(); } finally { //注意一定要在finally调用countDown，否则产生异常导致没调用到countDown造成程序死锁 countDownLatch.countDown(); } } public void setCountDownLatch(CountDownLatch countDownLatch) { this.countDownLatch = countDownLatch; } }
测试
欢迎大佬指点探讨！！！
