Sam Rogers

Total Home Score是一个产品，旨在帮助潜在的购房者和租房者在做出决定之前了解居住在特定房产中的感觉.

In order to scale this product and calculate scores for millions of properties, 我构建了一个大规模的数据管道来执行复杂的地理空间计算和聚合.

该管道包括使用Spark和EMR对道路交通数据进行计算，并生成驾驶员在特定道路长度上的典型驾驶方式的汇总. 然后将地址加载到Dask中，并跨数千个分区进行计算，以确定在给定地址的特定半径内存在多少“危险”道路.

Additionally, 我开发了一个管道来处理飞机位置数据(超过100亿个点)，并确定在特定属性处预期的飞机噪音水平.

一个基于python的应用程序，运行在AWS Lambda和Redis上，支持毫秒级的汇总产品使用数据的记录检索.

While working at a marketing technology provider, our product team wanted the ability to surface product usage data to our customers. 客户不希望等待在数据仓库中运行查询才能返回结果. 我设计的解决方案是运行一组预定义的聚合，并将它们放入缓存中，以便客户几乎可以立即接收和可视化结果.

Not only was this more rapid than running aggregations on demand, but it was also more cost-effective, instead of running thousands of aggregation queries in Snowflake per day, only one query needed to run to generate the output data and place it into our cache.

气流是一个开源工具，设计用于编排、调度和执行ETL作业. The tool was originally designed to run these processes within it as well. However, 由于气流执行实际的处理而不仅仅是在资源之间进行协调，可能会出现许多问题. For example, all dependencies need to be installed on the same instance, memory leaks can bring down the entire cluster, 如果客户数据被集群接触，可能需要额外的安全措施.

我的解决方案是让气流仅仅作为一个容器执行工具，而不是在应用程序中进行实际的处理. In addition, 要执行的作业和要传递给它的参数的所有配置仍然包含在气流代码中, but executed elsewhere. This makes for a simple interface for other data engineers to implement new pipelines.

For example, if an engineer has a file in S3 that they want to be loaded to a database on a schedule, they simply utilize the loader operator class that already exists in the Airflow repository. When executed, 该类提供一个在AWS Fargate中运行的任务，该任务使用传递给它的配置执行一个进程.